<jupyter_start><jupyter_text>We need to give OSM an area to consider for the query. We do this by creating a bounding box around the area we are interested in; in this case, Portland<jupyter_code># save the bounding box--PDX
bbox = {'xmax': -122.2575, 'ymax': 45.7859, 'xmin': -123.1095, 'ymin': 45.1453}
nw = shp.geometry.Point(bbox['xmin'], bbox['ymax'])
sw = shp.geometry.Point(bbox['xmin'], bbox['ymin'])
se = shp.geometry.Point(bbox['xmax'], bbox['ymin'])
ne = shp.geometry.Point(bbox['xmax'], bbox['ymax'])
pd.set_option('max_columns', None)<jupyter_output><empty_output><jupyter_text>OSM has a lot of data and we don't want to be overwhelmed. We can use the taginfo api to get useful meta data from OSM<jupyter_code># get the most frequently used amenity tag in OSM from the taginfo api
r = req.get('https://taginfo.openstreetmap.org/api/4/tags/popular?query=amenity')
r.status_code<jupyter_output><empty_output><jupyter_text>Convert the json to a dataframe for easy use<jupyter_code>tags = pd.concat(pd.DataFrame.from_dict(d, orient='index').transpose() for d in r.json()['data'])
tags.sort_values(by='count_nodes', ascending=False)
# the loader for some reason can't take a list of tags so we will have to iterate and concat
tag_pairs = tags.apply(lambda row: row['key'] + '=' + row['value'], axis=1).tolist()
tag_pairs
# we are only interested in some of these; let's filter now to spare the api
keepers = ['restaurant', 'cafe', 'pub', 'bar', 'theater', 'college', 'cinema', 'arts_centre', 'nightclub', 'university']
new_ls = []
for keeper in keepers:
    new_ls.append('amenity={}'.format(keeper))
new_ls<jupyter_output><empty_output><jupyter_text>Generally, the OSM api is not meant to be used over such a large area. In thise case since we are interested in such a small subset of data it will do. Still, the sleep time (t) might need adjusting in order to get the data back. Too long and you risk timing out, too short you might get throttled<jupyter_code>def pdx_amenities(pairs):
    t = 4
    dfs = []
    for pair in pairs:
        try:
            df = osm.node_query(bbox['ymin'], bbox['xmin'], bbox['ymax'], bbox['xmax'], tags=pair).reset_index()
            keep_cols = ['id', 'lat', 'lon']
            if 'name'  in df.columns.tolist():
                keep_cols.append('name')
            df = df[keep_cols]
            df['amenity'] = pair.split('=')[1]
            dfs.append(df)
        except RuntimeError as e:
            print(pair + ': ')
            print(e)
        # try to avoid auto throttle
        sleep(t)
    return pd.concat(dfs, ignore_index=True)
test = pdx_amenities(new_ls)
test.head(20)
test.groupby('amenity').size().sort_values(ascending=False)
test.id.size
# create some useful grouping
test['alias'] = test.apply(lambda row: 'food_drink' if row['amenity'] in ('restaurant', 'pub', 'bar') else 'entertainment' if row['amenity'] in ('cinema', 'arts_centre', 'nightclub', 'theater') else 'college' if row['amenity'] in ('college', 'university') else row['amenity'], axis=1)
test<jupyter_output><empty_output><jupyter_text>Write the data to a csv for later use<jupyter_code>test.to_csv('./data/osm.csv', index=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.14375195740682745, 'fraction_numerical': 0.010335108048856874, 'mean_word_length': 4.487972508591065, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '43471', 'license': 'MIT', 'n_tokens_mistral': 1124, 'n_tokens_neox': 1079, 'n_words': 325, 'path': 'osm.ipynb', 'repo_name': 'tadhgfendt/spatial_econometrics', 'size': '3193'}