<jupyter_start><jupyter_text>OpenStreetMap parser definition We use [`pyosmium`](https://github.com/osmcode/pyosmium), a open tool designed for parsing OSM objects within a Python environment (doc [here](http://docs.osmcode.org/pyosmium/latest/)).<jupyter_code>import pandas as pd
import osmium as osm<jupyter_output><empty_output><jupyter_text>OSM history parsing First we define a class that inherits from a Pyosmium handler.<jupyter_code>class TimelineHandler(osm.SimpleHandler):
    def __init__(self):
        osm.SimpleHandler.__init__(self)
        self.timeline = [] # Dictionnary of OSM elements
        
    def add_element(self, elem, elem_type):
        self.timeline.append([elem_type,
                              elem.id,
                              elem.version,
                              elem.visible,
                              pd.Timestamp(elem.timestamp),
                              elem.uid,
                              elem.changeset])
        
    def node(self, n):
        self.add_element(n, "node")

    def way(self, w):
        self.add_element(w, "way")
                                     
    def relation(self,r):
        self.add_element(r, "relation")<jupyter_output><empty_output><jupyter_text>Then we design an instance of this class, and call the `apply_file` function by passing the name of an OSM input file as a parameter. If this file is in the `osh.pbf` (respectively `osm.pbf`) format, we can have all local OSM history (resp. an up-to-date image of the API).<jupyter_code>tlhandler = TimelineHandler()
tlhandler.apply_file("../src/data/raw/bordeaux-metropole.osh.pbf")<jupyter_output><empty_output><jupyter_text>As we get the OSM objects under a file format, we can easily transform it into a `pandas` DataFrame, and manage it how we like.<jupyter_code>colnames = ['elem', 'id', 'version', 'visible', 'ts', 'uid', 'chgset']
elements = pd.DataFrame(tlhandler.timeline, columns=colnames)
elements = elements.sort_values(by=['elem', 'id', 'version'])
elements.head(10)
elements.sample(10)<jupyter_output><empty_output><jupyter_text>If desired, this structure may even be stored on the file system!<jupyter_code>elements.to_csv("bordeaux-metropole.csv", date_format='%Y-%m-%d')<jupyter_output><empty_output><jupyter_text>OSM tag set parsing As previously we design a parsing class, the only difference being in the appending function, in which we consider tag keys and values.<jupyter_code>class TagGenomeHandler(osm.SimpleHandler):
    
    def __init__(self):
        osm.SimpleHandler.__init__(self)
        self.taggenome = []

    def tag_inventory(self, elem, elem_type):
        for tag in elem.tags:
            self.taggenome.append([elem_type,
                                   elem.id,
                                   elem.version,
                                   tag.k,
                                   tag.v])

    def node(self, n):
        self.tag_inventory(n, "node")

    def way(self, w):
        self.tag_inventory(w, "way")

    def relation(self, r):
        self.tag_inventory(r, "relation")
taghandler = TagGenomeHandler()
taghandler.apply_file("/home/rde/data/osm-history/raw/bordeaux-metropole.osh.pbf")
tag_genome = pd.DataFrame(taghandler.taggenome)
tag_genome.columns = ['elem', 'id', 'version', 'tagkey', 'tagvalue']
tag_genome = tag_genome.sort_values(['elem', 'id', 'version'], ascending=False)
tag_genome.head(10)
tag_genome.sample(10)
tag_genome.to_csv("bordeaux-metropole-tags.csv")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11414745231995445, 'fraction_numerical': 0.002277255906632508, 'mean_word_length': 2.9262569832402234, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '466523', 'license': 'MIT', 'n_tokens_mistral': 1116, 'n_tokens_neox': 1076, 'n_words': 254, 'path': 'demo/osm_parsing.ipynb', 'repo_name': 'Oslandia/osm-data-classification', 'size': '3513'}