<jupyter_start><jupyter_text>Custom Parsers<jupyter_code>from mantisshrimp.imports import *
from mantisshrimp import *
import pandas as pd<jupyter_output>/home/lgvaz/anaconda3/envs/light2/lib/python3.7/site-packages/graphql/type/directives.py:55: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3,and in 3.9 it will stop working
  assert isinstance(locations, collections.Iterable), 'Must provide locations for directive.'
/home/lgvaz/anaconda3/envs/light2/lib/python3.7/site-packages/trains/backend_interface/metrics/events.py:27: DeprecationWarning: The usage of `cmp` is deprecated and will be removed on or after 2021-06-01.  Please use `eq` and `order` instead.
  @attr.attrs(cmp=False, slots=True)<jupyter_text>Understand the data format In this task we were given a `.csv` file with annotations, let's take a look at that.     **Note:**  Replace `source` with your own path for the dataset directory.<jupyter_code>source = Path("/home/lgvaz/.data/wheat")
df = pd.read_csv(source / "train.csv")
df.head()<jupyter_output><empty_output><jupyter_text>At first glance, we can make the following assumptions:  * Multiple rows with the same object_id, width, height  * A different bbox for each row  * source doesn't seem relevant right now   Once we know what our data provides we can create our custom `Parser`.    Create the Parser When creating a `Parser` we inherit from smaller building blocks that provides the functionallity we want:  * `DefaultImageInfoParser`: Will parse standard fields for image information, e.g. `filepath`, `height`, `width`  * `FasterRCNNParser`: Since we only need to predict bboxes we will use a `FasterRCNN` model, this will parse all the requirements for using such a model.   We can also specify exactly what fields we would like to parse, in fact, the parsers we are currently using are just helper classes that groups a collection of individual parsers.  We are going to see how to use individual parsers in a future tutorial.     **Note:**If you are using an IDE there is a little bit of magic than can happen. Once you created defined your class you can right click on it and select the option _"implement abstract methods"_, this will automatically populate your class with all the methods you need to override. If you are using a notebook, or your IDE does not support that, check the documentation to know what methods you should override.     **Important:**      Be sure to return the correct type on all overriden methods!<jupyter_code>class WheatParser(DefaultImageInfoParser, FasterRCNNParser):
    def __init__(self, df, source):
        self.df = df
        self.source = source
        self.imageid_map = IDMap()

    def __iter__(self):
        yield from self.df.itertuples()

    def __len__(self):
        return len(self.df)

    def imageid(self, o) -> int:
        return self.imageid_map[o.image_id]

    def filepath(self, o) -> Union[str, Path]:
        return self.source / f"{o.image_id}.jpg"

    def height(self, o) -> int:
        return o.height

    def width(self, o) -> int:
        return o.width

    def labels(self, o) -> List[int]:
        return [1]

    def bboxes(self, o) -> List[BBox]:
        return [BBox.from_xywh(*np.fromstring(o.bbox[1:-1], sep=","))]<jupyter_output><empty_output><jupyter_text>Defining the `__init__` is completely up to you, normally we have to pass our data (the `df` in our case) and the folder where our images are contained (`source` in our case). We then override `__iter__`, telling our parser how to iterate over our data. In our case we call `df.itertuples` to iterate over all `df` rows.`__len__` is not obligatory but will help visualizing the progress when parsing. And finally we override all the other methods, they all receive a single argument `o`, which is the object returned by `__iter__`. Now we just need to decide how to split our data and `Parser.parse`!<jupyter_code>data_splitter = RandomSplitter([.8, .2])
parser = WheatParser(df, source / "train")
train_rs, valid_rs = parser.parse(data_splitter)<jupyter_output><empty_output><jupyter_text>Let's take a look at one record.<jupyter_code>show_record(train_rs[0], label=False)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08862837045720985, 'fraction_numerical': 0.007033997655334115, 'mean_word_length': 4.366037735849057, 'pattern_counts': {'":': 0, '<': 20, '<?xml version=': 0, '>': 26, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '365644', 'license': 'Apache-2.0', 'n_tokens_mistral': 1283, 'n_tokens_neox': 1234, 'n_words': 506, 'path': 'tutorials/mantisshrimp_data/custom_parsers.ipynb', 'repo_name': 'ramaneswaran/mantisshrimp', 'size': '4265'}