<jupyter_start><jupyter_text>Utilisation des scriptsCe notebook vise à montrer l'utilisation des scripts du projet afin d'effectuer une augmentation de données sur une base de données.  Installation du projet `bfc` et accès aux donnéesLes login et mot de passe sont nécessaires pour accéder à des repos privés à partir de Google/Colab. Cela ne serait pas le cas en accès public.<jupyter_code>import os
from getpass import getpass
%%shell
pip uninstall opencv-python-headless 
pip install opencv-python-headless==4.1.2.30
user = getpass('GitHub user')
password = getpass('GitHub password')
os.environ['GITHUB_AUTH'] = user + ':' + password
%%shell

git clone https://$GITHUB_AUTH@github.com/tibocour/IA.git<jupyter_output>Cloning into 'IA'...
remote: Enumerating objects: 223, done.[K
remote: Counting objects: 100% (223/223), done.[K
remote: Compressing objects: 100% (171/171), done.[K
remote: Total 223 (delta 124), reused 112 (delta 44), pack-reused 0[K
Receiving objects: 100% (223/223), 102.31 MiB | 31.60 MiB/s, done.
Resolving deltas: 100% (124/124), done.<jupyter_text>Installation des dépendances<jupyter_code>%%shell

pip install -r IA/requirements.txt<jupyter_output>Requirement already satisfied: imgaug in /usr/local/lib/python3.7/dist-packages (from -r IA/requirements.txt (line 1)) (0.2.9)
Collecting pascal-voc-writer
  Downloading pascal_voc_writer-0.1.4-py2.py3-none-any.whl (4.0 kB)
Collecting tflite-model-maker-nightly
  Downloading tflite_model_maker_nightly-0.3.5.dev202202140605-py3-none-any.whl (617 kB)
[K     |████████████████████████████████| 617 kB 15.7 MB/s 
[?25hRequirement already satisfied: pycocotools in /usr/local/lib/python3.7/dist-packages (from -r IA/requirements.txt (line 4)) (2.0.4)
Requirement already satisfied: opencv-python in /usr/local/lib/python3.7/dist-packages (from -r IA/requirements.txt (line 5)) (4.1.2.30)
Requirement already satisfied: numpy>=1.15.0 in /usr/local/lib/python3.7/dist-packages (from imgaug->-r IA/requirements.txt (line 1)) (1.19.5)
Requirement already satisfied: Pillow in /usr/local/lib/python3.7/dist-packages (from imgaug->-r IA/requirements.txt (line 1)) (7.1.2)
Requirement already satisfied: scip[...]<jupyter_text>Augmentation de donnéesTéléchargement de la base de données à partir de la release GitHub (optionnel si utilisation des bases du répertoire `IA/data`)<jupyter_code>%%shell

wget https://github.com/tibocour/IA/releases/download/v5/project-5-at-2022-02-08-15-39-1926817e.zip
%%shell

ls IA/data<jupyter_output>1megot.mp4			      megots150images.zip
2megot.mp4			      megotsfeuilles.zip
bfc-labels.txt			      train_megots150images.zip
dechetsmilleimages.tflite	      valid_base.de.donnee.1000.images.zip
efficientdet-lite-bfc_edgetpu.tflite  valid_megots150images.zip
efficientdet-lite-bfc.tflite<jupyter_text>Pour choisir la base de données à augmenter<jupyter_code># database = os.path.join("IA", "data", "megots150images.zip")
# or
database = "project-5-at-2022-02-08-15-39-1926817e.zip"
os.environ.update({"database": database})<jupyter_output><empty_output><jupyter_text>Mise en cohérence de la base issue de `label-studio` et augmentation de données<jupyter_code>%%shell

rm -rf train_*.zip valid_*.zip

python IA/python/label_studio_voc_converter.py --zip $database<jupyter_output>created zip temporary directory: /tmp/tmpaos27xvq
extract zip file project-5-at-2022-02-08-15-39-1926817e.zip in /tmp/tmpaos27xvq
Traceback (most recent call last):
  File "IA/python/label_studio_voc_converter.py", line 171, in <module>
    raise FileExistsError(path)
FileExistsError: /tmp/tmpaos27xvq/Annotations<jupyter_text>Pour Télécharger les fichiers générés (peut être long...)<jupyter_code>filename = os.path.basename(database)
from google.colab import files
files.download(f"train_{filename}")
files.download(f"valid_{filename}")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.12833808659579984, 'fraction_numerical': 0.05859476276899144, 'mean_word_length': 5.334975369458128, 'pattern_counts': {'":': 1, '<': 22, '<?xml version=': 0, '>': 25, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '513764', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1598, 'n_tokens_neox': 1427, 'n_words': 323, 'path': 'notebooks/data_augmentation.ipynb', 'repo_name': 'tibocour/IA', 'size': '3857'}