<commit_before>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "nmea.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyM8YZccZVyxGZlm38W0z8m5"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XIdYAiaesqUI"
      },
      "outputs": [],
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#NMEA files processing"
      ],
      "metadata": {
        "id": "O2BTZIADsrRv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "NMEA stands for National Marine Electronics Association. Today NMEA is a standard data format supported by all GPS manufacturers. Take a look at into an NMEA [file](https://github.com/OSGeoLabBp/tutorials/blob/master/english/data_processing/lessons/code/nmea1.txt)\n",
        "You can collect your own data in NMEA file having a smart phone and a wide range of applications, like AndroSensor.\n",
        "\n",
        "In this lesson we read an NMEA file, get a statistics about the different sentences, visualize the trajectory and some further data.\n",
        "\n",
        "First off, you need to upload the file into your google drive to be able to read it in CoLab. Please click to the 'folder' icon on the left and upload the [file](https://https://github.com/OSGeoLabBp/tutorials/blob/master/english/data_processing/lessons/code/nmea1.txt)\n",
        "\n",
        "Read the file line by line and count the lines."
      ],
      "metadata": {
        "id": "b_zfGFM5s6i8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open('nmea1.txt') as fp:\n",
        "  i = 0\n",
        "  for line in fp:\n",
        "    i += 1\n",
        "  print(i)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vM2mpMI9uFM4",
        "outputId": "c6e0447e-8454-4203-eb55-cd6e579def2d"
      },
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "1712\n"
          ]
        }
      ]
    }
  ]
}<commit_msg>Created using Colaboratory<commit_after>{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "nmea.ipynb",
      "provenance": [],
      "authorship_tag": "ABX9TyPUZMrFhKlg6O7B7NpL305w"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "XIdYAiaesqUI"
      },
      "outputs": [],
      "source": [
        ""
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "#NMEA files processing"
      ],
      "metadata": {
        "id": "O2BTZIADsrRv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "NMEA stands for National Marine Electronics Association. Today NMEA is a standard data format supported by all GPS manufacturers. Take a look at into an NMEA [file](https://github.com/OSGeoLabBp/tutorials/blob/master/english/data_processing/lessons/code/nmea1.txt)\n",
        "You can collect your own data in NMEA file having a smart phone and a wide range of applications, like AndroSensor.\n",
        "\n",
        "In this lesson we read an NMEA file, get a statistics about the different sentences, visualize the trajectory and some further data.\n",
        "\n",
        "First off, you need to upload the file into your google drive to be able to read it in CoLab. Please click to the 'folder' icon on the left side and upload the [file](https://https://github.com/OSGeoLabBp/tutorials/blob/master/english/data_processing/lessons/code/nmea1.txt)\n",
        "\n",
        "Read the file line by line and count the lines."
      ],
      "metadata": {
        "id": "b_zfGFM5s6i8"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "with open('nmea1.txt') as fp:\n",
        "  i = 0\n",
        "  for line in fp:\n",
        "    i += 1\n",
        "fp.close()\n",
        "print(i)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "vM2mpMI9uFM4",
        "outputId": "5a4b29a5-931d-4538-891f-75e1c20a5103"
      },
      "execution_count": 22,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "63\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "A typical NMEA file contains several types of sentences. Count the various types using the python dictionary data structure. Sentence type is given by the 3..6 characters of each line, such as GGA, GSA, GLL ... and so. If you are less familiar with the NMEA sentences, feel free to google it up."
      ],
      "metadata": {
        "id": "-6hPvoFrP8Kq"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "fi = open('nmea1.txt', 'r') # input file\n",
        "for line in fi:\n",
        "  token = line[3:6]\n",
        "  if token not in tokens:\n",
        "    tokens[token] = 0   # create new item in dictionary\n",
        "  tokens[token] += 1\n",
        "fi.close()"
      ],
      "metadata": {
        "id": "dR43-s_QRF53"
      },
      "execution_count": 31,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Print out the dictionary. "
      ],
      "metadata": {
        "id": "WPN699uWSguK"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "print(tokens)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "gmQO7nfCT4Iy",
        "outputId": "90e97c9c-5bce-424e-dc6e-d46ab687f47e"
      },
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "{'GGA': 39, 'GLL': 55, 'RMC': 35}\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "Sort the items according to their occurence."
      ],
      "metadata": {
        "id": "YI-PpgGhUCIU"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "for t in sorted(tokens.items(), key=lambda x: x[1], reverse=True):\n",
        "    print(\"{}: {}\".format(t[0], t[1]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "Ni1EQc2gSos7",
        "outputId": "61d60f97-059a-4ced-e4d5-0e9826f03331"
      },
      "execution_count": 35,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "GLL: 55\n",
            "GGA: 39\n",
            "RMC: 35\n"
          ]
        }
      ]
    }
  ]
}
{'fraction_non_alphanumeric': 0.15905292479108635, 'fraction_numerical': 0.02743732590529248, 'mean_word_length': 1.7629857637552906, 'pattern_counts': {'":': 120, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 10, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '5708212', 'n_tokens_mistral': 2630, 'n_tokens_neox': 2493, 'n_words': 572}