<commit_before># Highway traffic analytics:

This repository consists of components to analyse publicly available traffic data.
Three components can be deployed independently.

* [Ingestion component](#ingestion-component)  
    Traffic data are downloaded and stored in a MongoDB database
  
* [Monitoring component](#monitoring-component)  
    Monitoring data are collected by Prometheus and visualized with Grafana

* [Analytics component](#analytics-component)  
    Componente to analyste the data with Python.
  
## Start all components
```
kubectl apply -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/ingestion/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/monitoring/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/analytics-workbench/k8s-jupyter-deployment.yaml
```
## Delete all components
```
kubectl delete -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/ingestion/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/monitoring/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/analytics-workbench/k8s-jupyter-deployment.yaml
```

## Ingestion component<commit_msg>Add data source node<commit_after># Highway traffic analytics:

This repository consists of components to analyse publicly available traffic data. All data are provided by [Landesbetrieb Straßenbau NRW, Verkehrszentrale](https://mcloud.de/web/guest/suche/-/results/suche/relevance/NRW/0/detail/_mcloudde_mdmgeschwindigkeitsdatennrw)  
Three components can be deployed independently.

* [Ingestion component](#ingestion-component)  
    Traffic data are downloaded and stored in a MongoDB database
  
* [Monitoring component](#monitoring-component)  
    Monitoring data are collected by Prometheus and visualized with Grafana

* [Analytics component](#analytics-component)  
    Componente to analyste the data with Python.
  
## Start all components
```
kubectl apply -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/ingestion/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/monitoring/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/analytics-workbench/k8s-jupyter-deployment.yaml
```
## Delete all components
```
kubectl delete -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/ingestion/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/monitoring/k8s-deployment.yaml -f https://raw.githubusercontent.com/scray/scray/feature/ingestion-examples/scray-examples/persistent-traffic-data/analytics-workbench/k8s-jupyter-deployment.yaml
```

## Ingestion component
{'fraction_non_alphanumeric': 0.10970081595648232, 'fraction_numerical': 0.003928679359323058, 'mean_word_length': 5.515748031496063, 'pattern_counts': {'":': 0, '<': 3, '<?xml version=': 0, '>': 3, 'https://': 13, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'dir': 'git-commits-cleaned', 'id': '6975549', 'n_tokens_mistral': 1079, 'n_tokens_neox': 1016, 'n_words': 167}