<jupyter_start><jupyter_text>Welcome to the Datenguide Python PackageWithin this notebook the functionality of the package will be explained and demonstrated with examples. Topics- Import- get region IDs- get statstic IDs- get the data    - for single regions    - for multiple regions  1. Import **Import the helper functions 'get_all_regions' and 'get_statistics'****Import the module Query for the main functionality**<jupyter_code># ONLY FOR TESTING LOCAL PACKAGE
# %cd ..

from datenguidepy.query_helper import get_all_regions, get_statistics
from datenguidepy import Query<jupyter_output>C:\Users\Alexandra\Documents\GitHub\datenguide-python<jupyter_text>**Import pandas and matplotlib for the usual display of data as tables and graphs**<jupyter_code>import pandas as pd
import matplotlib
%matplotlib inline

pd.set_option('display.max_colwidth', 150)<jupyter_output><empty_output><jupyter_text>2. Get Region IDs How to get the ID of the region I want to query Regionalstatistik - the database behind Datenguide - has data for differently granular levels of Germany. nuts:        1 – Bundesländer        2 – Regierungsbezirke / statistische Regionen        3 – Kreise / kreisfreie Städte.           lau:        1 - Verwaltungsgemeinschaften        2 - Gemeinden.the function `get_all_regions()` returns all IDs from all levels.<jupyter_code># get_all_regions returns all ids
get_all_regions()<jupyter_output><empty_output><jupyter_text>To get a specific ID, use the common pandas function `query()`<jupyter_code># e.g. get all "Bundesländer
get_all_regions().query("level == 'nuts1'")
# e.g. get the ID of Havelland
get_all_regions().query("name =='Havelland'")<jupyter_output><empty_output><jupyter_text>3. Get statistic IDs How to find statistics<jupyter_code># get all statistics
get_statistics()<jupyter_output><empty_output><jupyter_text>If you already know the statsitic ID you are looking for - perfect. Otherwise you can use the pandas `query()` function so search e.g. for specific terms.<jupyter_code># find out the name of the desired statistic about birth
get_statistics().query('long_description.str.contains("Statistik der Geburten")', engine='python')<jupyter_output><empty_output><jupyter_text>4. get the data The top level element is the Query. For each query fields can be added (usually statistics / measures) that you want to get information on. A Query can either be done on a single region, or on multiple regions (e.g. all Bundesländer).  Single RegionIf I want information - e.g. all births for the past years in Berlin:<jupyter_code># create a query for the region 11
query = Query.region('11')
# add a field (the statstic) to the query
field_births = query.add_field('BEV001')
# get the data of this query
query.results().head()<jupyter_output><empty_output><jupyter_text>To get the short description in the result data frame instead of the cryptic ID (e.g. "<NAME>" instead of BEV001) set the argument "verbose_statsitics"=True in the resutls:<jupyter_code>query.results(verbose_statistics =True).head()<jupyter_output><empty_output><jupyter_text>Now we only get the information about the count of births per year and the source of the data (year, value and source are default fields).But there is more information in the statistic that we can get information on.Let's look at the meta data of the statstic:<jupyter_code># get information on the field
field_births.get_info()<jupyter_output>[1mkind:[0m
OBJECT

[1mdescription:[0m
<NAME>

[1marguments:[0m
[4myear[0m: LIST of type SCALAR(Int)

[4mstatistics[0m: LIST of type ENUM(BEV001Statistics)
enum values:
R12612: Statistik der Geburten

[4mALTMT1[0m: LIST of type ENUM(ALTMT1)
enum values:
ALT000B20: unter 20 Jahre
ALT020B25: 20 bis unter 25 Jahre
ALT025B30: 25 bis unter 30 Jahre
ALT030B35: 30 bis unter 35 Jahre
ALT035B40: 35 bis unter 40 Jahre
ALT040UM: 40 Jahre und mehr
GESAMT: Gesamt

[4mGES[0m: LIST of type ENUM(GES)
enum values:
GESM: männlich
GESW: weiblich
GESAMT: Gesamt

[4mNATEL1[0m: LIST of type ENUM(NATEL1)
enum values:
NATAAO: Mutter und Vater Ausländer,  ohne Angabe der Nationalität
NATDDDO: Mutter und Vater Deutsche, Mutter Deutsche und Vater o.Angabe der Nat.
NATEETA: ein Elternteil Ausländer
GESAMT: Gesamt

[4mNAT[0m: LIST of type ENUM(NAT)
enum values:
NATA: Ausländer(innen)
NATD: Deutsche
GESAMT: Gesamt

[4mLEGIT2[0m: LIST of type ENUM(LEGIT2)
enum values:
LEGIT01A: Eltern mi[...]<jupyter_text>The arguments tell us what we can use for filtering (e.g. only data on baby girls (female)).The fields tell us what more information can be displayed in our results.<jupyter_code># add filter
field_births.add_args({'GES': 'GESW'})
# now only about half the amount of births are returned as only the results for female babies are queried
query.results().head()
# add the field NAT (nationality) to the results
field_births.add_field('NAT')<jupyter_output><empty_output><jupyter_text>**CAREFUL**: The information for the fields (e.g. nationality) is by default returned as a total amount. Therefore - if no argument "NAT" is specified in addition to the field, then only "None" will be displayed.In order to get information on all possible values, the argument "ALL" needs to be added:(the rows with value "None" are the aggregated values of all options)<jupyter_code>field_births.add_args({'NAT': 'ALL'})
query.results().head()<jupyter_output><empty_output><jupyter_text>To display the short description of the enum values instead of the cryptic IDs (e.g. Ausländer(innen) instead of NATA), set the argument "verbose_enums = True" on the results:<jupyter_code>query.results(verbose_enums=True).head()<jupyter_output><empty_output><jupyter_text>Multiple Regions To display data for multiple single regions, a list with region IDs can be used:<jupyter_code>query_multiple = Query.region(['01', '02'])
query_multiple.add_field('BEV001')
query_multiple.results().sort_values('year').head()<jupyter_output><empty_output><jupyter_text>To display data for e.g. all 'Bundesländer' or for all regions within a Bundesland, you can use the function `all_regions()`:- specify nuts level- specify lau level- specify parent ID (Careful: not only the regions for the next lower level will be returned, but all levels - e.g. if you specify a parent on nuts level 1 then the "children" on nuts 2 but also the "grandchildren" on nuts 3, lau 1 and lau 2 will be returned)<jupyter_code># get data for all Bundesländer
query_all = Query.all_regions(nuts=1)
query_all.add_field('BEV001')
query_all.results().sort_values('year').head(12)
# get data for all regions within Brandenburg
query_all = Query.all_regions(parent='12')
query_all.add_field('BEV001')
query_all.results().head()
# get data for all nuts 3 regions within Brandenburg
query_all = Query.all_regions(parent='12', nuts=3)
query_all.add_field('BEV001')
query_all.results().sort_values('year').head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09109341057204924, 'fraction_numerical': 0.019840695148443156, 'mean_word_length': 5.851190476190476, 'pattern_counts': {'":': 0, '<': 57, '<?xml version=': 0, '>': 57, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '49', 'license': 'MIT', 'n_tokens_mistral': 2350, 'n_tokens_neox': 2199, 'n_words': 796, 'path': 'use_case/01_intro_tutorial.ipynb', 'repo_name': 'elekt/datenguide-python', 'size': '6923'}