<jupyter_start><jupyter_text>Base dos dados Base dos Dados is a Brazilian project of consolidation of datasets in a common repository with easy to follow codes. Download *clean, integrated and updated* datasets in an easy way through SQL, Python, R or CLI (Stata in development). With Base dos Dados you have freedom to:- download whole tables- download tables partially- cross datasets and download final productThis Jupyter Notebook aims to help Python coders in main funcionalities of the platform. If you just heard about Base Dos Dados this is the place to come and start your journey. This manual is divided into three sections:- Installation and starting commands- Using Base dos Dados with Pandas package- ExamplesIt is recommended to have know about the package Pandas before using *basedosdados* since data will be stored as dataframes. If you have any feedback about this material please send me an email and I will gladly help: <EMAIL>  1. Installation and starting commands Base dos Dados are located in pip and to install it you have to run the following code:<jupyter_code>#!pip install basedosdados==1.3.0a3<jupyter_output><empty_output><jupyter_text>Using an exclamation mark before the command will pass the command to the shell (not to the Python interpreter). If you prefer you can type *pip install basedosdados* directly in your Command Prompt.Before using *basedosdados* you have to import it:<jupyter_code>import basedosdados as bd<jupyter_output><empty_output><jupyter_text>Now that you have installed and imported the package it is time to see what data is available. The following command will list all current datasets.Note: the first time you execute a command in *basedosdados* it will prompt you to login with your Google Cloud. This is because results are delivered through BigQuery (which belongs to Google). BigQuery provides a reliable and quick service for managing data. It is free unless you require more than 1TB per month. If this happens to you just follow the on-screen instructions.<jupyter_code>bd.list_datasets()<jupyter_output>dataset_id: br_abrinq_oca


dataset_id: br_ana_atlas_esgotos


dataset_id: br_bd_diretorios_brasil


dataset_id: br_camara_atividade_legislativa


dataset_id: br_camara_dados_abertos


dataset_id: br_cgu_servidores_executivo_federal


dataset_id: br_geobr_mapas


dataset_id: br_ibge_amc


dataset_id: br_ibge_censo2010


dataset_id: br_ibge_nomes_brasil


dataset_id: br_ibge_pib


dataset_id: br_ibge_pnad_covid


dataset_id: br_ibge_populacao


dataset_id: br_imprensa_nacional_dou


dataset_id: br_inep_ideb


dataset_id: br_inep_indicadores_educacionais


dataset_id: br_inep_sinopse_estatistica_educacao_basica


dataset_id: br_inpe_prodes


dataset_id: br_mapbiomas_estatisticas


dataset_id: br_mc_auxilio_emergencial


dataset_id: br_mc_indicadores


dataset_id: br_me_caged


dataset_id: br_me_rais


dataset_id: br_me_socios


dataset_id: br_mobilidados_indicadores


dataset_id: br_ms_sim


dataset_id: br_observatorio_crianca_indicadores


dataset_id: br_ponte_indicadores


dataset_id[...]<jupyter_text>Note: the first time you execute a command in *basedosdados* it will prompt you to login with your Google Cloud. This is because results are delivered through BigQuery (which belongs to Google). BigQuery provides a reliable and quick service for managing data. It is free unless you require more than 1TB per month. If the login screen appears to you just follow the on-screen instructions. You can check the data you want with the *get_dataset_description*. We will use *br_ibge_pib* as example:<jupyter_code>bd.get_dataset_description('br_ibge_pib')<jupyter_output>Dados sobre Produto Interno Bruto do IBGE.


Para saber mais acesse:
Website: https://basedosdados.org/dataset/br-ibge-pib
Github: https://github.com/basedosdados/mais/

Ajude a manter o projeto :)
Apoia-se: https://apoia.se/basedosdados

Instituição (Quem mantém os dados oficiais?)
-----------
Nome: instituto-brasileiro-de-geografia-e-estatistica

Onde encontrar os dados
-----------------------
- https://www.ibge.gov.br/explica/pib.php

Grupos
------
- economia

Idiomas
------
- portugues

Licença
-------
Nome: MIT<jupyter_text>Each of the *datasets* has tables inside and to see what are the available tables you can use the following command:<jupyter_code>bd.list_dataset_tables('br_ibge_pib')<jupyter_output>table_id: municipios<jupyter_text>To analyze which information is in that table you can run the following command.<jupyter_code>bd.get_table_description('br_ibge_pib', 'municipios')<jupyter_output>Produto Interno Bruto (PIB) municipal a preços correntes.


Para saber mais acesse:
Website: 
Github: https://github.com/basedosdados/mais/

Ajude a manter o projeto :)
Apoia-se: https://apoia.se/basedosdados

Nível da Observação (Colunas que identificam cada linha unicamente)
-------------------
- municipio
- ano

Cobertura Temporal
------------------
- 2002
- 2003
- 2004
- 2005
- 2006
- 2007
- 2008
- 2009
- 2010
- 2011
- 2012
- 2013
- 2014
- 2015
- 2016
- 2017
- 2018

Cobertura Espacial
------------------
- brasil

Frequencia de Atualização
-------------------------
1 ano

Tratado por
-----------
Nome: <NAME>
Código: https://github.com/basedosdados/mais/tree/master/bases/br_ibge_pib/code
Email: <EMAIL>

Publicado por
-------------
Nome: <NAME>
Código: https://github.com/basedosdados/mais/tree/master/bases/br_ibge_pib/code
Website: www.ricardodahis.com
Email: <EMAIL><jupyter_text>With these three commands you are set and know what data is available. You are now able to create your dataframe and start your analysis! In this example we will generate a dataframe called *df* with information regarding Brazilian municipalities' GDP.  The data is stored as a dataframe object and it will interact with the same commands used in Pandas.<jupyter_code>df = bd.read_table('br_ibge_pib', 'municipios')
# Note: depending on how you configured your Google Cloud setup, it might be necessary to add project ID in the request.
# Like `bd.read_table('br_ibge_pib', 'municipios', billing_project_id=<YOUR_PROJECT_ID>)<jupyter_output>Downloading: 100%|██████████████████████████████████████████████████████████| 94616/94616 [00:08<00:00, 10636.02rows/s]<jupyter_text>Example In this section we will do quick exploratory example on the dataset we have been using. First by let's define again the table and take a look at it:<jupyter_code>df.head()<jupyter_output><empty_output><jupyter_text>Example 1: São Paulo The cities are being indexed by their ID You can see the correspondence in the following website: https://www.ibge.gov.br/explica/codigos-dos-municipios.php We will take a look at São Paulo city with the following ID: 3550308Creating a dataframe with São Paulo information:<jupyter_code>sao_paulo = df[df.id_municipio == 3550308]<jupyter_output><empty_output><jupyter_text>Looking at the table created we can check that everything is working as expected.<jupyter_code>sao_paulo<jupyter_output><empty_output><jupyter_text>Pandas has some graph functions built in and we can use them to further visualize data:<jupyter_code>sao_paulo.plot(x='ano', y='PIB')<jupyter_output><empty_output><jupyter_text>Since the tables are in nominal prices it might be interesting to see results in logarithmic form. For that we can use the following code:<jupyter_code>import numpy as np

#Adding new columns:
df['log_pib'] = df.PIB.apply(np.log10)

#Separating Sao Paulo again, now it will come with the new columns:
sao_paulo = df[df.id_municipio == 3550308]<jupyter_output><empty_output><jupyter_text>Analyzing data the way we did before:<jupyter_code>sao_paulo
sao_paulo.plot(x='ano',y='log_pib')<jupyter_output><empty_output><jupyter_text>Example 2: 2015 If you want to check all value added to GDP by the services sector in the year 2015 you could use the following code:<jupyter_code># First separating 2015 data only:
municipios2015 = df[df.ano == 2015]

#Adding logarithmic column:
municipios2015['log_VA_servicos'] = municipios2015.VA_servicos.apply(np.log10)

#Visualizing dataframe as histogram:
municipios2015[municipios2015['log_VA_servicos'] > 0].hist('log_VA_servicos')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09034229828850855, 'fraction_numerical': 0.021026894865525673, 'mean_word_length': 5.146506386175807, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '14', 'dir': 'jupyter-structured-clean-dedup', 'id': '10376', 'license': 'MIT', 'n_tokens_mistral': 2877, 'n_tokens_neox': 2627, 'n_words': 883, 'path': 'artigos/bd_for_beginners.ipynb', 'repo_name': 'basedosdados/analises', 'size': '8243'}