<jupyter_start><jupyter_text>Collect AQI dataThe luftdaten.info webpage provides instruction to create an ESP8266 based AQI meter that uploads data to the cloud and measures PM2.5, PM10, humidity, temperature, air pressure and so on. You can connect a 1602 display as well. Powered by USB with micro USB. DatabaseThe measured data is availabe as .csv file to download from madavi.de. You get one file with all data for each day. That's 365 files per year to analyse - with data for every 145 seconds. Convert it to one database with values for each our and only one file is the goal of this notebook.Let's list all imported files:<jupyter_code>import os
for dirname, _, filenames in os.walk('/kaggle/input'):
    for filename in filenames:
        print(os.path.join(dirname, filename))<jupyter_output>/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-17.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-28.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-23.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-18.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-29.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-22.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-26.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-24.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-21.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-27.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-16.csv
/kaggle/input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-25.csv
/kaggle/input/aqi-first-data[...]<jupyter_text>Now let's use pandas to import the data.<jupyter_code>import time
import pandas as pd

start_time = time.time()

data = pd.read_csv('../input/aqi-first-data-from-april-2018/data-esp8266-1129419-2018-04-20.csv',sep=';',header=0,skiprows=1,index_col=False,names=['date','2','3','4','5','6','7','pm25','pm10','10','11','temp','hum','14','15','16','17','18','19','20','21'],usecols=['date','pm25','pm10','temp','hum'])
#data.drop(["Time","durP1","ratioP1","P1","durP2","SDS_P1","SDS_P2","BMP_temperature","BMP_pressure","BME280_temperature","BME280_humidity","BME280_pressure","Min_cycle","Max_cycle","Samples","Signal"],axis = 1, inplace = True)
print(data)

time_passed = time.time() - start_time
print('Import took %s seconds' % time_passed)<jupyter_output>date   pm25  pm10  temp   hum
0    2018/04/20 00:03:54  12.60  8.13  26.9  57.0
1    2018/04/20 00:06:23  11.30  7.77  27.0  53.3
2    2018/04/20 00:08:53  10.53  8.30  26.9  53.4
3    2018/04/20 00:11:23   8.33  7.20  27.0  56.0
4    2018/04/20 00:13:52  11.03  7.33  27.1  57.0
..                   ...    ...   ...   ...   ...
569  2018/04/20 23:47:50   5.00  3.70  26.8  53.4
570  2018/04/20 23:50:20   4.30  3.90  26.8  55.5
571  2018/04/20 23:52:49   5.03  4.10  26.9  55.8
572  2018/04/20 23:55:18   3.73  3.37  27.0  50.6
573  2018/04/20 23:57:50   5.03  3.47  27.0  55.5

[574 rows x 5 columns]
Import took 0.04047751426696777 seconds
{'fraction_non_alphanumeric': 0.1658385093167702, 'fraction_numerical': 0.2108695652173913, 'mean_word_length': 3.771851851851852, 'pattern_counts': {'":': 0, '<': 7, '<?xml version=': 0, '>': 7, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '69560', 'license': 'MIT', 'n_tokens_mistral': 1743, 'n_tokens_neox': 1336, 'n_words': 240, 'path': 'data/import-luftdaten-info-csv-into-sqlite3-database-1.ipynb', 'repo_name': 'kreier/aqi', 'size': '3220'}