<jupyter_start><jupyter_text>ANOVOS- TS Auto Detection**Following notebook shows the list of functions related to "ts_auto_detection" module provided under ANOVOS package**- [ts_loop_cols_pre](ts_loop_cols_pre)- [regex_date_time_parser](regex_date_time_parser)- [ts_preprocess](ts_preprocess)API specification of **ts_auto_detection** module can be found here: [API Specification](https://docs.anovos.ai/api/data_ingest/ts_auto_detection.html) **Setting Spark Session**<jupyter_code>from anovos.shared.spark import *<jupyter_output>2022-03-09 13:00:27.225 | INFO     | anovos.shared.spark:init_spark:54 - Getting spark session, context and sql context app_name: Anovos_pipeline<jupyter_text>**Input/Output Path**<jupyter_code>inputPath = "../data/time_series_data/csv"
outputPath = "../output/time_series_data/ts_autodetection"
from anovos.data_ingest.data_ingest import read_dataset
from anovos.shared.utils import ends_with
df = read_dataset(spark, file_path = inputPath, file_type = "csv",file_configs = {"header": "True", 
                                                                           "delimiter": "," , 
                                                                           "inferSchema": "True"})
df.toPandas().head(5)
from anovos.data_ingest.ts_auto_detection import regex_date_time_parser,ts_loop_cols_pre,ts_preprocess<jupyter_output><empty_output><jupyter_text>ts_loop_cols_pre- API specification of function **ts_loop_cols_pre** can be found here<jupyter_code>odf = ts_loop_cols_pre(idf=df, id_col='STATE')
odf
# List describing the names of different fields present in the dataset
odf[0]
# List describing Custom data type of different fields which help to identify the potential columns which can be passed for time series analysis
odf[1]
# List describing maximum character length of each field
odf[2]<jupyter_output><empty_output><jupyter_text>**Note:** Here we identified 'YR' column having 'int_c' data type with 4 character length which can be passed for time series analysis  regex_date_time_parser- API specification of function **regex_date_time_parser** can be found here<jupyter_code>odf=regex_date_time_parser(spark,idf=df,id_col="STATE",col="YR",tz="local",val_unique_cat=4,trans_cat="int_c")
odf.toPandas()<jupyter_output><empty_output><jupyter_text>ts_preprocess- API specification of function **ts_preprocess** can be found here<jupyter_code># Example 1 - by passing all arguments 
odf = ts_preprocess(spark, idf=df, id_col='STATE', output_path=outputPath, tz_offset="local", run_type="local")
odf.toPandas()
# Example 2 - with mandatory arguments (rest arguments have default values)
odf1 = ts_preprocess(spark, idf=df, id_col='STATE', output_path=outputPath)
odf1.toPandas()
# Example 3 - using tz_offset as gmt
odf = ts_preprocess(spark, idf=df, id_col='STATE', output_path=outputPath, tz_offset="gmt", run_type="local")
odf.toPandas()
# Example 4 - using tz_offset as utc
odf = ts_preprocess(spark, idf=df, id_col='STATE', output_path=outputPath, tz_offset="utc", run_type="local")
odf.toPandas()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10533159947984395, 'fraction_numerical': 0.010078023407022107, 'mean_word_length': 5.5747863247863245, 'pattern_counts': {'":': 3, '<': 20, '<?xml version=': 0, '>': 20, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '571406', 'license': 'Apache-2.0', 'n_tokens_mistral': 1034, 'n_tokens_neox': 984, 'n_words': 233, 'path': 'examples/notebooks/data_ingest__ts_auto_detection.ipynb', 'repo_name': 'ziedbouf/anovos', 'size': '3076'}