<jupyter_start><jupyter_text>Working with Snapshot FilesThis notebook shows how to read and use content from Snapshot files, either from a local folder or from a specific file.In this notebook...* [Dependencies and Initialisation](Dependencies-and-Initialisation)* [Supported File Formats]()* [Load a single AVRO file to a Pandas DataFrame]()* [Load all AVRO files in a folder to a Pandas DataFrame]()  Dependencies and InitialisationImport statements and environment initialisation using the package `dotenv`. More details in the [Configuration notebook](0.2_configuration.ipynb).<jupyter_code>from factiva.news import SnapshotFiles
print('Done!')<jupyter_output>Done!<jupyter_text>Supported File FormatsAt the moment, `factiva-*` packages only support `AVRO` files. The support for additional formats is currently under development.  Load a single AVRO file to a Pandas DataFrameThis operations uses the function `.read_file`.* **`filepath`**: Relative or absolute file path* **`only_stats`**: _Optional_. Specifies if only file metadata is loaded (True), or if the full article content is loaded (False). On average, only_stats loads about 1/10 and is recommended for quick metadata-based analysis. (Default is False)* **`merge_body`**: _Optional_. Specifies if the body field should be merged with the snippet and this last column being dropped. (default is False)<jupyter_code>sf = SnapshotFiles()
articles = sf.read_file('./data/part-000000000000.avro', only_stats=True)
articles.columns
articles[['an', 'publication_datetime', 'title', 'industry_codes', 'language_code']].head()<jupyter_output><empty_output><jupyter_text>Load all files within a folderThis operations loads all files with the same extention into a single Pandas DataFrame.* **`folderpath`**: Relative or absolute folder path* **`file_format`**: `AVRO` is the only format available at the moment* **`only_stats`**: _Optional_. Specifies if only file metadata is loaded (True), or if the full article content is loaded (False). On average, only_stats loads about 1/10 and is recommended for quick metadata-based analysis. (Default is False)* **`merge_body`**: _Optional_. Specifies if the body field should be merged with the snippet and this last column being dropped. (default is False)<jupyter_code>from factiva.news import SnapshotFiles
sf = SnapshotFiles()
articles = sf.read_folder(folderpath='./data/ztj2gkbldt', only_stats=False, merge_body=True)
articles.columns
articles[['an', 'publication_datetime', 'title', 'industry_codes', 'language_code']].head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09824561403508772, 'fraction_numerical': 0.008187134502923977, 'mean_word_length': 6.591715976331361, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '553999', 'license': 'MIT', 'n_tokens_mistral': 716, 'n_tokens_neox': 681, 'n_words': 286, 'path': '1.9_snapshot_files.ipynb', 'repo_name': 'dowjones/factiva-notebooks-packages', 'size': '2565'}