<jupyter_start><jupyter_text>Reading and Writing Parquet FilesParquet is a high-density and performance file organization, the following section demonstrates how to with Parquet and import/export it to iguazio databasesee more details:  https://arrow.apache.org/docs/python/parquet.html<jupyter_code>!conda update boost-cpp -y<jupyter_output><empty_output><jupyter_text>Make sure to restart the Jupyter kernel after running the conda update<jupyter_code>import pyarrow.parquet as pq
import pandas as pd
import pyarrow as pa
# read some data into a dataframe and show its data & metadata
data = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/breast-cancer-wisconsin/wdbc.data', header = None)

# specify columns extracted from wbdc.names
data.columns = ["id","diagnosis","radius_mean","texture_mean","perimeter_mean","area_mean","smoothness_mean",
                "compactness_mean","concavity_mean","concave points_mean","symmetry_mean","fractal_dimension_mean",
                "radius_se","texture_se","perimeter_se","area_se","smoothness_se","compactness_se","concavity_se",
                "concave points_se","symmetry_se","fractal_dimension_se","radius_worst","texture_worst",
                "perimeter_worst","area_worst","smoothness_worst","compactness_worst","concavity_worst",
                "concave points_worst","symmetry_worst","fractal_dimension_worst"] 

data.head()<jupyter_output><empty_output><jupyter_text>Write the data into a Parquet on iguazio file systemNote: iguazio file system is mounted under `/v3io/{data-container-name}`<jupyter_code>#write the dataframe into a parquet (on iguazio file system)
import os
filepath = os.path.join('/v3io/'+os.getenv('V3IO_HOME')+'/examples/example.parquet')

table = pa.Table.from_pandas(data)
pq.write_table(table, filepath)
# read the parquet into memory and print the head 
table2 = pq.read_table(filepath)
table2.to_pandas().head()<jupyter_output><empty_output><jupyter_text>Advanced usage<jupyter_code># read specific columns to save bandwidth/memory and accelerate load
pq.read_table(filepath, columns=['id', 'area_mean']).to_pandas().head()
# read specific columns, if previously written by Pandas (read extra metadata)
pq.read_pandas(filepath, columns=['id', 'area_mean']).to_pandas().head()
parquet_file = pq.ParquetFile(filepath)
parquet_file.schema
parquet_file.metadata
# write to multiple partitioned files 
pq.write_to_dataset(table, root_path='my_parq', partition_cols=['radius_mean'])
# read partitioned table
table3 = pq.read_table('my_parq')
table3.to_pandas().head()
# read with multiple threads to accelerate performance
pq.read_table('my_parq', use_threads=4).to_pandas().head()<jupyter_output><empty_output><jupyter_text>Remove Data<jupyter_code># unmark the comment
#!rm -rf $HOME/examples/*<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1193502824858757, 'fraction_numerical': 0.002824858757062147, 'mean_word_length': 6.804407713498622, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '532518', 'license': 'Apache-2.0', 'n_tokens_mistral': 955, 'n_tokens_neox': 926, 'n_words': 191, 'path': 'getting-started/parquet-read-write.ipynb', 'repo_name': 'anant-g/tutorials', 'size': '2832'}