<jupyter_start><jupyter_text>Writing DataCopyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License. It is possible to write out the data at any point in a Dataflow. These writes are added as steps to the resulting Dataflow and will be executed every time the Dataflow is executed. Since there are no limitations to how many write steps there are in a pipeline, this makes it easy to write out intermediate results for troubleshooting or to be picked up by other pipelines.It is important to note that the execution of each write results in a full pull of the data in the Dataflow. A Dataflow with three write steps will, for example, read and process every record in the dataset three times.  Writing to Files Data can be written to files in any of our supported locations (Local File System, Azure Blob Storage, and Azure Data Lake Storage). In order to parallelize the write, the data is written to multiple partition files. A sentinel file named SUCCESS is also output once the write has completed. This makes it possible to identify when an intermediate write has completed without having to wait for the whole pipeline to complete.> When running a Dataflow in Spark, attempting to execute a write to an existing folder will fail. It is important to ensure the folder is empty or use a different target location per execution.The following file formats are currently supported:- Delimited Files (CSV, TSV, etc.)- Parquet Files<jupyter_code>import azureml.dataprep as dprep

# We'll start by loading data into a Dataflow. We will re-use this with different formats.
t = dprep.smart_read_file('./data/fixed_width_file.txt')
t = t.to_number('Column3')
t.head(10)<jupyter_output><empty_output><jupyter_text>Delimited Files<jupyter_code># The line below creates a new Dataflow with a write step, but the actual write has not been
# executed yet. When the Dataflow is run, the write will take place.
write_t = t.write_to_csv(directory_path=dprep.LocalFileOutput('./test_out/'))

# We now execute the Dataflow, which executes the write operation.
write_t.run_local()

written_files = dprep.read_csv('./test_out/part-*')
written_files.head(10)<jupyter_output><empty_output><jupyter_text>The data we wrote out contains several errors in the numeric columns due to numbers that we were unable to parse. When written out to CSV, these are replaced with the string "ERROR" by default. We can parameterize this as part of our write call. In the same vein, it is also possible to set what string to use to represent null values.<jupyter_code>write_t = t.write_to_csv(directory_path=dprep.LocalFileOutput('./test_out/'), 
                         error='BadData',
                         na='NA')
write_t.run_local()
written_files = dprep.read_csv('./test_out/part-*')
written_files.head(10)<jupyter_output><empty_output><jupyter_text>Parquet Files<jupyter_code># Similarly to write_to_csv above write_to_parquet returns a new Dataflow with a write Parquet step
# which hasn't been executed yet.
write_parquet_t = t.write_to_parquet(directory_path=dprep.LocalFileOutput('./test_parquet_out/'),
                                     error='MiscreantData')

# We now execute the Dataflow, which executes the write operation.
write_parquet_t.run_local()

written_parquet_files = dprep.read_parquet_file('./test_parquet_out/part-*')
written_parquet_files.head(10)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06426056338028169, 'fraction_numerical': 0.002640845070422535, 'mean_word_length': 4.70066889632107, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 18, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '469192', 'license': 'MIT', 'n_tokens_mistral': 950, 'n_tokens_neox': 913, 'n_words': 418, 'path': 'API/writing_data.ipynb', 'repo_name': 'joelvisroman/MSDataPrepDocs', 'size': '3408'}