<jupyter_start><jupyter_text>d6tstack with pysparkPyspark is a great library for out-of-core computing. But if input files are not properly organized it quickly breaks. For example:1) if columns are different between files: [unlike dask](https://github.com/d6t/d6tstack/blob/master/examples-dask.ipynb) pyspark actually handles that2) if column order is rearranged between files it will read data, but into the wrong columns and you won't notice it3) if columns are named between files, you'll have to manually fix the situationPyspark can't easily handle those scenarios. With d6tstack you can easily fix the situation with just a few lines of code!For more instructions, examples and documentation see https://github.com/d6t/d6tstack<jupyter_code>import findspark
findspark.init(r'E:\progs.install\spark-2.2.0-bin-hadoop2.7')

import pyspark
sc = pyspark.SparkContext(appName="myAppName")
from pyspark.sql import SQLContext
sqlc = SQLContext(sc)<jupyter_output><empty_output><jupyter_text>Base Case: Columns are same between all filesAs a base case, we have input files which have consistent input columns and thus can be easily read in dask.<jupyter_code>sdf = sqlc.read.csv('test-data/input/test-data-input-csv-clean-*.csv', inferSchema=False, header=True)
sdf.toPandas()<jupyter_output><empty_output><jupyter_text>Problem Case 1: Columns are different between filesThat worked well. But what happens if your input files have inconsistent columns across files? Say for example one file has a new column that the other files don't have.[unlike dask](https://github.com/d6t/d6tstack/blob/master/examples-dask.ipynb) pyspark actually handles that. The new column got correctly added.<jupyter_code>sdf = sqlc.read.csv('test-data/input/test-data-input-csv-colmismatch-*.csv', inferSchema=False, header=True)
sdf.toPandas()<jupyter_output><empty_output><jupyter_text>Problem Case 2: Columns are reordered between filesThis is a sneaky case. The columns are the same but the order is different! Pyspark will read everything just fine without a warning but your data is totally messed up! You don't even notice it! You'll start using the data and at some point notice something weird is going on!In the example below, the "profit" column contains data from the "cost" column!<jupyter_code>sdf = sqlc.read.csv('test-data/input/test-data-input-csv-reorder-*.csv', inferSchema=False, header=True)
sdf.toPandas()<jupyter_output><empty_output><jupyter_text>Fixing the problem with d6stackAfter a while you'll get to the root of the problem, and then you can either manually process those files or use d6tstack to easily check for such a situation and fix it with a few lines of code - no manual processing required. Let's take a look!<jupyter_code>import glob
import d6tstack.combine_csv

cfg_fnames = list(glob.glob('test-data/input/test-data-input-csv-reorder-*.csv'))
c = d6tstack.combine_csv.CombinerCSV(cfg_fnames, all_strings=True)

# check columns
col_preview = c.preview_columns()
print('all columns equal?' , col_preview['is_all_equal'])
print('')
print('in what order do columns appear in the files?')
print('')
col_preview['df_columns_order'].reset_index(drop=True)<jupyter_output>all columns equal? False

in what order do columns appear in the files?<jupyter_text>Again, just a useful check before loading data into dask you can see that the columns don't line up. It's very fast to run because it only reads the headers, there's NO reason for you NOT to do it from a QA perspective.Same as above, the fix is the same few lines of code with d6stack.<jupyter_code># out-of-core combining
c2 = d6tstack.combine_csv.CombinerCSVAdvanced(c, cfg_col_sel=c.col_preview['columns_all'])
c2.combine_save('test-data/output/test-combined.csv')
sdf = sqlc.read.csv('test-data/output/test-combined.csv', inferSchema=False, header=True)
sdf.toPandas()<jupyter_output><empty_output><jupyter_text>Problem Case 3: Columns are renamed between filesIn this case a column gets renamed between files so you have two columns with partial NaNs that should really be the same column. You would have to manually inspect which columns this applies to and then manually edit them looking for NaNs.Instead you can use d6tstack to make your input files consistent.<jupyter_code># coming soon<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08358556461001164, 'fraction_numerical': 0.006519208381839348, 'mean_word_length': 5.895666131621188, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 3, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '19362', 'license': 'MIT', 'n_tokens_mistral': 1321, 'n_tokens_neox': 1292, 'n_words': 480, 'path': 'examples-pyspark.ipynb', 'repo_name': 'anujism/d6tstack', 'size': '4295'}