<jupyter_start><jupyter_text>Load and Explore Movie DataUsing the new Microsoft Data Preparation SDK (in public preview)<jupyter_code>movies = dprep.read_csv(path='data/ml-latest-small/movies.csv', 
                        inference_arguments=dprep.InferenceArguments.current_culture())
movies.get_profile()
movies.head(5)<jupyter_output><empty_output><jupyter_text>Extract GenresThis is the part where we extract all of the genres<jupyter_code>df = movies.to_pandas_dataframe()
g = set()
for item in df['genres']:
    g = g | set([i.strip() for i in str(item).split('|')])
print(g)
genres = ['Documentary', 'Children', 'Action', 'Film-Noir', 'Drama', 
          'Adventure', 'Fantasy', 'IMAX', 'Mystery', 'Musical', 'Sci-Fi', 
          'War', 'Crime', '(no genres listed)', 'Romance', 'Comedy', 'Thriller', 
          'Western', 'Animation', 'Horror']<jupyter_output><empty_output><jupyter_text>Add Column Per GenreThis is to add a one-hot style encoding to genres<jupyter_code>for item in genres:
    movies = movies.add_column(new_column_name=item, 
                               prior_column='genres', 
                               expression=dprep.col('genres').contains(item))
movies = movies.drop_columns(columns='genres')
movies.head(5)<jupyter_output><empty_output><jupyter_text>Save Data Preparation StepsCreate and save dprep file<jupyter_code>movies = movies.set_name(name='movies')
package = dprep.Package(arg=movies)
package = package.save(file_path='movies.dprep')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13747521480502314, 'fraction_numerical': 0.0013218770654329147, 'mean_word_length': 4.670411985018727, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '275116', 'license': 'MIT', 'n_tokens_mistral': 530, 'n_tokens_neox': 501, 'n_words': 101, 'path': 'clean.ipynb', 'repo_name': 'sethjuarez/Endorser', 'size': '1513'}