<jupyter_start><jupyter_text>Filtering Data<jupyter_code>df.filter('Close < 500').select(['Open', 'Close']).show()
df.filter(df['Close'] < 500).select(['Open', 'Close']).show()
df.filter((df['Close'] < 200) & ~(df['Open'] > 200)).select(['Open', 'Close']).show()
df.filter(df['Low'] == 197.16).show()<jupyter_output>+-------------------+------------------+----------+------+------+---------+---------+
|               Date|              Open|      High|   Low| Close|   Volume|Adj Close|
+-------------------+------------------+----------+------+------+---------+---------+
|2010-01-22 00:00:00|206.78000600000001|207.499996|197.16|197.75|220441900|25.620401|
+-------------------+------------------+----------+------+------+---------+---------+<jupyter_text>Converting Results to Rows and Dicts<jupyter_code>result = df.filter(df['Low'] == 197.16).collect()
row = result[0]
row.asDict()['Volume']<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.4368932038834951, 'fraction_numerical': 0.0970873786407767, 'mean_word_length': 1.7701492537313432, 'pattern_counts': {'":': 0, '<': 11, '<?xml version=': 0, '>': 9, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '596760', 'license': 'MIT', 'n_tokens_mistral': 361, 'n_tokens_neox': 294, 'n_words': 34, 'path': 'apache-spark/python/basics/DataFrame Operations.ipynb', 'repo_name': 'muhdlaziem/notes', 'size': '927'}