<jupyter_start><jupyter_text>$DS^2$: Operator SelectionIn this notebook, we'll query the `gh-2017` corpus to find the most used data science "operators." An operator, in this context, is either something you call _on a dataframe_ (e.g., `df.drop(...)`) or something you call and _pass a dataframe_ (e.g., `foo.fit_transform(df)`).There are many other possibile non-call operators. For example, one can filter a dataframe like so: `df[df.x == 12]`. We will also attempt to capture a sampling of such operators in our analysis but, in this notebook, we will stick to understanding what operators (based on calls) are widely used.<jupyter_code>%load_ext autoreload
%autoreload 2

from codebook.python import *
from codebook.semantics import DSNotebooks as DSN

# Use the gh-2017 dataset. ~760k unique python notebooks
Evaluator.use_ds_test_1k()

# flows_df = pd.read_csv('/data/gh-2017/results/flows-read-to-fit.csv')
# DSN.set_prefilters(set(flows_df.fpath.unique()))
tmp = execute(
  use_of(DSN.pandas_read()) 
  compile=True
)
drops, df1, df2, df3 = DSN.drops()
tmp = df1.set_index('gid_use').join(uses1.set_index('gid_use')[["source_text_cols"]])
tmp[tmp.source_text_cols != tmp.source_text_use]
# Let's save these so we can avoid re-querying later...
reads.to_csv('/data/gh-2017/results/pandas-reads-2.csv', encoding='utf-8', index=False)
# Let's look at any call where the _target_ of said call is a use of one of the `pd.read_*` calls
# This should capture things like df.drop(...) or df.head(...)
calls_on_dataframes = execute(
  call() % select('name') % 'call'
  |where| call_target()
    |is_| anything(from_set(DSN.use_of_pandas_read(), 'gid')),
  compile=True
)
# Let's save these so we can avoid re-querying later...
calls_on_dataframes.to_csv('/data/gh-2017/results/calls-on-dataframes-2.csv', encoding='utf-8', index=False)
# Let's look at any call where any _argment_ of said call is a use of one of the `pd.read_*` calls
# This should capture things like pd.merge(...df...) and x.fit_transform(df)
calls_with_dataframes_as_args = execute(
  call() % select('name') % 'call'
  |where| any_arg()
    |is_| anything(from_set(DSN.use_of_pandas_read(), 'gid')),
  compile=True
)
# Let's save these so we can avoid re-querying later...
calls_with_dataframes_as_args.to_csv('/data/gh-2017/results/calls-with-dataframes-as-args-2.csv', encoding='utf-8', index=False)
flows_df = DSN.flows_reads_to_fits()
flows_df.to_csv('/data/gh-2017/results/flows-read-to-fit-2.csv', encoding='utf-8', index=False)
df1 = execute(
  call(with_name('drop')) % 'call', 
  |where| the_first_arg()
    |is_| use_of(string() % 'cols'),
  compile=True
)
df2 = execute(
  call(with_name('drop')) % 'call', 
  |where| the_first_arg()
    |is_| use_of(
      list_(where_every_child_has_type('string')) % 'cols'
    )
  compile=True
)
df3 = execute(
  call(with_name('drop')) % 'call', 
  |where| the_first_arg()
    |isa| subscript()
    |where| the_value_is(
      attribute()
      |where| the_attribute() 
        |isa| identifier(with_text('columns'))
    )
    |and_w| the_subscript_is(use_of(
      list_(where_every_child_has_type('integer')) % 'col_ids'
    )),
  compile=True
)

tmp = execute(
  call(with_name('astype')) % 'call'
  |where| the_first_arg()
    |is_| anything() % 'arg',
  compile=True
)

tmp
tmp = DSN.join()
tmp.pretty.value_counts().head(30)
visualize(tmp[tmp.pretty.str.contains('Left_outer')], 'call')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15040650406504066, 'fraction_numerical': 0.0156794425087108, 'mean_word_length': 4.694214876033058, 'pattern_counts': {'":': 0, '<': 5, '<?xml version=': 0, '>': 5, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '520179', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 1296, 'n_tokens_neox': 1245, 'n_words': 304, 'path': 'applications/jupyter-extension/nteract_on_jupyter/notebooks/operator-selection-gh-2017.ipynb', 'repo_name': 'jjhenkel/nteract', 'size': '3444'}