<jupyter_start><jupyter_text>Analytics queries I. Count total number of records with each distinct source IP<jupyter_code>_field = "id.orig_h"
pd.concat([f[[_field]] for f in dfs 
           if _field in f.columns]) \
  .groupby(_field)\
  .size()\
  .reset_index(name='count')<jupyter_output><empty_output><jupyter_text>Search queries I. Find all records with IP 10.128.0.19, sort by timestamp, and return the first 5<jupyter_code># records in a single dataframe
_dfs = pd.concat([f for f in dfs 
           if _field in f.columns], sort=False)
_dfs[_dfs["id.orig_h"] == "10.128.0.19"].sort_values("ts").head(5)
# records as dataframes
def search(field, label, key, n):
    rows, results = list(), list()
    for f in dfs:
        if field in f.columns:
            rows += f[f[field] == label].iterrows()

    return sorted(rows, key=lambda x: x[1][key])[:n]

records = search("id.orig_h", "10.128.0.19", "ts", 5)
for r in records:
    display(r[1].to_frame().transpose())<jupyter_output><empty_output><jupyter_text>Data discovery queries I. Count the number of records with each different schema<jupyter_code>for df in dfs:
    print(df.dtypes, len(df))<jupyter_output>_path             object
_write_ts         object
ts                object
uid               object
id.orig_h         object
id.orig_p          int64
id.resp_h         object
id.resp_p          int64
proto             object
duration         float64
orig_bytes       float64
resp_bytes       float64
conn_state        object
missed_bytes       int64
history           object
orig_pkts          int64
orig_ip_bytes      int64
resp_pkts          int64
resp_ip_bytes      int64
service           object
dtype: object 1021952
_path                 object
_write_ts             object
ts                    object
uid                   object
id.orig_h             object
id.orig_p              int64
id.resp_h             object
id.resp_p              int64
trans_depth            int64
method                object
host                  object
uri                   object
version              float64
user_agent            object
request_body_len       int64
response_body_len      int64
status_code   [...]
{'fraction_non_alphanumeric': 0.08314193844740468, 'fraction_numerical': 0.03123564538355535, 'mean_word_length': 2.09375, 'pattern_counts': {'":': 0, '<': 12, '<?xml version=': 0, '>': 12, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '516179', 'license': 'Apache-2.0', 'n_tokens_mistral': 712, 'n_tokens_neox': 667, 'n_words': 190, 'path': 'query/pandas_queries.ipynb', 'repo_name': 'ananyaks/data_model', 'size': '2177'}