<jupyter_start><jupyter_text>Now you can get extra information for the profiler if you activate pass verbose= True to optimus<jupyter_code># Create optimus
from optimus import Optimus
op = Optimus(master="local[*]", app_name = "optimus" , checkpoint= True, queue_url="amqp://eujwlcwg:QwZVFnWSqsJFodlF-8xWCWi7Rg6WPSwj@chimpanzee.rmq.cloudamqp.com/eujwlcwg")
df = op.load.csv("data/Meteorite_Landings.csv").h_repartition()
df.table(10)<jupyter_output><empty_output><jupyter_text>Profiler dump mode (Faster). It just handle the column data type as present in the dataframe<jupyter_code>op.profiler.run(df, "name", infer=False)<jupyter_output><empty_output><jupyter_text>Profiler smart mode (Slower). It just try to infer the column data type and present extra data acordinly. From example datetype columns get extra histograms about minutes, day, week and month. Also can detect array types on data.<jupyter_code>op.profiler.run(df, "*",infer=True)<jupyter_output><empty_output><jupyter_text>Plot profile for a specific column<jupyter_code>op.profiler.run(df, "reclat")<jupyter_output><empty_output><jupyter_text>Output a json file  Plot histagram for multiple columns<jupyter_code>df.plots.hist(["id", "reclong"], 20)
df.plots.frequency(["id", "reclong"], 10)
df.plots.correlation(["id","mass (g)", "reclat"])
df.correlation(["id","mass (g)", "reclat"], output="array")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1317108088761632, 'fraction_numerical': 0.006442376521116679, 'mean_word_length': 8.32, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '258482', 'license': 'Apache-2.0', 'n_tokens_mistral': 519, 'n_tokens_neox': 498, 'n_words': 114, 'path': 'examples/new-api-profiler.ipynb', 'repo_name': 'gaybro8777/Optimus', 'size': '1397'}