<jupyter_start><jupyter_text>Integrating Vertica with Dask Imports and basic setup<jupyter_code>import numpy as np
import pandas as pd
import dask
import dask.dataframe as dd
from distributed.client import Client

import vertica_python
from verticapy.vdataframe import vDataFrame
from dask_vertica import (
    to_vertica,
    read_vertica,
)


schema = "coiled_schema"

vdb = {
    'host': 'aaa.bb.cc.ddd',
    'port': 5433,
    'user': '<vertica_db_username>',
    'password': '<<PASSWORD>>',
    'database': '<vertica_db_name>',
    'connection_load_balance': True,
    'session_label': 'py',
    'unicode_error': 'strict'
}
client = Client(n_workers=4)
client<jupyter_output><empty_output><jupyter_text>Create a large(-ish) time series dataframe (1 year of 30-sec data)<jupyter_code># larger DF with more partitions
demo_ts = dd.demo.make_timeseries(
    start="2000-01-01",
    end="2000-12-31",
    freq="30s",
    partition_freq="1W"
).reset_index()  # moves the date from the index to a column

demo_ts.head()<jupyter_output><empty_output><jupyter_text>Upload the dataframe to a Vertica database(this will overwrite any existing table of the same name)<jupyter_code>to_vertica(demo_ts, vdb, "demo_ts", schema=schema, if_exists="overwrite")<jupyter_output><empty_output><jupyter_text>Try to upload load again, this time with `if_exists = "error"` to prevent overwriting<jupyter_code>try:
    to_vertica(demo_ts, vdb, "demo_ts", schema=schema, if_exists="error")
except RuntimeError:
    print("Sucessfully avoided disaster")<jupyter_output><empty_output><jupyter_text>Lazily compute monthly means for each name and upload to DB<jupyter_code>means = (
    read_vertica(vdb, name="demo_ts", npartitions=50, schema=schema)
        .assign(month=lambda df: df["timestamp"].dt.month)
        .groupby(["name", "month"])["y"]
        .mean()
        .reset_index()
)
print(type(means))
to_vertica(means, vdb, "monthly_means", schema=schema, if_exists="overwrite")<jupyter_output><empty_output><jupyter_text>Now fetch the table we uploaded as a dataframe and do some calculations<jupyter_code>demo_means = read_vertica(vdb, "monthly_means", 5, schema=schema)
demo_means  # dask series, uncomputed<jupyter_output><empty_output><jupyter_text>Finally call `compute` on the retrieved data<jupyter_code>demo_means.compute()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1196763202725724, 'fraction_numerical': 0.012776831345826235, 'mean_word_length': 6.139817629179332, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '610049', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 851, 'n_tokens_neox': 813, 'n_words': 184, 'path': 'examples/example.ipynb', 'repo_name': 'phobson/dask-vertica', 'size': '2357'}