<jupyter_start><jupyter_text>Pangeo Billing on AWSThis binder should serve as a tool for looking at billing information for Pangeo's AWS account.Anyone with admin access (?) should be able to authenticate and use the binder.<jupyter_code>import boto3
import subprocess
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import matplotlib.dates as mdates 

%matplotlib inline<jupyter_output><empty_output><jupyter_text>Try some Boto3Okay. As long as AWS CLI is configured and you normally have access to the bucket, you can access it here.- AWS CLI won't be configured automatically in a Binder.    - Have that as part of the Docker Image, ask users to configure with their keys when they launch    - Or Find a temporary access method- On a JupyterHub, some users might already have AWS CLI configured    - Easier to use this code    - Data will persist        - Reduces costs for downloading from S3 many times        - Data is accessible to anyone who can get into your Hub (security issue)  Authenticate with Boto3The method in this notebook works for the following configurations:- Users who have configured the AWS CLI - Boto3 will use the access keys that you put in via 'aws configure'- Users who are on the us-east-1 pangeo JupyterHub - I added a policy to the role the cluster assumes that allows read access to the billing data bucket If neither of these apply, ie. you opened this as a binder, you will need to install and configure the AWS CLI- See https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html- Configure via the command 'aws configure'- Boto3 will then read your access keys as listed above  Billing Information BucketsThere are two billing information buckets- copydbr-pangeo- 783380859522-dlt-utilizationThe dlt bucket has more data over more time (from Nov 2018), but Rob declares the following about the copydbr bucket:"... this is cloudcheckr output which superseded the previous billing bucket: It was adopted by DLT as a better solution; and it has been more reliable."So, we are using the copydbr bucket by default, but permissions are in place if you want to look at the other one.<jupyter_code># Variables
region = 'us-east-1'
bucket = 'copydbr-pangeo'
file_keys = []
# Spin up base boto3 objects
s3_client = boto3.client('s3', region_name=region)
s3 = boto3.resource('s3', region_name=region)
csv_file_list = s3_client.list_objects(Bucket=bucket)
file_keys = [entry['Key'] for entry in csv_file_list['Contents']]
print("%d file keys" % len(file_keys))
file_keys<jupyter_output>2 file keys<jupyter_text>Download filesLet's grab the csv files themselvesCan for loop the cells below to download everything in file_keys<jupyter_code>for key in file_keys:
    download_loc = "/data/" + key
    s3_client.download_file(bucket, key, key)<jupyter_output><empty_output><jupyter_text>Unzip the file and watch the command line outputDon't need this right nowfor key in file_keys:    run_cmd_with_output(["unzip", key])  Move files via command line / subprocessYou don't actually need to unzip,Pandas read_csv will do it for you!<jupyter_code>def run_cmd_with_output(commands):
    print(subprocess.run(commands, capture_output=True
                        ).stdout.decode("utf-8"))<jupyter_output><empty_output><jupyter_text>Move data files to a new directory<jupyter_code>run_cmd_with_output(["mkdir", "data"])
# Need to run the whole command as one string with shell=True
# Because the wildcard '*' gets interpreted differently otherwise
subprocess.run(["mv *.zip data"], shell=True)
subprocess.run(["mv *.csv data"], shell=True)<jupyter_output><empty_output><jupyter_text>Can I circumvent downloading entirely?Might be able to just load in the data with boto3 into read_csv.Nope. Too annoying.  Create the dataframe<jupyter_code>df_list = [pd.read_csv("data/" + key, parse_dates=[14, 15]) for key in file_keys]
df = pd.concat(df_list)
df.head()<jupyter_output><empty_output><jupyter_text>Computations / Visualizations<jupyter_code>total_expenditures = df.BlendedCost.sum()
total_expenditures
df.groupby('ProductName').sum()['BlendedCost']<jupyter_output><empty_output><jupyter_text>Note: The Custom Charge seen in these cells is DLT Business Support.<jupyter_code>df[df['ProductName'] == 'Custom Charge']['ItemDescription'].values[0]
_ = df.groupby('ProductName').sum()['BlendedCost'].sort_values().plot(kind='barh')
_ = plt.xlabel('Cost ($)')<jupyter_output><empty_output><jupyter_text>Analysis Sections- Daily Costs (done)- Costs by Cluster- Monthly Costs (done)- Costs by User?<jupyter_code>df_time_idx = df.set_index("UsageEndDate")
df_time_idx.head()
# helper function to process start_time, end_time, and df
def process_start_and_end_time(df, start_time, end_time, cost_col):
    """
    Helper function to use start and end time
    Should function with either, both, or neither of start and end time specified
    """
    data_to_plot = df.resample("D").sum()
    if start_time and end_time:
        data_to_plot = data_to_plot.loc[start_time:end_time, cost_col]
    elif start_time and not end_time:
        data_to_plot = data_to_plot.loc[start_time:, cost_col]
    elif not start_time and end_time:
        data_to_plot = data_to_plot.loc[:end_time, cost_col]
    else:
        data_to_plot = data_to_plot.loc[:, cost_col]
        
    return data_to_plot
def plot_daily_costs(df, start_time=None, end_time=None, cost_col="BlendedCost"):
    """
    Plots a bar graph of daily costs over the specified range
    Can also specify the cost type
    """
    
    # Get the correct data to plot
    data_to_plot = process_start_and_end_time(df, start_time, end_time, cost_col)
    
    # Total Costs and Title / Label Formatting
    cost_total = data_to_plot.sum()
    title = "Daily Costs (Total Cost $%0.2f)" % cost_total
    ylabel = "Daily Costs ($ of %s)" % cost_col

    # Actual Plot
    fig, ax = plt.subplots()

    # Formatting for Dates
    days = mdates.DayLocator()
    months = mdates.MonthLocator()
    month_fmt = mdates.DateFormatter("%B")
    ax.xaxis.set_major_locator(months)
    ax.xaxis.set_minor_locator(days)
    ax.xaxis.set_major_formatter(month_fmt)

    # Plot with raw matplotlib
    ax.bar(data_to_plot.index, data_to_plot.values)
    plt.ylabel(ylabel)
    plt.xlabel("Usage End Date")
    plt.title(title)

    return fig
_ = plot_daily_costs(df_time_idx)
_ = plot_daily_costs(df=df_time_idx, start_time="2019-11-01")
_ = plot_daily_costs(df=df_time_idx, start_time="2019-10-01",
                     end_time="2019-10-31", cost_col="UnBlendedCost")
def plot_monthly_costs(df, start_time=None, end_time=None, cost_col="BlendedCost"):
    """
    Plots a bar graph of month costs over the specified range
    Can also specify the cost column
    """
    
    # Get the correct data to plot
    data_to_plot = process_start_and_end_time(df, start_time, end_time, cost_col)
    
    # Total Nov Costs and Title / Label Formatting
    cost_total = data_to_plot.sum()
    title = "Monthly Costs (Total Cost $%0.2f)" % cost_total
    ylabel = "Monthly Costs ($ of %s)" % cost_col
    
    # Actual Plot
    fig, ax = plt.subplots()

    # Formatting for Dates
    months = mdates.MonthLocator()
    month_fmt = mdates.DateFormatter("%B")
    ax.xaxis.set_major_locator(months)
    ax.xaxis.set_major_formatter(month_fmt)

    # Plot with raw matplotlib
    ax.bar(data_to_plot.index, data_to_plot.values)
    plt.ylabel(ylabel)
    plt.xlabel("Usage End Date")
    plt.title(title)

    return fig
_ = plot_monthly_costs(df_time_idx)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08918883154691015, 'fraction_numerical': 0.009130607383882492, 'mean_word_length': 4.476811594202899, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 36, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 6, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '228147', 'license': 'MIT', 'n_tokens_mistral': 2492, 'n_tokens_neox': 2384, 'n_words': 757, 'path': 'aws_billing_notebook.ipynb', 'repo_name': 'salvis2/costnotify', 'size': '7557'}