<jupyter_start><jupyter_text>Section 2: Working with Dask Arrays  Inspecting a Dask arrayA dask.array called energy is provided for you. The array energy contains electricity load in kWh in Texas sampled every 15 minutes for the year 2000. This dataset was provided by ERCOT.Your job is to determine the correct shape of the array energy and the total electricity (in kWh) consumed by Texas for the year 2000 (i.e., the sum of the electricity load). Remember to leverage both the .sum() and .compute() methods.<jupyter_code>import h5py

filename = '../datasets/Texas/Texas/texas.2000.hdf5'

with h5py.File(filename, 'r') as f:
    # List all groups
    print("Keys: %s" % f.keys())
    a_group_key = list(f.keys())[0]

    # Get the data
    data = list(f[a_group_key])

import dask.array as da

energy = da.from_array(data, chunks=1000)
energy.sum().compute()<jupyter_output><empty_output><jupyter_text>Chunking a NumPy arrayA NumPy array has been provided for you as energy. This is the electricity load in kWh for the state of Texas sampled every 15 minutes over the year 2000 (that's about 35 thousand samples).Your job is to convert the NumPy array into a dask.array; the dask.array should have chunks whose sizes are 1/4 of the number of elements of the array energy. You will then inspect the chunk sizes of the Dask array. Finally, you'll compute the mean electricity load in kWh in two ways (using the dask.array and using the NumPy array) to compare the results.<jupyter_code>import h5py
import numpy as np

filename = '../datasets/Texas/Texas/texas.2000.hdf5'

with h5py.File(filename, 'r') as f:
    # List all groups
    print("Keys: %s" % f.keys())
    a_group_key = list(f.keys())[0]

    # Get the data
    energy = np.array(f[a_group_key])

# Call da.from_array():  energy_dask
energy_dask = da.from_array(energy, chunks=len(energy)/4)

# Print energy_dask.chunks
print(energy_dask.chunks)

# Print Dask array average and then NumPy array average
print(energy_dask.mean().compute())
print(energy.mean())<jupyter_output><empty_output><jupyter_text>Timing Dask array computationsYour job now is to create two Dask arrays from energy using different chunksizes. You'll then measure the time required (in milliseconds) to compute the standard deviation of each Dask array.The NumPy array energy is provided as before and the module dask.array is imported for you as da.<jupyter_code>import h5py

filename = '../datasets/Texas/Texas/texas.2000.hdf5'

with h5py.File(filename, 'r') as f:
    # List all groups
    print("Keys: %s" % f.keys())
    a_group_key = list(f.keys())[0]

    # Get the data
    data = list(f[a_group_key])

import dask.array as da
# Import time
import time

# Call da.from_array() with arr: energy_dask4
energy_dask4 = da.from_array(energy, chunks=len(energy)/4)

# Print the time to compute standard deviation
t_start = time.time()
std_4 = energy_dask4.std().compute()
t_end = time.time()
print((t_end - t_start) * 1.0e3)
# Import time
import time

# Call da.from_array() with arr: energy_dask8
energy_dask8 = da.from_array(energy, chunks=len(energy)/8)

# Print the time to compute standard deviation
t_start = time.time()
std_8 = energy_dask8.std().compute()
t_end = time.time()
print((t_end - t_start) * 1.0e3)<jupyter_output><empty_output><jupyter_text>Predicting result of broadcastingBinary arithmetic operations between two arrays are possible only when their shapes are compatible. Being compatible means either that the two arrays have identical .shape attributes or that their shapes satisfy certain rules. The rules for broadcasting in NumPy are summarized in the documentation.Here, the NumPy arrays a, b, c, and d are provided for you. They differ in shape, number of dimensions, or in the extent of each dimension. Which of the possible sums listed below yields an array with shape (10,10)?```In [1]: a.shapeOut[1]:()In [2]:b.shape Out[2]:(10, 1, 1)In [3]:c.shapeOut[3]:(10, 10)In [4]:d.shapeOut[4]:(1, 10)```  Subtracting & broadcastingNow that you're comfortable with broadcasting rules, you're going to practice using the .reshape() method together with broadcasting.The one-dimensional array load_2001 holds the total electricity load for the state of Texas sampled every 15 minutes for the entire year 2001 (35040 samples in total). The one-dimensional array load_recent holds the corresponding data sampled for each of the years 2013 through 2015 (i.e., 105120 samples consisting of the samples from 2013, 2014, & 2015 in sequence). None of these years are leap years, so each year has 365 days. Observe also that there are 96 intervals of duration 15 minutes in each day.Your job is to compute the differences of the samples in the years 2013 to 2015 each from the corresponding samples of 2001.<jupyter_code># Import h5py and dask.array
import h5py
import dask.array as da
import numpy as np 

filename = '../datasets/Texas/Texas/texas.2001.hdf5'

with h5py.File(filename, 'r') as f:
    # List all groups
    print("Keys: %s" % f.keys())
    a_group_key = list(f.keys())[0]

    # Get the data
    data = list(f[a_group_key])

load_2001 = np.array(data)


filename = '../datasets/Texas/Texas/texas.2013.hdf5'

with h5py.File(filename, 'r') as f:
    # List all groups
    print("Keys: %s" % f.keys())
    a_group_key = list(f.keys())[0]

    # Get the data
    data = list(f[a_group_key])

load_2013 = np.array(data)


filename = '../datasets/Texas/Texas/texas.2014.hdf5'

with h5py.File(filename, 'r') as f:
    # List all groups
    print("Keys: %s" % f.keys())
    a_group_key = list(f.keys())[0]

    # Get the data
    data = list(f[a_group_key])

load_2014 = np.array(data)

filename = '../datasets/Texas/Texas/texas.2015.hdf5'

with h5py.File(filename, 'r') as f:
    # List all groups
    print("Keys: %s" % f.keys())
    a_group_key = list(f.keys())[0]

    # Get the data
    data = list(f[a_group_key])

load_2015 = np.array(data)

load_recent = np.array([load_2013, load_2014, load_2015])

# Reshape load_recent to three dimensions: load_recent_3d
load_recent_3d = load_recent.reshape((3,365,96))
# Reshape load_2001 to three dimensions: load_2001_3d
load_2001_3d = load_2001.reshape((1,365,96))
# Subtract the load in 2001 from the load in 2013 - 2015: diff_3d
diff_3d = load_recent_3d - load_2001_3d
# Print the difference each year on March 2 at noon
print(diff_3d[:, 61, 48])<jupyter_output><empty_output><jupyter_text>Computing aggregationsYou'll now compute summary statistics by aggregating NumPy arrays across various dimensions (in this case, giving trends in electricity usage).The load_recent_3d NumPy array from the last exercise is provided for you. It contains the electricity load (in kWh) sampled every 15 minutes from the start of 2013 to the end of 2015. The possible index values of the three-dimensional array correspond to year (from 0 to 2), day (from 0 to 364), and 15-minute interval (from 0 to 95); remember, NumPy arrays are indexed from zero. Thus, load_recent_3d[0,1,2] is the electricity load consumed on January 2nd, 2013 from 00:30:00 AM to 00:45:00 AM.<jupyter_code># Print maximum of load_recent_3d across 2nd & 3rd dimensions
print(load_recent_3d.max(axis=(1,2)))
# Compute sum along last dimension of load_recent_3d: daily_consumption
daily_consumption = load_recent_3d.sum(axis=-1)
# Print mean of 62nd row of daily_consumption
print(daily_consumption[:,61].mean())<jupyter_output><empty_output><jupyter_text>Reading the dataFor this exercise, the monthly average maximum temperature (degrees Celsius) data on a 10km grid has been downloaded from NOAA for the continental US.The data is stored in separate HDF5 files from 2008 to 2011. Your job is to use h5py to connect to the datasets and prepare a list of Dask arrays where each element is one year of data. The list of filenames is provided for you as filenames. The chunksizes have been chosen to optimize reading data from disk.For this exercise you'll utilize a list comprehension to quickly iterate through the filenames list and form a new list of h5py file handles and again to make a list of Dask arrays.<jupyter_code># Import h5py and dask.array
import h5py
import dask.array as da

# List comprehension to read each file: dsets
dsets = [h5py.File(f)["/tmax"] for f in filenames]

# List comprehension to make dask arrays: monthly
monthly = [da.from_array(d, chunks=(1,444,922)) for d in dsets]<jupyter_output><empty_output><jupyter_text>Stacking data & reading climatologyNow that we have a list of Dask arrays, your job is to call da.stack() to make a single Dask array where month and year are in separate dimensions. You'll also read the climatology data set, which is the monthly average max temperature again from 1970 to 2000.<jupyter_code># Compute the difference: diff
diff = (by_year - climatology)* 9/5
# Compute the average over last two axes: avg
avg = da.nanmean(diff, axis=(-1,-2)).compute()
# Plot the slices [:,0], [:,7], and [:11] against the x values
x = range(2008,2012)
f, ax = plt.subplots()
ax.plot(x,avg[:,0], label='Jan')
ax.plot(x,avg[:,7], label='Aug')
ax.plot(x,avg[:,11], label='Dec')
ax.axhline(0, color='red')
ax.set_xlabel('Year')
ax.set_ylabel('Difference (degrees Fahrenheit)')
ax.legend(loc=0)
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09687195583937655, 'fraction_numerical': 0.040480571490421044, 'mean_word_length': 4.696670776818742, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '82502', 'license': 'MIT', 'n_tokens_mistral': 3210, 'n_tokens_neox': 2887, 'n_words': 1136, 'path': 'parallel-programming-with-dask-in-python/2. Working with Dask Arrays/notebook_section_2.ipynb', 'repo_name': 'nhutnamhcmus/datacamp-playground', 'size': '9239'}