<jupyter_start><jupyter_text>Copyright (c) Microsoft Corporation. All rights reserved.Licensed under the MIT License.  Tutorial: Load demo data and enrich it with NOAA ISD Weather data.In this tutorial, you load the demo data (a parquet file in Azure Blob), check the data schema, enrich it with NOAA ISD Weather data.Prerequisites:> You must install the PyPi package on the cluster:> * azureml-contrib-opendatasetsLearn how to:> * Load the demo data from Azure Blob> * Check the demo data schema> * Initialize NoaaIsdWeather class to load weather data> * Enrich the demo data with weather data> * Display the joined result annd stats  Install azureml-opendatasets package<jupyter_code>!pip uninstall -y azureml-opendatasets
!pip install azureml-opendatasets<jupyter_output><empty_output><jupyter_text>Define a DemoData class to load demo parquet from Azure Blob<jupyter_code>from azure.storage.blob import BlockBlobService
import pyarrow.parquet as pq
from io import BytesIO

class DemoData:
    def __init__(self):
        self.blob_account_name = "azureopendatastorage"
        self.blob_container_name = "tutorials"
        self.blob_relative_path = 'noaa_isd_weather/demo.parquet'

    def to_pandas_dataframe(self):
        blob_service = BlockBlobService(account_name=self.blob_account_name)
        byte_stream = BytesIO()
        blob = blob_service.get_blob_to_stream(
            container_name=self.blob_container_name,
            blob_name=self.blob_relative_path,
            stream=byte_stream)

        return pq.read_table(source=byte_stream).to_pandas()<jupyter_output><empty_output><jupyter_text>Initialize a DemoData instance and load the pandas DataFrame and check the schema<jupyter_code>df = DemoData().to_pandas_dataframe()
df.dtypes<jupyter_output><empty_output><jupyter_text>Display the top 5 rows in the demo data dataframe<jupyter_code>df.head(5)<jupyter_output><empty_output><jupyter_text>Initialize NoaaIsdWeather class, get the enricher from it and enrich demo dataFor weather data, due to size, by default we allow reading from the last month if multiple months are passed.If you want to load more, please refer to `04-nyc-taxi-join-weather-in-pandas.ipynb.ipynb` under this folder for how.The logic for join:The join logic for Pandas version is using cKDTree to accelerate the speed of the process. We gather the public weather dataset as long/lat point array, pass it to create cKDTree. Then gather the customer dataset as long/lat point array, pass it to cKDTree query function, to find the closest point in cKDTree. After querying cKDTree, we join public weather dataset with customer dataset by the querying result, then grant ranking group id.<jupyter_code>from azureml.opendatasets.accessories.location_data import LatLongColumn
from azureml.opendatasets.accessories.location_time_customer_data import LocationTimeCustomerData
from azureml.opendatasets import NoaaIsdWeather
from datetime import datetime


_customer_data = LocationTimeCustomerData(df, LatLongColumn('lat', 'long'), 'datetime')
weather = NoaaIsdWeather(
    cols=["temperature", "windSpeed", "seaLvlPressure"],
    start_date=datetime(2015, 5, 1, 0, 0),
    end_date=datetime(2015, 5, 31, 23, 59))
weather_enricher = weather.get_enricher()
joined_data = weather_enricher.enrich_customer_data_with_agg(
    customer_data_object=_customer_data,
    location_match_granularity=5,
    time_round_granularity='day',
    agg='avg')<jupyter_output><empty_output><jupyter_text>Display the top 10 rows of the joined result<jupyter_code>joined_data.data.head(10)<jupyter_output><empty_output><jupyter_text>Check the stats of joined result<jupyter_code>joined_data.data.info()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07050243111831442, 'fraction_numerical': 0.007563479200432199, 'mean_word_length': 5.720508166969147, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 36, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '637729', 'license': 'MIT', 'n_tokens_mistral': 1191, 'n_tokens_neox': 1136, 'n_words': 342, 'path': 'tutorials/data-join/02-weather-join-in-pandas.ipynb', 'repo_name': 'Azure/OpenDatasetsNotebooks', 'size': '3702'}