<jupyter_start><jupyter_text>![LOGO](../img/MODIN_ver2_hrz.png)Scale your pandas workflows by changing one line of code  Exercise 4: Setting up cluster environment**GOAL**: Learn how to set up a cluster for Modin.**NOTE**: This exercise has extra requirements. Read instructions carefully before attempting. **This exercise instructs the user on how to start a 700+ core cluster, and it is not shut down until the end of Exercise 5. Read instructions carefully.** Often in practice we have a need to exceed the capabilities of a single machine. Modin works and performs well in both local mode and in a cluster environment. The key advantage of Modin is that your notebook does not change between local development and cluster execution. Users are not required to think about how many workers exist or how to distribute and partition their data; Modin handles all of this seamlessly and transparently.![Cluster](../img/modin_cluster.png)**Extra Requirements for this exercise**Detailed instructions can be found here: https://docs.ray.io/en/master/cluster/launcher.htmlFrom command line:- `pip install boto3`- `aws configure`- `ray up modin-cluster.yaml`Included in this directory is a file named `modin-cluster.yaml`. We will use this to start the cluster.<jupyter_code># !pip install boto3
# !aws configure<jupyter_output><empty_output><jupyter_text>Starting and connecting to the clusterThis example starts 1 head node (m5.24xlarge) and 7 workers (m5.24xlarge), 768 total CPUs.Cost of this cluster can be found here: https://aws.amazon.com/ec2/pricing/on-demand/.<jupyter_code># !ray up modin-cluster.yaml<jupyter_output><empty_output><jupyter_text>Connect to the cluster with `ray attach`<jupyter_code># !ray attach modin-cluster.yaml
# DO NOT CHANGE THIS CODE!
# Changing this code risks breaking further exercises

import time
time.sleep(600) # We need to give ray enough time to start up all the workers
import ray
ray.init(address="auto")
import modin.pandas as pd
assert pd.DEFAULT_NPARTITIONS == 768, "Not all Ray nodes are started up yet"
ray.shutdown()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0778414517669532, 'fraction_numerical': 0.012416427889207259, 'mean_word_length': 5.367781155015198, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '159816', 'license': 'ECL-2.0', 'n_tokens_mistral': 601, 'n_tokens_neox': 570, 'n_words': 263, 'path': 'examples/tutorial/tutorial_notebooks/cluster/exercise_4.ipynb', 'repo_name': 'staftermath/modin', 'size': '2094'}