<jupyter_start><jupyter_text>Connect to the clusters   STEPS: - connect to VPN and type in the terminal :  *ssh cleres@iccluster028.iccluster.epfl.ch*- see the list of datasets: hadoop fs -ls /datasets- Create your python script - **put the .py script on the cluster** : scp -r -p FILE <EMAIL>:/home/YOURNAME/FILE- **Run the script on the cluster** : spark-submit --master yarn --deploy-mode cluster --driver-memory 4G --num-executors 5 --executor-memory 4G --executor-cores 5 fetch_data_from_server.py- **Download the output from the cluster** : scp -r -p  <EMAIL>es@ic<EMAIL>.<EMAIL>.<EMAIL>.ch:/home/cleres/posts.parquet ./- See the size of the files that we dowload : **du -h posts.parquet**  TWITTER datasetWhat do we need to do : - The data is on the cluster in the **datasets/** directory - Pandas is perfect for data analysis and manipulation, while Spark can do the heavy job. The ideal workflow is to write your aggregation task in Spark, save the reduced dataset in JSON format and then load it on your laptop with Pandas.- How can I **access local files in the cluster**, I try to accessuser/greffe/data/mapping_country_codes.csvwith spark.read.csv("/home/greffe/data/mapping_country_codes.csv", header=True)but get the following error:pyspark.sql.utils.AnalysisException: u'Path does not exist: hdfs://iccluster028.iccluster.epfl.ch:8020/user/greffe/data/mapping_country_codes.csv;'EDIT: I understood the problem1) we have to put the files on hdfs "hadoop fs -put file"2) we access those at "hdfs:///user/USERNAME/FILE" from spark (edited)- **put the py file on the cluster** (in a new terminal window without ssh access) : scp -r -p FILE <EMAIL>:/home/YOURNAME/FILE- OUR VERSION : scp -r -p fetch_data_from_server.py <EMAIL>:/home/cleres/- spark-submit --master yarn --deploy-mode cluster --driver-memory 4G --num-executors 5 --executor-memory 4G --executor-cores 5 some_script.py script_arg1 script_arg2- OUR VERSION: **spark-submit --master yarn --deploy-mode cluster --driver-memory 4G --num-executors 5 --executor-memory 4G --executor-cores 5 fetch_data_from_server.py**- How could we **download the output from the cluster**? you have to transfering from the hdfs to your local file system with hadoop and then to use scp to get it locally (if you use files in the py script, they have to be on the hdfs by the way)Export the files from user/cleres/ to the local machine: **scp -r -p  <EMAIL>:/home/cleres/posts.parquet ./**- to do this you need to use copy-to-local before 1.hadoop fs -get HDFS file path      Local system directory path2.hadoop fs -copyToLocal HDFS file path    Local system directory pathNOTE: "./" exports the files in the directory where the UNIX command is launched. - see the size of the files that we dowload : **du -h posts.parquet**<jupyter_code>import os
import re

from pyspark.sql import *
from pyspark.sql.functions import unix_timestamp, udf, to_date
from pyspark.sql.types import ArrayType, StringType, IntegerType
from datetime import datetime

spark = SparkSession.builder.getOrCreate()
spark.conf.set('spark.sql.session.timeZone', 'UTC')
sc = spark.sparkContext

CLUSTER_DATA_DIR = '/datasets/twitter_internetarchive/2017/01/01/01/'
MY_DATA_DIR = '/data/'

files = os. listdir(path)
for name in files:
    print(path + name)
    posts_df = spark.read.json(CLUSTER_DATA_DIR + name)
    posts_df.write.parquet(MY_DATA_DIR + name + ".parquet", mode = "overwrite")<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10507982583454281, 'fraction_numerical': 0.011030478955007257, 'mean_word_length': 4.213313161875946, 'pattern_counts': {'":': 0, '<': 13, '<?xml version=': 0, '>': 13, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 1, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '1', 'dir': 'jupyter-structured-clean-dedup', 'id': '524821', 'license': 'MIT', 'n_tokens_mistral': 1114, 'n_tokens_neox': 1056, 'n_words': 411, 'path': 'research_code/fetch_data_from_server.ipynb', 'repo_name': 'nicolaslesimple/Investing_For_Social_Good', 'size': '3568'}