<jupyter_start><jupyter_text>Co-clustering in SparkThis noteboo attempts to execute in Spark the co-clustering algorithm presented in [co_clustering_R](../r/co_clustering_R.ipynb) which is the implementation of Bregman block average co-clustering algorithm (Banerjee et al., 2007) and it is inpired in the [single node impementation](https://github.com/fnyanez/bbac), [Copyright (c) 2016 <NAME>](https://github.com/fnyanez/bbac/blob/master/LICENSE.md).Next to the Python code the user will see the R code in Markdown format.  Dependencies<jupyter_code>#Add all dependencies to PYTHON_PATH
import sys
sys.path.append("/usr/lib/spark/python")
sys.path.append("/usr/lib/spark/python/lib/py4j-0.10.4-src.zip")
sys.path.append("/usr/lib/python3/dist-packages")

#Define environment variables
import os
os.environ["HADOOP_CONF_DIR"] = "/etc/hadoop/conf"
os.environ["PYSPARK_PYTHON"] = "python3"
os.environ["PYSPARK_DRIVER_PYTHON"] = "ipython"

#Load PySpark to connect to a Spark cluster
from pyspark import SparkConf, SparkContext

#from osgeo import gdal
#To read GeoTiffs as a ByteArray
from io import BytesIO
from rasterio.io import MemoryFile<jupyter_output><empty_output><jupyter_text>Connect to Spark<jupyter_code>appName = "co_clustering"
masterURL="spark://pheno0.phenovari-utwente.surf-hosted.nl:7077"

#A context needs to be created if it does not already exist
try:
    sc.stop()
except NameError:
    print("A  new Spark Context will be created.")
    
sc = SparkContext(conf = SparkConf().setAppName(appName).setMaster(masterURL))<jupyter_output>A  new Spark Context will be created.
{'fraction_non_alphanumeric': 0.10936517913262099, 'fraction_numerical': 0.01257071024512885, 'mean_word_length': 5.633333333333334, 'pattern_counts': {'":': 0, '<': 9, '<?xml version=': 0, '>': 9, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '2', 'dir': 'jupyter-structured-clean-dedup', 'id': '344981', 'license': 'Apache-2.0', 'n_tokens_mistral': 574, 'n_tokens_neox': 537, 'n_words': 144, 'path': 'applications/notebooks/romulo/co_clustering.ipynb', 'repo_name': 'phenology/infrastructure', 'size': '1597'}