<jupyter_start><jupyter_text>Cerebral Cortex Data Analysis AlgorithmsCerebral Cortex contains a library of algorithms that are useful for processing data and converting it into features or biomarkers.  This page demonstrates a simple GPS clustering algorithm.  For more details about the algorithms that are available, please see our [documentation](https://cerebralcortex-kernel.readthedocs.io/en/latest/).  These algorithms are constantly being developed and improved through our own work and the work of other researchers.  Initalize the system<jupyter_code>%reload_ext autoreload
from util.dependencies import *

CC = Kernel("/home/jovyan/cc_conf/", study_name="default")<jupyter_output><empty_output><jupyter_text>Generate some sample location dataThis example utilizes a data generator to protect the privacy of real participants and allows for anyone utilizing this system to explore the data without required institutional review board approvals. This is disabled for this demonstration to not create too much data at once.<jupyter_code>gps_stream = gen_location_datastream(user_id=USER_ID, stream_name="GPS--org.md2k.phonesensor--PHONE")
gps_stream.printSchema()
gps_stream.show(3)<jupyter_output>root
 |-- timestamp: timestamp (nullable = true)
 |-- localtime: timestamp (nullable = true)
 |-- user: string (nullable = true)
 |-- version: long (nullable = true)
 |-- latitude: double (nullable = true)
 |-- longitude: double (nullable = true)
 |-- altitude: long (nullable = true)
 |-- speed: double (nullable = true)
 |-- bearing: double (nullable = true)
 |-- accuracy: double (nullable = true)

+-------------------+-------------------+--------------------+-------+-----------------+------------------+--------+--------+----------+---------+
|          timestamp|          localtime|                user|version|         latitude|         longitude|altitude|   speed|   bearing| accuracy|
+-------------------+-------------------+--------------------+-------+-----------------+------------------+--------+--------+----------+---------+
|2019-09-01 11:35:59|2019-09-01 16:35:59|00000000-afb8-476...|      1|35.11071115468912|   -90.02262837146|      93|2.370432|159.166668| 24.43889|
|2019-09-0[...]<jupyter_text>Cluster the location dataCerebral Cortex makes it easy to apply built-in algorithms to data streams.  In this case, `gps_clusters` is imported from the algorithm library, then `compute` is utilized to run this algorithm on the `gps_stream` to generate a set of centroids. This is the general format for applying algorithm to datastream and makes it easy for researchers to apply validated and tested algorithms to his/her own data without the need to become an expert in the particular set of transformations needed.  _Note:_ the `compute` method engages the parallel computation capabilities of Cerebral Cortex, which causes all the data to be read from the data storage layer and processed on every computational core available to the system.  This allows the computation to run as quickly as possible and to take advantage of powerful clusters from a relatively simple interface.  This capability is critical to working with mobile sensor big data where data sizes can exceed 100s of gigabytes per datastream for larger studies.<jupyter_code>from cerebralcortex.algorithms.gps.clustering import cluster_gps
windowed_gps = gps_stream.window()
clusters = cluster_gps(windowed_gps)
clusters.printSchema()
clusters.show(truncate=False)<jupyter_output>2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,250 WARNING Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f67739e75b0>: Failed to establish a new connection: [Errno -2] Name or service not known')': /auth/tokens/demo
2020-11-10 21:19:16,250 WARNING Retrying (Retr[...]<jupyter_text>Visualize GPS Data  GPS Stream PlotGPS visualization requires dedicated plotting capabilities. Cerebral Cortex includes a library to allow for interactive exploration.  In this plot, use your mouse to drag the map around along with zooming in to explore the specific data points.<jupyter_code>from cerebralcortex.plotting.gps.plots import plot_gps_clusters
plot_gps_clusters(clusters, user_id=USER_ID, zoom=8)<jupyter_output>/opt/conda/lib/python3.8/site-packages/pkg_resources/__init__.py:1149: DeprecationWarning:

Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.

/opt/conda/lib/python3.8/site-packages/pkg_resources/__init__.py:1149: DeprecationWarning:

Use of .. or absolute path in a resource path is not allowed and will raise exceptions in a future release.
{'fraction_non_alphanumeric': 0.1314156796390299, 'fraction_numerical': 0.04267719496145892, 'mean_word_length': 3.97196261682243, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 17, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '468095', 'license': 'BSD-2-Clause', 'n_tokens_mistral': 1595, 'n_tokens_neox': 1433, 'n_words': 538, 'path': 'jupyter_demo/cc_algorithms.ipynb', 'repo_name': 'MD2Korg/CerebralCortex-Platform', 'size': '5319'}