<jupyter_start><jupyter_text>Hitchhiker's Guide to Hyperspace (Python) An Indexing Subsystem for Apache Spark™[Hyperspace](https://github.com/microsoft/hyperspace) introduces the ability for Apache Spark™ users to create indexes on their datasets (e.g., CSV, JSON, Parquet etc.) and leverage them for potential query and workload acceleration.In this notebook, we highlight the basics of Hyperspace, emphasizing on its simplicity and show how it can be used by just anyone.**Disclaimer**: Hyperspace helps accelerate your workloads/queries under two circumstances:  1. Queries contain filters on predicates with high selectivity (e.g., you want to select 100 matching rows from a million candidate rows)  2. Queries contain a join that requires heavy-shuffles (e.g., you want to join a 100 GB dataset with a 10 GB dataset)You may want to carefully monitor your workloads and determine whether indexing is helping you on a case-by-case basis.  SetupTo begin with, let's start a new Spark™ session. Since this notebook is a tutorial merely to illustrate what Hyperspace can offer, we will make a configuration change that allow us to highlight what Hyperspace is doing on small datasets. By default, Spark™ uses *broadcast join* to optimize join queries when the data size for one side of join is small (which is the case for the sample data we use in this tutorial). Therefore, we disable broadcast joins so that later when we run join queries, Spark™ uses *sort-merge* join. This is mainly to show how Hyperspace indexes would be used at scale for accelerating join queries.The output of running the cell below shows a reference to the successfully created Spark™ session and prints out '-1' as the value for the modified join config which indicates that broadcast join is successfully disabled.<jupyter_code>import random

session_id = random.randint(0,1000000)
data_path = "/hyperspace/data-{0}".format(session_id)
index_location = "/hyperspace/indexes-{0}".format(session_id)

# Use a random index location to avoid conflicts while using the notebook.
spark.conf.set("spark.hyperspace.system.path", index_location)
# Start your Spark session
spark

# Disable BroadcastHashJoin, so Spark will use standard SortMergeJoin. Currently Hyperspace indexes utilize SortMergeJoin to speed up query.
spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)

# Verify that BroadcastHashJoin is set correctly 
print(spark.conf.get("spark.sql.autoBroadcastJoinThreshold"))<jupyter_output><empty_output><jupyter_text>Data PreparationTo prepare our environment, we will create sample data records and save them as parquet data files. While we use Parquet for illustration, you can use other formats such as CSV. In the subsequent cells, we will also demonstrate how you can create several Hyperspace indexes on this sample dataset and how one can make Spark™ use them when running queries. Our example records correspond to two datasets: *department* and *employee*. You should configure "empLocation" and "deptLocation" paths so that on the storage account they point to your desired location to save generated data files. The output of running below cell shows contents of our datasets as lists of triplets followed by references to dataFrames created to save the content of each dataset in our preferred location.<jupyter_code>from pyspark.sql.types import StructField, StructType, StringType, IntegerType

# Sample department records
departments = [(10, "Accounting", "New York"), (20, "Research", "Dallas"), (30, "Sales", "Chicago"), (40, "Operations", "Boston")]

# Sample employee records
employees = [(7369, "SMITH", 20), (7499, "ALLEN", 30), (7521, "WARD", 30), (7566, "JONES", 20), (7698, "BLAKE", 30)]

# Create a schema for the dataframe
dept_schema = StructType([StructField('deptId', IntegerType(), True), StructField('deptName', StringType(), True), StructField('location', StringType(), True)])
emp_schema = StructType([StructField('empId', IntegerType(), True), StructField('empName', StringType(), True), StructField('deptId', IntegerType(), True)])

departments_df = spark.createDataFrame(departments, dept_schema)
employees_df = spark.createDataFrame(employees, emp_schema)

emp_Location = data_path + "/employees.parquet"
dept_Location = data_path + "/departments.parquet"

employees_df.write.mode("overwrite").parquet(emp_Location)
departments_df.write.mode("overwrite").parquet(dept_Location)<jupyter_output><empty_output><jupyter_text>Let's verify the contents of parquet files we created above to make sure they contain expected records in correct format. We later use these data files to create Hyperspace indexes and run sample queries.Running below cell, the output displays the rows in employee and department dataframes in a tabular form. There should be 14 employees and 4 departments, each matching with one of triplets we created in the previous cell.<jupyter_code># emp_Location and dept_Location are the user defined locations above to save parquet files
emp_DF = spark.read.parquet(emp_Location)
dept_DF = spark.read.parquet(dept_Location)

# Verify the data is available and correct
emp_DF.show()
dept_DF.show()<jupyter_output><empty_output><jupyter_text>Hello Hyperspace Index!Hyperspace lets users create indexes on records scanned from persisted data files. Once successfully created, an entry corresponding to the index is added to the Hyperspace's metadata. This metadata is later used by Apache Spark™'s Hyperspace-enabled optimizer during query processing to find and use proper indexes. Once indexes are created, users can perform several actions:  - **Refresh** If the underlying data changes, users can refresh an existing index to capture that.   - **Delete** If the index is not needed, users can perform a soft-delete i.e., index is not physically deleted but is marked as 'deleted' so it is no longer used in your workloads.  - **Vacuum** If an index is no longer required, users can vacuum it which forces a physical deletion of the index contents and associated metadata completely from Hyperspace's metadata.Below sections show how such index management operations can be done in Hyperspace.First, we need to import the required libraries and create an instance of Hyperspace. We later use this instance to invoke different Hyperspace APIs to create indexes on our sample data and modify those indexes.Output of running below cell shows a reference to the created instance of Hyperspace.<jupyter_code>from hyperspace import *

# Create an instance of Hyperspace
hyperspace = Hyperspace(spark)<jupyter_output><empty_output><jupyter_text>Create IndexesTo create a Hyperspace index, the user needs to provide 2 pieces of information:* An Apache Spark™ DataFrame which references the data to be indexed.* An index configuration object: IndexConfig, which specifies the *index name*, *indexed* and *included* columns of the index. As you might have noticed, in this notebook, we illustrate indexing using the [Covering Index](https://www.red-gate.com/simple-talk/sql/learn-sql-server/using-covering-indexes-to-improve-query-performance/), which is the default index in Hyperspace. In the future, we plan on adding support for other index types. We start by creating three Hyperspace indexes on our sample data: two indexes on the department dataset named "deptIndex1" and "deptIndex2", and one index on the employee dataset named 'empIndex'. For each index, we need a corresponding IndexConfig to capture the name along with columns lists for the indexed and included columns. Running below cell creates these indexConfigs and its output lists them.**Note**: An *index column* is a column that appears in your filters or join conditions. An *included column* is a column that appears in your select/project.For instance, in the following query:```sqlSELECT XFROM TableWHERE Y = 2```Y can be an *index column* and X can be an *included column*.<jupyter_code># Create index configurations

emp_IndexConfig = IndexConfig("empIndex1", ["deptId"], ["empName"])
dept_IndexConfig1 = IndexConfig("deptIndex1", ["deptId"], ["deptName"])
dept_IndexConfig2 = IndexConfig("deptIndex2", ["location"], ["deptName"])<jupyter_output><empty_output><jupyter_text>Now, we create three indexes using our index configurations. For this purpose, we invoke "createIndex" command on our Hyperspace instance. This command requires an index configuration and the dataFrame containing rows to be indexed.Running below cell creates three indexes.<jupyter_code># Create indexes from configurations

hyperspace.createIndex(emp_DF, emp_IndexConfig)
hyperspace.createIndex(dept_DF, dept_IndexConfig1)
hyperspace.createIndex(dept_DF, dept_IndexConfig2)<jupyter_output><empty_output><jupyter_text>List IndexesBelow code shows how a user can list all available indexes in a Hyperspace instance. It uses the `indexes` API which returns information about existing indexes as a Spark™'s DataFrame so you can perform additional operations. For instance, you can invoke valid operations on this DataFrame for checking its content or analyzing it further (for example filtering specific indexes or grouping them according to some desired property). Below cell uses DataFrame's `show` action to fully print the rows and show details of our indexes in a tabular form. For each index, we can see all the information Hyperspace has stored about it in its metadata. You will immediately notice the following:  - `config.indexName`, `config.indexedColumns`, `config.includedColumns` are the fields that a user normally provides during index creation.  - `status.status` indicates if the index is being actively used by the Spark's optimizer.  - `dfSignature` is automatically generated by Hyperspace and is unique for each index. Hyperspace uses this signature internally to maintain the index and exploit it at query time.   In the output below, all three indexes should have "ACTIVE" as status and their name, indexed columns, and included columns should match with what we defined in index configurations above.<jupyter_code>hyperspace.indexes().show()<jupyter_output><empty_output><jupyter_text>Delete IndexesA user can drop an existing index by using the `deleteIndex` API and providing the index name. Index deletion is a **soft-delete** operation i.e., only the index's status in the Hyperspace metadata from is changed from "ACTIVE" to "DELETED". This will exclude the deleted index from any future query optimization and Hyperspace no longer picks that index for any query. However, index files for a deleted index still remain available (since it is a soft-delete), so if you accidentally deleted the index, you could still restore it.The cell below deletes index with name "deptIndex2" and lists Hyperspace metadata after that. The output should be similar to above cell for "List Indexes" except for "deptIndex2" which now should have its status changed into "DELETED".<jupyter_code>hyperspace.deleteIndex("deptIndex2")

hyperspace.indexes().show()<jupyter_output><empty_output><jupyter_text>Restore IndexesA user can use the `restoreIndex` API to restore a deleted index. This will bring back the latest version of index into ACTIVE status and makes it usable again for queries. The cell below shows an example of `restoreIndex` API. We delete "deptIndex1" and restore it. The output shows "deptIndex1" first went into the "DELETED" status after invoking "deleteIndex" command and came back to the "ACTIVE" status after calling "restoreIndex".<jupyter_code>hyperspace.deleteIndex("deptIndex1")

hyperspace.indexes().show()

hyperspace.restoreIndex("deptIndex1")

hyperspace.indexes().show()<jupyter_output><empty_output><jupyter_text>Vacuum IndexesThe user can perform a **hard-delete** i.e., fully remove files and the metadata entry for a deleted index using the `vacuumIndex` API. Once done, this action is **irreversible** as it physically deletes all the index files associated with the index.The cell below vacuums the "deptIndex2" index and shows Hyperspace metadata after vaccuming. You should see metadata entries for two indexes "deptIndex1" and "empIndex" both with "ACTIVE" status and no entry for "deptIndex2".<jupyter_code>hyperspace.vacuumIndex("deptIndex2")
hyperspace.indexes().show()<jupyter_output><empty_output><jupyter_text>Enable/Disable HyperspaceHyperspace provides APIs to enable or disable index usage with Spark™.  - By using `enableHyperspace` API, Hyperspace optimization rules become visible to the Apache Spark™ optimizer and it will exploit existing Hyperspace indexes to optimize user queries.  - By using `disableHyperspace` command, Hyperspace rules no longer apply during query optimization. You should note that disabling Hyperspace has no impact on created indexes as they remain intact.Below cell shows how you can use these commands to enable or disable hyperspace. The output simply shows a reference to the existing Spark™ session whose configuration is updated.<jupyter_code># Enable Hyperspace
Hyperspace.enable(spark)

# Disable Hyperspace
Hyperspace.disable(spark)<jupyter_output><empty_output><jupyter_text>Index UsageIn order to make Spark use Hyperspace indexes during query processing, the user needs to make sure that Hyperspace is enabled. The cell below enables Hyperspace and creates two DataFrames containing our sample data records which we use for running example queries. For each DataFrame, a few sample rows are printed.<jupyter_code># Enable Hyperspace
Hyperspace.enable(spark)

emp_DF = spark.read.parquet(emp_Location)
dept_DF = spark.read.parquet(dept_Location)

emp_DF.show(5)
dept_DF.show(5)<jupyter_output><empty_output><jupyter_text>Hyperspace's Index TypesCurrently, Hyperspace can exploit indexes for two groups of queries: * Selection queries with lookup or range selection filtering predicates.* Join queries with an equality join predicate (i.e. Equi-joins).  Indexes for Accelerating FiltersOur first example query does a lookup on department records (see below cell). In SQL, this query looks as follows:```sqlSELECT deptName FROM departmentsWHERE deptId = 20```The output of running the cell below shows: - query result, which is a single department name.- query plan that Spark™ used to run the query. In the query plan, the "FileScan" operator at the bottom of the plan shows the datasource where the records were read from. The location of this file indicates the path to the latest version of the "deptIndex1" index. This shows  that according to the query and using Hyperspace optimization rules, Spark™ decided to exploit the proper index at runtime.<jupyter_code># Filter with equality predicate

eqFilter = dept_DF.filter("""deptId = 20""").select("""deptName""")
eqFilter.show()

hyperspace.explain(eqFilter, True)<jupyter_output><empty_output><jupyter_text>Our second example is a range selection query on department records. In SQL, this query looks as follows:```sqlSELECT deptName FROM departmentsWHERE deptId > 20```Similar to our first example, the output of the cell below shows the query results (names of two departments) and the query plan. The location of data file in the FileScan operator shows that 'deptIndex1" was used to run the query.<jupyter_code># Filter with range selection predicate

rangeFilter = dept_DF.filter("""deptId > 20""").select("deptName")
rangeFilter.show()

hyperspace.explain(rangeFilter, True)<jupyter_output><empty_output><jupyter_text>Our third example is a query joining department and employee records on the department id. The equivalent SQL statement is shown below:```sqlSELECT employees.deptId, empName, departments.deptId, deptNameFROM   employees, departments WHERE  employees.deptId = departments.deptId```The output of running the cell below shows the query results which are the names of 14 employees and the name of department each employee works in. The query plan is also included in the output. Notice how the file locations for two FileScan operators shows that Spark used "empIndex" and "deptIndex1" indexes to run the query.<jupyter_code># Join

eqJoin = emp_DF.join(dept_DF, emp_DF.deptId == dept_DF.deptId).select(emp_DF.empName, dept_DF.deptName)

eqJoin.show()

hyperspace.explain(eqJoin, True)<jupyter_output><empty_output><jupyter_text>Support for SQL SemanticsThe index usage is transparent to whether the user uses DataFrame API or Spark™ SQL. The following example shows the same join example as before but using Spark SQL, showing the use of indexes if applicable.<jupyter_code>from pyspark.sql import SparkSession

emp_DF.createOrReplaceTempView("EMP")
dept_DF.createOrReplaceTempView("DEPT")

joinQuery = spark.sql("SELECT EMP.empName, DEPT.deptName FROM EMP, DEPT WHERE EMP.deptId = DEPT.deptId")

joinQuery.show()
hyperspace.explain(joinQuery, True)<jupyter_output><empty_output><jupyter_text>Explain APISo far, you might have observed we have been using the explain API provided by Hyperspace. The `explain` API from Hyperspace is very similar to Spark's `df.explain` API but allows users to compare their original plan vs the updated index-dependent plan before running their query. You have an option to choose from html/plaintext/console mode to display the command output<jupyter_code>eqJoin = emp_DF.join(dept_DF, emp_DF.deptId == dept_DF.deptId).select(emp_DF.empName, dept_DF.deptName)

hyperspace.explain(eqJoin, True)<jupyter_output><empty_output><jupyter_text>Refresh IndexesIf the original data on which an index was created changes, then the index will no longer capture the latest state of data and hence will not be used by Hyperspace to provide any acceleration. The user can refresh such a stale index using the `refreshIndex` API. This causes the index to be fully rebuilt and updates it according to the latest data records.        Spoiler alert: if you are worried about fully rebuilding your index every time your data changes, don't worry! We will show you how to *incrementally refresh* your index in subsequent cells below.The two cells below show an example for this scenario:- First cell adds two more departments to the original departments data. It reads and prints list of departments to verify new departments are added correctly. The output shows 6 departments in total: four old ones and two new. Invoking "refreshIndex" updates "deptIndex1" so index captures new departments.- Second cell runs our range selection query example. The results should now contain four departments: two are the ones, seen before when we ran the query above, and two are the new departments we just added.<jupyter_code>extra_Departments = [(50, "Inovation", "Seattle"), (60, "Human Resources", "San Francisco")]

extra_departments_df = spark.createDataFrame(extra_Departments, dept_schema)
extra_departments_df.write.mode("Append").parquet(dept_Location)


dept_DFrame_Updated = spark.read.parquet(dept_Location)

dept_DFrame_Updated.show(10)

hyperspace.refreshIndex("deptIndex1")
newRangeFilter = dept_DFrame_Updated.filter("deptId > 20").select("deptName")
newRangeFilter.show()

hyperspace.explain(newRangeFilter, True)
hyperspace.indexes().show()
# Clean-up the remaining indexes
hyperspace.deleteIndex("empIndex1")
hyperspace.deleteIndex("deptIndex1")

hyperspace.vacuumIndex("empIndex1")
hyperspace.vacuumIndex("deptIndex1")<jupyter_output><empty_output><jupyter_text>Hybrid Scan for Mutable DatasetsOften times, if your underlying source data had some new files appended or existing files deleted, your index will get stale and Hyperspace decides not to use it. However, there are times where you just want to use the index without having to refresh it everytime. There could be multiple reasons for doing so:  1. You do not want to continuosly refresh your index but instead want to do it periodically since you understand your workloads the best.    2. You added/removed only a few files and do not want to wait for yet another refresh job to finish. To allow you to still use a stale index, Hyperspace introduces **Hybrid Scan**, a novel technique that allows users to utilize outdated or stale indexes (e.g., the underlying source data had some new files appended or existing files deleted), without refreshing indexes. To achieve this, when you set the appropriate configuration to enable Hybrid Scan, Hyperspace modifies the query plan to leverage the changes as following:- Appended files can be merged to index data by using **`Union` or `BucketUnion` (for join)**. Shuffling appended data can also be applied before merging, if needed.- Deleted files can be handled by injecting `Filter-NOT-IN` condition on **lineage column** of index data, so that the indexed rows from the deleted files can be excluded at query time. You can check the transformation of the query plan in below examples.  Hybrid Scan for appended files - non-partitioned dataNon-partitioned data is used in below example. In this example, we expect Join index can be used for the query and `BucketUnion` is introduced for appended files.<jupyter_code># GENERATE TEST DATA

testdata = [
    ("orange", 3, "2020-10-01"),
    ("banana", 1, "2020-10-01"),
    ("carrot", 5, "2020-10-02"),
    ("beetroot", 12, "2020-10-02"),
    ("orange", 2, "2020-10-03"),
    ("banana", 11, "2020-10-03"),
    ("carrot", 3, "2020-10-03"),
    ("beetroot", 2, "2020-10-04"),
    ("cucumber", 7, "2020-10-05"),
    ("pepper", 20, "2020-10-06")
]

testdata_location = data_path + "/productTable"
from pyspark.sql.types import StructField, StructType, StringType, IntegerType
testdata_schema = StructType([
    StructField('name', StringType(), True),
    StructField('qty', IntegerType(), True),
    StructField('date', StringType(), True)])

test_df = spark.createDataFrame(testdata, testdata_schema)
test_df.write.mode("overwrite").parquet(testdata_location)
test_df = spark.read.parquet(testdata_location)
# CREATE INDEX
hyperspace.createIndex(test_df, IndexConfig("productIndex2", ["name"], ["date", "qty"]))

spark.conf.set("spark.sql.autoBroadcastJoinThreshold", -1)
filter1 = test_df.filter("name = 'banana'")
filter2 = test_df.filter("qty > 10")
query = filter1.join(filter2, "name")

# Check Join index rule is applied properly.
hyperspace.explain(query, True)
# Append new files.
append_data = [
    ("orange", 13, "2020-11-01"),
    ("banana", 5, "2020-11-01")
]
append_df = spark.createDataFrame(append_data, testdata_schema)
append_df.write.mode("append").parquet(testdata_location)<jupyter_output><empty_output><jupyter_text>Hybrid scan is disabled by default. Therefore, you will see that since we appended new data, Hyperspace will decide NOT to use the index.In the output, you will see no plan differences (hence no highlighting).<jupyter_code># Hybrid Scan configs are false by default.
spark.conf.set("spark.hyperspace.index.hybridscan.enabled", "false")

test_df_with_append = spark.read.parquet(testdata_location)
filter1 = test_df_with_append.filter("name = 'banana'")
filter2 = test_df_with_append.filter("qty > 10")
query = filter1.join(filter2, "name")
hyperspace.explain(query, True)
query.show()<jupyter_output><empty_output><jupyter_text>Enable Hybrid Scan

In plan with indexes, you can see
`Exchange hashpartitioning` required only for appended files so that we could still utilize the "shuffled" index data with appended files. `BucketUnion` is used to merge "shuffled" appended files with the index data.<jupyter_code># Enable Hybrid Scan config.
spark.conf.set("spark.hyperspace.index.hybridscan.enabled", "true")
# Adjust the ratio of appended data threshold if needed. It's 0.3 (30%) by default.
spark.conf.set("spark.hyperspace.index.hybridscan.maxAppendedRatio", "0.9")

# Need to redefine query to recalculate the query plan.
query = filter1.join(filter2, "name")
hyperspace.explain(query, True)
query.show()<jupyter_output><empty_output><jupyter_text>Incremental Index RefreshWhen you ready to update your indexes but do not want to rebuild your entire index, Hyperspace supports updating indexes in an incremental manner using `hs.refreshIndex("name", "incremental")` API. This will allow eliminate the need for a full rebuild of index from scratch, utilizing previously created index files as well as updating indexes on only the newly added data.Of course, please be sure to use the complementary `optimizeIndex` API (shown below) periodically to make sure you do not see performance regressions. We recommend calling `optimize` at least once for every 10 times you call `refreshIndex(..., "incremental")`, assuming the data you added/removed is < 10% of the original dataset. For instance, if your original dataset is 100 GB, and you've added/removed data in increments/decrements of 1 GB, you can call `refreshIndex` 10 times before calling `optimizeIndex`. Please note that this example is simply used for illustration and you have to adapt this for your workloads.In the example below, notice the addition of a `Sort` node in the query plan when indexes are used. This is because partial indexes are created on the appended data files, causing Spark to introduce a `Sort`. Please also note that `Shuffle` i.e. `Exchange` is still eliminated from the plan, giving you the appropriate acceleration.<jupyter_code>def query():
    test_df_with_append = spark.read.parquet(testdata_location)
    filter1 = test_df_with_append.filter("name = 'banana'")
    filter2 = test_df_with_append.filter("qty > 10")
    return filter1.join(filter2, "name")
hyperspace.refreshIndex("productIndex2", "incremental")

hyperspace.explain(query(), True)
query().show()<jupyter_output><empty_output><jupyter_text>Optimize Index layoutAfter calling incremental refreshes multiple times on newly appended data (e.g. if the user writes to data in small batches or in case of streaming scenarios), the number of index files tend to become large affecting the performance of the index (large number of small files problem). Hyperspace provides `hyperspace.optimizeIndex("indexName")` API to optimize the index layout and reduce the large files problem.In the plan below, notice that Hyperspace has removed the additional `Sort` node in the query plan. Optimize can help avoiding sorting for any index bucket which contains only one file. However, this will only be true if ALL the index buckets have at most 1 file per bucket, after `optimizeIndex`.<jupyter_code># Append some more data and call refresh again.
append_data = [
    ("orange", 13, "2020-11-01"),
    ("banana", 5, "2020-11-01")
]
append_df = spark.createDataFrame(append_data, testdata_schema)
append_df.write.mode("append").parquet(testdata_location)

hyperspace.refreshIndex("productIndex2", "incremental")
# Call optimize. Ensure that Sort is removed after optimization (This is possible here because after optimize, in this case, every bucket contains only 1 file.).
hyperspace.optimizeIndex("productIndex2")

hyperspace.explain(query(), True)<jupyter_output><empty_output><jupyter_text>Optimize modesThe default mode for optimization is "quick" mode where files smaller than a predefined threshold are picked for optmization. To maximize the effect of optimization, Hyperspace allows another optimize mode "full" as shown below. This mode picks ALL index files for optimization irrespective of their file size and creates the best possible layout of the index. This is also slower than the default optimize mode because more data is being processed here.<jupyter_code>hyperspace.optimizeIndex("productIndex2", "full")

hyperspace.explain(query(), True)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0740808287131602, 'fraction_numerical': 0.010650714911000876, 'mean_word_length': 5.728098159509202, 'pattern_counts': {'":': 0, '<': 98, '<?xml version=': 0, '>': 103, 'https://': 2, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '256370', 'license': 'Apache-2.0', 'n_tokens_mistral': 7756, 'n_tokens_neox': 7194, 'n_words': 3323, 'path': 'notebooks/python/Hitchhikers Guide to Hyperspace.ipynb', 'repo_name': 'wentaowu/hyperspace', 'size': '27416'}