<jupyter_start><jupyter_text>PreliminariesThis notebook assumes that all setup steps from [GCP-readme.md](./GCP-readme.md) have been followed.This notebook does not demonstrate the full ETL pipeline used in production (executed on GCP Dataflow).We only investigate the first couple of major steps of the *production* pipeline (which is executed in Dataflow on the Google Cloud Platform) in order to demonstrate the general idea of the process.  Note that the *production* pipeline is really a sequence of (sub) pipelines that are daisychained together in a particular order, since latter pipelines depend on former pipelines.In this notebook, we demonstrate the first two pipelines, which accomplish the following: 1. Boostrap the video indexSubsteps are:    1. Download the video index (archive)    2. Extract it.    3. Write it to the destination directory as a CSVThe video index drives the entire full-blown *production* pipeline.It tells us the filenames of the target videos. 2. Download each video segment comprising final target videos. (The video index contains the URLs).Target videos are comprised of video segments since some of the final target videos can be rather large.Altgether, **the *production* pipeline (executed on GCP Dataflow) retrieves more than 2600 videos** produced by the research conducted by Boston and Rutgers Universities, jointly under the [National Center for Sign Language and Gesture Resources project](http://www.bu.edu/asllrp/ncslgr.html).This notebook will demonstrate retrieving 50 of those.The implementation of the download shall leverage Apache Beam's parallelism in order to avoid the amount of time it would take to accomplish doing it sequentially.  Note that when executed locally, the Apache Beam SDK uses Docker containers for worker node clusters.  A cluster in this case consists of 8 workers nodes since my local machine has 8 cores.In *production*, on GCP Dataflow, this can be scheduled to your heart's content (but this, of course, costs more money to do so). 3. Use the `OpenCV` library to extract all frames (from each segment) for each target video.This step leverages Apache Beam's parallelism as well.But we MUST take care to ensure that a single worker extracts the frames of each segment associated with the target video.This is because frames are ordered/sequenced.  Allowing two different workers to extract frames of different segments associated with the same final target video would likely result in frames being extracted out of order (due to parallelism).Therefore, we partition the extraction task by final target video in order to ensure a single worker handles all segments associated with a single target video.But we do want parallelism to occurr at the final target video level.In the end, **in production, the pipeline extracts more than 561,000 frames (images) from the source target videos**!Since these images are Of course in this demonstration we will be extracting much less - only 50 out of the more than 2600 videos available will be downloaded and processed (frames extracted).  Still, extracting from 50 videos will amount to thousands of frames.<jupyter_code>%load_ext autoreload

%autoreload 2

from __future__ import absolute_import

import apache_beam as beam
import apache_beam.runners.interactive.interactive_beam as ib

from api import beam__common, fileio, fidscs_globals
from api.fidscs_globals import disp_source
from api import data_extractor__beam

from apache_beam.options.pipeline_options import PipelineOptions<jupyter_output><empty_output><jupyter_text>Constants to be used in this notebook<jupyter_code>WORK_DIR = '/tmp'
MAX_TARGET_VIDEOS = 50  # set to -1 for all in production but not in this interactive notebook! That will result in extracting more than 561,000 images from more than 2600 videos! (onto your local machine)
PIPELINE_BASE_JOB_NAME = 'sc-fids-capstone-etl-demo'<jupyter_output><empty_output><jupyter_text>Use Apache Beam PipelineOptions for any global settingsWe MUST do this since the point of Apache Beam is to enable parallelism (in processing).How this is accomplished is beyond the scope of this notebook.But suffice it to say that any notion of a global variable cannot be implemented in the manner one is normally implemented - e.g. with Python global variables.However, a PipelineOptions object IS passed to each and every worker node by Apache Beam.Therefore, we accomplish global settings to be shared by all workers - e.g. the working directory and the final destination filepaths to be output by the pipeline - by passing would-be global settings to PipelineOptions, which are required to bootstrap each worker node by Apache Beam.  Custom Apache Beam Pipeline optionsThe `beam__common.FIDSCapstonePipelineOptions` class was written to do just that and allows us to create and use our own custom options in Apache Beam pipelines.Without it, attempting to set custom options on the Pipeline will fail since Apache Beam's PipelineOptions class will reject any options it doesn't recognize.<jupyter_code>disp_source(beam__common.FIDSCapstonePipelineOptions)<jupyter_output><empty_output><jupyter_text>PipelineOptions InitializationFor this notebook, we execute locally (vs. GCP Dataflow) - that is, we use the Apache Beam's `DirectRunner`.  Actually, we use a variant of - the `InteractiveRunner` - geared specifically for running in notebooks.  But it is still run locally.  Some `PipelineOptions` options differ (or are not needed), relative to the `DataflowRunner`.To see the full implementation on how this differs from using the `Dataflow` runner, start by inspecting [run_cloud__etl.py](./run_cloud__etl.py) and follow the code.Initializing the `dict` upon which `PipelineOptions` are based has been wrapped up within the `beam__common.make_fids_options_dict` function.<jupyter_code>disp_source(beam__common.make_fids_options_dict)<jupyter_output><empty_output><jupyter_text>PipelineOptions geared for the `InteractiveRunner`Note that `InteractiveRunner` is a variant of the `DirectRunner` that allows us to run an Apache Beam pipeline with a Jupyter Notebook.Documentation for `InteractiveRunner` can be found [here](https://cloud.google.com/dataflow/docs/guides/interactive-pipeline-development).First, it must be reiterated that we must use this runner in order to collect (reduce) data kept in Apache Beam `Pcollection`s for conversion to Pandas `DataFrame`s for display within this notebook.  Apache Beam `Pcollection`s can generally and *roughly* be thought of as Resilient Distributed Datasets.  The documentation for Apache Beam `Pcollection`s can be found in the **Apache Beam Programming Guide** located [here](https://beam.apache.org/documentation/programming-guide/).  But **`Pcollection`s are the basis for all processing within an Apache Beam pipeline**.Also note that the `InteractiveRunner` is not really meant to be used for enterprise (read: "Big Data") pipelines.The runner used for production in this project is the `DataFlow` Google Cloud Platform runner.The reader is reminded that the point of this notebook, however, is to present a demonstration of only a subset of the full Apache Beam pipeline (used in this project).<jupyter_code>options = {
    # 'runner': 'DirectRunner',
    'runner': 'InteractiveRunner',
    'environment_type': 'DOCKER',
    'direct_num_workers': 0, # 0 is use all available cores
    'direct_running_mode': 'multi_threading', # ['in_memory', 'multi_threading', 'multi_processing']
    'streaming': False # set to True if data source is unbounded (e.g. GCP PubSub),
}

options.update(beam__common.make_fids_options_dict(WORK_DIR, max_target_videos=MAX_TARGET_VIDEOS))<jupyter_output><empty_output><jupyter_text>Finally, instantiate the `PipelineOptions` (using the above `options` `dict`)<jupyter_code>job_suffix = 'boostrap-vid-index'
job_name = f"{PIPELINE_BASE_JOB_NAME}--{job_suffix}"
options.update({
    'job_name': job_name
})

pipeline_options = PipelineOptions(flags=[], **options) # easier to pass in options from command-line this way
print(f"PipelineOptions:\n{pipeline_options.get_all_options()}\n")<jupyter_output>PipelineOptions:
{'runner': 'InteractiveRunner', 'streaming': False, 'beam_services': {}, 'type_check_strictness': 'DEFAULT_TO_ANY', 'type_check_additional': '', 'pipeline_type_check': True, 'runtime_type_check': False, 'performance_runtime_type_check': False, 'direct_runner_use_stacked_bundle': True, 'direct_runner_bundle_repeat': 0, 'direct_num_workers': 0, 'direct_running_mode': 'multi_threading', 'dataflow_endpoint': 'https://dataflow.googleapis.com', 'project': 'sc-fids-capstone', 'job_name': 'sc-fids-capstone-etl-demo--boostrap-vid-index', 'staging_location': None, 'temp_location': None, 'region': None, 'service_account_email': None, 'no_auth': False, 'template_location': None, 'labels': None, 'update': False, 'transform_name_mapping': None, 'enable_streaming_engine': False, 'dataflow_kms_key': None, 'flexrs_goal': None, 'hdfs_host': None, 'hdfs_port': None, 'hdfs_user': None, 'hdfs_full_urls': False, 'num_workers': None, 'max_num_workers': None, 'autoscaling_algorithm': None, 'm[...]<jupyter_text>But before running the pipeline, create necessary filestructure within `WORK_DIR`<jupyter_code>if not fileio.dir_path_exists(options[fidscs_globals.OPT_NAME_DATA_DIR], options)[0]:
    fileio.make_dirs(options[fidscs_globals.OPT_NAME_DATA_DIR], options)

if not fileio.dir_path_exists(options[fidscs_globals.OPT_NAME_TMP_DIR], options)[0]:
    fileio.make_dirs(options[fidscs_globals.OPT_NAME_TMP_DIR], options)

if not beam__common.dataset_csv_files_exist(options):
    if not fileio.dir_path_exists(options[fidscs_globals.OPT_NAME_VIDEO_DIR], options)[0]:
        fileio.make_dirs(options[fidscs_globals.OPT_NAME_VIDEO_DIR], options)

if not fileio.dir_path_exists(options[fidscs_globals.OPT_NAME_STITCHED_VIDEO_FRAMES_DIR], options)[0]:
    fileio.make_dirs(options[fidscs_globals.OPT_NAME_STITCHED_VIDEO_FRAMES_DIR], options)<jupyter_output><empty_output><jupyter_text>We are now ready to execute the pipeline.  But before doing so, let's discuss how it works.There are two top-level functions used by the "boostrap-vid-index" pipeline, in this order:1. `data_extractor__beam.pl__1__bootstrap_target_video_index`2. `data_extractor__beam.pl__2__write_target_vid_index_csv`Let's examine the source code for `data_extractor__beam.pl__1__bootstrap_target_video_index`...The follow python source code illustrates the programming paradigm used in all Apache Beam (stands for **B**atch and Str**eam** processing) pipelines.<jupyter_code>disp_source(data_extractor__beam.pl__1__bootstrap_target_video_index)<jupyter_output><empty_output><jupyter_text>The gist of `data_extractor__beam.pl__1__bootstrap_target_video_index` is that it will ensure that the video index exists locally before any other dependent pipeline can execute.If it doesn't, it will download download/extract the contents of the video index archive from [http://www.bu.edu/asllrp/ncslgr-for-download/video_index-20120129.zip](http://www.bu.edu/asllrp/ncslgr-for-download/zip).The first noteable point to make is that it uses the custom class `data_extractor__beam.TargetVideoIndexBootstrapper`, which inherits `beam__common.PipelinePcollElementProcessor`, which in turn inherits from from Apache Beam's `DoFn` class.  Inheriting from Apache Beam's `DoFn` allows the inherited class to be used in Apache Beam pipelines via `beam.ParDo` (which stands for "**Par**allel **Do**").  Full documentation can be found [here](https://beam.apache.org/documentation/transforms/python/elementwise/pardo/).There is nothing particularly noteworthy about the internal implementation of `data_extractor__beam.TargetVideoIndexBootstrapper`.  It simply downloads the video index archive (to a memfile) and extracts its contents (in-memory).  Please see its implementation for details if you are interested.Source code for `data_extractor__beam.pl__2__write_target_vid_index_csv` is listed below.  It simply writes the bytes extracted from the archive to desintation path `/data/video_index-20120129.csv` (using an Apache Beam `schema` so that column names can easily be referenced/manipulated later).<jupyter_code>disp_source(data_extractor__beam.pl__2__write_target_vid_index_csv)<jupyter_output><empty_output><jupyter_text>We are now ready to execute the first step of the "boostrap-vid-index" pipeline!<jupyter_code>n_partitions = 8 # hardcoded for now but we need to retrieve this from beam to be the number of workers

pl = beam.Pipeline(options=pipeline_options)

full_target_vid_index_schemad_pcoll = data_extractor__beam.pl__1__bootstrap_target_video_index(pl)<jupyter_output><empty_output><jupyter_text>That seems fast!  That's because the pipeline wasn't acctually executed yet.  What Apache Beam did in this case was create the corresponding pipelines *execution graph* (which is actually a *Directed Acyclic Graph*).With the `InteractiveRunner`, the pipeline only gets executed when it is required.This happens in notebooks by calling `ib.collect` or `ib.show`, which essentially executes and then reduces the distributed `Pcollection`.In this case, we use `ib.collect` which also stuffs the results into a Pandas `DataFrame` for viewing purposes within notebooks.Note that this is NOT done in production (in the cloud, in GCP Dataflow) since Pandas `DataFrame`s aren't needed and are simply impractical for "Big Data" solutions.  Pandas `DataFrame`s really don't serve this purpose.  Can you imagine attempting to hold all the corresponding tensor bytes for 561,000+ images in memory??Anyway, moving on... before calling `ib.collect`, we must first tell Apache Beam to "record" all `Pcollection`s (up to a certain point) by calling `ib.watch(locals())`.  Note that only `Pcollection`s prior to calling `ib.watch(locals())` are eligible for "collection" (conversion to Pandas `DataFrame`s).<jupyter_code># we require this in order to make use of ib.show() (which provides visualization of the pcolls specified) or ib.collect() (which creates a pandas dataframe from a pcoll)
    # but all pcolls we wish to visualize must be created prior to executing the following line
ib.watch(locals())<jupyter_output><empty_output><jupyter_text>We can now collect the `full_target_vid_index_schemad_pcoll` `Pcollection` into a Pandas `DataFrame` for display in this notebook.<jupyter_code>df_full_target_vid_index = ib.collect(full_target_vid_index_schemad_pcoll)
df_full_target_vid_index<jupyter_output><empty_output><jupyter_text>Note that this isn't entirely useful yet since we don't have any corresponding column names (in the above Pandas `DataFrame`).  We have applied a `schema` to the `Pcollection` but that doesn't get applied to the Pandas `DataFrame` since applying a `schema` to a `Pcollection` is carried out by mappinng each row to a literal Apache Beam `Row` object, thereby effectively converting each element to an *unhashed* `dict`.  Thus, we cannot guarantee the ordering of the columns will be fixed.  We must therefore use the `schema` to refer to columns by name.But, we do see that there are 2,612 corresponding target videos to download.  Note that since target videos are actually comprised of segments, there may actually be more videos than that that we download in the end (if we were to download them all... which is exactly what is done in production, on GCP Dataflow).This is done inline while writing the `Pcollection` (collected into the above Pandas `DataFrame` just for viewing) to the destination `/data/video_index-20120129.csv` file path (by `data_extractor__beam.pl__2__write_target_vid_index_csv`).But, as a nuance of collecting a `Pcollection` into a `DataFrame`, we can't simply call `data_extractor__beam.pl__2__write_target_vid_index_csv` now if we want to view the resulting `Pcollection` as a `DataFrame`.  Recall that only `Pcollection`s prior to calling `ib.watch(locals())` are eligible for "collection" (conversion to Pandas `DataFrame`s), which we already did.  This means we must re-execute the first step (`data_extractor__beam.pl__1__bootstrap_target_video_index`), followed by `data_extractor__beam.pl__2__write_target_vid_index_csv`, call `ib.watch(locals())`, and then finally call `ib.collect` on each of the corresponding `Pcollection`s in order to view them.But won't that mean that `data_extractor__beam.pl__1__bootstrap_target_video_index` will re-download the video index?  ANSWER: no because it was written specifically to guard againstt that case.  Take a look at its source and you'll see.  If the video index exists locally, it will simply load it from the "tmp" directory and the resulting `Pcollection` is used as input for `data_extractor__beam.pl__2__write_target_vid_index_csv` (which will apply a `schema` and then write it to the final destination path `/data/video_index-20120129.csv`).Let's do that now... The full "boostrap-vid-index" pipeline<jupyter_code># create a new instance of the pipeline
pl = beam.Pipeline(options=pipeline_options)

full_target_vid_index_schemad_pcoll = data_extractor__beam.pl__1__bootstrap_target_video_index(pl)
_ = data_extractor__beam.pl__2__write_target_vid_index_csv(full_target_vid_index_schemad_pcoll, pl._options._all_options)<jupyter_output><empty_output><jupyter_text>We know that observing the `full_target_vid_index_schemad_pcoll` `Pcollection` won't be particularly useful and the `Pcollection` that `data_extractor__beam.pl__2__write_target_vid_index_csv` outputs simply has the destination path after it successfully writes `full_target_vid_index_schemad_pcoll` to `/data/video_index-20120129.csv`, which isn't particularly interesting.  But we need to be sure this pipeline completes before executing the more interesting "download-videos-extract-frames" pipeline.So instead of calling `ib.collect` to force the above pipeline to run, we'll simply call `pl.run` instead (since we are not particularly interested in viewing any `Pcollection`-to-`DataFrame` conversions from it).<jupyter_code>print(f"\n\n****************************** Starting pipeline job: {job_name} ******************************")
pl.run();
print(f"****************************** Finished pipeline job: {job_name} ******************************")<jupyter_output>****************************** Starting pipeline job: sc-fids-capstone-etl-demo--boostrap-vid-index ******************************
FOUND EXISTING SEL VID INDEX: /tmp/data/tmp/video_index-20120129/files_by_video_name.csv
TARGET-VIDEO-INDEX CSV WRITTEN TO STORAGE: /tmp/data/video_index-20120129.csv
****************************** Finished pipeline job: sc-fids-capstone-etl-demo--boostrap-vid-index ******************************<jupyter_text>The "download-videos-extract-frames" pipelineThe "download-videos-extract-frames" pipeline is comprised of four steps:1. `beam__common.pl__1__read_target_vid_index_csv`2. `data_extractor__beam.pl__2__filter_target_vid_index`3. `data_extractor__beam.pl__3__parallel_download_videos`4. `data_extractor__beam.pl__4__parallel_extract_target_video_frames` The function names used for each step suggest what they do.  So I will only show source code for `data_extractor__beam.pl__3__parallel_download_videos` and `data_extractor__beam.pl__4__parallel_extract_target_video_frames`, and provide short explanations for steps 1 and 2.Step 1 obviously reads `/data/video_index-20120129.csv` from storage into a `Pcollection` to be used as input for `data_extractor__beam.pl__2__filter_target_vid_index`, which simply selects the first `MAX_TARGET_VIDEOS` from the full list of records from the `full_target_vid_index_schemad_pcoll` `Pcollection` that `beam__common.pl__1__read_target_vid_index_csv` returns.  Note that `data_extractor__beam.pl__2__write_target_vid_index_csv`, in addition to applying a `schema`, also applies a row *id* and writes to `/data/video_index-20120129.csv` in the order of that index.  `beam__common.pl__1__read_target_vid_index_csv` returns the corresponding `Pcollection` ordered by this index.Let's now inspect source code for steps 3 and 4...<jupyter_code>disp_source(data_extractor__beam.pl__3__parallel_download_videos)<jupyter_output><empty_output><jupyter_text>Now things get really interesting with `data_extractor__beam.pl__3__parallel_download_videos`...What we do here is explicitly tell Apache Beam to create 8 independent *partitions*, each of which will download videos independently of one another, corresponding to worker nodes.Note that either threads or worker nodes.  How that plays out is beyond the scope of this notebook.  Suffice it to say that this results in much faster processing than simply executing sequentially.When they are all done, the results are merged (via `beam.Flatten`) into a single `Pcollection` to be supplied as input to `data_extractor__beam.pl__4__parallel_extract_target_video_frames`.<jupyter_code>disp_source(data_extractor__beam.pl__4__parallel_extract_target_video_frames)<jupyter_output><empty_output><jupyter_text>From the first part of this notebook...This step leverages Apache Beam's parallelism as well.But we MUST take care to ensure that a single worker extracts the frames of each segment associated with the target video.This is because frames are ordered/sequenced.  Allowing two different workers to extract frames of different segments associated with the same final target video would likely result in frames being extracted out of order (due to parallelism).Therefore, we partition the extraction task by final target video in order to ensure a single worker handles all segments associated with a single target video.But we do want parallelism to occurr at the final target video level.Before creating the pipeline execution graph, it is worth taking a deeper look into the internals of how we use the `OpenCV` library to process the videos (extract frames).The `data_extractor__beam.SegmentFrameExtractor` wraps the `data_extractor__beam.beam_extract_frames`, which houses the logic for this processing.  There are also a couple of helper functions that `` uses: `data_extractor__beam.capture_segment_video` and `data_extractor__beam.write_frame_to_file`.  These will be listed after `data_extractor__beam.beam_extract_frames`.<jupyter_code>disp_source(data_extractor__beam.beam_extract_frames)
disp_source(data_extractor__beam.capture_segment_video)
disp_source(data_extractor__beam.write_frame_to_file)<jupyter_output><empty_output><jupyter_text>We are now ready to execute the "download-videos-extract-frames" pipeline.  But first we must... Create the "download-videos-extract-frames" pipeline execution graph<jupyter_code>job_suffix = 'download-videos-extract-frames'
job_name = f"{PIPELINE_BASE_JOB_NAME}--{job_suffix}"
options.update({
    'job_name': job_name
})
pipeline_options = PipelineOptions(flags=[], **options) # easier to pass in options from command-line this way
print(f"PipelineOptions:\n{pipeline_options.get_all_options()}\n")

pl = beam.Pipeline(options=pipeline_options)

full_target_vid_index_schemad_pcoll = beam__common.pl__1__read_target_vid_index_csv(pl)
filtered_target_vid_index_schemad_pcoll = data_extractor__beam.pl__2__filter_target_vid_index(full_target_vid_index_schemad_pcoll, pl._options._all_options)
merged_download_results = data_extractor__beam.pl__3__parallel_download_videos(filtered_target_vid_index_schemad_pcoll, pl._options._all_options, n_partitions)
merged_extraction_results = data_extractor__beam.pl__4__parallel_extract_target_video_frames(merged_download_results, pl._options._all_options, n_partitions)<jupyter_output>PipelineOptions:
{'runner': 'InteractiveRunner', 'streaming': False, 'beam_services': {}, 'type_check_strictness': 'DEFAULT_TO_ANY', 'type_check_additional': '', 'pipeline_type_check': True, 'runtime_type_check': False, 'performance_runtime_type_check': False, 'direct_runner_use_stacked_bundle': True, 'direct_runner_bundle_repeat': 0, 'direct_num_workers': 0, 'direct_running_mode': 'multi_threading', 'dataflow_endpoint': 'https://dataflow.googleapis.com', 'project': 'sc-fids-capstone', 'job_name': 'sc-fids-capstone-etl-demo--download-videos-extract-frames', 'staging_location': None, 'temp_location': None, 'region': None, 'service_account_email': None, 'no_auth': False, 'template_location': None, 'labels': None, 'update': False, 'transform_name_mapping': None, 'enable_streaming_engine': False, 'dataflow_kms_key': None, 'flexrs_goal': None, 'hdfs_host': None, 'hdfs_port': None, 'hdfs_user': None, 'hdfs_full_urls': False, 'num_workers': None, 'max_num_workers': None, 'autoscaling_algorith[...]<jupyter_text>This time we would like to observe the results (collected into Pandas `DataFrame`s)...<jupyter_code># we require this in order to make use of ib.show() (which provides visualization of the pcolls specified) or ib.collect() (which creates a pandas dataframe from a pcoll)
    # but all pcolls we wish to visualize must be created prior to executing the following line
ib.watch(locals())<jupyter_output><empty_output><jupyter_text>And calling `ib.collect` forces the pipeline to actually run... Run the full "download-videos-extract-frames" pipeline We do this by collecting `Pcollection`s into Pandas `DataFrame`s for viewing with *Interactive Beam*.<jupyter_code>print(f"\n\n****************************** Starting pipeline job: {job_name} ******************************")
df_full_target_vid_index_schemad_pcoll = ib.collect(full_target_vid_index_schemad_pcoll)
df_filtered_target_vid_index_schemad_pcoll = ib.collect(filtered_target_vid_index_schemad_pcoll)
df_merged_download_results = ib.collect(merged_download_results)
df_merged_extraction_results = ib.collect(merged_extraction_results)
print(f"****************************** Finished pipeline job: {job_name} ******************************")
df_filtered_target_vid_index_schemad_pcoll
df_merged_download_results
df_merged_extraction_results.columns = ['segment_fname', 'frames', 'segment_dicts']
df_merged_extraction_results
print(f"We extracted {df_merged_extraction_results.frames.sum()} frames from {df_merged_extraction_results.segment_fname.count()} downloaded segments.")<jupyter_output>We extracted 4382 frames from 50 downloaded segments.
{'fraction_non_alphanumeric': 0.08681192660550459, 'fraction_numerical': 0.007186544342507645, 'mean_word_length': 6.816253361219002, 'pattern_counts': {'":': 0, '<': 77, '<?xml version=': 0, '>': 77, 'https://': 5, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 34, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '20', 'dir': 'jupyter-structured-clean-dedup', 'id': '530761', 'license': 'W3C', 'n_tokens_mistral': 7646, 'n_tokens_neox': 7312, 'n_words': 2657, 'path': 'etl_subset_demo.ipynb', 'repo_name': 'sacontreras/fids-capstone-asl-translation', 'size': '26160'}