<jupyter_start><jupyter_text>Pachyderm Combinator Component DemoIf you are reading this then you are about to take Pachyderm, a data lineage and pipelining tool, for a test drive.Pachyderm has been packaged as a [combinator component](https://combinator.ml/components/introduction/), which makes it easy to spin up a combination of MLOps components to make a stack. This notebook is running within the cluster, next to the Pachyderm installation.The following demo is a shortened version of [Pachyderm's official getting started demo](https://docs.pachyderm.com/latest/getting_started/beginner_tutorial/). I recommend reading that if you are interested in the details._Note: this demo only works reliably when in the default namespace (the default demo pipeline expects the default namespace)_ Download the `pachctl` ClientThe first task is to download and install the `pachctl` client to interact with the Pachyderm cluster running in the cluster.<jupyter_code>!wget -q -O /tmp/pachctl.tar.gz https://github.com/pachyderm/pachyderm/releases/download/v1.13.2/pachctl_1.13.2_linux_amd64.tar.gz 
!tar -xvf /tmp/pachctl.tar.gz -C /tmp 
!cp /tmp/pachctl_1.13.2_linux_amd64/pachctl /opt/conda/bin # Move the binary to somewhere on the PATH<jupyter_output>pachctl_1.13.2_linux_amd64/pachctl<jupyter_text>Connect to the Pachyderm ClusterOnce downloaded, we need to connect to the cluster. It's running in the same namespace, so we can use Kubernetes' internal service DNS to get the IP of the running `pachd` container. By default the `grpc` port runs on `650`._Note: `pachctl` is a binary, so you need to prepend all demo commands with a `!` to run the command in the shell_<jupyter_code>!pachctl config update context `pachctl config get active-context` --pachd-address=pachd:650
!pachctl list repo<jupyter_output>NAME CREATED SIZE (MASTER) DESCRIPTION<jupyter_text>Create a RepositoryYou haven't created any repositories yet (as seen above) so you need to create one to store the test data. Below I create a repository called images to store the incoming image data you will see shortly.<jupyter_code>!pachctl create repo images
!pachctl list repo<jupyter_output>NAME   CREATED      SIZE (MASTER) DESCRIPTION 
images 1 second ago 0B<jupyter_text>Add DataNow it is time to add some random image data. Note the new size of the repository. Why not try [inspecting the commit history](https://docs.pachyderm.com/latest/concepts/data-concepts/commit/) at this point? Or try to [download the file](https://docs.pachyderm.com/latest/concepts/data-concepts/file/)?<jupyter_code>!pachctl put file images@master:liberty.png -f http://imgur.com/46Q8nDz.png
!pachctl list repo
!pachctl list commit images<jupyter_output>REPO   BRANCH COMMIT                           FINISHED      SIZE     PROGRESS DESCRIPTION
images master ae640f0bc8494f0fbfd8a8435f91390d 2 seconds ago 57.27KiB -<jupyter_text>Create a PipelineA pipeline is like a pipe that connects one repository to another, with a processing step in-between. The example below connects to the `images` repository, performs some edge detection, and writes it to an `edges` repository. Take a look at the pipeline definition if you're interested and [the documentation](https://docs.pachyderm.com/latest/concepts/pipeline-concepts/).<jupyter_code>!pachctl create pipeline -f https://raw.githubusercontent.com/pachyderm/pachyderm/master/examples/opencv/edges.json<jupyter_output>WARNING: please specify a tag for the docker image in your transform.image spec.
For example, change 'python' to 'python:3' or 'bash' to 'bash:5'. This improves
reproducibility of your pipelines.<jupyter_text>Now the pipeline has been created, you will see it automatically start to process all files in the `master` branch of the `images` repository.<jupyter_code>!pachctl list pipeline edges<jupyter_output>NAME  VERSION INPUT     CREATED       STATE / LAST JOB   DESCRIPTION                                                                
edges 1       images:/* 3 seconds ago [32mrunning[0m / [33mstarting[0m A pipeline that performs image edge detection by using the OpenCV library.<jupyter_text>The flush command below is a useful way to get the notebook to wait until the pipeline that depends on this is complete. It might take a while to complete the first time because Kubernetes has to download the pipeline container.<jupyter_code>!pachctl flush job images@master
!pachctl list job<jupyter_output>ID                               PIPELINE STARTED        DURATION  RESTART PROGRESS  DL       UL       STATE   
81974c3c6b5f4fc49ba9b492047b7979 edges    40 seconds ago 3 seconds 0       1 + 0 / 1 57.27KiB 22.22KiB [32msuccess[0m<jupyter_text>View the Result of the PipelineThe pipeline created an output repository, which we can query to get a file. I'm using jupyter's inbuilt markdown processor to show the image.<jupyter_code>!pachctl get file edges@master:liberty.png > liberty.png<jupyter_output><empty_output><jupyter_text>![](liberty.png)  Create Another PipelineYou can create sophisticated graphs in Pachyderm. The example below is creating another pipeline called `montage` that uses the output of the `edges` pipeline.As before, I run a few commands to wait for it to complete.<jupyter_code>!pachctl create pipeline -f https://raw.githubusercontent.com/pachyderm/pachyderm/master/examples/opencv/montage.json
!pachctl flush job edges@master
!pachctl list job
!pachctl get file montage@master:montage.png > montage.png<jupyter_output><empty_output><jupyter_text>![](montage.png)  Data Driven PipelinesAutomatically trigger both pipelines by adding some more data to the `images` repository. Remember that all commits are retained, and you can move back to them at any time.<jupyter_code>!pachctl put file images@master:AT-AT.png -f http://imgur.com/8MN9Kg0.png
!pachctl put file images@master:kitten.png -f http://imgur.com/g2QnNqa.png
!pachctl flush job images@master
!pachctl list job
!pachctl get file montage@master:montage.png > montage.png<jupyter_output><empty_output><jupyter_text>![](montage.png)  Next StepsThat's it for this demo. I recommend looking through [the official documentation](https://docs.pachyderm.com/latest/).Also try more demos by creating a new notebook and walk through the [Pachyderm examples](https://docs.pachyderm.com/latest/how-tos/).Also try other [Combinator stacks](https://combinator.ml/stacks/introduction/) which integrate other products with Pachyderm for 10x multipliers!  Clean UpThe following command deletes all data and pipelines, in case you want to run through the demo notebook again.<jupyter_code>!pachctl delete repo --all && pachctl delete pipeline --all<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07301017354877319, 'fraction_numerical': 0.016906044284859367, 'mean_word_length': 4.723458904109589, 'pattern_counts': {'":': 0, '<': 38, '<?xml version=': 0, '>': 41, 'https://': 11, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '568312', 'license': 'Apache-2.0', 'n_tokens_mistral': 2131, 'n_tokens_neox': 1971, 'n_words': 746, 'path': 'examples/testfaster/demo.ipynb', 'repo_name': 'combinator-ml/pachyderm', 'size': '6684'}