<jupyter_start><jupyter_text>Install and deploy Kubeflow and Pachyderm Download the `mnist.npz` file to a blank directory on your local machine:<jupyter_code>!curl -O https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz<jupyter_output><empty_output><jupyter_text>Create a Pachyderm repo called `input-repo`:<jupyter_code>!pachctl config update context `pachctl config get active-context` --namespace "pachyderm"
!pachctl create repo input-repo<jupyter_output><empty_output><jupyter_text>We check-in our `mnist.npz` to `input-repo`:<jupyter_code>!pachctl put file input-repo@master:/mnist.npz -f mnist.npz<jupyter_output><empty_output><jupyter_text>This command copies the minst dataset from your local machine to the Pachyderm repo `input-repo` and Pachyderm will assign it a commit ID. Congratulations! Your data now has a HEAD commit, and Pachyderm has begun version-controlling the data!Confirm that the data is checked-in by running the following command:<jupyter_code>!pachctl list file input-repo@master<jupyter_output><empty_output><jupyter_text>Create a Pachyderm  pipeline to trigger the new 'mnist_pipeline':<jupyter_code>!pachctl create pipeline -f pipeline_steps/pachyderm/pipeline.yaml<jupyter_output><empty_output><jupyter_text>The Pachyderm pipeline starts, creates a Kubeflow run, and passes data to the newly-created Kubeflow workers. Also, you will see an output commit, containing the model parameters, with the input commit and trigger pipeline in its provenance.<jupyter_code>!pachctl list pipeline
!pachctl list job
!pachctl list repo<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08047411104179664, 'fraction_numerical': 0.0, 'mean_word_length': 7.354166666666667, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '151971', 'license': 'MIT', 'n_tokens_mistral': 535, 'n_tokens_neox': 518, 'n_words': 154, 'path': 'ai-pipeline/04-PachydermMNIST.ipynb', 'repo_name': 'chritter/jupyter-notebooks', 'size': '1603'}