<jupyter_start><jupyter_text>Otter-Grader Demo Notebook This notebook demonstrates how to use otter-grader. For installation details, see the [README](https://github.com/ucbds-infra/otter-grader).<jupyter_code>import pandas as pd
import os
from subprocess import PIPE
import subprocess<jupyter_output><empty_output><jupyter_text>Student Usage  NotebooksOtter supports in-notebook checks so that students can check their progress when working through an assignments. To use it, you need to create an instance of `otter.Notebook`, to which you (optionally) pass a path to a directory of tests; this defaults to `./tests`.<jupyter_code>import otter
grader = otter.Notebook()<jupyter_output><empty_output><jupyter_text>To check a student's work, use `Notebook.check`. Pass to this the question identifier, which is the filename (without the `.py` extension). For example, in this demo, `./tests/q4.py` tests a student's `square` function:<jupyter_code>with open("./tests/q4.py") as f:
    print(f.read())<jupyter_output>test = {
	"name": "q4",
	"points": 1,
	"suites": [ 
		{
			"cases": [ 
				{
					"code": r"""
					>>> square(5)
					25
					>>> square(2.5)
					6.25
					"""
				},
			],
			"scored": False,
			"setup": "",
			"teardown": "",
			"type": "doctest"
		}, 
	]
}<jupyter_text>If we have the square function defined below, we can run the test using `grader.check("q4")`:<jupyter_code>def square(x):
    return x**2
grader.check("q4")<jupyter_output><empty_output><jupyter_text>If we change the function so that the tests fail, then the grader outputs the failed test and the incorrect output.<jupyter_code>square = lambda x: x**3
grader.check("q4")<jupyter_output><empty_output><jupyter_text>Students can also run all tests at once using `Notebook.check_all`:<jupyter_code>grader.check_all()<jupyter_output><empty_output><jupyter_text>Students can also create their own PDF submissions using `Notebook.export`, to which you pass the path to the notebook. Otter uses nb2pdf to generate PDFs (more info [below](nb2pdf)). The filtering behavior of nb2pdf is turned on by default, although it can be turned off using the `filtering=False` argument.<jupyter_code>grader.export("demo.ipynb")<jupyter_output><empty_output><jupyter_text>ScriptsIf a student is working with Python scripts instead of notebooks, the otter command line utility has a `check` command that can run these tests as well. The help entry for this command is given below.<jupyter_code>!otter check -h<jupyter_output>usage: otter [-h] [-q QUESTION] [-t TESTS-PATH] file

positional arguments:
  file           Python file to grade

optional arguments:
  -h, --help     show this help message and exit
  -q QUESTION    Grade a specific test
  -t TESTS-PATH  Path to test files<jupyter_text>If we wanted to run a single check, we would pass the question identifier (the filename without `.py`) to the `-q` flag. The tests path is assumed to be `./tests` unless another path is provided to the `-t` flag.<jupyter_code>!otter check scripts/file0.py -q q4<jupyter_output>Unable to revert mtime: /Library/Fonts
All tests passed!<jupyter_text>As with notebooks, a failed test will output the test and the incorrect output.<jupyter_code>!otter check scripts/file0.py -q q2<jupyter_output>1 of 2 tests passed

Tests passed:
 possible 


Tests failed: 
*********************************************************************
Line 2, in tests/q2.py 0
Failed example:
    1 == 1
Expected:
    False
Got:
    True<jupyter_text>To check all of the tests at once, use `otter check` without a `-q` flag:<jupyter_code>!otter check scripts/file0.py<jupyter_output>4 of 6 tests passed

Tests passed:
 q1  q3  q4  q5 


Tests failed: 
*********************************************************************
Line 2, in tests/q2.py 0
Failed example:
    1 == 1
Expected:
    False
Got:
    True<jupyter_text>Grading  NotebooksOtter uses Docker containers to execute students' submissions in parallel containers. It accepts exports from Gradescope and Canvas, or the notebook files with a JSON- or YAML-formatted metadata file. If you're using the custom metadata file, it requires an entry for each notebook, with the notebook filename as the `filename` parameter and the student identifier as `identifier`. An example of the YAML metadata can be found in `notebooks/meta.yml`.<jupyter_code>with open("./notebooks/meta.yml") as f:
    for l in f.readlines()[0:10]:
        print(l[:-1])<jupyter_output>- identifier: 0
  filename: test-nb-0.ipynb
- identifier: 1
  filename: test-nb-1.ipynb
- identifier: 2
  filename: test-nb-2.ipynb
- identifier: 3
  filename: test-nb-3.ipynb
- identifier: 4
  filename: test-nb-4.ipynb<jupyter_text>RequirementsThe docker container comes with the following packages installed:* datascience* jupyter_client* ipykernel* matplotlib* pandas* ipywidgets* scipy* nb2pdf* otter-graderIf you have any other requirements besides these, create a `requirements.txt` file:<jupyter_code>with open("./requirements.txt") as f:
    print(f.read())<jupyter_output>tqdm<jupyter_text>Command-Line UsageNow let's examine the usage of the `otter` command:<jupyter_code>!otter -h<jupyter_output>usage: otter [-h] [-p NOTEBOOKS-PATH] [-t TESTS-PATH] [-o OUTPUT-PATH] [-g]
             [-c] [-j JSON] [-y YAML] [-s] [--pdf] [--filter-pdf]
             [-f FILES [FILES ...]] [-v] [-r REQUIREMENTS]
             [--containers NUM-CONTAINERS] [--image IMAGE]

Local autograder for Jupyter Notebooks and Python scripts. Launches parallel
Docker containers to grade notebooks/scripts and returns a CSV of grades.
Requires a metadata file if not exported from Gradescope or Canvas. Add
support files by putting them into the SUBMISSIONS-DIR folder or using the -f
flag. Can optionally create PDFs of Jupyter Notebooks for manual grading.

optional arguments:
  -h, --help            show this help message and exit
  -p NOTEBOOKS-PATH, --path NOTEBOOKS-PATH
                        Path to directory of submissions
  -t TESTS-PATH, --tests-path TESTS-PATH
                        Path to directory of tests
  -o OUTPUT-PATH, --output-path OUTPUT-PATH
                        Path to w[...]<jupyter_text>The `-p` flag should be the path to the directory that conains the notebooks (or the unzipped export from Canvas or Gradescope). The `g`, `c`, `y`, and `j` flags indicate the metadata type; the `y` and `j` flags require an argument that indicates the path to the metadata file. If you want to generate PDFs for manual grading, you can add the `--pdf` or `--filter-pdf` flags, which generate an unfiltered or filtered, respectively, PDF for each notebook. If you need a requirements file, pass that path to the `-r` flag. Optionally, you can set the number of docker containers or the docker image that the grader runs on using the `--containers` and `--image` flags, respectively.Some flag defaults are listed below. If a flag is not list, it has no default value.| Flag | Default ||-----|-----|| `-p` | `./` || `-t` | `./tests` || `-o` | `./` || `--containers` | 4 || `--image` | `ucbdsinfra/otter-grader` |**If your notebooks require any other files (e.g. data files)**, place these files into the notebooks path directory (whatever the value of `-p` is). These will automatically be copied into the docker containers.Let's run the autograder on the files in `./notebooks`. The command below tells otter that our metadata file can be found at `./notebooks/meta.yml`, that the notebooks can be found in `./notebooks`, that we have a requirements file at `./requirements.txt`, and to use verbose output.<jupyter_code>!otter -y notebooks/meta.yml -p notebooks -r requirements.txt -v<jupyter_output>Found YAML metadata...
Launching docker containers...
Launched container 1b4b6bf6b9a3...
Launched container 52a50e11981c...
Launched container 36aa3362f5ca...
Launched container 03eba9df2e0e...
Installing requirements in container 1b4b6bf6b9a3...
Installing requirements in container 52a50e11981c...
Installing requirements in container 36aa3362f5ca...
Installing requirements in container 03eba9df2e0e...
Grading notebooks in container 52a50e11981c...
Grading notebooks in container 03eba9df2e0e...
Grading notebooks in container 1b4b6bf6b9a3...
Grading notebooks in container 36aa3362f5ca...
Copying grades from container 03eba9df2e0e...
Copying grades from container 36aa3362f5ca...
Copying grades from container 52a50e11981c...
Copying grades from container 1b4b6bf6b9a3...
Stopping container 36aa3362f5ca...
Stopping container 03eba9df2e0e...
Stopping container 52a50e11981c...
Stopping container 1b4b6bf6b9a3...
Combining grades and saving...<jupyter_text>We should now have a CSV file with the scores and score breakdowns at `./final_grades.csv` (since we left `-o` with its default value).<jupyter_code>pd.read_csv("final_grades.csv").head()<jupyter_output><empty_output><jupyter_text>If the autograder ran correctly, all of the submissions should receive a 2/5, since three of the tests are written to fail on these notebooks.  nb2pdfWhen exporting notebook PDFs, otter uses the library nb2pdf, which in turn relies on nbpdfexporter. This requires that chromium be installed on the docker container (which is true if using `ucbdsinfra/otter-grader`, the default image) but it also must be installed on the JupyterHub distribution that students use if they are going to be exporting PDFs themselves.nb2pdf can filter notebooks in two ways: using cell tags or using HTML comments in Markdown cells. TagsWhen generating the PDF (if filtering is turned on, either by use of the `--tag-filter` flag or `filter_type="tags"` in `Notebook.export`), the following cells will be included by default:* Markdown cells* Code cells with images in the output* All cells tagged with `include`If a cell falls within one of the 3 categories listed above but you do not want to include it in the exported PDF, tag the cell with `ignore`. This cell will then not be added to the manual graded PDF. HTML CommentsIf you elect to use HTML comments, all content that is between the HTML comments `` and `` will be included. This filtering is triggered with the `--html-filter` PDF and is the default behavior of `Notebook.export`.  ScriptsOtter can also grade Python scripts using the addition of the `-s` flag. It uses the same metadata and path structure as with notebooks, so the only change to our call above will be to change the path to `./scripts`. *This will overwrite `final_grades.csv` from above since we aren't changing the output path.*<jupyter_code>!otter -sy scripts/meta.yml -p scripts -r requirements.txt -v<jupyter_output>Found YAML metadata...
Launching docker containers...
Launched container 79a0856c4dda...
Launched container d186eeff8a4e...
Launched container 772fcbbc1c2d...
Launched container 5304474cae57...
Installing requirements in container 79a0856c4dda...
Installing requirements in container d186eeff8a4e...
Installing requirements in container 5304474cae57...
Installing requirements in container 772fcbbc1c2d...
Grading scripts in container d186eeff8a4e...
Grading scripts in container 79a0856c4dda...
Grading scripts in container 5304474cae57...
Grading scripts in container 772fcbbc1c2d...
Copying grades from container 79a0856c4dda...
Stopping container 79a0856c4dda...
Copying grades from container 5304474cae57...
Copying grades from container d186eeff8a4e...
Copying grades from container 772fcbbc1c2d...
Stopping container 5304474cae57...
Stopping container d186eeff8a4e...
Stopping container 772fcbbc1c2d...
Combining grades and saving...<jupyter_text>Now, we should be able to read in the grades from the scripts. If all went well, they should each have a 4/5.<jupyter_code>pd.read_csv("final_grades.csv").head()<jupyter_output><empty_output><jupyter_text>GradescopeOtter is compatible with the Gradescope autograder, and has a command line tool to generate the zipfile that is used to configure the Docker container that Gradescope's autograder usage. The base command for this utility is `otter gen`, and its help entry is given below.<jupyter_code>!otter gen -h<jupyter_output>usage: otter [-h] [-t [TESTS-PATH]] [-o [OUTPUT-PATH]] [-r [REQUIREMENTS]]
             [--threshold THRESHOLD] [--points POINTS]
             [files [files ...]]

Generates zipfile to configure Gradescope autograder

positional arguments:
  files                 Other support files needed for grading (e.g. .py
                        files, data files)

optional arguments:
  -h, --help            show this help message and exit
  -t [TESTS-PATH], --tests-path [TESTS-PATH]
                        Path to test files
  -o [OUTPUT-PATH], --output-path [OUTPUT-PATH]
                        Path to which to write zipfile
  -r [REQUIREMENTS], --requirements [REQUIREMENTS]
                        Path to requirements.txt file
  --threshold THRESHOLD
                        Pass/fail score threshold
  --points POINTS       Points possible, overrides sum of test points<jupyter_text>The `otter gen` command creates a zipfile at `OUTPUT-PATH/autograder.zip` which contains the necessary files for the Gradescope autograder:* `run_autograder`: the script that executes otter* `setup.sh`: a file that instructs Ubuntu on how to install dependencies* `requirements.txt`: Python's list of necessary dependencies* `tests`: the folder of test cases* `files`: a folder containing any files needed for the notebooks to execute (e.g. data files)The requirements file create automatically includes the otter dependencies (see [Requirements](Requirements)), but you can optionally include your own, other ones by passing a filepath to the `-r` flag. Any files included that are not passed to a flag are automatically placed into the `files` folder in the zipfile and will be copied into the notebook directory in the Gradescope container.Let's generate a zipfile from this directory with the tests in `./tests`, the requirements in `./requirements.txt`, and `test-df.csv` as a data file.<jupyter_code>!otter gen -r requirements.txt test-df.csv<jupyter_output><empty_output><jupyter_text>Now, we have a file `./autograder.zip` in the working directory:<jupyter_code>"autograder.zip" in os.listdir()<jupyter_output><empty_output><jupyter_text>If we unzip the file, you'll see all of the files listed above as well as the CSV file we added in the `autograder/files` directory:<jupyter_code>unzipped = subprocess.run(["unzip", "autograder.zip", "-d", "autograder"], stdout=PIPE, stderr=PIPE)
print(unzipped.stdout.decode("utf-8"))
os.listdir("autograder")<jupyter_output><empty_output><jupyter_text>Pass/Fail ThresholdsThe generator also supports scoring assignments on a pass/fail basis using only some percentage threshold. If you include a threshold, pass a number between 0 and 1 to the `--threshold` flag. If a student scores below `--threshold`, then they will get 0 points; otherwise, they will get points for all questions.The command below creates an autograder that automatically gives students full points as long as they get at least an 80%.<jupyter_code>!otter gen -r requirements.txt test-df.csv --threshold 0.8<jupyter_output><empty_output><jupyter_text>Overriding Points PossibleOtter also supports overriding the number of points possible for an assignment from the default to the sum of the points possible for each test using the `--points` flag. This is done by scaling the new points possible by the percentage of points received as a decimal. For example, if a student passes a 2- and 1-point test but fails a 4-point test on an assignment with `--points` set to 2, they will receive a $\frac{2 + 1}{2 + 1 + 4} \cdot 2 = 0.85714285714$.The command below creates an autograder that scales the number of points down to 3.<jupyter_code>!otter gen -r requirements.txt test-df.csv --points 3<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09618051178620951, 'fraction_numerical': 0.022942140426068323, 'mean_word_length': 4.151623376623377, 'pattern_counts': {'":': 9, '<': 86, '<?xml version=': 0, '>': 92, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '606262', 'license': 'BSD-3-Clause', 'n_tokens_mistral': 5054, 'n_tokens_neox': 4681, 'n_words': 1909, 'path': 'demo/demo.ipynb', 'repo_name': 'TylerADavis/otter-grader', 'size': '15866'}