A Simple Autograder for Jupyter Notebooks

Easily grade Jupyter notebooks using test cases and generate detailed reports.

Creating an autogradable notebook

Creating an autogradable item is as simple as adding a cell with a test case name (_test_case) and points ( _points ) to the notebook.

Assume your student is tasked to add x and y, and store the result to a new variable named sum_xy.

# file: submissions/student1.ipynb
# Task: Calculate the sum of x and y,
# and store it to a new variable named `sum_xy`

x = 1
y = 2

# YOUR CODE BEGINS
sum_xy = x + y
# YOUR CODE ENDS

Add a cell with the following content after the code cell.

_test_case = "calculate-sum"
_points = 2

assert sum_xy == 3

Grading notebooks

from jupygrader import grade_notebooks

graded_results = grade_notebooks(['submissions/student1.ipynb'])

This will return a GradedResult object. Below is a sample output of the GradedResult object in JSON format.

{
  "filename": "student1.ipynb",
  "learner_autograded_score": 2,
  "max_autograded_score": 2,
  "max_manually_graded_score": 0,
  "max_total_score": 2,
  "num_autograded_cases": 1,
  "num_passed_cases": 1,
  "num_failed_cases": 0,
  "num_manually_graded_cases": 0,
  "num_total_test_cases": 1,
  "grading_finished_at": "2025-03-19 06:49 PM UTC",
  "grading_duration_in_seconds": 0.02,
  "test_case_results": [
    {
      "test_case_name": "sum_xy",
      "points": 2,
      "available_points": 2,
      "did_pass": true,
      "grade_manually": false,
      "message": ""
    }
  ],
  "submission_notebook_hash": "c98c747e12a0456033956759c9daf7a0",
  "test_cases_hash": "ea8037ca2aedc3c33ec7227962cc2464",
  "grader_python_version": "3.12.5",
  "grader_platform": "Linux...",
  "jupygrader_version": "..."
  // ... more fields here
}

Grading Multiple Notebooks

import glob
from jupygrader import grade_notebooks

# Select all Jupyter notebooks in the "submissions" folder
notebooks = glob.glob('submissions/*.ipynb')

# Grade notebooks
graded_results = grade_notebooks(notebooks)

📝 Summary

Jupygrader is a Python package for automated grading of Jupyter notebooks. It provides a framework to:

1. Execute and grade Jupyter notebooks containing student work and test cases
2. Generate comprehensive reports in multiple formats (JSON, HTML, TXT)
3. Extract student code from notebooks into separate Python files
4. Verify notebook integrity by computing hashes of test cases and submissions

✨ Key Features

• Executes notebooks in a controlled, temporary environment
• Preserves the original notebook while creating graded versions
• Adds grader scripts to notebooks to evaluate test cases
• Supports multiple grading modes:
• Automatic grading via assertions and tests
• Manual grading
• Hybrid
• Generates detailed grading results including:
• Individual test case scores
• Overall scores and summaries
• Success/failure status of each test
• Error messages for failed test cases
• Produces multiple output formats for instructors to review:
• CSV export for batch grading results
• Graded notebook (.ipynb)
• HTML report
• JSON result data
• Plaintext summary for quick review
• Extracted Python code
• Includes metadata like Python version, platform, and file hashes for verification

🔒 Security Features

• Executes notebooks in isolated temporary directories
• Computes hashes of both test cases and submissions for academic integrity and duplicate prevention
• Cleanup of temporary files after grading completes
• Option to obfuscate test case code to prevent students from seeing solutions

Warning

This package provides only a basic obfuscation, and students with technical knowledge can easily decode the string to reveal the original code. Supporting a password-based encryption method is planned for future releases.

Tip

For extra security, consider running the grading script inside a Docker container or a virtual machine. This ensures that the grading environment is isolated from the host system.

📊 Output Formats

Jupygrader generates multiple output formats for each graded notebook:

1. Graded Notebook (.ipynb) – The original notebook with test results
2. HTML Report – Interactive HTML with test case navigation and results highlighting
3. JSON Results – Structured data with comprehensive grading metrics
4. Text Summary – Plain text overview of scores and test outcomes
5. CSV Export – Tabular data for all notebooks in a batch
6. Extracted Code (.py) – Pure Python code extracted from student cells

🚀 Advanced Usage

Batch Processing with File Dependencies

from jupygrader import grade_notebooks, GradingItem

# Configure grading with dependencies
grading_configs = [
    GradingItem(
        notebook_path="assignments/hw1/student1.ipynb",
        output_path="results/hw1",
        copy_files=["data/dataset.csv", "utility_functions.py"]
    ),
    GradingItem(
        notebook_path="assignments/hw1/student2.ipynb",
        output_path="results/hw1",
        copy_files=["data/dataset.csv", "utility_functions.py"]
    )
]

# Grade all notebooks with their dependencies
results = grade_notebooks(grading_configs)

Jupygrader is designed for educational settings where instructors need to grade student work in Jupyter notebooks, providing automated feedback while maintaining records of submissions and grading results.