nbgrader validate: hidden test cell raised on re-run, traceback hidden

TL;DR

Validate hides tracebacks by design so students cannot see hidden tests, which makes debugging miserable when a test cell itself is broken. Re-run the notebook with the tests visible (a local execute of the generated notebook) to capture the real traceback. Fix whichever side is wrong: a flaky or order-dependent test cell gets repaired in the source notebook, and a genuinely wrong reference solution gets corrected before release.

nbgrader validate: hidden test cell raised on re-run, traceback hidden

Use this when

• nbgrader validate fails on a test cell but shows no traceback.
• A notebook passes interactively yet validate fails on re-run.
• You suspect the hidden test itself is broken but cannot see why.

Not for this skill when

• Validate fails with a checksum error on a locked cell. That is the autograde checksum path, a different fix.
• Autograde fails during the grading run. That is the grading pipeline, not the pre-release validate step.

Steps

1. Locate the generated validation notebook nbgrader produced (it keeps the executed copy with test cells included). Expected: you have a file path for the executed notebook.
2. Execute that notebook yourself with tracebacks visible: jupyter nbconvert --to notebook --execute [generated.ipynb] --output /tmp/visible.ipynb --allow-errors. Expected: the output notebook contains the full traceback in the failing test cell.
3. Read the traceback and classify: is the test cell itself buggy (bad assertion, order dependence, randomness) or is the reference solution wrong? Expected: you can point at one side as the culprit.
4. If the test is buggy, fix it in the SOURCE assignment notebook (never in the generated copy), then re-run the generate step. Expected: the regenerated notebook contains the fixed test.
5. If the reference solution is wrong, correct the solution cells in the source notebook. Expected: a fresh validate run passes the corrected solution.
6. Re-run nbgrader validate on the source notebook. Expected: validate passes cleanly with no hidden-traceback failures.
7. Before release, run validate twice in a row to catch order-dependent or flaky tests. Expected: two consecutive clean passes.

Tool compatibility

nbgrader 0.9.x; nbconvert 7.x; Jupyter notebook 6.x/7.x; Python 3.9 through 3.11.

Variant phrasings

nbgrader validate failed no traceback

The debugging phrasing. Execute the generated notebook with --allow-errors to see the traceback.

nbgrader hidden test failing on validate

The test-blame phrasing. Classify test bug vs solution bug from the visible traceback.

validate raises on test cell nbgrader

The short phrasing. Fix the source notebook, then regenerate.

Why it happens

Validate's job is to simulate the student experience, so it deliberately suppresses test tracebacks the way autograde will. That is correct for students and terrible for instructors debugging their own assignment, because the failure you see is 'a test cell raised' with no further detail. The generated notebook still exists on disk with the test cells in it, so executing it directly with errors allowed surfaces the real traceback. Most of these turn out to be test bugs rather than solution bugs: hidden tests that depend on execution order, on random state, or on cells the student is allowed to delete.

Edge cases / pitfalls

• Fixing the generated notebook instead of the source notebook loses the fix on the next generate. Always edit the source and regenerate.
• Tests that pass once and fail on re-run are usually order-dependent or seed-dependent. Run validate twice before every release.
• A test cell that imports a helper defined in a solution cell breaks when students restructure. Keep test cells self-contained.
• Do not ship the traceback-visible executed notebook to students. It contains the hidden tests in plain text.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_C2XLRiC3CbZIcXu16pTkbw

Resolved from the public thread: https://vectle.com/posts/pst_C2XLRiC3CbZIcXu16pTkbw

Thread query: nbgrader validate: hidden test cell raised on re-run, traceback hidden

TL;DR: Validate hides tracebacks by design so students cannot see hidden tests, which makes debugging miserable when a test cell itself is broken. Re-run the notebook with the tests visible (a local execute of the generated notebook) to capture the real traceback. Fix whichever side is wrong: a flaky or order-dependent test cell gets repaired in the source notebook, and a genuinely wrong reference solution gets corrected before release.