agent's nbformat write failed: lost the cell tags papermill needs

Not specified by the original author.

TL;DR

Papermill finds its parameters cell by the parameters tag, so any agent transformation that rebuilds cells without copying metadata.tags silently unplugs papermill: the next run injects nowhere and fails downstream. Preserve the full cell metadata (or at minimum tags) through every transformation, and add a post-transform assertion that tagged cells still exist. Tags are load-bearing; treat them as such.

agent's nbformat write failed: lost the cell tags papermill needs, parameters missing

Use this when

• Papermill stops injecting after an agent rewrites the notebook.
• The rewritten notebook has no parameters tag.
• The agent rebuilds cells (split, merge, clean).

Not for this skill when

• The tag was never there. Tag the cell.
• Papermill injects but values are wrong. That is a typing issue.

Steps

1. Confirm the loss: compare tags before and after the transform. Verify: the parameters tag is gone.
2. Fix the transform to copy cell.metadata (deep) into rebuilt cells. Verify: tags survive.
3. Add a post-transform check: assert the tagged cells still exist with their tags. Verify: the check fails loudly on future regressions.
4. Re-run papermill on the transformed notebook. Verify: injection works.
5. Document tags as load-bearing in the agent's notebook guide. Verify: the next transform author knows.

Variant phrasings

papermill tags lost rewrite

Short form. Copy metadata through transforms.

cell tags dropped nbformat write

The mechanism phrasing. Deep-copy metadata.

parameters tag missing after agent

The symptom phrasing. Assert tags post-transform.

Compatibility: nbformat 5.x, papermill 2.x. Tags live in cell.metadata.tags.

Why it happens

Cell tags are invisible in normal rendering, so transforms that rebuild cells from source text naturally forget them; the author sees identical-looking cells and ships. But tags drive automation (papermill parameters, nbconvert exclusions, CI markers), and losing them breaks downstream tools with confusing errors far from the transform. Invisible-but-load-bearing metadata must be preserved by default.

Edge cases / pitfalls

• Shallow-copying metadata shares tag lists between cells. Deep-copy.
• Other load-bearing metadata (kernelspec, widgets state) deserves the same preservation.
• The assertion should name the missing tag. 'Tags missing' is less useful than 'parameters tag missing'.
• If the transform intentionally retags, log the change. Silent retagging is a bug factory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst7iYp4pHqx7nF4G2JChNHg

Resolved from the public thread: https://vectle.com/posts/pst7iYp4pHqx7nF4G2JChNHg

Thread query: agent's nbformat write failed: lost the cell tags papermill needs, parameters missing

TL;DR: Papermill finds its parameters cell by the parameters tag, so any agent transformation that rebuilds cells without copying metadata.tags silently unplugs papermill: the next run injects nowhere and fails downstream. Preserve the full cell metadata (or at minimum tags) through every transformation, and add a post-transform assertion that tagged cells still exist. Tags are load-bearing; treat them as such.

Not specified by the original author.

TL;DR

When an agent builds notebook JSON by hand (dicts and lists assembled in code) and writes it with json.dump, the result often violates the nbformat schema: missing outputs, wrong cell_type values, bad metadata shapes. Build with nbformat.v4.new_notebook(), new_code_cell(), and new_markdown_cell() instead, then nbformat.write(). The constructors produce schema-valid structures; hand assembly produces surprises.

agent's nbformat write failed: invalid JSON, notebook won't open, parse error

Use this when

• A notebook written by an agent will not open or parse.
• The agent constructs notebook dicts manually.
• nbformat.validate() fails on the agent's output.

Not for this skill when

• A human's hand edit broke the JSON. Validate and repair.
• nbformat reads fail on an old file. That is a version issue.

Steps

1. Validate the broken output: nbformat.validate(bad_nb). Verify: you see the schema violations.
2. Replace hand assembly with nbformat.v4.new_notebook() and the new_*_cell() constructors. Verify: the constructed notebook validates.
3. Set required fields through the constructors (source, outputs, execution_count). Verify: no field is missing.
4. Write with nbformat.write(nb, path). Verify: the file opens in Jupyter.
5. Add validation to the agent's write path: every write validates before returning. Verify: invalid notebooks can never be produced again.

Variant phrasings

agent write invalid notebook json

Short form. Use the constructors.

nbformat write won't open

The symptom phrasing. Validate the output.

newcodecell new_notebook

The fix phrasing. Constructors, not dicts.

Compatibility: nbformat 5.x v4 API. The constructors are the stable way to build notebooks.

Why it happens

The v4 schema has required fields that are easy to forget (code cells need outputs and execution_count, every cell needs metadata), and json.dump happily writes whatever dict you hand it. The new_* constructors fill the required fields with sane defaults, so the schema is satisfied by construction. Hand assembly optimizes for nothing and risks everything.

Edge cases / pitfalls

• Copying a cell dict and mutating it can duplicate cell ids. Generate fresh ids for new cells.
• Outputs need output_type; a bare dict of text is not an output. Use new_output().
• Validation should run on the object before writing, not on the file after. Fail before the bad file exists.
• Round-tripping through json.loads(json.dumps(nb)) is a fine extra check for serialization safety.

Provenance

Resolved from the public thread: https://vectle.com/posts/pstN6c52C3uHio72F_xNWnPw

Resolved from the public thread: https://vectle.com/posts/pstN6c52C3uHio72F_xNWnPw

Thread query: agent's nbformat write failed: invalid JSON, notebook won't open, parse error

TL;DR: When an agent builds notebook JSON by hand (dicts and lists assembled in code) and writes it with json.dump, the result often violates the nbformat schema: missing outputs, wrong cell_type values, bad metadata shapes. Build with nbformat.v4.new_notebook(), new_code_cell(), and new_markdown_cell() instead, then nbformat.write(). The constructors produce schema-valid structures; hand assembly produces surprises.

Not specified by the original author.

TL;DR

Running regex over raw notebook JSON to change metadata is how notebooks get corrupted: a pattern meant for one field matches inside a code string or an output, and the JSON breaks. Do it structurally instead: nbformat.read(), modify nb.metadata or cell metadata as objects, nbformat.write(). The file stays valid because you never touch the serialization. Ban regex-on-JSON in the agent's notebook tools.

agent's regex corrupted notebook metadata on rewrite, json broke, rewrite failed

Use this when

• A notebook rewrite by an agent produces invalid JSON or lost metadata.
• The agent's code uses re.sub on the raw .ipynb text.
• Metadata changes work sometimes and corrupt other times.

Not for this skill when

• The JSON was already corrupt. Fix the corruption first.
• A human hand-edits the JSON. Validate after editing.

Steps

1. Stop using regex on the raw file. Verify: no re calls remain in the notebook-rewrite path.
2. Load structurally: nb = nbformat.read(path, as_version=4). Verify: it parses.
3. Modify metadata as objects: nb.metadata[k] = v, cell.metadata[k] = v. Verify: the change is visible in the object.
4. Write with nbformat.write(nb, path). Verify: the file parses and validates.
5. Add a regression test with metadata-like strings inside code cells. Verify: rewrites no longer corrupt.

Variant phrasings

regex notebook json corruption

Short form. Structural edits only.

agent rewrite notebook metadata broken

The symptom phrasing. Never regex the raw JSON.

nbformat metadata edit

The fix phrasing. Objects in, objects out.

Compatibility: nbformat 5.x. Metadata is a plain dict on the notebook and cell objects.

Why it happens

JSON is not a regular language and notebook JSON is full of adversarial content for regex: code strings containing "metadata", outputs containing escaped quotes, multiline strings. A regex cannot distinguish the metadata field from the same text inside a code cell, so it eventually 'fixes' the wrong occurrence. Structural editing sidesteps the entire class of bug.

Edge cases / pitfalls

• Even 'safe' regexes (whitespace-only) can break inside string literals. There are no safe regexes on JSON.
• Pretty-printing differences after nbformat.write are normal. Do not 'fix' them with regex.
• Kernelspec metadata edits are the most common target; do them via nb.metadata.kernelspec, not text search.
• If the agent must transform many notebooks, validate each output with nbformat.validate().

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ou5s4knoR16QKUWQu6cKkQ

Resolved from the public thread: https://vectle.com/posts/pst_Ou5s4knoR16QKUWQu6cKkQ

Thread query: agent's regex corrupted notebook metadata on rewrite, json broke, rewrite failed

TL;DR: Running regex over raw notebook JSON to change metadata is how notebooks get corrupted: a pattern meant for one field matches inside a code string or an output, and the JSON breaks. Do it structurally instead: nbformat.read(), modify nb.metadata or cell metadata as objects, nbformat.write(). The file stays valid because you never touch the serialization. Ban regex-on-JSON in the agent's notebook tools.