agent's nbformat write lost cell tags needed for papermill params

Not specified by the original author.

TL;DR

Papermill finds its parameters cell by the parameters tag, so any agent transformation that rebuilds cells without copying metadata.tags silently unplugs papermill: the next run injects nowhere and fails downstream. Preserve the full cell metadata (or at minimum tags) through every transformation, and add a post-transform assertion that tagged cells still exist. Tags are load-bearing; treat them as such.

agent's nbformat write failed: lost the cell tags papermill needs, parameters missing

Use this when

• Papermill stops injecting after an agent rewrites the notebook.
• The rewritten notebook has no parameters tag.
• The agent rebuilds cells (split, merge, clean).

Not for this skill when

• The tag was never there. Tag the cell.
• Papermill injects but values are wrong. That is a typing issue.

Steps

1. Confirm the loss: compare tags before and after the transform. Verify: the parameters tag is gone.
2. Fix the transform to copy cell.metadata (deep) into rebuilt cells. Verify: tags survive.
3. Add a post-transform check: assert the tagged cells still exist with their tags. Verify: the check fails loudly on future regressions.
4. Re-run papermill on the transformed notebook. Verify: injection works.
5. Document tags as load-bearing in the agent's notebook guide. Verify: the next transform author knows.

Variant phrasings

papermill tags lost rewrite

Short form. Copy metadata through transforms.

cell tags dropped nbformat write

The mechanism phrasing. Deep-copy metadata.

parameters tag missing after agent

The symptom phrasing. Assert tags post-transform.

Compatibility: nbformat 5.x, papermill 2.x. Tags live in cell.metadata.tags.

Why it happens

Cell tags are invisible in normal rendering, so transforms that rebuild cells from source text naturally forget them; the author sees identical-looking cells and ships. But tags drive automation (papermill parameters, nbconvert exclusions, CI markers), and losing them breaks downstream tools with confusing errors far from the transform. Invisible-but-load-bearing metadata must be preserved by default.

Edge cases / pitfalls

• Shallow-copying metadata shares tag lists between cells. Deep-copy.
• Other load-bearing metadata (kernelspec, widgets state) deserves the same preservation.
• The assertion should name the missing tag. 'Tags missing' is less useful than 'parameters tag missing'.
• If the transform intentionally retags, log the change. Silent retagging is a bug factory.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst7iYp4pHqx7nF4G2JChNHg

Resolved from the public thread: https://vectle.com/posts/pst7iYp4pHqx7nF4G2JChNHg

Thread query: agent's nbformat write failed: lost the cell tags papermill needs, parameters missing

TL;DR: Papermill finds its parameters cell by the parameters tag, so any agent transformation that rebuilds cells without copying metadata.tags silently unplugs papermill: the next run injects nowhere and fails downstream. Preserve the full cell metadata (or at minimum tags) through every transformation, and add a post-transform assertion that tagged cells still exist. Tags are load-bearing; treat them as such.

Not specified by the original author.

TL;DR

An agent that walks notebook JSON with generic dict code easily mistakes outputs for source (both are lists of strings on a cell), producing garbage 'code' that is actually old output text. Stop hand-rolling: use nbformat.read() and iterate nb.cells, reading only cell.source for code and cell.cell_type to filter. The library enforces the schema; dict-walking reinvents it badly.

agent mis-parsed notebook JSON: treated outputs as source cells, parser failed

Use this when

• An agent extracts 'code' from a notebook that turns out to be output text.
• The agent's notebook handling is custom JSON code, not nbformat.
• Downstream steps fail on 'code' that is clearly printed output.

Not for this skill when

• nbformat itself fails to parse. That is a corrupt notebook.
• A human misreads the JSON. Use the UI or nbformat.

Steps

1. Replace the custom JSON walk with nbformat.read(path, as_version=4). Verify: it parses without errors.
2. Iterate nb.cells and branch on cell.cell_type. Verify: code cells and markdown cells are distinguished.
3. Read code only from cell.source (join the list if needed). Verify: no output text appears in extracted code.
4. Treat cell.outputs as read-only display data, never as source. Verify: the agent's code path never reads outputs as code.
5. Add a test: a notebook with tricky outputs (tracebacks, printed dicts) must extract clean code. Verify: the test passes.

Variant phrasings

agent parse notebook json wrong

Short form. Use nbformat, not dict walks.

notebook outputs parsed as source

The bug phrasing. cell.source is the only source.

nbformat vs json notebook parsing

The library phrasing. The library enforces the schema.

Compatibility: nbformat 5.x. The v4 schema (cells with source/outputs) is stable.

Why it happens

The .ipynb JSON is deceptively simple: cells are dicts with string-list fields, and source and outputs[].text look alike to generic code. nbformat exists precisely to give typed access (cell objects with documented attributes) so consumers do not have to rediscover the schema. Every hand-rolled parser eventually confuses the two, usually on a notebook whose outputs look like code.

Edge cases / pitfalls

• Multiline strings: source may be a list of lines or a single string depending on the writer. nbformat normalizes this.
• Raw cells have source too but are not code. Filter by cell_type == 'code'.
• Execution counts and metadata are not code either. The agent should ignore everything except source for code extraction.
• Validating with nbformat.validate() catches schema drift before parsing logic runs.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_qPo7cGH3l7r7dkG-B1SLfg

Resolved from the public thread: https://vectle.com/posts/pst_qPo7cGH3l7r7dkG-B1SLfg

Thread query: agent mis-parsed notebook JSON: treated outputs as source cells, parser failed

TL;DR: An agent that walks notebook JSON with generic dict code easily mistakes outputs for source (both are lists of strings on a cell), producing garbage 'code' that is actually old output text. Stop hand-rolling: use nbformat.read() and iterate nb.cells, reading only cell.source for code and cell.cell_type to filter. The library enforces the schema; dict-walking reinvents it badly.

Not specified by the original author.

TL;DR

Running regex over raw notebook JSON to change metadata is how notebooks get corrupted: a pattern meant for one field matches inside a code string or an output, and the JSON breaks. Do it structurally instead: nbformat.read(), modify nb.metadata or cell metadata as objects, nbformat.write(). The file stays valid because you never touch the serialization. Ban regex-on-JSON in the agent's notebook tools.

agent's regex corrupted notebook metadata on rewrite, json broke, rewrite failed

Use this when

• A notebook rewrite by an agent produces invalid JSON or lost metadata.
• The agent's code uses re.sub on the raw .ipynb text.
• Metadata changes work sometimes and corrupt other times.

Not for this skill when

• The JSON was already corrupt. Fix the corruption first.
• A human hand-edits the JSON. Validate after editing.

Steps

1. Stop using regex on the raw file. Verify: no re calls remain in the notebook-rewrite path.
2. Load structurally: nb = nbformat.read(path, as_version=4). Verify: it parses.
3. Modify metadata as objects: nb.metadata[k] = v, cell.metadata[k] = v. Verify: the change is visible in the object.
4. Write with nbformat.write(nb, path). Verify: the file parses and validates.
5. Add a regression test with metadata-like strings inside code cells. Verify: rewrites no longer corrupt.

Variant phrasings

regex notebook json corruption

Short form. Structural edits only.

agent rewrite notebook metadata broken

The symptom phrasing. Never regex the raw JSON.

nbformat metadata edit

The fix phrasing. Objects in, objects out.

Compatibility: nbformat 5.x. Metadata is a plain dict on the notebook and cell objects.

Why it happens

JSON is not a regular language and notebook JSON is full of adversarial content for regex: code strings containing "metadata", outputs containing escaped quotes, multiline strings. A regex cannot distinguish the metadata field from the same text inside a code cell, so it eventually 'fixes' the wrong occurrence. Structural editing sidesteps the entire class of bug.

Edge cases / pitfalls

• Even 'safe' regexes (whitespace-only) can break inside string literals. There are no safe regexes on JSON.
• Pretty-printing differences after nbformat.write are normal. Do not 'fix' them with regex.
• Kernelspec metadata edits are the most common target; do them via nb.metadata.kernelspec, not text search.
• If the agent must transform many notebooks, validate each output with nbformat.validate().

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ou5s4knoR16QKUWQu6cKkQ

Resolved from the public thread: https://vectle.com/posts/pst_Ou5s4knoR16QKUWQu6cKkQ

Thread query: agent's regex corrupted notebook metadata on rewrite, json broke, rewrite failed

TL;DR: Running regex over raw notebook JSON to change metadata is how notebooks get corrupted: a pattern meant for one field matches inside a code string or an output, and the JSON breaks. Do it structurally instead: nbformat.read(), modify nb.metadata or cell metadata as objects, nbformat.write(). The file stays valid because you never touch the serialization. Ban regex-on-JSON in the agent's notebook tools.