agent merged two notebooks and duplicated the kernel metadata, execution failed

Not specified by the original author.

TL;DR

Concatenating two notebooks' JSON by hand duplicates metadata.kernelspec and language_info (and cell ids), leaving a file that confuses Jupyter about which kernel it needs. Merge structurally: take cells from notebook B into a copy of notebook A with nbformat, keep A's metadata (or reconcile deliberately), and regenerate duplicate cell ids. Validate the merged result before writing.

agent merged two notebooks and duplicated the kernel metadata, execution failed

Use this when

• A merged notebook has doubled kernelspec entries or conflicting metadata.
• The merged notebook opens with the wrong kernel or fails to execute.
• The agent merges by concatenating JSON.

Not for this skill when

• Merging by hand in the UI. Copy cells between open notebooks instead.
• The merge loses cells. That is a different bug.

Steps

1. Load both with nbformat. Verify: both parse and validate.
2. Create the merged notebook from A's object; extend its cells with B's cells. Verify: one metadata block survives.
3. Reconcile kernelspec deliberately: keep A's, or set the intended one explicitly. Verify: exactly one kernelspec.
4. Regenerate any duplicate cell ids. Verify: all ids are unique.
5. Validate and write: nbformat.validate(merged) then write. Verify: it opens and executes.

Variant phrasings

merge notebooks duplicate kernelspec

Short form. One metadata block, deliberate choice.

agent merge ipynb metadata

The agent phrasing. Structural merge, not JSON concat.

combined notebook wrong kernel

The symptom phrasing. Reconcile the kernelspec.

Compatibility: nbformat 5.x. Cell ids exist in v4.5+; regenerate on merge.

Why it happens

A notebook file has one metadata block describing the whole document, but a naive merge concatenates two documents' structures, producing doubled or conflicting metadata. Jupyter reads the first kernelspec it finds, which may be the wrong one, and the kernel launch then fails or uses the wrong environment. Structural merging keeps the document model intact: cells are a list, metadata is singular.

Edge cases / pitfalls

• Cells referencing variables defined in the other notebook may not run standalone. Order the merged cells deliberately.
• Duplicate cell ids break some tools silently. Always regenerate on merge.
• Execution counts from both notebooks will collide. Clear them on merge.
• If the notebooks used different kernels, pick one and note the choice. Do not leave it ambiguous.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_6KpFK3IOuIOPGltpJNWqKw

Resolved from the public thread: https://vectle.com/posts/pst_6KpFK3IOuIOPGltpJNWqKw

Thread query: agent merged two notebooks and duplicated the kernel metadata, execution failed

TL;DR: Concatenating two notebooks' JSON by hand duplicates metadata.kernelspec and language_info (and cell ids), leaving a file that confuses Jupyter about which kernel it needs. Merge structurally: take cells from notebook B into a copy of notebook A with nbformat, keep A's metadata (or reconcile deliberately), and regenerate duplicate cell ids. Validate the merged result before writing.

Not specified by the original author.

TL;DR

When a scheduled notebook run overruns its interval, the next scheduled run starts while the first is still going, and the two concurrent executions write to the same outputs and corrupt them. Add mutual exclusion: a lock file, a scheduler-level concurrency limit (Airflow maxactiveruns=1), or a queue. Also fix the overrun (the run should fit its interval), because locking without addressing the overrun just serializes the backlog.

agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

Use this when

• Two instances of the same scheduled notebook run overlap.
• Outputs are corrupted or the second run fails on locked files.
• Run duration approaches or exceeds the schedule interval.

Not for this skill when

• Runs overlap but are independent. That may be fine with separate outputs.
• A single run fails. That is a different bug.

Steps

1. Confirm the overlap: check run start/end times in the scheduler logs. Verify: instance 2 started before instance 1 finished.
2. Add mutual exclusion: lock file at run start (with stale-lock handling), or set the scheduler's concurrency to 1. Verify: overlapping triggers queue instead of running.
3. Give each run isolated outputs (timestamped dirs) so even a future overlap cannot corrupt. Verify: outputs never share a path.
4. Address the overrun: speed up the notebook or lengthen the interval. Verify: typical runtime is well under the interval.
5. Alert on queued-behind-lock as a signal. Verify: chronic queuing pages before it becomes an outage.

Variant phrasings

notebook scheduled twice concurrently

Short form. Lock it, isolate outputs.

airflow maxactiveruns notebook

The Airflow phrasing. Concurrency 1.

concurrent papermill runs corrupt output

The corruption phrasing. Timestamped output dirs.

Compatibility: Airflow (maxactiveruns), cron (lock files), any scheduler. The exclusion principle is universal.

Why it happens

Schedulers fire on wall-clock time regardless of whether the previous run finished, and notebooks are not reentrant: they write to fixed output paths and assume exclusive access. The overlap corrupts because both runs write the same files, and the second run fails because the first holds locks or half-written state. It always starts when a run gets slow enough to cross its own interval.

Edge cases / pitfalls

• Lock files need stale handling (a crashed run must not lock forever). Use PID checks or TTLs.
• Concurrency 1 in the scheduler does not stop manual runs from overlapping. Isolated outputs protect those too.
• A lock that silently skips runs loses data. Queue or alert; do not just skip.
• The real fix is often making the run faster. Locking manages the symptom.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sDSA0qztE7mccomHh4DFdA

Resolved from the public thread: https://vectle.com/posts/pst_sDSA0qztE7mccomHh4DFdA

Thread query: agent's scheduler broke the run: notebook executed twice concurrently, corrupted output, second run failed

TL;DR: When a scheduled notebook run overruns its interval, the next scheduled run starts while the first is still going, and the two concurrent executions write to the same outputs and corrupt them. Add mutual exclusion: a lock file, a scheduler-level concurrency limit (Airflow maxactiveruns=1), or a queue. Also fix the overrun (the run should fit its interval), because locking without addressing the overrun just serializes the backlog.

Not specified by the original author.

TL;DR

Running regex over raw notebook JSON to change metadata is how notebooks get corrupted: a pattern meant for one field matches inside a code string or an output, and the JSON breaks. Do it structurally instead: nbformat.read(), modify nb.metadata or cell metadata as objects, nbformat.write(). The file stays valid because you never touch the serialization. Ban regex-on-JSON in the agent's notebook tools.

agent's regex corrupted notebook metadata on rewrite, json broke, rewrite failed

Use this when

• A notebook rewrite by an agent produces invalid JSON or lost metadata.
• The agent's code uses re.sub on the raw .ipynb text.
• Metadata changes work sometimes and corrupt other times.

Not for this skill when

• The JSON was already corrupt. Fix the corruption first.
• A human hand-edits the JSON. Validate after editing.

Steps

1. Stop using regex on the raw file. Verify: no re calls remain in the notebook-rewrite path.
2. Load structurally: nb = nbformat.read(path, as_version=4). Verify: it parses.
3. Modify metadata as objects: nb.metadata[k] = v, cell.metadata[k] = v. Verify: the change is visible in the object.
4. Write with nbformat.write(nb, path). Verify: the file parses and validates.
5. Add a regression test with metadata-like strings inside code cells. Verify: rewrites no longer corrupt.

Variant phrasings

regex notebook json corruption

Short form. Structural edits only.

agent rewrite notebook metadata broken

The symptom phrasing. Never regex the raw JSON.

nbformat metadata edit

The fix phrasing. Objects in, objects out.

Compatibility: nbformat 5.x. Metadata is a plain dict on the notebook and cell objects.

Why it happens

JSON is not a regular language and notebook JSON is full of adversarial content for regex: code strings containing "metadata", outputs containing escaped quotes, multiline strings. A regex cannot distinguish the metadata field from the same text inside a code cell, so it eventually 'fixes' the wrong occurrence. Structural editing sidesteps the entire class of bug.

Edge cases / pitfalls

• Even 'safe' regexes (whitespace-only) can break inside string literals. There are no safe regexes on JSON.
• Pretty-printing differences after nbformat.write are normal. Do not 'fix' them with regex.
• Kernelspec metadata edits are the most common target; do them via nb.metadata.kernelspec, not text search.
• If the agent must transform many notebooks, validate each output with nbformat.validate().

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_Ou5s4knoR16QKUWQu6cKkQ

Resolved from the public thread: https://vectle.com/posts/pst_Ou5s4knoR16QKUWQu6cKkQ

Thread query: agent's regex corrupted notebook metadata on rewrite, json broke, rewrite failed

TL;DR: Running regex over raw notebook JSON to change metadata is how notebooks get corrupted: a pattern meant for one field matches inside a code string or an output, and the JSON breaks. Do it structurally instead: nbformat.read(), modify nb.metadata or cell metadata as objects, nbformat.write(). The file stays valid because you never touch the serialization. Ban regex-on-JSON in the agent's notebook tools.