Galileo: prototype in the playground, version code-run experiments

Galileo: prototype in the playground, version code-run experiments

TL;DR

Use the Galileo playground to iterate on a prompt, then move the winning setup into a code-run experiment for anything you need to reproduce. Playground outputs are saved to the experiment log stream, so the comparison history stays intact when you switch. Keep prompts, model names, and dataset versions in your own version control; the log stream records what ran but is not a substitute for code.

Use this when you run into the situation in the title.

When not to use this skill: unrelated tasks. It covers only the procedure above.

Compatibility

The steps above apply to the commands named in them. This skill does not pin a version, so if a flag looks different on your machine, check your installed version's docs first.

Details

When a playground result looks better than a code run, diff the exact prompt text and parameters, not your memory of them.

Context: Official docs (docs.galileo.ai, Running experiments): documents the difference that matters for reproducibility: playground runs are interactive, but code-run experiments are the ones you can version and rerun; playground outputs are still saved to the experiment log stream.

Official docs (docs.galileo.ai, Running experiments): documents the difference that matters for reproducibility: playground runs are interactive, but code-run experiments are the ones you can version and rerun; playground outputs are still saved to the experiment log stream. Resolved: Use the Galileo playground to iterate on a prompt, then move the winning setup into a code-run experiment for anything you need to reproduce. Playground outputs are saved to the experiment log stream, so the comparison history stays intact when you switch. Keep prompts, model names, and dataset versions in your own version control; the log stream records what ran but is not a substitute for code. When a playground result looks better than a code run, diff the exact prompt text and parameters, not your memory of them.