dagster code location failed to load, grpc server error
Fixes a Dagster code location that fails to load with a gRPC server error, covering the user code process crashing on import, port and host wiring, and version mismatch between webserver and code server. Use when a location shows an error badge. Not for a single run failure inside a location that loads fine.
TL;DR
The Dagster webserver and daemon do not import your code in their own processes. They talk over gRPC to a code server that runs your repository, and a load failure means that server crashed, usually on an import error in your definitions, or the host and port in the workspace config point at nothing healthy. Run the code server entry point directly, read its own traceback, and fix that error; the gRPC wording in the UI is a wrapper around it.
dagster code location failed to load, grpc server errorUse this when
- A code location shows failed to load in Dagster with a gRPC or server error.
- The failure appeared right after a code deploy or a dependency change.
- All assets from one repository vanished while other locations still work.
Not for this skill when
- The location loads and a specific run fails. That error lives in the run, not the code server.
- The webserver itself is down for every location. That is a platform outage, not one code location.
Steps
- Open the code location error detail in the UI and copy the underlying traceback if it includes one. Note whether a Python exception from your code is present. Verify: any wrapped user code error is extracted
- Run the same entry point the code server uses, the module or file in your code server command, in that deployment environment and watch it start. Note the first import or startup error it prints. Verify: the code server failure reproduces outside gRPC
- Fix the startup cause in order of likelihood: a missing dependency in the code image, an exception at import time in the definitions file, or a resource such as a file or credential read at import. Note the fix. Verify: the definitions import cleanly in the code environment
- Check the workspace configuration: the host, port, and location name the webserver uses must match the code server actually listening, and versions of Dagster should match across the two. Note any mismatch found. Verify: webserver and code server agree on where to talk and in what version
- Restart the code server, reload the location in the UI, and confirm assets appear and a test run can launch. Note the location state is healthy. Verify: the location loads and executes
Variant phrasings
dagster code location error GrpcServerProcess startup failed
The fuller gRPC startup wording.
dagster repository location failed to load after deploy
The deploy timed phrasing.
Compatibility: Dagster 1.x deployments with separate code servers, including the Helm and container setups. In-process test runs will not reproduce a code server wiring fault.
Why it happens
The process isolation is deliberate: user code crashes should not take down the webserver or the daemon, and teams can deploy code independently. The cost is indirection in errors. The UI reports that its gRPC call failed, which is true and almost useless, while the real traceback sits in the code server's own logs or appears when you start the server yourself. Version skew aggravates it, because the gRPC protocol between mismatched Dagster versions can fail in ways that look like code errors.
Edge cases / pitfalls
- Code that reads a database or a credential store at import time makes location loading depend on that system. Move such reads into ops or resources.
- A code server that starts but is slow to import can time out the webserver health check on big projects. Raise the startup timeout rather than splitting blindly.
- Kubernetes code server pods crash looping on a bad image look identical to an import error in the UI. Check pod state before reading Python tracebacks that may not exist.
- After fixing, an old error can linger in the UI until the location is reloaded. Reload explicitly before declaring failure.
Provenance
Resolved from the public thread: https://vectle.com/posts/pst_ZQpygLFlf-ANydVkiLwbPw