CockroachDB on Kubernetes with Istio: "connection refused" - how to run insecure statefulset with istio mTLS

TL;DR

Running CockroachDB (insecure mode) as a StatefulSet on GKE with Istio injected, cockroach sql --insecure failed with connection refused even though the pods were running.

Steps

1. Running CockroachDB (insecure mode) as a StatefulSet on GKE with Istio injected, cockroach sql --insecure failed with connection refused even though the pods were running. The root cause discussed in the thread: the install manifests name the 26257 port grpc, so the Istio sidecar tries to speak gRPC to it, but CockroachDB's SQL protocol is not TLS/gRPC-first and the handshake breaks.
2. Rename the 26257 port from grpc to tcp in the CockroachDB Service and StatefulSet manifests so the Istio sidecar treats the traffic as plain TCP instead of trying gRPC protocol sniffing. That is the fix multiple users confirmed (one via mike-holberger's manifests, another directly: changing grpc to tcp in the service and statefulset definitions made it work). On newer Istio you can alternatively set appProtocol: tcp on the port, per Istio's protocol-selection docs. Note the maintainer closed the issue as not-planned: CockroachDB is not compatible with Istio mTLS because the SQL protocol is not TLS-first, and production --insecure deployments are unsupported - so treat this as a workaround for dev/test setups, not a supported configuration. Source: https://github.com/cockroachdb/cockroach/issues/19667

When to use

You are seeing this: Mined from a verified thread: https://vectle.com/threads/thriZOrrCjByZvd0jO2kCYBg (original: https://github.com/cockroachdb/cockroach/issues/19667) Use this skill when you run into "CockroachDB on Kubernetes with Istio: "connection refused" - how to run insecure statefulset with istio mTLS".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Note the maintainer closed the issue as not-planned: CockroachDB is not compatible with Istio mTLS because the SQL protocol is not TLS-first, and production --insecure deployments are unsupported - so treat this as a workaround for dev/test setups, not a supported configuration.

Sources

• https://github.com/cockroachdb/cockroach/issues/19667

In the thread, the asker reported: Running CockroachDB (insecure mode) as a StatefulSet on GKE with Istio injected, cockroach sql --insecure failed with connection refused even though the pods were running. The root cause discussed in the thread: the install manifests name the 26257 port grpc, so the Istio sidecar tries to speak gRPC to it, but CockroachDB's SQL protocol is not TLS/gRPC-first and the handshake breaks. The verified answer that resolved it: Rename the 26257 port from grpc to tcp in the CockroachDB Service and StatefulSet manifests so the Istio sidecar treats the traffic as plain TCP instead of trying gRPC protocol sniffing. That is the fix multiple users confirmed (one via mike-holberger's manifests, another directly: changing grpc to tcp in the service and statefulset definitions made it work). On newer Istio you can alternatively set appProtocol: tcp on the port, per Istio's protocol-selection docs. Note the maintainer closed the issue as not-planned: CockroachDB is not compatible with Istio mTLS because the SQL protocol is not TLS-first, and production --insecure deployments are unsupported - so treat this as a workaround for dev/test setups, not a supported configuration. Source: https://github.com/cockroachdb/cockroach/issues/19667 Thread: https://vectle.com/threads/thriZOrrCjByZvd0jO2kCYBg