CockroachDB on Kubernetes with Istio: "connection refused" - how to run insecure statefulset with istio mTLS

CockroachDB on Kubernetes with Istio: "connection refused" - how to run insecure statefulset with istio mTLS

What is going on

Running CockroachDB (insecure mode) as a StatefulSet on GKE with Istio injected, cockroach sql --insecure failed with connection refused even though the pods were running. The root cause discussed in the thread: the install manifests name the 26257 port grpc, so the Istio sidecar tries to speak gRPC to it, but CockroachDB's SQL protocol is not TLS/gRPC-first and the handshake breaks.

What actually works

Rename the 26257 port from grpc to tcp in the CockroachDB Service and StatefulSet manifests so the Istio sidecar treats the traffic as plain TCP instead of trying gRPC protocol sniffing. That is the fix multiple users confirmed (one via mike-holberger's manifests, another directly: changing grpc to tcp in the service and statefulset definitions made it work). On newer Istio you can alternatively set appProtocol: tcp on the port, per Istio's protocol-selection docs. Note the maintainer closed the issue as not-planned: CockroachDB is not compatible with Istio mTLS because the SQL protocol is not TLS-first, and production --insecure deployments are unsupported - so treat this as a workaround for dev/test setups, not a supported configuration. Source: https://github.com/cockroachdb/cockroach/issues/19667

Source

Mined from a verified thread: https://vectle.com/threads/thriZOrrCjByZvd0jO2kCYBg (original: https://github.com/cockroachdb/cockroach/issues/19667)

In the thread, the asker reported: Running CockroachDB (insecure mode) as a StatefulSet on GKE with Istio injected, cockroach sql --insecure failed with connection refused even though the pods were running. The root cause discussed in the thread: the install manifests name the 26257 port grpc, so the Istio sidecar tries to speak gRPC to it, but CockroachDB's SQL protocol is not TLS/gRPC-first and the handshake breaks. The verified answer that resolved it: Rename the 26257 port from grpc to tcp in the CockroachDB Service and StatefulSet manifests so the Istio sidecar treats the traffic as plain TCP instead of trying gRPC protocol sniffing. That is the fix multiple users confirmed (one via mike-holberger's manifests, another directly: changing grpc to tcp in the service and statefulset definitions made it work). On newer Istio you can alternatively set appProtocol: tcp on the port, per Istio's protocol-selection docs. Note the maintainer closed the issue as not-planned: CockroachDB is not compatible with Istio mTLS because the SQL protocol is not TLS-first, and production --insecure deployments are unsupported - so treat this as a workaround for dev/test setups, not a supported configuration. Source: https://github.com/cockroachdb/cockroach/issues/19667 Thread: https://vectle.com/threads/thriZOrrCjByZvd0jO2kCYBg