Cloud SQL Proxy v2: "failed to connect to instance: Dial error: failed to dial ... i/o timeout"

Not specified by the original author.

What was going on After upgrading the Cloud SQL Auth Proxy from v1 to v2 (2.2.0) as a GKE sidecar, the proxy logged many "failed to connect to instance: Dial error: failed to dial (connection name = ...): dial tcp ...:3307: i/o timeout" errors across multiple PHP applications. Downgrading to v1 made the errors go away, the database was not under heavy load, and only ~5% of the connection limit was in use. The maintainer noted such timeouts usually indicate a network problem or a CPU-starved proxy container.

What fixed it The reporter resolved this by switching the Cloud SQL instance to a private IP and running the proxy with the --private-ip flag (they also switched GKE to Cloud DNS at the same time, so they could not isolate which change fixed it). The working theory is that the v2 proxy's dials were going through egress NAT/DNS in a way that timed out, and private IP removed that path. If you hit these dial i/o timeouts on proxy v2, try: 1) give the proxy container more CPU (the default 100m request is lean), and 2) switch to private IP with --private-ip on both the instance and the proxy. The reporter confirmed the errors stopped after the migration and the maintainer closed the issue. Source: https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/1840

In the thread, the question was: After upgrading the Cloud SQL Auth Proxy from v1 to v2 (2.2.0) as a GKE sidecar, the proxy logged many "failed to connect to instance: Dial error: failed to dial (connection name = ...): dial tcp ...:3307: i/o timeout" errors across multiple PHP applications. Downgrading to v1 made the errors go away, the database was not under heavy load, and only ~5% of the connection limit was in use. The maintainer noted such timeouts usually indicate a network problem or a CPU-starved proxy container. The verified resolution: The reporter resolved this by switching the Cloud SQL instance to a private IP and running the proxy with the --private-ip flag (they also switched GKE to Cloud DNS at the same time, so they could not isolate which change fixed it). The working theory is that the v2 proxy's dials were going through egress NAT/DNS in a way that timed out, and private IP removed that path. If you hit these dial i/o timeouts on proxy v2, try: 1) give the proxy container more CPU (the default 100m request is lean), and 2) switch to private IP with --private-ip on both the instance and the proxy. The reporter confirmed the errors stopped after the migration and the maintainer closed the issue. Source: https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/1840 Thread: https://vectle.com/threads/thr_vuAXT0mTm-7dsmZjFD4TTQ