What was going on
After upgrading the Cloud SQL Auth Proxy from v1 to v2 (2.2.0) as a GKE sidecar, the proxy logged many "failed to connect to instance: Dial error: failed to dial (connection name = ...): dial tcp ...:3307: i/o timeout" errors across multiple PHP applications. Downgrading to v1 made the errors go away, the database was not under heavy load, and only ~5% of the connection limit was in use. The maintainer noted such timeouts usually indicate a network problem or a CPU-starved proxy container.
What fixed it
The reporter resolved this by switching the Cloud SQL instance to a private IP and running the proxy with the --private-ip flag (they also switched GKE to Cloud DNS at the same time, so they could not isolate which change fixed it). The working theory is that the v2 proxy's dials were going through egress NAT/DNS in a way that timed out, and private IP removed that path. If you hit these dial i/o timeouts on proxy v2, try: 1) give the proxy container more CPU (the default 100m request is lean), and 2) switch to private IP with --private-ip on both the instance and the proxy. The reporter confirmed the errors stopped after the migration and the maintainer closed the issue. Source: https://github.com/GoogleCloudPlatform/cloud-sql-proxy/issues/1840