Grafana Prometheus data source: connection refused in Docker (loopback mixup)

TL;DR

Fix: point the data source at an address that resolves from the Grafana container. In Docker Compose use the service hostname (prometheus:9090 instead of the loopback address), or the Prometheus container's IP; on Docker Desktop you can also use the host-gateway hostname.

Steps

1. The Prometheus UI works from your browser, but Grafana doesnt query from your browser: the Grafana server makes the request itself. The loopback address of one container is not the loopback address of another, even with published ports, so a data source URL pointing at the container's own loopback can never reach Prometheus.
2. Fix: point the data source at an address that resolves from the Grafana container. In Docker Compose use the service hostname (prometheus:9090 instead of the loopback address), or the Prometheus container's IP; on Docker Desktop you can also use the host-gateway hostname. The reporter confirmed: "i configure the ip of prometheus in the grafana, it worked!"

When to use

You are seeing this: After adding Prometheus as a Grafana data source, the save/test step failed with Error reading Prometheus:... dial tcp... connect: connection refused, even though the Prometheus UI opened fine in the browser and all scrape targets were up. Use this skill when you run into "Grafana Prometheus data source: connection refused in Docker (loopback mixup)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

After adding Prometheus as a Grafana data source, the save/test step failed with Error reading Prometheus:... dial tcp... connect: connection refused, even though the Prometheus UI opened fine in the browser and all scrape targets were up.

In the thread Grafana Prometheus data source: connection refused in Docker (loopback mixup), the question was: After adding Prometheus as a Grafana data source, the save/test step failed with Error reading Prometheus: ... dial tcp ... connect: connection refused, even though the Prometheus UI opened fine in the browser and all scrape targets were up. The reporter was running Grafana and Prometheus in Docker. The verified resolution: The Prometheus UI works from your browser, but Grafana doesnt query from your browser: the Grafana server makes the request itself. The loopback address of one container is not the loopback address of another, even with published ports, so a data source URL pointing at the container's own loopback can never reach Prometheus.

Fix: point the data source at an address that resolves from the Grafana container. In Docker Compose use the service hostname (prometheus:9090 instead of the loopback address), or the Prometheus container's IP; on Docker Desktop you can also use the host-gateway hostname. The reporter confirmed: "i configure the ip of prometheus in the grafana, it worked!" Original discussion: https://vectle.com/threads/thr_nEOZn9AfVVvWGCBJLkciEw