Grafana Datasource error with Loki: connection debugging

Not specified by the original author.

TL;DR

Datasource errors for Loki are connectivity or auth between Grafana and Loki: wrong URL, auth headers missing or expired, or a proxy in between breaking the connection. Test the datasource from Grafana's server side (not your browser), check what changed since it worked, and verify the exact URL Grafana uses. Browser-side vs server-side access mode is the classic confusion.

The query

Grafana "Datasource error" with Loki: connection debugging

Use this when

• Loki panels show datasource errors
• A working datasource suddenly breaks
• Setting up the Loki datasource for the first time
• After Grafana or Loki upgrades

Not for when

• LogQL syntax or query errors (different message)
• Slow queries (performance, different topic)
• Loki ingestion problems

Steps

Step 1: Use Save and Test and read the message

Grafana's datasource Save and Test button returns the specific error: connection refused, 401, 404, timeout. Each maps to a different layer. Start here, not in the panel. Expected output: the specific connection error classified.

Step 2: Check server vs browser access mode

Grafana can query Loki from its backend (server access) or from your browser (browser access). Browser mode fails on CORS and network reachability from your laptop; server mode fails on Grafana-to-Loki connectivity. Know which mode you configured. Expected output: the mode identified; the failing network path known.

Step 3: Verify the URL Grafana uses

Check the datasource URL character by character: scheme, host, port, path. Common misses: http vs https, missing port, a path prefix the Loki deployment requires. Test the URL from the Grafana host directly. Expected output: the URL proven reachable from the right network location.

Step 4: Check authentication

Verify auth headers, basic auth credentials, or tokens configured on the datasource. Rotated credentials break working datasources silently; the error looks like a connection problem. Expected output: valid credentials confirmed; auth errors resolved.

Step 5: Look for proxies in the path

Reverse proxies between Grafana and Loki (auth proxies, ingress) can break the connection with their own errors. Check proxy logs when direct connections work but the datasource fails. Expected output: the proxy hop identified and fixed or ruled out.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_JjpVCl9426qbHM3gZK1teQ

Resolved from the public thread: https://vectle.com/posts/pst_JjpVCl9426qbHM3gZK1teQ

Thread query: Grafana "Datasource error" with Loki: connection debugging

TL;DR: Datasource errors for Loki are connectivity or auth between Grafana and Loki: wrong URL, auth headers missing or expired, or a proxy in between breaking the connection. Test the datasource from Grafana's server side (not your browser), check what changed since it worked, and verify the exact URL Grafana uses. Browser-side vs server-side access mode is the classic confusion.

Not specified by the original author.

TL;DR

Test the data source connection, fix the query time range and filters, and confirm the underlying incident data exists before changing the panel. Do this before posting workarounds to customers: confirm the fault in Grafana, fix the single mismatched value or setting described in the steps, then repeat the original action and check the public page as a logged-out visitor. Most cases of this error are a configuration or value mismatch in Grafana status panel data source, not a provider outage, so a methodical check clears it in minutes.

Grafana status panel not reflecting incident: data source query failed error

Use this when

• The exact error above appears while working with Grafana status panel data source during an incident or a test incident.
• The public status page does not reflect what responders saved in Grafana.
• An automation, integration, or responder hits this failure repeatedly with the same payload or setting.
• You need customer communication moving again and have admin access to the status page.

Not for this skill when

• Grafana reports a confirmed platform outage on its own status page. Wait for the provider and post a holding update instead.
• You are designing a new status page or choosing between providers. This skill only repairs the named failure.
• The failure is in the underlying product outage itself, not in status communication. Fix the product incident first.

Steps

1. Reproduce the failure once and capture the exact request, response, and timestamp from Grafana. Note the page id, component or incident id, and the account or automation that sent the request, because most status page faults are scoped to one page or one credential. Verify: you have the full error body saved, not just a status code screenshot
2. Run the same query in Explore and note any error the panel hides. Work in the admin console or with a read-only API call first so you do not change customer-visible state while diagnosing. Verify: the setting or value you found matches what the automation believes it is using, or you have identified the exact mismatch
3. Test the data source and re-enter credentials if the test fails. Change one thing at a time and keep the previous value noted so you can roll back if the incident is live. Verify: the change is saved in Grafana and visible on a fresh page load, not only in the editor
4. Widen the time range and remove filters temporarily to confirm incident rows return. Prefer the smallest customer-safe test: a test incident, a test subscriber, or a staging page where one exists. Verify: the test produces the expected public result without touching the live incident
5. Repeat the original failing action exactly as the responder or automation performed it, using the same account, page, and payload shape with the corrected values. Verify: the action succeeds and the public status page shows the intended state to a logged-out visitor
6. Record what changed in the incident log or runbook: the faulty value, the corrected value, and where the automation reads it from. Add a pre-incident check for this setting to the responder checklist if the failure blocked comms during a live outage. Verify: the runbook entry exists and the next on-call can find the check in under a minute

Why it happens

The panel only renders what its query returns. A failing data source, an expired credential, or a filter that excludes the current incident makes the panel look stale while incidents exist. The failure tends to surface mid-incident because that is when new automations, new responders, and new values meet the page for the first time under time pressure. Configuration that was copied from another page, another provider, or a test environment carries values the live page does not accept, and the error message usually names the symptom rather than the mismatched setting. Treating it as a value or permission mismatch, and checking the live page configuration against what the sender used, resolves the great majority of cases without provider support.

Variants and edge cases

Grafana status panel not reflecting incident - variant with a different tool in the chain

The same failure arriving through an integration (for example a bridge from an incident tool) instead of the admin console. Diagnose on the Grafana side first, because the bridge usually collapses the real error into a generic one, then fix the mapping or credential the bridge uses.

Same error on a second page or staging page

Multi-page accounts repeat this fault when configuration is copied between pages. Compare the working page and the failing page setting by setting, especially ids, roles, and allowed values, rather than assuming the fix from the first page transfers unchanged.

Error returns after working for weeks

A sudden recurrence usually means a rotation, rename, or cleanup changed the underlying value: a key was rotated, a component renamed, a template edited, or a group membership changed. Check recent admin audit history for the affected page before changing anything else.

Compatibility: Applies to Grafana current web console and API as of October 2026, and to common bridges from PagerDuty, Opsgenie, Slack, Datadog, and incident tools that publish to it. Confirm field names against the provider changelog if your page behaves differently.

Provenance

Resolved from the public thread: https://vectle.com/posts/pst_sAWYcVM93vzOTRYguTCtyg

Resolved from the public thread: https://vectle.com/posts/pst_sAWYcVM93vzOTRYguTCtyg

Thread query: Grafana status panel not reflecting incident: data source query failed error

TL;DR: Test the data source connection, fix the query time range and filters, and confirm the underlying incident data exists before changing the panel. Do this before posting workarounds to customers: confirm the fault in Grafana, fix the single mismatched value or setting described in the steps, then repeat the original action and check the public page as a logged-out visitor. Most cases of this error are a configuration or value mismatch in Grafana status panel data source, not a provider outage, so a methodical check clears it in minutes.

Not specified by the original author.

TL;DR

Fix: point the data source at an address that resolves from the Grafana container. In Docker Compose use the service hostname (prometheus:9090 instead of the loopback address), or the Prometheus container's IP; on Docker Desktop you can also use the host-gateway hostname.

Steps

1. The Prometheus UI works from your browser, but Grafana doesnt query from your browser: the Grafana server makes the request itself. The loopback address of one container is not the loopback address of another, even with published ports, so a data source URL pointing at the container's own loopback can never reach Prometheus.
2. Fix: point the data source at an address that resolves from the Grafana container. In Docker Compose use the service hostname (prometheus:9090 instead of the loopback address), or the Prometheus container's IP; on Docker Desktop you can also use the host-gateway hostname. The reporter confirmed: "i configure the ip of prometheus in the grafana, it worked!"

When to use

You are seeing this: After adding Prometheus as a Grafana data source, the save/test step failed with Error reading Prometheus:... dial tcp... connect: connection refused, even though the Prometheus UI opened fine in the browser and all scrape targets were up. Use this skill when you run into "Grafana Prometheus data source: connection refused in Docker (loopback mixup)".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

After adding Prometheus as a Grafana data source, the save/test step failed with Error reading Prometheus:... dial tcp... connect: connection refused, even though the Prometheus UI opened fine in the browser and all scrape targets were up.

In the thread Grafana Prometheus data source: connection refused in Docker (loopback mixup), the question was: After adding Prometheus as a Grafana data source, the save/test step failed with Error reading Prometheus: ... dial tcp ... connect: connection refused, even though the Prometheus UI opened fine in the browser and all scrape targets were up. The reporter was running Grafana and Prometheus in Docker. The verified resolution: The Prometheus UI works from your browser, but Grafana doesnt query from your browser: the Grafana server makes the request itself. The loopback address of one container is not the loopback address of another, even with published ports, so a data source URL pointing at the container's own loopback can never reach Prometheus.

Fix: point the data source at an address that resolves from the Grafana container. In Docker Compose use the service hostname (prometheus:9090 instead of the loopback address), or the Prometheus container's IP; on Docker Desktop you can also use the host-gateway hostname. The reporter confirmed: "i configure the ip of prometheus in the grafana, it worked!" Original discussion: https://vectle.com/threads/thr_nEOZn9AfVVvWGCBJLkciEw