Grafana 12.1 upgrade issue - db migration

TL;DR

While attempting to upgrade from Grafana 12.0 to 12.1, the database migration gets stuck indefinitely on the following step: ``` logger=sqlstore t=2025-08-03T11:51:43.548045485Z level=info msg="Connecting to DB" dbtype=postgres logger=migrator t=2025-08-03T11:51:43.564028186Z level=info msg="Locking database" logger=migrator t=2025-08-03T11:51:43.565415943Z level=info msg="Starting DB migrations" logger=migrator t=2025-08-03T11:51:43.583652895Z level=info msg="Executing...

Steps

1. What was reported:

Issue grafana/grafana#109101 (closed, 29 comments): ### What happened?

2. We are running Grafana in Kubernetes with a PostgreSQL 14.5 database since Grafana 10.4.
3. While attempting to upgrade from Grafana 12.0 to 12.1, the database migration gets stuck indefinitely on the following step:
4. `

logger=sqlstore t=2025-08-03T11:51:43.548045485Z level=info msg="Connecting to DB" dbtype=postgres logger=migrator t=2025-08-03T11:51:43.564028186Z level=info msg="Locking database" logger=migrator t=2025-08-03T11:51:43.565415943Z level=info msg="Starting DB migrations" logger=migrator t=2025-08-03T11:51:43.583652895Z level=info msg="Executing...

5. What works:

If the Grafana 12.0 to 12.1 upgrade gets stuck on the annotation migration with the pod crashlooping (Kubernetes restarts the pod mid-migration, leaving several copies of the slow query running), the fix shipped in v12.2.3 and v12.3.1 via PR #113589, so upgrade past that. If you are mid-upgrade: scale Grafana down to one replica and run the backfill manually (UPDATE annotation SET dashboarduid = dashboard.uid FROM dashboard WHERE annotation.dashboardid = dashboard.id AND annotation.dashboardid != 0 AND annotation.dashboarduid IS NULL), then let the migration finish. Also raise readiness and liveness probes to 10 minutes or more so Kubernetes does not kill the pod mid-migration, and trim the annotations table first with the maxage and maxannotationstokeep settings.

Expected: What works: If the Grafana 12.0 to 12.1 upgrade gets stuck on the annotation migration with the pod crashlooping (Kubernetes restarts the pod mid-migration, leaving several copies of the slow query running), the fix shipped in v12.2.3 and v12.3.1 via PR #113589, so upgrade past that.

When to use

You are seeing this: What was reported: Issue grafana/grafana#109101 (closed, 29 comments): ### What happened? Use this skill when you run into "Grafana 12.1 upgrade issue - db migration".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: v12.2.3, v12.3.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Edge cases

• Also raise readiness and liveness probes to 10 minutes or more so Kubernetes does not kill the pod mid-migration, and trim the annotations table first with the maxage and maxannotationstokeep settings.

In the GitHub issue, the reporter asked: Grafana 12.1 upgrade issue - db migration. The working answer that came out of the thread: If the Grafana 12.0 to 12.1 upgrade gets stuck on the annotation migration with the pod crashlooping (Kubernetes restarts the pod mid-migration, leaving several copies of the slow query running), the fix shipped in v12.2.3 and v12.3.1 via PR #113589, so upgrade past that. If you are mid-upgrade: scale Grafana down to one replica and run the backfill manually (UPDATE annotation SET dashboarduid = dashboard.uid FROM dashboard WHERE annotation.dashboardid = dashboard.id AND annotation.dashboardid != 0 AND annotation.dashboarduid IS NULL), then let the migration finish. Also raise readiness and liveness probes to 10 minutes or more so Kubernetes does not kill the pod mid-migration, and trim the annotations table first with the maxage and maxannotationstokeep settings. Source: https://github.com/grafana/grafana/issues/109101