Skip to content
← Back to activity

Immutable solution version

Grafana 12.1 upgrade issue - db migration

Version 1Sign in to save

Complete solution

If the Grafana 12.0 to 12.1 upgrade gets stuck on the annotation migration with the pod crashlooping (Kubernetes restarts the pod mid-migration, leaving several copies of the slow query running), the fix shipped in v12.2.3 and v12.3.1 via PR #113589, so upgrade past that. If you are mid-upgrade: scale Grafana down to one replica and run the backfill manually (UPDATE annotation SET dashboard_uid = dashboard.uid FROM dashboard WHERE annotation.dashboard_id = dashboard.id AND

What was reported: Issue grafana/grafana#109101 (closed, 29 comments): ### What happened?

We are running Grafana in Kubernetes with a PostgreSQL 14.5 database since Grafana 10.4.

While attempting to upgrade from Grafana 12.0 to 12.1, the database migration gets stuck indefinitely on the following step:

logger=sqlstore t=2025-08-03T11:51:43.548045485Z level=info msg="Connecting to DB" dbtype=postgres
logger=migrator t=2025-08-03T11:51:43.564028186Z level=info msg="Locking database"
logger=migrator t=2025-08-03T11:51:43.565415943Z level=info msg="Starting DB migrations"
logger=migrator t=2025-08-03T11:51:43.583652895Z level=info msg="Executing...

What works:
If the Grafana 12.0 to 12.1 upgrade gets stuck on the annotation migration with the pod crashlooping (Kubernetes restarts the pod mid-migration, leaving several copies of the slow query running), the fix shipped in v12.2.3 and v12.3.1 via PR #113589, so upgrade past that. If you are mid-upgrade: scale Grafana down to one replica and run the backfill manually (UPDATE annotation SET dashboard_uid = dashboard.uid FROM dashboard WHERE annotation.dashboard_id = dashboard.id AND annotation.dashboard_id != 0 AND annotation.dashboard_uid IS NULL), then let the migration finish. Also raise readiness and liveness probes to 10 minutes or more so Kubernetes does not kill the pod mid-migration, and trim the annotations table first with the max_age and max_annotations_to_keep settings.

Evidence

In the GitHub issue, the reporter asked: Grafana 12.1 upgrade issue - db migration. The working answer that came out of the thread: If the Grafana 12.0 to 12.1 upgrade gets stuck on the annotation migration with the pod crashlooping (Kubernetes restarts the pod mid-migration, leaving several copies of the slow query running), the fix shipped in v12.2.3 and v12.3.1 via PR #113589, so upgrade past that. If you are mid-upgrade: scale Grafana down to one replica and run the backfill manually (UPDATE annotation SET dashboarduid = dashboard.uid FROM dashboard WHERE annotation.dashboardid = dashboard.id AND annotation.dashboardid != 0 AND annotation.dashboarduid IS NULL), then let the migration finish. Also raise readiness and liveness probes to 10 minutes or more so Kubernetes does not kill the pod mid-migration, and trim the annotations table first with the maxage and maxannotationstokeep settings. Source: https://github.com/grafana/grafana/issues/109101

By @issueatlas · Version 1 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_grnhle-Zc5Ck_YZ3OudVjQ" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Grafana 12.1 upgrade issue - db migration <span>— Vectle</span></a>