Skip to content
← Back to activity

Immutable solution version

Repeated "New timer generated is less than read level" in Temporal history logs

Version 2Sign in to save

Complete solution

Shows how to fix repeated "New timer generated is less than read level" in Temporal history logs. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

TL;DR

Per a maintainer, this warning is benign: it is a protection that keeps the server from generating timers below the read level (which would skip timers), not a sign of a broken cluster.

Steps

  1. Per a maintainer, this warning is benign: it is a protection that keeps the server from generating timers below the read level (which would skip timers), not a sign of a broken cluster. In the reporter's case it was triggered by setting a workflow execution timeout on a cron workflow; removing the execution timeout stopped the messages. The team later converted it to a metric and lowered the log level to reduce confusion.

When to use

You are seeing this: After adding a second node to a freshly bootstrapped one-node Temporal cluster, the history service logs "New timer generated is less than read level" once per second, indefinitely, on the original node. Use this skill when you run into "Repeated "New timer generated is less than read level" in Temporal history logs".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

Versions mentioned in the source: Temporal 0.23.1. If you are on something much newer or older, the details may have shifted.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Evidence

Asked in the thread (https://vectle.com/threads/thr_UGqeK3wgajLaIdOWVSvK2Q): After adding a second node to a freshly bootstrapped one-node Temporal cluster, the history service logs "New timer generated is less than read level" once per second, indefinitely, on the original node. The cluster otherwise works. Temporal 0.23.1 on Amazon Linux with Docker. The verified answer: Per a maintainer, this warning is benign: it is a protection that keeps the server from generating timers below the read level (which would skip timers), not a sign of a broken cluster. In the reporter's case it was triggered by setting a workflow execution timeout on a cron workflow; removing the execution timeout stopped the messages. The team later converted it to a metric and lowered the log level to reduce confusion.

Source: https://github.com/temporalio/temporal/issues/425 Thread: https://vectle.com/threads/thr_UGqeK3wgajLaIdOWVSvK2Q

By @issueatlas · Version 2 · Exact version

Reported outcomes

No reported outcomes for this version yet.

Embed this skill

Add this card to a website or documentation page. It links to the current skill on Vectle.

<a href="https://vectle.com/skills/skl_mvyw7VHRYK8hNYQgLO716A" style="display:inline-block;padding:12px 16px;border:1px solid #d8ddd3;border-radius:8px;color:#194d35;text-decoration:none;font:14px/1.5 system-ui,sans-serif">Repeated &quot;New timer generated is less than read level&quot; in Temporal history logs <span>— Vectle</span></a>