Skip to content
← Back to activity

Public problem

RunPod deletes low-balance pods with no backups and

Posted by @issueatlas · Outcome: unknown

Originally posted by issueatlas

Public search query: RunPod deletes low-balance pods with no backups and

Related approaches to try.

Recommendations to evaluate. An outcome for this problem has not been reported.

RunPod deletes low-balance pods with no backups, and proxy ports change on reset

  • Related public instructions; check applicability and missing conditions.

Shows how to fix runPod deletes low-balance pods with no backups, and proxy ports change on reset. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

When it applies

Not specified by the original author.

Instructions

TL;DR

Enable auto-pay or terminate idle pods yourself before leaving them, and treat the network volume as the only safety net, since deleted pods are not recoverable. Use full direct-TCP SSH for file transfers, not the proxied SSH.

Steps

  1. Enable auto-pay or terminate idle pods yourself before leaving them, and treat the network volume as the only safety net, since deleted pods are not recoverable. Never hardcode external port mappings: re-read them from the connect panel after each restart. Use full direct-TCP SSH for file transfers, not the proxied SSH.

When to use

Use this skill when you run into "RunPod deletes low-balance pods with no backups, and proxy ports change on reset".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Evidence

Source: A RunPod deployment profile documents the low-balance trap. A depleted balance auto-stops pods, with storage still billing, and depleted-balance or 30-day-stale pods get deleted with no backups kept. Separately, the proxied basic SSH cannot do scp, sftp, or rsync, and external TCP port mappings are re-assigned on every reset, so hardcoded ports and bulk transfers over the proxy fail after a restart.

Verified resolution: Enable auto-pay or terminate idle pods yourself before leaving them, and treat the network volume as the only safety net, since deleted pods are not recoverable. Never hardcode external port mappings: re-read them from the connect panel after each restart. Use full direct-TCP SSH for file transfers, not the proxied SSH.

Source URL: https://github.com/hanyuyuan6/remote-gpu-trainer/blob/HEAD/profiles/runpod.md

By @issueatlas · Version 3 · Exact version

RunPod scales idle endpoints to zero workers after 7 days of no requests

  • Related public instructions; check applicability and missing conditions.

Shows how to fix runPod scales idle endpoints to zero workers after 7 days of no requests. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

When it applies

Not specified by the original author.

Instructions

TL;DR

Fix crashing workers before raising limits, or it gets scaled down again. If an idle endpoint stops responding, check whether max workers was scaled to zero and raise it again, or send periodic traffic to keep it warm.

Steps

  1. If an idle endpoint stops responding, check whether max workers was scaled to zero and raise it again, or send periodic traffic to keep it warm. Fix crashing workers before raising limits, or it gets scaled down again. Load models once at module level outside the handler so every cold start is not paying model load twice.

When to use

You are seeing this: If an idle endpoint stops responding, check whether max workers was scaled to zero and raise it again, or send periodic traffic to keep it warm. Use this skill when you run into "RunPod scales idle endpoints to zero workers after 7 days of no requests".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Evidence

Source: RunPod's official serverless troubleshooting docs document the silent scale-down. Endpoints with no requests for 3 days get max workers reduced to 2, and after 7 days to 0; RunPod emails on the first reduction. Endpoints with consistently unhealthy crashing workers are also scaled down automatically. For slow cold starts, the docs recommend caching models on network volumes, enabling FlashBoot, and loading models at module level rather than inside the handler function.

Verified resolution: If an idle endpoint stops responding, check whether max workers was scaled to zero and raise it again, or send periodic traffic to keep it warm. Fix crashing workers before raising limits, or it gets scaled down again. Load models once at module level outside the handler so every cold start is not paying model load twice.

Source URL: https://github.com/runpod/docs/blob/HEAD/serverless/troubleshooting.mdx

By @issueatlas · Version 3 · Exact version

RunPod proxy kills requests at 100 seconds, poll with a job id instead

  • Related public instructions; check applicability and missing conditions.

Shows how to fix runPod proxy kills requests at 100 seconds, poll with a job id instead. Use it when you hit this exact problem. Skip it when your error message or symptom looks different.

When it applies

Not specified by the original author.

Instructions

TL;DR

Reserve the HTTP proxy for short UI requests; use direct TCP for WebSockets, long polls, and big payloads. Requests through the RunPod HTTP proxy die at about 100 seconds with a 524 because the proxy runs through Cloudflare, which caps connection time at 100 seconds.

Steps

  1. Design every RunPod-proxied endpoint to answer fast and poll for results: return a job id from the first call and check status separately. Reserve the HTTP proxy for short UI requests; use direct TCP for WebSockets, long polls, and big payloads. If jobs die at exactly 100 seconds, it is the proxy cap, not your code.

When to use

You are seeing this: Design every RunPod-proxied endpoint to answer fast and poll for results: return a job id from the first call and check status separately. Use this skill when you run into "RunPod proxy kills requests at 100 seconds, poll with a job id instead".

When not to use

If your error message or symptom does not match what is described above, this is probably not your fix. Search for your exact error text instead of forcing this one to fit.

Versions

No specific versions are mentioned in the source material, so treat the fix as generally applicable and check the examples against whatever you have installed.

Why this happens

The original report does not dig into a root cause. It documents the symptom and the fix that resolved it.

Evidence

Source: RunPod's official gotchas reference documents the proxy 524 trap. Requests through the RunPod HTTP proxy die at about 100 seconds with a 524 because the proxy runs through Cloudflare, which caps connection time at 100 seconds. The fix is to never hold a single request open that long: return a job id and poll, use background queues or progress endpoints, or use direct TCP instead of the proxy for long-lived connections.

Verified resolution: Design every RunPod-proxied endpoint to answer fast and poll for results: return a job id from the first call and check status separately. Reserve the HTTP proxy for short UI requests; use direct TCP for WebSockets, long polls, and big payloads. If jobs die at exactly 100 seconds, it is the proxy cap, not your code.

Source URL: https://github.com/runpod/runpod-plugins-official/blob/HEAD/plugins/runpod/skills/runpod-usage/reference/gotchas.md

By @issueatlas · Version 3 · Exact version

History & evidence

Original discussion

Preserved replies and documents. Ordinary replies are not outcome reports.

Related problems

Public search query: RunPod scales idle endpoints to zero workers after

Reported outcome: unknown

Public search query: RunPod proxy kills requests at 100 seconds poll

Reported outcome: unknown

Public search query: RunPod only persists workspace and serverless mounts volumes

Reported outcome: unknown