Symptom: the same message delivered multiple times while the handler is still running it, or gRPC DEADLINE_EXCEEDED on the subscriber.
Cause: ack deadline (default 10s) expired before ack. Pub/Sub redelivered. Your handler is fine; its lease is too short.
Confirm:
1. Check the subscription ack deadline: `gcloud pubsub subscriptions describe [SUB]`.
2. Measure handler time: log start/end per message_id. If p99 exceeds the ack deadline, diagnosis confirmed.
3. Check Monitoring for redelivery counts spiking alongside slow handlers.
Fix:
1. Raise the ack deadline to cover p99 with headroom: `gcloud pubsub subscriptions update [SUB] --ack-deadline [SECONDS]` (max 600).
2. Prefer the streaming-pull client library, which extends leases while the handler runs. Manual pull loops never extend.
3. Set flow control (max outstanding messages) so slow handlers do not pile up redeliveries.
4. Add a dead-letter topic with max_delivery_attempts. One poison message that always exceeds the deadline will otherwise redeliver forever.
Do not set 600s blindly for every subscription. Long deadlines delay redelivery of genuinely failed messages. Match the deadline to the handler.
Make handlers idempotent regardless: dedupe on message_id. At-least-once means duplicates are always possible, even with perfect tuning.
Verify: redelivery count in Monitoring drops to near zero after the change, and each message_id is processed once in your logs.