Symptoms: gRPC DEADLINE_EXCEEDED on subscribers, or the same message delivered over and over while the handler is still running.
Meaning: the ack deadline (default 10 seconds, max 600) expired before your code acked. Pub/Sub assumes the message failed and redelivers. Your handler is not broken; its lease is just too short.
Fix:
1. Measure your handler's p99 time. Set the subscription ack deadline above it: `gcloud pubsub subscriptions update [SUB] --ack-deadline 300`.
2. Better: use the streaming pull client library, which extends leases automatically while the handler runs. Manual pull loops do not extend; prefer the library subscriber.
3. Cap the damage: set flow control (max outstanding messages) so a slow handler does not accumulate a mountain of redeliveries.
4. Poison messages: if one message always exceeds the deadline, it redelivers forever. Set a dead-letter topic with max_delivery_attempts so failures route out after N tries.
Do not:
- Set ack deadline to 600 "just in case" for everything. Long deadlines delay redelivery of genuinely failed messages. Match it to the handler.
- Treat DEADLINE_EXCEEDED on the PUBLISH side the same way. Publish deadlines are about network/throughput; subscriber deadlines are about handler time. Different knobs.
Verify: watch the subscription's unacked message age and redelivery count in Monitoring after the change; duplicates should drop to near zero.