A service that just dies is a service that loses data. An abrupt kill doesn’t wait for in-flight requests to finish, doesn’t notify dependents, and doesn’t drain the connection pool. In a distributed system, a sloppy shutdown cascades — timeouts spike, requests fail, queues back up, and downstream services start retrying in a panic.
Graceful shutdown is the difference between a coordinated handoff and a car crash. It’s not optional if you’re running anything that matters.
This post walks through how to actually implement it — the signals you listen for, the drain patterns that work, the trade-offs between fast shutdowns and safe ones, and the gotchas that bite teams in production.
What Graceful Shutdown Actually Is
Graceful shutdown is a multi-phase process. A signal arrives (SIGTERM from Kubernetes, a deploy event, a manual drain). Your service stops accepting new work, finishes what it’s already doing, and closes connections cleanly. Dependents have time to route around you. Databases and queues see clean disconnects, not aborted transactions.
The phases look like this:
- Signal Reception — The system receives a shutdown signal (SIGTERM, SIGINT, or a health endpoint going down).
- Stop Accepting Work — HTTP servers stop listening for new connections. Workers stop pulling from queues. Load balancers remove the instance from rotation.
- Drain Existing Work — In-flight requests finish. Active jobs complete. Connections flush.
- Graceful Exit — The process exits with code 0, allowing orchestrators (Kubernetes, systemd) to know the shutdown was clean.
Sounds simple. The details kill you.
Most teams skip the drain phase or set timeouts that are way too short. A 5-second shutdown grace period in Kubernetes is a joke if your average request takes 8 seconds. Your service gets SIGKILL’d mid-request, and you lose state.
Signal Handling and Listener Shutdown
The first step is actually catching the shutdown signal. In Node.js, that’s straightforward:
import http from 'http';
import process from 'process';
const server = http.createServer((req, res) => {
// request handler
});
let isShuttingDown = false;
process.on('SIGTERM', () => {
console.log('SIGTERM received, starting graceful shutdown');
isShuttingDown = true;
server.close(() => {
console.log('HTTP server closed');
process.exit(0);
});
// Force exit after 30 seconds
setTimeout(() => {
console.error('Forced shutdown after timeout');
process.exit(1);
}, 30000);
});
server.listen(3000);
The key detail: call server.close() to stop accepting new connections, but keep the server alive until existing requests finish. The server won’t exit until all active connections are gone. That’s the drain window.
In Go, the pattern is similar but you have to manage the context explicitly:
package main
import (
"context"
"net/http"
"os"
"os/signal"
"syscall"
"time"
)
func main() {
server := &http.Server{Addr: ":8080"}
go func() {
if err := server.ListenAndServe(); err != nil && err != http.ErrServerClosed {
panic(err)
}
}()
sigChan := make(chan os.Signal, 1)
signal.Notify(sigChan, syscall.SIGTERM, syscall.SIGINT)
<-sigChan
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
if err := server.Shutdown(ctx); err != nil {
panic(err)
}
}
Here, server.Shutdown(ctx) is the graceful close call. It stops accepting new connections and waits for existing ones to finish, but only up to the timeout. If the context deadline passes, it returns an error and the process exits anyway.
The timeout is critical. Set it based on your p99 request latency, not your average. If your slowest legitimate request takes 15 seconds, your shutdown grace period should be at least 20 seconds. In Kubernetes, set terminationGracePeriodSeconds to match or exceed this.
Draining Workers and Background Jobs
HTTP servers are only part of the picture. Most real systems have background workers — things pulling from queues, processing events, running periodic tasks. Those need to drain too.
If you're using a queue like Bull (Redis-backed) in Node.js, you need to gracefully close the queue processor:
import Bull from 'bull';
const emailQueue = new Bull('emails', {
redis: { host: 'localhost', port: 6379 }
});
emailQueue.process(async (job) => {
// Process the job
await sendEmail(job.data);
});
process.on('SIGTERM', async () => {
console.log('Closing queue processor');
// Stop accepting new jobs
await emailQueue.close();
// Wait for active jobs to finish (respects job timeout)
// This is built into Bull
});
The problem: if a job is in the middle of processing when SIGTERM arrives, you have a race condition. The job might complete after the process is killed. Or it might get halfway through and get aborted, leaving the email half-sent and the job marked as failed.
The solution is idempotency. Make your jobs idempotent so that if they run twice, the second run is a no-op. That way, if a job is interrupted, it can be retried safely. Also, set a job timeout that's shorter than your shutdown grace period. If a job takes longer than 20 seconds and your grace period is 30 seconds, the job will timeout during shutdown and get retried on the next instance.
For Kafka consumers, the pattern is similar but the mechanics are different. You're polling for messages, processing them, and committing offsets. On shutdown:
const consumer = kafka.consumer({ groupId: 'my-group' });
await consumer.subscribe({ topic: 'my-topic' });
await consumer.run({
eachMessage: async ({ topic, partition, message }) => {
// Process message
},
});
process.on('SIGTERM', async () => {
console.log('Disconnecting Kafka consumer');
await consumer.disconnect();
process.exit(0);
});
When you call consumer.disconnect(), the Kafka client stops polling, commits any pending offsets, and leaves the consumer group. Other instances in the group will rebalance and pick up the partition. Clean handoff.
The gotcha: if you're in the middle of processing a message when disconnect() is called, the message won't be committed. It'll be redelivered to another consumer. This is why idempotency matters here too.
Load Balancer and Orchestrator Integration
Your code handling SIGTERM is only half the story. The orchestrator (Kubernetes, a load balancer, a reverse proxy) also needs to know the service is shutting down.
In Kubernetes, the shutdown sequence is:
- Kubelet sends SIGTERM to the container.
- At the same time, the endpoint is removed from the Service, and the pod moves to the Terminating phase.
- The load balancer (kube-proxy or a service mesh like Istio) stops routing new traffic to the pod.
- Your code has terminationGracePeriodSeconds to finish existing requests (default 30 seconds).
- If the container doesn't exit after the grace period, Kubelet sends SIGKILL.
The problem: there's a race between step 2 (endpoint removal) and step 1 (SIGTERM). A client might send a request that's already been routed to your pod's IP address before the endpoint is removed. Your pod receives the request after it's already stopped accepting new connections. The request fails.
This is called the connection draining problem. The fix is to introduce a small delay between receiving SIGTERM and actually closing the server. In your shutdown handler, sleep for 5 seconds before calling server.close(). This gives the load balancer time to propagate the endpoint removal.
process.on('SIGTERM', async () => {
console.log('SIGTERM received');
// Give load balancer time to drain requests
await new Promise(resolve => setTimeout(resolve, 5000));
isShuttingDown = true;
server.close(() => {
process.exit(0);
});
// Hard timeout
setTimeout(() => {
console.error('Forced shutdown');
process.exit(1);
}, 30000);
});
For a reverse proxy (nginx, HAProxy, Envoy), you can use health checks to signal shutdown. When you receive SIGTERM, immediately start returning 503 Service Unavailable from your health endpoint. The proxy detects this and stops routing traffic. Then proceed with graceful shutdown.
let isHealthy = true;
app.get('/health', (req, res) => {
if (isHealthy) {
res.status(200).json({ status: 'ok' });
} else {
res.status(503).json({ status: 'shutting down' });
}
});
process.on('SIGTERM', () => {
isHealthy = false; // Health checks now fail
// Proxy detects this and drains connections
setTimeout(() => {
server.close();
}, 5000);
});
Timeouts and Forced Termination
You cannot wait forever. If a request is stuck (database query hanging, external API timeout), the entire shutdown stalls. You need a hard timeout that forces termination.
The timeout hierarchy should be:
- Individual request timeout — Each HTTP request should have a timeout (e.g., 30 seconds). This is set on the server or router.
- Shutdown grace period — The time between SIGTERM and forced exit. Should be longer than individual request timeouts (e.g., 60 seconds in Kubernetes).
- Orchestrator force-kill timeout — Kubernetes will SIGKILL after terminationGracePeriodSeconds expires.
In Node.js, you can set request timeouts on the server:
const server = http.createServer((req, res) => {
// 30-second timeout per request
req.setTimeout(30000, () => {
res.statusCode = 503;
res.end('Request timeout');
});
// handle request
});
server.requestTimeout = 30000;
Then in your shutdown handler, use a longer timeout:
process.on('SIGTERM', () => {
server.close();
// 60-second hard timeout for graceful shutdown
const shutdownTimeout = setTimeout(() => {
console.error('Forced shutdown after grace period');
process.exit(1);
}, 60000);
server.on('close', () => {
clearTimeout(shutdownTimeout);
process.exit(0);
});
});
If a request hits its 30-second timeout, it fails but doesn't block the shutdown. The shutdown process continues and exits cleanly after 60 seconds at most.
In practice, set your Kubernetes terminationGracePeriodSeconds to 90-120 seconds. Set your shutdown grace timeout to 60 seconds. Set individual request timeouts to 30 seconds. This gives you layered safety.
Common Failures and Anti-Patterns
Anti-pattern 1: Ignoring SIGTERM. Some services don't catch SIGTERM at all. Kubernetes waits 30 seconds (the default grace period), then sends SIGKILL. The process dies mid-request. Fix: always catch SIGTERM and close your listeners.
Anti-pattern 2: Grace period too short. The default terminationGracePeriodSeconds in Kubernetes is 30 seconds. If your p99 request latency is 20 seconds, this is not enough. You'll lose requests. Measure your actual latency and set the grace period accordingly. Add a buffer for variance.
Anti-pattern 3: Not draining background workers. You close the HTTP server but keep pulling from the queue. New jobs arrive, you start processing, then get killed mid-job. The job is marked failed and you've lost work. Stop accepting new work before the shutdown grace period starts.
Anti-pattern 4: Blocking shutdown on external dependencies. If your shutdown handler tries to notify an external service (e.g., deregister from a service registry), and that service is slow or unreachable, your entire shutdown blocks. Set a timeout on external calls during shutdown. If they take too long, timeout and proceed.
Anti-pattern 5: Not logging shutdown events. When SIGTERM arrives, log it. When the server closes, log it. When the timeout fires and you're forced to exit, log it. This is your signal that something went wrong in production. Without logs, you won't know if shutdowns are graceful or forced.
A real example: a service processing large file uploads. The HTTP request timeout is 60 seconds. The shutdown grace period is 30 seconds. A client uploads a 10GB file slowly over 45 seconds. SIGTERM arrives. The request is still in flight. The grace period expires. The process is killed. The upload is lost, the file is partially written, the database transaction is rolled back. The client has to retry.
The fix: make sure your grace period is longer than your request timeout. Or better, stream the upload to object storage (S3, GCS) instead of keeping it in-memory, so the request can finish quickly even if the file is large.
Testing Graceful Shutdown
Most teams don't test this. They find out during a deploy that shutdown is broken.
Set up a simple test. Start your service. Send a slow request (one that takes 5 seconds to complete). While it's in flight, send SIGTERM to the process. Verify that the request completes and returns a 200 status code. The process should exit with code 0.
Then test the timeout. Send a request that never completes (hang the request handler). Send SIGTERM. Wait for the grace period to expire. Verify that the process exits with code 1 (forced exit) after the timeout.
In a staging environment, deploy your service and then trigger a rolling restart. Watch the logs. You should see SIGTERM received, requests draining, and clean exits. No 503 errors (unless requests legitimately timeout). No stuck processes.
If you're using Kubernetes, test pod eviction. Run kubectl drain on a node. The pods should shut down gracefully. Verify that no requests are lost and no jobs are interrupted mid-process.
Load testing tools like k6 or wrk can help here. Send traffic to your service, then trigger a shutdown. Measure how many requests fail. In a well-implemented system, the failure rate during shutdown should be near zero.
Graceful shutdown is one of those things that's invisible when it works and catastrophic when it doesn't. A service that shuts down cleanly absorbs deploys without data loss or client errors. One that crashes hard creates cascading failures across your system. The difference is a few hundred lines of code and some careful thinking about timeouts.
If your team is regularly losing data during deploys or seeing timeouts spike when services restart, graceful shutdown is worth a hard look. It's the kind of operational detail that separates systems that hold up under real load from ones that fall apart. We help teams architect this kind of resilience into their systems — if you're building something that can't tolerate restarts, the application to apply for an engagement takes ten minutes.
