Health Checks & Graceful Shutdown

Health Check Endpoints

Health check endpoints are HTTP routes that expose the current state of your application to external systems like load balancers, orchestrators (Kubernetes, ECS), and monitoring tools. A well-designed health endpoint returns structured JSON indicating whether the service is alive, ready to serve traffic, and the status of its dependencies.

Anatomy of a Health Endpoint

A production health endpoint should check multiple dimensions:

• Liveness: Is the process running and not deadlocked?
• Readiness: Can the service accept new requests right now?
• Dependencies: Are databases, caches, and external APIs reachable?

Implementation Example

import express from 'express';
import { Pool } from 'pg';
import Redis from 'ioredis';

const app = express();
const dbPool = new Pool({ connectionString: process.env.DATABASE_URL });
const redis = new Redis(process.env.REDIS_URL);

interface HealthStatus {
  status: 'healthy' | 'degraded' | 'unhealthy';
  uptime: number;
  timestamp: string;
  checks: {
    database: CheckResult;
    redis: CheckResult;
    memory: CheckResult;
  };
}

interface CheckResult {
  status: 'pass' | 'fail';
  latencyMs: number;
  error?: string;
}

async function checkDatabase(): Promise<CheckResult> {
  const start = Date.now();
  try {
    await dbPool.query('SELECT 1');
    return { status: 'pass', latencyMs: Date.now() - start };
  } catch (err) {
    return {
      status: 'fail',
      latencyMs: Date.now() - start,
      error: err instanceof Error ? err.message : 'Unknown error',
    };
  }
}

async function checkRedis(): Promise<CheckResult> {
  const start = Date.now();
  try {
    await redis.ping();
    return { status: 'pass', latencyMs: Date.now() - start };
  } catch (err) {
    return {
      status: 'fail',
      latencyMs: Date.now() - start,
      error: err instanceof Error ? err.message : 'Unknown error',
    };
  }
}

function checkMemory(): CheckResult {
  const start = Date.now();
  const mem = process.memoryUsage();
  const heapUsedMB = mem.heapUsed / 1024 / 1024;
  const heapTotalMB = mem.heapTotal / 1024 / 1024;
  const usagePercent = (heapUsedMB / heapTotalMB) * 100;
  return {
    status: usagePercent > 90 ? 'fail' : 'pass',
    latencyMs: Date.now() - start,
  };
}

app.get('/health', async (_req, res) => {
  const [database, redis, memory] = await Promise.all([
    checkDatabase(),
    checkRedis(),
    checkMemory(),
  ]);

  const checks = { database, redis, memory };
  const allPass = Object.values(checks).every(c => c.status === 'pass');
  const anyFail = Object.values(checks).some(c => c.status === 'fail');

  const status: HealthStatus['status'] = allPass ? 'healthy' : anyFail ? 'unhealthy' : 'degraded';

  const statusCode = status === 'healthy' ? 200 : 503;
  res.status(statusCode).json({
    status,
    uptime: process.uptime(),
    timestamp: new Date().toISOString(),
    checks,
  });
});

Best Practices

• Always return a non-200 status code when the service is unhealthy so load balancers stop routing traffic
• Keep health checks lightweight—avoid expensive queries or external API calls
• Use timeouts on dependency checks to prevent the health endpoint itself from hanging
• Separate /health/live (liveness) from /health/ready (readiness) for Kubernetes environments

Readiness Probes

Readiness probes determine whether a service is prepared to handle incoming requests. During startup, a service may need to establish database connections, warm caches, load configuration, or complete migrations. The readiness probe should return a non-ready status until all startup tasks are complete, preventing traffic from being routed prematurely.

Kubernetes Integration

Kubernetes uses readiness probes to decide whether to include a Pod in Service endpoints. If the readiness probe fails, the Pod is removed from the load balancer pool but the container continues running.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api-server
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api-server
  template:
    metadata:
      labels:
        app: api-server
    spec:
      containers:
        - name: api
          image: myapp:latest
          ports:
            - containerPort: 3000
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 3000
            initialDelaySeconds: 5
            periodSeconds: 10
            failureThreshold: 3
            successThreshold: 1
          livenessProbe:
            httpGet:
              path: /health/live
              port: 3000
            initialDelaySeconds: 15
            periodSeconds: 20
            failureThreshold: 3

Implementation with Startup State Tracking

enum ServiceState {
  Starting = 'starting',
  Ready = 'ready',
  Draining = 'draining',
  Stopped = 'stopped',
}

class Application {
  private state: ServiceState = ServiceState.Starting;
  private dependenciesReady = false;

  async initialize(): Promise<void> {
    try {
      await this.connectDatabase();
      await this.warmCache();
      await this.loadConfiguration();
      this.dependenciesReady = true;
      this.state = ServiceState.Ready;
      console.log('Application is ready to serve traffic');
    } catch (err) {
      console.error('Failed to initialize:', err);
      this.state = ServiceState.Stopped;
      process.exit(1);
    }
  }

  async checkReadiness(): Promise<{ ready: boolean; reason?: string }> {
    if (this.state === ServiceState.Starting) {
      return { ready: false, reason: 'Application is still starting up' };
    }
    if (this.state === ServiceState.Draining) {
      return { ready: false, reason: 'Application is draining connections' };
    }
    if (this.state === ServiceState.Stopped) {
      return { ready: false, reason: 'Application has stopped' };
    }
    if (!this.dependenciesReady) {
      return { ready: false, reason: 'Dependencies are not ready' };
    }
    return { ready: true };
  }
}

const app = new Application();

app.get('/health/ready', async (_req, res) => {
  const { ready, reason } = await app.checkReadiness();
  res.status(ready ? 200 : 503).json({ ready, reason });
});

Key Considerations

• Set initialDelaySeconds long enough for your application to complete startup initialization
• Tune periodSeconds based on how quickly your service can recover from a temporary dependency failure
• Use successThreshold: 1 so the Pod is re-added to the pool immediately when it becomes healthy again
• Readiness probes should not check external dependencies aggressively—they only need to confirm internal state

Graceful Shutdown

Graceful shutdown ensures that when a process receives a termination signal, it stops accepting new work, finishes processing in-flight requests, cleans up resources, and then exits cleanly. Without graceful shutdown, active connections are dropped, database transactions are left in inconsistent states, and users experience errors during deployments.

Signal Handling

Node.js receives SIGTERM when a container is being stopped. You must listen for this signal and coordinate shutdown:

import express from 'express';
import http from 'http';
import { Pool } from 'pg';

const app = express();
const dbPool = new Pool({ connectionString: process.env.DATABASE_URL });
const server = http.createServer(app);

let isShuttingDown = false;
const DRAIN_TIMEOUT_MS = 30_000;

app.get('/api/orders', async (_req, res) => {
  const result = await dbPool.query('SELECT * FROM orders WHERE status = $1', ['pending']);
  res.json(result.rows);
});

app.get('/health/live', (_req, res) => {
  if (isShuttingDown) {
    return res.status(503).json({ alive: false });
  }
  res.json({ alive: true });
});

app.get('/health/ready', (_req, res) => {
  if (isShuttingDown) {
    return res.status(503).json({ ready: false });
  }
  res.json({ ready: true });
});

function gracefulShutdown(signal: string) {
  console.log(`Received ${signal}. Starting graceful shutdown...`);
  isShuttingDown = true;

  const shutdownTimer = setTimeout(() => {
    console.error('Drain timeout exceeded. Forcing shutdown.');
    process.exit(1);
  }, DRAIN_TIMEOUT_MS);

  server.close(async () => {
    console.log('HTTP server closed. No new connections accepted.');

    try {
      await dbPool.end();
      console.log('Database pool closed.');
    } catch (err) {
      console.error('Error closing database pool:', err);
    }

    clearTimeout(shutdownTimer);
    console.log('Graceful shutdown complete.');
    process.exit(0);
  });
}

process.on('SIGTERM', () => gracefulShutdown('SIGTERM'));
process.on('SIGINT', () => gracefulShutdown('SIGINT'));

server.listen(3000, () => {
  console.log('Server listening on port 3000');
});

How It Works Step by Step

1. SIGTERM received — Set isShuttingDown = true so health probes return 503
2. Load balancer stops sending traffic — Kubernetes removes the Pod from endpoints because readiness probe fails
3. server.close() — Stop accepting new HTTP connections while existing ones finish
4. Drain in-flight requests — Let the event loop process remaining work
5. Clean up resources — Close database pools, Redis connections, message queue consumers
6. Exit — Call process.exit(0) to signal clean termination

Common Pitfalls

• Forgetting to stop the health probe — If the health endpoint still returns 200 after SIGTERM, the load balancer keeps sending traffic
• No timeout — If cleanup hangs forever, the orchestrator will force-kill the process (SIGKILL), defeating the purpose
• Long-lived connections — WebSocket or SSE connections need explicit close logic with a timeout
• Background tasks — Jobs in queues should acknowledge completion or requeue before shutdown

Liveness Probes

Liveness probes detect whether an application is alive and functioning correctly. Unlike readiness probes which check if the service can handle new traffic, liveness probes check if the process itself is healthy. A failing liveness probe tells the orchestrator to restart the container because the application is in an unrecoverable state—such as a deadlock, infinite loop, or memory leak that prevents it from serving requests.

When to Use Liveness vs Readiness

Implementation Example

import express from 'express';

const app = express();

let lastHeartbeat = Date.now();
let requestCount = 0;

function recordActivity(): void {
  lastHeartbeat = Date.now();
  requestCount++;
}

app.use((_req, _res, next) => {
  recordActivity();
  next();
});

app.get('/health/live', (_req, res) => {
  const now = Date.now();
  const secondsSinceActivity = (now - lastHeartbeat) / 1000;
  const HEARTBEAT_THRESHOLD = 60;

  if (secondsSinceActivity > HEARTBEAT_THRESHOLD) {
    return res.status(503).json({
      alive: false,
      reason: `No activity for ${secondsSinceActivity.toFixed(1)}s`,
    });
  }

  res.json({
    alive: true,
    uptime: process.uptime(),
    requestCount,
    heapUsedMB: (process.memoryUsage().heapUsed / 1024 / 1024).toFixed(1),
  });
});

setInterval(() => {
  const mem = process.memoryUsage();
  const heapUsedMB = mem.heapUsed / 1024 / 1024;
  if (heapUsedMB > 500) {
    console.warn(`High memory usage detected: ${heapUsedMB.toFixed(0)}MB`);
  }
}, 30_000);

app.get('/api/work', async (_req, res) => {
  const result = await performComputation();
  res.json(result);
});

Kubernetes Liveness Probe Configuration

livenessProbe:
  httpGet:
    path: /health/live
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 10
  timeoutSeconds: 5
  failureThreshold: 3

The initialDelaySeconds should be longer than your application's worst-case startup time to avoid premature restarts. The failureThreshold determines how many consecutive failures trigger a restart—setting it to 3 prevents restarts on transient issues.

Liveness Probe Anti-Patterns

• Checking external dependencies — If your database is temporarily down, a liveness probe will restart every Pod, causing a cascading failure. Use readiness probes for dependency checks instead
• Expensive computations — Liveness checks run frequently; keep them under 5ms
• Returning 200 when deadlocked — A deadlocked process may still respond to HTTP if the event loop thread is partially available. Use thread-level health checks or watchdog timers for more accurate detection