Anatomy of Graceful Shutdown: Part 6 - nginx's Master-Worker Coordination

nginx is one of the most widely deployed web servers and reverse proxies in the world, handling millions of requests per second across countless production environments. Its reputation for reliability comes not just from its performance characteristics, but from how gracefully it handles operations like configuration reloads and shutdowns.

When you run nginx -s quit or send a SIGQUIT signal to nginx, you’re triggering a sophisticated coordination dance between the master process and its worker processes. Unlike simpler web servers that might just close all connections immediately, nginx implements a multi-phase graceful shutdown that ensures existing requests complete while preventing new ones from being accepted.

This coordination is particularly interesting because nginx operates as a multi-process architecture where a single master process manages multiple worker processes. Each worker process handles thousands of concurrent connections using an event-driven model, making the shutdown coordination significantly more complex than single-threaded servers.

nginx Architecture: The Master-Worker Model

nginx’s architecture centers around a master-worker process model designed for both performance and robustness. The master process acts as a coordinator, managing worker processes that handle the actual request processing.

Core Components

The nginx codebase is organized into several key layers:

• Core (src/core/): Configuration parsing, memory management, and fundamental data structures
• Event System (src/event/): Event-driven I/O with support for epoll, kqueue, and other platform-specific mechanisms
• HTTP Module (src/http/): HTTP request processing pipeline with filtering capabilities
• Process Management (src/os/unix/): Master-worker coordination, signal handling, and process lifecycle

Process Relationships

The master process spawns a configurable number of worker processes (typically matching CPU cores) and coordinates their lifecycle. Each worker process operates independently, handling its own set of client connections through an event loop.

Master Process (PID 1234)
├── Worker Process 1 (PID 1235) - handles connections 0-999
├── Worker Process 2 (PID 1236) - handles connections 1000-1999
├── Worker Process 3 (PID 1237) - handles connections 2000-2999
└── Worker Process 4 (PID 1238) - handles connections 3000-3999

When a graceful shutdown begins, the master process must coordinate with all worker processes to ensure they complete their active work before the entire system shuts down.

Signal Handling and Shutdown Entry Points

nginx’s graceful shutdown begins with signal handling in the master process. The signal handling system is implemented in src/os/unix/ngx_process.c and establishes handlers for various UNIX signals.

Signal Registration

During startup, nginx registers signal handlers for shutdown coordination:

Signal initialization (src/os/unix/ngx_process.c:39-83):

ngx_signal_t signals[] = {
    { ngx_signal_value(NGX_SHUTDOWN_SIGNAL),    // SIGQUIT
      "SIG" ngx_value(NGX_SHUTDOWN_SIGNAL),
      "quit",
      ngx_signal_handler },
    
    { ngx_signal_value(NGX_TERMINATE_SIGNAL),   // SIGTERM  
      "SIG" ngx_value(NGX_TERMINATE_SIGNAL),
      "stop", 
      ngx_signal_handler },
    // Additional signals...
};

The key distinction between signals:

• SIGQUIT (NGX_SHUTDOWN_SIGNAL): Graceful shutdown - finish existing requests
• SIGTERM (NGX_TERMINATE_SIGNAL): Immediate termination - close all connections

Master Signal Handler

The main signal handler processes shutdown signals and sets global flags that the master process event loop monitors:

Signal handler logic (src/os/unix/ngx_process.c:319-467):

static void ngx_signal_handler(int signo) {
    ngx_int_t      ignore;
    ngx_err_t      err;
    ngx_signal_t  *sig;

    ignore = 0;
    err = ngx_errno;

    for (sig = signals; sig->signo != 0; sig++) {
        if (sig->signo == signo) {
            break;
        }
    }

    switch (ngx_process) {
    case NGX_PROCESS_MASTER:
        switch (signo) {
        case ngx_signal_value(NGX_SHUTDOWN_SIGNAL):
            ngx_quit = 1;  // Set graceful shutdown flag
            break;
        case ngx_signal_value(NGX_TERMINATE_SIGNAL):
            ngx_terminate = 1;  // Set immediate termination flag
            break;
        }
        break;
    }
}

The signal handler sets atomic flags (ngx_quit, ngx_terminate) that the master process checks in its main event loop.

Master Process Shutdown Coordination

The master process runs a continuous event loop that monitors for signals, manages worker processes, and coordinates shutdown operations. The main coordination logic is in ngx_master_process_cycle() in src/os/unix/ngx_process_cycle.c.

Master Event Loop

Master process coordination (src/os/unix/ngx_process_cycle.c:74-275):

void ngx_master_process_cycle(ngx_cycle_t *cycle) {
    char              *title;
    u_char            *p;
    size_t             size;
    ngx_int_t          i;
    ngx_uint_t         sigio;
    sigset_t           set;
    struct itimerval   itv;
    ngx_uint_t         live;
    ngx_msec_t         delay;
    ngx_core_conf_t   *ccf;

    // Signal masking setup...
    sigemptyset(&set);
    sigaddset(&set, SIGCHLD);
    sigaddset(&set, SIGALRM);
    sigaddset(&set, SIGIO);
    sigaddset(&set, SIGINT);
    sigaddset(&set, ngx_signal_value(NGX_RECONFIGURE_SIGNAL));
    sigaddset(&set, ngx_signal_value(NGX_REOPEN_SIGNAL));
    sigaddset(&set, ngx_signal_value(NGX_NOACCEPT_SIGNAL));
    sigaddset(&set, ngx_signal_value(NGX_TERMINATE_SIGNAL));
    sigaddset(&set, ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
    sigaddset(&set, ngx_signal_value(NGX_CHANGEBIN_SIGNAL));

    // Main event loop
    for ( ;; ) {
        if (ngx_quit) {
            // Begin graceful shutdown sequence
            ngx_signal_worker_processes(cycle, ngx_signal_value(NGX_SHUTDOWN_SIGNAL));
            ngx_close_listening_sockets(cycle);
            continue;
        }

        if (ngx_terminate) {
            // Immediate termination
            ngx_signal_worker_processes(cycle, ngx_signal_value(NGX_TERMINATE_SIGNAL));
            ngx_master_process_exit(cycle);
        }

        // Continue monitoring worker processes...
        live = ngx_reap_children(cycle);
        if (live == 0 && (ngx_terminate || ngx_quit)) {
            ngx_master_process_exit(cycle);
        }
    }
}

Worker Process Communication

The master process communicates with worker processes through two mechanisms:

1. Inter-process channels: nginx creates socketpair channels for fast communication
2. UNIX signals: Fallback mechanism using kill() system calls

Worker process signaling (src/os/unix/ngx_process_cycle.c:432-530):

static void ngx_signal_worker_processes(ngx_cycle_t *cycle, int signo) {
    ngx_int_t      i;
    ngx_err_t      err;
    ngx_channel_t  ch;

    ch.command = NGX_CMD_QUIT;
    ch.fd = -1;

    for (i = 0; i < ngx_last_process; i++) {
        if (ngx_processes[i].detached || ngx_processes[i].pid == -1) {
            continue;
        }

        if (ngx_processes[i].channel[0] == -1) {
            // Use signal if no channel available
            if (kill(ngx_processes[i].pid, signo) == -1) {
                ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                              "kill(%P, %d) failed", 
                              ngx_processes[i].pid, signo);
            }
            continue;
        }

        // Use channel for communication
        if (ngx_write_channel(ngx_processes[i].channel[0],
                              &ch, sizeof(ngx_channel_t), cycle->log)
            == NGX_OK) {
            continue;
        }

        // Channel write failed, fall back to signal
        if (kill(ngx_processes[i].pid, signo) == -1) {
            ngx_log_error(NGX_LOG_ALERT, cycle->log, ngx_errno,
                          "kill(%P, %d) failed", 
                          ngx_processes[i].pid, signo);
        }
    }
}

The dual communication approach ensures reliability - if the faster channel communication fails, nginx falls back to traditional signals.

Worker Process Graceful Shutdown

Each worker process runs its own event loop that handles client connections and monitors for shutdown signals from the master process. The worker shutdown logic is implemented in ngx_worker_process_cycle().

Worker Event Loop and Shutdown Detection

Worker process main loop (src/os/unix/ngx_process_cycle.c:699-749):

static void ngx_worker_process_cycle(ngx_cycle_t *cycle, void *data) {
    ngx_int_t worker = (intptr_t) data;

    ngx_worker_process_init(cycle, worker);

    for ( ;; ) {
        if (ngx_exiting) {
            // Already in shutdown phase
            if (ngx_event_no_timers_left() == NGX_OK) {
                ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
                ngx_worker_process_exit(cycle);
            }
        }

        if (ngx_quit) {
            // Begin graceful shutdown
            ngx_quit = 0;
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, 
                          "gracefully shutting down");
            ngx_setproctitle("worker process is shutting down");

            if (!ngx_exiting) {
                ngx_exiting = 1;
                ngx_set_shutdown_timer(cycle);
                ngx_close_listening_sockets(cycle);
                ngx_close_idle_connections(cycle);
            }
        }

        if (ngx_terminate) {
            // Immediate termination
            ngx_log_error(NGX_LOG_NOTICE, cycle->log, 0, "exiting");
            ngx_worker_process_exit(cycle);
        }

        // Process events
        ngx_process_events_and_timers(cycle);
    }
}

Connection State Management

The worker process maintains several types of connections during shutdown:

1. Listening sockets: Closed immediately to stop accepting new connections
2. Active connections: Allowed to complete their current requests
3. Idle connections: Closed immediately to free resources

Listening socket closure (src/core/ngx_connection.c:1100-1175):

void ngx_close_listening_sockets(ngx_cycle_t *cycle) {
    ngx_uint_t         i;
    ngx_listening_t   *ls;
    ngx_connection_t  *c;

    if (ngx_event_flags & NGX_USE_IOCP_EVENT) {
        return;
    }

    ngx_accept_disabled = 1;

    ls = cycle->listening.elts;
    for (i = 0; i < cycle->listening.nelts; i++) {
        c = ls[i].connection;

        if (c) {
            if (c->read->active && !c->read->accept) {
                ngx_del_event(c->read, NGX_READ_EVENT, 0);
            }

            c->read->handler = ngx_empty_event_handler;
            c->write->handler = ngx_empty_event_handler;

            if (ngx_close_socket(c->fd) == -1) {
                ngx_log_error(NGX_LOG_EMERG, cycle->log, ngx_socket_errno,
                              ngx_close_socket_n " %V failed",
                              &ls[i].addr_text);
            }
        }
    }
}

Idle connection cleanup (src/core/ngx_connection.c:1434-1450):

void ngx_close_idle_connections(ngx_cycle_t *cycle) {
    ngx_uint_t         i;
    ngx_connection_t  *c;

    c = cycle->connections;

    for (i = 0; i < cycle->connection_n; i++) {
        if (c[i].fd != (ngx_socket_t) -1 && c[i].idle) {
            c[i].close = 1;
            c[i].read->handler(c[i].read);
        }
    }
}

Shutdown Timer and Timeout Management

nginx implements a configurable shutdown timeout to prevent worker processes from hanging indefinitely on slow-closing connections. This is controlled by the worker_shutdown_timeout directive.

Shutdown Timer Implementation

Timer setup (src/core/ngx_cycle.c:1429-1443):

void ngx_set_shutdown_timer(ngx_cycle_t *cycle) {
    ngx_core_conf_t  *ccf;

    ccf = (ngx_core_conf_t *) ngx_get_conf(cycle->conf_ctx, ngx_core_module);

    if (ccf->shutdown_timeout) {
        ngx_shutdown_event.handler = ngx_shutdown_timer_handler;
        ngx_shutdown_event.log = cycle->log;
        ngx_shutdown_event.cancelable = 1;

        ngx_add_timer(&ngx_shutdown_event, ccf->shutdown_timeout);
    }
}

Timeout handler (src/core/ngx_cycle.c:1447-1476):

static void ngx_shutdown_timer_handler(ngx_event_t *ev) {
    ngx_uint_t         i;
    ngx_cycle_t       *cycle;
    ngx_connection_t  *c;

    cycle = ev->data;

    ngx_log_error(NGX_LOG_WARN, ev->log, 0,
                  "worker shutdown timeout");

    /* force connections to close */

    c = cycle->connections;

    for (i = 0; i < cycle->connection_n; i++) {
        if (c[i].fd == (ngx_socket_t) -1 
            || c[i].read == NULL
            || c[i].read->accept
            || c[i].read->channel
            || c[i].read->resolver) {
            continue;
        }

        ngx_log_debug1(NGX_LOG_DEBUG_CORE, ev->log, 0,
                       "*%uA shutdown timeout", c[i].number);

        c[i].close = 1;
        c[i].error = 1;

        c[i].read->handler(c[i].read);
    }
}

Timer-Based Exit Condition

Worker processes use timer state to determine when it’s safe to exit:

Timer check for exit (src/event/ngx_event_timer.c:100-126):

ngx_int_t ngx_event_no_timers_left(void) {
    ngx_event_t        *ev;
    ngx_rbtree_node_t  *node, *root, *sentinel;

    sentinel = ngx_event_timer_rbtree.sentinel;

    if (ngx_event_timer_rbtree.root == sentinel) {
        return NGX_OK;
    }

    for (node = ngx_rbtree_min(ngx_event_timer_rbtree.root, sentinel);
         node;
         node = ngx_rbtree_next(&ngx_event_timer_rbtree, node)) {

        ev = ngx_rbtree_data(node, ngx_event_t, timer);

        if (!ev->cancelable) {
            return NGX_AGAIN;
        }
    }

    return NGX_OK;
}

This function checks if any non-cancelable timers remain. Only when all critical timers have finished does the worker process consider it safe to exit.

Complete Shutdown Flow

The complete graceful shutdown sequence follows these phases:

Phase 1: Signal Reception and Propagation

1. Master process receives SIGQUIT signal
2. Master sets ngx_quit = 1 flag
3. Master signals all worker processes via channels or signals
4. Master closes its own listening sockets

Phase 2: Worker Shutdown Initiation

1. Workers receive shutdown signal and set ngx_exiting = 1
2. Workers close listening sockets to stop accepting new connections
3. Workers immediately close idle connections to free resources
4. Workers start shutdown timer (if configured)

Phase 3: Active Connection Handling

1. Workers continue processing active HTTP requests
2. New connection attempts are rejected (listening sockets closed)
3. Workers monitor timer state to determine exit readiness

Phase 4: Shutdown Completion

1. Workers wait for all non-cancelable timers to complete OR timeout
2. If timeout expires, workers force-close remaining connections
3. Workers perform final cleanup and exit
4. Master reaps worker processes and exits when all workers terminate

Key Design Insights

nginx’s graceful shutdown implementation demonstrates several important design patterns for building robust server software:

Signal-Driven State Management

Rather than using complex inter-process communication for coordination, nginx relies on simple atomic flags (ngx_quit, ngx_terminate, ngx_exiting) that can be safely set from signal handlers. This approach is both thread-safe and robust against signal delivery timing issues.

Dual Communication Channels

The master process uses both socketpair channels and UNIX signals to communicate with workers. This redundancy ensures shutdown commands reach worker processes even when the faster channel mechanism fails.

Connection State Classification

nginx categorizes connections into listening, active, and idle states during shutdown. This classification allows for intelligent resource management - immediately closing idle connections while preserving active request processing.

Configurable Timeout Boundaries

The worker_shutdown_timeout directive provides operators control over the tradeoff between graceful handling and shutdown speed. In production environments with strict deployment windows, this timeout prevents hung connections from blocking system updates.

Timer-Based Exit Conditions

Rather than simply checking connection counts, nginx uses timer state to determine exit readiness. This approach accounts for internal event system state and ensures clean shutdown even with complex event interactions.

The nginx graceful shutdown implementation serves as an excellent example of how to coordinate complex multi-process systems during shutdown operations. Its combination of signal handling, process coordination, connection management, and timeout mechanisms provides both reliability and operator control over the shutdown process.

For systems architects building distributed services, nginx’s approach offers valuable lessons in managing service lifecycle operations across process boundaries while maintaining system reliability under varying operational conditions.