What Really Happens When 10,000 Requests Hit a Node.js Server

By Vivek Kumar 5 min read

What Really Happens When 10,000 Requests Hit a Node.js Server

Most engineers think Node.js “handles 10,000 requests concurrently.”

It doesn’t.

What actually happens is far more interesting — and far more dangerous if you misunderstand it.

When a traffic spike hits your service, Node.js is only one piece of the story.

The real system includes:

  • the OS kernel
  • socket queues
  • epoll/kqueue
  • the event loop
  • downstream dependencies

This article breaks down the full lifecycle of those 10,000 requests — and where things actually break.

The Problem

You deploy a Node.js service.

It works fine under normal load.

Then traffic spikes:

  • 10,000 incoming requests in seconds
  • latency jumps
  • some requests time out
  • some never reach your app

CPU usage?

Still low.

This is where most engineers get confused.

Why It Happens

Node.js is not a thread-per-request model.

Instead:

  • One thread runs the event loop
  • The OS handles incoming connections
  • Node reacts only when sockets are ready

This is called a readiness-based I/O model.

Key implication:

Node.js does not process all requests instantly — it processes them when the event loop gets to them.

Internal Architecture

Let’s follow a request from the network to your code.

Step 1: Incoming Connections Hit the Kernel

Before Node.js sees anything, the OS is already buffering connections.

flowchart LR
    Client --> NIC[Network Interface]
    NIC --> Kernel
    Kernel --> SocketQueue[Socket Accept Queue]
    SocketQueue --> Node

The kernel maintains:

  • SYN queue (half-open connections)
  • accept queue (fully established connections)

If these queues fill up:

  • new connections are dropped
  • clients see timeouts or connection resets

Node.js is not even involved yet.

Step 2: Node.js Uses epoll/kqueue

Node doesn’t poll sockets manually.

It asks the OS:

“Tell me when something is ready.”

This is done via:

  • epoll (Linux)
  • kqueue (macOS/BSD)
flowchart TD
    Kernel -->|ready events| epoll
    epoll --> EventLoop
    EventLoop --> JSHandlers

Important detail:

  • epoll returns ready file descriptors
  • not “requests to process”
  • not “tasks to execute”

Node still has to:

  • accept the connection
  • read data
  • parse HTTP
  • run your handler

Step 3: Event Loop Processes Requests Sequentially

Here’s the critical bottleneck.

sequenceDiagram
    participant C1 as Client 1
    participant C2 as Client 2
    participant EL as Event Loop

    C1->>EL: Request
    C2->>EL: Request

    EL->>EL: Handle C1
    EL->>EL: Handle C2

Even with 10,000 connections:

  • only one callback executes at a time
  • no parallel JS execution

Concurrency ≠ parallelism.

Example Scenario

Let’s simulate reality.

Case 1: Fast Handler (Ideal Case)

app.get("/", (req, res) => {
  res.send("ok");
});
  • event loop is free quickly
  • can cycle through thousands of requests
  • high throughput

Case 2: Slow Downstream Call

app.get("/", async (req, res) => {
  const data = await db.query("SELECT * FROM users");
  res.send(data);
});

Now:

  • each request waits on DB
  • sockets remain open
  • responses pile up
flowchart LR
    Requests --> Node
    Node --> DB
    DB --> SlowResponse
    SlowResponse --> Backlog

This creates:

  • connection buildup
  • memory pressure
  • increased latency

Case 3: Blocking Code (Worst Case)

app.get("/", (req, res) => {
  const data = JSON.parse(hugeString); // blocks event loop
  res.send(data);
});

Now:

  • event loop is blocked
  • all 10,000 requests stall

Where Things Break

1. Kernel Queue Overflow

If traffic > accept capacity:

  • SYN queue fills
  • accept queue fills
  • connections dropped

No logs in Node.

Just lost traffic.

2. Event Loop Saturation

If handlers are slow:

  • event loop lag increases
  • requests wait longer
  • tail latency explodes

3. Downstream Backpressure

If DB or APIs are slow:

  • open connections accumulate
  • memory usage grows
  • retries amplify load

4. File Descriptor Limits

Each connection = 1 file descriptor.

Limits:

  • ulimit -n
  • container limits

Hit the limit → server stops accepting connections.

Diagram: End-to-End Flow

flowchart LR
    Clients --> KernelQueue
    KernelQueue --> epoll
    epoll --> EventLoop
    EventLoop --> AppLogic
    AppLogic --> DB
    DB --> AppLogic
    AppLogic --> EventLoop
    EventLoop --> Clients

Key Takeaways

  • Node.js does not process all requests simultaneously
  • The OS kernel is your first bottleneck
  • epoll gives readiness, not execution
  • Event loop is the true throughput limiter
  • Slow downstreams create hidden queues
  • Blocking code freezes everything

Practical Debugging Tips

When your system struggles under load:

Measure the right things

  • event loop lag (p95, p99)
  • open connections
  • socket queue drops (netstat -s)
  • file descriptor usage

Tune the system

  • increase backlog (server.listen(backlog))
  • raise ulimit -n
  • tune kernel params (somaxconn)

Protect the system

  • implement timeouts
  • use circuit breakers
  • apply backpressure

Final Thoughts

Node.js scales incredibly well — but only if you understand where the real limits are.

The biggest mistake engineers make is thinking:

“Node handles concurrency automatically.”

It doesn’t.

Concurrency is a system-level property:

  • kernel queues
  • event loop scheduling
  • downstream capacity

If you ignore any one of these, your system will fail — silently and unpredictably.

The next time someone says:

“Node can handle 10,000 requests easily”

You’ll know the real answer:

“It depends — on everything around Node.”

Next in the Series

Continue reading: From Socket to Callback — The Hidden Journey of Every Node.js Request