MongoDB Write Concern: Why an "Acknowledged" Write May Still Be Lost

Most engineers working with MongoDB have seen configurations like:

{ w: "majority", j: true }

This is commonly treated as a safe default for production writes. However, very few engineers actually understand what these options do internally — and what guarantees they really provide.

When a client writes to MongoDB, the data goes through multiple layers before it is considered durable:

1. WiredTiger cache
2. Journal
3. Oplog replication
4. Majority acknowledgment

Understanding this pipeline is critical when designing high-throughput or highly reliable systems.

Let’s break down what actually happens.

The Write Path in MongoDB

When a client sends a write request, the operation traverses a well-defined pipeline inside the primary node before any acknowledgment reaches the client.

graph TD
    A["Client Application"] -->|"write request"| B["Primary Node"]
    B --> C["WiredTiger Cache (RAM)"]
    C --> D["Journal (WAL on disk)"]
    D --> E["Oplog Entry"]
    E --> F["Replication to Secondaries"]
    F --> G["Majority Acknowledgment"]
    G -->|"ack"| A

    style A fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style B fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style C fill:#302020,stroke:#f97316,color:#e6edf3
    style D fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style E fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style F fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style G fill:#1f2937,stroke:#58a6ff,color:#e6edf3

Each step adds different guarantees — and trade-offs — between latency, durability, and consistency.

Most confusion around MongoDB durability comes from mixing up these layers. A write can be acknowledged at different points in this pipeline depending on your write concern configuration, and each point provides a fundamentally different level of safety.

WiredTiger Cache

MongoDB’s storage engine, WiredTiger, uses an in-memory cache to manage reads and writes. By default, this cache is allocated approximately 50% of available RAM (minus 1 GB).

When a write occurs:

1. The document is modified in memory within the WiredTiger cache
2. The change is recorded in the storage engine’s internal data structures
3. The write may still not be on disk yet

At this stage, the data is entirely vulnerable to process crashes or machine failures. If mongod terminates unexpectedly, anything that exists only in the WiredTiger cache is lost.

This is precisely why MongoDB relies on journaling.

Journaling

The journal acts as a write-ahead durability log (WAL). Before acknowledging certain writes, MongoDB ensures that the operation has been written to a journal file on disk.

If MongoDB crashes, the recovery process is straightforward:

journal → replay → restore last consistent state

Without journaling, writes that existed only in memory would be permanently lost.

What j: true Means

{ j: true }

This option tells MongoDB: do not acknowledge this write until it has been flushed to the on-disk journal.

However, there is an important implementation detail that many engineers overlook:

MongoDB batches journal commits, typically around every 100 milliseconds.

So the actual flow becomes:

graph TD
    A["Write arrives at primary"] --> B["Added to journal buffer (RAM)"]
    B --> C["Journal commit batch (~100ms)"]
    C --> D["Flushed to disk"]
    D --> E["Acknowledgment sent to client"]

    style A fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style B fill:#302020,stroke:#f97316,color:#e6edf3
    style C fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style D fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style E fill:#1f2937,stroke:#58a6ff,color:#e6edf3

With j: true, the write waits for this flush to complete before acknowledging. Without it (j: false), the acknowledgment may return while the data is still only in the journal buffer — volatile memory.

This batching significantly improves throughput but also introduces a small durability window for writes that do not request j: true.

The Oplog

Replication in MongoDB is driven by the oplog (operations log) — a capped collection stored on the primary node.

Each write operation produces an oplog entry similar to:

{
  "ts": { "$timestamp": { "t": 1710460800, "i": 1 } },
  "op": "i",
  "ns": "mydb.orders",
  "o": {
    "_id": "ObjectId('...')",
    "item": "widget",
    "qty": 25
  }
}

Secondary nodes continuously replicate by:

1. Tailing the primary’s oplog
2. Applying each operation locally in order

The oplog serves as the source of truth for replication. Conceptually, it plays a role similar to replication logs in other systems:

System	Replication Log
MongoDB	Oplog
MySQL	Binlog
PostgreSQL	WAL
Kafka	Partition log

If a secondary falls behind, it catches up by reading oplog entries it hasn’t yet applied. If it falls too far behind (beyond the oplog’s capped size), it requires a full resync — an expensive operation.

Write Concern Explained

Write concern determines when MongoDB considers a write successful and sends an acknowledgment back to the client.

The most common options:

Write Concern	Meaning
w: 0	Fire and forget — no acknowledgment
w: 1	Primary acknowledged the write (default)
w: "majority"	Majority of the replica set acknowledged
j: true	Journal commit required before acknowledgment

These options compose. The strongest commonly used configuration:

{ w: "majority", j: true }

This means:

1. The write is journaled on the primary (flushed to disk)
2. The write is replicated to a majority of replica set members
3. Those members confirm the write
4. Only then is the operation acknowledged to the client

Each additional guarantee adds latency, but also closes a specific class of failure.

What Majority Actually Means

Consider a replica set with 5 nodes:

┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐  ┌─────────┐
│ Primary  │  │Secondary│  │Secondary│  │Secondary│  │Secondary│
│  (P)     │  │  (S1)   │  │  (S2)   │  │  (S3)   │  │  (S4)   │
└─────────┘  └─────────┘  └─────────┘  └─────────┘  └─────────┘

Majority = ⌊n/2⌋ + 1 = 3 nodes

A w: "majority" write is acknowledged only after 3 nodes (including the primary) have confirmed the operation:

graph LR
    C["Client"] -->|"write"| P["Primary (P)"]
    P -->|"replicate"| S1["Secondary (S1)"]
    P -->|"replicate"| S2["Secondary (S2)"]
    P -.->|"replicate"| S3["Secondary (S3)"]
    P -.->|"replicate"| S4["Secondary (S4)"]

    P -->|"ack ✓"| C
    S1 -->|"ack ✓"| P
    S2 -->|"ack ✓"| P

    style C fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style P fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style S1 fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style S2 fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style S3 fill:#1f2937,stroke:#6b7280,color:#8b949e
    style S4 fill:#1f2937,stroke:#6b7280,color:#8b949e

The write is acknowledged once P + S1 + S2 (any majority combination) have confirmed. S3 and S4 will eventually receive the write, but the client doesn’t wait for them.

This provides strong guarantees against primary node failures: even if the primary goes down immediately after acknowledgment, at least two secondaries have the data. A new primary election will choose a node that has the write.

Subtle Durability Details

One detail that many engineers overlook:

w: "majority" does not, by itself, guarantee disk persistence on the secondaries.

Replication can happen from memory before a journal flush on the secondary nodes, depending on configuration. The write concern w: "majority" confirms that the secondaries have received the write — not necessarily that they have journaled it.

MongoDB introduced the writeConcernMajorityJournalDefault setting to control this behavior:

• When set to true (default in modern deployments), majority writes wait for journal commits on the majority of nodes
• When set to false, majority acknowledgment may come from in-memory replication

In modern MongoDB deployments (4.x+), majority writes are typically journaled by default. But this detail matters when reasoning about durability guarantees — especially in older deployments or custom configurations.

A helpful mental model:

Replication  =  availability guarantee  (data exists on multiple nodes)
Journaling   =  durability guarantee    (data survives process crashes)

You need both for complete protection against data loss.

Performance Trade-offs

Durability guarantees always come with latency costs. The stronger the guarantee, the more round trips and disk operations are required before the client receives an acknowledgment.

Configuration	Latency	Crash Safety	Replica Safety	Use Case
w: 0	Minimal	None	None	Telemetry, fire-and-forget logs
w: 1, j: false	Lowest	Weak	None	Non-critical writes
w: 1, j: true	Medium	Strong	None	Single-node durability
w: "majority"	Higher	Varies	Strong	Most production workloads
w: "majority", j: true	Highest	Strong	Strong	Critical data (payments, state)

For high-throughput systems, engineers sometimes choose weaker write concerns for non-critical data:

// Non-critical: analytics events, access logs
db.analytics.insertOne(event, { writeConcern: { w: 1, j: false } });

// Critical: payment records, state transitions
db.payments.insertOne(payment, { writeConcern: { w: "majority", j: true } });

The key insight is that write concern should vary by operation criticality, not be a blanket setting across the entire application.

Failure Scenarios

Let’s walk through what happens under different configurations when the primary crashes.

Case 1: w: 1, j: false — No Journaling

graph TD
    A["Client sends write"] --> B["Write lands in WiredTiger cache (RAM)"]
    B --> C["Acknowledgment sent to client ✓"]
    C --> D["Primary crashes before journal flush"]
    D --> E["Write is permanently lost ✗"]

    style A fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style B fill:#302020,stroke:#f97316,color:#e6edf3
    style C fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style D fill:#3a1a1a,stroke:#ef4444,color:#e6edf3
    style E fill:#3a1a1a,stroke:#ef4444,color:#e6edf3

The write existed only in volatile memory. The crash destroys it entirely. The client believes the write succeeded.

Case 2: w: 1, j: true — Journaling Enabled

graph TD
    A["Client sends write"] --> B["Write lands in WiredTiger cache"]
    B --> C["Journal flushed to disk"]
    C --> D["Acknowledgment sent to client ✓"]
    D --> E["Primary crashes"]
    E --> F["Recovery: replay journal"]
    F --> G["Write restored ✓"]

    style A fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style B fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style C fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style D fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style E fill:#302020,stroke:#f97316,color:#e6edf3
    style F fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style G fill:#1a3a2a,stroke:#22c55e,color:#e6edf3

The journal allows recovery — but only on the same node. If the disk is destroyed or the node is permanently lost, the write is gone because it was never replicated.

Case 3: w: "majority", j: true — Full Protection

graph TD
    A["Client sends write"] --> B["Write journaled on primary"]
    B --> C["Write replicated to majority of secondaries"]
    C --> D["Acknowledgment sent to client ✓"]
    D --> E["Primary crashes permanently"]
    E --> F["New primary elected from secondaries"]
    F --> G["Data preserved on new primary ✓"]

    style A fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style B fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style C fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style D fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style E fill:#302020,stroke:#f97316,color:#e6edf3
    style F fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style G fill:#1a3a2a,stroke:#22c55e,color:#e6edf3

Even if the primary is permanently lost, the data survives. A new primary is elected from the secondaries that already have the write. This is the only configuration that guarantees no data loss during failover.

Key Takeaways

In distributed databases, durability is rarely a single guarantee — it is a composition of multiple mechanisms, each addressing a different failure mode.

In MongoDB, a successful durable write is the result of multiple layers working together:

graph TD
    A["WiredTiger Cache (RAM)"] -->|"crash recovery"| B["Journal (WAL on disk)"]
    B -->|"replication"| C["Oplog (replication log)"]
    C -->|"fault tolerance"| D["Majority Acknowledgment"]

    A -.- E["Protects against: nothing (volatile)"]
    B -.- F["Protects against: process crashes"]
    C -.- G["Protects against: single-node failure"]
    D -.- H["Protects against: minority-node failures"]

    style A fill:#302020,stroke:#f97316,color:#e6edf3
    style B fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style C fill:#1f2937,stroke:#58a6ff,color:#e6edf3
    style D fill:#1a3a2a,stroke:#22c55e,color:#e6edf3
    style E fill:#1f2937,stroke:#6b7280,color:#8b949e
    style F fill:#1f2937,stroke:#6b7280,color:#8b949e
    style G fill:#1f2937,stroke:#6b7280,color:#8b949e
    style H fill:#1f2937,stroke:#6b7280,color:#8b949e

Each layer exists to solve a different failure mode:

• WiredTiger cache provides fast in-memory writes — but offers no durability
• Journaling provides crash recovery on a single node — but doesn’t protect against node loss
• Oplog replication provides data redundancy across nodes — but doesn’t guarantee disk persistence
• Majority acknowledgment provides fault tolerance — ensuring data survives minority failures

When we configure w: "majority" or j: true, we’re not just toggling database settings — we’re making explicit trade-offs between latency, durability, and system resilience.

Understanding these trade-offs is what separates simply using a database from operating it reliably in production.

---

Further Reading

• Cache Thrashing in Databases — The Silent Performance Killer