Imagine a Tuesday morning. A corporate client is on the phone, and they’re not happy. $50,000 left their account last week. Nobody at their company authorized it.

You open the account. The transfer is sitting right there. $50K, debited, settled, three business days ago. But who initiated it? At what time, exactly? Was it a duplicate of an authorized transfer earlier that day? A retry after a network glitch? A manual correction by a teammate who forgot to leave a note?

The database has the current balance. It doesn’t have the path that got there. In the model you built around, each state change overwrote the current row. The trail, if it exists, lives somewhere else.

So you open three terminals, pull from Loki, stare at a Grafana dashboard that shows you where things ended, and squint at timestamps that disagree by eighty milliseconds. At some point, refunding first and investigating later starts to look safer than giving the client a story you cannot prove. By the time you give up, it’s late, you’ve refunded the $50K to be safe, and the client has filed you under “still maturing.”

You have a number. You don’t have a story. And in payments, the story is the thing that costs you money.

The row keeps replacing the past

The default move is to store the current balance. UPDATE accounts SET balance = 90 WHERE id = 123. I’d written some version of that statement a hundred times before I stopped to ask what I was throwing away. State updated in place. Want concurrency safety? Wrap it in a lock. Want history? Stick a row in transactions_audit. Move on.

Then someone said “store events, not state” and I didn’t get it. Isn’t that just an audit log? I had one. Most systems do — usually called transactions, history, or audit_log. Looking at it for the first time, I struggled to see what problem event sourcing was actually solving.

Audit logs are shadows

In CRUD systems, state is the source of truth and history is optional. In event sourcing, history is the source of truth and state is optional.

In CRUD, the audit log was a shadow — written by code that ran alongside the main write path. The two could drift apart. Someone runs a one-off UPDATE to quietly fix a balance, forgets to insert the matching audit row, and suddenly the history is incomplete. I’ve been the one who did it. Once, I patched a status field directly in the database and skipped the endpoint that would’ve emitted the ELK log. The state changed with no trace of why.

In an event-sourced system, the accounts row is not the source of truth. The events are. A balance table may still exist, but it’s a projection — something you can rebuild from the log. Events are never updated; new facts are always appended instead. That isn’t a limitation. That’s the design.

The current value is a result

Then it hit me: isn’t git also just this? Every commit is an immutable append to history. A bank statement has the same shape: the movements explain the number. Event sourcing gave me that feeling, applied to business data. The current value is not a thing you overwrite; it is the result of the history behind it.

That immutability is a rule the system has to enforce — through storage permissions, append-only APIs, and operational controls. It isn’t something the database guarantees; with enough access, an administrator can still mutate data. The auditability the rest of the piece relies on depends on enforcement, not just modeling it.

Event sourcing doesn’t mean you never store current state. It means current state is not the truth. If a derived view burns down, the event log should be enough to rebuild it.

CRUD asks what is true right now? and stores the answer. Event sourcing asks what happened, in order? and derives the current answer.

My first reaction: this was cleaner than I expected. My second: wait, a busy account has millions of events. We replay all of them, every read?

I Thought Snapshots Were Cheating

Until I learned the difference between truth and a cache.

Replay has a cost

Event sourcing solved the audit problem. It created a new one.

Picture an operating account at a corporate payments platform. Payroll on the 25th, supplier payouts on the 1st, internal transfers weekly. In a payments platform, one payment can fan out into many events: authorization, validation, fee, ledger movement, settlement, notification. A few thousand events on a working day sounds small until it runs for eighteen months. For a busy account, the relevant stream can reach the low millions.

The CFO opens the balance dashboard on a Monday morning. Under the hood, the system starts replaying events.

Event one: debit. Event two: credit. Event three: a fee. It will do this a million times.

The dashboard takes six seconds.

The CFO calls.

Six seconds doesn’t sound like much when you read it. It does when every filter change, every refresh, every interaction sits behind it.

If you derive the balance at read time, the cost grows with the stream. Eventually that growth becomes user-visible. More I/O and more deserialization also mean more chances to hit a timeout or dropped connection. The system may have to restart or retry a large fold from the beginning.

For an account balance, you cannot skip the events that define the balance. You can optimize the read path, but semantically the answer still depends on the relevant stream.

So the fix could not be making the mutable balance row authoritative again. It had to make replay cheaper without changing what counted as truth.

A checkpoint, not a replacement

The answer has a name: snapshots.

A snapshot is a periodic checkpoint of derived state, tagged with the version of the last event it includes. At some point you take the result of folding events 1 through 50,000 and write it down. Not as truth. As a cache. The next read loads the snapshot and folds only events 50,001 onward. Maybe five events. Maybe five hundred. Not a million.

The version is the load-bearing part. The snapshot says: I am the result of folding this log up to event 50,000. Without it, you don’t know where to resume. But a snapshot usually needs more than the last event number — it also needs to know which projection code and schema produced it. If the function changes, the cached result may no longer mean what the new code thinks it means.

The burn test

My immediate reaction was: this feels like cheating. Didn’t I just replace one stored balance with another stored balance, except now with more steps?

A CRUD row is truth. If that row is the only source of truth, deleting it loses the balance.

A snapshot is not truth. It’s a derived view. Delete it: the million events are still there. Rebuild from scratch: same number. The math reruns.

A useful way to tell the difference is the burn test. Can I delete it and still recover? If yes, it’s a cache. If no, it’s truth. Snapshots pass that test.

A simplified example of burn test:

const events = [
  { type: 'credit', amount: 100 },
  { type: 'debit', amount: 30 },
  { type: 'credit', amount: 25 },
  { type: 'fee', amount: 5 },
];

const fold = (events) =>
  events.reduce((balance, event) => {
    if (event.type === 'credit') return balance + event.amount;
    if (event.type === 'debit') return balance - event.amount;
    if (event.type === 'fee') return balance - event.amount;
    return balance;
  }, 0);

let snapshot = {
  afterEvent: 2,
  balance: fold(events.slice(0, 2)),
};

const beforeBurn = snapshot.balance + fold(events.slice(snapshot.afterEvent));

// Throw away the snapshot.

const afterBurn = fold(events);

console.log('Before burn:', beforeBurn);
console.log('After burn: ', afterBurn);

Output:

Before burn: 90
After burn:  90

The snapshot disappeared. The answer did not, because the events were still there.

(Rebuildable doesn’t mean free to rebuild. A projection can take hours to backfill and be operationally important to keep warm. The burn test sorts what’s truth from what’s derived — not what’s cheap from what’s expensive.)

Once I accepted that distinction, the next problem was the one hiding inside it: what happens when the thing that rebuilds the cache changes?

At that point, the system has a new kind of problem. The same truth can produce different answers depending on how it is interpreted.

I Thought CQRS Was Overkill

Until one read model could be wrong while the log stayed right.

CQRS: Command Query Responsibility Segregation, means the path that changes state and the path that reads state do not have to use the same model. Commands express intent and are checked against business rules. Queries return answers shaped for reading. Sometimes that split is two code paths. Sometimes it is two tables. Sometimes it is two databases.

When the read model falls behind

Imagine a balance projection built before reversal events existed. At the time, it handled the events everyone thought mattered:

• MoneyCredited
• MoneyDebited
• FeeCharged

Later, reversals are added. If a debit fails downstream, the system appends DebitReversed to put the money back. The write side records the new event correctly, but the balance projection is not updated.

This can happen when the write path and read model are deployed separately. The service that appends events now knows about DebitReversed; the old projection code still falls through and ignores event types it does not recognize.

That is not a broken log. It is a read model that no longer understands every fact it needs.

Same log, different answer

Now imagine a small account stream where the snapshot is taken at event 10:

The real balance should include +1,000 credited, -400 debited, -20 in fees, and +70 reversed.

But the old reducer ignores reversals, so the cached snapshot says:

1_000 - 400 - 20 // 580

Then someone notices the mismatch and fixes the reducer:

const balance = events.reduce((balance, event) => {
  if (event.type === 'MoneyCredited') return balance + event.amount;
  if (event.type === 'MoneyDebited') return balance - event.amount;
  if (event.type === 'FeeCharged') return balance - event.amount;
  if (event.type === 'DebitReversed') return balance + event.amount; // added this
  return balance;
}, 0);

Another 10 events arrive. To keep the example short, say those events net out to:

Now there are two ways to answer the same question:

• Resume from the old snapshot: $580 + $200 - $100 - $5 + $10 = $685
• Burn and replay from event one: ($1,000 - $400 - $20 + $70) + $200 - $100 - $5 + $10 = $755

Same event log. Same events. $70 apart.

Nothing alarms, because the snapshot is not corrupt. It really is the result of folding events 1 through 10. The problem is that it was folded by an older read model.

This is where CQRS started to make sense to me. Not because CQRS magically fixes stale snapshots, but because it gives the problem the right shape: the facts I write and the views I read have different jobs. The log can be correct while a projection is wrong. In practice, this is why projection versions and rebuild strategies matter. If projections are not disposable, versioned, and rebuildable, stale derived state starts pretending to be truth.

The read side needs its own plan

What I learned from this example was smaller than the jargon around it.

In CRUD, the table is usually the shared reality. If I add a reversed_amount column, the write path and read path tend to move together through the same migration.

With event sourcing and CQRS, that is not guaranteed. The write side can start appending a new fact like DebitReversed before an old projection knows what that event means. That split can be useful, but it means the read side needs a plan for new event types. Otherwise it can silently ignore facts that should change the answer.

Snapshots need a reducer version

Snapshots also need more than an event number. A snapshot is not just “balance after event 10.” It is “balance after event 10, as calculated by this version of the reducer.” If I fix the reducer to handle DebitReversed, old snapshots built by the previous reducer may be stale even though the data inside them looks valid.

That is the part I had missed: changing projection code can invalidate old derived state. If the snapshot does not record a projection version, schema version, or some other marker, the system may resume from a value that was correct under old logic and wrong under new logic.

Rebuildability is the escape hatch

The log was not the broken part here. It recorded the reversal. The projection was the broken part. That is why read models have to be rebuildable. If I cannot delete a projection and replay the log to get it back, I have started treating derived state as truth.

The dangerous failure is quiet. Nothing crashes. The snapshot is not corrupted bytes. The reducer just starts from an old interpretation and then applies the new events correctly. That is how the system can confidently show $685 when replaying from the log would show $755.

Unknown events should fail fast

The reducer shape matters too. Returning the same balance for unknown events is fine only when the event is explicitly irrelevant to this view. For a critical financial projection, silently ignoring an unknown money event is too risky. I would rather make the projection’s event handling explicit and treat unknown subscribed events as failures, or force them through an explicit versioning path.

Replaying history with new questions

That opens a question the earlier version was covering up. What if I want to reinterpret history?

Marketing asks what revenue would have been at $0.50 from day one. Compliance asks for the books under last year’s schedule. The ledger projection doesn’t change — the customer paid what they paid — but the log can still answer both questions with different reducers.

That was the part that made the split click for me. The log keeps the facts. A projection answers one question about those facts. Delete the projection and replay the events; the answer comes back. Delete the log and there is nothing left to replay.

The failure mode is letting a projection become truth by accident.

This is more work than CRUD. Sometimes a lot more. You write more code, manage an append-only log, and reason about projections that can go stale. None of that is free.

The case for it isn’t elegance. It’s the day someone asks you to explain a $50K charge and you can see the retries, reversals, fees, and state changes that led there before deciding to refund. It’s the day a CFO asks why the dashboard number changed and the answer is the log, not a git commit. It’s also the day the read model can serve the dashboard without making the write side carry that shape.

If your system never has those days, this is overkill. If it does, the extra work may be cheaper than not knowing what happened.