Git, Linux & DevOps
Distributed Systems Basics
A distributed system is one where components run on separate machines and communicate over a network rather than through memory on a single computer. That single fact - the network in between - introduces problems that don't exist in a program running on one machine: messages can be delayed, lost, or arrive out of order, machines can fail independently of each other, and there is no single, instantaneous view of 'now' that every part agrees on. Most of what's studied under this heading is really about living with those constraints rather than eliminating them, since they can't be eliminated.
Why it matters
- Almost every real system is distributed once it has more than one machine
- A web server talking to a separate database is already a distributed system, even if nobody thinks of it that way, and it inherits the same failure modes as a much larger one.
- Network failures are normal, not exceptional
- A request can fail because the network dropped it, not because anything is wrong with either side - code that assumes every call either fully succeeds or cleanly fails misses this middle case.
- It explains why adding a cache or a replica isn't free
- Every copy of data introduces a question of how and when the copies agree with each other, which is a real design decision, not a detail.
- It underlies microservices, distributed databases and most cloud infrastructure
- The vocabulary here - consistency, partition, consensus - is the shared foundation those more specific topics build on.
The network is not reliable, instant or free
It's tempting to write code as if a network call behaves like a function call - it either returns a result or throws an error, right away. In practice a request can be delayed, dropped, duplicated, or arrive out of order relative to another request, and bandwidth and latency are real, non-zero constraints rather than implementation details to ignore. Code that quietly assumes otherwise tends to work fine in testing and then fail in ways that are hard to reproduce once it runs on a real network under real conditions.
The CAP theorem, briefly
When a network partition happens - some machines can't reach others - a system has to choose between staying consistent (every read reflects the latest write, even if that means refusing some requests) and staying available (every request gets a response, even if it might be slightly stale). Partitions are a fact of real networks, not a rare edge case, so this is a genuine, ongoing design decision that different systems resolve differently depending on what matters more for their use case.
Idempotency and retries
Because a network call can fail without the caller knowing whether the other side actually completed the work, the standard way to handle a failed or timed-out call is to retry it. That's only safe if the operation is idempotent - doing it twice has the same effect as doing it once. An idempotency key, a value the caller generates once and sends with every retry, lets the receiving side recognize a repeated request and return the original result instead of repeating the effect.
POST /charges
{
"amount": 2000,
"currency": "usd",
"idempotency_key": "order-48213-attempt-1"
}
# if the client times out and retries with the SAME key,
# the server recognizes it and returns the original result
# instead of charging the customer twiceMistakes people make here
- Assuming a network call either fully succeeds or cleanly fails
- a request can time out after the server actually processed it - the response was lost, not the operation - so 'no response' isn't the same as 'nothing happened', and code that assumes otherwise double-processes things on retry.
- Treating clocks on different machines as synchronized
- clock drift between machines is real and ordinary; relying on comparing timestamps from two different machines to decide what happened first is a common source of subtle bugs.
- Ignoring what happens during a network partition
- it's tempting to design only for the case where every machine can reach every other machine; a real system eventually has to decide what a node does when it can't reach the others, and assuming it just won't happen isn't a plan.
- Confusing 'distributed' with 'more reliable'
- adding more machines adds more independent things that can fail; reliability comes from how failures are handled - redundancy, retries, failover - not from the mere fact of being distributed.
- Retrying a non-idempotent operation on failure
- retrying a request that charges a card or sends an email, without an idempotency mechanism, risks doing it twice if the first attempt actually succeeded but its response was lost.
Strengths and trade-offs
Where it is strong
- Adding machines lets a system handle more load and survive individual machine failures - neither is possible with a single machine, however powerful.
- Understanding these problems well transfers directly to designing reliable APIs, even ones most people wouldn't call a 'distributed system'.
- Redundancy across machines and locations is what makes real high availability possible at all.
The trade-offs
- There's no way around the fundamental limits here - you can only choose which trade-offs to accept, not eliminate them.
- Debugging is genuinely harder: a bug that only shows up when two machines' timing interacts a certain way is much harder to reproduce than a single-process bug.
- Every additional machine or network hop is another independent failure point to plan for, which is real ongoing design and operational effort.
Who needs this
Anyone building or operating a system with more than one machine involved, which includes most backend work once a database is on a separate host from the application, benefits from these ideas, even without formally studying distributed systems. Building the underlying infrastructure - databases, message queues, consensus systems - needs it much more deeply.
Questions about distributed systems basics
- Is my app a 'distributed system' if it just talks to a database?
- Yes, in the sense that matters here: the application and the database are on separate machines communicating over a network, so it inherits the same basic issues - the call can be slow, can fail, can succeed without the response arriving - as a much larger system. The scale differs, the underlying problems don't.
- What is the CAP theorem actually saying?
- That when a network partition happens - some machines can't reach others - a system has to choose between staying consistent (every read reflects the latest write, even if that means refusing some requests) and staying available (every request gets a response, even if it might be slightly stale). Partitions are a fact of real networks, so this choice is a real, ongoing design decision, not an academic edge case.
- What does 'idempotent' mean and why does it matter so much?
- An operation is idempotent if doing it twice has the same effect as doing it once, like setting a value, versus incrementing it. It matters because retries are the standard way to handle a failed or timed-out network call, and retrying safely requires that doing the operation again doesn't cause harm, which is only true if it's idempotent, or made idempotent with a mechanism like an idempotency key.
- Do I need to understand consensus algorithms to work with distributed systems?
- Not to use them - most developers rely on a database or coordination service that already implements consensus internally, the same way most developers use TCP without implementing it themselves. Understanding consensus in depth matters more if you're building the underlying infrastructure, like a distributed database or coordination service.