What Makes a System "Distributed" & Why It Is Hard

Leslie Lamport famously gave the quintessential definition of distributed computing:

Formally, a distributed system is a network of autonomous computing nodes (virtual machines, containers, or bare-metal servers) that coordinate their state and execute tasks solely by exchanging messages over an asynchronous, unreliable network. To end users and client applications, the cluster ideally appears as a single coherent, unified system.

Why We Build Distributed Systems:

1. Physical Scale (Capacity): Datasets in modern web systems exceed petabytes and cannot fit on any commercially available single physical server.
2. High Availability & Fault Tolerance: If a single server catches fire, another node in a different availability zone seamlessly assumes the workload.
3. Geographic Locality (Latency): Placing computation and data close to users (e.g., in Tokyo, Frankfurt, and Oregon) minimizes packet round-trip times bounded by the speed of light.

Originally formulated by L. Peter Deutsch and Sun Microsystems fellows, these eight false assumptions represent the most common pitfalls software engineers encounter when transitioning from monolithic to distributed systems:

1. The Network is Reliable: Fiber optic cables get severed by backhoes; Top-of-Rack (ToR) switches experience transient power loss; TCP connections reset unpredictably. Systems must assume packet loss is inevitable.
2. Latency is Zero: In-memory pointer dereferencing takes ~100 nanoseconds; sending a packet cross-continent (New York to London) takes ~ 70ms purely due to light speed in glass fiber (200,000 km/s).
3. Bandwidth is Infinite: Cross-service RPC calls serialize massive JSON payloads, saturating network interface cards (NICs) and causing queue drop bottlenecks.
4. The Network is Secure: Messages traverse public backbones, intermediate cloud routers, and VPC gateways. Mutual TLS (mTLS) and envelope encryption are mandatory.
5. Topology Does Not Change: Cloud auto-scaling, spot instance terminations, and Kubernetes pod rescheduling continuously alter IP routes and node availability.
6. There is One Administrator: Different microservices are owned by separate engineering teams across multiple organizations, clouds, and compliance regimes.
7. Transport Cost is Zero: Marshaling, unmarshaling (JSON/Protobuf), socket buffer copying, and kernel context switches consume substantial CPU cycles.
8. The Network is Homogeneous: Clusters comprise heterogeneous CPU architectures (x86 vs ARM), diverse OS kernels, and mixed network interfaces.

In single-node software architectures, systems fail deterministically: either the process is executing instructions correctly, or it has crashed completely (Crash-Stop).

In distributed systems, failures are partial and non-deterministic:

• Ambiguous RPC Responses: When Node A sends a network request to Node B and experiences a 5-second timeout, Node A cannot distinguish between three fundamentally different realities:
  1. The request was dropped on the wire before reaching Node B (operation never ran).
  2. Node B crashed while processing the request (operation partially ran).
  3. Node B executed the operation successfully, but the acknowledgment response was dropped on the return path (operation ran, but Node A thinks it timed out).
• Physical Clock Drift: Computer quartz crystals drift by milliseconds every hour. Without specialized synchronization hardware (like Google Spanner's TrueTime atomic clocks), server timestamps cannot be relied upon to establish strict global causality.