25 Top Papers

This paper marked the beginning of modern web search, demonstrating that the web's hyperlink graph measures page authority (captured by the PageRank algorithm) and greatly improves ranking. The Google prototype also revealed the significant descriptive power of anchor text, successfully utilizing these external labels to improve accuracy. We all take ranking for granted, but this paper showed how to get superior search relevance over content-only methods. Last but not least, their prototype demonstrated one of the key WSC approaches by sharding their web index across inexpensive commodity hardware.

By 2003, Google's architecture had evolved quite a bit from the 1998 prototype that maxed out at two queries per second, and the conceptual ideas of the original prototype had been proven to work in practice. Clusters of 15,000 commodity PCs used sharding to provide horizontal scaling and redundancy. Some of the more frequent hardware failures were now handled automatically. The servers themselves were managed in a relatively manual way because Borg and Linux containers had not been conceived yet. Last but not least, the paper is one of the first ones to discuss power and energy as important system attributes.

The Google File System (GFS) was the first real storage system designed at Google, and the first horizontally scalable file system tailored for large-scale, data-intensive applications. Before GFS, the only high-capacity storage systems available required putting expensive high-performance disks with RAID controllers into a rack. In contrast, GFS assembled a sea of disks, distributed across thousands of machines, into a file system. Files were split into individual chunks, and each chunk was placed on multiple disks to avoid data loss and increase read throughput. A single master node managed metadata. GFS presented a POSIX-like API but relaxed consistency semantics to simplify the system and enhance performance for its target applications. GFS was instrumental in allowing Google to significantly increase its search index size above that of other search engines, and prepared it for storage-heavy future applications like Gmail and YouTube.

MapReduce complements GFS: whereas GFS made it easy to store large amounts of data across many devices, MapReduce made it easy to process large datasets across many commodity servers. Users specify “map” functions that process key/value pairs to generate intermediate pairs, and “reduce” functions that merge intermediate values associated with the same key. The runtime system transparently handles parallelization, fault tolerance, data distribution, and load balancing. This abstraction allowed programmers without distributed systems expertise to leverage large clusters effectively. The paper demonstrated MapReduce's utility across diverse tasks like distributed sorting, web access log analysis, and inverted index construction, significantly simplifying large-scale data processing.

Complementing GFS and MapReduce, Bigtable was the initial solution to hyperscale databases and the start of the NoSQL movement that sacrificed strong transactional consistency in exchange for horizontal scalability. Bigtable stores its data across thousands of commodity servers and exposes a sparse, distributed, persistent multi-dimensional sorted map data model. Data is indexed by row key, column key, and timestamp. Bigtable uses GFS for underlying storage and provides high scalability, availability, and performance. Its strength in providing low-latency reads/writes and efficient scans over massive datasets strongly influenced subsequent NoSQL designs.

WSC services need a way to bootstrap their configuration data, and multiple replicas of a single service need to agree who's the leader. Chubby provides this functionality in a single, simple API. It uses Paxos for fault-tolerant consensus among its own replica servers. Its client-side caching mechanism, coupled with keep-alive sessions and event notification, minimizes server load while maintaining consistency.

Computer systems, particularly servers in data centers, should consume power in proportion to the amount of work performed. Most servers aren't energy proportional: they consume a significant fraction of their peak power even when idle or operating at low utilization. This short paper defined the problem and argued that better energy proportionality would significantly reduce data center operational costs and environmental impact.

Dremel implements interactive, ad-hoc querying of extremely large, read-only, nested datasets. Its high-throughput parallel query execution engine may use thousands of servers for a single query. Columnar storage allows better compression and significantly reduces I/O load because queries read only partial rows, skipping unused record fields. Its query engine uses a massively parallel tree architecture to execute SQL-like queries over petabytes of data in seconds. For many analytics tasks, Dremel reduced the programming effort and the execution time by several orders of magnitude compared to MapReduce.

The Dapper tracing infrastructure observes cluster-scale applications with low overhead, application-level transparency, and scalability. It provides developers visibility into requests as they propagate through large microservice-based architectures. By assigning globally unique IDs to requests and propagating context, Dapper reconstructs causal paths and timings across different services. The system employs adaptive sampling to manage data volume while capturing representative traces. Without Dapper (which has evolved into OSS as OpenTelemetry) it would be difficult to understand why a service is slow today, or why some requests are much slower than others.

Google-Wide Profiling (GWP) provides continuous, low-overhead profiling for WSC applications. It collects performance data (CPU usage, memory allocation, synchronization contention) from running jobs with minimal performance impact, using sampling. Its ubiquitous data collection identifies performance bottlenecks, attributes resource consumption accurately across and inside services, and tracks performance regressions over time. GWP provides the fleet-wide visibility essential for improving the efficiency of warehouse-scale applications without requiring application-specific instrumentation. GWP's in-depth, accurate profiling data supports many use cases, including understanding hardware differences, enabling feedback-directed compiler optimization, and discovering frequently used library code across all binaries.

Spanner is the first database that uniquely combines transactional consistency (external consistency, the highest consistency standard) with high availability and horizontal scalability across geographically distributed data centers. It achieves this through features like automatic sharding, replication using Paxos, and synchronous replication. A key innovation is the TrueTime API, which uses GPS and atomic clocks to generate tightly synchronized timestamps with bounded uncertainty, enabling consistent distributed transactions, consistent snapshot-reads across the database, and lock-free reads. Spanner supports a relational data model and SQL-like query language, providing familiar database semantics at a global scale.

The B4 paper details Google's experience designing, deploying, and operating a private Software-Defined Wide Area Network (SDN WAN) to connect its global data centers. The system separates the control plane from the data plane, using centralized traffic engineering controllers to manage bandwidth allocation and routing across the physical network. B4 employs custom switch hardware and OpenFlow protocols, demonstrating SDN's effectiveness in managing large, complex WANs for improved efficiency, flexibility, and cost-effectiveness compared to traditional WAN architectures.

Tail latency—high-percentile latency—in large-scale distributed systems has a disproportionate impact on user experience. In systems involving thousands of servers for a single request, even rare, small delays on individual components often dominate overall response time due to fan-out. Many factors cause latency variability, including resource sharing, background activities, and queueing delays. The paper outlines several techniques to mitigate tail latency, such as hedging requests (sending the same request to multiple replicas), latency-induced probation, and micro-partitioned data.

This study complemented Google-Wide Profiling by collecting significantly more data, mostly at the instruction level. The authors analyze instruction mix, CPI (Cycles Per Instruction), cache misses, branch mispredictions, and front-end stalls across diverse workloads. They demonstrate how microarchitectural features interact with software at scale, revealing bottlenecks like front-end stalls that are often masked in smaller studies.

Borg is a large-scale cluster manager responsible for scheduling, deploying, monitoring, and managing applications across tens of thousands of machines. It handles both long-running services and batch jobs, focusing on high reliability, scalability, and efficient resource utilization through techniques like resource reclamation and fine-grained sharing. Borg uses a logically centralized primary scheduler that is replicated for fault tolerance. Distributed agents (Borglets) provide fine-grained resource management on each machine. Borg allocates resources based on application priorities and quotas, supports declarative job specifications, and provides features like rolling updates and health checking.

Managing thousands of services across myriads of servers shouldn't be an art, it should be a science. Site Reliability Engineering (SRE) embeds software engineering principles into infrastructure and operations tasks, focusing on automation, reliability, and scalability. It provides a blueprint for organizations aiming to build and maintain highly reliable, scalable software systems. Crucially, absolute reliability is a non-goal; rather, it's balancing reliability and velocity, and learning from failures via blameless postmortems.

This paper disclosed Google's first-generation Tensor Processing Unit (TPU), the first custom ASIC deployed at scale to accelerate AI inference workloads. The authors describe the TPU's architecture, especially its large matrix multiplier unit and on-chip memory optimized for deep learning computations. TPUs demonstrate significantly higher performance and energy efficiency on production AI workloads relative to contemporary CPUs and GPUs. The analysis highlights the benefits of domain-specific hardware acceleration for critical large-scale workloads, and summarizes key learnings underpinning the substantial improvements in throughput and energy efficiency for machine learning inference tasks.

WSCs need to optimize for microsecond-level latencies, driven by faster data center networks, new memory hierarchies, and accelerators. Using two case studies (how to waste a fast data center network, and how to waste a fast data center processor), the paper points out how system optimizations targeting nanosecond- and millisecond-scale events are inadequate for events in the microsecond range. New techniques can achieve high performance at microsecond latencies but need co-design across hardware, kernel, and application layers.

Andromeda describes Google Cloud Platform's network virtualization stack. Its SDN (software-defined network) design provides high-performance, secure networking for virtual machines. Andromeda decouples the control plane from a distributed data plane implemented partially on-host and partially offloaded to hardware. This architecture enables rapid provisioning of network services (VPCs, firewalls, load balancers), offers performance approaching bare-metal networks, and ensures strong security isolation between tenants. The design prioritizes product velocity and operational agility, simplifying new network features and scaling infrastructure efficiently.

Network hardware provides hundreds of Gbps, but traditional kernel APIs impose too high an overhead to actually use this capacity. Snap implements a high-performance host networking stack outside the kernel using a microkernel-inspired architecture. This modularity enhances security by isolating components and allows deploying specialized network functions (e.g., custom protocols, hardware offloads) without modifying the core kernel.

Thunderbolt enforces power caps in heavily oversubscribed large data centers. It uses workload classification and QoS feedback to apply power caps intelligently, prioritizing latency-sensitive tasks and selectively throttling background or batch jobs. This approach maximizes useful work and hides the existence of power capping from the most important users, improving data center efficiency by safely running power infrastructure near its limits when necessary. (If you have time, you should pair your reading of this paper with the original “Power provisioning for a warehouse-sized computer” paper that introduced power oversubscription at Google.)

TCP has dealt with internet congestion effectively by using packet loss to detect congestion. However, it doesn't work well in data center networks which exhibit very high speeds combined with very low latencies and small-buffer switches. Swift shows that a very simple protocol based on accurate measurements of packet delays outperforms all others. Its simple, end-host-based approach achieves extremely low network queues, high link utilization, and fast convergence.

Video Coding Units (VCUs) dramatically accelerate video processing, and represent the most popular WSC accelerator after AI accelerators. The authors describe the VCU hardware, software drivers, server integration, and job management systems, as well as the challenges of integrating specialized hardware into a massive, general-purpose computing environment, including fault tolerance, monitoring, and programming models.

Starting from a traditional Clos topology, the Jupiter network evolved to a more flexible direct-connect architecture, with MEMS-based Optical Circuit Switches (OCS) that provide dedicated optical paths to dynamically reconfigure the network topology. WSC networks are continually growing and changing, and thus many design aspects of Jupiter support flexibility, incremental updates, sophisticated traffic engineering and dynamically reconfigured traffic flows Automated network operations support incremental capacity delivery and topology engineering without disrupting live services.

Traditional ML training runs a large cluster's accelerators in a highly synchronous, lock-step way that requires all accelerators to be the same. However, models have become too large to fit into a single cluster. Pathways is a new orchestration layer for accelerator systems to train and serve even larger-scale AI models. Pathways' architecture utilizes asynchronous distributed dataflow and routes computation across heterogeneous hardware accelerators (CPUs, GPUs, TPUs), in combination with a single-controller model and centralized scheduling. This approach allows moving beyond current dense, single-task models towards sparse, multi-task models that can learn many tasks simultaneously and leverage sparsity for efficiency.