From a51fd07857d837cd80223c48326b296e7cc7fc5d Mon Sep 17 00:00:00 2001 From: Feng Ruohang Date: Sun, 10 Aug 2025 16:45:07 +0800 Subject: [PATCH] update part index --- content/en/part-i.md | 38 +++- content/en/part-ii.md | 52 ++++-- content/en/part-iii.md | 2 +- content/en/preface.md | 44 ++--- content/en/toc.md | 216 ++++++++++++++++++++--- content/zh/part-i.md | 63 ++++--- content/zh/part-ii.md | 135 +++++++------- content/zh/part-iii.md | 50 +++--- content/zh/toc.md | 389 +++++++++++++++++++++++++++++++++++++---- 9 files changed, 775 insertions(+), 214 deletions(-) diff --git a/content/en/part-i.md b/content/en/part-i.md index 1fc6445..9dd6d5a 100644 --- a/content/en/part-i.md +++ b/content/en/part-i.md @@ -8,7 +8,7 @@ breadcrumbs: false This page is from the 1st edition, 2nd edition is not available yet. {{< /callout >}} -The first four chapters go through the fundamental ideas that apply to all data sys‐ tems, whether running on a single machine or distributed across a cluster of machines: +The first five chapters go through the fundamental ideas that apply to all data systems, whether running on a single machine or distributed across a cluster of machines: 1. [Chapter 1](/en/ch1) introduces the tradeoffs that data systems must make, such as the balance between consistency and availability, and how these tradeoffs affect system design. @@ -18,18 +18,40 @@ The first four chapters go through the fundamental ideas that apply to all data 4. [Chapter 4](/en/ch4) turns to the internals of storage engines and looks at how databases lay out data on disk. Different storage engines are optimized for different workloads, and choosing the right one can have a huge effect on performance. -5. [Chapter 5](/en/ch5) compares various formats for data encoding (serialization) and espe‐ cially examines how they fare in an environment where application requirements change and schemas need to adapt over time. +5. [Chapter 5](/en/ch5) compares various formats for data encoding (serialization) and especially examines how they fare in an environment where application requirements change and schemas need to adapt over time. Later, [Part II](/en/part-ii) will turn to the particular issues of distributed data systems. -## Index +## [1. Trade-offs in Data Systems Architecture](/en/ch1) +- [Analytical versus Operational Systems](/en/ch1#sec_introduction_analytics) +- [Cloud versus Self-Hosting](/en/ch1#sec_introduction_cloud) +- [Distributed versus Single-Node Systems](/en/ch1#sec_introduction_distributed) +- [Data Systems, Law, and Society](/en/ch1#sec_introduction_compliance) +- [Summary](/en/ch1#summary) -- [1. Tradeoffs in Data Systems Architecture](/en/ch1) -- [2. Defining NonFunctional Requirements](/en/ch2) -- [3. Data Models and Query Languages](/en/ch3) -- [4. Storage and Retrieval](/en/ch4) -- [5. Encoding and Evolution](/en/ch5) +## [2. Defining Nonfunctional Requirements](/en/ch2) +- [Case Study: Social Network Home Timelines](/en/ch2#sec_introduction_twitter) +- [Describing Performance](/en/ch2#sec_introduction_percentiles) +- [Reliability and Fault Tolerance](/en/ch2#sec_introduction_reliability) +- [Scalability](/en/ch2#sec_introduction_scalability) +- [Maintainability](/en/ch2#sec_introduction_maintainability) +- [Summary](/en/ch2#summary) +## [3. Data Models and Query Languages](/en/ch3) +- [Relational Model versus Document Model](/en/ch3#sec_datamodels_history) +- [Graph-Like Data Models](/en/ch3#sec_datamodels_graph) +- [Event Sourcing and CQRS](/en/ch3#sec_datamodels_events) +- [Dataframes, Matrices, and Arrays](/en/ch3#sec_datamodels_dataframes) +- [Summary](/en/ch3#summary) +## [4. Storage and Retrieval](/en/ch4) +- [Storage and Indexing for OLTP](/en/ch4#sec_storage_oltp) +- [Data Storage for Analytics](/en/ch4#sec_storage_analytics) +- [Multidimensional and Full-Text Indexes](/en/ch4#sec_storage_multidimensional) +- [Summary](/en/ch4#summary) +## [5. Encoding and Evolution](/en/ch5) +- [Formats for Encoding Data](/en/ch5#sec_encoding_formats) +- [Modes of Dataflow](/en/ch5#sec_encoding_dataflow) +- [Summary](/en/ch5#summary) diff --git a/content/en/part-ii.md b/content/en/part-ii.md index ec0f160..40bfbe7 100644 --- a/content/en/part-ii.md +++ b/content/en/part-ii.md @@ -89,24 +89,52 @@ Replication can also help improve performance. We discuss replication in [Chapte Splitting a big database into smaller subsets called *partitions* so that different partitions can be assigned to different nodes (also known as *sharding*). We discuss partitioning in [Chapter 7](/en/ch7). -These are separate mechanisms, but they often go hand in hand, as illustrated in Figure II-1. +These are separate mechanisms, but they often go hand in hand, as illustrated in [Figure II-1](#fig_replication_partitioning). -![](/fig/ddia_08.png) +{{< figure src="/fig/ddia_08.png" id="fig_replication_partitioning" caption="*Figure II-1. A database split into two partitions, with two replicas per partition." class="w-full my-4" >}} -> *Figure II-1. A database split into two partitions, with two replicas per partition.* +With an understanding of those concepts, we can discuss the difficult trade-offs that you need to make in a distributed system. +We’ll discuss *transactions* in [Chapter 8](/en/ch8), as that will help you understand all the many things that can go wrong in a data system, and what you can do about them. +We’ll conclude this part of the book by discussing the fundamental limitations of distributed systems in [Chapters 9](/en/ch9) and [10](/en/ch10). -With an understanding of those concepts, we can discuss the difficult trade-offs that you need to make in a distributed system. We’ll discuss *transactions* in Chapter 7, as that will help you understand all the many things that can go wrong in a data system, and what you can do about them. We’ll conclude this part of the book by discussing the fundamental limitations of distributed systems in Chapters 8 and 9. - -Later, in Part III of this book, we will discuss how you can take several (potentially distributed) datastores and integrate them into a larger system, satisfying the needs of a complex application. But first, let’s talk about distributed data. +Later, in [Part III](/en/part-iii) of this book, we will discuss how you can take several (potentially distributed) datastores and integrate them into a larger system, +satisfying the needs of a complex application. But first, let’s talk about distributed data. -## Index +## [6. Replication](/en/ch6) +- [Single-Leader Replication](/en/ch6#sec_replication_leader) +- [Problems with Replication Lag](/en/ch6#sec_replication_lag) +- [Multi-Leader Replication](/en/ch6#sec_replication_multi_leader) +- [Leaderless Replication](/en/ch6#sec_replication_leaderless) +- [Summary](/en/ch6#summary) + +## [7. Sharding](/en/ch7) +- [Pros and Cons of Sharding](/en/ch7#sec_sharding_reasons) +- [Sharding of Key-Value Data](/en/ch7#sec_sharding_key_value) +- [Request Routing](/en/ch7#sec_sharding_routing) +- [Sharding and Secondary Indexes](/en/ch7#sec_sharding_secondary_indexes) +- [Summary](/en/ch7#summary) + +## [8. Transactions](/en/ch8) +- [What Exactly Is a Transaction?](/en/ch8#sec_transactions_overview) +- [Weak Isolation Levels](/en/ch8#sec_transactions_isolation_levels) +- [Serializability](/en/ch8#sec_transactions_serializability) +- [Distributed Transactions](/en/ch8#sec_transactions_distributed) +- [Summary](/en/ch8#summary) + +## [9. The Trouble with Distributed Systems](/en/ch9) +- [Faults and Partial Failures](/en/ch9#sec_distributed_partial_failure) +- [Unreliable Networks](/en/ch9#sec_distributed_networks) +- [Unreliable Clocks](/en/ch9#sec_distributed_clocks) +- [Knowledge, Truth, and Lies](/en/ch9#sec_distributed_truth) +- [Summary](/en/ch9#summary) + +## [10. Consistency and Consensus](/en/ch10) +- [Linearizability](/en/ch10#sec_consistency_linearizability) +- [ID Generators and Logical Clocks](/en/ch10#sec_consistency_logical) +- [Consensus](/en/ch10#sec_consistency_consensus) +- [Summary](/en/ch10#summary) -- [6. Replication](/en/ch6) -- [7. Partitioning](/en/ch7) -- [8. Transactions](/en/ch8) -- [9. The Trouble with Distributed Systems](/en/ch9) -- [10. Consistency and Consensus](/en/ch10) ### References diff --git a/content/en/part-iii.md b/content/en/part-iii.md index 73b0034..a1b48ce 100644 --- a/content/en/part-iii.md +++ b/content/en/part-iii.md @@ -57,7 +57,6 @@ By being clear about which data is derived from which other data, you can bring This point will be a running theme throughout this part of the book. - ## Overview of Chapters We will start in [Chapter 11](/en/ch11) by examining batch-oriented dataflow systems such as MapReduce, and see how they give us good tools and principles for building large- scale data systems. @@ -70,3 +69,4 @@ In [Chapter 12](/en/ch12) we will take those ideas and apply them to data stream - [11. Batch Processing](/en/ch11) (WIP) - [12. Stream Processing](/en/ch12) (WIP) - [13. Doing the Right Thing](/en/ch13) (WIP) + diff --git a/content/en/preface.md b/content/en/preface.md index 38c6517..cd43074 100644 --- a/content/en/preface.md +++ b/content/en/preface.md @@ -8,15 +8,15 @@ breadcrumbs: false This page is from the 1st edition, 2nd edition is not available yet. {{< /callout >}} -If you have worked in software engineering in recent years, especially in server-side and backend systems, you have probably been bombarded with a plethora of buzz‐ words relating to storage and processing of data. NoSQL! Big Data! Web-scale! Sharding! Eventual consistency! ACID! CAP theorem! Cloud services! MapReduce! Real-time! +If you have worked in software engineering in recent years, especially in server-side and backend systems, you have probably been bombarded with a plethora of buzzwords relating to storage and processing of data. NoSQL! Big Data! Web-scale! Sharding! Eventual consistency! ACID! CAP theorem! Cloud services! MapReduce! Real-time! -In the last decade we have seen many interesting developments in databases, in dis‐ tributed systems, and in the ways we build applications on top of them. There are various driving forces for these developments: +In the last decade we have seen many interesting developments in databases, in distributed systems, and in the ways we build applications on top of them. There are various driving forces for these developments: - Internet companies such as Google, Yahoo!, Amazon, Facebook, LinkedIn, Microsoft, and Twitter are handling huge volumes of data and traffic, forcing them to create new tools that enable them to efficiently handle such scale. - Businesses need to be agile, test hypotheses cheaply, and respond quickly to new market insights by keeping development cycles short and data models flexible. - Free and open source software has become very successful and is now preferred to commercial or bespoke in-house software in many environments. - CPU clock speeds are barely increasing, but multi-core processors are standard, and networks are getting faster. This means parallelism is only going to increase. -- Even if you work on a small team, you can now build systems that are distributed across many machines and even multiple geographic regions, thanks to infra‐ structure as a service (IaaS) such as Amazon Web Services. +- Even if you work on a small team, you can now build systems that are distributed across many machines and even multiple geographic regions, thanks to infrastructure as a service (IaaS) such as Amazon Web Services. - Many services are now expected to be highly available; extended downtime due to outages or maintenance is becoming increasingly unacceptable. *Data-intensive applications* are pushing the boundaries of what is possible by making use of these technological developments. We call an application *data-intensive* if data is its primary challenge—the quantity of data, the complexity of data, or the speed at which it is changing—as opposed to *compute-intensive*, where CPU cycles are the bottleneck. @@ -27,21 +27,21 @@ The buzzwords that fill this space are a sign of enthusiasm for the new possibil Fortunately, behind the rapid changes in technology, there are enduring principles that remain true, no matter which version of a particular tool you are using. If you understand those principles, you’re in a position to see where each tool fits in, how to make good use of it, and how to avoid its pitfalls. That’s where this book comes in. -The goal of this book is to help you navigate the diverse and fast-changing landscape of technologies for processing and storing data. This book is not a tutorial for one particular tool, nor is it a textbook full of dry theory. Instead, we will look at examples of successful data systems: technologies that form the foundation of many popular applications and that have to meet scalability, performance, and reliability require‐ ments in production every day. +The goal of this book is to help you navigate the diverse and fast-changing landscape of technologies for processing and storing data. This book is not a tutorial for one particular tool, nor is it a textbook full of dry theory. Instead, we will look at examples of successful data systems: technologies that form the foundation of many popular applications and that have to meet scalability, performance, and reliability requirements in production every day. -We will dig into the internals of those systems, tease apart their key algorithms, dis‐ cuss their principles and the trade-offs they have to make. On this journey, we will try to find useful ways of *thinking about* data systems—not just *how* they work, but also *why* they work that way, and what questions we need to ask. +We will dig into the internals of those systems, tease apart their key algorithms, discuss their principles and the trade-offs they have to make. On this journey, we will try to find useful ways of *thinking about* data systems—not just *how* they work, but also *why* they work that way, and what questions we need to ask. -After reading this book, you will be in a great position to decide which kind of tech‐ nology is appropriate for which purpose, and understand how tools can be combined to form the foundation of a good application architecture. You won’t be ready to build your own database storage engine from scratch, but fortunately that is rarely necessary. You will, however, develop a good intuition for what your systems are doing under the hood so that you can reason about their behavior, make good design decisions, and track down any problems that may arise. +After reading this book, you will be in a great position to decide which kind of technology is appropriate for which purpose, and understand how tools can be combined to form the foundation of a good application architecture. You won’t be ready to build your own database storage engine from scratch, but fortunately that is rarely necessary. You will, however, develop a good intuition for what your systems are doing under the hood so that you can reason about their behavior, make good design decisions, and track down any problems that may arise. ## Who Should Read This Book? -If you develop applications that have some kind of server/backend for storing or pro‐ cessing data, and your applications use the internet (e.g., web applications, mobile apps, or internet-connected sensors), then this book is for you. +If you develop applications that have some kind of server/backend for storing or processing data, and your applications use the internet (e.g., web applications, mobile apps, or internet-connected sensors), then this book is for you. -This book is for software engineers, software architects, and technical managers who love to code. It is especially relevant if you need to make decisions about the architec‐ ture of the systems you work on—for example, if you need to choose tools for solving a given problem and figure out how best to apply them. But even if you have no choice over your tools, this book will help you better understand their strengths and weaknesses. +This book is for software engineers, software architects, and technical managers who love to code. It is especially relevant if you need to make decisions about the architecture of the systems you work on—for example, if you need to choose tools for solving a given problem and figure out how best to apply them. But even if you have no choice over your tools, this book will help you better understand their strengths and weaknesses. -You should have some experience building web-based applications or network serv‐ ices, and you should be familiar with relational databases and SQL. Any non- relational databases and other data-related tools you know are a bonus, but not required. A general understanding of common network protocols like TCP and HTTP is helpful. Your choice of programming language or framework makes no dif‐ ference for this book. +You should have some experience building web-based applications or network services, and you should be familiar with relational databases and SQL. Any non-relational databases and other data-related tools you know are a bonus, but not required. A general understanding of common network protocols like TCP and HTTP is helpful. Your choice of programming language or framework makes no difference for this book. If any of the following are true for you, you’ll find this book valuable: @@ -56,25 +56,25 @@ Sometimes, when discussing scalable data systems, people make comments along the ## Scope of This Book -This book does not attempt to give detailed instructions on how to install or use spe‐ cific software packages or APIs, since there is already plenty of documentation for those things. Instead we discuss the various principles and trade-offs that are funda‐ mental to data systems, and we explore the different design decisions taken by differ‐ ent products. +This book does not attempt to give detailed instructions on how to install or use specific software packages or APIs, since there is already plenty of documentation for those things. Instead we discuss the various principles and trade-offs that are fundamental to data systems, and we explore the different design decisions taken by different products. In the ebook editions we have included links to the full text of online resources. All links were verified at the time of publication, but unfortunately links tend to break frequently due to the nature of the web. If you come across a broken link, or if you are reading a print copy of this book, you can look up references using a search engine. For academic papers, you can search for the title in Google Scholar to find open-access PDF files. Alternatively, you can find all of the references at [*https:// github.com/ept/ddia-references*](https:// github.com/ept/ddia-references), where we maintain up-to-date links. -We look primarily at the *architecture* of data systems and the ways they are integrated into data-intensive applications. This book doesn’t have space to cover deployment, operations, security, management, and other areas—those are complex and impor‐ tant topics, and we wouldn’t do them justice by making them superficial side notes in this book. They deserve books of their own. +We look primarily at the *architecture* of data systems and the ways they are integrated into data-intensive applications. This book doesn’t have space to cover deployment, operations, security, management, and other areas—those are complex and important topics, and we wouldn’t do them justice by making them superficial side notes in this book. They deserve books of their own. Many of the technologies described in this book fall within the realm of the *Big Data* buzzword. However, the term “Big Data” is so overused and underdefined that it is not useful in a serious engineering discussion. This book uses less ambiguous terms, such as single-node versus distributed systems, or online/interactive versus offline/ batch processing systems. -This book has a bias toward free and open source software (FOSS), because reading, modifying, and executing source code is a great way to understand how something works in detail. Open platforms also reduce the risk of vendor lock-in. However, where appropriate, we also discuss proprietary software (closed-source software, soft‐ ware as a service, or companies’ in-house software that is only described in literature but not released publicly). +This book has a bias toward free and open source software (FOSS), because reading, modifying, and executing source code is a great way to understand how something works in detail. Open platforms also reduce the risk of vendor lock-in. However, where appropriate, we also discuss proprietary software (closed-source software, software as a service, or companies’ in-house software that is only described in literature but not released publicly). ## Outline of This Book This book is arranged into three parts: -1. In [Part I](/en/part-i), we discuss the fundamental ideas that underpin the design of data- intensive applications. We start in [Chapter 1](/en/ch1) by discussing what we’re actually trying to achieve: reliability, scalability, and maintainability; how we need to think about them; and how we can achieve them. In [Chapter 2](/en/ch2) we compare sev‐ eral different data models and query languages, and see how they are appropriate to different situations. In [Chapter 3](/en/ch3) we talk about storage engines: how databases arrange data on disk so that we can find it again efficiently. [Chapter 4](/en/ch4) turns to formats for data encoding (serialization) and evolution of schemas over time. -2. [In Part II](/en/part-ii), we move from data stored on one machine to data that is distributed across multiple machines. This is often necessary for scalability, but brings with it a variety of unique challenges. We first discuss replication ([Chapter 5](/en/ch5)), parti‐ tioning/sharding ([Chapter 6](/en/ch6)), and transactions ([Chapter 7](/en/ch7)). We then go into more detail on the problems with distributed systems ([Chapter 8](/en/ch8)) and what it means to achieve consistency and consensus in a distributed system ([Chapter 9](/en/ch9)). +1. In [Part I](/en/part-i), we discuss the fundamental ideas that underpin the design of data-intensive applications. We start in [Chapter 1](/en/ch1) by discussing what we’re actually trying to achieve: reliability, scalability, and maintainability; how we need to think about them; and how we can achieve them. In [Chapter 2](/en/ch2) we compare several different data models and query languages, and see how they are appropriate to different situations. In [Chapter 3](/en/ch3) we talk about storage engines: how databases arrange data on disk so that we can find it again efficiently. [Chapter 4](/en/ch4) turns to formats for data encoding (serialization) and evolution of schemas over time. +2. [In Part II](/en/part-ii), we move from data stored on one machine to data that is distributed across multiple machines. This is often necessary for scalability, but brings with it a variety of unique challenges. We first discuss replication ([Chapter 5](/en/ch5)), partitioning/sharding ([Chapter 6](/en/ch6)), and transactions ([Chapter 7](/en/ch7)). We then go into more detail on the problems with distributed systems ([Chapter 8](/en/ch8)) and what it means to achieve consistency and consensus in a distributed system ([Chapter 9](/en/ch9)). -3. In [Part III](/en/part-iii), we discuss systems that derive some datasets from other datasets. Derived data often occurs in heterogeneous systems: when there is no one data‐ base that can do everything well, applications need to integrate several different databases, caches, indexes, and so on. In [Chapter 10](/en/ch10) we start with a batch pro‐ cessing approach to derived data, and we build upon it with stream processing in [Chapter 11](/en/ch11). Finally, in [Chapter 12](/en/ch12) we put everything together and discuss approaches for building reliable, scalable, and maintainable applications in the future. +3. In [Part III](/en/part-iii), we discuss systems that derive some datasets from other datasets. Derived data often occurs in heterogeneous systems: when there is no one database that can do everything well, applications need to integrate several different databases, caches, indexes, and so on. In [Chapter 10](/en/ch10) we start with a batch processing approach to derived data, and we build upon it with stream processing in [Chapter 11](/en/ch11). Finally, in [Chapter 12](/en/ch12) we put everything together and discuss approaches for building reliable, scalable, and maintainable applications in the future. @@ -88,7 +88,7 @@ Most of what we discuss in this book has already been said elsewhere in some for [Safari](http://oreilly.com/safari) (formerly Safari Books Online) is a membership-based training and reference platform for enterprise, government, educators, and individuals. -Members have access to thousands of books, training videos, Learning Paths, interac‐ tive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Pro‐ fessional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others. +Members have access to thousands of books, training videos, Learning Paths, interactive tutorials, and curated playlists from over 250 publishers, including O’Reilly Media, Harvard Business Review, Prentice Hall Professional, Addison-Wesley Professional, Microsoft Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, and Course Technology, among others. For more information, please visit http://oreilly.com/safari. @@ -105,9 +105,9 @@ O’Reilly Media, Inc. We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at *http://bit.ly/designing-data-intensive-apps*. -To comment or ask technical questions about this book, send email to *bookques‐ tions@oreilly.com*. +To comment or ask technical questions about this book, send email to *bookquestions@oreilly.com*. -For more information about our books, courses, conferences, and news, see our web‐ site at *http://www.oreilly.com*. +For more information about our books, courses, conferences, and news, see our website at *http://www.oreilly.com*. * Find us on Facebook: [http://facebook.com/oreilly](http://facebook.com/oreilly) * Follow us on Twitter: [http://twitter.com/oreillymedia](#http://twitter.com/oreillymedia) @@ -119,14 +119,14 @@ For more information about our books, courses, conferences, and news, see our we Acknowledgments -This book is an amalgamation and systematization of a large number of other peo‐ ple’s ideas and knowledge, combining experience from both academic research and industrial practice. In computing we tend to be attracted to things that are new and shiny, but I think we have a huge amount to learn from things that have been done before. This book has over 800 references to articles, blog posts, talks, documenta‐ tion, and more, and they have been an invaluable learning resource for me. I am very grateful to the authors of this material for sharing their knowledge. +This book is an amalgamation and systematization of a large number of other people’s ideas and knowledge, combining experience from both academic research and industrial practice. In computing we tend to be attracted to things that are new and shiny, but I think we have a huge amount to learn from things that have been done before. This book has over 800 references to articles, blog posts, talks, documentation, and more, and they have been an invaluable learning resource for me. I am very grateful to the authors of this material for sharing their knowledge. -I have also learned a lot from personal conversations, thanks to a large number of people who have taken the time to discuss ideas or patiently explain things to me. In particular, I would like to thank Joe Adler, Ross Anderson, Peter Bailis, Márton Balassi, Alastair Beresford, Mark Callaghan, Mat Clayton, Patrick Collison, Sean Cribbs, Shirshanka Das, Niklas Ekström, Stephan Ewen, Alan Fekete, Gyula Fóra, Camille Fournier, Andres Freund, John Garbutt, Seth Gilbert, Tom Haggett, Pat Hel‐ land, Joe Hellerstein, Jakob Homan, Heidi Howard, John Hugg, Julian Hyde, Conrad Irwin, Evan Jones, Flavio Junqueira, Jessica Kerr, Kyle Kingsbury, Jay Kreps, Carl Lerche, Nicolas Liochon, Steve Loughran, Lee Mallabone, Nathan Marz, Caitie, McCaffrey, Josie McLellan, Christopher Meiklejohn, Ian Meyers, Neha Narkhede, Neha Narula, Cathy O’Neil, Onora O’Neill, Ludovic Orban, Zoran Perkov, Julia Powles, Chris Riccomini, Henry Robinson, David Rosenthal, Jennifer Rullmann, Matthew Sackman, Martin Scholl, Amit Sela, Gwen Shapira, Greg Spurrier, Sam Stokes, Ben Stopford, Tom Stuart, Diana Vasile, Rahul Vohra, Pete Warden, and Brett Wooldridge. +I have also learned a lot from personal conversations, thanks to a large number of people who have taken the time to discuss ideas or patiently explain things to me. In particular, I would like to thank Joe Adler, Ross Anderson, Peter Bailis, Márton Balassi, Alastair Beresford, Mark Callaghan, Mat Clayton, Patrick Collison, Sean Cribbs, Shirshanka Das, Niklas Ekström, Stephan Ewen, Alan Fekete, Gyula Fóra, Camille Fournier, Andres Freund, John Garbutt, Seth Gilbert, Tom Haggett, Pat Helland, Joe Hellerstein, Jakob Homan, Heidi Howard, John Hugg, Julian Hyde, Conrad Irwin, Evan Jones, Flavio Junqueira, Jessica Kerr, Kyle Kingsbury, Jay Kreps, Carl Lerche, Nicolas Liochon, Steve Loughran, Lee Mallabone, Nathan Marz, Caitie, McCaffrey, Josie McLellan, Christopher Meiklejohn, Ian Meyers, Neha Narkhede, Neha Narula, Cathy O’Neil, Onora O’Neill, Ludovic Orban, Zoran Perkov, Julia Powles, Chris Riccomini, Henry Robinson, David Rosenthal, Jennifer Rullmann, Matthew Sackman, Martin Scholl, Amit Sela, Gwen Shapira, Greg Spurrier, Sam Stokes, Ben Stopford, Tom Stuart, Diana Vasile, Rahul Vohra, Pete Warden, and Brett Wooldridge. Several more people have been invaluable to the writing of this book by reviewing drafts and providing feedback. For these contributions I am particularly indebted to Raul Agepati, Tyler Akidau, Mattias Andersson, Sasha Baranov, Veena Basavaraj, David Beyer, Jim Brikman, Paul Carey, Raul Castro Fernandez, Joseph Chow, Derek Elkins, Sam Elliott, Alexander Gallego, Mark Grover, Stu Halloway, Heidi Howard, Nicola Kleppmann, Stefan Kruppa, Bjorn Madsen, Sander Mak, Stefan Podkowinski, Phil Potter, Hamid Ramazani, Sam Stokes, and Ben Summers. Of course, I take all responsibility for any remaining errors or unpalatable opinions in this book. -For helping this book become real, and for their patience with my slow writing and unusual requests, I am grateful to my editors Marie Beaugureau, Mike Loukides, Ann Spencer, and all the team at O’Reilly. For helping find the right words, I thank Rachel Head. For giving me the time and freedom to write in spite of other work commit‐ ments, I thank Alastair Beresford, Susan Goodhue, Neha Narkhede, and Kevin Scott. +For helping this book become real, and for their patience with my slow writing and unusual requests, I am grateful to my editors Marie Beaugureau, Mike Loukides, Ann Spencer, and all the team at O’Reilly. For helping find the right words, I thank Rachel Head. For giving me the time and freedom to write in spite of other work commitments, I thank Alastair Beresford, Susan Goodhue, Neha Narkhede, and Kevin Scott. -Very special thanks are due to Shabbir Diwan and Edie Freedman, who illustrated with great care the maps that accompany the chapters. It’s wonderful that they took on the unconventional idea of creating maps, and made them so beautiful and com‐ pelling. +Very special thanks are due to Shabbir Diwan and Edie Freedman, who illustrated with great care the maps that accompany the chapters. It’s wonderful that they took on the unconventional idea of creating maps, and made them so beautiful and compelling. Finally, my love goes to my family and friends, without whom I would not have been able to get through this writing process that has taken almost four years. You’re the best. \ No newline at end of file diff --git a/content/en/toc.md b/content/en/toc.md index f75e1ae..de5760b 100644 --- a/content/en/toc.md +++ b/content/en/toc.md @@ -10,17 +10,28 @@ breadcrumbs: false ![](/title.jpg) + ## [Preface](/en/preface) +- [Who Should Read This Book?](/en/preface#who-should-read-this-book) +- [Scope of This Book](/en/preface#scope-of-this-book) +- [Outline of This Book](/en/preface#outline-of-this-book) +- [References and Further Reading](/en/preface#references-and-further-reading) +- [O'Reilly Safari](/en/preface#oreilly-safari) +- [How to Contact Us](/en/preface#how-to-contact-us) +- [Acknowledgments](/en/preface#acknowledgments) ## [1. Trade-offs in Data Systems Architecture](/en/ch1) - - [Analytical versus Operational Systems](/en/ch1#sec_introduction_analytics) - [Characterizing Transaction Processing and Analytics](/en/ch1#sec_introduction_oltp) - [Data Warehousing](/en/ch1#sec_introduction_dwh) + - [From data warehouse to data lake](/en/ch1#from-data-warehouse-to-data-lake) + - [Beyond the data lake](/en/ch1#beyond-the-data-lake) - [Systems of Record and Derived Data](/en/ch1#sec_introduction_derived) - [Cloud versus Self-Hosting](/en/ch1#sec_introduction_cloud) - [Pros and Cons of Cloud Services](/en/ch1#sec_introduction_cloud_tradeoffs) - [Cloud-Native System Architecture](/en/ch1#sec_introduction_cloud_native) + - [Layering of cloud services](/en/ch1#layering-of-cloud-services) + - [Separation of storage and compute](/en/ch1#sec_introduction_storage_compute) - [Operations in the Cloud Era](/en/ch1#sec_introduction_operations) - [Distributed versus Single-Node Systems](/en/ch1#sec_introduction_distributed) - [Problems with Distributed Systems](/en/ch1#sec_introduction_dist_sys_problems) @@ -28,10 +39,9 @@ breadcrumbs: false - [Cloud Computing versus Supercomputing](/en/ch1#id17) - [Data Systems, Law, and Society](/en/ch1#sec_introduction_compliance) - [Summary](/en/ch1#summary) - + - [References](/en/ch1#references) ## [2. Defining Nonfunctional Requirements](/en/ch2) - - [Case Study: Social Network Home Timelines](/en/ch2#sec_introduction_twitter) - [Representing Users, Posts, and Follows](/en/ch2#id20) - [Materializing and Updating Timelines](/en/ch2#sec_introduction_materializing) @@ -42,6 +52,8 @@ breadcrumbs: false - [Reliability and Fault Tolerance](/en/ch2#sec_introduction_reliability) - [Fault Tolerance](/en/ch2#id27) - [Hardware and Software Faults](/en/ch2#sec_introduction_hardware_faults) + - [Tolerating hardware faults through redundancy](/en/ch2#tolerating-hardware-faults-through-redundancy) + - [Software faults](/en/ch2#software-faults) - [Humans and Reliability](/en/ch2#id31) - [Scalability](/en/ch2#sec_introduction_scalability) - [Describing Load](/en/ch2#id33) @@ -52,70 +64,111 @@ breadcrumbs: false - [Simplicity: Managing Complexity](/en/ch2#id38) - [Evolvability: Making Change Easy](/en/ch2#sec_introduction_evolvability) - [Summary](/en/ch2#summary) - + - [References](/en/ch2#references) ## [3. Data Models and Query Languages](/en/ch3) - - [Relational Model versus Document Model](/en/ch3#sec_datamodels_history) - [The Object-Relational Mismatch](/en/ch3#sec_datamodels_document) + - [Object-relational mapping (ORM)](/en/ch3#object-relational-mapping-orm) + - [The document data model for one-to-many relationships](/en/ch3#the-document-data-model-for-one-to-many-relationships) - [Normalization, Denormalization, and Joins](/en/ch3#sec_datamodels_normalization) + - [Trade-offs of normalization](/en/ch3#trade-offs-of-normalization) + - [Denormalization in the social networking case study](/en/ch3#denormalization-in-the-social-networking-case-study) - [Many-to-One and Many-to-Many Relationships](/en/ch3#sec_datamodels_many_to_many) - [Stars and Snowflakes: Schemas for Analytics](/en/ch3#sec_datamodels_analytics) - [When to Use Which Model](/en/ch3#sec_datamodels_document_summary) + - [Schema flexibility in the document model](/en/ch3#sec_datamodels_schema_flexibility) + - [Data locality for reads and writes](/en/ch3#sec_datamodels_document_locality) + - [Query languages for documents](/en/ch3#query-languages-for-documents) + - [Convergence of document and relational databases](/en/ch3#convergence-of-document-and-relational-databases) - [Graph-Like Data Models](/en/ch3#sec_datamodels_graph) - [Property Graphs](/en/ch3#id56) - [The Cypher Query Language](/en/ch3#id57) - [Graph Queries in SQL](/en/ch3#id58) - [Triple-Stores and SPARQL](/en/ch3#id59) + - [The RDF data model](/en/ch3#the-rdf-data-model) + - [The SPARQL query language](/en/ch3#the-sparql-query-language) - [Datalog: Recursive Relational Queries](/en/ch3#id62) - [GraphQL](/en/ch3#id63) - [Event Sourcing and CQRS](/en/ch3#sec_datamodels_events) - [Dataframes, Matrices, and Arrays](/en/ch3#sec_datamodels_dataframes) - [Summary](/en/ch3#summary) - + - [References](/en/ch3#references) ## [4. Storage and Retrieval](/en/ch4) - - [Storage and Indexing for OLTP](/en/ch4#sec_storage_oltp) - [Log-Structured Storage](/en/ch4#sec_storage_log_structured) + - [The SSTable file format](/en/ch4#the-sstable-file-format) + - [Constructing and merging SSTables](/en/ch4#constructing-and-merging-sstables) + - [Bloom filters](/en/ch4#bloom-filters) + - [Compaction strategies](/en/ch4#sec_storage_lsm_compaction) - [B-Trees](/en/ch4#sec_storage_b_trees) + - [Making B-trees reliable](/en/ch4#sec_storage_btree_wal) + - [B-tree variants](/en/ch4#b-tree-variants) - [Comparing B-Trees and LSM-Trees](/en/ch4#sec_storage_btree_lsm_comparison) + - [Read performance](/en/ch4#read-performance) + - [Sequential vs. random writes](/en/ch4#sidebar_sequential) + - [Write amplification](/en/ch4#write-amplification) + - [Disk space usage](/en/ch4#disk-space-usage) - [Multi-Column and Secondary Indexes](/en/ch4#sec_storage_index_multicolumn) + - [Storing values within the index](/en/ch4#sec_storage_index_heap) - [Keeping everything in memory](/en/ch4#sec_storage_inmemory) - [Data Storage for Analytics](/en/ch4#sec_storage_analytics) - [Cloud Data Warehouses](/en/ch4#sec_cloud_data_warehouses) - [Column-Oriented Storage](/en/ch4#sec_storage_column) + - [Column Compression](/en/ch4#sec_storage_column_compression) + - [Sort Order in Column Storage](/en/ch4#sort-order-in-column-storage) + - [Writing to Column-Oriented Storage](/en/ch4#writing-to-column-oriented-storage) - [Query Execution: Compilation and Vectorization](/en/ch4#sec_storage_vectorized) - [Materialized Views and Data Cubes](/en/ch4#sec_storage_materialized_views) - [Multidimensional and Full-Text Indexes](/en/ch4#sec_storage_multidimensional) - [Full-Text Search](/en/ch4#sec_storage_full_text) - [Vector Embeddings](/en/ch4#id92) - [Summary](/en/ch4#summary) - + - [References](/en/ch4#references) ## [5. Encoding and Evolution](/en/ch5) - - [Formats for Encoding Data](/en/ch5#sec_encoding_formats) - [Language-Specific Formats](/en/ch5#id96) - [JSON, XML, and Binary Variants](/en/ch5#sec_encoding_json) + - [JSON Schema](/en/ch5#json-schema) + - [Binary encoding](/en/ch5#binary-encoding) - [Protocol Buffers](/en/ch5#sec_encoding_protobuf) + - [Field tags and schema evolution](/en/ch5#field-tags-and-schema-evolution) - [Avro](/en/ch5#sec_encoding_avro) + - [The writer’s schema and the reader’s schema](/en/ch5#the-writers-schema-and-the-readers-schema) + - [Schema evolution rules](/en/ch5#schema-evolution-rules) + - [But what is the writer’s schema?](/en/ch5#but-what-is-the-writers-schema) + - [Dynamically generated schemas](/en/ch5#dynamically-generated-schemas) - [The Merits of Schemas](/en/ch5#sec_encoding_schemas) - [Modes of Dataflow](/en/ch5#sec_encoding_dataflow) - [Dataflow Through Databases](/en/ch5#sec_encoding_dataflow_db) + - [Different values written at different times](/en/ch5#different-values-written-at-different-times) + - [Archival storage](/en/ch5#archival-storage) - [Dataflow Through Services: REST and RPC](/en/ch5#sec_encoding_dataflow_rpc) + - [Web services](/en/ch5#sec_web_services) + - [The problems with remote procedure calls (RPCs)](/en/ch5#sec_problems_with_rpc) + - [Load balancers, service discovery, and service meshes](/en/ch5#sec_encoding_service_discovery) + - [Data encoding and evolution for RPC](/en/ch5#data-encoding-and-evolution-for-rpc) - [Durable Execution and Workflows](/en/ch5#sec_encoding_dataflow_workflows) + - [Durable execution](/en/ch5#durable-execution) - [Event-Driven Architectures](/en/ch5#sec_encoding_dataflow_msg) + - [Message brokers](/en/ch5#message-brokers) + - [Distributed actor frameworks](/en/ch5#distributed-actor-frameworks) - [Summary](/en/ch5#summary) - + - [References](/en/ch5#references) ## [6. Replication](/en/ch6) - - [Single-Leader Replication](/en/ch6#sec_replication_leader) - [Synchronous Versus Asynchronous Replication](/en/ch6#sec_replication_sync_async) - [Setting Up New Followers](/en/ch6#sec_replication_new_replica) - [Handling Node Outages](/en/ch6#sec_replication_failover) + - [Follower failure: Catch-up recovery](/en/ch6#follower-failure-catch-up-recovery) + - [Leader failure: Failover](/en/ch6#leader-failure-failover) - [Implementation of Replication Logs](/en/ch6#sec_replication_implementation) + - [Statement-based replication](/en/ch6#statement-based-replication) + - [Write-ahead log (WAL) shipping](/en/ch6#write-ahead-log-wal-shipping) + - [Logical (row-based) log replication](/en/ch6#logical-row-based-log-replication) - [Problems with Replication Lag](/en/ch6#sec_replication_lag) - [Reading Your Own Writes](/en/ch6#sec_replication_ryw) - [Monotonic Reads](/en/ch6#sec_replication_monotonic_reads) @@ -123,24 +176,44 @@ breadcrumbs: false - [Solutions for Replication Lag](/en/ch6#id131) - [Multi-Leader Replication](/en/ch6#sec_replication_multi_leader) - [Geographically Distributed Operation](/en/ch6#sec_replication_multi_dc) + - [Multi-leader replication topologies](/en/ch6#sec_replication_topologies) + - [Problems with different topologies](/en/ch6#problems-with-different-topologies) - [Sync Engines and Local-First Software](/en/ch6#sec_replication_offline_clients) + - [Real-time collaboration, offline-first, and local-first apps](/en/ch6#real-time-collaboration-offline-first-and-local-first-apps) + - [Pros and cons of sync engines](/en/ch6#pros-and-cons-of-sync-engines) - [Dealing with Conflicting Writes](/en/ch6#sec_replication_write_conflicts) + - [Conflict avoidance](/en/ch6#conflict-avoidance) + - [Last write wins (discarding concurrent writes)](/en/ch6#sec_replication_lww) + - [Manual conflict resolution](/en/ch6#manual-conflict-resolution) + - [Automatic conflict resolution](/en/ch6#automatic-conflict-resolution) - [CRDTs and Operational Transformation](/en/ch6#sec_replication_crdts) + - [What is a conflict?](/en/ch6#what-is-a-conflict) - [Leaderless Replication](/en/ch6#sec_replication_leaderless) - [Writing to the Database When a Node Is Down](/en/ch6#id287) + - [Catching up on missed writes](/en/ch6#sec_replication_read_repair) + - [Quorums for reading and writing](/en/ch6#sec_replication_quorum_condition) - [Limitations of Quorum Consistency](/en/ch6#sec_replication_quorum_limitations) + - [Monitoring staleness](/en/ch6#monitoring-staleness) - [Single-Leader vs. Leaderless Replication Performance](/en/ch6#sec_replication_leaderless_perf) + - [Multi-region operation](/en/ch6#multi-region-operation) - [Detecting Concurrent Writes](/en/ch6#sec_replication_concurrent) + - [The “happens-before” relation and concurrency](/en/ch6#sec_replication_happens_before) + - [Capturing the happens-before relationship](/en/ch6#capturing-the-happens-before-relationship) + - [Version vectors](/en/ch6#version-vectors) - [Summary](/en/ch6#summary) - + - [References](/en/ch6#references) ## [7. Sharding](/en/ch7) - - [Pros and Cons of Sharding](/en/ch7#sec_sharding_reasons) - [Sharding for Multitenancy](/en/ch7#sec_sharding_multitenancy) - [Sharding of Key-Value Data](/en/ch7#sec_sharding_key_value) - [Sharding by Key Range](/en/ch7#sec_sharding_key_range) + - [Rebalancing key-range sharded data](/en/ch7#rebalancing-key-range-sharded-data) - [Sharding by Hash of Key](/en/ch7#sec_sharding_hash) + - [Hash modulo number of nodes](/en/ch7#hash-modulo-number-of-nodes) + - [Fixed number of shards](/en/ch7#fixed-number-of-shards) + - [Sharding by hash range](/en/ch7#sharding-by-hash-range) + - [Consistent hashing](/en/ch7#sec_sharding_consistent_hashing) - [Skewed Workloads and Relieving Hot Spots](/en/ch7#sec_sharding_skew) - [Operations: Automatic or Manual Rebalancing](/en/ch7#sec_sharding_operations) - [Request Routing](/en/ch7#sec_sharding_routing) @@ -148,72 +221,169 @@ breadcrumbs: false - [Local Secondary Indexes](/en/ch7#id166) - [Global Secondary Indexes](/en/ch7#id167) - [Summary](/en/ch7#summary) - + - [References](/en/ch7#references) ## [8. Transactions](/en/ch8) - - [What Exactly Is a Transaction?](/en/ch8#sec_transactions_overview) - [The Meaning of ACID](/en/ch8#sec_transactions_acid) + - [Atomicity](/en/ch8#sec_transactions_acid_atomicity) + - [Consistency](/en/ch8#sec_transactions_acid_consistency) + - [Isolation](/en/ch8#sec_transactions_acid_isolation) + - [Durability](/en/ch8#durability) - [Single-Object and Multi-Object Operations](/en/ch8#sec_transactions_multi_object) + - [Single-object writes](/en/ch8#sec_transactions_single_object) + - [The need for multi-object transactions](/en/ch8#sec_transactions_need) + - [Handling errors and aborts](/en/ch8#handling-errors-and-aborts) - [Weak Isolation Levels](/en/ch8#sec_transactions_isolation_levels) - [Read Committed](/en/ch8#sec_transactions_read_committed) + - [No dirty reads](/en/ch8#no-dirty-reads) + - [No dirty writes](/en/ch8#sec_transactions_dirty_write) + - [Implementing read committed](/en/ch8#sec_transactions_read_committed_impl) - [Snapshot Isolation and Repeatable Read](/en/ch8#sec_transactions_snapshot_isolation) + - [Multi-version concurrency control (MVCC)](/en/ch8#sec_transactions_snapshot_impl) + - [Visibility rules for observing a consistent snapshot](/en/ch8#sec_transactions_mvcc_visibility) + - [Indexes and snapshot isolation](/en/ch8#indexes-and-snapshot-isolation) + - [Snapshot isolation, repeatable read, and naming confusion](/en/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - [Preventing Lost Updates](/en/ch8#sec_transactions_lost_update) + - [Atomic write operations](/en/ch8#atomic-write-operations) + - [Explicit locking](/en/ch8#explicit-locking) + - [Automatically detecting lost updates](/en/ch8#automatically-detecting-lost-updates) + - [Conditional writes (compare-and-set)](/en/ch8#sec_transactions_compare_and_set) + - [Conflict resolution and replication](/en/ch8#conflict-resolution-and-replication) - [Write Skew and Phantoms](/en/ch8#sec_transactions_write_skew) + - [Characterizing write skew](/en/ch8#characterizing-write-skew) + - [More examples of write skew](/en/ch8#more-examples-of-write-skew) + - [Phantoms causing write skew](/en/ch8#sec_transactions_phantom) + - [Materializing conflicts](/en/ch8#materializing-conflicts) - [Serializability](/en/ch8#sec_transactions_serializability) - [Actual Serial Execution](/en/ch8#sec_transactions_serial) + - [Encapsulating transactions in stored procedures](/en/ch8#encapsulating-transactions-in-stored-procedures) + - [Pros and cons of stored procedures](/en/ch8#sec_transactions_stored_proc_tradeoffs) + - [Sharding](/en/ch8#sharding) + - [Summary of serial execution](/en/ch8#summary-of-serial-execution) - [Two-Phase Locking (2PL)](/en/ch8#sec_transactions_2pl) + - [Implementation of two-phase locking](/en/ch8#implementation-of-two-phase-locking) + - [Performance of two-phase locking](/en/ch8#performance-of-two-phase-locking) + - [Predicate locks](/en/ch8#predicate-locks) + - [Index-range locks](/en/ch8#sec_transactions_2pl_range) - [Serializable Snapshot Isolation (SSI)](/en/ch8#sec_transactions_ssi) + - [Pessimistic versus optimistic concurrency control](/en/ch8#pessimistic-versus-optimistic-concurrency-control) + - [Decisions based on an outdated premise](/en/ch8#decisions-based-on-an-outdated-premise) + - [Detecting stale MVCC reads](/en/ch8#detecting-stale-mvcc-reads) + - [Detecting writes that affect prior reads](/en/ch8#sec_detecting_writes_affect_reads) + - [Performance of serializable snapshot isolation](/en/ch8#performance-of-serializable-snapshot-isolation) - [Distributed Transactions](/en/ch8#sec_transactions_distributed) - [Two-Phase Commit (2PC)](/en/ch8#sec_transactions_2pc) + - [A system of promises](/en/ch8#a-system-of-promises) + - [Coordinator failure](/en/ch8#coordinator-failure) + - [Three-phase commit](/en/ch8#three-phase-commit) - [Distributed Transactions Across Different Systems](/en/ch8#sec_transactions_xa) + - [Exactly-once message processing](/en/ch8#sec_transactions_exactly_once) + - [XA transactions](/en/ch8#xa-transactions) + - [Holding locks while in doubt](/en/ch8#holding-locks-while-in-doubt) + - [Recovering from coordinator failure](/en/ch8#recovering-from-coordinator-failure) + - [Problems with XA transactions](/en/ch8#problems-with-xa-transactions) - [Database-internal Distributed Transactions](/en/ch8#sec_transactions_internal) + - [Exactly-once message processing revisited](/en/ch8#exactly-once-message-processing-revisited) - [Summary](/en/ch8#summary) - + - [References](/en/ch8#references) ## [9. The Trouble with Distributed Systems](/en/ch9) - - [Faults and Partial Failures](/en/ch9#sec_distributed_partial_failure) - [Unreliable Networks](/en/ch9#sec_distributed_networks) - [The Limitations of TCP](/en/ch9#sec_distributed_tcp) - [Network Faults in Practice](/en/ch9#sec_distributed_network_faults) - [Detecting Faults](/en/ch9#id307) - [Timeouts and Unbounded Delays](/en/ch9#sec_distributed_queueing) + - [Network congestion and queueing](/en/ch9#network-congestion-and-queueing) - [Synchronous Versus Asynchronous Networks](/en/ch9#sec_distributed_sync_networks) + - [Can we not simply make network delays predictable?](/en/ch9#can-we-not-simply-make-network-delays-predictable) - [Unreliable Clocks](/en/ch9#sec_distributed_clocks) - [Monotonic Versus Time-of-Day Clocks](/en/ch9#sec_distributed_monotonic_timeofday) + - [Time-of-day clocks](/en/ch9#time-of-day-clocks) + - [Monotonic clocks](/en/ch9#monotonic-clocks) - [Clock Synchronization and Accuracy](/en/ch9#sec_distributed_clock_accuracy) - [Relying on Synchronized Clocks](/en/ch9#sec_distributed_clocks_relying) + - [Timestamps for ordering events](/en/ch9#sec_distributed_lww) + - [Clock readings with a confidence interval](/en/ch9#clock-readings-with-a-confidence-interval) + - [Synchronized clocks for global snapshots](/en/ch9#sec_distributed_spanner) - [Process Pauses](/en/ch9#sec_distributed_clocks_pauses) + - [Response time guarantees](/en/ch9#sec_distributed_clocks_realtime) + - [Limiting the impact of garbage collection](/en/ch9#sec_distributed_gc_impact) - [Knowledge, Truth, and Lies](/en/ch9#sec_distributed_truth) - [The Majority Rules](/en/ch9#sec_distributed_majority) - [Distributed Locks and Leases](/en/ch9#sec_distributed_lock_fencing) + - [Fencing off zombies and delayed requests](/en/ch9#sec_distributed_fencing_tokens) + - [Fencing with multiple replicas](/en/ch9#fencing-with-multiple-replicas) - [Byzantine Faults](/en/ch9#sec_distributed_byzantine) + - [Weak forms of lying](/en/ch9#weak-forms-of-lying) - [System Model and Reality](/en/ch9#sec_distributed_system_model) + - [Defining the correctness of an algorithm](/en/ch9#defining-the-correctness-of-an-algorithm) + - [Safety and liveness](/en/ch9#sec_distributed_safety_liveness) + - [Mapping system models to the real world](/en/ch9#mapping-system-models-to-the-real-world) - [Formal Methods and Randomized Testing](/en/ch9#sec_distributed_formal) + - [Model checking and specification languages](/en/ch9#model-checking-and-specification-languages) + - [Fault injection](/en/ch9#sec_fault_injection) + - [Deterministic simulation testing](/en/ch9#deterministic-simulation-testing) - [Summary](/en/ch9#summary) - + - [References](/en/ch9#references) ## [10. Consistency and Consensus](/en/ch10) - - [Linearizability](/en/ch10#sec_consistency_linearizability) - [What Makes a System Linearizable?](/en/ch10#sec_consistency_lin_definition) - [Relying on Linearizability](/en/ch10#sec_consistency_linearizability_usage) + - [Locking and leader election](/en/ch10#locking-and-leader-election) + - [Constraints and uniqueness guarantees](/en/ch10#sec_consistency_uniqueness) + - [Cross-channel timing dependencies](/en/ch10#cross-channel-timing-dependencies) - [Implementing Linearizable Systems](/en/ch10#sec_consistency_implementing_linearizable) + - [Linearizability and quorums](/en/ch10#sec_consistency_quorum_linearizable) - [The Cost of Linearizability](/en/ch10#sec_linearizability_cost) + - [The CAP theorem](/en/ch10#the-cap-theorem) + - [Linearizability and network delays](/en/ch10#linearizability-and-network-delays) - [ID Generators and Logical Clocks](/en/ch10#sec_consistency_logical) - [Logical Clocks](/en/ch10#sec_consistency_timestamps) + - [Lamport timestamps](/en/ch10#lamport-timestamps) + - [Hybrid logical clocks](/en/ch10#hybrid-logical-clocks) + - [Lamport/hybrid logical clocks vs. vector clocks](/en/ch10#lamporthybrid-logical-clocks-vs-vector-clocks) - [Linearizable ID Generators](/en/ch10#sec_consistency_linearizable_id) + - [Implementing a linearizable ID generator](/en/ch10#implementing-a-linearizable-id-generator) + - [Enforcing constraints using logical clocks](/en/ch10#enforcing-constraints-using-logical-clocks) - [Consensus](/en/ch10#sec_consistency_consensus) - [The Many Faces of Consensus](/en/ch10#sec_consistency_faces) + - [Single-value consensus](/en/ch10#single-value-consensus) + - [Compare-and-set as consensus](/en/ch10#compare-and-set-as-consensus) + - [Shared logs as consensus](/en/ch10#sec_consistency_shared_logs) + - [Fetch-and-add as consensus](/en/ch10#fetch-and-add-as-consensus) + - [Atomic commitment as consensus](/en/ch10#atomic-commitment-as-consensus) - [Consensus in Practice](/en/ch10#sec_consistency_total_order) + - [Using shared logs](/en/ch10#sec_consistency_smr) + - [From single-leader replication to consensus](/en/ch10#from-single-leader-replication-to-consensus) + - [Subtleties of consensus](/en/ch10#subtleties-of-consensus) + - [Pros and cons of consensus](/en/ch10#pros-and-cons-of-consensus) - [Coordination Services](/en/ch10#sec_consistency_coordination) + - [Allocating work to nodes](/en/ch10#allocating-work-to-nodes) + - [Service discovery](/en/ch10#service-discovery) - [Summary](/en/ch10#summary) + - [References](/en/ch10#references) +## [11. Batch Processing](/en/ch11) +- [……](/en/ch11#) +- [Summary](/en/ch11#summary) + - [References](/en/ch11#references) + +## [12. Stream Processing](/en/ch12) +- [……](/en/ch12#) +- [Summary](/en/ch12#summary) + - [References](/en/ch12#references) + +## [13. Do the Right Thing](/en/ch13) +- [……](/en/ch13#) +- [Summary](/en/ch13#summary) + - [References](/en/ch13#references) -## [11. Batch Processing](/en/ch11) (WIP) -## [12. Stream Processing](/en/ch12) (WIP) -## [13. Doing the Right Thing](/en/ch13) (WIP) ## [Glossary](/en/glossary) -## [Colophon](/en/colophon) + +## [Colophon](/en/colophon) +- [About the Author](/en/colophon#about-the-author) +- [Colophon](/en/colophon#colophon) diff --git a/content/zh/part-i.md b/content/zh/part-i.md index ff407f1..4f54a25 100644 --- a/content/zh/part-i.md +++ b/content/zh/part-i.md @@ -10,33 +10,44 @@ breadcrumbs: false 本书前四章介绍了数据系统底层的基础概念,无论是在单台机器上运行的单点数据系统,还是分布在多台机器上的分布式数据系统都适用。 -1. [第一章](/ch1) 将介绍本书使用的术语和方法。**可靠性,可伸缩性和可维护性** ,这些词汇到底意味着什么?如何实现这些目标? -2. [第二章](/ch2) 将对几种不同的 **数据模型和查询语言** 进行比较。从程序员的角度看,这是数据库之间最明显的区别。不同的数据模型适用于不同的应用场景。 -3. [第三章](/ch3) 将深入 **存储引擎** 内部,研究数据库如何在磁盘上摆放数据。不同的存储引擎针对不同的负载进行优化,选择合适的存储引擎对系统性能有巨大影响。 -4. [第四章](/ch4) 将对几种不同的 **数据编码** 进行比较。特别研究了这些格式在应用需求经常变化、模式需要随时间演变的环境中表现如何。 +1. [第一章](/ch1) 将介绍 **数据系统架构中的利弊权衡**。我们将讨论不同类型的数据系统(例如,分析型与事务型),以及它们在云环境中的运行方式。 +2. [第二章](/ch2) 将介绍非功能性需求的定义。。**可靠性,可伸缩性和可维护性** ,这些词汇到底意味着什么?如何实现这些目标? +3. [第三章](/ch3) 将对几种不同的 **数据模型和查询语言** 进行比较。从程序员的角度看,这是数据库之间最明显的区别。不同的数据模型适用于不同的应用场景。 +4. [第四章](/ch4) 将深入 **存储引擎** 内部,研究数据库如何在磁盘上摆放数据。不同的存储引擎针对不同的负载进行优化,选择合适的存储引擎对系统性能有巨大影响。 +5. [第五章](/ch5) 将对几种不同的 **数据编码** 进行比较。特别研究了这些格式在应用需求经常变化、模式需要随时间演变的环境中表现如何。 -第二部分将专门讨论在 **分布式数据系统** 中特有的问题。 +[第二部分](/part-ii) 将专门讨论在 **分布式数据系统** 中特有的问题。 -## 索引 +## [1. 数据系统架构中的权衡](/ch1) +- [分析型与事务型系统](/ch1#sec_introduction_analytics) +- [云服务与自托管](/ch1#sec_introduction_cloud) +- [分布式与单节点系统](/ch1#sec_introduction_distributed) +- [数据系统、法律与社会](/ch1#sec_introduction_compliance) +- [总结](/ch1#summary) -* [第一章:可靠性、可伸缩性和可维护性](/ch1) - * [关于数据系统的思考](/ch1#关于数据系统的思考) - * [可靠性](/ch1#可靠性) - * [可伸缩性](/ch1#可伸缩性) - * [可维护性](/ch1#可维护性) - * [本章小结](/ch1#本章小结) -* [第二章:数据模型与查询语言](/ch2) - * [关系模型与文档模型](/ch2#关系模型与文档模型) - * [数据查询语言](/ch2#数据查询语言) - * [图数据模型](/ch2#图数据模型) - * [本章小结](/ch2#本章小结) -* [第三章:存储与检索](/ch3) - * [驱动数据库的数据结构](/ch3#驱动数据库的数据结构) - * [事务处理还是分析?](/ch3#事务处理还是分析) - * [列式存储](/ch3#列式存储) - * [本章小结](/ch3#本章小结) -* [第四章:编码与演化](/ch4) - * [编码数据的格式](/ch4#编码数据的格式) - * [数据流的类型](/ch4#数据流的类型) - * [本章小结](/ch4#本章小结) +## [2. 定义非功能性需求](/ch2) +- [案例研究:社交网络首页时间线](/ch2#sec_introduction_twitter) +- [描述性能](/ch2#sec_introduction_percentiles) +- [可靠性与容错](/ch2#sec_introduction_reliability) +- [可伸缩性](/ch2#sec_introduction_scalability) +- [可运维性](/ch2#sec_introduction_maintainability) +- [总结](/ch2#summary) + +## [3. 数据模型与查询语言](/ch3) +- [关系模型与文档模型](/ch3#sec_datamodels_history) +- [图数据模型](/ch3#sec_datamodels_graph) +- [事件溯源与 CQRS](/ch3#sec_datamodels_events) +- [数据框、矩阵与数组](/ch3#sec_datamodels_dataframes) +- [总结](/ch3#summary) + +## [4. 存储与检索](/ch4) +- [OLTP 系统的存储与索引](/ch4#sec_storage_oltp) +- [分析型数据存储](/ch4#sec_storage_analytics) +- [多维索引与全文索引](/ch4#sec_storage_multidimensional) +- [总结](/ch4#summary) + +## [5. 编码与演化](/ch5) +- [编码数据的格式](/ch5#sec_encoding_formats) +- [数据流的模式](/ch5#sec_encoding_dataflow) +- [总结](/ch5#summary) diff --git a/content/zh/part-ii.md b/content/zh/part-ii.md index 28241fa..28e0b00 100644 --- a/content/zh/part-ii.md +++ b/content/zh/part-ii.md @@ -15,106 +15,111 @@ breadcrumbs: false ------- -在本书的 [第一部分](/part-i) 中,我们讨论了数据系统的各个方面,但仅限于数据存储在单台机器上的情况。现在我们到了 [第二部分](/part-ii),进入更高的层次,并提出一个问题:如果 **多台机器** 参与数据的存储和检索,会发生什么? +在本书的 [第一部分](/part-i) 中,我们讨论了数据系统的各个方面,但仅限于数据存储在单台机器上的情况。 +现在我们到了 [第二部分](/part-ii),进入更高的层次,并提出一个问题:如果 **多台机器** 参与数据的存储和检索,会发生什么? 你可能会出于各种各样的原因,希望将数据库分布到多台机器上: -* 可伸缩性 +可伸缩性 +: 如果你的数据量、读取负载、写入负载超出单台机器的处理能力,可以将负载分散到多台计算机上。 - 如果你的数据量、读取负载、写入负载超出单台机器的处理能力,可以将负载分散到多台计算机上。 +容错 / 高可用性 +: 如果你的应用需要在单台机器(或多台机器,网络或整个数据中心)出现故障的情况下仍然能继续工作,则可使用多台机器,以提供冗余。一台故障时,另一台可以接管。 -* 容错 / 高可用性 +延迟 +: 如果在世界各地都有用户,你也许会考虑在全球范围部署多个服务器,从而每个用户可以从地理上最近的数据中心获取服务,避免了等待网络数据包穿越半个世界。 - 如果你的应用需要在单台机器(或多台机器,网络或整个数据中心)出现故障的情况下仍然能继续工作,则可使用多台机器,以提供冗余。一台故障时,另一台可以接管。 +## 伸缩至更高的负载 -* 延迟 +如果你需要的只是伸缩至更高的 **负载(load)**,最简单的方法就是购买更强大的机器(有时称为 **垂直伸缩**,即 vertical scaling,或 **向上伸缩**,即 scale up)。许多处理器,内存和磁盘可以在同一个操作系统下相互连接,快速的相互连接允许任意处理器访问内存或磁盘的任意部分。在这种 **共享内存架构(shared-memory architecture)** 中,所有的组件都可以看作一台单独的机器。 - 如果在世界各地都有用户,你也许会考虑在全球范围部署多个服务器,从而每个用户可以从地理上最近的数据中心获取服务,避免了等待网络数据包穿越半个世界。 - -## 伸缩至更高的载荷 - -如果你需要的只是伸缩至更高的 **载荷(load)**,最简单的方法就是购买更强大的机器(有时称为 **垂直伸缩**,即 vertical scaling,或 **向上伸缩**,即 scale up)。许多处理器,内存和磁盘可以在同一个操作系统下相互连接,快速的相互连接允许任意处理器访问内存或磁盘的任意部分。在这种 **共享内存架构(shared-memory architecture)** 中,所有的组件都可以看作一台单独的机器 [^i]。 - -[^i]: 在大型机中,尽管任意处理器都可以访问内存的任意部分,但总有一些内存区域与一些处理器更接近(称为 **非均匀内存访问(nonuniform memory access, NUMA)**【1】)。为了有效利用这种架构特性,需要对处理进行细分,以便每个处理器主要访问临近的内存,这意味着即使表面上看起来只有一台机器在运行,**分区(partitioning)** 仍然是必要的。 +> [!NOTE] +> 在大型机中,尽管任意处理器都可以访问内存的任意部分,但总有一些内存区域与一些处理器更接近(称为 **非均匀内存访问(nonuniform memory access, NUMA)** [^1])。为了有效利用这种架构特性,需要对处理进行细分,以便每个处理器主要访问临近的内存,这意味着即使表面上看起来只有一台机器在运行,**分区(partitioning)** 仍然是必要的。 共享内存方法的问题在于,成本增长速度快于线性增长:一台有着双倍处理器数量,双倍内存大小,双倍磁盘容量的机器,通常成本会远远超过原来的两倍。而且可能因为存在瓶颈,并不足以处理双倍的载荷。 共享内存架构可以提供有限的容错能力,高端机器可以使用热插拔的组件(不关机更换磁盘,内存模块,甚至处理器)—— 但它必然囿于单个地理位置的桎梏。 -另一种方法是 **共享磁盘架构(shared-disk architecture)**,它使用多台具有独立处理器和内存的机器,但将数据存储在机器之间共享的磁盘阵列上,这些磁盘通过快速网络连接 [^ii]。这种架构用于某些数据仓库,但竞争和锁定的开销限制了共享磁盘方法的可伸缩性【2】。 +另一种方法是 **共享磁盘架构(shared-disk architecture)**,它使用多台具有独立处理器和内存的机器,但将数据存储在机器之间共享的磁盘阵列上,这些磁盘通过快速网络连接。这种架构用于某些数据仓库,但竞争和锁定的开销限制了共享磁盘方法的可伸缩性 [^2]。 -[^ii]: 网络附属存储(Network Attached Storage, NAS),或 **存储区网络(Storage Area Network, SAN)** +> [!NOTE] +> 网络附属存储(Network Attached Storage, NAS),或 **存储区网络(Storage Area Network, SAN)** ### 无共享架构 -相比之下,**无共享架构**【3】(shared-nothing architecture,有时被称为 **水平伸缩**,即 horizontal scaling,或 **向外伸缩**,即 scaling out)已经相当普及。在这种架构中,运行数据库软件的每台机器 / 虚拟机都称为 **节点(node)**。每个节点只使用各自的处理器,内存和磁盘。节点之间的任何协调,都是在软件层面使用传统网络实现的。 +相比之下,**无共享架构** [^3](shared-nothing architecture,有时被称为 **水平伸缩**,即 horizontal scaling,或 **向外伸缩**,即 scaling out)已经相当普及。 +在这种架构中,运行数据库软件的每台机器 / 虚拟机都称为 **节点(node)**。每个节点只使用各自的处理器,内存和磁盘。节点之间的任何协调,都是在软件层面使用传统网络实现的。 -无共享系统不需要使用特殊的硬件,所以你可以用任意机器 —— 比如性价比最好的机器。你也许可以跨多个地理区域分布数据从而减少用户延迟,或者在损失一整个数据中心的情况下幸免于难。随着云端虚拟机部署的出现,即使是小公司,现在无需 Google 级别的运维,也可以实现异地分布式架构。 +无共享系统不需要使用特殊的硬件,所以你可以用任意机器 —— 比如性价比最好的机器。你也许可以跨多个地理区域分布数据从而减少用户延迟,或者在损失一整个数据中心的情况下幸免于难。 +随着云端虚拟机部署的出现,即使是小公司,现在无需 Google 级别的运维,也可以实现异地分布式架构。 -在这一部分里,我们将重点放在无共享架构上。它不见得是所有场景的最佳选择,但它是最需要你谨慎从事的架构。如果你的数据分布在多个节点上,你需要意识到这样一个分布式系统中约束和权衡 —— 数据库并不能魔术般地把这些东西隐藏起来。 +在这一部分里,我们将重点放在无共享架构上。它不见得是所有场景的最佳选择,但它是最需要你谨慎从事的架构。 +如果你的数据分布在多个节点上,你需要意识到这样一个分布式系统中约束和权衡 —— 数据库并不能魔术般地把这些东西隐藏起来。 + +虽然分布式无共享架构有许多优点,但它通常也会给应用带来额外的复杂度,有时也会限制你可用数据模型的表达力。 +在某些情况下,一个简单的单线程程序可以比一个拥有超过 100 个 CPU 核的集群表现得更好 [^4]。另一方面,无共享系统可以非常强大。接下来的几章,将详细讨论分布式数据会带来的问题。 -虽然分布式无共享架构有许多优点,但它通常也会给应用带来额外的复杂度,有时也会限制你可用数据模型的表达力。在某些情况下,一个简单的单线程程序可以比一个拥有超过 100 个 CPU 核的集群表现得更好【4】。另一方面,无共享系统可以非常强大。接下来的几章,将详细讨论分布式数据会带来的问题。 ### 复制 vs 分区 数据分布在多个节点上有两种常见的方式: -* 复制(Replication) +复制(Replication) +: 在几个不同的节点上保存数据的相同副本,可能放在不同的位置。复制提供了冗余:如果一些节点不可用,剩余的节点仍然可以提供数据服务。复制也有助于改善性能。[第六章](/ch6) 将讨论复制。 - 在几个不同的节点上保存数据的相同副本,可能放在不同的位置。复制提供了冗余:如果一些节点不可用,剩余的节点仍然可以提供数据服务。复制也有助于改善性能。[第五章](/ch5) 将讨论复制。 +分区 (Partitioning) +: 将一个大型数据库拆分成较小的子集(称为 **分区**,即 partitions),从而不同的分区可以指派给不同的 **节点**(nodes,亦称 **分片**,即 sharding)。[第七章](/ch7) 将讨论分区。 -* 分区 (Partitioning) +复制和分区是不同的机制,但它们经常同时使用。如 [图 II-1](#fig_replication_partitioning) 所示。 - 将一个大型数据库拆分成较小的子集(称为 **分区**,即 partitions),从而不同的分区可以指派给不同的 **节点**(nodes,亦称 **分片**,即 sharding)。[第六章](/ch6) 将讨论分区。 +{{< figure src="/fig/ddia_08.png" id="fig_replication_partitioning" caption="图 II-1 一个数据库切分为两个分区,每个分区都有两个副本" class="w-full my-4" >}} -复制和分区是不同的机制,但它们经常同时使用。如 [图 II-1](/img/figii-1.png) 所示。 -![](/img/figii-1.png) - -**图 II-1 一个数据库切分为两个分区,每个分区都有两个副本** - -理解了这些概念,就可以开始讨论在分布式系统中需要做出的困难抉择。[第七章](/ch7) 将讨论 **事务(Transaction)**,这对于了解数据系统中可能出现的各种问题,以及我们可以做些什么很有帮助。[第八章](/ch8) 和 [第九章](/ch9) 将讨论分布式系统的根本局限性。 +理解了这些概念,就可以开始讨论在分布式系统中需要做出的困难抉择。[第八章](/ch8) 将讨论 **事务(Transaction)**,这对于了解数据系统中可能出现的各种问题,以及我们可以做些什么很有帮助。 +[第九章](/ch9) 和 [第十章](/ch10) 将讨论分布式系统的根本局限性。 在本书的 [第三部分](/part-iii) 中,将讨论如何将多个(可能是分布式的)数据存储集成为一个更大的系统,以满足复杂的应用需求。但首先,我们来聊聊分布式的数据。 -## 索引 +## [6. 复制](/ch6) +- [单主复制](/ch6#sec_replication_leader) +- [复制延迟的问题](/ch6#sec_replication_lag) +- [多主复制](/ch6#sec_replication_multi_leader) +- [无主复制](/ch6#sec_replication_leaderless) +- [总结](/ch6#summary) -* [第五章:复制](/ch5) - * [领导者与追随者](/ch5#领导者与追随者) - * [复制延迟问题](/ch5#复制延迟问题) - * [多主复制](/ch5#多主复制) - * [无主复制](/ch5#无主复制) - * [本章小结](/ch5#本章小结) -* [第六章:分区](/ch6) - * [分区与复制](/ch6#分区与复制) - * [键值数据的分区](/ch6#键值数据的分区) - * [分区与次级索引](/ch6#分区与次级索引) - * [分区再平衡](/ch6#分区再平衡) - * [请求路由](/ch6#请求路由) - * [本章小结](/ch6#本章小结) -* [第七章:事务](/ch7) - * [事务的棘手概念](/ch7#事务的棘手概念) - * [弱隔离级别](/ch7#弱隔离级别) - * [可串行化](/ch7#可串行化) - * [本章小结](/ch7#本章小结) -* [第八章:分布式系统的麻烦](/ch8) - * [故障与部分失效](/ch8#故障与部分失效) - * [不可靠的网络](/ch8#不可靠的网络) - * [不可靠的时钟](/ch8#不可靠的时钟) - * [知识、真相与谎言](/ch8#知识真相与谎言) - * [本章小结](/ch8#本章小结) -* [第九章:一致性与共识](/ch9) - * [一致性保证](/ch9#一致性保证) - * [线性一致性](/ch9#线性一致性) - * [顺序保证](/ch9#顺序保证) - * [分布式事务与共识](/ch9#分布式事务与共识) - * [本章小结](/ch9#本章小结) +## [7. 分片](/ch7) +- [分片的利与弊](/ch7#sec_sharding_reasons) +- [键值数据的分片](/ch7#sec_sharding_key_value) +- [请求路由](/ch7#sec_sharding_routing) +- [分片与二级索引](/ch7#sec_sharding_secondary_indexes) +- [总结](/ch7#summary) + +## [8. 事务](/ch8) +- [事务到底是什么?](/ch8#sec_transactions_overview) +- [弱隔离级别](/ch8#sec_transactions_isolation_levels) +- [可串行化](/ch8#sec_transactions_serializability) +- [分布式事务](/ch8#sec_transactions_distributed) +- [总结](/ch8#summary) +- [参考](/ch8#参考) + +## [9. 分布式系统的麻烦](/ch9) +- [故障与部分失效](/ch9#sec_distributed_partial_failure) +- [不可靠的网络](/ch9#sec_distributed_networks) +- [不可靠的时钟](/ch9#sec_distributed_clocks) +- [知识、真相和谎言](/ch9#sec_distributed_truth) +- [总结](/ch9#summary) + +## [10. 一致性与共识](/ch10) +- [线性一致性](/ch10#sec_consistency_linearizability) +- [ID 生成器和逻辑时钟](/ch10#sec_consistency_logical) +- [共识](/ch10#sec_consistency_consensus) +- [总结](/ch10#summary) -## 参考文献 +### 参考 -1. Ulrich Drepper: “[What Every Programmer Should Know About Memory](https://people.freebsd.org/~lstewart/articles/cpumemory.pdf),” akka‐dia.org, November 21, 2007. -1. Ben Stopford: “[Shared Nothing vs. Shared Disk Architectures: An Independent View](http://www.benstopford.com/2009/11/24/understanding-the-shared-nothing-architecture/),” benstopford.com, November 24, 2009. -1. Michael Stonebraker: “[The Case for Shared Nothing](http://db.cs.berkeley.edu/papers/hpts85-nothing.pdf),” IEEE Database EngineeringBulletin, volume 9, number 1, pages 4–9, March 1986. -1. Frank McSherry, Michael Isard, and Derek G. Murray: “[Scalability! But at What COST?](http://www.frankmcsherry.org/assets/COST.pdf),” at 15th USENIX Workshop on Hot Topics in Operating Systems (HotOS),May 2015. \ No newline at end of file +[^1]: Ulrich Drepper: “[What Every Programmer Should Know About Memory](https://people.freebsd.org/~lstewart/articles/cpumemory.pdf),” akka‐dia.org, November 21, 2007. +[^2]: Ben Stopford: “[Shared Nothing vs. Shared Disk Architectures: An Independent View](http://www.benstopford.com/2009/11/24/understanding-the-shared-nothing-architecture/),” benstopford.com, November 24, 2009. +[^3]: Michael Stonebraker: “[The Case for Shared Nothing](http://db.cs.berkeley.edu/papers/hpts85-nothing.pdf),” IEEE Database EngineeringBulletin, volume 9, number 1, pages 4–9, March 1986. +[^4]: Frank McSherry, Michael Isard, and Derek G. Murray: “[Scalability! But at What COST?](http://www.frankmcsherry.org/assets/COST.pdf),” at 15th USENIX Workshop on Hot Topics in Operating Systems (HotOS),May 2015. diff --git a/content/zh/part-iii.md b/content/zh/part-iii.md index d5e9cdd..d9bd0ff 100644 --- a/content/zh/part-iii.md +++ b/content/zh/part-iii.md @@ -18,13 +18,13 @@ breadcrumbs: false 从高层次上看,存储和处理数据的系统可以分为两大类: -* 记录系统(System of record) +权威记录系统(System of record) +: **记录系统**,也被称为 **真相源(source of truth)**,持有数据的权威版本。当新的数据进入时(例如,用户输入)首先会记录在这里。 + 每个事实正正好好表示一次(表示通常是 **正规化的**,即 normalized)。如果其他系统和 **记录系统** 之间存在任何差异,那么记录系统中的值是正确的(根据定义)。 - **记录系统**,也被称为 **真相源(source of truth)**,持有数据的权威版本。当新的数据进入时(例如,用户输入)首先会记录在这里。每个事实正正好好表示一次(表示通常是 **正规化的**,即 normalized)。如果其他系统和 **记录系统** 之间存在任何差异,那么记录系统中的值是正确的(根据定义)。 - -* 派生数据系统(Derived data systems) - - **派生系统** 中的数据,通常是另一个系统中的现有数据以某种方式进行转换或处理的结果。如果丢失派生数据,可以从原始来源重新创建。典型的例子是 **缓存(cache)**:如果数据在缓存中,就可以由缓存提供服务;如果缓存不包含所需数据,则降级由底层数据库提供。非规范化的值,索引和物化视图亦属此类。在推荐系统中,预测汇总数据通常衍生自用户日志。 +派生数据系统(Derived data systems) +: **派生系统** 中的数据,通常是另一个系统中的现有数据以某种方式进行转换或处理的结果。如果丢失派生数据,可以从原始来源重新创建。 + 典型的例子是 **缓存(cache)**:如果数据在缓存中,就可以由缓存提供服务;如果缓存不包含所需数据,则降级由底层数据库提供。非规范化的值,索引和物化视图亦属此类。在推荐系统中,预测汇总数据通常衍生自用户日志。 从技术上讲,派生数据是 **冗余的(redundant)**,因为它重复了已有的信息。但是派生数据对于获得良好的只读查询性能通常是至关重要的。它通常是非规范化的。可以从单个源头衍生出多个不同的数据集,使你能从不同的 “视角” 洞察数据。 @@ -39,21 +39,25 @@ breadcrumbs: false 我们将从 [第十一章](/ch11) 开始,研究例如 MapReduce 这样 **面向批处理(batch-oriented)** 的数据流系统。对于建设大规模数据系统,我们将看到,它们提供了优秀的工具和思想。 [第十二章](/ch12) 将把这些思想应用到 **流式数据(data streams)** 中,使我们能用更低的延迟完成同样的任务。[第十三章](/ch13) 将对本书进行总结,探讨如何使用这些工具来构建可靠,可伸缩和可维护的应用。 -## 索引 -* [第十章:批处理](/ch11) - * [使用Unix工具的批处理](/ch11#使用Unix工具的批处理) - * [MapReduce和分布式文件系统](/ch11#MapReduce和分布式文件系统) - * [MapReduce之后](/ch11#MapReduce之后) - * [本章小结](/ch11#本章小结) -* [第十一章:流处理](/ch12) - * [传递事件流](/ch12#传递事件流) - * [数据库与流](/ch12#数据库与流) - * [流处理](/ch12#流处理) - * [本章小结](/ch12#本章小结) -* [第十三章:做正确的事](/ch13) - * [数据集成](/ch13#数据集成) - * [分拆数据库](/ch13#分拆数据库) - * [将事情做正确](/ch13#将事情做正确) - * [做正确的事情](/ch13#做正确的事情) - * [本章小结](/ch13#本章小结) +## [第十一章:批处理](/ch11) +- [使用Unix工具的批处理](/ch11#使用unix工具的批处理) +- [MapReduce和分布式文件系统](/ch11#mapreduce和分布式文件系统) +- [MapReduce之后](/ch11#mapreduce之后) +- [本章小结](/ch11#本章小结) +- [参考文献](/ch11#参考文献) + +## [第十二章:流处理](/ch12) +- [传递事件流](/ch12#传递事件流) +- [数据库与流](/ch12#数据库与流) +- [流处理](/ch12#流处理) +- [本章小结](/ch12#本章小结) +- [参考文献](/ch12#参考文献) + +## [第十三章:数据系统的未来](/ch13) +- [数据集成](/ch13#数据集成) +- [分拆数据库](/ch13#分拆数据库) +- [将事情做正确](/ch13#将事情做正确) +- [做正确的事情](/ch13#做正确的事情) +- [本章小结](/ch13#本章小结) +- [参考文献](/ch13#参考文献) diff --git a/content/zh/toc.md b/content/zh/toc.md index f004c60..3a13fc3 100644 --- a/content/zh/toc.md +++ b/content/zh/toc.md @@ -11,16 +11,25 @@ breadcrumbs: false ## [序言](/preface) +- [本书的目标读者](/preface#本书的目标读者) +- [本书涉及的领域](/preface#本书涉及的领域) +- [本书纲要](/preface#本书纲要) +- [参考文献与延伸阅读](/preface#参考文献与延伸阅读) +- [O‘Reilly Safari](/preface#oreilly-safari) +- [致谢](/preface#致谢) ## [1. 数据系统架构中的权衡](/ch1) - - [分析型与事务型系统](/ch1#sec_introduction_analytics) - [事务处理与分析的特征](/ch1#sec_introduction_oltp) - [数据仓库](/ch1#sec_introduction_dwh) + - [从数据仓库到数据湖](/ch1#from-data-warehouse-to-data-lake) + - [超越数据湖](/ch1#beyond-the-data-lake) - [权威数据源与派生数据](/ch1#sec_introduction_derived) - [云服务与自托管](/ch1#sec_introduction_cloud) - [云服务的利弊](/ch1#sec_introduction_cloud_tradeoffs) - [云原生系统架构](/ch1#sec_introduction_cloud_native) + - [云服务的分层](/ch1#layering-of-cloud-services) + - [存储与计算的分离](/ch1#sec_introduction_storage_compute) - [云时代的运维](/ch1#sec_introduction_operations) - [分布式与单节点系统](/ch1#sec_introduction_distributed) - [分布式系统的问题](/ch1#sec_introduction_dist_sys_problems) @@ -28,10 +37,9 @@ breadcrumbs: false - [云计算与超级计算](/ch1#id17) - [数据系统、法律与社会](/ch1#sec_introduction_compliance) - [总结](/ch1#summary) - + - [参考](/ch1#参考) ## [2. 定义非功能性需求](/ch2) - - [案例研究:社交网络首页时间线](/ch2#sec_introduction_twitter) - [表示用户、帖子与关注关系](/ch2#id20) - [时间线的物化与更新](/ch2#sec_introduction_materializing) @@ -42,80 +50,123 @@ breadcrumbs: false - [可靠性与容错](/ch2#sec_introduction_reliability) - [容错](/ch2#id27) - [硬件与软件故障](/ch2#sec_introduction_hardware_faults) + - [通过冗余容忍硬件故障](/ch2#tolerating-hardware-faults-through-redundancy) + - [软件故障](/ch2#software-faults) - [人类与可靠性](/ch2#id31) - [可伸缩性](/ch2#sec_introduction_scalability) - [描述负载](/ch2#id33) - [共享内存、共享磁盘与无共享架构](/ch2#sec_introduction_shared_nothing) - [可伸缩性原则](/ch2#id35) -- [可维护性](/ch2#sec_introduction_maintainability) - - [可操作性:让运维更轻松](/ch2#id37) +- [可运维性](/ch2#sec_introduction_maintainability) + - [可运维性:让运维更轻松](/ch2#id37) - [简单性:管理复杂度](/ch2#id38) - [可演化性:让变化更容易](/ch2#sec_introduction_evolvability) - [总结](/ch2#summary) - + - [参考](/ch2#参考) ## [3. 数据模型与查询语言](/ch3) - - [关系模型与文档模型](/ch3#sec_datamodels_history) - [对象关系不匹配](/ch3#sec_datamodels_document) + - [对象关系映射(ORM)](/ch3#object-relational-mapping-orm) + - [用于一对多关系的文档数据模型](/ch3#the-document-data-model-for-one-to-many-relationships) - [规范化、反规范化与连接](/ch3#sec_datamodels_normalization) + - [规范化的权衡](/ch3#trade-offs-of-normalization) + - [社交网络案例研究中的反规范化](/ch3#denormalization-in-the-social-networking-case-study) - [多对一与多对多关系](/ch3#sec_datamodels_many_to_many) - [星型与雪花型:分析模式](/ch3#sec_datamodels_analytics) - [何时使用哪种模型](/ch3#sec_datamodels_document_summary) + - [文档模型中的模式灵活性](/ch3#sec_datamodels_schema_flexibility) + - [读写的数据局部性](/ch3#sec_datamodels_document_locality) + - [文档的查询语言](/ch3#query-languages-for-documents) + - [文档和关系数据库的融合](/ch3#convergence-of-document-and-relational-databases) - [图数据模型](/ch3#sec_datamodels_graph) - [属性图](/ch3#id56) - [Cypher 查询语言](/ch3#id57) - [SQL 中的图查询](/ch3#id58) - [三元组存储与 SPARQL](/ch3#id59) + - [RDF 数据模型](/ch3#the-rdf-data-model) + - [SPARQL 查询语言](/ch3#the-sparql-query-language) - [Datalog:递归关系查询](/ch3#id62) - [GraphQL](/ch3#id63) - [事件溯源与 CQRS](/ch3#sec_datamodels_events) - [数据框、矩阵与数组](/ch3#sec_datamodels_dataframes) - [总结](/ch3#summary) - + - [参考](/ch3#参考) ## [4. 存储与检索](/ch4) - - [OLTP 系统的存储与索引](/ch4#sec_storage_oltp) - [日志结构存储](/ch4#sec_storage_log_structured) + - [SSTable 文件格式](/ch4#the-sstable-file-format) + - [构建和合并 SSTable](/ch4#constructing-and-merging-sstables) + - [布隆过滤器](/ch4#bloom-filters) + - [压实策略](/ch4#sec_storage_lsm_compaction) - [B 树](/ch4#sec_storage_b_trees) + - [使 B 树可靠](/ch4#sec_storage_btree_wal) + - [B 树变体](/ch4#b-tree-variants) - [比较 B 树与 LSM 树](/ch4#sec_storage_btree_lsm_comparison) + - [读取性能](/ch4#read-performance) + - [顺序与随机写入](/ch4#sidebar_sequential) + - [写放大](/ch4#write-amplification) + - [磁盘空间使用](/ch4#disk-space-usage) - [多列索引与二级索引](/ch4#sec_storage_index_multicolumn) + - [在索引中存储值](/ch4#sec_storage_index_heap) - [全内存存储](/ch4#sec_storage_inmemory) - [分析型数据存储](/ch4#sec_storage_analytics) - [云数据仓库](/ch4#sec_cloud_data_warehouses) - [列式存储](/ch4#sec_storage_column) + - [列压缩](/ch4#sec_storage_column_compression) + - [列存储中的排序顺序](/ch4#sort-order-in-column-storage) + - [写入列式存储](/ch4#writing-to-column-oriented-storage) - [查询执行:编译与向量化](/ch4#sec_storage_vectorized) - [物化视图与多维数据集](/ch4#sec_storage_materialized_views) - [多维索引与全文索引](/ch4#sec_storage_multidimensional) - [全文检索](/ch4#sec_storage_full_text) - [向量嵌入](/ch4#id92) - [总结](/ch4#summary) - + - [参考](/ch4#参考) ## [5. 编码与演化](/ch5) - - [编码数据的格式](/ch5#sec_encoding_formats) - [特定语言的格式](/ch5#id96) - [JSON、XML 及其二进制变体](/ch5#sec_encoding_json) + - [JSON 模式](/ch5#json-schema) + - [二进制编码](/ch5#binary-encoding) - [Protocol Buffers](/ch5#sec_encoding_protobuf) + - [字段标签与模式演化](/ch5#field-tags-and-schema-evolution) - [Avro](/ch5#sec_encoding_avro) + - [写入者模式与读取者模式](/ch5#the-writers-schema-and-the-readers-schema) + - [模式演化规则](/ch5#schema-evolution-rules) + - [但什么是写入者模式?](/ch5#but-what-is-the-writers-schema) + - [动态生成的模式](/ch5#dynamically-generated-schemas) - [模式的优点](/ch5#sec_encoding_schemas) - [数据流的模式](/ch5#sec_encoding_dataflow) - [流经数据库的数据流](/ch5#sec_encoding_dataflow_db) + - [不同时间写入的不同值](/ch5#different-values-written-at-different-times) + - [归档存储](/ch5#archival-storage) - [流经服务的数据流:REST 与 RPC](/ch5#sec_encoding_dataflow_rpc) + - [Web 服务](/ch5#sec_web_services) + - [远程过程调用(RPC)的问题](/ch5#sec_problems_with_rpc) + - [负载均衡器、服务发现和服务网格](/ch5#sec_encoding_service_discovery) + - [RPC 的数据编码与演化](/ch5#data-encoding-and-evolution-for-rpc) - [持久化执行与工作流](/ch5#sec_encoding_dataflow_workflows) + - [持久化执行](/ch5#durable-execution) - [事件驱动的架构](/ch5#sec_encoding_dataflow_msg) + - [消息代理](/ch5#message-brokers) + - [分布式 actor 框架](/ch5#distributed-actor-frameworks) - [总结](/ch5#summary) - + - [参考](/ch5#参考) ## [6. 复制](/ch6) - - [单主复制](/ch6#sec_replication_leader) - [同步复制与异步复制](/ch6#sec_replication_sync_async) - [设置新的副本](/ch6#sec_replication_new_replica) - [处理节点故障](/ch6#sec_replication_failover) + - [从节点故障:追赶恢复](/ch6#follower-failure-catch-up-recovery) + - [领导者故障:故障转移](/ch6#leader-failure-failover) - [复制日志的实现](/ch6#sec_replication_implementation) + - [基于语句的复制](/ch6#statement-based-replication) + - [预写日志(WAL)传输](/ch6#write-ahead-log-wal-shipping) + - [逻辑(基于行)日志复制](/ch6#logical-row-based-log-replication) - [复制延迟的问题](/ch6#sec_replication_lag) - [读己之写](/ch6#sec_replication_ryw) - [单调读](/ch6#sec_replication_monotonic_reads) @@ -123,24 +174,44 @@ breadcrumbs: false - [复制延迟的解决方案](/ch6#id131) - [多主复制](/ch6#sec_replication_multi_leader) - [跨地域运行](/ch6#sec_replication_multi_dc) + - [多主复制拓扑](/ch6#sec_replication_topologies) + - [不同拓扑的问题](/ch6#problems-with-different-topologies) - [同步引擎与本地优先软件](/ch6#sec_replication_offline_clients) + - [实时协作、离线优先和本地优先应用](/ch6#real-time-collaboration-offline-first-and-local-first-apps) + - [同步引擎的利弊](/ch6#pros-and-cons-of-sync-engines) - [处理写入冲突](/ch6#sec_replication_write_conflicts) + - [冲突避免](/ch6#conflict-avoidance) + - [最后写入者胜(丢弃并发写入)](/ch6#sec_replication_lww) + - [手动冲突解决](/ch6#manual-conflict-resolution) + - [自动冲突解决](/ch6#automatic-conflict-resolution) - [CRDT 与操作变换](/ch6#sec_replication_crdts) + - [什么是冲突?](/ch6#what-is-a-conflict) - [无主复制](/ch6#sec_replication_leaderless) - [当节点故障时写入数据库](/ch6#id287) + - [追赶错过的写入](/ch6#sec_replication_read_repair) + - [读写仲裁](/ch6#sec_replication_quorum_condition) - [仲裁一致性的局限](/ch6#sec_replication_quorum_limitations) + - [监控陈旧性](/ch6#monitoring-staleness) - [单主与无主复制的性能](/ch6#sec_replication_leaderless_perf) + - [多地区操作](/ch6#multi-region-operation) - [检测并发写入](/ch6#sec_replication_concurrent) + - ["先发生"关系与并发](/ch6#sec_replication_happens_before) + - [捕获先发生关系](/ch6#capturing-the-happens-before-relationship) + - [版本向量](/ch6#version-vectors) - [总结](/ch6#summary) - + - [参考](/ch6#参考) ## [7. 分片](/ch7) - - [分片的利与弊](/ch7#sec_sharding_reasons) - [面向多租户的分片](/ch7#sec_sharding_multitenancy) - [键值数据的分片](/ch7#sec_sharding_key_value) - [按键的范围分片](/ch7#sec_sharding_key_range) + - [重新平衡键范围分片数据](/ch7#rebalancing-key-range-sharded-data) - [按键的哈希分片](/ch7#sec_sharding_hash) + - [哈希取模节点数](/ch7#hash-modulo-number-of-nodes) + - [固定数量的分片](/ch7#fixed-number-of-shards) + - [按哈希范围分片](/ch7#sharding-by-hash-range) + - [一致性哈希](/ch7#sec_sharding_consistent_hashing) - [倾斜的工作负载与缓解热点](/ch7#sec_sharding_skew) - [运维:自动/手动再均衡](/ch7#sec_sharding_operations) - [请求路由](/ch7#sec_sharding_routing) @@ -148,71 +219,321 @@ breadcrumbs: false - [本地二级索引](/ch7#id166) - [全局二级索引](/ch7#id167) - [总结](/ch7#summary) - + - [References](/ch7#references) ## [8. 事务](/ch8) - - [事务到底是什么?](/ch8#sec_transactions_overview) - [ACID 的含义](/ch8#sec_transactions_acid) + - [原子性](/ch8#sec_transactions_acid_atomicity) + - [一致性](/ch8#sec_transactions_acid_consistency) + - [隔离性](/ch8#sec_transactions_acid_isolation) + - [持久性](/ch8#durability) - [单对象与多对象操作](/ch8#sec_transactions_multi_object) + - [单对象写入](/ch8#sec_transactions_single_object) + - [多对象事务的需求](/ch8#sec_transactions_need) + - [处理错误和中止](/ch8#handling-errors-and-aborts) - [弱隔离级别](/ch8#sec_transactions_isolation_levels) - [读已提交](/ch8#sec_transactions_read_committed) + - [没有脏读](/ch8#no-dirty-reads) + - [没有脏写](/ch8#sec_transactions_dirty_write) + - [实现读已提交](/ch8#sec_transactions_read_committed_impl) - [快照隔离与可重复读](/ch8#sec_transactions_snapshot_isolation) + - [多版本并发控制(MVCC)](/ch8#sec_transactions_snapshot_impl) + - [观察一致快照的可见性规则](/ch8#sec_transactions_mvcc_visibility) + - [索引与快照隔离](/ch8#indexes-and-snapshot-isolation) + - [快照隔离、可重复读和命名混淆](/ch8#snapshot-isolation-repeatable-read-and-naming-confusion) - [防止丢失更新](/ch8#sec_transactions_lost_update) + - [原子写操作](/ch8#atomic-write-operations) + - [显式锁定](/ch8#explicit-locking) + - [自动检测丢失的更新](/ch8#automatically-detecting-lost-updates) + - [条件写入(比较并设置)](/ch8#sec_transactions_compare_and_set) + - [冲突解决与复制](/ch8#conflict-resolution-and-replication) - [写偏斜与幻读](/ch8#sec_transactions_write_skew) + - [描述写偏斜](/ch8#characterizing-write-skew) + - [更多写偏斜的例子](/ch8#more-examples-of-write-skew) + - [导致写偏斜的幻读](/ch8#sec_transactions_phantom) + - [物化冲突](/ch8#materializing-conflicts) - [可串行化](/ch8#sec_transactions_serializability) - [实际串行执行](/ch8#sec_transactions_serial) + - [将事务封装在存储过程中](/ch8#encapsulating-transactions-in-stored-procedures) + - [存储过程的利弊](/ch8#sec_transactions_stored_proc_tradeoffs) + - [分片](/ch8#sharding) + - [串行执行总结](/ch8#summary-of-serial-execution) - [两阶段锁定(2PL)](/ch8#sec_transactions_2pl) + - [两阶段锁定的实现](/ch8#implementation-of-two-phase-locking) + - [两阶段锁定的性能](/ch8#performance-of-two-phase-locking) + - [谓词锁](/ch8#predicate-locks) + - [索引范围锁](/ch8#sec_transactions_2pl_range) - [可串行化快照隔离(SSI)](/ch8#sec_transactions_ssi) + - [悲观并发控制与乐观并发控制](/ch8#pessimistic-versus-optimistic-concurrency-control) + - [基于过时前提的决策](/ch8#decisions-based-on-an-outdated-premise) + - [检测陈旧的 MVCC 读取](/ch8#detecting-stale-mvcc-reads) + - [检测影响先前读取的写入](/ch8#sec_detecting_writes_affect_reads) + - [可串行化快照隔离的性能](/ch8#performance-of-serializable-snapshot-isolation) - [分布式事务](/ch8#sec_transactions_distributed) - [两阶段提交(2PC)](/ch8#sec_transactions_2pc) + - [系统性的承诺](/ch8#a-system-of-promises) + - [协调器故障](/ch8#coordinator-failure) + - [三阶段提交](/ch8#three-phase-commit) - [跨不同系统的分布式事务](/ch8#sec_transactions_xa) + - [精确一次消息处理](/ch8#sec_transactions_exactly_once) + - [XA 事务](/ch8#xa-transactions) + - [存疑时持有锁](/ch8#holding-locks-while-in-doubt) + - [从协调器故障中恢复](/ch8#recovering-from-coordinator-failure) + - [XA 事务的问题](/ch8#problems-with-xa-transactions) - [数据库内部的分布式事务](/ch8#sec_transactions_internal) + - [再谈精确一次消息处理](/ch8#exactly-once-message-processing-revisited) - [总结](/ch8#summary) - +- [参考](/ch8#参考) ## [9. 分布式系统的麻烦](/ch9) - - [故障与部分失效](/ch9#sec_distributed_partial_failure) - [不可靠的网络](/ch9#sec_distributed_networks) - [TCP 的局限性](/ch9#sec_distributed_tcp) - - [实践中的网络故障](/ch9#sec_distributed_network_faults) - - [故障检测](/ch9#id307) - - [超时与无界延迟](/ch9#sec_distributed_queueing) - - [同步网络与异步网络](/ch9#sec_distributed_sync_networks) + - [网络故障的实践](/ch9#sec_distributed_network_faults) + - [检测故障](/ch9#id307) + - [超时和无界延迟](/ch9#sec_distributed_queueing) + - [网络拥塞和排队](/ch9#network-congestion-and-queueing) + - [同步与异步网络](/ch9#sec_distributed_sync_networks) + - [我们不能简单地使网络延迟可预测吗?](/ch9#can-we-not-simply-make-network-delays-predictable) - [不可靠的时钟](/ch9#sec_distributed_clocks) - [单调时钟与日历时钟](/ch9#sec_distributed_monotonic_timeofday) - - [时钟同步与准确性](/ch9#sec_distributed_clock_accuracy) + - [日历时钟](/ch9#time-of-day-clocks) + - [单调时钟](/ch9#monotonic-clocks) + - [时钟同步和准确性](/ch9#sec_distributed_clock_accuracy) - [对同步时钟的依赖](/ch9#sec_distributed_clocks_relying) + - [用于事件排序的时间戳](/ch9#sec_distributed_lww) + - [带置信区间的时钟读数](/ch9#clock-readings-with-a-confidence-interval) + - [用于全局快照的同步时钟](/ch9#sec_distributed_spanner) - [进程暂停](/ch9#sec_distributed_clocks_pauses) -- [知识、真相与谎言](/ch9#sec_distributed_truth) + - [响应时间保证](/ch9#sec_distributed_clocks_realtime) + - [限制垃圾回收的影响](/ch9#sec_distributed_gc_impact) +- [知识、真相和谎言](/ch9#sec_distributed_truth) - [多数派原则](/ch9#sec_distributed_majority) - - [分布式锁与租约](/ch9#sec_distributed_lock_fencing) + - [分布式锁和租约](/ch9#sec_distributed_lock_fencing) + - [隔离僵尸进程和延迟请求](/ch9#sec_distributed_fencing_tokens) + - [多副本隔离](/ch9#fencing-with-multiple-replicas) - [拜占庭故障](/ch9#sec_distributed_byzantine) + - [弱形式的谎言](/ch9#weak-forms-of-lying) - [系统模型与现实](/ch9#sec_distributed_system_model) - - [形式化方法与随机测试](/ch9#sec_distributed_formal) + - [定义算法的正确性](/ch9#defining-the-correctness-of-an-algorithm) + - [安全性与活性](/ch9#sec_distributed_safety_liveness) + - [将系统模型映射到现实世界](/ch9#mapping-system-models-to-the-real-world) + - [形式化方法和随机测试](/ch9#sec_distributed_formal) + - [模型检查与规范语言](/ch9#model-checking-and-specification-languages) + - [故障注入](/ch9#sec_fault_injection) + - [确定性模拟测试](/ch9#deterministic-simulation-testing) - [总结](/ch9#summary) - + - [参考](/ch9#参考) ## [10. 一致性与共识](/ch10) - - [线性一致性](/ch10#sec_consistency_linearizability) - [什么使系统具有线性一致性?](/ch10#sec_consistency_lin_definition) - [依赖线性一致性](/ch10#sec_consistency_linearizability_usage) + - [锁定与领导者选举](/ch10#locking-and-leader-election) + - [约束与唯一性保证](/ch10#sec_consistency_uniqueness) + - [跨通道时序依赖](/ch10#cross-channel-timing-dependencies) - [实现线性一致性系统](/ch10#sec_consistency_implementing_linearizable) + - [线性一致性与仲裁](/ch10#sec_consistency_quorum_linearizable) - [线性一致性的代价](/ch10#sec_linearizability_cost) -- [ID 生成器与逻辑时钟](/ch10#sec_consistency_logical) + - [CAP 定理](/ch10#the-cap-theorem) + - [线性一致性与网络延迟](/ch10#linearizability-and-network-delays) +- [ID 生成器和逻辑时钟](/ch10#sec_consistency_logical) - [逻辑时钟](/ch10#sec_consistency_timestamps) + - [Lamport 时间戳](/ch10#lamport-timestamps) + - [混合逻辑时钟](/ch10#hybrid-logical-clocks) + - [Lamport/混合逻辑时钟 vs. 向量时钟](/ch10#lamporthybrid-logical-clocks-vs-vector-clocks) - [线性一致的 ID 生成器](/ch10#sec_consistency_linearizable_id) + - [实现线性一致的 ID 生成器](/ch10#implementing-a-linearizable-id-generator) + - [使用逻辑时钟强制约束](/ch10#enforcing-constraints-using-logical-clocks) - [共识](/ch10#sec_consistency_consensus) - [共识的多面性](/ch10#sec_consistency_faces) + - [单值共识](/ch10#single-value-consensus) + - [比较并设置作为共识](/ch10#compare-and-set-as-consensus) + - [共享日志作为共识](/ch10#sec_consistency_shared_logs) + - [获取并增加作为共识](/ch10#fetch-and-add-as-consensus) + - [原子提交作为共识](/ch10#atomic-commitment-as-consensus) - [共识的实践](/ch10#sec_consistency_total_order) - - [协调服务](/ch10#sec_consistency_coordination) + - [使用共享日志](/ch10#sec_consistency_smr) + - [从单主复制到共识](/ch10#from-single-leader-replication-to-consensus) + - [共识的微妙之处](/ch10#subtleties-of-consensus) + - [共识的利弊](/ch10#pros-and-cons-of-consensus) - [总结](/ch10#summary) + - [参考文献](/ch10#参考文献) +## [第十一章:批处理](/ch11) +- [使用Unix工具的批处理](/ch11#使用unix工具的批处理) + - [简单日志分析](/ch11#简单日志分析) + - [命令链与自定义程序](/ch11#命令链与自定义程序) + - [排序 VS 内存中的聚合](/ch11#排序-vs-内存中的聚合) + - [Unix哲学](/ch11#unix哲学) + - [统一的接口](/ch11#统一的接口) + - [逻辑与布线相分离](/ch11#逻辑与布线相分离) + - [透明度和实验](/ch11#透明度和实验) +- [MapReduce和分布式文件系统](/ch11#mapreduce和分布式文件系统) + - [MapReduce作业执行](/ch11#mapreduce作业执行) + - [分布式执行MapReduce](/ch11#分布式执行mapreduce) + - [MapReduce工作流](/ch11#mapreduce工作流) + - [Reduce侧连接与分组](/ch11#reduce侧连接与分组) + - [示例:用户活动事件分析](/ch11#示例用户活动事件分析) + - [排序合并连接](/ch11#排序合并连接) + - [把相关数据放在一起](/ch11#把相关数据放在一起) + - [分组](/ch11#分组) + - [处理偏斜](/ch11#处理偏斜) + - [Map侧连接](/ch11#map侧连接) + - [广播散列连接](/ch11#广播散列连接) + - [分区散列连接](/ch11#分区散列连接) + - [Map侧合并连接](/ch11#map侧合并连接) + - [MapReduce工作流与Map侧连接](/ch11#mapreduce工作流与map侧连接) + - [批处理工作流的输出](/ch11#批处理工作流的输出) + - [建立搜索索引](/ch11#建立搜索索引) + - [键值存储作为批处理输出](/ch11#键值存储作为批处理输出) + - [批处理输出的哲学](/ch11#批处理输出的哲学) + - [Hadoop与分布式数据库的对比](/ch11#hadoop与分布式数据库的对比) + - [存储多样性](/ch11#存储多样性) + - [处理模型的多样性](/ch11#处理模型的多样性) + - [针对频繁故障设计](/ch11#针对频繁故障设计) +- [MapReduce之后](/ch11#mapreduce之后) + - [物化中间状态](/ch11#物化中间状态) + - [数据流引擎](/ch11#数据流引擎) + - [容错](/ch11#容错) + - [关于物化的讨论](/ch11#关于物化的讨论) + - [图与迭代处理](/ch11#图与迭代处理) + - [Pregel处理模型](/ch11#pregel处理模型) + - [容错](/ch11#容错) + - [并行执行](/ch11#并行执行) + - [高级API和语言](/ch11#高级api和语言) + - [向声明式查询语言的转变](/ch11#向声明式查询语言的转变) + - [专业化的不同领域](/ch11#专业化的不同领域) +- [本章小结](/ch11#本章小结) +- [参考文献](/ch11#参考文献) + +## [第十二章:流处理](/ch12) +- [传递事件流](/ch12#传递事件流) + - [消息传递系统](/ch12#消息传递系统) + - [直接从生产者传递给消费者](/ch12#直接从生产者传递给消费者) + - [消息代理](/ch12#消息代理) + - [消息代理与数据库的对比](/ch12#消息代理与数据库的对比) + - [多个消费者](/ch12#多个消费者) + - [确认与重新传递](/ch12#确认与重新传递) + - [分区日志](/ch12#分区日志) + - [使用日志进行消息存储](/ch12#使用日志进行消息存储) + - [日志与传统的消息传递相比](/ch12#日志与传统的消息传递相比) + - [消费者偏移量](/ch12#消费者偏移量) + - [磁盘空间使用](/ch12#磁盘空间使用) + - [当消费者跟不上生产者时](/ch12#当消费者跟不上生产者时) + - [重播旧消息](/ch12#重播旧消息) +- [数据库与流](/ch12#数据库与流) + - [保持系统同步](/ch12#保持系统同步) + - [变更数据捕获](/ch12#变更数据捕获) + - [变更数据捕获的实现](/ch12#变更数据捕获的实现) + - [初始快照](/ch12#初始快照) + - [日志压缩](/ch12#日志压缩) + - [变更流的API支持](/ch12#变更流的api支持) + - [事件溯源](/ch12#事件溯源) + - [从事件日志中派生出当前状态](/ch12#从事件日志中派生出当前状态) + - [命令和事件](/ch12#命令和事件) + - [状态、流和不变性](/ch12#状态流和不变性) + - [不可变事件的优点](/ch12#不可变事件的优点) + - [从同一事件日志中派生多个视图](/ch12#从同一事件日志中派生多个视图) + - [并发控制](/ch12#并发控制) + - [不变性的局限性](/ch12#不变性的局限性) +- [流处理](/ch12#流处理) + - [流处理的应用](/ch12#流处理的应用) + - [复合事件处理](/ch12#复合事件处理) + - [流分析](/ch12#流分析) + - [维护物化视图](/ch12#维护物化视图) + - [在流上搜索](/ch12#在流上搜索) + - [消息传递和RPC](/ch12#消息传递和rpc) + - [时间推理](/ch12#时间推理) + - [事件时间与处理时间](/ch12#事件时间与处理时间) + - [知道什么时候准备好了](/ch12#知道什么时候准备好了) + - [你用的是谁的时钟?](/ch12#你用的是谁的时钟) + - [窗口的类型](/ch12#窗口的类型) + - [流连接](/ch12#流连接) + - [流流连接(窗口连接)](/ch12#流流连接窗口连接) + - [流表连接(流扩充)](/ch12#流表连接流扩充) + - [表表连接(维护物化视图)](/ch12#表表连接维护物化视图) + - [连接的时间依赖性](/ch12#连接的时间依赖性) + - [容错](/ch12#容错) + - [微批量与存档点](/ch12#微批量与存档点) + - [原子提交再现](/ch12#原子提交再现) + - [幂等性](/ch12#幂等性) + - [失败后重建状态](/ch12#失败后重建状态) +- [本章小结](/ch12#本章小结) +- [参考文献](/ch12#参考文献) + +## [第十三章:数据系统的未来](/ch13) +- [数据集成](/ch13#数据集成) + - [组合使用派生数据的工具](/ch13#组合使用派生数据的工具) + - [理解数据流](/ch13#理解数据流) + - [派生数据与分布式事务](/ch13#派生数据与分布式事务) + - [全序的限制](/ch13#全序的限制) + - [排序事件以捕获因果关系](/ch13#排序事件以捕获因果关系) + - [批处理与流处理](/ch13#批处理与流处理) + - [维护衍生状态](/ch13#维护衍生状态) + - [应用演化后重新处理数据](/ch13#应用演化后重新处理数据) + - [Lambda架构](/ch13#lambda架构) + - [统一批处理和流处理](/ch13#统一批处理和流处理) +- [分拆数据库](/ch13#分拆数据库) + - [组合使用数据存储技术](/ch13#组合使用数据存储技术) + - [创建索引](/ch13#创建索引) + - [一切的元数据库](/ch13#一切的元数据库) + - [开展分拆工作](/ch13#开展分拆工作) + - [分拆系统vs集成系统](/ch13#分拆系统vs集成系统) + - [少了什么?](/ch13#少了什么) + - [围绕数据流设计应用](/ch13#围绕数据流设计应用) + - [应用代码作为衍生函数](/ch13#应用代码作为衍生函数) + - [应用代码和状态的分离](/ch13#应用代码和状态的分离) + - [数据流:应用代码与状态变化的交互](/ch13#数据流应用代码与状态变化的交互) + - [流处理器和服务](/ch13#流处理器和服务) + - [观察派生数据状态](/ch13#观察派生数据状态) + - [物化视图和缓存](/ch13#物化视图和缓存) + - [有状态、可离线的客户端](/ch13#有状态可离线的客户端) + - [将状态变更推送给客户端](/ch13#将状态变更推送给客户端) + - [端到端的事件流](/ch13#端到端的事件流) + - [读也是事件](/ch13#读也是事件) + - [多分区数据处理](/ch13#多分区数据处理) +- [将事情做正确](/ch13#将事情做正确) + - [数据库的端到端原则](/ch13#数据库的端到端原则) + - [正好执行一次操作](/ch13#正好执行一次操作) + - [抑制重复](/ch13#抑制重复) + - [操作标识符](/ch13#操作标识符) + - [端到端原则](/ch13#端到端原则) + - [在数据系统中应用端到端思考](/ch13#在数据系统中应用端到端思考) + - [强制约束](/ch13#强制约束) + - [唯一性约束需要达成共识](/ch13#唯一性约束需要达成共识) + - [基于日志消息传递中的唯一性](/ch13#基于日志消息传递中的唯一性) + - [多分区请求处理](/ch13#多分区请求处理) + - [及时性与完整性](/ch13#及时性与完整性) + - [数据流系统的正确性](/ch13#数据流系统的正确性) + - [宽松地解释约束](/ch13#宽松地解释约束) + - [无协调数据系统](/ch13#无协调数据系统) + - [信任但验证](/ch13#信任但验证) + - [维护完整性,尽管软件有Bug](/ch13#维护完整性尽管软件有bug) + - [不要盲目信任承诺](/ch13#不要盲目信任承诺) + - [验证的文化](/ch13#验证的文化) + - [为可审计性而设计](/ch13#为可审计性而设计) + - [端到端原则重现](/ch13#端到端原则重现) + - [用于可审计数据系统的工具](/ch13#用于可审计数据系统的工具) +- [做正确的事情](/ch13#做正确的事情) + - [预测性分析](/ch13#预测性分析) + - [偏见与歧视](/ch13#偏见与歧视) + - [责任与问责](/ch13#责任与问责) + - [反馈循环](/ch13#反馈循环) + - [隐私和追踪](/ch13#隐私和追踪) + - [监视](/ch13#监视) + - [同意与选择的自由](/ch13#同意与选择的自由) + - [隐私与数据使用](/ch13#隐私与数据使用) + - [数据资产与权力](/ch13#数据资产与权力) + - [回顾工业革命](/ch13#回顾工业革命) + - [立法与自律](/ch13#立法与自律) +- [本章小结](/ch13#本章小结) +- [参考文献](/ch13#参考文献) -## [11. 批处理](/ch11)(未发布) -## [12. 流处理](/ch12)(未发布) -## [13. 做正确的事](/ch13)(未发布) ## [术语表](/glossary) + ## [后记](/colophon) +- [关于作者](/colophon#关于作者) +- [关于译者](/colophon#关于译者) +- [后记](/colophon#后记)