We’ve just launched Sequence support in DSQL and we’re excited about it. Up until now our recommendation has been to use UUIDs, and for truly massive scale it still is. But we recognize that there’s plenty of places where you’d like to use a unique number for an identity on a table.

If you just want to get started with sequences, here’s how you make one in DSQL:

CREATE SEQUENCE my_sequence CACHE 65536;
SELECT nextval('my_sequence'); 

Or if you want to use it as an identity in a column:¹

CREATE TABLE orders (
  id BIGINT GENERATED BY DEFAULT AS IDENTITY (CACHE 65536) PRIMARY KEY,
  customer_name TEXT
);

This blog is going to go more into the details of why sequences look like they do in DSQL, but to get started that’s all you need to know!²

Sequences in DSQL

In single box (or single writer) SQL systems, sequences are very lightweight. They provide unique values without a unique index and without taking heavy locks. In Postgres, sequences are stored like any other data: in a table. That table is then stored for durability on disk, with writes and updates going through a log for crash recovery. When a backend process calls nextval() it reads the value, increments it, and writes the new value back. To avoid going to disk too much, backends can also cache a number of values, set by the CACHE value of the sequence, we’ll come back to that later.

In a distributed architecture things are a little less simple, but not by much! Marc’s blog on the circle of life is a good primer on DSQL’s architecture. When you call nextval(), the read goes to storage, checks the latest value, and increments the value before writing the update to the journal. So far so simple? The important thing to remember is that getting the next set of values in a sequence goes through a full circle of life.

In distributed systems, creating scalable applications is a mutual responsibility³. The big problem with scaling sequences is they’re a classic hot key. DSQL’s advantage is that we can support endless horizontal scaling at every component, but to be able to scale DSQL needs to be able to spread your changes out across workers. For typical inserts we do that by partitioning the data. However, you can’t partition a single row table.

CACHE to the rescue

Remember the CACHE value from sequences in Postgres? This sets how many values a given backend gets on each call to nextval(). So with CACHE=3 a backend would fetch 3 values on every call to nextval(), which they can then use in requests without performing extra expensive IO.

                         ┌─────────────┐
                         │    Disk     │
                         │  seq = 10   │
                         └──────┬──────┘
                                │
           ┌────────────────────┼────────────────────┐
           │                    │                    │
           ▼                    ▼                    ▼
    ┌─────────────┐      ┌─────────────┐      ┌─────────────┐
    │  Backend A  │      │  Backend B  │      │  Backend C  │
    │ cache: 1-3  │      │ cache: 4-6  │      │ cache: 7-9  │
    └─────────────┘      └─────────────┘      └─────────────┘
           │                    │                    │
           ▼                    ▼                    ▼
      nextval()=1          nextval()=4          nextval()=7
      nextval()=2          nextval()=5          nextval()=8
      nextval()=3          nextval()=6          nextval()=9

Each backend reserves a chunk of sequence values on its first call. Subsequent nextval() calls return from the local cache without going to disk. If a backend crashes, those values from the sequence are discarded, so higher cache values can result in gaps in sequences.

DSQL parallelises our compute in the form of the Query Processor. Instead of individual backends, statements are executed by QPs. Here, CACHE functions the same as in Postgres. When a QP calls nextval() it gets a cached set of values, and hands them out. So now to the elephant in the room for DSQL support, we only support CACHE=1 or CACHE>=65536.

The point of these values is to highlight the decision for the developer. Either you want your sequence to be densely packed and low throughput, or you want it to be able to scale. With large cache values (>=65k), sequences are rarely a bottleneck in DSQL transactions.

What if I don’t need scale though?

That is totally fine too! Not every project is trying to hit 100k TPS. We also know there’s plenty of applications where you have a slow rate of inserts and would prefer a dense, increasing sequence. That’s why DSQL supports CACHE=1.

To put some numbers on it, I ran some experiments⁴.

Each of these I tested with CACHE 1, CACHE 65536, and I provided an example with UUID for a value that doesn’t require coordination. Since UUID is always locally generated, there’s no way for it to conflict and serves as a good baseline.

This is what the DDL for my first test looks like:

CREATE SEQUENCE seq_cache_1 CACHE 1;
CREATE SEQUENCE seq_cache_65536 CACHE 65536;

Here in the first test I’m creating two sequences, one with CACHE=1, and one with CACHE=65536. We’re then fetching new values serially, so we’re making one request to get a new value, waiting until we get it back, and then making another. The majority of the time is spent in network time waiting for the request to go from my laptop to DSQL’s QP and back. You’ll notice that the high cache value is faster, because the QP I’m connected to isn’t having to fetch an update from Storage every time, but it’s not faster by much. Comparing with UUID, you can see that’s pretty much the same as our high cache option.

------------------------------------------------------------
Experiment 1: Individual nextval() calls (100 iterations)
------------------------------------------------------------

CACHE=1:
  Mean:  20.26 ms
  Min:   17.94 ms
  Max:   97.10 ms
  Total: 2025.65 ms

CACHE=65536:
  Mean:  13.60 ms
  Min:   11.53 ms
  Max:   26.95 ms
  Total: 1360.24 ms

UUID:
  Mean:  13.31 ms
  Min:   11.81 ms
  Max:   27.11 ms
  Total: 1330.73 ms

Speedup with CACHE=65536 vs CACHE=1: 1.5x faster
Speedup with UUID vs CACHE=1: 1.5x faster

Okay great! But that doesn’t really tell us that much about how it scales. We’re just using a single connection and fetching one value at a time. Let’s look now at the case of a bulk insert. So here, we’re inserting 1000 rows into a table with a sequence:

-- Creating the table
CREATE TABLE bench_table (id BIGINT, data TEXT);

-- My insert statements look like this:
INSERT INTO bench_table (id, data)
SELECT nextval('seq_cache_1'), 'row ' || g
FROM generate_series(1, 1000) g;

We’re still running on one connection, but now we’re running a bulk insert of 1000 rows instead of fetching the nextval a bunch of times. So what does that look like?

------------------------------------------------------------
Experiment 2: Bulk INSERT with 1000 rows
------------------------------------------------------------

CACHE=1:     6229.11 ms total
CACHE=65536: 83.16 ms total
UUID:        77.24 ms total

Sequence CACHE=65536 vs CACHE=1: 74.9x faster
UUID vs CACHE=1: 80.6x faster

Well that’s a big difference! The reason for this is that typically incrementing sequences don’t follow the transaction semantics that we have for other values. It would be strange if something like:

BEGIN;
select nextval('my_seq'); -- returns 4
ROLLBACK;
select nextval('my_seq'); -- returns 4 again ??

were to happen. To preserve the expectation that sequences always return a unique, incrementing value, under the hood they are created by special internal transactions. This means that every call to fetch a nextval is going through the DSQL circle of life. With that in mind, the results for the bulk insert on a single connection make sense! For CACHE=1, even though we’re only on one connection, the QP has to go through the full loop for each row, fetching a value from storage, writing back to the journal, waiting for the transaction to finish before the next value can be read. With a large CACHE value, our QP only needs to do that once. This is on a single region cluster, but on a multi-region cluster the difference would be even more marked, because we’d need to wait for the write to be committed to our second region.

Now that was still on a single connection, what about when we actually want to scale? How do they behave when we throw more connections at them? This experiment is the same as experiment one, except we’re running contested. Instead of just one connection, let’s create 100, and have each of those fetch 100 next values:

------------------------------------------------------------
Experiment 3: Conflict test (100 workers x 100 nextvals each)
------------------------------------------------------------

CACHE=1:
  Total time:  51589.87 ms
  Throughput:  193.8 calls/sec
  Mean:        353.03 ms
  Min:         17.16 ms
  Max:         21182.18 ms
  Errors:      0

CACHE=65536:
  Total time:  3020.29 ms
  Throughput:  3310.9 calls/sec
  Mean:        17.46 ms
  Min:         10.57 ms
  Max:         1538.64 ms
  Errors:      0

UUID:
  Total time:  2902.25 ms
  Throughput:  3445.6 calls/sec
  Mean:        15.00 ms
  Min:         10.86 ms
  Max:         1439.94 ms
  Errors:      0

Throughput speedup with CACHE=65536 vs CACHE=1: 17.1x
Throughput speedup with UUID vs CACHE=1: 17.8x

The throughput difference is again just as marked. In the CACHE=1 case, the majority of internal transactions to fetch a cache value are conflicting. DSQL hides the internal details where these would show up as OCC errors, instead this shows up as latency, as conflicts would in regular Postgres. With high cache values we have almost no contention. Comparing to our baseline of UUIDs we can see the difference is minimal.

So what do I use?

If you want to use a sequence our recommendation is to use a high cache value. It’s going to keep up with your scale and avoid being a bottleneck in your system. If you really want densely packed sequences and you don’t expect your table to ever be running higher than a few transactions per second, then CACHE=1 will work just fine. If you change your mind or see it becoming a blocker down the line, you can always go back and fix it with:

ALTER SEQUENCE my_seq CACHE 65536;

But if you truly don’t want to worry about scale, just use UUIDs:

CREATE TABLE orders (
  id UUID DEFAULT gen_random_uuid() PRIMARY KEY,
  --...
);

This position paper from John Ousterhout sets out everything that’s wrong with TCP and exactly how we should fix it. It’s an interesting and purposefully polemical paper¹. Ousterhout has serious pedigree in distributed systems. He was one of the co-authors of In Search of an Understandable Consensus Algorithm, the paper that introduced Raft, created the Tcl scripting language, and has led a number of teams to impressive results over the years, so he’s talking from a place of experience.

The paper proposes that there are core issues with TCP that can’t be fixed. It argues that these issues are so core to TCP as to require breaking changes, at which point you might as well fix everything at once. It then goes on to discuss Homa, a protocol designed specifically for the datacenter, that fixes all of these issues.

What’s wrong with TCP?

Let’s run through the properties that the author sets out as needing to be reworked:

First, TCP is stream oriented. Work comes in as bytes, but in the datacenter it’s typically executed in complete messages, which have to be read from the stream and reconstructed. This means messages can’t be rerouted to available cores. Over time network speeds have increased to the point that server cores can’t keep up. To make full use of a network link you need to spread the load equally across them, but stream orientation makes that difficult to do. The stream is tied to whichever core is reading from the stream and you either need to then dispatch work to other cores or process whatever incoming message you have, blocking further work on the same stream. This quote highlights the issue well:

The fundamental problem with streaming is that the units in which data is received (range of bytes) do not correspond to dispatchable units of work (messages)

In a similar vein, TCP is also connection oriented. This adds overhead, each open remote connection on a Linux server requires around 2000 bytes of overhead in the kernel. Connections have non trivial time to setup too, with 1 RTT to connect. While connections made sense previously when clients and hosts were long lived, now many applications are serverless. Paying the connection cost makes less sense in that world.²

TCP requires in order delivery, although as Ousterhout admits, with a limited amount of reordering allowed. This prevents techniques for reducing load throughout the network, such as packet spraying, where packets are routed via different network pathways, reducing congestion at the hot nodes.

Congestion control is highlighted as another problem point. TCP’s congestion control is driven by senders reacting to backpressure. This means there must be packet queueing when the network is loaded.

Bandwidth sharing (or “fair scheduling”) shares bandwidth on a host link equally between active connections. But Ousterhout argues that this impacts short messages disproportionately, leading to much higher tail latencies under load.

Why is Homa better?

The main thrust of Homa is that it fixes all of these issues. It’s message oriented, so work arrives in dispatchable units. It’s connectionless, so there’s no setup or ongoing overhead. It can be delivered out of order, allowing packet spraying–balancing load evenly across network links. It also lets receivers control congestion through a kind of token bucket method. Senders can only send packets in response to grants from a receiver, so the receiver can limit congestion and use grants to prioritize certain (shorter) messages.

The only data provided in this paper is a graph displaying the 99th percentile slowdown on a loaded network. This took me a little while to parse so I’m going to talk my way through my understanding. As this is the slowdown, it’s graphing the ratio between the latency of messages in an unloaded, vs loaded network. It’s essentially showing how much slower the p99 is when the network is loaded, vs unloaded for each of these protocols, so we can see that for Homa, messages are about 6-10X slower under load for the p99, while for TCP it’s over 100X for small messages, dropping down to a little under 20X for 1M messages.

I found this a little confusing of a way to present the information, but it gets the message across! Homa is clearly designed to benefit tail latencies for smaller messages.

But what about encryption?

My big question reading this paper was the lack of any mention of encryption. TCP works very well with TLS and there are so many easy to set up integrations. There’s no excuse or reason to have datacenter traffic containing customer data communicating over plaintext. Although I believe there are existing standards for protocols like this, such as DTLS for datagrams and UDP, it would be great to have some kind of mention about how encryption fits into the picture.

What else is out there?

There are other protocols in the space. Discussing this paper with colleagues, I learned about the SRD protocol which is used by Elastic Block Store for high throughput. SRD takes advantage of many of the same improvements that Homa does, such as packet spraying through a network. This means packets can arrive unordered (which the protocol then handles) but the common case is in order. In cases like this, the work is already being done, however, EBS (even within AWS) is relatively unique in its needs. This paper does mention other alternatives, mainly Infiniband, but it doesn’t mention SRD.

Conclusion

I found this a fun read. Like a lot of foundational software, TCP becomes one of those things that you rarely think about not using. It’s a strong default for many good reasons. TCP is incredibly common, almost any host that you can think of has a TCP implementation, and because its so common its been heavily optimised over the years.

But it’s always worth revisiting building blocks as things change, particularly the most common ones. Saving resources at the lower levels can be so powerful because they are so common. Saving 1% on a $100M cost can justify spending a lot more engineering time than saving 90% on $1000.

However, I don’t know if this seems to be one of those cases. TCP is just so ubiquitous, and so well optimised already, that for 95% of use cases it’s not worth the effort to switch. With every new technology there are new operational scars to learn. The further out on the bleeding edge you are, the more you have to debug yourself. I think for extremely high throughput systems where you have control over the vertical system it will make sense. I definitely agree with the position that integrating any such protocol with a few major RP frameworks is the best start to get things off the ground. I’ll be interested to see over the next few years how this space continues to develop.

Related reading

• Attack of the Killer Microseconds
• A Cloud-Optimized Transport Protocol for Elastic and Scalable HPC
• It’s TCP vs. RPC All Over Again

this ability to take a property that the language does not know about and “teach” it to Rust, so that now it is enforced at compile time, is why he likes to call Rust an “X-safe” language. It’s not just memory-safe or thread-safe, but X-safe for any X that one takes the time to implement in the type system.

Rust is a language where you can use the type system to enforce safety guarantees at compile time. This isn’t exclusive to Rust, it’s a feature of any language with a strong type system. I think the ergonomics of Rust are particularly well suited to it, definitely over other systems languages.

I find myself coming back to this when we have implementation decisions. For example, if you are working on a service that has to handle encryption keys, you can use the type system to craft APIs and structs that simplify their handling. You can use a struct to ensure that your keys are never logged accidentally, by creating custom Display and Debug implementations that censor the plaintext. If you have multiple different kinds of encryption keys, you can craft APIs that require a key of type Key<ComponentA>, so you can’t accidentally pass in the key for ComponentB. There are all sorts of nice things you can do here, and it’s important to take advantage of them!

I think a good rule of thumb is when you find someone saying “as long as…”. As long as we use it in this way… As long as we pass it in the same way we receive it… That just means we know how we should use it, so we should encode that in the type system! People forget, people make mistakes. Lets make it easy on ourselves by making it harder to make mistakes than to just use the API as intended.

Apache Kafka is a system for streaming logs with the aim of providing high throughput over strict guarantees around delivery. This was an interesting paper to read ~13 years on, as Kafka has become more and more ubiquitous in system design.

Design

This paper presents Kafka as a relatively simple system, consisting of brokers and consumers. Brokers receive messages from producers. Consumers poll brokers for messages that they care about, which are broken up into topics. Messages are opaque byte strings, allowing consumers to define their own formats. LinkedIn, for example, used Avro encoding, a binary serialization format with a versioned schema¹.

Producers send messages to brokers on particular topics. To distribute load across the nodes, topics are sharded into a number of partitions. Each broker stores one or more of these. Consumers subscribe to a topic by creating 1+ message streams, which each provide an iterator over a stream of messages. Producers can publish a message to either a randomly selected partition, or one determined by a partition key and a partition function.

Let someone else do the hard work

In a few key areas the Kafka developers chose simplicity over complexity in their design, letting someone else do the hard work. This first comes up in the caching (or lack of). They decided to rely on the file system’s page cache instead of caching messages in memory. This has the benefit of avoiding double buffering as well as maintaining a cache between broker process restarts. It’s also simpler to implement and maintain.

Similarly, the developers use the sendfile API to eliminate additional copying on the host. This eliminates 2 buffer copies and 1 system call from a standard approach of sending bytes. This adds up to make Kafka more efficient for throughput.

Instead of organising their own consensus mechanism, Kafka uses Zookeeper for the coordination primitives. This saves reinventing the wheel and keeps the coordination of the system much simpler. Zookeeper mimics a simple file system’s page cache for an API. Paths can be created, have their values set, read, or deleted. Nodes can register to watch a path - meaning a watcher can be notified when the children of a path or its value have been changed. This is used by Kafka for nodes to know when they need to reconfigure.

Paths can also be created as ephemeral, so when the client that creates them disappears the path is automatically removed. By offboarding the complexity of this management to Zookeeper, Kafka again priorities simplicity. Instead of having a centralized main node, the consumers and producers can coordinate in a decentralized way.

Whenever brokers or consumers are added, a rebalancing process is triggered, which spreads the partitions of a topic over the new set of consumers. This is a pretty simple algorithm that runs determinastically on each consumer. Because Kafka only guarantees “at least once” delivery, there’s no real correctness issues that crop up here. But it’s important for developers to be aware that they should implement their own idempotency mechanism if its needed.

Pull vs Push

Kafka operates on a “pull” model for messages, meaning consumers are in charge of the state necessary to pull messages. Each message is stored in memory in the broker while being identified by its memory offset in the log. Logs are partitions of topics implemented as a set of files , each split into roughly the same size.

Instead of giving each message a unique ID, it is identified by its offset in the log file. This is a pretty interesting implementation! It means that the brokers have much less state or information to manage for each message. Consumers start reading messages from the queue, and request the next message by sending the end of the offset they got to. This means that the brokers don’t need to store any state for consumers, they can just feed them the next messages when requested. Another benefit of this is easy checkpointing. If a consumer fails while processing, it can pick up from its last successful checkpoint.

To trim messages, brokers wait for a specific time period, e.g., 7 days. This allows replaying messages over a longer term for consumers, they can just roll back to an earlier point in the queue. Since message queues are saved to files on the brokers, the only added pressure is to disk utilization – which is a lot cheaper than memory.

Why did Kafka become so successful?

I struggled to find any hard data showing the market adoption of Kafka aside from this 2023 Stack Overflow survey where 10% of professional devs reported using it - however that still put it behind RabbitMQ at 12%. There are a lot of articles claiming that Kafka has ‘won’ the message queueing space, and it certainly seems to be used more and more each year, either as a system or a compatible protocol.

There’s probably a lot of people who are more authorised to draw conclusions here than I am. From my understanding, the simplicity of Kafka seems really advantageous. As an API, it’s super simple to work with. You have a continuous iteration of messages which simply hangs when waiting for new ones along the stream. It’s also quite powerful for development. Thanks to the pull model, consumers can replay messages for up to a configurable time limit. This means from an operations standpoint, it’s much simpler to re-read a queue than it is to re-push a queue of messages (as you would in the case of SNS).

The performance benefits of this approach are pretty clear from the paper, at least against the existing top log processors of the time. Since 2011 a number of other data streaming services have emerged, both open and closed source. Kafka’s by no means the only option, but it’s a good example of what the authors put forward: by having a specialized system you can get a good deal of extra performance.

This took me all of 8:00am to 9:30am today to get a working solution. This feels very much in a similar vein to Marc Brooker’s experience with gen AI coding tools. It’s great for small, greenfield tasks where you want a solution, but don’t have the time to dive into all of the weeds yourself.

Here I was using Claude 3.5 Sonnet to iterate. I’ve added the chatbot log to the repo to keep a record of what it was like to actually iterate and reach the solution. There’s a few things I need to do first, but hopefully I’ll be able to share it soon.

The code itself isn’t particularly long, and it’s mostly glue that’s supported by incredible open source projects (mdBook and the whole Rust ecosystem). However, I still find it super impressive that this is where we are at. It feels like a step change in tooling. I’m having more and more of these moments when it comes to reaching to LLMs to help me fill a gap in my available tools.

Long term, would I blindly use this crate? No. I think I would go through the code with a line by line review before publishing, add some TODOs and make notes of the sketchier parts that are likely to bite you. But to get a solution off the ground in no time at all, it’s awesome.

Now I can get back to the blog writing I actually meant to do.

Wouldn’t it be nice to write some software and confidently say that you know it’s right? That, as long as some assumptions about the world hold, it’s going to do exactly what you want it to, no matter what strange permutations or combinations of failures happen. In broad strokes that’s the promise of formal verification and proofs in software.

This paper, in particular, is about formally verifying controllers. More specifically, how to verify liveness for Kubernetes cluster controllers. While the researchers start there, I think their contributions are more broadly applicable for other control plane architectures too.

What’s TLA?

Let’s start with a quick diversion/refresher, feel free to skip if you already know! TLA stands for temporal logic of actions. It’s a way of thinking, step by step, how things change over time as the result of actions. Leslie Lamport came up with the field as a way to describe concurrent programs in a formal, logical way.

We use formal methods like TLA to be able to make concrete mathematical statements about things that we can then prove or disprove. Something like “the ground is dry” is a logical proposition about the world. If then an action happens like “it rains” we can know that in the next state “the ground is dry” is false. TLA allows us to draw these same kinds of logical conclusions about concurrent systems.

It’s really useful when we want to describe properties of systems. These could be safety properties, or liveness properties. A safety property says that our system doesn’t do a bad thing, while a liveness property says that our system eventually does a good thing, or the thing we want it to do. Liveness is a tricky thing! Safety properties are usually easier to prove because a system that does nothing can be perfectly safe. Liveness properties require the system to keep making progress towards a goal.

What does this paper provide?

This paper provides a property called eventually stable reconciliation (ESR). They claim that this property precludes a lot of bugs in controllers. It also presents Anvil, a framework for developing kubernetes controllers and verifying that they meet this property. They then use Anvil to verify three Kubernetes controllers and show they have comparable performance to their un-verified counterparts.

ESR is a TLA formula, it’s a statement about a system that can be expressed in TLA. It essentially says that if what the controller wants stops changing, then the system should eventually reach that state.

That seems super reasonable as a goal to me! Intuitively, if a controller is able to update a system to match its desired state then it seems like it’s doing its job. If you don’t have familiarity with controllers or control plane software, you can think of them as being like a thermostat. You set some kind of desired state through configuration (turn the thermometer to 20C), and then the controller makes adjustments to the environment (turns your central heating on/off) while monitoring for changes (checking the current temp) until its desired state is reached.

In the case of a Kubernetes controller, this looks something more like creating service configuration files, or spinning up containerized nodes. The overall principle is the same, but the amount of input or ongoing state increases, and the number of ways in which things can go wrong multiplies.

Formally ESR is represented by the TLA formula:

\(\forall d.\Box(\Box \text{desire}(d) \implies \Diamond \Box \text{match}(d))\)¹

This asserts that for all desired states $d$, if the controller always desires $d$, then eventually $d$ will always match the current state. It seems clear that this will prevent a whole range of bugs that prevent the controller from moving towards its goal.

How do they prove things?

The authors use Verus, a language that allows writing proofs in the form of preconditions and postconditions. Verus leverages Rust’s type system and reduces the problem to an SMT-solvable form. The team extended Verus with a set of simple temporal logic constructs to handle the temporal aspects of their proofs.

The advantage of using Verus is that it allows for modular proofs, breaking down complex properties into smaller, more manageable pieces. This approach makes it easier to reason about and verify complex systems like cluster controllers. Like Dafny, and in contrast to a model language like TLA+, it allows verifying the actual code that you run in production.

The Anvil toolkit provides a good deal of work towards proving ESR for Kubernetes controllers. Proving implementations with Anvil is still a manual process, you still have to write proof code to go with your implementation, but Anvil does provide a lot of handy looking lemmas² to help.

By applying their ESR property and proof methodology, the authors were able to identify and fix bugs in existing controllers that had been missed by extensive testing. This shows the power of formal verification in catching subtle issues that can be nearly impossible to uncover with traditional testing.

The Future

A big part of why I find this exciting is that it shows a future where more and more of our software is written proof first. With projects like this, it doesn’t seem hard to imagine a world where we start our software with a given proven core and then extend it to match the service at hand. I think this could be extremely powerful for Control Plane software where the problems are relatively common and abstractable: identify what’s wrong, make changes to fix it.

We’re already seeing other aspects of software trend in this formally verified direction. The AWS crypto team have delivered huge speed improvements by formally verifying RSA. By building on the foundation of a correct solution they were able to make more aggressive optimisations.

To be a bit more pedantic, I see this happening for software at the largest scales. At AWS scale events that would be one in a billion are happening every hour. Formal correctness can help eliminate these. A big benefit from that is that, much like with testing, formal verification can give teams the ability to make big changes more confidently. If you know that you have a process to catch issues, it’s much easier to be bold in making big changes.

This paper is about building a network primitive to speed up consensus protocols. Like other papers in this area, MRTOM builds on the fact that network ordered protocols can have much higher throughput than standard consensus protocols. MRTOM takes this one step further by offloading not just packet ordering, but also the fast path of consensus protocols to programmable switches.

The consensus problem

It’s probably a good idea to have a quick refresher on what consensus problems are, and why network ordered protocols can be faster.

Broadly, the problem of consensus is agreeing with a group what’s happened. It’s like if you were sending a series of letters to 20 different people in different countries. Some of your letters might get lost or intercepted before they get there, or one of the people you’re sending to might be missing a letter and would disagree with the others on what you’ve said.

Even worse, the letters need to have an agreed order. If you send three messages, the end meaning might be completely different if they arrive in a different order:

• I’ll be arriving at midnight
• I’ll be arriving at noon
• Ignore that last message, the next one will be my new arrival time

These three messages have different meanings depending on when they arrive. We could understand that you’ll be arriving at midnight, or at noon, or we could be unsure of what time you’re arriving at all.¹

Without a way of deciding what letters all of the group has received and in what order, we can’t agree on exactly what you’ve said. Consensus protocols like Paxos solve this by ensuring a consensus group agrees on what messages they’ve received and in what order. Typically a Paxos like protocol has a leader that accepts requests from clients, while followers can be used to read committed data or will forward new requests onto the leader. During the process of agreeing on a message, the members of a Paxos group have to send messages back and forth with each other, which takes time, meaning we can process fewer messages and respond slower than if there was just one recipient.

The benefits of having a group of recipients are that because the data is replicated, it is more resilient. One of the recipients could become unavailable, but we’re still able to provide that data when someone comes to read it.

If you know what you’re expecting next, you can act fast

Network ordered consensus protocols, like NOPaxos, exploit the fact that if you have a total ordering for incoming client messages, you can get consensus pretty quickly! Typically consensus incurs a penalty, as a group of servers incur a communication overhead in agreeing what they’ve received. Network ordered protocols make the observation that if you have a monotonically increasing dense ID² assigned to incoming requests, 1) its very easy to know that you have things in order, and 2) its very obvious if you’re missing a message. If the packets reliably arrive and in order³, then servers in a consensus group don’t need to communicate to agree on what they’ve received and in what order.

Since not doing something is always cheaper and faster than doing something, this lets network ordered protocols do as little as possible. They only have to communicate when packets don’t arrive. This is the separation of the fast and the slow path of network ordered consensus protocols. The fast path is when everything gets there on time and in order, while the slow path happens when a server is missing a message.

The downside of network ordered protocols is they need something to order the messages. Typically this is a single network switch which all messages to the group have to pass through. This switch will then assign each message a monotonically increasing ID. The negative of this is that we’re introducing a single point of failure (bad) and a scaling bottleneck (also not great). There’s no such thing as a free lunch.

MRTOM Design

MRTOM (pronounced Mr. Tom) is based on the observation that the fast path of many consensus protocols “is precisely the reliable, ordered, and acknowledged delivery of messages to a set of nodes”. By focusing on offloading that fast path to the network, the core idea of the paper is to free up server capacity for handling the slow path and other application logic.

MRTOM works by having a MRTOM instance, typically a programmable switch, in between clients and the group of servers running the consensus protocol. This is so far so familiar for the network ordered story. The MRTOM instance provides an ordering for packets, which is then used to speed up consensus.

It differs in two important ways, first that it tries to increase the reliability of delivery by maintaining a loopback of packets. Once a packet has been acked by all servers in the group, MRTOM considers it delivered and can discard it. If it’s not acknowledged within a certain time then MRTOM re-sends the packet.

Second, and more importantly, MRTOM offloads the fast path from the server group, instead running it on the switch through eBPF. This is the main aspect that gives the protocol a big throughput and latency advantage over NOPaxos or other protocols.

eBPF usage

MRTOM allows offloading the fast path of protocols to network switches. They do this using eBPF, which is a way of using Linux kernel capabilities without needing to change kernel source code.

Extended BPF developed from Berkeley Packet Filtering, but is mostly referred to as eBPF now, as the capabilities go beyond just packet filtering. eBPF programs let you run code in the kernel userspace through verified bytecode. This means they can attach hooks to the kernel without recompiling or creating kernel modules. As a result, we can get very efficient program execution as it cuts out the syscall middle man. This in turn cuts CPU & memory overheads by avoiding allocations for packet processing. eBPF also provides means for kernel level programs to have data structures that can interact with user space programs, which is how the switch can then push packets into the slow path.

In the case of MRTOM-Paxos, the fast path is integrated into the MRTOM edge interface. This interface handles aggregating acknowledgement responses from servers in the MRTOM group. Once a majority of servers in the group (including the leader) have responded, the switch sends back an acknowledgement to the client. This aggregation reduces the number of messages and coordination between the client and servers in the group.

Single switch

This paper uses a single switch, much like NOPaxos, which gives a single bottleneck. However, we’ve seen recently other protocols like Hydra demonstrate the ability to scale the number of sequencers while maintaining a total consistent order. I’d like to see a combination of these ideas, to see if there would be a way to merge the multiple sequencer approach with MRTOM’s fast path offloading. My intuition is that it wouldn’t be possible to mix them, given Hydra’s reliance on the receiver’s reconstructing the order.

Summary

MRTOM isn’t a real theoretical advance, but it presents a nice practical idea for increasing throughput and decreasing latency. Doing more at the network level, or even on the server while bypassing the typical overhead of a linux stack with eBPF/XDP, can help us be fast.

However, I’m not sure if the trend of the industry is heading in this direction. While programmable switches have been around for a while, there’s very few commercial cloud operators that will let you use them, and when you do it’s certainly not easy. The trend seems to be to more abstracted and easily fungible hardware, rather than investing time programming switches that will need to be replaced in 4-5 years.

Majority quorum systems are useful in providing a simple mechanism for consensus. To accept a value, you need a majority of servers to agree to accepting it. Weighted majority quorum services (WQMS) take this approach and recognise that some servers are going to have better performance than others, so they should get more voting power.

The main contribution of this paper is defining three ways of reassigning weights in a WQMS. The first two are shown to be as difficult as consensus, while the third can be performed consensus-free. I’m not fully sure how practically useful this is, but that’s also because I don’t know much about the real world uses of WQMS¹! As far as the paper goes, it’s very clearly written, with some nicely presented proofs.

Weight Reassignment

The paper presents the problem of weight reassignment. Solving this problem means providing an algorithm to update the weights of a static set of servers running a WQMS.

Weight reassignment has some restrictions. With the system, we want to be able to tolerate up to $f$ servers (out of a total $n$) crashing at once. This means that we need to place a limit on the total weight of the $f$ most weighted servers so they have less than half of the total weight. This way, if any $f$ servers fail, the remaining $n - f$ can continue to make progress. We call this restriction integrity.

This means that we can’t reassign weights in a way that would violate integrity. Say we had a quorum of 5 servers, each with a voting weight of $1$, and we wanted to tolerate up to two server failures. If we were to add $2$ to the weight of any of the servers, now the $f$ top weighted servers would have a total weight of $4$, and the total weight of all servers would be $7$. Since the total weight of the top $f$ servers is greater than half of the total weight, this would violate integrity. For brevity going forward, we’ll call the set of the top $f$ servers $F$.

There are three other restrictions on the weight reassignment algorithm that are simpler:

• Validity-I - when a weight change is proposed, if it violates integrity then a no-op change (a change with zero weight difference) is created. If it doesn’t violate integrity then the proposed change is created.
• Validity-II - there’s an API clients can use to check the weights of a server $s$, read_changes(s). When it’s called, a client gets a list of all of the weight changes made to $s$, from which they can reconstruct the current weight of $s$ by summing the changes.
• Livesness - if a server calls reassign, the operation will eventually complete, and the server will get back a message indicating the set of changes made.

Equivalence to consensus

Laid out in this way, the authors go on to show that this problem is equivalent to consensus, meaning it’s at least as difficult to solve. Given we’re using a quorum system to solve consensus, that sounds like it would defeat the point!

I really liked the proof they use to demonstrate this. The authors construct a scenario in which every server proposes a weight change. These changes are constructed such that one, and only one, of the weight changes can succeed with a non-zero weight change. If two or more of the changes succeeded, then integrity would be violated.

For the proof, the servers are divided into two disjoint sets, $F$ and $S \backslash F$, where $F$ is the set of servers $F = { s_1, s_2, …, s_f}$. $S$ is the set of all servers, so $S \backslash F$ is the set of all servers not in $F$. Note that the sets have the following lengths $|S| = n, |F| = f, |S \backslash F| = n - f$. The initial weight of each server $s \in F$ is $\frac{n - 1}{2f}$ while the weight of every server $s \in S \backslash F$ is $\frac{n+1}{2(n - f)}$. We can call the total weight of a set $S$ at time $t$: $\texttt{W}_{S,t}$. Based on the initial weights of each server in the sets, $\texttt{W}_{F,0} = \frac{n - 1}{2f} \times f$ and $\texttt{W}_{S \backslash F,0} = \frac{n+1}{2(n - f)} \times (n - f)$. Since $\frac{n - 1}{2} < \frac{n+1}{2}$, we can see that the total weight of $F$ is less than the total weight of $S \backslash F$ and integrity is satisfied by the initial weights.

Each of the servers $s_i$ proposes a weight change. For the servers in $F$, they propose adding $0.5$ to their weight: $\texttt{reassign}(s_i, 0.5)$, while all servers in $S \backslash F$ propose subtracting $0.5$ from their weight: $\texttt{reassign}(s_i, -0.5)$. From this, we can see that accepting one of these changes would not violate integrity. E.g., if we accept one change from $F$, then the new total weight of $F$ becomes $\frac{n - 1}{2} + 0.5$, which is still less than $\frac{n+1}{2}$. Similarly, if we accept one change from $S \backslash F$, then $\texttt{W}_{S \backslash F,1} = \frac{n+1}{2} - 0.5$.

However, it’s clear that if we accept more than one change in any combination then integrity will be violated. E.g., if we accept one change from both sets, we can see that $\frac{n - 1}{2} + 0.5 = \frac{n+1}{2} - 0.5$, which would violate integrity. Since if a change doesn’t violate integrity we have to accept it, then we must accept one and only one change, which is the same as deciding consensus on a value between the group. The paper also provides an algorithm for solving consensus by proposing weight changes and deciding on one, which is fun but probably not as interesting or useful as the equivalence proof.

Pairwise weight reassignment

The authors then tried restricting the problem to make it easier to solve. If it’s hard to arbitrarily re-assign weights up and down among the servers, what about only allowing pairs of servers to exchange weights? E.g., for server $s_2$ to gain a weight, some other server $s_4$ needs to lose the same amount of weight. This means that the total weight of all servers can remain constant throughout.

Apart from this change, weight reassignment remains the same. The integrity requirement is still in place. Instead of a server proposing $\texttt{reassign}$, they can propose to $\texttt{transfer}(s_i, s_j, \Delta)$, where the $\Delta$ change in weight is taken from $s_i$ and given to $s_j$ if integrity is not violated. If integrity would be violated by the change, then two zero weight changes are created (one for $s_i$ and one for $s_j$).

The authors then show in much the same way that this is also equivalent to consensus. Just like before, they craft a scenario based on the sets of servers $F$ and $S \backslash F$ where one and only one weight $\texttt{transfer}$ can complete with a non-zero weight, showing that the problem is equivalent to consensus. The proof is pretty similar to the previous one so I won’t go in to the exact scenario. Again, they provide an algorithm for consensus based on this impossibility proof.

Restricted pairwise weight reassignment

Finally the authors introduce restricted pairwise weight reassignment, which can be performed without consensus. There are two restrictions they place on transfer:

1. Only $s_i$ can call $\texttt{transfer}(s_i, *, \Delta)$. So only $s_i$ can transfer away some of its weight
2. The weight of $s_i$ has to always be greater than $\frac{\texttt{W}_{S,0}}{2(n - f)}$. That means there’s a floor on the weight of each server, such that if each server in $S \backslash F$ had this weight, they would have the majority vote.

The authors assert that if the first condition holds, then the second is locally verifiable, meaning it can be done without consensus. I found this argument pretty straight forward! Any server can give away its weight, but only while remaining greater than the floor weight, leaving all of the servers not in $F$ with enough weight to continue as a quorum. This is proved with the inequality:

So $\texttt{W}_{S \backslash F, t} > \frac{\texttt{W}_{S, 0}}{2}$ and integrity is preserved at all times.

The reason no other server can take away another’s weight without consensus is that if we have any two servers trying to take away weight from some server $s_i$, then while either of them could make a transfer recognizing the second condition, there’s no way for both transfers to succeed without some form of consensus to decide which succeeds.

Thoughts

The authors proved this result for a static set of servers. It would be great to see if the results could be extended for a dynamic set of servers, but given all of the proofs relied on a static set I’d imagine that would be hard. At the minimum you could imagine using consensus to decide weights as you add/remove servers, before switching back to the consensus free weight reassignment.

How useful is it? You can easily imagine a scenario where a server doesn’t know that it’s holding up progress for a quorum, and since only servers can give away their own weight, that might prove tricky to use in practice. It would be great to see an experimental evaluation to see how helpful this could be.

Replicated systems typically pretty much always have some overhead in comparison to unreplicated systems, at least if you want strong consistency for your data. We need to do extra work in order to make sure that we get the same result across all nodes. The fastest systems minimise or avoid that coordination, but where we can’t avoid it, we need an algorithm to manage that consensus.

Network ordered distributed protocols can be surprisingly performant compared to unreplicated systems. Network-Ordered Paxos (NOPaxos) is able to achieve throughput within 2% of an unreplicated system, while for comparison Paxos only achieves around 25%¹. However, they have drawbacks. NOPaxos requires sending all packets for a consensus group through a single point. This leaves it difficult to scale the size of groups within a data centre, and can increase the time to recover when a sequencer fails.

This paper provides an algorithm for consistent network packet ordering with drop detection over a parallel set of sequencers. This means we can get the benefits of packet sequencing (higher throughput and lower latency consensus algorithms) while avoiding the single point of failure in a system.

How it works

Single sequencer

Let’s start with the case of a single sequencer, as in NOPaxos. A single node sequencer works by maintaining a counter. When it receives a packet from a sender it adds a header with the current counter, increments the counter, and sends² the packet to a group of receivers.

When the receivers get the packet, they can then recreate the same ordering that the sequencer saw pretty simply. Because the packets each have a number that’s monotonically increasing (1, 2, 3…) it’s easy to sort the sequence. Similarly, this is how drop detection comes in. If a receiver has received a set of packets with sequence counts: [1, 2, 4], then it can tell it’s missing a packet with sequence count 3. The receiver will then send a drop notification to the other members of its group, who then can decide to either permanently ignore the packet, or accept it and then resend to all the members that are missing it. They do that through an elected leader, which initiates a round of agreement. Leaders are regularly elected in a process similar to Paxos, but we don’t need to go into that for the differences with Hydra.

Together this provides consistent ordering and drop detection of packets.

Hydra

Hydra takes this protocol and adds the ability to run multiple sequencers. This means you’re not limited by the throughput of a single switch or host when scaling your service. The paper shows a roughly linear increase in throughput as the number of switches increases. If one switch was limited to a throughput of 200k messages per second across a group, then with two you should have a throughput of 400k.

Just like in the single sequencer case, each sequencer maintains a monotonically increasing counter (1, 2, 3…). This counter is maintained locally by each sequencer. When they receive a packet, they add the counter as a header, increment their counter, and forward the packet on to the receiver group. The sequencers also have their own ID, which they add to the packet and is used to determinsitically settle the order of packets at the receivers. If we took this naive approach then we would lose drop detection. Imagine a receiver’s getting packets from one sequencer, but missing those from another. How can it tell that a packet from the other sequencer has been dropped?

To resolve this issue, the paper introduces a combination of sequence numbers and physical clocks. Each sequencer also has a physical clock tracking real time. When forwarding packets on, as well as adding the local counter to the packet, the sequencer also adds its current clock value and its sequencer ID. Because the physical clocks are monontonically increasing, the protocol is able to guarantee that each message broadcast by the sequencers has a consistent partial ordering:

Partial ordering definition - $\S 4.3.1$

For messages $m_1$ and $m_2$ sent to the same recipients, with respective clock values $c_1$ and $c_2$, sequenced by sequencers with IDs $i$ and $j$, $m_1$ is ordered before $m_2$ if $c_1 < c_2 \vee (c_1 = c_2 \wedge i < j)$

This essentially means when a receiver gets two messages from different sequencers, the one with the lower clock value is ordered first, and if the clock values are the same then ties are broken using the sequencer ID.

Since the stamps on each packet are consistent for each receiver, the ordering is the same among all of them. So that’s great! But how does that help us with drop detection?

Drop Detection

This is where the sequence numbers come back in. Remember how the single sequencer scenario uses these to detect drops? Hydra uses them in a similar way. Each Hydra receiver first buffers the messages it gets, and only “delivers” them (logically to the application) once they have determined that no message with a lower clock value from another sequencer will be delivered.

To do that, the receivers track two values for each sequencer: the largest sequence number, and the largest clock value seen in its messages. The receiver will only deliver messages up to the point when it knows that all other receivers have a higher clock value.

Let’s take an example where a receiver is listening to two sequencers with IDs $1$ and $2$. If a receiver has received three messages:

• $m_1$ that has a clock value $c = 14$, a sequencer ID $s_{id} = 2$ and a sequencer count of $s_c = 1$
• $m_2$ with $c = 12, s_{id} = 1, s_c = 1$
• $m_3$ with $c = 20, s_{id} = 1, s_c = 2$

The first two messages can be delivered in the order $m_2, m_1$, because the receiver knows that the time on sequencer 1 is at least 20, and the time on sequencer 2 is at least 14. Therefore the minimum time at all sequencers is 14, and it can deliver all messages with a physical time up to that point. It can’t yet deliver $m_3$ because it hasn’t received a message from sequencer 2 with a time of 20 or greater.

If the sequencer then received another message, $m_4$, with $c = 30, s_{id} = 2, s_c = 3$, it would know that it’s missing the message $s_{id} = 2 \wedge s_c = 2$. At that point it delivers a drop notification for $s_{id} = 2 \wedge s_c = 2$ to the application.

Because we’re now waiting for updates from all sequencers before we can deliver messages, it’s clear there could be some issues. What if one sequencer gets fewer messages to forward on than others? A receiver could be waiting a while to get an update from a slow sequencer, while a load of messages from other sequencers are queueing up. To fix this progress issue, the paper presents configurable flush messages. This is a kind of heartbeat notification, where the sequencer sends a packet with its current physical clock, and its current sequence number (without incrementing it). This allows all receivers to be updated on the minimum time across all sequencers, so they can send out messages before that time.

It should be clear enough that the safety of this protocol isn’t affected by clock drift, just the performance. If the sequencers have a huge difference in their physical clocks then receivers may be waiting a long time for all receivers to catch up to a high water mark time. However, the messages still have consistent ordering, and drop detection is not affected by a slow physical clock.

Thoughts

I thought this was a really interesting paper! It was great to dig into the network ordering that enables NOPaxos. The contributions here definitely extend the work in practical and useful directions. It’s pretty rare that you need to implement a system like this, but interesting to know that network ordering can be scaled beyond a single sequencer.

Getting an interview

Tech companies look for engineers that can work with data. They want people that know how it can tell a story, how to get the right data, what’s important, and how you can use it to support your arguments. Your CV should reflect this. Try to identify nuggets of data about your previous experiences to include.

What was the impact of your actions in your projects? This doesn’t even have to be technical projects. If you were part of a student society that held events, did you grow their attendance by some %-age year on year? Companies want to see that you can set goals for improvement and hit them, or at least have awareness of it. Include this in both your CV bullet points and in your interview prep.

When it comes to applications, a referral is at least 10X more effective than applying to a portal – at least when it comes to getting you through the first screening. After that it’s up to you, but it’s the best way to get an introduction to a company. There’s so many low quality submissions that come in through online application portals that it’s very hard for recruiters to screen, particularly with the rise of job application bots and LLM usage.

If you have a target company you want to work at, try and get a referral instead of applying for a job just through a portal. Look up people that work there on LinkedIn. You can filter by some criteria that’s connected to you to make an easier introduction. Look for mutual connections, that you went to the same school, etc. that’s not the key part but it helps people have familiartiy. Reach out to them with a short message about yourself, what you’re aiming for, and ask if they can refer you.

If there’s a position advertised you’re looking at, reach out to the hiring manager and ask for more information. Hiring managers are much more likely to hire candidates that they know. It’s a good way to show interest and that you’re a real person before putting an application in. This stuff is a little scary at first, but it gets easier when you understand what people want. Hiring managers want good quality candidates that won’t waste their time. Hiring is an incredibly expensive process! You have something to offer them as a friendly, competent, understanding individual.

Interview prep

Have a 50/50 emphasis split on leadership questions (stories about what work you’ve done in the past and how they show certain characteristics the company is looking for) vs DSA coding skills (Leetcode or whatever other platform). You should prepare the stories for interviews in much the same way you prepare your coding skills.

For coding prep I have no other advice than to do it. I did a bunch prior to my first role, and I really enjoyed a lot of the learning. Leetcode Easy and Mediums will serve you well for entry level roles. It does suck, but it’s a bit of a necessary evil. No one needs you to solve DSA problems under time pressure in your day to day job, but like exams at school, it’s a way of demonstrating you can play the game.

Write out a grid of all of the various experiences you had against all of the public values that your target company advertises¹. For each box write a few bulletpoints in the STAR² format (situation, task, action, result) about the story, talking about how you demonstrated that principle. People care most about the action and the result, while the situation and the task are the context so they can understand. You should be able to relay these stories in around 5 minutes.

When it comes to the interview process, performing that you understand and can demonstrate what they want to see is as good as actually doing it. Interviewers want to know that you can play the game of internal values, because that’s how companies organise themselves without pulling in many different directions. They’re used in all company decisions, hiring, project prioritisation. Playing a role in a company is as good as being the role.

When it comes to interviewing, come with a few questions. It’s your chance to learn about the place too, what the day to day is like. Be curious about the team and the work. It’s a good way to demonstrate you are engaged in the process and evaluating your options carefully.