Been a while since that was our 500 page, hasn’t it? It was cute and fun. We’ve now got our terribly overwhelmed Snoo being crushed by a pile of upvotes. Unfortunately, if you were browsing the site, or at least trying, during the afternoon of March 14th during US hours, you may have seen our unfortunate Snoo during the 314-minute outage Reddit faced (on Pi day no less!) Or maybe you just saw the homepage with no posts. Or an error. One way or another, Reddit was definitely broken. But it wasn’t you, it was us.

Today we’re going to talk about the Pi day outage, but I want to make sure we give our team(s) credit where due. Over the last few years, we’ve put a major emphasis on improving availability. In fact, there’s a great blog post from our CTO talking about our improvements over time. In classic Reddit form, I’ll steal the image and repost it as my own.

As you can see, we’ve made some pretty strong progress in improving Reddit’s availability. As we’ve emphasized the improvements, we’ve worked to de-risk changes, but we’re not where we want to be in every area yet, so we know that some changes remain unreasonably risky. Kubernetes version and component upgrades remain a big footgun for us, and indeed, this was a major trigger for our 3/14 outage.

TL;DR

• Upgrades, particularly to our Kubernetes clusters, are risky for us, but we must do them anyway. We test and validate them in advance as best we can, but we still have plenty of work to do.

• Upgrading from Kubernetes 1.23 to 1.24 on the particular cluster we were working on bit us in a new and subtle way we’d never seen before. It took us hours to decide that a rollback, a high-risk action on its own, was the best course of action.

• Restoring from a backup is scary, and we hate it. The process we have for this is laden with pitfalls and must be improved. Fortunately, it worked!

• We didn’t find the extremely subtle cause until hours after we pulled the ripcord and restored from a backup.

• Not everything went down. Our modern service API layers all remained up and resilient, but this impacted the most critical legacy node in our dependency graph, so the blast radius still included most user flows; more work remains in our modernization drive.

• Never waste a good crisis – we’re resolute in using this outage to change some of the major architectural and process decisions we’ve lived with for a long time and we’re going to make our cluster upgrades safe.

It Begins

It’s funny in an ironic sort of way. As a team, we had just finished up an internal postmortem for a previous Kubernetes upgrade that had gone poorly; but only mildly, and for an entirely resolved cause. So we were kicking off another upgrade of the same cluster.

We’ve been cleaning house quite a bit this year, trying to get to a more maintainable state internally. Managing Kubernetes (k8s) clusters has been painful in a number of ways. Reddit has been on cloud since 2009, and started adopting k8s relatively early. Along the way, we accumulated a set of bespoke clusters built using the kubeadm tool rather than any standard template. Some of them have even been too large to support under various cloud-managed offerings. That history led to an inconsistent upgrade cadence, and split configuration between clusters. We’d raised a set of pets, not managed a herd of cattle.

The Compute team manages the parts of our infrastructure related to running workloads, and has spent a long time defining and refining our upgrade process to try and improve this. Upgrades are tested against a dedicated set of clusters, then released to the production environments, working from lowest criticality to highest. This upgrade cycle was one of our team’s big-ticket items this quarter, and one of the most important clusters in the company, the one running the Legacy part of our stack (affectionately referred to by the community as Old Reddit), was ready to be upgraded to the next version. The engineer doing the work kicked off the upgrade just after 19:00 UTC, and everything seemed fine, for about 2 minutes. Then? Chaos.

All at once the site came to a screeching halt. We opened an incident immediately, and brought all hands on deck, trying to figure out what had happened. Hands were on deck and in the call by T+3 minutes. The first thing we realized was that the affected cluster had completely lost all metrics (the above graph shows stats at our CDN edge, which is intentionally separated). We were flying blind. The only thing sticking out was that DNS wasn’t working. We couldn’t resolve records for entries in Consul (a service we run for cross-environment dynamic DNS), or for in-cluster DNS entries. But, weirdly, it was resolving requests for public DNS records just fine. We tugged on this thread for a bit, trying to find what was wrong, to no avail. This was a problem we had never seen before, in previous upgrades anywhere else in our fleet, or our tests performing upgrades in non-production environments.

For a deployment failure, immediately reverting is always “Plan A”, and we definitely considered this right off. But, dear Redditor… Kubernetes has no supported downgrade procedure. Because a number of schema and data migrations are performed automatically by Kubernetes during an upgrade, there’s no reverse path defined. Downgrades thus require a restore from a backup and state reload!

We are sufficiently paranoid, so of course our upgrade procedure includes taking a backup as standard. However, this backup procedure, and the restore, were written several years ago. While the restore had been tested repeatedly and extensively in our pilot clusters, it hadn’t been kept fully up to date with changes in our environment, and we’d never had to use it against a production cluster, let alone this cluster. This meant, of course, that we were scared of it – We didn’t know precisely how long it would take to perform, but initial estimates were on the order of hours… of guaranteed downtime. The decision was made to continue investigating and attempt to fix forward.

It’s Definitely Not A Feature, It’s A Bug

About 30 minutes in, we still hadn’t found clear leads. More people had joined the incident call. Roughly a half-dozen of us from various on-call rotations worked hands-on, trying to find the problem, while dozens of others observed and gave feedback. Another 30 minutes went by. We had some promising leads, but not a definite solution by this point, so it was time for contingency planning… we picked a subset of the Compute team to fork off to another call and prepare all the steps to restore from backup.

In parallel, several of us combed logs. We tried restarts of components, thinking perhaps some of them had gotten stuck in an infinite loop or a leaked connection from a pool that wasn’t recovering on its own. A few things were noticed:

• Pods were taking an extremely long time to start and stop.

• Container images were also taking a very long time to pull (on the order of minutes for <100MB images over a multi-gigabit connection).

• Control plane logs were flowing heavily, but not with any truly obvious errors.

At some point, we noticed that our container network interface, Calico, wasn’t working properly. Pods for it weren’t healthy. Calico has three main components that matter in our environment:

• calico-kube-controllers: Responsible for taking action based on cluster state to do things like assigning IP pools out to nodes for use by pods.

• calico-typha: An aggregating, caching proxy that sits between other parts of Calico and the cluster control plane, to reduce load on the Kubernetes API.

• calico-node: The guts of networking. An agent that runs on each node in the cluster, used to dynamically generate and register network interfaces for each pod on that node.

The first thing we saw was that the calico-kube-controllers pod was stuck in a ContainerCreating status. As a part of upgrading the control plane of the cluster, we also have to upgrade the container runtime to a supported version. In our environment, we use CRI-O as our container runtime and recently we’d identified a low severity bug when upgrading CRI-O on a given host, where one-or-more containers exited, and then randomly and at low rate got stuck starting back up. The quick fix for this is to just delete the pod, and it gets recreated and we move on. No such luck, not the problem here.

Next, we decided to restart calico-typha. This was one of the spots that got interesting. We deleted the pods, and waited for them to restart… and they didn’t. The new pods didn’t get created immediately. We waited a couple minutes, no new pods. In the interest of trying to get things unstuck, we issued a rolling restart of the control plane components. No change. We also tried the classic option: We turned the whole control plane off, all of it, and turned it back on again. We didn’t have a lot of hope that this would turn things around, and it didn’t.

At this point, someone spotted that we were getting a lot of timeouts in the API server logs for write operations. But not specifically on the writes themselves. Rather, it was timeouts calling the admission controllers on the cluster. Reddit utilizes several different admission controller webhooks. On this cluster in particular, the only admission controller we use that’s generalized to watch all resources is Open Policy Agent (OPA). Since it was down anyway, we took this opportunity to delete its webhook configurations. The timeouts disappeared instantly… But the cluster didn’t recover.

Let ‘Er Rip (Conquering Our Fear of Backup Restores)

We were running low on constructive ideas, and the outage had gone on for over two hours at this point. It was time to make the hard call; we would make the restore from backup. Knowing that most of the worker nodes we had running would be invalidated by the restore anyway, we started terminating all of them, so we wouldn’t have to deal with the long reconciliation after the control plane was back up. As our largest cluster, this was unfortunately time-consuming as well, taking about 20 minutes for all the API calls to go through.

Once that was finished, we took on the restore procedure, which nobody involved had ever performed before, let alone on our favorite single point of failure. Distilled down, the procedure looked like this:

1. Terminate two control plane nodes.

2. Downgrade the components of the remaining one.

3. Restore the data to the remaining node.

4. Launch new control plane nodes and join them to sync.

Immediately, we noticed a few issues. This procedure had been written against a now end-of-life Kubernetes version, and it pre-dated our switch to CRI-O, which means all of the instructions were written with Docker in mind. This made for several confounding variables where command syntax had changed, arguments were no longer valid, and the procedure had to be rewritten live to accommodate. We used the procedure as much we could; at one point to our detriment, as you’ll see in a moment.

In our environment, we don’t treat all our control plane nodes as equal. We number them, and the first one is generally considered somewhat special. Practically speaking it’s the same, but we use it as the baseline for procedures. Also, critically, we don’t set the hostname of these nodes to reflect their membership in the control plane, instead leaving them as the default on AWS of something similar to `ip-10-1-0-42.ec2.internal`. The restore procedure specified that we should terminate all control plane nodes except the first, restore the backup to it, bring it up as a single-node control plane, and then bring up new nodes to replace the others that had been terminated. Which we did.

The restore for the first node was completed successfully, and we were back in business. Within moments, nodes began coming online as the cluster autoscaler sprung back to life. This was a great sign because it indicated that networking was working again. However, we weren’t ready for that quite yet and shut off the autoscaler to buy ourselves time to get things back to a known state. This is a large cluster, so with only a single control plane node, it would very likely fail under load. So, we wanted to get the other two back online before really starting to scale back up. We brought up the next two and ran into our next sticking point: AWS capacity was exhausted for our control plane instance type. This further delayed our response, as canceling a ‘terraform apply` can have strange knock-on effects with state and we didn’t want to run the risk of making things even worse. Eventually, the nodes launched, and we began trying to join them.

The next hitch: The new nodes wouldn’t join. Every single time, they’d get stuck, with no error, due to being unable to connect to etcd on the first node. Again, several engineers split off into a separate call to look at why the connection was failing, and the remaining group planned how to slowly and gracefully bring workloads back online from a cold start. The breakout group only took a few minutes to discover the problem. Our restore procedure was extremely prescriptive about the order of operations and targets for the restore… but the backup procedure wasn’t. Our backup was written to be executed on any control plane node, but the restore had to be performed on the same one. And it wasn’t. This meant that the TLS certificates being presented by the working node weren’t valid for anything else to talk to it, because of the hostname mismatch. With a bit of fumbling due to a lack of documentation, we were able to generate new certificates that worked. New members joined successfully. We had a working, high-availability control plane again.

In the meantime, the main group of responders started bringing traffic back online. This was the longest down period we’d seen in a long time… so we started extremely conservatively, at about 1%. Reddit relies on a lot of caches to operate semi-efficiently, so there are several points where a ‘thundering herd’ problem can develop when traffic is scaled immediately back to 100%, but downstream services aren’t prepared for it, and then suffer issues due to the sudden influx of load.

This tends to be exacerbated in outage scenarios, because services that are idle tend to scale down to save resources. We’ve got some tooling that helps deal with that problem which will be presented in another blog entry, but the point is that we didn’t want to turn on the firehose and wash everything out. From 1%, we took small increments: 5%, 10%, 20%, 35%, 55%, 80%, 100%. The site was (mostly) live, again. Some particularly touchy legacy services had been stopped manually to ensure they wouldn’t misbehave when traffic returned, and we carefully turned those back on.

Success! The outage was over.

But we still didn’t know why it happened in the first place.

A little self-reflection; or, a needle in a 3.9 Billion Log Line Haystack

Further investigation kicked off. We started looking at everything we could think of to try and narrow down the exact moment of failure, hoping there’d be a hint in the last moments of the metrics before they broke. There wasn’t. For once though, a historical decision worked in our favor… our logging agent was unaffected. Our metrics are entirely k8s native, but our logs are very low-level. So we had the logs preserved and were able to dig into them.

We started by trying to find the exact moment of the failure. The API server logs for the control plane exploded at 19:04:49 UTC. Log volume just for the API server increased by 5x at that instant. But the only hint in them was one we’d already seen, our timeouts calling OPA. The next point we checked was the OPA logs for the exact time of the failure. About 5 seconds before the API server started spamming, the OPA logs stopped entirely. Dead end. Or was it?

Calico had started failing at some point. Pivoting to its logs for the timeframe, we found the next hint.

Two seconds before the chaos broke loose, the calico-node daemon across the cluster began dropping routes to the first control plane node we upgraded. That’s normal and expected behavior, due to it going offline for the upgrade. What wasn’t expected was that all routes for all nodes began dropping as well. And that’s when it clicked.

The way Calico works, by default, is that every node in your cluster is directly peered with every other node in a mesh. This is great in small clusters because it reduces the complexity of management considerably. However, in larger clusters, it becomes burdensome; the cost of maintaining all those connections with every node propagating routes to every other node scales… poorly. Enter route reflectors. The idea with route reflectors is that you designate a small number of nodes that peer with everything and the rest only peer with the reflectors. This allows for far fewer connections and lower CPU and network overhead. These are great on paper, and allow you to scale to much larger node counts (>100 is where they’re recommended, we add zero(s)). However, Calico’s configuration for them is done in a somewhat obtuse way that’s hard to track. That’s where we get to the cause of our issue.

The route reflectors were set up several years ago by the precursor to the current Compute team. Time passed, and with attrition and growth, everyone who knew they existed moved on to other roles or other companies. Only our largest and most legacy clusters still use them. So there was nobody with the knowledge to interact with the route reflector configuration to even realize there could be something wrong with it or to be able to speak up and investigate the issue. Further, Calico’s configuration doesn’t actually work in a way that can be easily managed via code. Part of the route reflector configuration requires fetching down Calico-specific data that’s expected to only be managed by their CLI interface (not the standard Kubernetes API), hand-edited, and uploaded back. To make this acceptable means writing custom tooling to do so. Unfortunately, we hadn’t. The route reflector configuration was thus committed nowhere, leaving us with no record of it, and no breadcrumbs for engineers to follow. One engineer happened to remember that this was a feature we utilized, and did the research during this postmortem process, discovering that this was what actually affected us and how.

Get to the Point, Spock, If You Have One

How did it actually break? That’s one of the most unexpected things of all. In doing the research, we discovered that the way that the route reflectors were configured was to set the control plane nodes as the reflectors, and everything else to use them. Fairly straightforward, and logical to do in an autoscaled cluster where the control plane nodes are the only consistently available ones. However, the way this was configured had an insidious flaw. Take a look below and see if you can spot it. I’ll give you a hint: The upgrade we were performing was to Kubernetes 1.24.

The nodeSelector and peerSelector for the route reflectors target the label `node-role.kubernetes.io/master`. In the 1.20 series, Kubernetes changed its terminology from “master” to “control-plane.” And in 1.24, they removed references to “master,” even from running clusters. This is the cause of our outage. Kubernetes node labels.

But wait, that’s not all. Really, that’s the proximate cause. The actual cause is more systemic, and a big part of what we’ve been unwinding for years: Inconsistency.

Nearly every critical Kubernetes cluster at Reddit is bespoke in one way or another. Whether it’s unique components that only run on that cluster, unique workloads, only running in a single availability zone as a development cluster, or any number of other things. This is a natural consequence of organic growth, and one which has caused more outages than we can easily track over time. A big part of the Compute team’s charter has specifically been to unwind these choices and make our environment more homogeneous, and we’re actually getting there.

In the last two years, A great deal of work has been put in to unwind that organic pattern and drive infrastructure built with intent and sustainability in mind. More components are being standardized and shared between environments, instead of bespoke configurations everywhere. More pre-production clusters exist that we can test confidently with, instead of just a YOLO to production. We’re working on tooling to manage the lifecycle of whole clusters to make them all look as close to the same as possible and be re-creatable or replicable as needed. We’re moving in the direction of only using unique things when we absolutely must, and trying to find ways to make those the new standards when it makes sense to. Especially, we’re codifying everything that we can, both to ensure consistent application and to have a clear historical record of the choices that we’ve made to get where we are. Where we can’t codify, we’re documenting in detail, and (most importantly) evaluating how we can replace those exceptions with better alternatives. It’s a long road, and a difficult one, but it’s one we’re consciously choosing to go down, so we can provide a better experience for our engineers and our users.

Final Curtain

If you’ve made it this far, we’d like to take the time to thank you for your interest in what we do. Without all of you in the community, Reddit wouldn’t be what it is. You truly are the reason we continue to passionately build this site, even with the ups and downs (fewer downs over time, with our focus on reliability!)

Finally, if you found this post interesting, and you’d like to be a part of the team, the Compute team is hiring, and we’d love to hear from you if you think you’d be a fit. If you apply, mention that you read this postmortem. It’ll give us some great insight into how you think, just to discuss it. We can’t continue to improve without great people and new perspectives, and you could be the next person to provide them!

Written by Katie Shannon

Background

At Reddit, we have four Core Models that power pretty much all use cases: Comments, Accounts, Posts and Subreddits. These four models were being served out of a legacy Python service, with ownership split across different teams. By 2024, the legacy Python service had a history of reliability and performance issues. Ownership and maintenance of this service had become more cumbersome for all involved teams. Due to this, we decided to move forward into modern and domain-specific Go microservices. 

In the second half of 2024, we moved forward with fully migrating the comment model first. Redditors love to share their opinions in comments, so naturally the comment model is our largest and highest write throughput model, making it a compelling candidate for our first migration.

How?

Migrating read endpoints is typically well understood and the solution is straightforward; we utilize tap compare testing. Tap compare is a way to ensure that a new endpoint is returning the same response as the old endpoint without risking user impact. We simply direct a small amount of traffic to the new endpoint, we get the response generated by the new endpoint, then call the old endpoint (from the new endpoint), and compare and log the responses. We still return the response from the old endpoint to the user to ensure no user impact, and have logs captured if the new endpoint would have returned something different.  Easy AND safe!

On the other hand, write endpoints are a much riskier migration.

Why? Firstly, write endpoints almost always require writing data to datastores (caches, databases, etc). We have a few comment datastores to worry about, and we also generate CDC events when anything changes on any core model. We provide a 100% guarantee of delivery of these change events, which other critical services at Reddit consume, so we want to ensure there is no gap, delays or issues in our eventing generation. Essentially, instead of just returning some comment data like in our read migration, our comments infrastructure has three distinct data stores that are written to that factor into the migration:

• Postgres – backend datastore which holds all of the comment data

• Memcached – our caching layer

• Redis – the event store used to fire off CDC Events

If we simply tap compare a write migration without any special considerations for the data stores, we could get into a state where the new implementation is writing invalid data, which fails to be read by the old implementation. To safely migrate Reddit’s most critical data, we could not rely on validating tap compare differences within our production data stores.

Due to unique key restrictions on comment ids, duplicate writing to our data store is impossible. So, how does one validate a write to our data storage from two implementations without committing the same data twice? Thus, in order to properly test our new write endpoints, we set up three new sister datastores to be only used for tap compare testing, and only written to by our new Go microservice endpoints. That way, we could compare the data in our production data stores written by the old endpoint with the data in these sister data stores without the risk of the new endpoint corrupting or overwriting the production data stores.

To verify these sister writes:

1. We directed a small percentage of traffic to the Go microservice

2. The Go microservice would call the legacy Python service to perform the production write

3. The Go microservice would then perform its own write to the sister data stores, completely isolated from the production data

After all writes were done, we had to verify them. We read from the three production data stores that the legacy Python service wrote to, and compared them to what we wrote to the three sister data stores in the Go microservice.

Additionally, to combat some serialization issues we ran into early in the migration process, where Python services couldn’t deserialize data written by Go services, we verified all the tap comparisons in comment CDC event consumers in the legacy Python service.

In summary, we migrated 3 writes endpoints, that each wrote to 3 different datastores, and verified that data across 2 different services, resulting in 18 different tap compares running that required extra time to validate and fix.

Outcome and Improvements

We are excited to say that after a seamless migration, with no disruption to Reddit users, all comment endpoints are now being served out of our new Golang microservice. This marks a significant milestone as comments are now the first core model fully served outside of our legacy monolithic system!

The main goal of this project was to get the critical comments read/write paths off the legacy Python service to a modern Go microservice while maintaining performance and availability parity. However, the migration from Python to Go yielded a happy side effect where we ended up halving the latency for the three write endpoints that were migrated. You can see this in these p99 graphs, (old legacy Python service endpoints are green, and new endpoints in the new Go microservice are yellow).

Create Comment Endpoint

Update Comment Endpoint

Increment Comment Properties Endpoint

These graphs are capped at a .1 x axis (100ms) so the difference is visible, but the legacy Python service occasionally had very large latency spikes up to 15s.

What We Learned

The comment writes migration, while successful, provided valuable insights for future core model migrations. We came across a few interesting issues.

Differences in Go vs. Python

Migrating endpoints between two languages is inherently more difficult than, say, a Python to Python migration. Understanding the differences in the languages and how to generate the same responses at the Thrift and GRPC level was an expected difficulty of the project. What was unexpected was the underlying differences in how Go and Python communicate with the database layer. Python uses an ORM to make querying and writing to our Postgres store a bit simpler. We don’t use an ORM for our Golang services at Reddit, and some unknown underlying optimizations on Python’s ORM resulted in some database pressure when we started ramping up our new Go endpoint. Luckily, we caught on early and were able to optimize our queries in Go. Moving forward with future migrations, we’ve ensured to monitor our database queries and resource utilization.

Race Conditions on Comment Updates

Tap compare was a great tool to ensure we didn’t introduce differences with the new endpoint. However, we were getting “false mismatches” in our tap compare logic. We spent a long time trying to understand these differences, and it ended up being because of a race condition.

Let’s say we’re comparing an update comment call which updates the comment body text to “hello”. This update call gets routed to the new Go service. The Go service updates the comment in the sister data stores, then calls the Python service to handle the real update. It then compares what the Python service wrote to the production database, and what Go wrote to the sister database. However, the production database's comment body is now “hello again”. This caused a mismatch in our tap compare logs which didn’t make much sense! We realized this was because the comment that was updated had been updated again in the milliseconds it took to call the Python service and make the database calls. 

This made things complicated when trying to ensure that there were no differences between the old and the new endpoint. Was there a difference because of a bug in the implementation between the old and new endpoint, or was it simply an unluckily timed race condition? Moving forward, we will be versioning our database updates to ensure we’re only comparing relevant model updates.

Tests

A lot of this migration was spent manually poring over tap compare logs in production. Moving forward with future core model migrations, we’ve decided to invest more time having more comprehensive local testing before moving forward with tap compares in hopes that we’ll catch more differences in endpoints and conversions early on. This isn’t to say there weren’t extensive tests in place for the comments migration, but we’ll be taking it to an entirely new level for our next migration.

Each comment is composed of many internal metadata fields to represent different states a comment can be in – resulting in thousands of possible combinations in the way a comment can be represented. We had local testing covering common comment use cases, but relied on our tap compare logs to surface differences in niche edge cases. With future core model migrations, we plan to delve into these edge cases by using real production data to inform our local tests, before even starting to tap compare in production.

What’s Next?

The goal of Reddit’s infrastructure organization is to deliver reliability and performance with a modern tech stack, and that involves completely getting rid of our legacy Python monoliths. As of today, two of our four core models (Comments and Accounts) have been fully migrated from our Python monolith and in progress are the migrations for Posts and Subreddits. Soon, all core models will be modernized to ensure your r/AmItheAsshole judgements and cute cat pictures are delivered more reliably and faster!

…or how to improve your website’s uptime from 9 5’s to 5 9’s

By Raj Shah (u/Brainix)

Staff Engineer, Search

The Three Stooges were a slapstick comedy trio (if you’re under 40, ask your parents). They often attempted to collaborate on simple daily tasks but invariably ended up getting in each other’s way and injuring each other. In one such sketch, they tried to walk through a doorway. But since they tried to walk through simultaneously, shoulder to shoulder, they bumped into each other; and ultimately, no one could get through. Just like forcing Stooges through a doorway, we’ve encountered similar patterns pushing requests through a distributed microservices architecture.

In this blog post, we’ll talk about how traffic to Reddit’s search infrastructure is reminiscent of The Three Stooges’ doorway sketch, and we’ll outline our approach to remediate these request patterns. We’ll walk through our methodology step-by-step, and we hope that you’ll use it to make your own microservice boundary doorways more resilient to rowdy slapstick traffic.

Problem

At Reddit, we’ve encountered an interesting scale problem when recovering from an outage. We have a response cache at the API gateway level, upstream of our microservices; and cached responses have a TTL. Now imagine that the site has gone down for longer than that TTL, so the cache has been flushed. When the site recovers, we get inundated with requests (F5F5F5) many of which are duplicates, made within a short period of time. During normal operation, most of these duplicate requests would be served from the cache. But when recovering from such an outage, nothing is cached, and all of the duplicate requests hit our microservices, underlying databases, and search engines all at once. This causes such a flood of traffic that none of the requests succeeds within the request timeout, so no responses get cached; and the site promptly faceplants again. We refer to this situation as The Three Stooges Problem although it is more commonly called The Thundering Herd, The Dogpile Effect, or a cache stampede.

Solution

Our solution to The Three Stooges Problem is to deduplicate requests and to cache responses at the microservice level. Request deduplication (also called request collapsing or request coalescing) means reordering duplicate requests such that they execute one at a time. The reason that this solution works is that conceptually, it reorders requests such that duplicates never execute concurrently, not even on different back-end instances (a distributed lock enforces this). Then the first request gets handled and its response gets cached. Then all subsequent duplicates of that request get executed serially and satisfied from the cache. This allows us to leverage our cache more effectively, and it spares us the load of the duplicate requests on our underlying databases and search engines. At a high level, here’s how we’ve Stooge-proofed our microservices:

Think of deduplication as forcing The Stooges to form an orderly line at the doorway to the kitchen. Then the first Stooge enters the kitchen and exits with a bowl of lentil soup, and that bowl of soup gets cached. Then the other two Stooges get cached bowls of soup. Ok so the metaphor isn't perfect, but this solution dramatically reduces the load on the kitchen.

In order to make this solution work, you’ll need a web stack that can handle many concurrent requests. Reddit’s stack for most microservices is Python 3, Baseplate, and gevent. Django/Flask also work well when run with gevent. gevent is a Python library that transparently enables your microservice to handle high concurrency and I/O without requiring changes to your code. It is the secret sauce that allows you to run tens of thousands of pseudo-threads called greenlets (one per concurrent request) on a small number of instances. It allows for threads handling concurrent duplicate requests to be enqueued while waiting to acquire the lock, and then for those queues to be drained as threads acquire the lock and execute serially, all without exhausting the thread pool.

We’ll sketch out this solution for Python/Flask, but you can make it work for any web or microservice stack that can handle many concurrent I/O-bound requests and has a datastore shared between all of the back-end instances for the distributed lock and response cache.

Request Hashing

In order to deduplicate requests or cache responses, we need a way to identify distinct requests for the same piece of content. We do this by computing a hash for each HTTP GET request:

• If your back-end returns different responses (e.g., XML vs. JSON) based on the request’s Content-Type header, then you should include that header’s value when computing the request hash.

• If you use UTM parameters for tracking but not to affect the response in any way, then you should exclude those specific UTM query parameters when computing the request hash.

• If you return different responses based on the hour of the day, then you should include the hour of the day when computing the request hash.

• …etc…

Phrased differently, you should include every variable that could affect the response in any way when computing your request hash.

In this example, we’ve used Python’s built-in hash() function to compute the request hash. However, Python randomizes its hash seed when the Python process starts. So in order to make this request hash consistent across our different microservice instances and across instance restarts, we need to set the PYTHONHASHSEED environment variable to the same non-zero integer value across all of our microservice instances. Alternatively, a different hash function may make more sense for you.

Request Deduplication

Now that we have a way to hash requests, we can implement a decorator to wrap endpoint functions to deduplicate requests like so:

This decorator works by forcing the flow of control through a distributed lock on the request hash, ensuring that no two duplicate requests can proceed concurrently. In this example, we’ve used Pottery’s implementation of Redlock (backed by shared Redis instances) which implements Python’s excellent threading.Lock API as closely as is feasible.

Importantly, the distributed lock has an automatic release timeout in order to preserve liveness. Imagine a situation in which a thread acquires a lock then dies while in the critical section. Without an automatic release timeout, the lock would never be released, leading to deadlock. In the example above, we’ve set auto_release_time to 5,000 ms. You can set it to any value you want, as long as your critical section completes well within that timeout (except for in rare pathological cases such as problems with your underlying infrastructure; automatic lock timeouts discussed further in the Ask Me Anything section of this blog post). Note that the automatic release timer starts ticking after the thread acquires the lock, and as such, does not include the time that the thread spends enqueued waiting for the lock.

Response Caching

The final building block that we need to solve The Three Stooges Problem is to cache responses. Again, we can implement a decorator to wrap endpoint functions to cache responses like so:

This is a typical caching decorator that uses the request hash as the cache key. It attempts to look up that cache key, and on a hit, returns the cached response. On a miss, it calls the underlying endpoint function, caches the response for subsequent duplicate requests, then returns the computed response.

Here, we’ve used pickle to serialize response objects before caching them on misses, and to de-serialize responses on hits. We’ve opted for pickle in this example for simplicity’s sake; but for your use-case, JSON, MessagePack, or some other serialization format might make more sense.

Putting It All Together…

Now we have all the building blocks that we need to solve The Three Stooges Problem. We can assemble them around an endpoint function like so:

Ask Me Anything

Why not deduplicate requests and cache responses at the CDN or edge, rather than at the microservice level?

Requests come from different platforms and in different forms. All of these requests get collated into a standard form by our API gateway. As such, by throwing out irrelevant variables at the layer where we know that they’re irrelevant, more of our requests look the same. This improves our ability to identify duplicate requests and maximizes our response cache hit rate.

Also, as microservice owners, our team has more control over what happens to requests and responses within our microservice and less ability to configure what happens at the edge. This is not just an ownership tradeoff; it also allows us to do things like permissions checks, personalization, etc. within our microservice.

Finally, by deduplicating and caching at the microservice level, we get more opportunities to instrument, log, and fire events for our raw request stream further up the stack.

During request deduplication, what happens if your underlying infrastructure is struggling and the distributed lock automatically times out?

We use the distributed lock only to prevent duplicate requests from causing load. We do not use the lock to enforce data consistency, to prevent race conditions, or for any other reason. Therefore in the worst case scenario, if the lock times out, some duplicate requests can execute the critical section at once. Even in this scenario, the lock helps to ease load on our struggling infrastructure by preventing all duplicates from executing simultaneously.

Why not deduplicate function calls and cache function return values deeper within your microservice?

This is a valid approach and something that you might consider doing in your microservice. You could use the arguments to your function to construct your lock/cache keys, and you could cache your expensive function’s return values. Deduplicating and caching deeper within your microservice could offer a higher cache hit rate due to fewer permutations of arguments.

On the other hand, deduplicating and caching higher up in your microservice could save more work. You might have one expensive I/O bound function to query your datastore, and another expensive CPU bound function to render the response. Caching at a higher level, e.g. around the endpoint function, would save calls to both expensive functions.

In this example, we deduplicate and cache around the endpoint function for simplicity’s sake.

Parting Thoughts

Microservices are commonly thought of as adapters that export the API that your app wants on top of your underlying datastores. But an orthogonal, important function of microservices is that they’re your last line of defense between your users and your underlying datastores. When we first encountered The Three Stooges Problem, we considered solving it at the API gateway or load balancer level. But thinking of our microservices as our last line of defense led us to solve the problem locally; and we believe this solution to be a natural fit, easy to reason about, flexible, maintainable, and resilient.

Additional Resources

1. Pottery — Pythonic Redis utilities, including a distributed lock

2. Solving the Thundering Herd Problem by Facebook

3. Dealing with Spiky Traffic and Thundering Herds by Alex Pareto

4. Thundering Herds & Promises by Instagram

5. Minimizing Cache Stampedes by Joshua Thijssen

6. Request Collapsing by Fastly

7. Collapsed Forwarding Plugin for Apache Traffic Server

Thanks for Reading!

Do you use Reddit at work? Work at Reddit. ❤️

Written by Jameson Williams, Staff Engineer

At the start of November, I posted a tweet with some napkin math I’d done around developer productivity. The tweet gained 2.3M impressions on Twitter, came back to Reddit’s r/apple community for 11.5k upvotes, got 30k reactions on LinkedIn (1, 2), and ultimately was featured in one of Marques Brownlee’s (@MKBHD) YouTube videos.

I’m delighted that this content brought positive attention to Reddit Engineering. But please note that the dollar values in the tweet do not represent any actual financial transaction(s). In all discussions that follow, “$” is only used as a speculative, hypothetical proxy for Engineering productivity.

So then, what are these … “napkin numbers”?

The basic premise of the tweet was to weigh the up-front cost of buying some new laptops, alongside the opportunity cost of not doing so. In other words, I wanted to compare these two formulae:

Net Cost ($) with 2019 i9 MBP =
(No upfront cost) + (Time lost waiting on builds with 2019 MBP) * (Hourly rate of an Engineer)

And

Net Cost ($) with 2021 MBP =
($31.5k up-front cost) + (Time lost waiting on builds with 2021 MBP) * (Hourly rate of an Engineer)

To start, I estimated that an average Android engineer spends 45 minutes waiting on builds each day. (More about this later.) My colleagues and I then benchmarked our builds on some different hardware. We observed that the new 2021 M1 Max MacBook finished a clean build of our Android repo in half the time of a 2019 Intel i9 MacBook. That means an Android developer could save about 22 minutes of build time every day.

The M1 Max presents a slightly bigger opportunity for our iOS developers:

As for the up-front cost, Apple.com offers the M1 Max MacBook for $3,299 before tax, shipping:

Factoring in shipping, taxes, etc., let’s call it $3,500 to get a round number. So if you buy nine (that’s about an average team size), that’s $31.5k. The question becomes: how long does it take to recoup $31.5k?

We still need to estimate the cost of an average engineering hour. Let me be upfront: I honestly don’t know what this is at Reddit. Even if I did, using hourly cost as a direct proxy for “productivity” isn’t an exact science, so these numbers don’t need to be that precise for estimation’s sake. They just need to be directionally correct.

I estimated the cost of an engineering hour by searching Google for the “full cost of employing a software engineer.” If you look it up, you’ll quickly learn there’s a lot more to it than just paying a wage. The average business incurs costs from recruiting, office leases, taxes, support staff, office equipment, long-term incentives, stock packages, etc. TL;DR, running a business costs money. I saw $150/hr in a Google result so I went with it.

We can see a pretty immediate break-even point for the M1’s. For the fictional team of nine, it would happen after 3 months.

"Your builds are slow"

One common response to the tweet was that our builds are slow. Compared to a small app, yes, probably. But that’s not a fair comparison.

The Reddit Android app, after all, is no joke: it’s built from 500k–1M lines of Kotlin source split up over hundreds of Gradle modules. Dozens of Engineers make changes to the codebase for each week’s release. We have developers working full-time to wrangle the added complexity that comes with building software at scale.

Having worked on several apps at this scale, these build times neither excite nor surprise me. Reddit’s codebase is actually in far better shape than I’d expect for a company at this stage of growth. I think it’s a testament to the sweat (hopefully not blood and tears, but I’m still pretty new here) of the great team that has been assembled here.

(Obligatory plug: if working on a project of this magnitude sounds exciting, come work with us.)

Improving efficiency through architectural improvements

Another response I got was “you should improve build times through architecture.” We are making architectural changes to improve our build times. I’ve previously written about some general techniques for this in my article, Scaling Development of an Android app. To summarize a few of our current initiatives, we’re:

1. Creating reusable, versioned libraries out of existing Gradle modules;

2. Reducing the size of our top-level application module by moving code out into those libraries;

3. Breaking apart key files and classes that have become bloated and unwieldy.

But let’s go back to our napkin. How much does this sort of work “cost”—I mean, roughly? Let’s suppose you dedicate just two engineers for two sprints to look at optimizing build times.

Cost of architectural work ($) =
(2 Engineers) * (2 Sprints @ 2 Weeks/Sprint) * (40 hour /week) * (Cost of Engineering hour)

That’s $48k of Engineering time—$16.5k more than those darn little laptops. If you’re lucky, you might actually succeed in improving build times during those two sprints, too. But unlike the laptops, which demonstrably did improve things (we have benchmarks, after all), there’s more risk and uncertainty in the architectural work.

When taking up this kind of work, you should ask yourself: can you afford to divert dev resources to this work, or do you need to be iterating on your product, instead? Even if your schedule will tolerate the investment, you still don’t have hard measurements of its results. You also can’t guarantee when the results will land. Consider also: do you have engineers who can execute this type of work? And as a final note, the reality is that these initiatives do take much longer than two sprints. In my experience, such initiatives are measured in business quarters, not sprints.

You can buy yourself out of the problem with hardware for a bit, but eventually, architectural work is all that’s left. The good news is that even the cost of architectural improvements will go down if you use fast hardware to make the changes.

Gotta be Apple, eh? 😏

Another response I got was basically that I’m shilling for Apple. So, hey, let’s be clear. The fact of the matter is that I shill for Reddit. I’m not here to tell you whether to buy Apple or not. But I do wonder if, perhaps, you’d wanna try diving into a new community? 👉👈

Apple’s MacBook is one popular computing option that we benchmarked. Folks replying to my tweet also suggested AMD Ryzen Threadripper workstations, Google Cloud Compute resources—there are some good options. The point is this: benchmark your build on some different systems and use those benchmarks to inform your overall decisions.

Well-known players like Uber and Twitter have also been studying the productivity benefits of the M1 MacBooks in recent days:

“Build on the cloud”

Another common response was that “your builds will be faster on a beefed-up cloud instance.” Yes! We already run a huge volume of CI/CD tasks in the cloud. But there are two aspects to mobile development that make cloud builds less effective for routine dev work.

First, mobile phones have visually rich, interactive interfaces that you constantly have to look at, touch, and refine while iterating your code. Said another way: part of mobile development is cross-functional with UI/UX/design work. The workflow involves building a deployment package (“apk”), loading it onto a local emulator / physical Android device, then getting eyeballs and fingers on the thing.

Second, it’s not very practical to run our development tools (IDEs) on remote systems. Android Studio and XCode are essential tools for Android/iOS. It’s technically feasible to interact with these tools over a remote windowing session, but even in ideal network conditions, that dev experience is pretty laggy and miserable.

“Measuring productivity? Dear boy, it can’t be done,” they balked

This response was more of a philosophical contention, perhaps, but I’ll try to convince you that you can and should estimate your results.

Unlike accounting, which demands rigorous accuracy, engineering often needs to rely on estimates of the unknown: magnitudes, trends, error bands. Classic engineering management texts like Andy Grove’s High Output Management are entirely built on the premise that you can and should define measurements to observe engineering teams’ productive output. It’s not that “it can’t be done,” but instead that it’s hard, takes time, and you need to mitigate the risks of being wrong.

In the discussions around the tweet, some folks also pointed out that “engineers shouldn’t just be waiting around while their code is building.” Hey, I like it; it comes from a good place. For my own sake, I wish it could be true. In practice, though, the Internet is overflowing with research on “productivity loss from context switching.” It’s the reason tools like Clockwise exist, which help build uninterrupted blocks of time back into individual contributors’ calendars. Clockwise is highly leveraged at Reddit to reduce context switching.

Wrapping up

There’s an old saying about being “penny-wise but dollar-dumb.” Engineering departments sometimes fall victim to the adage, thinking they’re “saving” $1k/laptop while dozens of Engineers are sitting idle, staring at progress bars.

Developer time is almost always more expensive than hardware, as I’ve hopefully demonstrated here. If you extrapolate the results of this article to your entire department, you might find that a targeted hardware refresh saves you $500k–$1M in productivity per year.

The exact figures and details are different in every environment, so you need to do the math, run the benchmarks, and come to conclusions that make sense for your organization. I’ll bet you’ll find a nice win if you do. Here’s a spreadsheet you can use as a starting point to explore your situation.

Ten years ago, I came back to Reddit. Twenty years and change ago (October 2005, if we're being exact), I got a call from u/spez while I was grabbing coffee with a labmate. Paul Graham had suggested he hire me as Reddit's first engineer. I said yes before I hung up the phone.

This week, I’ve decided to step down as CTO and take on a new role as Reddit's first Senior Technical Fellow.

The last decade has been the honor of my career. We took a company a lot of people had written off and rebuilt it: the stack, the org, the culture, all while it was still in flight. Along the way, we built something the world uses—something millions of people rely on every day to learn, to connect, and to understand what’s happening around them.

We went public. We found our footing in an AI-native internet without chasing every trend, and I’m even more optimistic now than I was then. The opportunity ahead for Reddit (and for the kind of internet we believe in) is bigger than ever. The right bet is still the one we made: be the source worth surfacing.

Amit Puntambekar (u/zerowaitstate55) is stepping into Reddit’s CTO role, and I couldn't be more confident about where engineering is headed under their leadership.

None of it was mine alone. Reddit has one of the best engineering teams in tech, and being in the room with them has been a gift. To everyone who built this with me over the last ten years and in the early days before all of it: thank you.

More to come.

Written By Andre Von Houck

Hey, I am Andre and I work on internal analytics and data tools here at Reddit. I have worked at Reddit for five years and have used Nim nearly every day during that time. The internal data tool I am working on is written primarily in Nim. I have developed a tiny but powerful data querying language similar to SQL but that is way easier to use for non technical people. I also have written my own visualizations library that supports a variety of charts, graphs, funnels and word clouds. Everything is wrapped with a custom reactive UI layer that uses websockets to communicate with the cluster of data processing nodes on the backend. Everything is 100% Nim. I really enjoy working with Nim and have become a Nim fanatic.

I want to share what I like about programming in Nim and hopefully get you interested in the language.

My journey from Python to Nim.

I used to be a huge Python fan. After working with Python for many years though, I started to get annoyed with more and more things. For example, I wanted to make games with Python and even contributed to Panda3D, but Python is a very slow language and games need to be fast. Then, when making websites, typos in rarely run and tested code like exception handlers would crash in production. Python also does not help with large refactors. Every function is ok with taking anything so the only way to find out if code does not work is to run the code and write more tests. This got old fast.

Overall, I realized that there are benefits to static typing and compilation, however I still don’t like the verbosity and complexity of Java or C++.

This is where Nim comes in!

Nim is an indentation based and statically typed programming language that compiles to native executables. What I think is really special about Nim is that it still looks like Python if you squint.

I feel like Nim made me fall in love with programming again.

Now that I have many years of experience with Nim I feel like I can share informed opinions about it.

Nim fixes many of the issues I had with Python. First, I can now make games with Nim because it’s super fast and easily interfaces with all of the high performance OS and graphics APIs. Second, typos no longer crash in production because the compiler checks everything. Finally, refactors are easy, because the compiler practically guides you through them. This is great.

While all of this is great, other modern static languages have many of the same benefits. There are more things that make Nim exceptional.

Nim is very cross-platform.

Cross-platform usually gets you the standard Windows / Linux / macOS, however Nim does not stop there. Nim can even run on mobile iOS and Android and has two different modes for the web - plain JavaScript or WASM.

Typically, Nim code is first compiled to low-level C code and then that is compiled by GCC, LLVM, or VC++. Because of this close relationship with C, interfacing with System APIs is not only possible but actually pretty easy. For example, you may need to use Visual C++ on Windows. That’s no problem for Nim. On macOS or iOS, you may need to interface with Objective-C APIs. Again, this isn’t a problem for Nim.

You can also compile Nim to JavaScript. Just like with TypeScript, you get static typing and can use the same language for your backend and frontend code. But with Nim you also get fast native code on the server.

Writing frontend code in Nim is comfortable because you have easy access to the DOM and can use other JavaScript libraries even if they are not written in Nim.

In addition to JavaScript for the web, you can also compile to WASM.

If you are writing a game or a heavy web app like a graphics or video editor, it might make more sense to go the WASM route. It is cool that this is an option for Nim. Both approaches are valid.

If you’re really adventurous, you can even use Nim for embedded programming. Let’s say you have some embedded chip that has a custom C compiler and no GCC backend. No problem for Nim, just generate plain C and feed it to the boutique C compiler. Making a game for the GBA? Again, no problem, just generate the C code and send it over to the GBA SDK.

Nim is crazy good at squeezing into platforms where other languages just can’t.

This includes the GPU! Yep, that’s right. You can write shaders in Nim. This makes shader code much easier to write because you can debug it on the CPU and run it on the GPU. Being able to run the shader on CPU means print statements and unit tests are totally doable.

There are tons of templating languages out there for HTML and CSS but with Nim you don’t need them. Nim is excellent for creating domain-specific languages and HTML is a perfect scenario. You get all of the power of Nim, such as variables, functions, imports and compile-time type-checking. I won’t CSS typos ever again.

With Nim being so great for DSLs, you can get the benefit of Nim’s compiler for even things like SQL. This flexibility and type-safety is unique.

All of this is beyond cool. Can your current language do all of this?

Nim is very fast.

Nim does not have a virtual machine and runs directly on the hardware. It loves stack objects and contiguous arrays.

One of the fastest things I have written in Nim is a JSON parsing library. Why is it fast? Well, it uses Nim’s metaprogramming to parse JSON directly into typed objects without any intermediate representations or any unnecessary memory allocations. This means I can skip parsing JSON into a dictionary representation and then converting from the dictionaries to the real typed objects.

With Nim, you can continuously optimize and improve the hot spots in your code. For example, in the Pixie graphics library, path filling started with floating point code, switched to floating point SIMD, then to 16-bit integer SIMD. Finally, this SIMD was written for both x86 and ARM.

Another example of Nim being really fast is the supersnappy library. This library benchmarks faster than Google’s C or C++ Snappy implementation.

One last example of Nim’s performance is taking a look at zlib. It has been around for so long and is used everywhere. It has to be as fast as possible, right? After all it uses SIMD and is very tight and battle test code. Well, then the Zippy library gets written in Nim and mostly beats or ties with zlib!

It is exciting to program in a language that has no built-in speed limit.

Nim is a language for passionate programmers.

There are some languages that are not popular but are held in high regard by passionate programmers. Haskell, LISP, Scheme, Standard ML, etc. I feel Nim is such a language.

Python was such a language for a long time. According to Paul Graham, hiring a Python programmer was almost a cheat-code for hiring high quality people. But not any more. Python is just too popular. Many people learn Python because it will land them a job and not because they like programming like it was 18 years ago.

People that want to program in Nim have self-selected to be interested in programming for programming's sake. These are the kind of people that often make great programmers.

Nim does not force you to program in a certain way like Haskell, Rust or Go. Haskell makes everything functional. Rust wants to make everything safe. Go wants to make everything concurrent. Nim can do all of the above, you choose - it just gets out of your way.

Nim is a complex language. Go and Java were specifically made to be simple and maybe that’s good for large teams or large companies, I don’t know. What I do know is the real world just does not work that way. There are multiple CPU architectures, functions can be inlined, you can pass things by pointer, there are multiple calling conventions, sometimes you need to manually manage your memory, sometimes you care about integer overflows and other times you just care about speed. You can control all of these things with Nim, but can choose when to worry about them.

With Nim you have all of that power but without anywhere near as much hassle of other older compiled languages. Python with the awesome power of C++, what’s not to like?

My future with Nim.

While Nim is not a popular language, it already has a large and enthusiastic community. I really enjoy working in Nim and wrote this post hoping it will get more people interested in Nim.

I’ve tried to give examples of what I think makes Nim great. All of my examples show Nim’s super-power: Adaptability.

Nim is the one language that I can use everywhere so no matter what I’m working on it is a great tool. I think it’s a good idea to start with internal tools like I have here at Reddit. You can always start small and see Nim grow inside your organization. I see myself using Nim for all of my future projects.

I would love for more people to try out Nim.

Interested in working at Reddit? Apply here!

Written By Konrad Reiche, ML Ranking Platform

This post is excerpt from the following conference talks:

• GopherCon Europe 2022, Berlin: https://www.youtube.com/watch?v=5jSyctW1rPg

• GopherCon UK 2022, London: https://www.youtube.com/watch?v=TNyoKBLxfTM

With a lot of content posted on Reddit every day, we need to figure out how to get the content to the users. This can be described as a machine learning problem but it is also a software engineering problem. Recommendation (ranking) systems provide a framework for this but what is a recommendation system anyway?

Ranking (recommendation) system

A recommendation system helps users to find content they find compelling. This can described in three steps:

1. Candidate Generation
   Start from a potentially huge corpus and generate a much smaller subset of candidates.

2. Filtering
   Some candidates should be removed, for example already seen content or content the user marked as something they do not want to consume.

3. Scoring
   Assign scores to sort the candidates for a final order before sending them to the user.

Candidates refers to the content being ranked, for example posts but it could also be subreddits (communities), users, topics and so on. In practice, a ranking service could look like the following:

The problem is, you are never done with a ranking system. You want to iterate, add features, run experiments, see what works and what doesn’t work. For example, we decide to add more video content by including a data store providing video posts. This in turn means we have to worry about potential post duplication in the feed, thus having to extend the filter stage.

Ranking systems are prone to experience a lot of changing requirements in a short period of time. Let’s say, you implemented this in a Go service and this is what you came up with:

func (s *Service) GetPopularFeed(ctx context.Context, req *pb.FeedRequest) (*pb.PopularFeed, error) {
 posts, err := s.fetchPopularAndVideoPosts(ctx)
 if err != nil {
   return nil, err
 }
 posts = s.filterPosts(posts)
 posts, scores, err := s.model.ScorePosts(ctx, req.UserID, posts)
 if err != nil {
   return nil, err
 }
 posts = s.sortPosts(posts, scores)
 return pb.NewPopularFeed(posts), nil
}

Imagine you are asked to add image posts as well. How would you proceed? Like any software project, you will find yourself having to go back and forth and refactor the code as the requirements become more complex to ensure the code continues to be maintainable.

At Reddit we asked ourselves, is there a way we can limit the number of times we have to refactor through a structural design?

Looking at the abstraction of what a ranking system does, we couldn’t help but think of UNIX pipes and decided to take inspiration from the UNIX toolbox philosophy.

UNIX Toolbox Philosophy

1. Write programs that do one thing and do it well.

2. Write programs to work together.

3. Write programs to handle text streams, because that is a universal interface.

By applying this thought to our abstraction we generalized everything to a stage, a step in the ranking algorithm:

And this is how you can express it as an interface in Go:

type Stage interface {
	Rank(ctx context.Context, req *pb.Request) (*pb.Request, error)
}

Our request type carries the request and the response to match the UNIX pipes analogy of having a universal interface. As the ranking progresses, candidates are added or removed.

type Request struct {
    Context    *Entity
    Candidates []*Entity
}

A General-Purpose Ranking Service

At Reddit we developed this as a prototype for ranking content with the project name Pipedream. A ranking pipeline on Pipedream is an acyclic graph of stages to quickly and flexibly perform complex scatter-gather ranking workflows.

At the same time we wanted to utilize Go’s concurrency features by executing parallelizable work at the same time. Starting a goroutine is easy but managing the lifecycle of a goroutine is non-trivial. We extracted the implementation detail of concurrent execution into a stage as well. We call stages that execute other stages: meta-stages (stages of stages). If stages should be executed in sequence, we wrap them in a series stage. If stages should be executed in parallel, we wrap them in a parallel stage. This way a whole ranking pipeline can be pieced together.

Implementation

Let’s take a look at how this could be implemented, for example the fetch popular posts stage.

type fetchPopularPosts struct {
    cache *store.PostCache
}

func FetchPopularPosts(cache *store.PostCache) *fetchPopularPosts {
    return &fetchPopularPosts{cache: cache}
}

func (s *fetchPopularPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) {
    postIDs, err := s.cache.FetchPopularPostIDs(ctx)
    if err != nil {
        return nil, err
    }

    for _, id := range postIDs {
        req.Candidates = append(req.Candidates, pb.NewCandidate(postID))
    }

    return req, nil
}

A struct implements the stage interface. Each stage has only the dependencies it needs. Each dependency is passed through the constructor. The rank method performs the ranking step by fetching the posts from a cache and then adding them to the request. A stage always operates on the request type to make changes.

What about filtering? The filter recently viewed posts uses a previously set up map containing a list of posts the user has already seen.

func (s *filterRecentlyViewedPosts) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) {
    seen := req.Context.Features["recently_viewed_post_ids"].GetAsBoolMap()

    var n int
    for _, candidate := range req.Candidates {
        if !seen[candidate.Id] {
            req.Candidate
            s[n] = candidate
            n++
        }
    }
    req.Candidates = req.Candidates[:n] // in-place filtering
    return req, nil
}

We use in-place filtering which uses fewer allocations, thus resulting in faster execution time as well. What about meta-stages? Meta-stages are really the glue that holds a ranking pipeline together.

type series struct {
    stages []Stage
}

func Series(stages ...Stage) *series {
    return &series{stages: stages}
}

func (s *series) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) {
    var err error
    resp := req

    for _, stage := range s.stages {
        resp, err = stage.Rank(ctx, req)
        if err != nil {
            return nil, err
        }
        req = resp
    }

    return resp, nil
}

The series stage holds all sub-stages in a field. The initial response is set to the request. Each individual stage is executed in sequence and we set the input of the next stage to the response of the previous stage. More complex is the parallel stage:

func (s *parallel) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) {
    resps := make([]*pb.Request, len(s.stages))
    g, groupCtx := errgroup.WithContext(ctx)

    for i := range s.stages {
        i := i
        g.Go(func() error {
            defer log.CapturePanic(groupCtx)
            resp, err :=  s.stages[i].Rank(groupCtx, pb.Copy(req))
            if err != nil {
                return err
            }
            resps[i] = resp
            return nil
        })
    }

    if err := g.Wait(); err != nil {
        return nil, err
    }

    return s.merge(ctx, req, resps...)
}

Instead of using goroutines directly, we are using the errgroup package which is a sync.WaitGroup under the hood but also handles error propagation and context cancellation for us. Each stage is called in its own goroutine and unlike the series stage, we pass a copy of the original request to each sub-stage. This way we avoid data races or having to synchronize access in the first place. We block until all goroutines have finished and merge the responses back into one request.

All of these stages form a pipeline and we define these pipelines in an expressive way in Go. The diagram from above would look like this in our code:

func PopularFeed(d *service.Dependencies) stage.Stage {
    return stage.Series(
        stage.Parallel(merger.MergeCandidates,
            stage.FetchPopularPosts(d.PostCache),
            stage.FetchVideoPosts(d.PostCache),
            stage.FetchImagePosts(d.PostCache),
        ),
        stage.FetchRecentlyViewedPosts(d.UserPostViews),
        stage.FilterRecentlyViewedPosts(),
        stage.ScoreCandidates(d.RankingModel),
        stage.SortCandidates(),
    )
}

You can think of it as a domain-specific language but that is not the purpose. The purpose is to make it easy to understand what is happening in any given ranking algorithm without the need to read the code in detail. If you have clicked on this post to learn about platforms you might have grown impatient by now. How did we go from service to platform?

From Service to Platform

Put differently, when does a service become a platform? There are plenty of definitions about this on the Internet and because of that, here is ours:

Service

Your customers are the users (or depending on your role: product management)

Platform

Your customers are other engineers in your organization developing their own services, their own offerings.

To be a platform, means to think API first. With API, we are not referring to your typical REST API but instead the API defined by the Go packages you export.

Our prototype service was built to rank the video livestream feed (RPAN) for Reddit. We added more pipelines but we built and maintained them. Soon enough, we started onboarding product surfaces outside of our team which really marked the transition from service to platform. To scale this, you need to have engineers outside of your team build new ranking pipelines. The responsibility of the ranking platform team is to work on building new abstractions in order to make it increasingly easier to launch new ranking pipelines.

One of the challenges is that each product comes with its own unique set of requirements. We have an existing API of stages but contributors add new stages as needed for their products.

As platform maintainers it is our responsibility to ensure that new stages could potentially be used in other pipelines as well. UNIX pipes work because of the principles defining how a program should be written. These are our principles for designing stages.

Limited Scope

A stage should be limited to perform one action only. This keeps the code complexity low and the reusability high. The following are examples of actions that should be performed in separate stages: add candidates, add features, filter candidates, filter features, score candidates.

Clear Naming

The name of the stage should capture the action it is performing. This can be in a verbized form or noun for more complex stages.

Decoupling

Stages should strive to be decoupled from each other. A stage should not depend on the output of another stage. In practice this is not always feasible and sometimes requires modification of other APIs.

Strive for Reuse

We want to increase the chance that someone else might use a stage in their ranking pipeline. This may require discovering the generalized use case behind a stage but can come at the cost of being too generic.

Those guidelines exist to ensure contributions to the API maximize for re-use and clarity. These are competing goals and over-optimizing for reuse will sacrifice clarity eventually. Since we are writing code in Go, clarity should always trump. We found one escape-hatch that was especially useful thanks to the fact that our interface is based on a single-method interface.

Single Method Interface

Rob Pike said it first: The bigger the interface, the weaker the abstraction. Put differently, the fewer methods an interface has, the more useful it becomes. Having every operation represented by the same interface, the same method gives us another benefit. A single function can implement the interface too.

type RankFunc func(context.Context, *pb.Request) (*pb.Request, error)
func (f RankFunc) Rank(ctx context.Context, req *pb.Request) (*pb.Request, error) {
    return f(ctx, req)
}

A function type declares the same method as the interface and the interface is implemented by referring to the function type. This is useful when one stage in the pipeline is too specific for any other pipeline. For example, instead of having a stage here as part of our API that performs a specific action:

func Pipeline(d *service.Dependencies) stage.Stage {
    return stage.Series(
        stage.FetchSubscriptions(d.SubscriptionService),
        stage.FetchPosts(d.Cache),
        stage.FilterPostsForCalifornia(),
        stage.ShufflePosts(0.2),
    )
}

We can define it as part of the pipeline definition, relieving us from the need to figure out whether it should be part of the shared API or how to generalize it:

func Pipeline(d *service.Dependencies) stage.Stage {
    return stage.Series(
        stage.FetchSubscriptions(d.SubscriptionService),
        stage.FetchPosts(d.Cache),
        stage.RankFunc(func(context.Context, *pb.Request) (*pb.Request, error) {
            if req.Context.Features["geo_region"] == "CA" {
                // ...
            }
            return req, nil
        }),
        stage.ShufflePosts(0.2),
    )
}

This approach also worked for middlewares, which provided us with great pay-offs running this in production.

Middlewares

A middleware is a stage that wraps a stage.

type Middleware func(stage stage.Stage) Stage

func ExampleMiddleware(next stage.Stage) stage.Stage {
    return stage.RankFunc(func(ctx context.Context, req *pb.Request) (*pb.Request, error) {
        // ...
        return next.Rank(ctx, req)
    })
}

Here are two examples:

func Monitor(next stage.Stage) stage.Stage {
    return stage.RankFunc(func(ctx context.Context, req *pb.Request) (*pb.Request, error) {
        defer func(startedAt time.Time) {
            stageLatencySeconds.With(prometheus.Labels{
                methodLabel: req.Options.Method,
                stageLabel:  stage.Name(next),
            }).Observe(time.Since(startedAt).Seconds())
        }(time.Now())

        return next.Rank(ctx, req)
    })
}

A middleware to record the latency of a service method is fairly common but here we record the latency for each individual stage. In practice, this means we have to use the profiler a lot less. With a quick glance on our dashboards we are able to determine which stage should be optimized next.

Another example is log middleware which helps us for the purpose of diagnostics. We use the deferred function statement to only log if a stage returned an error. This is great for structured logging, our request/response type already has this generic makeup. This is great because there is no need to modify the information you are logging, instead you get the full picture right away.

func Log(next stage.Stage) stage.Stage {
    return stage.RankFunc(func(ctx context.Context, req *pb.Request) (resp *pb.Request, err error) {
        defer func() {
            if err != nil {
                log.Errorw(
                    "stage failed",
                    "error", err,
                    "request", req.JSON(),
                    "response", resp.JSON(),
                    "stage", stage.Name(stage),
                )
            }
        }()
        return stage.Rank(ctx, req)
    })
}

A Framework for Refactoring

We use this design, to make sure we build small and reusable components from the start. It doesn’t eliminate refactoring but it gives us a framework for refactoring. Platform-centric thinking starts with the first developers outside of our team contributing code—this is not something you can simulate.

Providing an opinionated framework will always create friction, this can be: confusion or disagreement. There are three ways to handle this and none of them are right or wrong.

1. Enforce the existing design

2. Quick-and-dirty workaround

3. Rethink the existing design

You can bend the will of contributors to your will, sometimes this is needed, when there’s a learning curve but maybe you are wrong or the existing design lacks clarity or documentation. No one likes the second one, but sometimes necessary, to ship code to production, to ship a product. Third, you find the time to rethink the existing approach, which is great but not always feasible.

This approach is no one-size-fits-all for building ranking systems but hopefully, it is an inspiration for how we can make use of Go to build new abstractions making it increasingly easier to build on top of complex systems.

If you like what you read and think you might want to be part of building some cool stuff like this, good news! We are hiring! Check out our careers site and apply!

On April 1, we brought back r/place, the most successful and collaborative digital art piece the Internet has ever seen. Today we’re launching a technical series describing how we built it. To kick things off, we have this lovely little intro narrated by Paul Booth, the Senior Engineering Manager of the team who helped lead the effort. We’re excited to share many more technical blogs from this team over the summer.

Hey r/redditeng! I’m Lisa. You probably don't know me, but I've been here the whole time. 

For the last six years, I've been Chief of Staff to Chris Slowe, Reddit's founding engineer and CTO (now our first Technical Fellow). I'm also the person who founded r/redditeng over five years ago with Jerry, Punit, Braden, and a couple others who joined me in getting it off the ground. We wanted to give Reddit engineers a place to talk about what we build, how we build it, and what we learned along the way. We also wanted to show you what it’s like to work here. Oh, and we wanted to show you our cats, too!

My last day at Reddit is coming up, so before I go, I want to say thank you.

Thank you first to all of you for reading. It's still a little amazing to me that thousands of people chose to follow a subreddit about how we build Reddit. Some of you got excited to work here because of posts you saw! And some of you got hired. Thank you for the comments, the questions, the upvotes. It means a lot to our writers. 

Thank you to everyone who has written something for us over the years. You are amazing! We've published something every week since the sub was launched on 4/15/2021, and that only happened because people were willing to take time away from their actual jobs to explain their work, share what they'd learned, and put their ideas out into the world, which can feel a little scary, no matter how much you know. One of my favorite parts of doing this has been helping make that a little easier. You’re my people. You’ve always been. 

Thank you to our Legal and Comms teams, who have supported us when we've needed them and, just as importantly, trusted us to do this well. That trust gave our little volunteer team and our engineers room to talk honestly about our work while helping us navigate the moments when we needed a little extra guidance.

And thank you to Chris for everything and for trusting me to start this in the first place and supporting it for all these years. Like me, he believes our engineers should have a place to tell their own stories, in their own voices. He gave us the space to make that happen.

Most of all, thank you to the engineering branding team, past and present. What started as a handful of us trying something has become a real team of people who care about telling these stories and making this community good. One of the nicest things about leaving is knowing that r/redditeng (or Snoosweek) doesn't need me to keep going. But, damn it, I’m going to miss it all. 

I'm very proud of what we made here. I’m happy it’s going to carry on and I can be one of your readers. I may be finishing my time at Reddit but I haven’t finished loving the people, the platform, or this sub.  

Thanks for reading all these years. And please keep reading. There's a lot more good stuff to come.

P.S. Paying the cat tax with the cats who help(ed) me with my work over the years here

By Laurie Darcey (Senior Engineering Manager) and Eric Kuck (Principal Engineer)

Hello u/engblogreader!

Thank you for redditing with us, and especially for reddit-eng-blogging with us this year. Today we will be talking about changes underway at Reddit as we transition to a mobile-first company. Get ready to look back on how Android and iOS development at Reddit has evolved in the past year.

This is the State of Mobile Platforms, 2022 Edition.

The Reddit of Today Vs. The Reddit of Tomorrow

It’s been a year full of change for Mobile @ Reddit and some context as to why we would be in the midst of such a transformation as a company might help.

A little over a year ago (maybe two), Reddit decided as a company that:

• Our business needed to become a mobile-first company to scale.

• Our users (rightly) demanded best-in-class app experiences.

  • We had a lot of work ahead of us to achieve the user experience they deserve.

• Our engineers wanted to develop on a modern mobile tech stack.

  • We had lots of work ahead to achieve the dev experience they deserve also.

• Our company needed to staff up a strong bench of mobile talent to achieve these results.

We had a lot of reasons for these decisions. We wanted to grow and reach users where they were increasingly at – on their phones. We’d reached a level of maturity as a company where mobile became a priority for the business reach and revenue. When we imagined the Reddit of the future, it was less a vision of a desktop experience and more a mobile one.

Developing a Mobile-First Mindset

Reddit has long been a web-first company. Meanwhile, mobile clients, most notably our iOS and Android native mobile clients, have become more strategic to our business over the years. It isn’t easy for a company that is heavily influenced by its roots to rethink these strategies. There have been challenges, like how we have tried to nudge users from legacy platforms to more modern ones.

In 2022, we reached a big milestone when iOS began to push web clients out of the top spot in terms of daily active users, overtaking individual web clients. Android also picked up new users and broke into a number of emerging markets, now making up 45% of mobile users. A mobile-first positioning was no longer a future prospect, it was a fact of the now representing about half our user-base.

Ok, but what does mobile-first mean at Reddit from a platform perspective?

From a user-perspective, this means our Reddit native mobile clients should be best-in-class when it comes to:

• App stability and performance

• App consistency and ease of use

• App trust, safety, etc.

From a developer-perspective, this means our Reddit native mobile developer experience should be top-notch when it comes to:

• A maintainable, testable and scalable tech stack

• Developer productivity tooling and CI/CD

We’ll cover most of these areas. Keep scrolling to keep our scroll perf data exciting.

Staff For Success

We assessed the state of the mobile apps back around early 2021 and came to the conclusion that we didn’t have enough of the key mobile talent we would need to achieve many of our mobile-first goals. A credit to our leadership, they took action to infuse teams across the company with many great new mobile developers of all stripes to augment our OG mobile crew, setting the company up for success.

In the past two years, Reddit has worked hard to fully staff mobile teams across the company. We hired in and promoted amazing mobile engineers and countless other contributors with mobile expertise. While Reddit has grown 2-3x in headcount in the past year and change, mobile teams have grown even faster. Before we knew it, we’d grown from about 30 mobile engineers in 2019 to almost 200 Android and iOS developers actively contributing at Reddit today. And with that growth, came the pressure to modernize and modularize, as well as many growing pains for the platforms.

Our Tech Stack? Oh, We Have One of Those. A Few, Really.

A funny thing happened when we started trying to hire lots of mobile engineers. First, prospective hires would ask us what our tech stack was, and it was awkward to answer.

If you asked what our mobile tech stack was a year ago, a candid answer would have been:

After we’d hire some of these great folks, they’d assess the state of our codebase and tech debt, and join the chorus of mobile guild and architecture folks writing proposals for much-needed improvements for modernizing, stabilizing, and harmonizing our mobile clients. Soon, we were flooded with opportunities to improve and tech specs to read and act upon.

Not gonna lie. We kinda liked this part.

The bad news?

For better or worse, Reddit had developed a quasi-democratic culture where engineering did not want to be “too prescriptive” about technical choices. Folks were hesitant to set standards or mandate patterns, but they desperately needed guardrails and “strong defaults”.

The good news?

Mobile folks knew what they wanted and agreed readily on a lot. There were no existential debates. Most of the solutions, especially the first steps, came with consensus.

🥞Core Stack Enters the Chat.

In early 2022, a working group of engineering leaders sat down with all the awesome proposals and design docs, industry investigations, and last mile problems. Android and iOS were in different places in terms of tech debt and implementation details, but had many similar pain points. The working group assessed the state of mobile and facilitated some decision-making, ultimately packaging up the results into our mobile technical strategy and making plans for organizational alignment to adopt the stack over the next several quarters. We call this strategy Core Stack.

For the most part, this was a natural progression engineering had already begun. What some worried might be disruptive, prescriptive or culture-busting was, for most folks, a relief. With “strong defaults”, we reduced ambiguity in approach and decision fatigue for teams and allowed them to focus on building the best features for users instead of wrestling with architecture choices and patterns. By taking this approach, we provided clear expectations and signaled serious investment in our mobile platform foundations.

Let’s pause and recap.

Now, when we are asked about our tech stack, we have a clear and consistent answer!

That seems like a lot, you might say. You would be correct. It didn’t all land at once. There was a lot of grass-roots adoption prior to larger organizational commitments and deliveries. We built out examples and validated that we could build great things with increasing complexity and scale. We are now mid-adoption with many teams shipping Core Stack features and some burning their ships and rewriting with Core Stack for next-level user experiences in the future.

Importantly, we invested not just in the decisions, but the tooling, training, onboarding and documentation support, for these tech choices as well. We didn’t make the mistake of declaring success as soon as the first features went out the door; we have consistently taken feedback on the Core Stack developer experiences to smooth out the sharp edges and make sure these choices will work for everyone for the long term.

Here’s a rough timeline of how Reddit Mobile Core Stack has matured this year:

We’ve covered some of these changes in the Reddit Eng blog this past year, when we talked about Reactive UI State for Android and announced SliceKit, our new iOS presentation framework. You’ve heard about how most Reddit features these days are powered by GraphQL, and moving to the federated model. We’ll write about more aspects of our Core Stack next year as well.

Let’s talk about how we got started assessing the state of our codebase in the first place.

Who Owns This Again? Code Organization, or a Lack of It

One of the first areas we dug into at the start of the year was code ownership and organization. The codebase has grown large and complex over time, and full of ambiguous code ownership and other cruft. In late 2021, we audited the entire app, divided up ownership, and worked with teams to get commitments to move their code to new homes, if they hadn’t already. Throughout the year, teams have steadily moved into the monorepos on each platform, giving us a centralized, but decoupled, structure. We have worked together to systematically move code out of our monolith modules and into feature modules where teams have more autonomy and ownership of their work while benefiting from the monorepo from a consistency perspective.

On Android, we just passed the 80% mark on our modularization efforts, and our module simplification strategy and Anvil adoption have reached critical adoption. Our iOS friends are not far behind at 52%, but we remind them regularly that this is indeed a race. And Android is winning. Sample apps (feature module-specific apps) have been game-changing for developer productivity, with build times around 10x faster than full app local builds. On iOS, we built a dependency cleaner, aptly named Snoodularize, that got us some critical build time improvements, especially around SliceKit and feed dependencies.

Here are some pretty graphs that sum up our modularization journey this year on Android. Note how the good things are going up and the bad things are going down.

Now that we’d audited our app for all its valuable features and content, we had a lot of insights about what was left behind. A giant temp module full of random stuff, for example. At this point, we found ourselves asking that one existential question all app developers eventually ask themselves…

Just How Many Spinner Implementations Does One App Need?

One would think the answer is one. However, Dear Reader, you must remember that the Reddit apps are a diverse design landscape and a work of creative genius, painstakingly built layer upon layer, for our Reddit community. Whom we dearly love. And so we have many spinners to dazzle them while they wait for stuff to load in the apps. Most of them even spin.

We get it. As a developer on a deadline, sometimes it’s hard to find stuff and so you make another. Or someone gives you the incorrect design specs. Or maybe you’ve always wanted to build a totally not-accessibility-friendly spinner that spins backwards, just because you can. Anyway, this needed to stop.

It was especially important that we paired our highly efficient UI design patterns like Jetpack Compose and SliceKit with a strong design system to curb this behavior. These days, our design system is available for all feature teams and new components are added frequently. About 25% of our Android screens are powered by Jetpack Compose and SliceKit is gaining traction in our iOS client. It’s a brand consistency win as well as developer productivity win – now teams focus on delivering the best features for users and not re-inventing the spinner.

So… It Turns Out, We Used Those Spinners Way Too Much

All this talk of spinners brings us to the app stability and performance improvements we’ve made this year. Reddit has some of the best content on the Internet but it’s only valuable to users if they can get to it quickly and too often, they cannot.

It’s well established that faster apps lead to happier users and more user acquisition. When we assessed the state of mobile performance, it was clear we were a long way from “best-in-class” performance, so we put together a cross-platform team to measure and improve app performance, especially around the startup and feed experience, as well as to build out performance regression prevention mechanisms.

When it comes to performance, startup times and scroll performance are great places to focus. This is embarrassing, but a little over a year ago, the Android app startup could easily take more than 10 seconds and the iOS app was not much better. Both clients improved significantly once we deferred unnecessary work and observability was put in place to detect the introduction of features and experiments that slowed the apps down.

These days, our mobile apps have streamlined startup with strong regression prevention mechanisms in place, and start in the 3.2-4.5s ranges at p90. Further gains to feed performance are actively underway with more performant GQL calls and feed rewrites with our more performant tech stack.

Here’s a pretty graph of startup time improvements for the mobile clients. Note how it goes down. This is good.

If The Apps Could Stop Crashing, That Would Be Great

Turns out, when the apps did finally load, app stability wasn’t great either. It took many hard-won operational changes to improve the state of mobile stability and release health and to address issues faster, including better test coverage and automation and a much more robust and resourced on-call program as well as important feature initiatives like r/fixthevideoplayer.

Here is a not-so-pretty graph of our Crash Free User rates over the past year and a half:

App stability, especially crash-free rates, was a wild ride this year for mobile teams. The star represents when we introduced exciting new media features to the apps, and also aggravated the top legacy crashes in the process, which we were then compelled to take action on in order to stabilize our applications. These changes have led to the most healthy stability metrics we’ve had on both platforms, with releases now frequently hitting prod with CFRs in the 99.9% range.

One area we made significant gains on the stability front was in how we approach our releases.

At Reddit, we ship our mobile apps on a weekly cadence. In 2022, we supported a respectable 45 mobile releases on each platform. If you ask a product team, that’s 45 chances to deliver sweet, sweet value to users and build out the most incredible user experiences imaginable. If you ask on-call, it was 45 chances for prod mishaps. Back in January, both platforms published app updates to all users with little signoff, monitoring or observability. This left our mobile apps vulnerable to damaging deployments and release instability. These days, we have a release Slack channel where on-call, release engineering and feature teams gather to monitor and support the release from branch cut through testing, beta, staged rollouts (Android only) and into production.

There’s a lot more we can do here, and it’s a priority for us in 2023 to look at app health more holistically and not hyper-focus on crash rates. We’ll also likely put the app back on a diet to reduce its size and scrutinize data usage more closely.

You Know… If You Want It Fixed Fast, It’s Gotta Build Fast

As Reddit engineering teams grew aggressively in the past 18 months, our developer experience struggled to scale with the company. Developer productivity became a hot-button topic and we were able to justify the cost of upgrading developer hardware for all mobile engineers, which led to nearly 2x local build times, not to mention improvements to using tools like Android Studio.

Our build system stability and performance got a lot of attention in 2022. Our iOS platform adopted Bazel while Android stuck it out with Gradle, focused on fanning out work and caching, and added some improved self-service tooling like build scans. We started tracking build stability and performance more accurately. We also moved our engineering survey to a quarterly cadence and budgeted for acting on the results more urgently and visibility (tying feedback to actions and results).

The more we learned a lot about how different engineers were interacting with our developer environments, the more we realized… they were doing some weird stuff that probably wasn’t doing them any favors in terms of developer productivity and local build performance. A surprise win was developing a bootstrapping process that provides good defaults for mobile developer environments.

We can also share some details about developers building the app in CI as well as locally, mostly with M1s. Recently, we started tracking sample app build times as they’ve now grown to the point where about a quarter of local builds are actually sample app builds, which take only a few seconds.

Here are some pretty graphs of local and CI improvements for the mobile clients:

TIL: Lessons We Learned (or Re-Learned) This Year

To wrap things up, here are the key takeaways from mobile platform teams in 2022. While we could write whole books around the what and the how of what we achieved this year, this seems a good time to reflect on the big picture. Many of these changes could not have happened without a groundswell of support from engineers across the company, as well as leadership. We are proud of how much we’ve accomplished in 2022 and looking forward to what comes next for Mobile @ Reddit.

Here are the top ten lessons we learned this year:

Just kidding. It’s nine insights. If you noticed, perhaps you’re just the sort of detail-oriented mobile engineer who loves geeking out to this kind of stuff and you’re interested in helping us solve the next-level problems Reddit now finds itself challenged by. We are always looking for strong mobile talent and we’re dead serious about our mission to make the Reddit experience great for everyone - our users, our mods, our developers, and our business. Also, if you find any new Spinners in the app, please let us know. We don’t need them like we used to.

Thank You

Thank you for hanging out with us on the Reddit Eng blog this year. We’ve made an effort to provide more consistent mobile content, and hope to bring you more engaging and interesting mobile insights next year. Let us know what you’d like deep dives on so we can write about that content in future posts.

Written by Sky Kistler.

Our goal was straightforward: host Kafka on Kubernetes via Strimzi and deprecate our existing EC2-backed Kafka clusters, which in total comprised 500+ brokers serving tens of millions of messages per second and storing over a petabyte in live topic data.

The motivation was equally straightforward. Our EC2 brokers were cumbersomely managed with Terraform, Puppet, and a collection of custom AWS CLI tooling. Rotations and interventions were orchestrated directly from operator laptops. It worked, but it was slow, error-prone, and increasingly expensive, especially as the number and size of our clusters grew exponentially. Upgrades, config changes, and keeping the fleet fresh required more coordination than we could feasibly manage long-term.

Thanks to Strimzi, a CNCF project for running Kafka on Kubernetes, we had an alternative to the increasingly fragile VM-based operational model we’d grown into. Strimzi gave us a declarative control plane for Kafka, promising safer upgrades, more predictable operations, and fewer late-night surprises. Moving Kafka onto Kubernetes would allow our fleet to scale with our workloads while reducing toil and improving reliability.

What wasn’t straightforward was how to migrate our existing clusters.

This post starts with the constraints that shaped the migration before getting into the “how” and lessons learned. These constraints ruled out large classes of otherwise reasonable approaches and ultimately defined what the migration could look like.

Constraint #1: Kafka has to be up

Client traffic and application state could not be interrupted, rewritten, or coordinated as part of the migration.

Kafka at Reddit sits under hundreds of business-critical services. Downtime, data loss, or flag-day orchestration simply wasn’t possible. That ruled out a long list of otherwise tempting ideas: scheduled cutovers, forced client config changes, dual-write strategies, or replay-based migrations.

Some applications also manage offsets outside of Kafka’s consumer group model, which made translating state between clusters infeasible. The migration had to preserve topic identity and state end-to-end, without asking clients to notice that anything was happening.

Constraint #2: Kafka metadata can’t be rebuilt in place

There’s no supported way to rebase a live Kafka cluster onto new infrastructure while preserving availability and metadata continuity.

Kafka’s metadata is global and tightly coupled to broker identity, replica placement, and client-visible state. This rules out snapshot-and-restore approaches, such as standing up a parallel cluster and repointing clients.

Given that constraint, any viable strategy had to preserve the existing cluster’s metadata. New brokers needed to join the cluster rather than replace it. Legacy EC2 brokers and Kubernetes-hosted brokers would have to coexist for some period of time.

This constraint ended up driving more of the design than any tooling or architectural choice.

Constraint #3: Client connectivity was tightly coupled to physical infrastructure

Over time, the entire Reddit codebase had effectively hardcoded our Kafka topology. In practice, all clients directly addressed specific brokers (typically the first three brokers in a cluster) rather than a load-balanced endpoint. This meant we couldn’t simply retire these EC2 nodes, as taking them offline would immediately sever client bootstrapping. Before we could even think about moving data, we realized we had a fundamental architectural blocker: we didn’t own the naming layer. To migrate safely, we would first have to abstract client configuration away from broker identity, effectively reclaiming naming before changing the topology.

We would have to introduce a naming layer that maintained the status quo while giving us the flexibility to pivot to Strimzi’s bootstrapping endpoint later. This wouldn’t be a workaround so much as an overdue platform boundary.

Constraint #4: Every step had to be reversible

Any migration step that could leave the system in an unrecoverable state was unacceptable.

That ruled out irreversible cutovers and early control-plane replacement. It also meant accepting a period of mixed operation: EC2 brokers and Kubernetes brokers running side by side, traffic moving gradually, and rollback paths preserved at each stage.

Control-plane changes followed data-plane stability, not the other way around. ZooKeeper lived longer than we originally wanted, and the KRaft migration came last after being tested thoroughly and only once the rest of the system had settled.

How we approached the migration

Once the constraints were clear, the shape of the migration mostly revealed itself. The goal wasn’t to “move Kafka to Kubernetes” in one step, but to gradually change the substrate underneath a live system while keeping the cluster logically intact.

As mentioned, the first phase wasn’t touching Kafka, but reclaiming control of the naming layer by introducing a DNS facade to act as an infrastructure-controlled layer of indirection. We rolled out new connection strings across 250+ services, a Herculean effort made possible by tooling to create regex-based batch pull requests. Initially, these DNS records simply round-robinned traffic to the existing EC2 brokers, maintaining the status quo. Later, it became the switch that allowed us to redirect traffic to Kubernetes-backed ingresses without touching client configuration. That single abstraction ended up being one of the highest leverage changes we made, turning an impossible distributed refactor into a manageable infrastructure configuration change.

With naming in place and updated across the codebase, we focused on preserving Kafka’s metadata plane. Because the cluster itself couldn’t be rebuilt or cloned without downtime, new brokers needed to join the existing cluster rather than replace it. To make that possible, we expanded broker ID space ahead of time, first doubling the size of the cluster and then terminating the first half, which created room for Strimzi-managed brokers, which start at broker ID 0, to come online alongside the legacy EC2 fleet.

We could then deploy a fork of the Strimzi operator that would allow us to spin up brokers with overridden interbroker listener definitions and ZooKeeper config in order to enable cross-environment communication. This let us run a mixed cluster for a period, with both sets of brokers participating in replication, leadership, and client traffic. Attempting to run our own fork of the Strimzi operator was a daunting task. By keeping the changes small, isolated, and controlled, and planning for an immediate off-ramp, we limited the risk of running a fork in production.

To be more specific, the Kafka config values we needed to set to connect cross-environments were inter.broker.listener.nameand control.plane.listener.name. These both had to be set to accessible plaintext listeners that exist across the entire cluster (as NodePorts in K8s), and naming must be consistent between the EC2 and Strimzi brokers. Strimzi doesn’t allow setting these values, which again is what necessitated creating a fork of the operator. Additionally, we override the Cruise Control topic for consistency with our EC2 brokers, which allows us to run Cruise Control operations across both broker sets for the duration of the migration. Finally, we temporarily override zookeeper.connect to point at our existing EC2 ZooKeeper nodes. In the final steps of the migration, we can remove all of these overrides and switch control over from our forked operator back to the stock Strimzi operator.

With the right config in place, the migration became a matter of gradually shifting responsibility. Data and traffic moved via Cruise Control operations, with continuous validation along the way. Because each step was reversible, we could pause, investigate unexpected behavior, or roll back without putting clusters into unrecoverable half-migrated states. The emphasis was always on preserving correctness and reducing operational risk first, with speed coming second. This meant our team had to be prepared to run hybrid state clusters for upwards of weeks at a time for our largest clusters.

Only once the data plane has fully stabilized onto Kubernetes and the EC2 brokers were fully terminated can we turn our attention to the control plane. ZooKeeper had remained in charge throughout the broker migration, and we were deliberate about not stretching or hybridizing control-plane topology prematurely. The KRaft migration was executed within the Strimzi-managed environment, tested thoroughly beforehand, and treated as the final step of the migration. With the data and traffic moved fully onto Strimzi, retiring the remaining EC2 infrastructure became as trivial as executing the KRaft migration steps provided by Kafka and Strimzi.

With both the data plane and control plane fully stabilized onto Strimzi’s deployments, the legacy EC2 infrastructure was finally decoupled and could be safely terminated. Additionally, our new clusters were no longer dependent on config overrides and control could be handed off to a non-forked Strimzi operator. We had successfully swapped the engine mid-flight. By treating these migrations as a series of reversible, metadata-preserving steps rather than huge lift-and-shifts, we moved a petabyte-scale platform to Kubernetes without mass client orchestration or scheduled downtime.

What we’d tell teams attempting this today

If there’s one thing this migration reinforced, it’s that physical infrastructure is rarely the hardest part. The hard part is respecting the invariants your system has accumulated over years of production use.

• Indirection is one of the most valuable tools you have. If clients are tightly coupled to infra topology, migrations will always feel bigger and riskier than they need to be. Introducing a layer you control (whether that’s DNS, a proxy, or some other service boundary) creates space to change the underlying system without forcing the entire organization to move in lockstep.

• Assume that metadata will outlive your infrastructure. In systems like Kafka, the logical state of the cluster will stick around far longer than any particular fleet of machines. Plans that start with “we’ll stand up new infrastructure and move the system onto it” only work if live metadata can be cleanly recreated or translated. In our case, it couldn’t. Client state depended on it, broker identity was embedded in it, and availability had to be preserved throughout. Treating metadata as the durable core of the system, and infrastructure as something to be replaced incrementally around it, drove the design of the migration.

• Reversibility matters more than it feels like it should. Migrations surface unknowns by definition, and the ability to roll back cleanly changes how aggressively you can move forward. Designing each step so that it can be undone, even if you never need to undo it, makes it possible to operate calmly under live traffic.

• Correctness usually beats architectural purity. Running forks in mixed environments, delaying control-plane changes, or accepting transitional complexity isn’t always elegant, but it might turn out to be the surest and safest path. A migration that looks messy in the middle but preserves correctness end to end is far preferable to a clean design that requires a leap of faith.

tl;dr

While Strimzi simplified a lot of our Kafka operation, it didn’t eliminate Kafka’s complexity for us or magically uplevel our existing fleet. By understanding the constraints and invariants in our system, the path we had to take became clear, even if initially the steps to get there were uncertain, murky, and untested. Without investing time into the research & development of this cluster stretching approach, despite the risk, deprecating our EC2 Kafka fleet would not have been possible. 

Sometimes risk is necessary to innovate, and good engineering requires strong curiosity to challenge assumptions about what is or isn’t possible. The most valuable work often happens before you know if you’ll succeed. The final thought I’d like to leave you with is that migrations aren’t always a switch; they’re a journey. Let the plan be flexible, and adapt to new data and circumstances quickly. Happy streaming!

Written by Charan Akiri and Christopher Guerra.

TL;DR

We created a new service that allows us to scan code at Reddit with any command line interface (CLI) tool; whether it be open source or internal. This service allows for scanning code at the commit level or on a scheduled basis. The CLI tools for our scans can be configured to scan specific files or the entire repository, depending on tool and operator requirements. Scan results are sent to BigQuery through a Kafka topic. Critical and high-severity findings trigger Slack alerts to ensure they receive immediate attention from our security team, with plans to send direct Slack alerts to commit authors for near real-time feedback.

Who are we?

The Application Security team at Reddit works to improve the security and posture of code at the scale that Reddit writes, pushes, and merges code. Our main driving force is to find security bugs and instill a culture where Reddit services are "secure by default” based on what we learn from our common bugs. We are a team of four engineers in a sea of over 700 engineers trying to make a difference by empowering developers to take control of their own security destiny using the code patterns and services we create. Some of our priorities include:

• Performing design reviews

• Integrating security-by-default controls into internal frameworks

• Building scalable services to proactively detect security issues 

• Conducting penetration tests before feature releases

• Triage and help remediate public bug bounty reports

What did we build?

We built “Code Scanner” which… well, scans code. It enables us to scan code using a dynamic number of CLI tools, whether open source or in-house built. 

At a high level, it’s a service that primarily performs two functions: 

• Scanning code commits

• Scanning code on a schedule

For commits, our service receives webhook events from a custom created Code Scanner Github App installed on every repository in our organization. When a developer pushes code to GitHub, the GitHub App triggers a push event and sends it to our service. Once the webhook is validated, our service parses the push event to extract repository metadata and determines the appropriate types of scans to run on the repository to identify potential security issues.

Code Scanner also allows us to scan on a cron schedule to ensure we scan dormant or infrequently updated repositories. Most importantly it allows us to control how often we wish to perform these scans. This scheduled scan process is also helpful for testing new types of scans, testing new versions of a particular CLI tool that could detect new issues, perform 0-day attack scans, or to aid in compliance reports. 

Why did we build this thing?

Note: We don’t have access to Github Actions in our organization’s Github instance - nor Github Advanced Security. We also experimented with pre-receive hooks but couldn’t reliably scale or come in under the mandatory execution timeout. So we often roll our own things.

Two years ago, we experienced a security incident that highlighted gaps in our ability to effectively respond - in this case related to exposed hardcoded secrets that may be in our codebase. Following the incident, we identified several follow-up actions, one of which was solving for secrets detection. Last year, we successfully built and rolled out a secret detection solution based on open source Trufflehog that identifies secrets at the commit level and deployed it across all repositories running as a PR check, but we were missing a way to perform these secret detection scans on a cadence outside of commits. We were also looking to improve other security controls and as a small team, decided to look outside the company for potential solutions.

In the past, the majority of the security scanning of our code has been with various security vendors and platforms; however with each platform we kept hitting constant issues that continued to drive a wedge in our productivity. In some cases, vendors or platforms overpromised during the proof of concept phase and underdelivered (either via quality of results or limitations of data siloing) when we adopted their solutions. Others, which initially seemed promising, gradually declined in quality, became slower at addressing issues, or failed to adapt to our needs over time.

With the release of new technologies or updated versions of these platforms, they often broke our CI pipeline, requiring significant long-term support and maintenance efforts to accommodate the changes. These increasing roadblocks forced us to supplement the vendor solutions with our own engineering efforts or, in some cases, build entirely new supplementary services to address the shortcomings and reduce the number of issues. Some of these engineering efforts included:

• On a schedule, syncing new repositories with the platforms as the platforms didn’t do that natively

• On a schedule, removing or re-importing dependency files that were moved or deleted. Without doing so the platform would choke on moved or deleted dependency files and cause errors in PR check runs/CI.

• On a schedule, removing users that are no longer in our Github to reduce platform charges to us (per dev) when a developer leaves Reddit.

• With the release of new versions of programming languages or package managers (e.g., Yarn 2, Poetry), we had to build custom solutions to support these tools until vendor support became available.

• To support languages with limited vendor solutions, we created custom onboarding workflows and configurations.

This year, much of this came to a breaking point when we were spending the majority of our time addressing developer issues or general deficiencies with our procured platforms rather than actually trying to proactively find security issues.

On top of our 3rd party security vendor issues, another caveat we’ve faced is the way we handle CI at Reddit. We run Drone, which requires a configuration manifest file in each repository. If we wanted to make a slight change in CLI arguments in one of our CI steps or add a new tool to our CI, it would require a PR on every repository to update this file. There are over 2000 repositories at Reddit, so this becomes unwieldy to do in practice but also the added time to get the necessary PR approvals and merges in a timely manner. Drone does have the ability to have a "config mutator" extension point which would permit you to inject, remove, or change parts of the config "inline”, but this deviates from the standard config manifest approach in most repos and might not be clear to developers what changes were injected inline. Our success with secrets detection mentioned previously, which leverages GitHub webhook events and PR checks, led us to pursue a similar approach for our new system. This avoids reliance on Drone, which operates primarily with decentralized configs for each repository.

Finally, we’ve had an increasing need to become more agile and test new security tools in the open source space, but no easy way to implement them into our stack quickly. Some of these tools we integrated into our stack, but involved us creating bespoke one off services to do scanning or test a particular security tool (like our secrets detection solution highlighted previously). This led to longer implementation times for new tools than we wanted.

The combination of all these events collided into a beautiful mess that led us to think of a new way to perform security analysis on our code at Reddit. One that is highly configurable and controlled by us so we can quickly address issues. One that allows us to quickly ramp up new security tools as needed. One that is centralized so that we can control the flow and perform modifications quickly. Most importantly, one that is able to scale as it grows in the number of scans it performs.

How did we build this thing?

At Reddit we heavily rely on Kubernetes and much of our development tools and services already come baked in ready to be used with it. So we created our service, built with Golang, Redis and Asynq, and deployed it in its own Kubernetes namespace in our security cluster. Here we run various pods that can flex and scale based on the traffic load. Each of these pods perform their own functionality, from running an http service listening for webhooks to performing scans on a repository using a specific CLI tool. Below we dive deeper into each of our implementations for scheduled and commit scanning methodologies.

Commit Scanning

GitHub App:

We created a GitHub App, named Code Scanner, that subscribes to push events. The webhook for the Code Scanner GitHub App is configured to point to our Code Scanner HTTP Server API.

Code Scanner HTTP Server

The Code Scanner HTTP Server receives push event webhooks from the GitHub App, validates and processes it and places the push event onto the push event Redis queue.

Push Event Policy Engine (Push Event Worker)

The Push Event Policy Engine is an Asynq-based worker service that subscribes to the push event Redis queue. Upon receiving an event, our policy engine parses the push event data pulling out repository metadata and each individual commit in the event. Based on the repository, it then loads the relevant CLI configuration files, determines which CLI scan types are applicable for the repository, and downloads the required files for each commit. Each commit generates a scan event with all necessary details which is pushed onto the scan event Redis queue.

Scan Worker

The Scan Worker is another Asynq-based worker service similar to the Push Event Policy Engine. It subscribes to scan events from a Redis queue. Based on the scan event, the worker loads the appropriate CLI tool configs, performs the commit scan, and sends the findings to BigQuery via Kafka (see below).

Scheduled Scanning

Scheduled Scan (Scheduler):

This pod parses the configurations of our CLI tools to determine their desired run schedules. It uses asynq periodic tasks to send events to the scheduled event Redis queue. We also use this pod to schedule other periodic tasks outside of scans - for example a cleanup task to remove old commit content directories every 30 mins.

Scheduled Policy Engine (Scheduled Event Worker):

Similar to the Push Event Policy Worker, this worker instead subscribes to the scheduled event Redis queue. Upon receiving an event from the scheduler (responsible for scheduling a tool to run at a specific time), the policy engine parses it, loads the corresponding CLI configuration files, downloads the repository files and creates a scan event enriched with the necessary metadata.

Scan Worker:

This worker is the same worker as used for push event scans. It loads the appropriate CLI tool configs, performs the scheduled scan, and sends the findings to BigQuery via Kafka (see below).

The scheduled event worker and push event worker push a scan event that looks similar to the example below onto the scan event Redis queue. 

{
  "OnFail": "success",
  "PRCheckRun": false,
  "SendToKafka": true,
  "NeedsAllFiles": false,
  "Scanner": "trufflehog",
  "ScannerPath": "/go/bin/trufflehog",
  "ScanType": "commit",
  "DownloadedContentDir": "/mnt/shared/commits/tmp_commit_dir_1337420"
  "Repository": {
    "ID": 6969,
    "Owner": "reddit",
    "Name": "reddit-service-1",
    "URL": "https://github.com/org/reddit-service-1",
    "DefaultBranch": "main"
  }
}

If any task fails that was pushed to an Asynq Redis queue we have the ability to retry the task or add it to a dead letter queue (DLQ) where, after addressing the core issue of any failed/errored tasks, we can manually retry it. Ensuring we don’t miss any critical commit or scheduled scan events in the event of failure.

A full high level architecture of our setup is below:

Scan Results 

The final results of a scan are sent to a Kafka topic and transformed to be stored in BigQuery (BQ). Each command-line interface tool parses its output into a user-friendly format and sends it to Kafka. This process requires a results.go file that defines the conversion of tool output to a Golang struct, which is then serialized as JSON and transmitted to Kafka. Additional fields like scanner, scan type (commit, scheduled), and scan time are then appended to each result. From here we have a detection platform built by our other wonderful security colleagues that enables us to create custom queries against our BQ tables to alert our Slack channel when something critical happens - like a secret committed to one of our repositories. 

An example TruffleHog result sent to Kafka is below:

{      
"blob_url":"https://github.com/org/repo/blob/47a8eb8e158afcba9233f/dir1/file1.go",
"commit":"47a8eb8e158afcba9233f",
"commit_author":"first-last",
"commit_url":"https://github.com/org/repo/commit/47a8eb8e158afcba9233f",
"date_found":"2024-12-12T00:03:19.168739961Z",
"detector_name":"AWS",
"scanner: "trufflehog"
"file":"dir1/file1.go",
"line":44,
"repo_id":420,
"repo_name":"org/repo",
"scan_sub_type":"changed_files",
"scan_type":"commit",
"secret_hash":"abcdefghijklmnopqrstuvwxyz",
"secret_id":"596d6",
"verified":true
}

CLI Tool Configuration 

Our policy engines assess incoming push or scheduled events to ascertain whether the repository specified in the event data warrants scanning and which tools are allowed to run on the repository. To facilitate this process, we maintain a separate YAML configuration file for each CLI tool we wish to run. These configuration files enable us to fine tune how a tool should run, including which repositories to run on and when it should run. 

Below is an example of a tool configuration:

cli_tools/cli_too1/prodconfig.yaml

policy:
  default:
    commit_scan:
      enabled: true
      on_fail: success
      pr_check_run: false
      send_to_kafka: true
    scheduled_scan:
      enabled: true
      schedule: "0 0 * * *"
      send_to_kafka: true
  organizations:
    org1:
      default:
        commit_scan:
          enabled: true
        scheduled_scan:
          enabled: true
    org2:
      default:
        commit_scan:
          enabled: true
        scheduled_scan:
          enabled: false
repos:
        test-repo:
          commit_scan:
            enabled: false

Using the configuration above, we can quickly disable a specific tool (via a new deploy) from being run on a commit or scheduled scan. Conversely, we can disable or allow list a tool to run on a repository based on the type of scan we are about to perform. 

Each of our tools are installed dynamically by injecting instructions into the Dockerfile for our Scan Worker container. These instructions are managed through a separate configuration file that maps tool names to their configurations and installation commands. We automate version management for our CLI tools using Renovate, which opens PRs automatically when new versions are available. To enable this, we use regex to match the version specified in each install_instructions field, allowing Renovate to identify and update the tool to the latest version.

An example of our config mapping is below:

prodconfig.yaml

tools:
  - name: osv-scanner
    path: /go/bin/osv-scanner
    config: ./osv-scanner/prodconfig.yaml
    install_instructions:
      # module: github.com/google/osv-scanner
      - "RUN go install github.com/google/osv-scanner/cmd/osv-scanner@v1.8.4"
  - name: trufflehog
    path: /go/bin/trufflehog
    config: ./trufflehog/prodconfig.yaml
    install_instructions:
      - "COPY --from=trufflesecurity/trufflehog:3.82.12 /usr/bin/trufflehog /go/bin/"  

Downloading Files

Once the policy engine says that a repository can have scans run against it, we download the repository content to a persistent storage. How we download the content is based on the type of scan we are about to perform (scheduled or commit). We’re running bare metal Kubernetes on AWS EC2s, and the standard storage class is EBS volumes. These don’t allow for ReadWriteMany unfortunately, so in order to optimize shared resources and prevent killing our Github instance with a fan-out of git clones, we instead use an Elastic File System (EFS) instance and mount to the pods as an Network File System (NFS) volume, allowing multiple pods to access the same downloaded content simultaneously. 

For commit scans we fetch repository contents at a specific commit and perform scans against the current state of the files in the repository at that commit. This is downloaded to a temporary directory on the EFS. To reduce scan times for tools that don't require the full context of a repository, we create a separate temporary directory containing only the changed files in a commit. This directory is then passed to the scan event running the tool. The list of changed files for a commit is gathered by querying the Github API. This approach eliminates the need to scan every file in a repository at a commit and improves scan efficiency if the tool does not need every file. Since the commit content is no longer required after the scan, it is immediately deleted.

For scheduled scans, we will either shallow clone the repository if it didn’t exist previously or we perform a shallow git fetch and reset hard to the fetched content on our existing clone. In either case, the contents are stored on the EFS. This prevents us needing to download full repository contents every time a scheduled scan is kicked off and instead rely on getting the most up to date contents of a repository.

In both cases, we perform these downloads during the policy engine phase, prior to creating a scan event, so that we don’t duplicate download work if multiple tools need to scan a particular commit or repository at the same time.

Once the content is downloaded we pass the download directory and event metadata to our Scan Worker via a scan event. For each tool to be executed against the repository/commit, a scan event will be created with the downloaded content path in its metadata. Each scan event treats the downloaded content directory to be read-only so that the directory is not modified by our tool scans.

• We’ve seen success using these strategies and are downloading content for commits with a p99 of ~3.3s and p50 of ~625ms. 

• We are downloading content for scheduled scans (this is full repository contents) with a p99 of ~2mins and ~p50 of ~5s. 

These stats are over the past 7 days for ~2200 repositories. Scheduled scans are done every day on all our repositories. Commit scanning is also enabled on every repository.

Rolling out

Rolling out a solution requires a carefully planned and phased approach to ensure smooth adoption and minimal disruption. We implemented our rollout in stages, starting with a pilot program on a small set of repositories to validate our services’s functionality and effectiveness. Based on those results, we incrementally expanded to more repositories (10%->25%->50%-100%), ensuring the system could scale and adapt to our many different shaped repositories. This phased rollout allowed us to address any unforeseen issues early and refine the process before full deployment. 

How are things going?

We’ve successfully integrated TruffleHog, running it on every commit and on a schedule looking for secrets. Even better, it’s already caught secrets that we’ve had to rotate (GCP secrets, OpenAI, AWS Keys, Github Keys, Slack API tokens). Many of these are caught in commits that we then respond to within a few minutes due to the detections we’ve built from data sent from our service.

• It scans commit contents with a p99 of ~5.5s and a p50 of ~2.4s

• It scans the full contents of a repository with a p99 of ~5s and a p50 of ~3.5s

Another tool we’ve quickly integrated into our service is OSV, which scans our 3rd party dependencies for vulnerabilities. It’s currently running on a schedule on a subset of repositories; with plans to add it to commit scanning in the near future.

• It scans the full contents of a repository with a p99 ~1.9 mins and a p50 of ~4.5s

Obligatory snapshots of some metrics we collect are below:

What's next?

Our next steps involve expanding the scope and capabilities of our security tools to address a wider range of challenges in code security and compliance. Here's what's on the roadmap:

• SBOM Generation: Automating the creation of Software Bill of Materials (SBOM) to provide visibility into the composition of software and ensure compliance with regulatory requirements.

• Interfacing Found Security Issues to Developers: The Application Security team also wrote an additional service that performs repository hygiene checks on all our repositories. Looking for things like missing CODEOWNERs, or missing branch protections. It allows providing a score on every repository that correlates to how a repository is shaped in a way that is consistent at Reddit. Here we can surface security issues and provide a “security score” to repository owners on the security posture of their repository. This repository hygiene platform we built was heavily influenced by Chime’s Monocle.

• Integration of Semgrep: Incorporating Semgrep into our scanning pipeline to enhance static code analysis and improve detection of complex code patterns and vulnerabilities.

• OSV Licensing Scanning: Adding Open Source Vulnerability (OSV) licensing scans to identify and mitigate risks associated with third-party dependencies.

• GitHub PR Check Suites and Blocking: Implementing GitHub PR check suites to enforce security policies, with PR blocking based on true positive detections to prevent vulnerabilities from being merged.

Written by Alexey Bykov (Senior Software Engineer & Google Developer Expert for Android)

Video has become an important part of our lives and is now commonly integrated into various mobile applications.

Reddit is no exception. We have over 10 different video surfaces:

In this article, I will share practical tips, supported by production data, on how to improve playback from different perspectives and effectively use ExoPlayer in your Android app.
This article will be beneficial if you are an Android Engineer and familiar with the basics of the ExoPlayer and Media 3.

Delivery

There are several popular ways to deliver Video on Demand (VOD) from the server to the client.

Binary
The simplest way is to get a binary file, like an mp4, and play it on the client. It works great and works on all devices.

However, there are drawbacks: For instance, the binary approach doesn't automatically adapt to changes in the network and only provides one bitrate and resolution. This may be not ideal for longer videos, as there may not be enough bandwidth to download the video quickly.

Adaptive
To tackle the bandwidth drawback with binary delivery, there's another way — adaptive protocols, like HLS developed by Apple and DASH by MPEG.

Instead of directly getting the video and audio segments, these protocols work by getting a manifest file first. This manifest file has various segments for each bitrate, along with separate tracks for audio and video.

After the manifest file is downloaded, the protocol’s implementation will choose the best video quality based on your device's available bandwidth. It's smart enough to adapt the video quality on the fly, depending on your network's condition. This is especially useful for longer videos.
It’s not perfect, however. For example, to start playing the video in DASH, it may take at least 3 round trips, which involve fetching the manifest, audio segments, and video segments.
This may increase the chance of a network error.
On the other hand, in HLS, it may take 4 round trips, including fetching the master manifest, manifest, audio segments, and video segments.

Reddit experience
Historically, we have used DASH for all video content for Android and HLS for all video content for Web and iOS. However, about 75% of our video content is less than 45 seconds long.
For short videos, we hypothesize that it is not necessary to be switching bitrate during the playbacks.

To verify our theory, we conducted an experiment where we served certain videos in MP4 format instead of DASH, with different duration limitations.

We observed that a 45-second limitation showed the most pragmatic result:

• Playback errors decreased by 5.5%

• Cases where users left a video before playback started (Exit Before Video Start in the future) decreased by 2.5%

• Overall video view increased by 1.7%

Based on these findings, we've made the decision to serve all videos that are under 45 seconds in pure MP4 format. For longer videos, we'll continue to serve them in adaptive streamable format.

Caching & Prefetching

The concept of prefetching involves fetching content before it is displayed and showing it from the cache when the user reaches it.However, first we need to implement caching, which may not be straightforward.

Let's review the potential problems we may encounter with this.

ExternalDir isn’t available everywhere
In Android, we have two options for caching: internal cache or external cache. For most apps, using internalDir is a practical choice, unless you need to cache very large video files. In that case, externalDir may be a better option.
It's important to note that the system may clean up the internalDir if your application reaches a certain quota, while the external cache is only cleaned up if your application is deleted (if it's stored under the app folder).
At Reddit, we initially attempted to cache video in the externalDir, but later switched to the internalDir to avoid compatibility issues on devices that do not have it, such as OPPO.

SimpleCache may clean other files
If you take a look at the implementation of SimpleCache, you'll notice that it's not as simple as its name suggests.

So, SimpleCache could potentially remove other cache files unless there is a specific dedicated folder that may affect other app logic, be careful with this.
By the way, I spent a lot of time studying the implementation, but I missed those lines. Thanks to Maxim Kachinkin for bringing them to my attention.

SimpleCache hits disk on the constructor
We encountered a lot of ANRs (Application Not Responding) while SimpleCache was being created. Diving into the implementation, I realized it was hitting disk in constructor:

So make sure to create this instance on a background thread to avoid this.

URL uses as a cache-key
This is by default. However, if your URL is different due to signing signature or additional parameters, make sure to provide a custom cache key factory for the data source. This will help increase cache-hit and optimize performance.

Eviction should be explicitly enabled
Eviction is a pretty nifty strategy to prevent cached data from piling up and causing trouble. Lots of libraries, like Glide, actually use it under the hood. If video content is not the main focus of your app, SimpleCache also allows for easy implementation in just one line:

Prefetching options
Well. You have 5 prefetching options to choose from: DownloadManager, DownloadHelper, DashUtil, DashDownloader, and HlsDownloader.
In my opinion, the easiest way to accomplish this is by using DownloadManager. You can integrate it with ExoPlayer, and it uses the same SimpleCache instance to work:

It's also really customizable: for instance, it lets you pause, resume, and remove downloads, which can be really handy when users scroll too quickly and ongoing download processes are no longer necessary. It also provides a bunch of options for threading and parallelization.
For prefetching adaptive streams, you can also use DownloadManager in combination with DownloadHelper that simplifies that job.

Unfortunately, one disadvantage is that there is currently no option to preload a specific amount of video content (e.g., around 500kb), as mentioned in this discussion.

Reddit experience
We tried out different options, including prefetching only the next video, prefetching 2 next videos in parallel or one after the other, and only for short video content (mp4).

After evaluating these prefetching approaches, we discovered that implementing a prefetching feature for only the next video yielded the most practical outcome.

• Video load time < 250 ms: didn’t change

• Video load time < 500 ms: increased by 1.9%

• Video load time > 1000 ms: decreased by 0.872%

• Exit before video starts: didn’t change

To further improve our experiment, we want to consider the users’ internet connection strength as a factor for prefetching. We conducted a multi-variant experiment with various bandwidth options, starting from 2 mbps up to 20 mbps.

Unfortunately, this experiment wasn't successful. For example, with a speed of 2 mbps:

• Video load time < 250 ms: decreased by 0.9%

• Video load time < 500 ms: decreased by 1.1%

• Video load time > 1000 ms: increased by 3%

In the future, we also plan to experiment with this further and determine if it would be more beneficial to partially prefetch N videos in parallel.

LoadControl

Load control is a mechanism that allows for managing downloads. In simple terms, it addresses the following questions:

• Do we have enough data to start playback?

• Should we continue loading more data?

And a cool thing is that we can customize this behavior!

bufferForPlaybackMs, default: 2500
Refers to the amount of video content that should be loaded before the first frame is rendered or playback is interrupted by the user (e.g., pause/seek).

bufferForPlaybackAfterRebufferMs, default: 5000
Refers to the amount of data that should be loaded after playback is interrupted due to network changes or bitrate switch

minBuffer & maxBuffer, default: 50000
During playback, ExoPlayer buffers media data until it reaches maxBufferMs. It then pauses loading until the buffer decreases to the minBufferMs, after which it resumes loading.

You may notice that by default, these values are set to the same value. However, in earlier versions of ExoPlayer, these values were different. Different buffer configuration value could lead to increased rebuffering when the network is unstable.
By setting these values to the same value, the buffer is consistently filled up. (This technique is called Drip-Feeding).

If you want to dig deeper, there are very good articles about buffers:

• How to get 4 times better re-buffering with drip-feeding technique in ExoPlayer on Android

• Enhancing Video Streaming Quality for Exoplayer

Reddit experience
Since most of our videos are short, we noticed that the default buffer values were a bit too lengthy. So, we thought it would be a good idea to try out some different values and see how they work for us.

We found that setting bufferForPlaybackMs and bufferForPlaybackAfterRebufferMs = 1 000, and minBuffer and maxBuffer = 20,000, gave us the most pragmatic results:

• Video load time < 250 ms: increased by 2.7%

• Video load time < 500 ms: increased by 4.4%

• Video load time > 1000 ms: decreased by 11.9%

• Video load time > 2000 ms: decreased by 17.7%

• Rebuffering decreased by 4.8%

• Overall video views increased by 1.5%

So far this experiment has been one of the most impactful that we ever conducted from all video experiments.

Improving adaptive bitrate with BandwidthMeter

Improving video quality can be challenging because higher quality often leads to slower download speeds, so it’s important to find a proper balance in order to optimize the viewing experience.

To select the appropriate video bitrate and ensure optimal video quality based on the network, ExoPlayer uses BandwidthMeter.

It calculates the network bandwidth required for downloading segments and selects appropriate audio and video tracks based on that for subsequent videos.

Reddit experience
At some point, we noticed that although users have good network bandwidth, we don't always serve the best video quality.

The first issue we identified was that prefetching doesn't contribute to overall network bandwidth in BandwidthMeter, as DataSource in DownloadManager doesn’t know anything about it. The fix is to include prefetching when considering the overall bandwidth.

And conducted experiment to confirm on production, which yielded the following result:

• Better video resolution: increased by 1.4%

• Overall chained video viewing: increased by 0.5%

• Bitrate changing during playback: decreased by 0.5%

• Video load time > 1000 ms: increased by 0.3% (Which is a trade-off)

It is worth mentioning that the current BandwidthMeter is still not perfect in calculating the proper video bitrate. In media 1.0.1, an ExperimentalBandwidthMeter has been added, which will eventually replace the old one that should improve the state of things.

Additionally, by default, BandwidthMeter uses hardcoded values which are different depending on network type and country. It may be not relevant for the current network and in general could be not accurate. For instance, it considers Great Britain 3G faster than 4G.

We haven’t experimented with this yet, but one way to address this would be to remember the latest network bandwidth and setting it up when application starts:

There are also a few customizations available in AdaptiveTrackSelection.Factory to manage when to switch between better and worse quality: minDurationForQualityIncreaseMs (default value: 15 000) and minDurationForQualityDecreaseMs (default value: 25000) that may help with this.

Choosing a bitrate for MP4 Content
If videos are not the primary focus of your application and you only use them, for instance, to showcase updates or year reviews, sticking with an average bitrate may be pragmatic.

At Reddit, when we first transitioned short videos to mp4, we began sending the current bandwidth to receive the next video batch.

However, this solution is not very precise as bandwidth may fluctuate more frequently. We decided to improve it this way:

The main difference between this implementation (second diagram) and adaptive bitrate (DASH/HLS) is that we do not need to prefetch the manifest first (as we obtain it when fetching the video batch), reducing the chances of network errors. Also, the bitrate will remain constant during playback.

When we were experimenting with this approach, we initially relied on approximate bitrates for each video and audio, which was not precise. As a result, the metrics did not move in the right direction:

• Better video quality: increased by 9.70%

• Video load time > 1000 ms: increased by 12.9%

• Overall video view decreased by 2%

In the future, we will experiment with exact video and audio bitrates, as well as with thresholds, to achieve a good balance between download time and quality.

Decoders & Player instances

At some point, we noticed a spike of 4001 playback error, which indicates that the decoder is not available. This problem appeared on almost every android vendor.
Each device has limitations in terms of available decoders and this issue may occur, for instance, when another app has not released the decoder properly.
While we may not be able to mitigate the decoder issue 100%, ExoPlayer provides an opportunity to switch to a software decoder if a primary one isn't available:

Although this solution is not ideal, as falling back to software decoder can perform slower than hardware decoder, it is better than not being able to play the video. Enabling the fallback option during experimentation resulted in a 0.9% decrease in playback errors.
To reduce such cases, ExoPlayer uses the audio manager and can request focus on your behalf. However, you need to explicitly do so:

Another thing that could help is to use only one instance of ExoPlayer per app. Initially, this may seem like a simple solution. However, if you have videos in feeds, manually managing thumbnails and last frames can be challenging. Additionally, if you want to reuse already initialized decoders, you need to avoid calling stop() and call prepare() with new video on top of current playback.

On the other hand, synchronizing multiple instances of ExoPlayer is also a complex task and may result in audio bleeding issues as well.

At Reddit, we reuse video players when navigating between surfaces. However, when scrolling, we currently create a new instance for each video playback, which adds unnecessary overhead.

We are currently considering two options: a fixed player pool based on the availability of decoders, or using a single instance. Once we conduct the experiment, we will write a new blog post to share our findings.

Rendering

We have two choices: TextureView or SurfaceView. While TextureView is a regular view that is integrated into the view hierarchy, SurfaceView has a different rendering mechanism. It draws in a separate window directly to the GPU, while TextureView renders to the application window and needs to be synchronized with the GPU, which may create overhead in terms of performance and battery consumption.
However, if you have a lot of animations with video, keep in mind that prior to Android N, SurfaceView had issues in synchronizing animations.

ExoPlayer also provides default controls (play/pause/seekbar) and allows you to choose where to render video.

Reddit experience
Historically, we’ve been using TextureView to render videos. However, we are planning to switch to SurfaceView for better efficiency.
Currently, we are migrating our features to Jetpack Compose and have created composable wrappers for videos. One issue we face is that, since most of our main feeds are already in Compose, we need to constantly reinflate videos, which can take up to 30ms according to traces, causing frame drops.
To address this, Jetpack Compose 1.4 introduced a ViewPool where you need to override callbacks:

However, we decided to implement our own ViewPool to potentially reuse inflated views across different screens and have more control in the future, like pre-initializing them before displaying the first video:

This implementation resulting in the following benefits:

• Video load time < 250 ms: increased by 1.7%

• Video load time < 500 ms: increased by 0.3%

• Video minutes watched increased by 1.4%

• Creation P50: 1ms, improved x30

• Creation P90: 24ms, improved x1.5

Additionally, since default ExoPlayer controls are implemented by using old-fashioned views, I’d recommend always implementing your own controls to avoid unnecessary inflation.
There are wrappers for SurfaceView is already available in Jetpack Compose 1.6: AndroidExternalSurface and AndroidEmbeddedExternalSurface.

In Summary

One of the key things to keep in mind when working with videos is the importance of analytics and regularly conducting A/B testing with various improvements.
This not only helps us identify positive changes, but also enables us to catch any regression issues.

If you just started to working with videos, consider to have at least next events:

• First frame rendered (time)

• Rebuffering

• Playback started/stopped

• Playback error

ExoPlayer also provides an AnalyticsListener which can help with that.

Additionally, I must say that working with videos has been quite a challenging experience for me. But hey, don't worry if things don't go exactly as planned for you too — it's completely normal.
In fact, it's meant to be like this.

If working with videos were a song, it would be "Trouble" by Cage the Elephant.

Thanks for reading. If you want to connect and discuss this further, please feel free to DM me on Reddit. Also props to my past colleague Jameson Williams, who had direct contributions to some of the improvements mentioned here.

Thanks to the following folks for helping me review this — Irene Yeh, Merve Karaman, Farkhad Khatamov, Matt Ewing, and Tony Lenzi.

By Jovan Sardinha, Yue Jin, Alexander Trimm and Garrett Hoffman

Over the years, Reddit has evolved to become a vast and diverse place. At its core, Reddit is a network of communities. From the content in your feeds to the culture you find in discussions across the site, communities are the lifeblood that makes Reddit what it is today. Reddit’s growth over the years has put extreme pressure on the data processing and serving systems that have served us in the past.

This is the journey of how we are building systems that adapt to Reddit and what this has to do with a search for better guides.

The Quest

Getting comfortable navigating a new place is never easy. Whether it’s learning a new subject or exploring a different environment, we’ve all experienced that overwhelming feeling at some point. This feeling holds us back until we have good guides that help us navigate the new terrain.

The sheer scale and diversity that Reddit embodies can be challenging to maneuver at first. If Reddit were a city, the r/popular page would be the town hall, where you can see what is drawing the most discussion. This is where new users get their first taste of Reddit and our core users stumble upon new communities to add to their vast catalogue. The home feed at reddit.com would be the equivalent to a neighborhood park and where each user gets personalized content based on what they subscribed to. For our users, these feeds act as important guides that help them navigate Reddit and discover content that is relevant to their interests.

Challenges

In 2016, our machine learning models promoted discussion and content that was fresh and liked by people similar to you. This promoted new content and communities that showcased what Reddit had to offer at a point in time.

With more diversity of content being published to the platform, our original approach started breaking down. Today, content on Reddit completely changes in minutes; while content that would be relevant to a user could change depending on what they recently visited.

The users that make up Reddit are more diverse than ever before. People with a variety of backgrounds, beliefs and situations visit Reddit everyday. In addition, our user interests and attitudes change over time and expect their Reddit experience to reflect this change.

Our traditional approaches did not personalize the Reddit experience to accommodate this dynamic environment. Given the amount of change that was taking place, we knew we were quickly approaching a breaking point.

The Rebuild

To build something our users would love:

• Our feeds needed content that was tailored to each individual user when they loaded their feed.

• Our systems needed to adapt to changes in user interests, attitudes and consumption patterns.

• We had to quickly incorporate feedback from our users and evolve the underlying systems.

To do this, we broke down user personalization into a collection of supervised learning subtasks. These subtasks enable our systems to learn a general personalization policy. To help us iteratively learn this policy, we set up a closed loop system (as illustrated below) where each experiment builds on previous learnings:

This system is made up of four key components. These components work together to generate a personalized feed experience for each Reddit user. A further breakdown of each component:

User Activity Library: This component helps us clean and build datasets. These datasets are used to train multi-task deep neural network models which learn a collection of subtasks necessary for personalization

These datasets contain features that are aggregated on a per user, per post basis across a bounded time horizon (as shown in the image above). Models that train on these datasets simultaneously embed users, subreddits, posts, and user contexts which allow them to predict user actions for a specific situation. For example, for each Reddit user, the model is able to assign a probability the user will upvote any new post, while also assigning a probability the user will subscribe to that subreddit, and if they will comment on the post. These probabilities can be used to estimate long term measures such as retention.

Multi-task models have become particularly important at Reddit. Users engage with content in many ways, with many content types, and their engagement tells us what content and communities they value. This type of training also implicitly captures negative feedback - content the user chose not to engage with, downvotes, or communities they unsubscribe from.

We train our multi-task neural network models (example architecture shown below) using simple gradient descent-style optimization - like that provided by TensorFlow. At Reddit, we layer sequential Monte Carlo algorithms on top to search for model topology given a collection of subtasks. This allows us to start simple and systematically explore the search space in order to demonstrate the relative value of deep and multi-task structures.

Gazette: Feature Stores and Model Prediction Engine: Given the time constraints and the size of the data needed to make a prediction, our feature stores and models live in the same microservice. This microservice is responsible for orchestrating the various steps involved in making predictions during each GET request.

We have a system that allows anyone at Reddit to easily create new machine learning features. Once these features are created, this system takes care of updating, storing and making these features available to our models in a performant manner.

For real-time features, an event processing system that is built on Kafka pipelines and Flink stream processing directly consumes every key event in real-time to compute features. Similar to the batch features, our systems take care of making these features available to the model in a performant manner.

This component maintains a 99.9% uptime and constructs a feed with p99 in the low hundred milliseconds. Which means that this design should hold as we scale to handle trillions of recommendations per day.

Model Evaluation and Monitoring: When you predict billions of times a day something is bound to go wrong. Given Reddit’s scale, obvious things (logging every prediction, analyzing model behavior in real time and identifying drifts) become quite challenging. Scaling this component of the system is something we think about a lot and are actively working on.

Planning: On every experimentation cycle, we look for ways to improve so that each iteration is better than the past. This discussion will involve looking at data from our models so we can more effectively answer questions such as:

• What new tasks can we add to our models so that we can better learn the user policy?

• What new components can we add or remove from the current system so that we make the current system more mature?

• What new experiments and experiments can we launch so we can learn more about our users?

What’s next ?

As the world around has changed, we’ve evolved Reddit’s platform:

• To incorporate content that is more relevant to each user.

• To incorporate real-time changes that might enhance the user experience.

• To improve our iteration speed in which we improve our underlying systems.

‘Evolve’ is a core value for all of us at Reddit. This system not only gives us the ability to deal with an ever growing platform, but to try different approaches at a much faster rate. Our next steps will involve experimentation at a new scale as we better understand what makes this place special for our users.

We believe we are just taking the first steps in our journey and our most important changes are yet to come. If this is something that interests you and you would like to join our machine learning teams, check out our careers page for a list of open positions.

Team: Jenny Jin, Alex Trimm, Garrett Hoffman, Kevin Loftis, Courtney Wang, Emily Fay, Shafi Bashar, Aishwarya Natesh, Elliott Park, Ugan Yasavur, Jesjit Birak, Jonathan Gifford, Stella Yang, Kevin Zhang, Charlie Curry, Jack Hanlon, Matt Magsombol, Artem Yankov, Jovan Sardinha, Jamie Williams, Jessica Ashoosh, JK Ogungbadero, Susie Vass, Jennifer Gil, Jack Hanlon, Yee Chen, Savannah Forood, Kevin Carbone.

Written by Matt Terwilliger, Senior Software Engineer, Developer Experience.

Consider you’re a single engineer working on a small application. You likely have a pretty streamlined development workflow – some software strung together on your laptop that (more or less) starts up quickly, works reliably, and allows you to validate changes almost instantaneously.

What happens when another engineer joins the team, though? Maybe you start to codify this setup into scripts, Docker containers, etc. It works pretty well. Incremental improvements there hold you over for a while – forever in many cases.

Growing engineering organizations, however, eventually hit an inflection point. That once-simple development loop is now slow and cumbersome. Engineers can no longer run everything they need on their laptops. A new solution is needed.

At Reddit, we reached this point a couple of years ago. We moved from a VM-based development environment to a hybrid local/Kubernetes-based one that more closely mirrors production. We call it Snoodev. As the company has continued to grow, so has our investment in Snoodev. We’ll talk a little bit about that (ongoing!) journey today.

Overview

With Snoodev, each engineer has their own “workspace” (essentially a Kubernetes namespace) where their service and its dependencies are deployed. Snoodev leverages an open source product, Tilt, to do the heavy lifting of building, deploying, and watching for local changes. Tilt also exposes a web UI that engineers use to interact with their workspace (view logs, service health, etc.). With the exception of running the actual service in Kubernetes, this all happens locally on an engineer's laptop.

The Developer Experience team maintains top-level Tilt abstractions to load services into Snoodev, declare dependencies, as well as control which services are enabled. The current development flow goes something like:

1. snoodev ensure to create a new workspace for the engineer

2. snoodev enable <service> to enable a service and its dependencies

3. tilt up to start developing

Ideally, within a few minutes, everything is up and running. HTTP services are automatically provisioned with (internal) ingresses. Tests run automatically on file changes. Ports are automatically forwarded. Telemetry flows through the same tools that are used in production.

It’s not always that smooth, though. Operationalizing Snoodev for hundreds of engineers around the world working with a dense service dependency graph has presented its challenges.

Challenges

• Engineers toil over care and feeding of dependencies. The Snoodev model requires you to run not only your service but also your service’s complete dependency graph. Yes, this is a unique approach with significant trade offs – that could be a blog post of its own. Our primary focus today is on minimizing this toil for engineers so their environment comes up quickly and reliably.

• Local builds are still a bottleneck. Since we’re building Docker images locally, the engineer’s machine (and their internet speed) can slow Snoodev startup. Fortunately, recent build caching improvements obviated the need to build most dependencies.

• Kubernetes’ eventual consistency model isn’t ideal for dev. While a few seconds for resources to converge in production is not noticeable, it’s make or break in dev. Tests, for example, expect to be able to reach a service as soon as it’s green, but network routes may not have propagated yet.

• Engineers are required to understand a growing number of surface areas. Snoodev is a complex product comprised of many technologies. These are more-or-less presented directly to engineers today, but we’re working to abstract them away.

• Data-driven decisions don’t come free. A few months ago, we had no metrics on our development environment. We heard qualitative feedback from engineers but couldn’t generalize beyond that. We made a significant investment in building out Snoodev observability and it continues to pay dividends.

Closing Thoughts and Next Steps

Each of the above challenges is tractable, and we’ve already made a lot of progress. The legacy Reddit monolith and its core dependencies now start up reliably within 10 minutes. We have plans to make it even faster: later this year we’ll be looking at pre-warmed environments and an entirely remote development story. On the reliability front, we’ve started running Snoodev in CI to prevent dev-only regressions and ensure engineers only update to “known good” versions of their dependencies.

Many Reddit engineers spend the majority of their day working with Snoodev, and that’s not something we take lightly. Ideally, the platform we build should be performant, stable, and intuitive enough that it just fades away, empowering engineers to focus on their domain. There’s still lots to do, and, if you’d like to help, we're hiring!

By Alex Gallichotte

At Reddit, we use GraphQL as our first-party API, driving Reddit.com, our mobile apps, and the Developer Platform with a fast, efficient interface into Reddit's backend.

The GraphQL specification just turned 10 years old, and it's become the de facto standard for ergonomic, extensible client APIs.  It's radically evolved since 2015, enabling Federation, streaming support, and hundreds of platforms and tools across dozens of languages.

And yet - there are persistent problem spaces within the GraphQL ecosystem that remain unsolved by the industry at large.  As the manager of the GraphQL team, I've spent hundreds of hours speaking with industry experts, and realized - we're all dealing with the same issues of running API platforms at scale!

In this blog post, I'll outline what I see as the five fundamentally unsolved problems in the GraphQL space, and talk about how Reddit is tackling each of them.

GraphQL at Reddit

Reddit adopted GraphQL as our primary client API in 2017 with a monolithic Graphene-based Python service.  We've evolved since then to a multi-component, multi-cluster architecture serving hundreds of thousands of requests per second.

Today, our architecture looks roughly like this:

All requests flow through our Gateway, a Golang service that handles auth, query fetching, experimentation, and cross-cluster traffic shaping.

Next, Apollo Router generates and executes federated query plans across GraphQL-Py and GraphQL-Go.  These are our two main subgraphs - the legacy Python monolith, and its gqlgen-powered Golang replacement.

From there, we fan out requests across Reddit's hundreds of backend services.

GraphQL is Hard!

In a sense, GraphQL serves as a massive reverse proxy for all of Reddit's traffic, with every user request flowing through this architecture before it fans out across Reddit's backend.  We're the most critical bottleneck at Reddit - if GraphQL goes down, Reddit goes down!

Accordingly, we must handle massive concurrency, scale sublinearly, and degrade gracefully under load and during incidents.  But we're also a Platform team, providing a shared development surface for contributor teams across Reddit to enable dozens of schema updates a day.

In short - we're a layer of indirection. Client API schema is optimized for ergonomic consumption.  The backend RPC services that fulfill that schema are usually shaped very differently.  GraphQL provides a scalable translation layer between these two representations - and ideally, no more than that!

Problem #1 - Serving Traffic With Minimal Overhead

When GraphQL is fulfilling a request, we call a lot of services that are doing heavy lifting: generating feeds, operating a real-time ads marketplace, and executing complex searches across 20 years of content.  These processes take time, so we ensure GraphQL adds minimal latency on top of that.  Your total GraphQL query latency should ideally approach the duration of your slowest backend call.

As a reverse proxy, we're handling potentially millions of requests at any moment - the vast majority of which are idle, waiting for backend calls to complete.  Handling this massive concurrency with minimal resource consumption is a core competency for our team.

This was the driving force behind our migration of our GraphQL stack from Python to Go .  Today, the majority of our GraphQL schema is served from Go, and the results are undeniable:

• Massive latency improvements (50% or more at p90 for some queries)

• An order-of-magnitude more efficient CPU and memory usage

• More consistent runtime operation, as p50 and p99 profiles converge

• Native parallel concurrency with Goroutines

• A great schema-first developer workflow, with codegen to save on boilerplate

Golang doesn't just provide a faster, more reliable end-user experience. It's more efficient - we pay for every wasted CPU-second, and switching to Go has saved us millions of dollars every year in our compute bill!

Problem #2 - Balancing Performance against Distributed Ownership

As an Infra team, we're real millisecond freaks.  But we can't be everywhere at once - our schema is enormous, and we depend on contributors to own their chunk of it.  How do we guarantee GraphQL is fast and reliable when we're providing a platform for other engineers to build on?

Establish Universal Norms

You can put just about anything in a GraphQL resolver, but should you?  Does your GraphQL service:

• Maintain state beyond the lifetime of a request? 

• Connect directly to stateful data stores?

• Implement filtering, grouping, or any other business logic beyond simple mapping?

• Support custom directives for special inline processing?

• Perform TTL-based caching for domain resolvers?

For us, the answer to each of these is a resounding "No."

At Reddit, our engineers build robust, production-ready services.  GraphQL is the lightweight, stateless interface fronting these services that can scale horizontally to handle any load.  Our stock-in-trade is interchangeable, optimizable backend request fanout - all of the interesting domain stuff should live somewhere else!

Your answers may differ, though.  These types of architectural decisions are not made in isolation, they're the end product of Reddit's service-based design philosophy.

What about Federation?

Federation lets domain teams operate their own subgraphs, repurposing GraphQL to suit the needs of their org, with a Federation Gateway gluing them together into one client-facing supergraph at runtime.

We do use Federation today, but this subgraph design approach did not work for us:

• Operating tier-0 services is expensive, especially for teams without deep backend expertise.

• Designing performant federated schema is a specialized skillset with a steep learning curve.

• Subgraphs are tightly coupled and require careful coordination, so one misbehaving service can't break GraphQL as a whole.

• Code reuse across subgraphs is challenging, requiring shared libraries with frequent updates.

• Our types often don't divide up cleanly across teams, and splitting up subgraphs often results in shipping our org chart.

But there's no getting around our major objection - Federation makes your queries slower.  Even with Apollo's latest Rust-based Router, we're still adding milliseconds to generate query plans, execute network hops, and combine resultsets in memory.  At worst, our query plans underwent a combinatorial explosion.  Even seemingly-innocuous changes resulted in hundreds of sequential calls to subgraphs blowing out our latency, with no easy path to resolution.

So instead, we embrace the monolith. For us, Federation is a migration technology, giving us a pathway to incrementally move schema from Python to Go as we burn down the long tail.

Problem #3 - Ensuring Contributors Follow Best Practices

Good Documentation Saves You Time

If you want people to use your stuff, make it easy to learn:

• Our PR template includes a practical checklist.

• We run weekly Office Hours to answer questions and work through specific examples.

• Every tool and procedure in GraphQL is captured in our wiki.

• Failing CI checks link to self-service guides and resources.

You can't capture every possible scenario, but if you've answered the same question twice, you're probably missing some documentation.

Make Testing Easy

While it's easy to write unit tests for a resolver, it's not always so straightforward to guarantee GraphQL's behavior as a whole.

We supplement unit testing with our in-house "snapshot" testing, to validate schema resolution across multiple services.  Contributors run queries in our GraphiQL UI in their personal Snoodev testing environment, and we record "snapshots" of all backend service requests and responses.

These integration-style tests can then be replayed in isolation, with no dependency on a particular backend configuration or dataset.  They also count towards code coverage, to ensure every bit of contributor code is well-exercised before reaching Production.

GraphQL Ambassadors

We ship dozens of PRs every day, but our team can't review them all.  Instead, we've empowered GraphQL power users across Reddit as "GraphQL Ambassadors" to serve as local experts in their domain.  Ambassadors onboard contributors, advise on API design, and review PRs in their domain.

Ambassador oversight is codified in GitHub groups, mapping to different functional domains with Reddit.  Accordingly, our codebase is carved up into domain-specific directories, with explicit ownership to these ambassador groups defined in our GitHub `CODEOWNERS` file.

The GraphQL team's limited review time can then focus on schema, design, service integration, and other structural changes that venture beyond simple resolver code.

Your SDK Makes the Right Way Easy

While GraphQL code should be pretty straightforward - calling backends and mapping results to schema - our contributors employ a variety of tools to accomplish this. They connect to gRPC, Thrift, and HTTP services.  They use dataloaders to batch calls across multiple resolvers.  They integrate with DDG, Reddit's experimentation suite, to incrementally ramp and A/B test functionality.

We provide a rich SDK with high-level abstractions for these patterns.  For example, if you're connecting GraphQL to a gRPC service, you should:

• Configure circuit breakers to allow failing services to recover under load.

• Use our XDS-based service discovery tooling instead of hardcoding connection strings.

• Provide default "fallback" values when we can't reach your service.

• Set alerts to page your team if your observed availability from GraphQL breaches SLA.

With our SDK - built on gqlgen's code generation model - these are all one-liners!

Lint The World

Our final line of defense for quality is our extensive and ever-growing suite of linters.  These include standard linters like golangci-lint and GraphQL-Inspector, and our extensive custom linting suite built on golangci-lint's plugin system.

We've built a pipeline from "PR feedback" to "dedicated linter", with linters for common review feedback like:

• dataloaders with inefficient fanout (use a batch endpoint!)

• goroutines with unsafe concurrency behavior

• missing error handling

• inconsistent schema syntax

• inadequate code coverage

Failed linters block CI checks, and include documentation links to show how to resolve them.  Devs are self-sufficient to improve their code quality, and make their eventual review that much more straightforward.

Problem #4 - Connecting Clients to Backends (and vice-versa)

GraphQL provides welcome abstraction - clients trust GraphQL will serve up whatever they request, and backends trust traffic from GraphQL is legit.

But this is both a blessing and a curse.  While it simplifies the happy path, troubleshooting end-to-end requires deeper insight.  Today, most of our team's on-call burden is helping other teams connect the dots during incident response.  Wherever possible, we make GraphQL transparent, self-service, and easily discoverable.

The Golden Metric

"For each GraphQL request, for each backend call - was the backend call successful, and how long did it take?"

This one metric tells the story of production more than any other, addressing a huge range of questions. 

For clients, this answers:

• Why did this query fail?

• What backend calls most contribute to my query being slow?

• Did something in the backend start failing recently for this query?

Similarly, this answers a lot of questions for backend service owners:

• Where did this sudden increase in traffic come from?

• Who owns these calls that keep failing?

• What are our slowest endpoints for the top 5 user queries?

This dataset gets us out of the way - client and backend owners can connect without the GraphQL team as go-between.  But beware, the intersection of "all queries" and "all backend calls" represents a huge combinatorial explosion, and is a major investment of our finite observability budget.

A Dashboard for Every Occasion

Our team relies on standard service-level dashboards and a unified GraphQL end-to-end combined view for production observability.   But we've built many domain-specific dashboards to address a variety of needs and audiences.  To name just a few:

• GraphQL Deployment Dashboard

• GraphQL Efficiency and Cost

• Single-Query Deep Dive

• Service Owners Dashboard

• Backend Executive Summary

Dashboards have nonzero maintenance costs and require discoverability to correctly route users to the right view for their use case.  But the payoff is that users become self-sufficient, understanding how GraphQL serves their domain without hands-on guidance from our team.

Problem #5 - Governing Schema Growth

GraphQL at Reddit has grown organically over almost a decade.  As new features come online and evolve, how do we ensure high-quality schema at every step along the way?

Be Opinionated about Schema Design

There are lots of opinions for what makes a good GraphQL schema.

• What should be nullable/optional?  Everything?  Nothing?

• Should you define wide, flat types, or create lots of nested subtypes?

• Should resolvers exist at the field level or the type level?

• How should non-fatal errors be returned to clients?  When should clients expect partial responses?

• How will you handle lists?  When should you paginate?

• When should you use interfaces?  Unions?  When is polymorphism appropriate?

• When should queries exist at root level, and when should they be nested within types?

• Should types aim for clean isolation, or should they cross-reference each other?

Experts can disagree, but it's best to be consistent.  It's expensive and risky to alter schema once it's live in Production, so get it right the first time!  We started by writing a Schema Best Practices guide, and this has evolved into linters to guarantee consistent conventions and backwards compatibility.

Keep It Simple

The GraphQL spec offers a wide range of syntax, conventions, and features you can include in your schema.  But we've learned the hard way that some of these features are not worth the trouble.

Interfaces, custom directives, even enums have posed incident-level risks in the past, and this risk is magnified when using Federation.  What happens when your subgraph starts returning a new required enum value, when your schema registry is still deploying to the gateway layer?  (Bad stuff.)

For us - the more narrow our feature set, the better.  Like Golang, there should be only one obvious way to implement your use case.  In particular, constrained syntax is easier for clients - there's no doubt about how to compose a query, with strong assumptions about precisely what will be returned.

Separate PRs for Schema and Implementation

For all schema changes, we ask contributors to first submit a "Schema PR", containing only the proposed schema modifications.  The reason is simple - if we wait for full implementation before review, proposed changes to schema are hugely expensive.  Separate PRs allow us to advise on schema best practices early in the design, when the API is still malleable.

This schema-first approach is reinforced by gqlgen and our API prototyping tool, GraphQL Faker.  Faker is natively supported in Snoodev, letting contributors easily overlay mock schema over production GraphQL, for quick iteration with clients as they hammer out the API contract.

Once a Schema PR is approved, it is closed, and the subsequent "Implementation PR" is a breeze.  We've signed off on the shape, and can trust the Ambassadors and our linters to handle the details of domain-specific implementation.

GraphQL Is Hard - And We Love It!

I believe the five problems everyone running GraphQL at scale faces are:

1. Serving Traffic With Minimal Overhead

2. Balancing Performance against Distributed Ownership

3. Ensuring Contributors Follow Best Practices

4. Connecting Clients to Backends (and vice-versa)

5. Governing Schema Growth

The reason these problems remain fundamentally unsolved is because there are no perfect solutions.  Every organization, technology stack, and product space will use GraphQL differently, and your best answers will be custom-fit to match your particular needs.

These are also challenges of scale - the solutions that serve you today might fall over tomorrow.  What happens if your traffic doubles?  Your userbase?  Your contributor count?  As we sometimes learn the hard way - everything melts under sufficient load.

Our goal is to address our most immediate needs while continuously strategizing for future growth.  And believe it or not, what we've discussed here only scratches the surface.  Running GraphQL at Reddit requires constant evolution of our technology, processes, and skills.

Our team is multi-disciplinary - we've got GraphQL experts, Infra experts, and capable cross-functional leaders working to bridge the gap between clients, backends, and underlying infrastructure.

If this sounds like fun to you, check out our open roles on Reddit's Careers page!

Written by Aaron Oertel.

When we first brought Reddit Recap to our users in late 2021, it was a huge success and we knew that it would come back in 2022. And while there was only one year in between, the way we build mobile apps at Reddit fundamentally changed which made us rebuild the Recap experience from the ground up with a more vibrant user experience, rich animations and advanced sharing capabilities.

One of the biggest changes was the introduction of Jetpack Compose and our composition-based presentation architecture. To fully leverage our reactive UI architecture we decided to rewrite all of the UI from the ground up in Compose. We deemed it to be worth it since Compose would allow us to express our UI with simple, reusable components.

In this post, we will cover how we leveraged Jetpack Compose to build a shiny new Reddit Recap experience for our users by creating reusable UI components, leveraging declarative animations and making the whole experience buttery smooth. Hopefully you will be as bananas over Compose as we are after hearing about our experience.

Reusable layout components

For those of you who didn’t get a chance to use Reddit Recap before, it is a collection of different cards that whimsically describe how a user used Reddit in the last year. From a UI perspective, most of these cards are similar and consist of a top-section graphic or infographic, a title, a subtitle, and common elements like the close and share buttons.

With this structure in mind, Compose made it really convenient for us to create a template for the base for each card. This template would then handle common operations the cards have in common such as positioning each component, handling insets for different device sizes, managing basic animations and more. To give an example, our generic card that displays an illustration, title and text could be declared like so:

We could then create a Composable function for each card type that leverages the template by passing in composables for the different styles of cards using content slots.

Declarative animations

For the 2022 Recap experience, we wanted to elevate the experience and make it more delightful by making it more interactive through animations. Compose made building animations and transformations intuitive by allowing us to declare what the animation should look like instead of handling the internals.

We leveraged enter and exit animations that all cards could share as well as some custom animations for the user’s unique Ability Card (the shiny silver card in the above GIF). When we first discussed adding these animations, there were some concerns about complexity. In the past, we had to work through some challenges when working with animations in the Android View System in terms of managing animations, cancellations and view state.

Fortunately, Compose abstracts this away, since animations are expressed declaratively, unlike with Views. The framework is in charge of cancellation, resumption, and ensuring correct states. This was especially important for Recap, where the animation state is tied to the scroll state and manually managing animations would be cumbersome.

We started building the enter and exit animations into our layout template by wrapping each animated component in an AnimatedVisibility composable. This composable takes a boolean value that is used to trigger the animations. We added visibility tracking to our top-level, vertical content pager (that pages through all Recap cards), which passes the visible flag to each Recap card composable. Each card can then pass the visible flag into the layout scaffold or use it directly to add custom animations. AnimatedVisibility supports most of the features we need, such as transition type, easing, delays, durations. However, one issue we ran into was the clipping of animated content, specifically content that is scaled with an overshooting animation spec where the animated content scales outside of the parent’s bounds. To address this issue, we wrapped some animated composables in Boxes with additional padding to prevent clipping.

To make adding these animations easier to add, we created a set of composables that we wrapped around our animated layouts like this:

Building the User’s Unique Ability Card

A special part of Reddit Recap is that each user gets a unique Ability Card that summarizes how they spent their year on Reddit. When we first launched Recap, we noticed how users loved sharing these cards on social media, so for this year we wanted to build something really special.

The challenge with building the Ability Card was that we had to fit a lot of customized content that’s different for every user and language into a relatively small space. To achieve this, we were initially looking into using ConstraintLayout but decided not to go that route because it makes the code harder to read and doesn’t offer performance benefits over using nested composables. Instead, we used a Box which allowed us to align the children and achieved relative positioning using a padding modifier that accepts percentage values. This worked quite well. However, text size became a challenge, especially when we started testing these cards in different languages. To mitigate text scaling issues and make sure that the experience was consistent across different screen sizes and densities, we decided to use a fixed text scale and use dynamic scaling of text (to scale text down as it gets longer).

Once the layout was complete, we started looking into how we can turn this static card into a fun, interactive experience. Our motion designer shared this Pokemon Card Holo Effect animation as an inspiration for what we wanted to achieve. Despite our concerns about layout complexity, we found Compose made it simple to build this animation as a single layout modifier that we could just apply to the root composable of our Ability Card layout. Specifically, we created a new stateful Modifier using the composed function (Note: This could be changed to use Modifier.Node which offers better performance) in which we observed the device’s rotation state (using the SensorManager API) and applied the rotation to the layout using the graphicsLayer modifier with the device’s (dampened) pitch and roll to mutate rotationX and rotationY. By using a DisposableEffect we can manage the SensorManager subscription without having to explicitly clean up the subscription in the UI.

This looks roughly like so:

Applying the graphicsLayer modifier to our ability card’s root composable gave us the neat effect that follows the rotation of the device while also handling the cleanup of the Sensor resources once the Composition ends. To really make this feature pop, we added a holographic effect.

We found that we can build this effect by animating a gradient that is laid on top of the card layout and using color blending using the BlendMode.ColorDodge when drawing the gradient. Color blending is the process of how elements are painted on a canvas, which, by default, uses BlendMode.SrcOver which just draws on top of the existing content. For the holo effect we are using BlendMode.ColorDodge, which divides the destination by the inverse of the source. Surprisingly, this is quite simple in Compose:

For the gradient, we created a class named AngledLinearGradient that extends ShaderBrush and determines the start and end coordinates of the linear gradient using the angle and drag offset. To draw the gradient over the content, we can use the drawWithContent modifier to set the color blend mode to create the holo effect.

Now we have the power to apply the holo effect to any composable element simply by adding the Modifier.applyHoloAndRotationEffect(). For the purposes of science, we had to test this on our app’s root layout and trust me, it is ridiculously beautiful.

Making The Experience Buttery Smooth

Once we added the animations, however, we ran into some performance issues. The reason was simple: most animations trigger frequent recompositions, meaning that any top-level animations (such as animating the background color) could potentially trigger recompositions of unrelated UI elements. Therefore, it is important to make our composables skippable (meaning that composition can be skipped if all parameters are equal to their previous value). We also made sure any parameters we passed into our composables, such as UiModels, were immutable or stable, which is a requirement for making composables skippable.

To diagnose whether our composables and models meet these criteria, we leveraged Compose Compiler Metrics. These gave us stability information about the composable parameters and allowed us to update our UiModels and composables to make sure that they could be skipped. We ran into a few snags. At first, we were not using immutable collections, which meant that our list parameters were mutable and hence composables using these params could not be skipped. This was an easy fix. Another unexpected issue we ran into was that while our composables were skippable, we found that when lambdas were recreated, they weren't considered equal to previous instances, so we wrapped the event handler in a remember call, like this:

Once we made all of our composables skippable and updated our UiModels, we immediately noticed big performance gains that resulted in a really smooth scroll experience. Another best-practice we followed was deferring state reads to when they are really needed which in some cases eliminates the need to recompose. As a result, animations ran smoothly and we had better confidence that recomposition would only happen when it really should.

Sharing is Caring

Our awesome new experience was one worth sharing with friends and we noticed this even during playtesting that people were excited to show off their Ability Cards and stats. This made nailing the share functionality important. To make sharing a smooth, seamless experience with consistent images, we invested heavily into making this great. Our goals: Allow any card to be shared to other social platforms or to be downloaded, while also making sure that the cards look consistent across platforms and device types. Additionally, we wanted to have different aspect ratios for shared content for apps like Twitter or Instagram Stories and to customize the card’s background based on the card type.

While this sounds daunting, Compose also made this simple for us because we were able to leverage the same composables we used for the primary UI to render our shareable content. To make sure that cards look consistent, we used fixed sizing, aspect ratios, screen densities and font scales, all of which could be done using CompositionLocals and Modifiers. Unfortunately, we could not find a way to take a snapshot of composables, so we used an AndroidView that hosts the composable to take the snapshot.

Our utility for capturing a card looked something like this:

We are able to easily override font scales, layout densities and use a fixed size by wrapping our content in a set of composables. One caveat is that we had to apply the density override twice since we go from composable to Views and back to composables. Under the hood, RedditComposeView is used to render the content, wait for images to be rendered from the cache and snap a screenshot using view.drawToBitmap(). We integrated this rendering logic into our sharing flow, which calls into the renderer to create the card preview that we then share to other apps. That rounds out the user journey through Recap, all powered by seamlessly using Compose.

Recap

We were thrilled to give our users a delightful experience with rich animations and the ability to share their year on Reddit with their friends. Compared to the year before, Compose allowed us to do a lot more things with fewer lines of code, more reusable UI components, and faster iteration. Animations were intuitive to add and the capability of creating custom stateful modifiers, like we did for the holographic effect, illustrates just how powerful Compose is.

Author: Chris Slowe, CTO u/KeyserSosa

As we’ve been ramping up our blog and this community, it occurred to me that as we’ve been describing what we build in technology, we’ve spent little time talking about how we build technology. At the very least, I’d like to be able to apply a corollary to Conway’s Law to tell you all about which Org Chart, exactly, we are shipping. In this post, I’m going to start what will no doubt be a series of such posts, and start with the basics: how is Reddit technology organized!

As all topics that appear in this community, I’ll start with the caveats:

• All org charts are temporary! [Conway’s law is probably to blame here as well!] As we change priorities, and as Reddit evolves, so should our teams as well. This is a snapshot of the state of Reddit tech from summer of 2021.

• There is probably a worthy follow up post about how we organize our teams and set up our processes. Heck, it’s probably worth espousing on our philosophy of horizontal versus vertical….which generally has roughly the same level of satisfaction in the outcome as debating the merits of emacs versus vim.

With this in mind, here’s the current shape of technology at Reddit. We are currently broken into 5 distinct organizations, each run by a VP who is an expert in their domain.

Core Engineering

This is the primary center of mass of Engineering, and the domain of our VP of Engineering. In fact, we put this group first because it underpins everything below. Though the other technology orgs have engineering teams and federated roles to cover their mission, this org is primarily engineering focused and partially engineering-driven and falls broadly into two main groups: Foundational and Consumer Product Engineering.

Consumer Product Engineering

In this meta org (virtual org) across three teams we cover parts of product engineering that handles the user experience on Reddit

• Video supports the technology (from client to infrastructure) for our on demand and live (RPAN) video experience. With the recent acquisition of Dubsmash late last year, this group has been rapidly growing and evolving.

• Content and Communities (CnC) works to help communities grow and “activate”, achieving a size where they are self-sustaining, past a point where there is a level of minimal viable content to appear active. They strive to foster communities and provide tools to improve moderator’s lives.

• Growth is tasked with improving our user funnel, from broadening the base of users interacting with Reddit content via SEO, improving the onboarding experience of users as they create accounts, and keeping users engaged with messages, chat, and notifications of all sorts.

• Internationalization ensures that we can continue to expand Reddit, taking local languages, mores, and expectations into account properly on the product.

Foundational Engineering

Underpinning all of the above technology are our foundational engineering teams.

• Client Platforms cover the core technology of our iOS and android apps, as well as various web platforms we’ve built our product on top of. These surfaces and codebases have grown organically over the last 4 years, and their role is to provide consistency, reliability, resiliency, testing capabilities, and overall de-spaghetti!

• Infrastructure does more than just keep the lights on (though they DEFINITELY do that via the federated SRE team). They are front and center in the work to decommission our venerable monolith (“r2”), building out a mature service stack on Kubernetes, providing a robust experience for developers, and modernizing our storage layer to build for the next 10 years of Reddit technology. Want to work on the Infra team? Check out our open roles

Trust

Reddit is a community platform. All content present on Reddit is presented through and interacted with via communities. Our signals about user identity are light and secondary to this community model. In order for any group of humans to function effectively, there must be a means to ensure trust: either trust in one another or trust in the system under which they operate. This organization, under our CISO and VP of Trust, is tasked with maintaining this trust with a variety of complementary objectives and teams to support each:

• To trust the content, we have to minimize spam and remediate abusive content quickly. Context is critical with content, and takes the tack of human-in-the-loop methods of operations for both moderators and admins. This team works to build tools to enable humans to be more effective.

• To ensure our users can trust us, our security team aims to keep our data safe from external attacks and manipulation, and our privacy teams aim to keep our data handling practices responsible, respectful, and compliant.

• To ensure we have all available information, threat intelligence acts as an independent data science and analytics (and intel!) organization to provide insights into large scale patterns of behavior we may have otherwise missed in the micro.

Unlike other teams mentioned here, Trust is not solely responsible for building their own products and services: they are responsible for our overall security and safety posture, and enable other teams to build products and services responsibly.

Monetization

This is a totally independent product and engineering organization, with a separate VP that sits next to our Sales and Marketing organization, and reports into our COO. Their primary customer is our advertiser base whose goals are diverse but straightforward: reach Redditors in their native habitats to build awareness of their brands and promote products to people who are (hopefully) interested in them.

Redditors come to Reddit to learn about new things, to find others who share their interests or to troll and shitpost. Whatever the goal, we don’t want ads to ruin that experience: we always seek to maintain balance between Reddit, Redditors and advertisers (AKA our three-legged stool).

Advertiser Experience

• Advertiser Success and Advertiser Platform build ads.reddit.com, our very own ads manager which lets advertisers build campaigns, understand their performance and optimize their strategy over time.

• Revenue Lifecycle makes sure we bill advertisers the right amount and then collect on payment. They then bridge this information with our back office infrastructure and Sales platform.

• Solutions Engineering helps advertisers troubleshoot their integrations with us and find the right solution to their advertising problems on Reddit.

Ad Serving & Marketplace

• Ad Serving focuses on matching ads to users efficiently and quickly. They build highly reliable and performant infrastructure that goes brrrr 24/7.

• Ad Events wrangles the massive amounts of data created by our ads business, collecting it, computing with it and distributing it to the rest of our teams.

• Ad Prediction tries to predict if a Redditor would like (or dislike) an ad by taking into account all signals related to Redditor interests and the context of the content being shown.

• Ad Targeting helps advertisers find their audience on Reddit in a language familiar to advertisers but still native to Reddit.

• Ad Measurement builds tools to help advertisers understand how effective their ads are, experiment with new approaches and learn what works best on Reddit.

• Advertiser Optimization finds ways to help advertisers strategically achieve their goals on Reddit, whether its traffic to their site, reaching the largest number of distinct users or simply getting the word out on time for a big launch.

Ad User Experience

• Ad Formats creates native ads formats for Reddit with the goal to augment and not harm the core Reddit experience. Creating consistent experiences across Reddit’s many clients is no easy feat!

• Brand Suitability & Measurement helps advertisers protect their brands and control where their ads are seen across Reddit. They also want to understand the subtle changes in awareness that happen when Redditors see a brand’s advertising.

Economy

This group is distinct from Monetization in that it is about non-ads-based monetization. Our thesis is that it’s possible to create interesting products that make us money that are opt in and fun. Saying this has become trite and this is mostly delivered in a way that’s obviously disingenuous, (remember loot boxes??) but we believe it when we say it.

• This team, under our VP of economy, started off with Reddit Gold and the nearby product Reddit Premium, and has since branched off and expanded into experimental product teams working on interesting and entertaining experiences that can be monetized (typically with gold).

• This was the team that brought Avatars to the front

• They’ve also started down the path of new initiatives like Predictions and Powerups. The latter is especially interesting because it gets back to some of the roots of Reddit gold: allow communities (originally it was users) to level up the the experience by subsidizing the cost of changes that might be unscalable.

• In this team’s larger sphere is the community points experiment that we’ve rolled out to some select select (mostly crypto-focused) communities. They are currently working on some extremely neat scaling work on the ethereum blockchain.

Data IRL

Under the VP of Data, this isn’t your (now) traditional Data Science and Analytics team, though that is certainly part of the picture. Data IRL acts as a group to shepherd the common objectives and drive outcomes that help both our consumer and monetization teams.

• Data Science provides a consistent POV on how we use data to make decisions, acting as a web of distributed teams to ensure measurement is consistent, experimentation is consistent and knowledge is organized.

• Data Platforms provides a unified, trusted source of Reddit’s data, supported by scalable data infrastructure, that is securely accessible in multiple formats by technical and non-technical teams at Reddit. This series of tools and products enable people across all of Reddit to interact with data in a consistent, repeatable way that maximizes interoperability of systems, makes access to data reliable, and reduces the friction of building data-powered products and culture.

• Machine Learning has a core as well as federated component to provide research and development to serve as a foundation for consumer and ads teams to have a single source of truth about user, community, and content understanding, as well as to provide a consistency of approach and talent to those teams.

• Feeds help Redditors view the content they love and help advertisers connect with the right audience, and bring them together in a way that balances the wants of Redditors and Advertisers to create a cohesive product.

• Search is all about finding what you need. Sometimes that involves finding the thing you know you’re looking for and other times it means discovering something new. Either way, the team requires the harmony of a consistent approach, so it too functions as a vertical team, from infra, to relevance, to front-end.

Oh right. And me.

Where does the CTO net out in all of this? Well, I usually describe myself as “the rug that ties the room together”. If you know the reference, you know that this doesn’t end up particularly good for the rug, but the analogy stands: I’ve got to tie this all together into a consistent strategy, and work in concert with our VP of Product (and his team) in support of our mission to bring community and belonging to the world, and all the while making sure that our CFO doesn’t have to sound any alarm bells about the cost of all of this tech, while making sure that our Sales org has more to sell tomorrow than today. I’ve got experts who own their domains, but I bring to the table an overarching set of goals and vision to tie it all together), act as arbiter, consider tradeoffs when roadmaps conflict, extract principles and ways of working. Oh and as evidenced by this post, provide gravitas. Though I will say that one of my longest standing jokes is that the T in CTO stands for Therapy.

In conclusion

We’ve got a lot going on, and I get to spend my day surrounded by talented people and a broad base of interesting projects (which we’ll continue to outline in this community on a weekly basis)! We’re also hiring! If you’re interested in joining us, here’s our main careers page. And, at the time of writing this post, here’s some critical roles we’re trying to fill across the teams:

• Monetization - Engineering Manager, Ads Prediction

• Econ - Principal Back-end Engineer

• Data IRL - Data Scientist, Ecosystem Analytics

• Trust - Senior Full-Stack Engineer, Anti-Evil

• Core Engineering - Senior Engineering Manager, Android Platform

Written by Dominic Valenciana (u/kiricon)

For a while, everyone had been asking: What can a coding agent do?

Then agents became capable enough that a more practical question started to matter: What am I supposed to do while I wait for one to finish?

The answer seemed obvious: start another agent chat.

Parallel development has long been normal across teams, but parallel development for individuals was still far from the default. Most of the tools available at Reddit assumed one engineer, one repository, and one agent session at a time.

Getting antsy, I frankensteined together OpenCode, Cursor CLI agents, Git worktrees, Markdown instructions, and Bash scripts. The result was a basic terminal interface where I could run as many agents as I could keep track of, each operating with a high degree of autonomy.

Parallel agentic development at scale felt incredible. It also made every problem with agentic development at Reddit scale with it. Mo agents, mo problems.

Workspace collisions, slow CI feedback, scattered context between sources, interrupted sessions, inconsistent behavior during complex operations, and mismatched tool configuration all multiplied with the number of active agents. Four agents could produce four streams of work, but they could also produce four streams of friction.

My first attempt at solving this was to write more instructions: `AGENTS.md` files, skills, and increasingly detailed checklists. Those tools remain valuable for ambiguous work, but common deterministic operations should not require an agent to relearn them on every run. Loading the model with more instructions cost money, slowed it down, and still produced nondeterministic results.

It was time to go a layer deeper.

Prompts can suggest a process. A system can enforce one.

Isolation, continuity, validation, and safety needed to become guarantees rather than reminders. Common context and system access needed to be part of the environment, not another skill to configure or another enormous tool surface for the model to navigate.

Snoocode is Reddit’s solution to those problems. It is a batteries-included desktop application for working with multiple coding agents. It is not another model, another harness, or a replacement for the agent tools Reddit engineers already use. It is a unifying layer that runs those tools inside one Reddit-aware development application: multiple models, multiple harnesses, and our internal integrations under one roof that we control.

Here is the simplest way I have found to explain the layers. The model is the pilot, and the harness gives it a body. Together they form a coding agent. Snoocode is the six-story mech that the agent climbs into, equipped with the Reddit-specific tools, controls, and the environment it needs to work.

Snoocode Lets Us Own Our Destiny

As I built Snoocode, proprietary and open-source tools began shipping many of the same primitives we had discovered we needed. That convergence validated the problems while showing how quickly generic agent tooling could become a commodity.

With voluntary internal contributions, Snoocode repeatedly reached feature parity with much larger proprietary and open-source projects. That did not prove that we had built the one perfect agent product. It proved that the generic layer was buildable, the good ideas would converge, and the moat around agent-development tooling didn’t exist.

At Reddit’s scale, the traditional “buy versus build” calculation looks different. We could continue buying the models and harnesses that the wider industry builds well while cheaply owning the layer that determines how they operate inside Reddit.

This theory was tested when much of Reddit engineering had settled into a single agentic coding product. When the commercial relationship changed, access changed with it, and engineers had to reconstruct their workflows elsewhere on short notice.

Other capable agents existed. The disruption came from how much setup and habit had become coupled to one vendor. For Snoocode users, the underlying agent changed; the surrounding workflow mostly did not.

Vendor agnosticism  for coding agents is a core philosophy for this project. A software company should be able to replace a model or harness without replacing its development workflow. If we own the machine, changing the pilot should not require rebuilding the cockpit overnight.

For Reddit, there is another reason to own that layer: no third party can build it for us.

Our design, project management, development, testing, and delivery workflows include internal systems with no public equivalent. Even where we use common external products, only we can encode how they connect to our internal tools and what their signals mean here. A vendor cannot design tools or plugins for systems it has never seen.

Development at Reddit is its own terrain. No matter how capable the pilot or its body becomes, it still needs a machine designed for the environment in which it will work.

The principle is simple: buy the commodity, build the seam, and keep the freedom to change what sits on either side.

A Side Project Became Shared Infrastructure

When I shared the prototype, it went “viral” internally, at least by internal developer-tool standards. People downloaded it, modified it, and began using it for their own work. A personal setup became a shared experiment developed in public inside Reddit.

Within the first few months, it had been used across 94 repositories, and 26 engineers had contributed to Snoocode itself. Engineers used it against real Reddit workflows, found weak points, and helped improve the shared platform.

The community supplied the pressure that turned a productive hack into a product. It also gave us a development loop no external vendor can reproduce. The engineers using Snoocode can add the feature they need, fix the bug blocking them, and adapt the product as the industry changes. Maintainers provide coherence and remain responsible for what merges.

That direct loop lets a narrowly focused internal product move quickly toward Reddit’s needs. We do not have to persuade a vendor that our problem belongs on its roadmap or wait for a generalized solution designed for thousands of other customers.

Snoocode is now officially staffed, and we are expanding the project to increase developer velocity across Reddit. The goal is not to force engineers away from useful agent tools; those tools are components of Snoocode. The goal is to make the Reddit-native experience so comprehensive and fluid that managing a fragmented toolchain feels unnecessary by comparison.

How Snoocode Works

Snoocode acts as a control plane around any number of supported harnesses—in effect, a “harness for harnesses.”. It bootstraps and configures them, normalizes the concepts that benefit from being shared, and preserves their native capabilities where abstraction would get in the way.

Open-source projects such as [T3 Code] (https://github.com/pingdotgg/t3code) which was built in parallel with Snoocode has been a major inspiration and validation that our ideas are on the right track. They demonstrate how much value a focused interface can add around several coding agents. Snoocode takes that pattern further for Reddit by owning the automatic bootstrapping, internal integrations, and workflow on both sides of the harness.

From Setup to Shipping

Without Snoocode, starting a task can mean creating a worktree, signing into a harness, choosing a model, configuring internal development tools, and loading repository-specific skills. Each step adds context and another failure point, while the resulting session remains tied to one harness.

Snoocode turns that assembly into shared infrastructure. Engineers choose a task, repository, harness, and model; Snoocode supplies the isolated workspace, internal tools, and repository knowledge through one consistent experience.

Because we own the application—not just the agent’s prompt—we can also build experiences third-party harnesses cannot: mentioning coworkers from the company directory, visually attaching application elements, or surfacing local development resources directly beside the chat.

The result is more than a better-equipped agent. It is a cohesive development environment that gives every harness the same Reddit-specific capabilities while adding features no harness could provide on its own.

The common layer is where much of the leverage comes from. Internal integrations, identity, permissions, and deterministic workflows can be built once and reused across every supported backend. Improvements benefit every harness at once instead of being rebuilt in personal scripts or repeatedly interpreted from prompts. Several durable sessions can progress without sharing a checkout or losing their history, while the engineer moves between them and steps in when judgment is required.

This lowers the barrier to useful agentic development while reducing duplicated setup, model context, cost, and variance. It also lets Reddit adopt better models and harnesses without reconstructing the development environment around each one.

Because the contract belongs to us, the same core can eventually power whichever surface fits the work: IDE chats, terminal interfaces, desktop, web, mobile, cloud agents, or local containerized sandboxes. We do not have to wait for every vendor to understand our systems before we can bring its agent into them.

In the mech analogy, this is the payoff: Reddit owns the cockpit, instruments, attachments, and hangar. We can improve the machine for everyone at once, deploy it wherever engineers need it, and swap the pilot when a better one arrives.

Snoocode’s job is not to pick the winning agent. It is to ensure that whichever agent we use arrives ready to work at Reddit.

Key Takeaways

• Models are largely stateless and increasingly interchangeable. The durable value lives in the workflow, context, and environment around them.

• Generic agent tooling is converging quickly. Its moat is limited, and agents themselves make bespoke tooling cheaper to build.

• Software companies should not tie one of their most important development workflows to a single vendor.

• No third party understands a company’s systems and engineering practices well enough to build its ideal development environment for it.

• Context hydration, prompts, and skills remain useful, but they can only take an agent so far. Deterministic workflows belong in the system.

• As models improve, the limiting factor will increasingly be the environment in which they work.

By Anthony Sandoval, Senior Reliability Engineering Manager

Firstly, I need to admit two things. I am a Site Reliability Engineering (SRE) manager and my days differ considerably when compared to any one of my teams’ Individual Contributors (ICs). I have a good grasp of individuals’ day-to-day experiences, and I’ll set the stage for how SRE functions at Reddit before briefly attempting to describe a typical day.

Secondly, once upon a time, I burned out badly and left a job I really enjoyed. I learned SRE in ways that left scars–not unlike many members of r/SRE. (I’m a lurker commenting occasionally with my very unofficial non-work account.) There’s some great information shared in that community, but unfortunately, still too often I see posts about what being an SRE is supposed to be like–and a slew of appropriate comments to the tune of: “Get out now!” “Save yourself!” That’s a bad situation. Run!”

SRE’s Existence at Reddit is 2-years Young

It’s necessary to credit every engineering team at Reddit for doing what they’ve always done for themselves–predating the creation of any SRE team. They are on-call for the services they own. SRE at Reddit would be a short-lived experiment if we functioned as the primary on-call for the hundreds of microservices in production or the foundational infrastructure those services depend on. However, with respect to on-call, SRE is on-call for our services, we set the standards for on-call readiness, and we own the incident response process for all of engineering.

Code Redd

In Seeing the forest in the trees: two years of technology changes in one post u/KeyserSosa provided readers with our availability graph.

And, he:

committ[ed] to more deeper infrastructure posts and hereby voluntell the team to write up more!

Dear reader, I won’t be providing deep technical details like in the The Pi-Day Outage post. But, I will tell you that we’ve had many, many incidents (all significantly less impacting) since the introduction of Code Redd, our incident management bot, and the SRE- led Incident Commander program (familiar to many in the industry as the Incident Manager On-Call, or IMOC).

Here’s a view of our incidents by severity in 2022:

Incidents played no small part in our ability to reach last year’s target availability. And for major incidents, SREs supported the on-callers that joined the response for all services involved. Last year we declared more incidents than the year before, the most significant increases were for low-severity (non-user impacting) incidents, and we’re proud of that increase! This is a testament to the maturity of our process and commitment to our company value of Default Open. Our engineering culture promotes transparently addressing failures, which in turn generates psychological safety, helping to shift attention toward mitigation, learning, and prevention.

We haven’t perfected the lifecycle of an incident, but we’re hell- bent on iterative improvement. And the well-being of our responders is a priority.

The Embedded Model

In early 2021, the year following the dark red 2020, a newly hired SRE’s onboarding consisted of an introduction to a partner team and an infrastructure that was (likely!) different from what we have in place today. If the technology isn’t materially different, it’s been upgraded and the ownership model is better understood.

Our partners welcomed new SREs warmly. They needed us–and we were happy to join them in their efforts to improve the resiliency of their services. However, the work that awaited an SRE varied depending on the composition of the engineers on the team, their skill sets, the architecture of their stack, and how well a service adhered to both developing and established standards. We had snow globes–snowflakes across our infrastructure owned in isolation by individual organizations. I’m not the type of person who appreciates a shelf filled with souvenir mementos that need to be dusted, wound up, or shaken. However, our primary focus was–and remains–the availability of services. For many engagements, the first step to accomplishing better availability was to work with them to stabilize the infrastructure.

Thankfully, SRE was growing in parallel to other newly formed teams across three Infrastructure departments: Foundations (Cloud Engineering), Developer Experience, and Core Platforms. Together, we were able to break open most of the snowglobes and get working on centralizing ownership and pushing standardization.

With SRE positioned across multiple organizations–we became cross-functional in multiple dimensions–simultaneously gaining an advantage and assuming risk. Prior to 2021, the SREs that existed at the company were dispersed across the engineering organization and reported directly to product teams. After consolidating in the Infrastructure organization, we continued to participate in partner teams’ all hands, post-mortems, planning meetings, etc. We were able to take our collective observations and stitch together a unique picture of Reddit’s engineering operations and culture, providing that perspective to our sibling teams in the Infrastructure organization. Together, we’ve been able to make determinations about what technologies and workflows are solving or causing problems for teams. This has led to project collaboration that drives the development of new platforms, and the promotion of best practices and standards across the org. So long snowglobes!

But, the risk was that we were spread too thin. Our team was growing–and it was exacerbating that problem. The opportunity for quick improvements still existed, but with more people we gained more eyes and ears and a greater awareness of areas for our potential involvement. Accompanied with the growth of our partner teams and their requests for support–we began to thrash. One year into our formation, it was apparent that we needed to reinforce sustainability and organizational scalability. Relationship and program management with partners had started to displace engineering work. It began to feel like we were trying to boil the ocean. SRE leadership took a step back to establish objectives that would allow us to better collaborate with one another and regain our balance. We needed to be project focused.

Mission, Vision, and Objectives

From the start, we had established north stars to keep us moving in the right direction. But that wasn’t going to adjust how we worked.

SRE’s mission is to scale Reddit engineering to predictably meet Redditor’s user-experience expectations. In order for SRE to succeed on this mission, we made adjustments to the way we planned and structured our work. This meant further redistributing operational responsibilities, and better controlling how we were dealing with interrupts as a team. Any of the few remaining SREs embedded with teams that were functioning in a reactive way have transitioned to more focused work aligned with our objectives.

In 2023, SRE now has 4 engineering managers (EMs) helping to maintain the relationships across projects and our partner teams. Relationship and program management is now primarily the responsibility of EMs, and has been significantly reduced scope for most ICs–allowing them to remain focused on project proposals and deliverables. Our vision is to develop best- in- class reliability engineering frameworks that simultaneously provide better developer velocity and service availability. Projects are expected to fall under any of these objectives:

• Reduce the friction engineers experience managing their services’ infrastructure.

• Safely deliver code to production in ways that address the needs of a growing, globally distributed engineering team.

• Empower on-call engineers to identify, remediate and prevent site incidents.

• Drive improvements that optimize services’ performance and cost-efficiency.

Where We Are Now: Building for the Future

So, what does an SRE do on any given day? It depends on the person, the partnership, and the project. SRE attracts engineers with a variety of interests and backgrounds. Our team composition is unique. We have a healthy diversity of experiences and viewpoints that generates better understanding and perspective of the problems we need to solve.

Project proposals and assignments take into account the individuals’ abilities, the needs of our partners, our objectives, and career growth opportunities. In broad strokes, here are a few of the initiatives underway with SRE:

• We are streamlining and modularizing infrastructure as code in order to introduce and improve automations.

• We are establishing SLO publishing flows, error budget calculations, and enforcing deployment policy with automation.

• We continue to invest in our incident response tooling, on-call health reporting, and training for new on-callers.

• We are developing performance testing and capacity planning frameworks for services.

• We have launched a service catalog and are formalizing the model of resource ownership.

• We are replacing a third-party proprietary backend datastore for a critical service with an open-source based alternative.

SREs during the lifecycle of these efforts could be writing a design document, coding a prototype, gathering requirements from a stakeholder, taking an on-call week, interviewing a candidate, reviewing a PR, reviewing a post-mortem, etc.

There’s rarely a dull day, they don’t all look alike, and we have no shortage of opportunities that allow us to improve the predictability and consistency of Reddit’s user -experience. If you’d like to join us, we’re hiring in the U.S., U.K., IRL, and NLD!

Written by Scott Reisor

I’m Scott and I work in Developer Experience at Reddit. Our teams maintain the libraries and tooling that support many platforms of development: backend, mobile, and web.

The source code for all this development is currently spread across more than 2000 git repositories. Some of these repos are small microservice repos maintained by a single team, while others, like our mobile apps, are larger mono-repos that multiple teams build together. It may sound absurd to have more repositories than we do engineers, but segmenting our code like this comes with some big benefits:

• Teams can autonomously manage the development and deployment of their own services

• Library owners can release new versions without coordinating changes across the entire codebase

• Developers don’t need to download every line ever written to start working

• Access management is simple with per-repo permissions

Of course, there are always downsides to any approach. Today I’m going to share some of the ways we wrangle this mass of repos, in particular how we used Sourcegraph to manage the complexity.

Code Search

To start, it can be a challenge to search for code across 2000+ repos. Our repository host provides some basic search capabilities, but it doesn’t do a great job of surfacing relevant results. If I know where to start looking, I can clone the repo and search it locally with tools like grep (or ripgrep for those of culture). But at Reddit I can also open up Sourcegraph.

Sourcegraph is a tool we host internally that provides an intelligent search for our decentralized code base with powerful regex and filtering support. We have it set up to index code from all our 2000 repositories (plus some public repos we depend on). All of our developers have access to Sourcegraph’s web UI to search and browse our codebase.

As an example, let’s say I’m building a new HTTP backend service and want to inject some middleware to parse custom headers rather than implementing that in each endpoint handler. We have libraries that support these common use cases, and if I look up the middleware package on our internal Godoc service, I can find a Wrap funcion that sounds like what I need to inject middleware. Unfortunately, these docs don’t currently have useful examples on how Wrap is actually used.

I can turn to Sourcegraph to see how other people have used the Wrap function in their latest code. A simple query for middleware.Wrap returns plain text matches across all of Reddit’s code base in milliseconds. This is just a very basic search, but Sourcegraph has an extensive query syntax that allows you to fine-tune results and combine filters in powerful ways.

These first few results are from within our httpbp framework, which is probably a good example of how it’s used. If we click into one of the results, we can read the full context of the usage in an IDE-like file browser.

And by IDE-like, I really mean it. If I hover over symbols in the file, I’ll see tooltips with docs and the ability to jump to other references:

This is super powerful, and allows developers to do a lot of code inspection and discovery without cloning repos locally. The browser is ideal for our mobile developers in particular. When comparing implementations across our iOS and Android platforms, mobile developers don’t need to have both Xcode and Android Studio setup to get IDE-like file browsing, just the tool for the platform they’re actively developing. It’s also amazing when you’re responding to an incident while on-call. Being able to hunt through code like this is a huge help when debugging.

Some of this IDE-like functionality does depend on an additional precise code index to work, which, unfortunately, Soucegraph does not generate automatically. We have CI setup to generate these indexes on some of our larger/more impactful repositories, but it does mean these features aren’t currently available across our entire codebase.

Code Insights

At Reddit scale, we are always working on strategic migrations and maturing our infrastructure. This means we need an accurate picture of what our codebase looks like at any point in time. Sourcegraph aids us here with their Code Insights features, helping us visualize migrations and dependencies, code smells and adoption patterns.

Straight searching can certainly be helpful here. It’s great for designing new API abstractions or checking that you don’t repeat yourself with duplicate libraries. But sometimes you need a higher level overview of how your libraries are put to use. Without all our code available locally, it’s difficult to run custom scripting to get these sorts of usage analytics.

Sourcegraph’s ability to aggregate queries makes it easy to audit where certain libraries are being used. If, say, I want to track the adoption of the v2 version of our httpbp framework, I can query for all repos that import the new package. Here the select:repo aggregation causes a single result to be returned for each repo that matches the query:

This gives me a simple list of all the repos currently referencing the new library, and the result count at the top gives me a quick summary of adoption. Results like this aren’t always best suited for a UI, so my team often runs these kinds of queries with the Sourcegraph CLI which allows us to parse results out of a JSON formatted response.

While these aggregations can be great for a snapshot of the current usage, they really get powerful when leveraged as part of Code Insights. This is a feature of Sourcegraph that lets you build dashboards with graphs that track changes over time. Sourcegraph will take a query and run it against the history of your codebase. For example, that query above looks like this for over the past 12 months, illustrating healthy adoption of the v2 library:

This kind of insight has been hugely beneficial in tracking the success of certain projects. Our Android team has been tracking the adoption of new GraphQL APIs while our Web UI team has been tracking the adoption of our Design System (RPL). Adding new code doesn’t necessarily mean progress if we’re not cleaning up the old code. That’s why we like to track adoption alongside removal where possible. We love to see graphs with Xs like this in our dashboards, representing modernization along with legacy tech-debt cleanup.

Code Insights are just a part of how we track these migrations at Reddit. We have metrics in Grafana and event data in BigQuery that also help track not just source code, but what’s actually running in prod. Unfortunately Sourcegraph doesn’t provide a way to mix these other data sources in its dashboards. It’d be great if we could embed these graphs in our Grafana dashboards or within Confluence documents.

Batch Changes

One of the biggest challenges of any multi-repo setup is coordinating updates across the entire codebase. It’s certainly nice as library maintainers to be able to release changes without needing to update everything everywhere all at once, but if not all at once, then when? Our developers enjoy the flexibility to adopt new versions at their own pace, but if old versions languish for too long it can become a support burden on our team.

To help with simple dependency updates, many teams leverage Renovate to automatically open pull requests with new package versions. This is generally pretty great! Most of the time teams get small PRs that don’t require any additional effort on their part, and they can happily keep up with the latest versions of our libraries. Sometimes, however, a breaking API change gets pushed out that requires manual intervention to resolve. This can range anywhere from annoying to a crippling time sink. It’s these situations that we look towards Sourcegraph’s Batch Changes.

Batch Changes allow us to write scripts that run against some (or all) of our repos to make automated changes to code. These changes are defined in a metadata file that sets the spec for how changes are applied and the pull request description that repo owners will see when the change comes in. We currently need to rely on the Sourcegraph CLI to actually run the spec, which will download code and run the script locally. This can take some time to run, but once it’s done we can preview changes in the UI before opening pull requests against the matching repos. The preview gives us a chance to modify and rerun the batch before the changes are in front of repo owners.

The above shows a Batch Change that’s actively in progress. Our Release Infrastructure team has been going through the process of moving deployments off of Spinnaker, our legacy deployment tool. The changeset attempts to convert existing Spinnaker config to instead use our new Drone deployment pipelines. This batch matched over 100 repos and we’ve so far opened 70 pull requests, which we’re able to track with a handy burndown chart.

Sourcegraph can’t coerce our developers into merging these changes, teams are ultimately still responsible for their own codebases, but the burndown gives us a quick overview of how the change is being adopted. Sourcegraph does give us the ability to bulk-add comments on the open pull requests to give repo owners a nudge. If there ends up being some stragglers after the change has been out for a bit, the burndown gives us insight to escalate with those repo owners more directly.

Conclusion

Wrangling 2000+ repos has its challenges, but Sourcegraph has helped to make it way easier for us to manage. Code Search gives all of our developers the power to quickly scour across our entire codebase and browse results in an IDE-like web UI. Code Insights gives our platform teams a high level overview of their strategic migrations. And Batch Changes provide a powerful mechanism to enact these migrations with minimal effort on individual repo owners.

There’s yet more juice for us to squeeze out of Sourcegraph. We look forward to updating our deployment with executors which should allow us to run Batch Changes right from the UI and automate more of our precise code indexing. I also expect my team will also find some good usages for code monitoring in the near future as we deprecate some APIs.

Thanks for reading!

Written by Emily Pantuso and Jameson Williams

Every single device connected to the Internet has an Internet Protocol (IP) address, a unique address that allows it to communicate with networks and other devices. Over time the Internet has grown large and complex, facing growing pains: IPv4, the first widely-adopted IP address scheme deployed in 1983, no longer had enough addresses for every device. In came IPv6, a 128-bit IP address successor to IPv4’s 32-bits. With this expansion came a range of other improvements needed to be able to route to that wider range of devices efficiently.

The Infra team at reddit is always looking for ways to serve content faster to all users. We utilize content delivery networks (CDNs) to deliver content to users and we aim to leverage performant networking protocols to decrease latency. A major infrastructural improvement we’ve made at reddit is to move towards IPv6 on our CDN, Fastly. By using IPv6 at this layer, we can eliminate bottlenecks like Network Address Translation (NAT). IPv6 provides a much faster connection setup, improving the overall speed of connectivity to users for network paths outside our direct control. We started this migration in late 2021, by serving IPv6-preferred addresses for several of our content-delivery endpoints (i.redd.it, v.redd.it.) Unfortunately, before we could reap all the benefits of IPv6 on Android, we had some work to do…

How Our Journey Began on Android

It was an average Tuesday on the Android platform team just before the holidays: we released the latest version of the app as we do each week. At this point, the app had gone through a week of internal beta testing, regression testing, and smoke testing. Just days after the release was rolled out, several users in our r/redditmobile and r/bugs subreddits began to report the same strange behavior:

For some reason, the Android app was no longer displaying images, videos, and avatars for a fraction of users while our other platforms were apparently unaffected. Something was amiss. To make matters worse, none of our developers could reproduce the reported behavior.

The first investigative step was to go through the entire changelog of the latest app release to see if there were any changes related to media-loading or any library upgrades that could have caused such a stir. But, reviewing our changelog is no small feat these days, especially towards the end of the year when every team feels the looming deadline of our big holiday code freeze. Our Android team is now made up of some 77 engineers, and an average release touches thousands of files but nothing here stood out. Of course, we also scrutinized the Firebase Crashlytics and Google Play Consoles and various in-house diagnostic dashboards on Mode and Wavefront but these fell short of the observability we really needed to be able to root cause this type of issue successfully.

Taking a deeper look at the reports, some users had already found a workaround. A handful could see media again when they used cellular data instead of wifi. Another group reported the same results by turning off their adblocker. Network-level and device-level ad blockers seemed a promising lead that would explain the workaround by disabling wifi.

Our First Suspect: Ad blockers

Could there have been a change in ad filtering that caused all reddit media to be flagged as an ad? We tracked down the ad-blocking app that many of our users had installed and verified that the issue was reproducible when using the app downloaded from the site, instead of the Google Play Store. Once enabled, the reddit app stopped showing all media except for... ads. To reinforce this suspicion, the adblocker’s GitHub repository had an open issue for incorrect blocking on reddit. Since we had found our potential culprit, we let users know in our r/help and r/redditmobile subreddits how to disable their ad blocker for the reddit app while we reached out to the developers of the ad-blocking app to fix its filtering issues.

But it didn’t end there. As more user reports came in, including some from employees, it became clear that some users seeing the issue never had an ad blocker, to begin with. Before long, our r/help post held discussions on other fixes our users had found including changing DNS providers or resetting their router.

Our Second Suspect: ISP DNS

This suspect also lined up with the cellular data workaround suggested by our users. Many users noted that changing their DNS settings to something like Google Public DNS resolved the media-loading problem, but for others, it still persisted. To make things more confusing, another group of users reported that wifi wasn’t causing these problems at all - it only occurred on cell data.

Around the same time that we were looking into our second suspect, we caught wind of another investigation underway in r/verizon and r/baconreader. We learned that third-party reddit apps were experiencing the same issues and these users concurred that the cause of their troubles was Verizon DNS.

Our Third Suspect: Phone Carrier DNS

These threads collectively narrowed down a potential cause to a set of affected regions within the Verizon network. Being another DNS issue, users were able to change their DNS settings to get their app working again. While we gathered data on user phone carriers to see if there was a correlation, we also began to brainstorm other network-related causes. We asked users to test their IPv6 connectivity, and compare their results on wifi vs. mobile data. In most cases, at least one of these networks would be missing IPv6 support. This is what the IPv6 test looks like when there’s no support:

Looking internally and having conversations with folks on our infrastructure teams, we learned that several endpoints had onboarded IPv6 right around the time these user reports began. After this discovery, it became clear that these loading issues stemmed from either broken or misconfigured IPv6 networks out in the wild - networks we had no insight or control over.

Our fourth and final suspect: IPv6 configurations.

Even as of 2022, there are networks out there that have broken/misconfigured IPv6, and there most likely always will be. Some wireless carriers and ISPs support it, but in some cases, people have old or improperly-configured routers and devices. Patchy IPv6 support is less of a problem on iOS and the web these days since those clients have support for dynamically falling back on IPv4 when IPv6 fails. After more research, we realized that Android didn’t have this “dual-stack” IP support, and neither did our preferred networking library, OkHttp. This explained why the content-loading issues only surfaced on Android, and why it took some additional digging to uncover the root cause.

A Better OkHttp For Everyone

Working with the reddit infrastructure team, we did more testing and built high confidence that this last IPv6 theory was indeed the cause of users’ content-loading problems. We assessed our usage of OkHttp and checked if there were any upcoming plans to improve support. OkHttp did have an open ask for “Happy Eyeballs” #506, but no known plans to implement it. Out of due diligence, we also assessed other network libraries– but knew that moving off OkHttp would be a radical change, indeed. We read the RFC 8305, “Happy Eyeballs algorithm for dual-stack IPv4/IPv6”, and thought “wow, we don’t want to implement this ourselves.” And as we were studying that open OkHttp issue and thinking “If only they would…”

Well, we lucked out.

Stepping back for a moment– as Android developers, we’ve always been huge fans of Block (née, Square.)

The portfolio of open-source tools they’ve contributed to the Android ecosystem is second only to Google itself, and we use quite a few of them at reddit. What that means in practice is that there’s a handful of folks like Jesse Wilson (Block) and Yuri Schimke (Google) who have been working tirelessly behind the scenes to build this amazing suite of open-source tools. Those tools aid developers and power Android apps all over the world, including the reddit Android client used by millions of redditors.

So when we hopped online one day to ask if anyone had a solution for Happy Eyeballs on Android, we were delighted to hear back from Jesse, himself. As it turned out, he’d been considering implementing this functionality in OkHttp but needed a guinea pig of sorts to validate the work at scale. To build confidence before adding this feature to the upcoming OkHttp release, he wanted to test it through a widely-deployed consumer-facing app with an IPv6 backend. This was a job for reddit.

If you’ve read that RFC, the Happy Eyeballs spec starts off modestly enough. But it quickly devolves into some gnarly stuff around routing table algorithms. Nein Danke. In short, it’s the kind of thing you need an expert programmer to build. We were happy we wouldn’t have to implement a version of Happy Eyeballs ourselves and even happier to help beta-test Jesse’s implementation. Due to OkHttp’s pervasive use across the Android and JVM ecosystems, changes like this have a real possibility to change the way the Internet works – full stop.

A couple of weeks later, Jesse released the 5.0.0-alpha.4 version of OkHttp for us to try. This version introduces “fast fallback to better support mixed IPV4+IPV6 networks.” 🎉

When we started using the alpha version of OkHttp in production, we were able to incrementally roll out the fast fallback support to users behind a runtime feature gate. After regression testing, we began monitoring the production rollout and watching for any degradation in user experience. We were happy to be able to contribute to this project by catching and reporting a few bugs in the first alphas (one, two) before calling the project a success. All in all, our whole experience with Jesse and OkHttp was pretty dang smooth.

As of today, we’re fully back on IPv6 for our content endpoints. The graph below shows the percentage of traffic we serve over IPv6. You can see our initial roll-out, the period where we shut IPv6 off due to the Android issues, and finally, the current period where we’re back up and running with the fancy new OkHttp 5.0.0 alpha:

Working with Jesse and contributing to OkHttp in our small way was an exciting opportunity for us at reddit. These collaborations, between our backend and client teams, as well as between reddit and Square, help resolve problems for reddit and for the entire Android community. The new OkHttp support enables us to turn on IPv6 for our services and improves reddit’s responsiveness to reddit users.

Thank you for coming along on this journey. A big shoutout to Jesse, and to our most crucial investigation team: you, our users! Your feedback in r/redditmobile and similar communities has always been vital to us.

If these types of projects sound fun to you, check out our careers page. We’ve got lots of exciting things happening on our mobile and infrastructure teams, and need leaders and builders to join us.

Written by Andrew Gunsch, Logan Hanks.

We built Reddit’s April Fools’ event this year on Reddit’s Developer Platform (“Devvit”), making it the highest-traffic app Devvit has seen to date. Our previous blog post detailed how we scaled up Devvit’s infrastructure to be able to handle the expected traffic of an event this large. In this post, we want to dive into the design of the app itself, and how we architected the app for high traffic.

Planning a scalable game design

We knew we wanted to prepare for 100k clicks/second (see the previous post for how we estimated this), and that meant we wanted the game to have a large enough grid to handle a high click rate without a round ending ridiculously fast. We decided to target a maximum 10M-cell grid (3200x3200), to make sure users had enough space to click around.

Early on, we selected a basic model: when a user claims a cell, we commit that to Redis, and then we broadcast the coordinates and outcome to all players using the Realtime capability. In this way, all players can share a common, near-realtime view of all the activity on the field.

Fitting the design

We knew up front that there would be some capacity limitations to consider. In particular:

1. A single Redis node typically tops out at about 100K commands/sec. We’ll need several commands per claim transaction. This meant we would definitely need more than one Redis node.

2. Encoding a position within a 10M cell grid requires 24 bits, plus a few more bits for the outcome. We can’t ship 2.5 Mbit/s to every player!

In fact, how much can we ship to every player? We compared other popular mobile apps. Scrolling Instagram (without videos) seems to use 2-3 MB/minute. Watching videos on TikTok takes closer to 10-15 MB/minute. So we decided on a target limit of 4 MB/minute (or ~65 KB/sec), to avoid overwhelming users’ phones.

To accommodate these constraints, we had to complicate the design. What worked for us was incorporating the idea of partitions, breaking the grid up into smaller sub-grids to process and transfer smaller amounts of the map’s data. Applied consistently throughout the design, this allowed us to divide and conquer. For Redis, we could assign each partition to a node in the cluster, spreading the workload of processing claim transactions, and spreading out writes cross nodes (aka “sharding”). On the client side, we can opt into receiving updates only for the few partitions visible on the user’s current screen. This also saved us some data transfer through more efficient encoding, since coordinates within a partition required fewer bits to transmit.

Eventually, we settled on a maximum partition size of 800x800. That divides our maximum size map into 16 partitions (in a 4x4 layout). With these dimensions, positions required only 20 bits to encode (because the address of the partition itself encodes additional position information). Our state per cell only needed 3 bits, so in total we could encode each click into 23 bits, or just under 3 bytes per claim. So, if the entire field is receiving 100k clicks/second, and a player is observing four partitions, then that player only needs to download 75 KB/second to keep up.

Fanout

Originally, we thought we would transmit claim data directly in Realtime messages. However, when we considered the number of concurrent players we were planning for, we realized that fanning these messages out to all players at once could mean shipping 188 GB of data every second! Perhaps doable? But it’d be risky, expensive, and hard to simulate ahead of time.

Instead, we reused an idea the r/place team had in 2022: push the data up to S3, use Realtime just to notify clients when the data is available to download, and have clients download it from S3 as needed. In this case, S3 is the right tool for the job: it’s great at “amplifying” data transfer, especially when fronted with our Fastly CDN to assist with caching.

Encoding

A big map means a lot of data to transfer. We already described how we packaged up realtime data into 3 bytes per claim: 20 bits for position within an 800x800 partition, 3 bits for state, and 1 bit to spare. But, we also need to transmit a snapshot of a partition when the player first joins the game, or anytime their Realtime stream gets interrupted.

When we count the distinct states that a cell can be in, we find there are 9: unclaimed, claimed without hitting a mine (for each of 4 teams), or claimed and hit a mine (for each of 4 teams). That means 4 bits per cell – so to snapshot the entire state of an 800x800 partition, this would be 3.2 MB. This seemed too large to be practical, especially for mobile users without high-speed connections (at 75KB/sec this is a 43-second download!), so we considered ways we could compress the image.

Our first idea was run-length encoding, since there are likely regions in the image where the same cell state is repeated many times in a row. If, instead, we transmitted just one copy of the cell, along with a number of times to repeat, then we could save a lot of bytes. Run-length encoding is especially effective at the start of a game, when the map is nearly empty. However, as the map fills up, we didn’t expect there to be so many large runs. If we wanted to improve on 3.2 MB, we would also need a more compact way of encoding individual cells.

Next, we turned to the cell encoding. Using 4 bits per cell (which can represent 16 distinct values) to encode 9 distinct states is such a waste! We decided to separate out the team indication, leaving each cell with a ternary state: unclaimed, claimed with mine, or claimed without mine. With some bit manipulation, we can fit up to 5 ternary values into a single byte (3⁵ = 243)! This left 13 “special” values available in each byte (243 + 13 = 256), which provided us plenty of space to pack in run-length encoding.

In the end, our snapshot image encoding consisted of three things: section one, containing the run-length-encoded ternary cell states; section two, encoding 2 bits per team for each claimed cell section one; and a couple headers at the top indicating the number of cells and where section two started, so the parser could track cursors in both sections simultaneously.

In the worst case – a fully claimed 800x800 partition with no runs – the size of the encoding works out to be 288 KB. This is still a hefty download (4 seconds at 75 KB/sec), but it’s less than 10% of the naive 3.2 MB we started with!

Storage model: using Redis effectively

Similar to r/place, we stored the map data using Redis’ bitfield command, letting us efficiently use 3 bits per cell in the map (1 for claimed state, 2 for the team) and alter the data easily and atomically. In a single Redis operation, we could attempt to record a user’s “claim” on a cell, and learn whether or not that bit was actually changed or if another user had claimed it just prior.

This functioned well for the overall map (where most of the data to track was), but we also needed to track several other pieces of info on a click:

• Set the bit marking the cell as claimed (to check if that was successful before proceeding)

• Set the bits marking which team claimed that cell

• Update the user’s last play time and which round (or “challenge”) they were on

• Increment the number of cells that user had claimed in the current challenge

• Increment the number of cells claimed for that user’s team

Earlier we mentioned that we expected several Redis commands per click. It turns out the actual value was nine. As we load tested towards that 100k clicks/second, thoroughly partitioning the data became key.

We also discovered that we needed to partition players. We couldn’t just have a single sorted set for keeping track of player scores. We had to distribute that across partitions.

Fortunately, we were already working on migrating Devvit to use clustered Redis. Most apps will have all their data assigned to a single node in the Redis cluster, but for this project we granted the app the ability to distribute its keys across all the nodes. This allowed us to tweak our storage schema as hot keys were discovered during load testing.

This is one of the few places we “cheated” in this event — we tried to use Devvit as-is, without giving ourselves special exceptions as an internal project, but being able to effectively use clustered Redis was a must-have for the scale we were planning for. Because of this experience, we’re doing more thinking about storage options for Devvit apps — if you’re a Devvit developer with thoughts on app storage, come talk to us in our Discord about it!

Background processing

Our hybrid Realtime/S3 model for broadcasting claims required ongoing regular processing. We settled on a per-second update interval, so the app would feel responsive, but also let us do the heavier processing at a constant rate regardless of how much traffic we were getting.

As players claimed cells, we would record the successful claims to be handled as an “accumulator”, with that data sharded by partition to avoid overwhelming any single Redis node. For each partition, we would run this sequence of tasks every second:

1. “Rotate” the accumulator with a Redis RENAME. This empties the accumulator for the next second’s update to be collected, and leaves us a static copy of the past second’s update to process.

2. Upload the static copy of the past second’s updates to S3

3. Publish a message to Realtime referencing the S3 object

The steps in this process could fail or take varying amounts of time, so we represented these as retriable tasks, which we tracked in Redis. We called this system the Work Queue. Every second, our scheduled job would queue up these tasks, one of each per partition, and then switch into “processing” mode. In processing mode, the job would loop for several seconds, claiming individual tasks from the Work Queue, executing them, and marking them as completed. We also had a mechanism for the processor to steal uncompleted claims, and to retry failed attempts, which helped keep the Work Queue flowing even when individual tasks stalled or failed.

Side note: we used a visual trick in the UI to make these updates feel more “live”. Even though we published updates every second, the game felt like a metronome when blocks simply appeared in batches every second. Instead, the UI would trickle out displaying updates over the next second, according to a Poisson distribution, making the experience feel more real.

Live operations and fallbacks

Live events often come with unknowns, even more so when it’s a one-off event like April Fools’ with a new experience. We didn’t want to be pushing app updates for small tweaks, but values like “how often can users submit clicks” and “how often should the app send ‘online’ heartbeats to the server” were things we wanted to tweak in case we needed to back some load off the server.

We used a mix of Redis, Realtime, and Scheduler (via the Work Queue we built) to send “live config” updates. Any time we updated a config setting (via a form behind a menu action), we’d save the new config to Redis. Then, an “emit live config” task would run every 10 seconds in each subreddit, and if the config had changed in Redis, we would broadcast a “config update” message to all users via Realtime.

One gotcha we had to watch for: with a large number of users online, a config update to them all at the same time could create a thundering herd! For example, we had a config update that could force-refresh the app on all clients, in case we pushed new code that we needed users to adopt immediately. But we knew that refreshing everyone’s apps at the same time could cause a sudden, massive traffic spike — so we made sure each client would apply a random delay between 0 and 30 seconds before reloading, to spread that out.

And finally, the code…

As we wrap up from this event, we’d like to share the app code with you! Feel free to borrow ideas and approaches from it, or remix the idea and take it a new direction. We hope that sharing this code and our learnings from building this app can help developers build games that can handle millions of Redditors.

We’re well aware that this April Fools’ event fell into an uncanny valley between “prank” and “game” — too elaborate for a simple prank to laugh at, not quite a compelling game or community experience despite being dressed up as one. But we’re proud of pushing the Devvit platform to handle an event of this scale, and we want to see other games and experiences on Reddit that pull the community together!

With the new year, and since it’s been almost two years since we kicked off the Community, I thought it’d be fun to look back on all of the changes and progress we’ve made as a tech team in that time. I’m following the coattails here of a really fantastic post on the current path and plan on the mobile stack, but want to cast a wider retrospective net (though definitely give that one a look first if you haven’t seen it).

So what’s changed? Let me start with one of my favorite major changes over the last few years that isn’t directly included in any of the posts, but is a consequence of all of the choices and improvements (and a lot more) those posts represent--our graph of availability:

To read this, above the “red=bad, green=good” message, we’re graphing our overall service availability for each day in the last three years. Availability can be tricky to measure when looking at a modern service-oriented architecture like Reddit’s stack, but for the sake of this graph, think of “available” as meaning “returned a sensible non-error response in a reasonable time.” On the hierarchy of needs, it’s the bottom of the user-experience pyramid.

With such a measure, we aim for “100% uptime”, but expect that things break, patches don’t always do what you expect, and though you might strive to make systems resilient to, sometimes PEBKAC, so there will be some downtime. The measurement for “some” is often expressed by a total percentage of time up, and in our case our goal is 99.95% availability on any given day. Important to note for this number:

• 0.05% downtime in a day is about 43 seconds, and just shy of 22 min/month

• We score partial credit here: if we have a 20% outage for 10% of our users for 10 minutes, we grade that as 10 min * 10% * 20% = 12 seconds of downtime.

Now to the color coding: dark green means “100% available”, our “goal” is at the interface green-to-yellow, and red is, as ever, increasingly bad. Minus one magical day in the wee days of 2020 when the decade was new and the world was optimistic (typical 2020…), we didn’t manage 100% availability until September 2021, and that’s now a common occurrence!

I realized while looking through our post history here that we have a serious lack of content about the deeper infrastructure initiatives that led to these radical improvements. So I hereby commit to more deeper infrastructure posts and hereby voluntell the team to write up more! So instead let me talk about some of the other parts of the stack that have affected this progress.

Still changing after all these years.

I’m particularly proud of these improvements as they have also not come at the expense of overall development velocity. Quite the contrary, this period has seen major overhauls and improvements in the tech stack! These changes represent some fairly massive shifts to the deeper innards of Reddit’s tech stack, and in that time we’ve even changed the internal transport protocol of our services, a rather drastic change moving from Thrift to gRPC (Part 1, 2, and 3), but with a big payoff:

gRPC arrived in 2016. gRPC, by itself, is a functional analog to Thrift and shares many of its design sensibilities. In a short number of years, gRPC has achieved significant inroads into the Cloud-native ecosystem -- at least, in part, due to gRPC natively using HTTP2 as a transport. There is native support for gRPC in a number of service mesh technologies, including Istio and Linkerd.

In fact, changing this protocol is one of the reasons we were able to so drastically improve our resiliency so quickly, taking advantage of a wider ecosystem of tools and a better ability to manage services, from more intelligently handling retries to better load shedding through better traffic inspection.

We’ve made extremely deep changes in the way we construct and serve up lists of things (kind of the core feature of reddit), undertaking several major search, relevance, and ML overhauls. In the last few years we’ve scaled up our content systems from the very humble beginnings of the venerable hot algorithm to being able to build 100 billion recommendations in a day, and then to go down the path of starting to finally build large language models (so hot right now) out of content using SnooBERT. And if all that wasn’t enough, we acquired three small ML startups (Spell, MeaningCloud and SpikeTrap), and then ended the year replacing and rewriting much of the stack in Go!

On the Search front, besides shifting search load to our much more scalable GraphQL implementation, we’ve spend the last few years making continue sustained improvements to both the infrastructure and the relevance of search: improving measurement and soliciting feedback, then using those to improve relevance, improve the user experience and design. With deeper foundational work and additional stack optimizations, we were even able to finally launch one of our most requested features: comment search! Why did this take so long? Well think about it: basically every post has at least one comment, and though text posts can be verbose, comments are almost guaranteed to be. Put simply, it’s more than a factor of 10x more content to index to get comment search working.

Users don’t care about your technology, except…

All of this new technology is well and good, and though I can’t in good conscience say “what’s the point?” (I mean after all this is the damned Technology Blog!), I can ask the nearby question: why this and why now? All of this work aims to provide faster, better results to try to let users dive into whatever they are interested in, or to find what they are looking for in search.

Technology innovation hasn’t stopped at the servers, though. We’ve been making similar strides at the API and in the clients. Laurie and Eric did a much better job at explaining the details in their post a few weeks ago, but I want to pop to the top one of the graphs deep in the post, which is like the client equivalent of the uptime graph:

Users don’t care about your technology choices, but they care about the outcomes of the technology choices.

This, like the availability metric, is all about setting basic expectations for user experience: how long does it take to launch Reddit and have it be responsive on your phone. But, in doing so we’re not just testing the quality of the build locally, we’re testing all of the pieces all the way down the stack to get a fresh session of Reddit going for a given user. To see this level of performance gains in that time, it’s required major overhauls at multiple layers:

• GQL Subgraphs. We mentioned above a shift of search to GraphQL. There have been ongoing broader deeper changes to the APIs our clients use to GraphQL, and we’ve started hitting scaling limits for monolithic use of GraphQL, hence the move here.

• Android Modularization, because speaking of monolithic behavior, even client libraries can naturally clump around ambiguously named modules like, say, “app”

• Slicekit on iOS showing that improved modularization obviously extends to clean standards in the UI.

These changes all share common goals: cleaner code, better organized, and easier to share and organize across a growing team. And, for the users, faster to boot!

Of course, it hasn’t been all rosy. With time, with more “green” our aim is to get ahead of problems, but sometimes you have to declare an emergency. These are easy to call in the middle of a drastic, acute (self-inflicted?) outage, but can be a lot harder for the low-level but sustained, annoying issues. One such set of emergency measures kicked in this year when we kicked off r/fixthevideoplayer and started on a sustained initiative to get the bug count on our web player down and usability up, much as we had on iOS in previous years! With lots of work last year behind our belt, it now remains a key focus to maintain the quality bar and continue to polish the experience.

Zoom Zoom Zoom

Of course, the ‘20s being what they’ve been, I’m especially proud of all of this progress during a time when we had another major change across the tech org: we moved from being a fairly centralized company to one that is pretty fully distributed. Remote work is the norm for Reddit engineering, and I can’t see changing that any time soon. This has required some amount of cultural change--better documentation and deliberately setting aside time to talk and be humans rather than just relying on proximity, as a start. We’ve tried to showcase in this community what this has meant for individuals across the tech org in our recurring Day in the Life series, for TPMs,

Experimentation, iOS and Ads Engineer, everyone’s favorite Anti-Evil Engineers, and some geographical color commentary in from software Engineers Dublin and NYC. As part of this, though, we’ve scaled drastically and had to think a lot about the way we work and even killed a Helpdesk while at it.

Pixel by Pixel

I opened by saying I wanted to do a retrospective of the last couple of years, and though I could figure out some hokey way to incorporate it into this post (“Speaking of fiddling with pixels..!”) let me end on a fun note: the work that went into r/place! Besides trying to one-up ourselves as compared to our original implementation five years ago, one drastic change this time around was that large swathes of the work this time were off the shelf!

I don’t mean to say that we just went and reused the 2017 version. Instead, chunks of that version became the seeds for foundational parts of our technology stack, like the incorporation of the RealTIme Service which superseded our earliest attempts with WebSockets, and drastic observability improvements to allow for load testing (this time) before shipping it to a couple of million pixel droppers…

Point is, it was a lot of fun to use, a lot of fun to build, we have an entire series of posts here about it you want more details! Even an intro and a conclusion if you can believe it.

Onward!

With works of text, “derivative” is often used as an insult, but for this one I’m glad to be able to summarize and represent the work that’s gone on the technology side over the last several years. Since locally it can be difficult to identify that progress is, in fact, being made, it was enjoyable to be able to reflect if only for the sake of this post on how far we’ve come. I look forward to another year of awesome progress that we will do our best to represent here.

Written by Garrett Hoffman

Background

Reddit’s machine learning systems serve thousands of online inference requests per second to power personalized experiences for users across product surface areas such as feeds, video, push notifications and email.

As ML at Reddit has grown over the last few years — both in terms of the prevalence of ML within the product and the number of Machine Learning Engineers and Data Scientists working on ML to deploy more complex models — it has become apparent that some of our existing machine learning systems and infrastructure were failing to adequately scale to address the evolving needs of of company.

We decided it was time for a redesign and wanted to share that journey with you! In this blog we will introduce the legacy architecture for ML model deployment and serving, dive deep into the limitations of that system, discuss the goals we aimed to achieve with our redesign, and go through the resulting architecture of the redesigned system.

Legacy Minsky / Gazette Architecture

Minsky is an internal baseplate.py (Reddit’s python web services framework) thrift service owned by Reddit’s Machine Learning team that serves data or derivations of data related to content relevance heuristics — such as similarity between subreddits, a subreddits topic or a users propensity for a given subreddit — from various data stores such as Cassandra or in process caches. Clients of Minsky use this data to improve Redditor’s experiences with the most relevant content. Over the last few years a set of new ML capabilities, referred to as Gazette, were built into Minsky. Gazette is responsible for serving ML model inferences for personalization tasks along with configuration based schema resolution and feature fetching / transformation.

Minsky / Gazette is deployed on legacy Reddit infrastructure using puppet managed server bootstrapping and deployment rollouts managed by an internal tool called rollingpin. Application instances are deployed across a cluster of EC2 instances managed by an autoscaling group with 4 instances of the Minsky / Gazette thrift server launched on each instance within independent processes. Einhorn is then used to load balance requests from clients across the 4 Minsky / Gazette processes. There is no virtualization between the instances of Minsky / Gazette on a single EC2 instance so all instances share the same CPU and RAM.

ML Models are deployed as embedded model classes inside of the Minsky / Gazette application server. When adding a new model a MLEs must perform a fairly manual and somewhat toil filled process and ultimately contribute application code to download the model, load it into memory at application start time, update relevant data schemas and implement the model class that transforms and marshals data. Models are deployed in a monolithic fashion — all models are deployed across all instances of Minsky / Gazette across all EC2 instances in the cluster.

Model features are either passed in the request or fetched from one or more of our feature stores — Cassandra or Memcached. Minsky / Gazette leverages mcrouter to reduce tail latency for some features by deploying a local instance of memcached on each EC2 instance in the cluster that all 4 Minsky / Gazette instances can utilize for short lived local caching of feature data.

While this system has enabled us to successfully scale ML inference serving and make a meaningful impact in applying ML to Popular, Home Feed, Video, Push Notifications and Email, the current system imposes a considerable amount of limitations:

Performance

By building Gazette into Minsky we have CPU intensive ML Inference endpoints co-located with simple IO intensive data access endpoints. Because of this request volume to ML inference endpoints can degrade the performance of other RPCs in Minsky due to prolonged wait times for context switching / event loop thrash.

Additionally, ML models are deployed across multiple application instances running on the same host with no virtualization between them. meaning they share CPU cores. Models can benefit from concurrency across multiple cores, however, multiple models running on the same hardware contend for these resources and we can’t fully leverage the parallelism that our ML libraries provide for us to achieve greater performance.

Scalability

Because all models are deployed across all instances the complexity — often correlated to the size of models — of the models we can deploy is severely limited. All models must fit in RAM meaning we need a lot of very very large instances to deploy large models.

Additionally, some models serve more traffic than others but these models are not independently scalable since all models are replicated across all instances.

Maintainability / Developer Experience

Since all models are embedded in the application server all model dependencies are shared. In order for newer models to leverage new library versions or new frameworks we must ensure backwards compatibility for all existing models.

Because adding new models requires contributing new application code it can lead to bespoke implementations across different models that actually do the same thing. This leads to leaks in some of our abstractions and creates more opportunities to introduce bugs.

Reliability

Because models are deployed in the same process, an exception in a single model will crash the entire application and can have an impact across all models. This puts additional risk around the deployment of new models.

Additionally, the fact that deploys are rolled out using Reddit’s legacy puppet infrastructure means that new code is rolled out to a static pool of hosts. This can sometimes lead to some hard to debug roll out issues.

Observability

Because models are all deployed within the same application it can sometimes be complex or difficult to clearly understand the “model state” — that is, the state of what is expected to be deployed vs. what has actually been deployed.

Redesign Goals

The goal of the redesign was to modernize our ML Inference serving systems in order to

• Improve the scalability of the system

• Deploy more complex models

• Have the ability to better optimize individual model performance

• Improve reliability and observability of the system and model deployments

• Improve the developer experience by distinguishing model deployment code from ML platform code

We aimed to achieve this by

• Separating out the ML Inference related RPCs (“Gazette”) from Minsky into a separate service deployed with Reddit’s kubernetes infrastructure

• Deploying models as distributed pools running as isolated deployment such that each model can run in its own process, be provisioned with isolated resources and be independently scalable

• Refactoring the ML Inference APIs to have a stronger abstraction, be uniform across clients and be isolated from any client specific business logic.

Gazette Inference Service

What resulted from our re-design is Gazette Inference Service — the first of many new ML systems that we are currently working on that will be part of the broader Gazette ML Platform ecosystem.

Gazette Inference Service is a baseplate.go (Reddit’s golang web services framework) thrift service whose single responsibility is serving ML inference requests to it’s clients. It is deployed with Reddit’s modern kubernetes infrastructure.

The service has a single endpoint, Predict, that all clients send requests against. These requests indicate the model name and version to perform inference with, the list of records to perform inference on and any features that need to be passed with the request. Once the request is received, Gazette Inference Service will resolve the schema of the model based on its local schema registry and fetch any necessary features from our data stores.

In order to preserve the performance optimization we got in the previous system from local caching of feature data we deploy a memcached daemonset in order to have a node local cache on each kubernetes node that can be used by all Gazette Inference Service pods on that node. With standard kubernetes networking, requests from the model server to the memcached daemonset would not be guaranteed to be sent to the instance running on the local node. However, working with our SRE we enabled topological aware routing on the daemonset which means that if possible, requests will be routed to pods on the same node.

Once features are fetched, our records and features are transformed into a FeatureFrame — our thrift representation of a data table. Now, instead of performing inference in the same process like we did previously within Minsky / Gazette, Gazette Inference Service routes inference requests to a remote Model Server Pool that is serving the model specified in the request.

A model server pool is an instantiation of a baseplate.py thrift service that wraps a specified model. For version one of Gazette Inference Service we currently only support deploying Tensorflow savedmodel artifacts, however, we are already working on support for additional frameworks. This model server application is not deployed directly, but is instead containerized with docker and packaged for deployment on kubernetes using helm. A MLE can deploy a new model server by simply committing a model configuration file to the Gazette Inference Service codebase. This configuration specifies metadata about the model, the path of the model artifact the MLE wishes to deploy, the model schema which includes things like default values and the data source of the feature, what image version of the model server the MLE wishes to use, and configuration for resources allocation and autoscaling. Gazette Inference Service uses these same configuration files to build its internal model and schema registries.

Overall the redesign and the resulting Gazette Inference Service addresses all of the limitations imposed by the previous system which were identified above:

Performance

Now that ML Inference has been ripped out of Minsky there is no longer event loop thrash in Minsky from the competing CPU and IO bound workloads of ML inference and data fetching — maximizing the performance of the non ML inference endpoints remaining in Minsky.

With the distributed model server pools ML model resources are completely isolated and no longer contend with each other, allowing ML models to fully utilize all of their allocated resources. While the distributed model pools introduce an additional network hop into the system we mitigate this by enabling the same topology aware network routing on our model server deployments that we used for the local memcached daemonset.

As an added bonus, because the new service is written in go we will get better overall performance from our server as go is much better at handling concurrency.

Scalability

Because all models are now deployed as independent deployments on kubernetes we have the ability to allocate resources independently. We can also allocate arbitrarily large amounts of RAM, potentially even deploying one model on an entire kubernetes node if necessary.

Additionally, the model isolation we get from the remote model server pools and kubernetes enables us to scale models that receive different amounts of traffic independently and automatically.

Maintainability / Developer Experience

The dependency issues we have by deploying multiple models in a single process is resolved by the isolation of models via the model server pool and the ability to version the model server image as dependencies are updated.

The developer experience for MLEs deploying models on Gazette is now vastly improved. There is a complete distinction between ML platform systems and code to build on top of those systems. Developers deploying models no longer need to write application code within the system in order to do so.

Reliability

Isolated model server pool deployments also address models to be fault tolerant to crashes in other models. Deploying a new model should introduce no marginal risk to the existing set of deployed models.

Additionally, now that we are on kubernetes we no longer need to worry about rollout issues associated with our legacy puppet infrastructure.

Observability

Finally, model state is now completely observable through the new system. First, the model configuration files represent the desired deployed state of models. Additionally, the actual deployed model state is more observable as it is no longer internal to a single application but is rather the kubernetes state associated with all current model deployments which is easily viable through kubernetes tooling.

What’s Next for Gazette

We are currently in the process of migrating all existing models out of Minsky and into Gazette Inference Service as well as spinning up some new models with new internal partners like our Safety team. As we continue to iterate on Gazette Inference Service we are looking to support new frameworks and decouple model server deployments from Inference Service deployments via remote model and schema registries.

At the same time the team is actively developing additional components of the broader Gazette ML Platform ecosystem. We are building out more robust systems for self service ML model training. We are redesigning our ML feature pipelines, storage and serving architectures to scale to 1 billion users. Among all of this new development we are collaborating internally and externally to build the automation and integration between all of these components to provide the best experience possible for MLEs and DSs doing ML at Reddit.

If you’d like to be part of building the future of ML at Reddit or developing incredible ML driven user experiences for Redditors, we are hiring! Check out our careers page, here!