Take your business to the next level with our features

January 2020

I was the tech lead for a project to replace Dropbox’s aging monolith with a modern platform. The monolith story is similar to many in the industry. There were many initiatives to improve the monolith situation, including a company-wide Service-Oriented Architecture (SOA) push a few years prior. These initiatives all failed to make lasting changes due to conflicting incentives and priorities.

Our mandate was to improve developer velocity, as it had become impossible to make changes on the monolith. For example, a single web route took over a day to set up! Along with this mandate, our team was also accountable for the reliability of the system, as we had spent 2019, “SRE”-ing (for lack of a better word) the system and to get it stable. We saw an opportunity to push both frontiers forward and believed we had a novel approach; instead of asking for a full rewrite, we were going to build a platform completely on a modern tech stack with the ability to run legacy code with minimal changes. This would give everyone the full benefit of SOA (isolation, independent pushes, low-cost new routes, etc.) without asking them for a migration.

For the first time, we had a line of sight into replacing the monolith, which could be accomplished by the hard work of a small, surgical team of engineers.

Mid-2020

We had a prototype that worked. The new platform was a technological marvel. It featured the best of Dropbox’s tech stack with a modern gRPC-based serving framework, significant (10x) performance improvement on simple route overhead, autoscaling, automatically safe releases with canary analysis, independent releases, and failure isolation. And, it was able to run the legacy monolith code completely unchanged with an adapter.

We started demo-ing this to engineers in the company to solicit feedback. Surprisingly to us, the feedback could not have been more negative. Product engineers (also referred to as product) did not understand why their workflows were changing and how this was an improvement. In fact, they felt that the new platform worsened their experience; they had to learn new concepts (e.g. gRPC) for seemingly no gain.

I was quite frustrated. Weren’t the technologies like gRPC standard in the company at this point? Wouldn’t they have to learn these technologies anyways to get anything else done here? Why couldn’t the product teams see the effort that went into the project, and all the wins they would get from it? Why didn’t they appreciate my team’s efforts more?

I would later find out what the product teams were feeling. This was yet another infra-led initiative being forced on them. It did not fix any fundamental pain points. It did not involve the product teams in the decision-making process. As one product engineer put it, product teams were tired of the “just” attitude from infra. As in, “Why didn’t product teams just learn gRPC? Why didn’t they ‘just’ run services?” without taking into account all of the other complexities that product teams deal with that infra teams do not see. I, myself, was guilty of this attitude, as you saw above.

Like so many initiatives on the monolith before it, my project looked set to fail.

The Industry at Large

The story above is emblematic of the industry at large. Infrastructure keeps getting increasingly complex, and I don’t mean that in a positive way. At a minimum, most companies deal with Kubernetes and Infra-as-code toolings (e.g.Terraform) that back it. Then, in order to run anything real end to end, these companies inevitably need to pull in more complexities: release managers like argo rollouts or flagger, workflow engines like tekton or argo workflows, service meshes and ingresses, to name a few. This does not even go into the data layer, e.g. having to deal with DB migrations and backups.

These are all highly complex systems to learn, and none of them have anything to do with actually shipping products and delivering value to customers. Product engineers are left with the choice: (1) “be full-stack” and become an expert in these techs, in which case they would not be able to spend time doing their actual work OR (2) ignore the complexity and rely on an infra team to be formed to deal with this when the time is right, which creates a situation not unlike the Dropbox situation above.

Many products have tried to solve the complexity and largely fall into two trends. One is a set of infra-based products(e.g. argo, tekton) focused on delivering key building blocks. These products often end up being too complicated for the average engineer to learn, and inevitably, these products target platform/devops engineers instead of focusing on product wins. The other trend is product-oriented products (e.g. heroku) that attempt to hide away infrastructure complexity entirely and focus on product needs. These products will often fail for a different reason; the minute companies need to invest in the infrastructure for any reason (scaling, compliance, etc.), the products become untenable from the infra perspective and companies chart a migration off of them.

The key lies somewhere in the middle. Things do need to get better, but the only way it gets better is through intentional collaboration: infra, and product, together. One side cannot work alone to solve this.

Bridging Gaps

I have a long storied history with bridging gaps. Growing up with divorced parents and having strong relationships with both of them, I have always found significant power in empathy. Throughout my career, I have found myself on the edge of infrastructure teams and would have to regularly interface up and down the stack. Empathy is being able to shift perspective at will and use that to bridge gaps and align interests. I have found that leveraging this ability allowed me to be more effective in situations involving multiple differing perspectives than others.

Going back to the opening story, my team took the feedback from the product engineering teams and doubled down on customer outreach. We started doing workflow interviews where we prompted product engineers to write a new route in front of us using the tutorials we wrote. We took extra care to refrain from being defensive and instead focused on what pain points the product teams were facing. Most importantly, we engaged the product engineers as partners, asked for continuous feedback, and worked with them to achieve the results we all wanted.

Our efforts resulted in a positive attitude shift for our project. For the first time, product engineers felt like they had a partner they could trust in infrastructure. The same product engineer who pointed out the “just” attitude stated that he felt like there was a “bit flip” in how we approached building platforms and hoped that other infra teams started doing the same.

From a technology perspective, the final product that we shipped was not very different from what we initially presented to the product team. We accomplished everything we set out to do from the infrastructure point of view. However, the workflow was a marked shift.

The end state was significantly more thoughtful and intentionally designed to how a product engineer would use the platform. Building with empathy for our internal users led us to become the champion for product workflows and product interests across other infra and platform teams.

Oh, and over a decade since the founding of Dropbox, the monolith was finally no more. We’d call it a success. For technical details, see the Dropbox blog post.

Upshot

The Dropbox monolith story may be over for the time being, but the industry at large is still struggling with too much complexity. Infrastructure keeps getting more sophisticated and complicated, creating more bottlenecks on infrastructure talent and causing product development to suffer. Someone needs to bridge the gap and align incentives across infrastructure and product.

This is why I decided to start Prodvana with Andrew Fong. We believe in zero overhead across the stack, and we exist to build a platform that eliminates unnecessary cloud complexities. Put another way, our mission is to align the product and infrastructure interests and allow both sides to focus on innovating where they are best at.

Follow our company to learn more about what we are doing!