Build and Deploy Code in Small Chunks to Reduce Risk

by Edmond Lau

Photo credit: peosoldier

In late 2011, soon after the company had IPO’ed, LinkedIn paused feature development for two entire months. The motivation? The engineering team needed to restructure its entire code deployment process to streamline development. Up until that point, engineers would develop a new feature by forking a branch off of trunk (the main branch), building and testing the feature over the course of days or weeks, and then merging large batches of code changes back into trunk. They’d then have to fix painful merge conflicts and re-test all their changes to ensure that they didn’t introduce any incompatibilities due to any new code checked into trunk since feature development started. 1

The old branch-based workflow was slow and error-prone. To ship code by a certain day, engineers would merge their code into trunk three weeks before a release, to leave buffer room to iron out any kinks and bugs. Even with the three weeks of leeway, it was not uncommon for an actual deployment to take upwards of 7 hours to complete because the frantic merges would lead to instability and often require last-minute hot-fixes. 2

And so, LinkedIn invested in the tools and the processes necessary to migrate the entire company to trunk-based development, where all engineers commit to one shared branch (trunk). It’s a process that other companies like Google and Facebook have scaled successfully. 3 Rather than building entire features in isolation over days or weeks, engineers worked instead on much smaller changes that were committed directly into trunk.

The benefits of this more incremental approach are enormous:

  • You’re incentivized to work on smaller, logical chunks so that you can more quickly and easily commit code into the main branch without painful merge conflicts.
  • The “merge hell” that normally takes place before a release goes away because code changes are no longer built on top of forks that were many days or weeks old.
  • Context switches for engineers drop significantly — engineers develop and test their code prior to committing and don’t have to go through the same process several days or weeks later when at a future merge point.
  • Code changes, because they’re smaller, become easier to review — leading to higher code quality.
  • More engineers can provide valuable feedback earlier because the code is checked into the main branch.
  • Overall risk decreases, because there is a smaller inventory of code that is committed but not deployed.

Kevin Scott, LinkedIn’s Senior VP of Engineering, told Wired magazine, “Without having done all this work to change how we built our software, it literally would be impossible for us to be building endorsements, and the new influencers product, and the new version of profile and the things that are happening in mobile, and the upgrades we made to the recruiter product, and all of these dozens of significant changes that are rolling out.” [1] The incremental, trunk-based approach made it possible for them to focus more of their time on actual feature work and less on coordinating merges.

When I worked at Quora, we took this incremental approach to the extreme — we used continuous deployment, where every commit that passed a battery of tests would be immediately and automatically deployed into production. It would be quite normal for our team to deploy code upwards of 50 times a day. 4 And even at Quip, the startup where I work now, we deploy software several times a day. Far from increasing risk, the focus on smaller and incremental changes make the code much higher quality and dramatically reduces any coordination overhead. Building and deploying code in smaller chunks means that we can quickly and easily validate changes in front of our users. And that, in turn, means that we’re more likely to succeed.

Posted:

“A comprehensive tour of our industry's collective wisdom written with clarity.”

— Jack Heart, Engineering Manager at Asana

“Edmond managed to distill his decade of engineering experience into crystal-clear best practices.”

— Daniel Peng, Senior Staff Engineer at Google

“A comprehensive tour of our industry's collective wisdom written with clarity.”

— Jack Heart, Engineering Manager at Asana

“Edmond managed to distill his decade of engineering experience into crystal-clear best practices.”

— Daniel Peng, Senior Staff Engineer at Google

Grow Your Skills Beyond the Book

Listen to podcast interviews with top software engineers and watch master-level videos of techniques previously taught only in workshops and seminars.

Leave a Comment