Everywhere one looks in March 2020, they see big companies being forced into involuntary experiments. For the data-minded among us, this represents an unprecedented opportunity to study how widespread changes translate into measurable results. With so many historical inventions attributable to weird combinations of chance, some of these involuntary experiments are destined to yield transformative benefits.
Microsoft has recently distinguished itself for being on the leading edge of dealing with the coronavirus crisis. They earned accolades from the New York Times for being among the first companies to recognize the urgency of the coronavirus pandemic. By early March, they recommended employees work from home. Shortly thereafter, they published an excellent, comprehensive guide to working remotely. This guide offers a fascinating peek behind the curtains. One of the largest tech employers in the world has been forced into a massive involuntary experiment. And you won't believe what happened next...
Across work items, commits, and pull requests, we’re not seeing any declines. If anything has changed, it’s a shift in activity during the day, with activity starting earlier and finishing later, and with lower “peaks” in the middle of the day.
In a word: nothing. 🙈
But how could that be true? Tens of thousands of engineers upend their routine, abdicate the spaces that have been set up at great expense to optimize their productivity, and the result is... nothing happened? This narrative feels suspect, so we cross-checked it against our own data on Microsoft's productivity.
linkRemote working: the trend
Before we jump into the Microsoft data, a quick detour regarding its applicability. Microsoft's foray into working from home is but one tendril of a much larger pro-WFH movement already underway. In the last five years, remote work has seen a gradual uptick in interest. This month, the "gradual" in that uptick was tossed out the window:
Remote work has been trending upward, but March is a sea change.
How has this (now tidal) wave impacted the software development process? When it comes to coronavirus-driven change, it's really too early to know. A change of this magnitude needs more time and more sample size before we can interpret it confidently. Nevertheless, we posit that "some data" beats "no data," so long as the limitations of "some data" are made clear.
The data we'll review below uses the "Line Impact" metric. It's built to measure the degree to which each commit evolves the repo in which it resides. This is accomplished by identifying and ignoring the 95% of lines of code comprised of trivial changes like moved code, white space, keywords, find-and-replace, third-party libraries, etc. Strip it all out, and you're left with a stable indicator of how much product development is happening .
linkData suggests productivity is off by ~15%
Now let's dig into the data. With a sample size of about 20 developers, the Microsoft VS Code team is a globally distributed team. Eyeballing their last three months of activity, it looks to roughly corroborate their statement that it's "business as usual" at Microsoft. Nothing is dramatically off here:
Microsoft VS Code Line Impact by week, before and after near-global quarantine. The color bands represent various developers on the team
In fact, their Line Impact change over January-March is +52%, but that's mostly because the base comparison point (September-December) is comparatively light. Let's zoom in on the specific period since quarantine began:
Microsoft VS Code Line Impact by day since the quarantine. The big dips are the weekends. The color bands represent individual developers on the team.
Compared to the 15 days before the outbreak, team productivity is off pace by 22%. The key stat in the before-and-after picture of Microsoft's developer throughput is their daily Line Impact. For the first three months of this year, they were averaging almost 3,700 daily LI (for a team of < 20 active developers, an impressive result!). In the last two weeks, they've sunk to 3,200 daily, down 14%.
Might their dip be a seasonal phenomenon? Nope:
Microsoft VS Code team Line Impact circa March 2019
Near March of 2019, they averaged about 4,200 daily Line Impact. Relative to the 3,200 they've been averaging in the last two weeks, this appears to be a small -- but possibly significant -- blip on the radar.
linkMicrosoft relative to other companies
An obvious next question is how this small-but-maybe-significant difference we see in Microsoft compares to others? To gauge this, I ran a few database queries to check the Line Impact accumulated across the hundreds of repos that comprise our customers who have opted into anonymous data sharing. The results lined up with Microsoft's, to a surprising degree: globally, we have seen productivity across customer repos sag by 15% over the last two weeks, relative to the same period a month earlier.
If writing this article entitles me to an opinion on cause: I think people might be a bit distracted.
linkThe "classic" working from home adjustment
All things being equal, our research to date has suggested that the average team sees a negligible difference (positive or negative) in overall volume of work authored when working from home. However, the cadence at which work is delivered varies markedly.
Here is the "classic" work from home pattern that we've seen among teams like ours that worked from home one day per week:
Alloy's Line Impact by hour over last three months. Wednesday is our work from home day.
What usually happens is that employees get somewhat (5-20%) less done per hour when working from home, but that difference is offset by the longer range of hours that developers work when they have schedule flexibility. In general, it's a win for employers and employees when a policy can afford greater personal flexibility without a measurable cost in terms of code authorship. Our observation that their workday will stretch is consistent with Microsoft's own takeaway.
linkReplicating effective "work from home" developer teams
While there is great prior art on the state of WFH, there's still little research that explores the specific shortcomings of working from home for tech teams. What do engineers and project managers miss, besides the general camaraderie and free food? Here are three noteworthy omissions:
linkStand Up Meetings
One of the first tangible impacts felt by many product teams. Stand Up Meetings offer a quick way for a product team to get on the same page. There are a few different options to replace this. The most literal translation of "Stand Up Meeting" to its online equivalent is via Slack or Zoom. This works alright, but often takes longer than its real world equivalent. For users of GitClear, the Commit Activity Browser emulates the sort of transparency created by Stand Up Meetings:
Commit Activity Browser shows recent commit activity (and lack thereof), and tickets being worked on, per developer
A visualization like this helps Project Managers stay abreast of tickets underway. It helps Lead Developers spot which developers might benefit from a Slack visit (the ones who have been stuck longer than usual without yet requesting help). By keeping managers continuously appraised of "status" and "blockers," the need for a dedicated meeting on those subjects is reduced.
linkPR state awareness
Pull requests are another area where real-world discussion have historically driven progress. Most commonly, this takes the form of a developer opening up a PR and everyone forgetting about it for a couple days until someone speaks up about the oversight. To ensure that PRs don't slip through the cracks, GitClear provides a notifications system that catches various situations in which a pull request warrants review. It looks something like this:
Daily notifications keep managers updated on latest opportunities to optimize throughput
linkSprint planning meetings
Normally, agile teams will meet up at least once per sprint to diagnose which issues should be worked on, and by whom. Without the opportunity to gather in real life, it can be more difficult for managers to match tickets from the queue to the best-fit developer available. GitClear helps by keeping a live list of which committers have been most prolific in the code domains spanned by all the company's repos.
A few of the Domain Experts in the Facebook React repo. React's full list spans 30 different code domains from our preconfigured defaults.
By maintaining an up-to-the-minute roster of which developers have illustrated proficiency in various areas of the code, it becomes quicker to filter candidates for a sprint ticket. For more technical managers, they can even browse the specific directories where a developer has contributed, to match a Jira ticket to the developer who is most familiar with the applicable directories.
linkWork for a data-minded company?
From the look of it, this wide-scale work from home experiment won't be ending any time soon. As the situation develops, we'll aim to deliver another report on the "State of Work from Home" in April. If you'd like to see how your team has changed since transitioning to working from home, we'd love to include your results (anonymously) in a future update. Visit here for a demo or you can just sign up and start poking around.