linkAbstract

Modern code review holds a multifaceted role in improving code quality, finding defects, and sharing knowledge between team members. CodeGrip's 2022 survey of "1,000+ CxOs and developers" indicates that some form of code review is utilized by 84% of companies [1]. As a business process, pull requests have surged to near ubiquity. Unfortunately, that surge not been paired with a corresponding flourish in tooling improvements. GitHub, Bitbucket, Gitlab, et. al rely on the archaic Myers diff algorithm (published 1986), to characterize the changes within a pull request. This means that every line change has a binary designation as either "addition" or "deletion," just as it has since 1986.

In this research, we analyze the extent to which updating the venerable Myers algorithm might reduce code review time. According to CodeGrip, the median developer in 2022 spent 2-5 hours per week on code review, with 30% of respondents reporting spending more than 5 hours per week on review [1]. In a 40 hour work week, this implies more than 10% of the entire week is consumed by code review.

To demonstrate how an evolved Myers algorithm could function, GitClear in June 2024 released a pull request review tool that recognizes six types of code changes (add, delete, move, find/replace, copy/paste, update). Possessing more granular change detail allows many lines that would be designated as "changes" by Myers algorithm to be recognized as "effectively no-op."

This paper utilizes a two-prong method to assess the efficacy of applying precise change data to reduce pull request size:

Finally, we conclude by assessing the actionability of the results collected by this research. We further assess how emerging strategies such as "AI summarization" and "data benchmarking" may further expedite the code review process.

linkTable of contents

linkModern Code Review: State of Current Tooling

Peer code review, a manual source code inspection by developers other than the author, is generally recognized as a valuable tool for reducing software defects and improving the quality of software projects [3] .

The original ICICLE tool or “Intelligent Code Inspection in a C Language Environment” developed by the researchers at Bellcore, paved a new way for developers to review code, and many of the original concepts are still used in today's most popular tools.

open_in_new

open_in_new

Rietveld Code Review Tool 2008 GitHub Code Review Tool 2024

linkBuilt on the Venerable Myers Algorithm

In 1986, Eugene Myers published An O(ND) Difference Algorithm and Its Variations, which unified the problems of finding the longest common subsequence of two sequences and finding the shortest edit script for transforming one sequence into another. Myers showed that these problems were equivalent to finding the shortest path over an "edit graph." [7]

His algorithm improved the popular diff utility, a data comparison tool that displays the smallest set of line-by-line deletions and insertions to transform one file into another [7]. The diff utility is incorporated in the git version control system, a dominant choice for developers today, with up to 90% of professional developers listing it as a regular part of their toolset [8], [9] .

open_in_new 2022 Developer Survey - Stack Overflow

open_in_new 2023 Developer Ecosystem - JetBrains

On top of git, git diff, and the Myers Algorithm, modern code review tools came forth to provide an informal, asynchronous, and convenient way to adopt the code review process.

As from the 2023 JetBrains survey, GitHub, GitLab, and BitBucket hold the top ranks as "most popular version control systems," and de facto diff tools. Azure DevOps joins these stalwarts as the fourth most popular tool for code review in 2023:

open_in_new

open_in_new

Due to their intrinsic reliance on the Myers algorithm - the default algorithm used to process diffs - GitHub, GitLab, and BitBucket are referred to as "standard diff viewers" in the scope of this research.

linkGitClear's "Commit Cruncher" Diff Algorithm

In 2020, GitClear began building a novel diff algorithm, referred to as the "Commit Cruncher," to present code changes in terms more akin to modern developers. The six types of changed operations found in GitClear's diff viewer are detailed in GitClear's documentation [10]. They are "Added," "Deleted," "Updated," "Moved," "Find/Replaced," and "Copy/Pasted" (the latter featured prominently in GitClear's popular AI Code Quality research)

The following diff visually illustrates how recognizing precise change operations allows reductions to "changed lines to review":

open_in_new

This is the exact same code diff, presented with GitHub (left) vs GitClear (right)

The extent to which Commit Cruncher reduces lines to read vs. Myers algorithm depends on the prevalence of code operations that GitClear can distill. In the following section, we will use a large sample of pull requests to calculate the frequency of change operations that can be condensed without reducing change fidelity.

Another example where Myers falls short of the mark is when white space is involved, such as in this change:

open_in_new

The equivalent diff with Commit Cruncher:

open_in_new

The latter is able to recognize whitespace changes as trivial, so the extra indent created by wrapping the method body is not presented as a broad sea of red and green.

linkEmpirical Comparison Method: Delta in Changed Lines Shown

To assess the impact of GitClear's Commit Cruncher diff algorithm on the code review process, GitClear collected measurements for how many changed lines were shown across 12,000+ GitHub pull requests. The GitHub API's compare endpoint was utilized to capture the count of "added" and "deleted" lines to be shown to the user within each pull request (aka "set of sha endpoints").

link12,638 Pull Requests Suggest Median 27% Less Lines to Review

GitHub's "changed line count" was compared to the changed lines of code as measured by GitClear's Commit Cruncher (diff processing engine). The "changed line count" for a GitClear diff is defined as "the count of lines that would be seen if the user opened a pull request on GitClear with the default user settings." Since GitClear began collecting this comparative data in mid-May 2024, about 13,000 pull requests have been cataloged with their "changed line count" as derived from GitClear and GitHub. It is a standard GitClear feature that this count is shown by default atop the diff viewer, with a detailed file-by-file breakdown of the changed line comparison. Aggregating the 12,638 pull requests by their overall size (as approximated by Diff Delta), shows an increasing percentage reduction in the "changed lines to review count" as the size of the pull request increases:

open_in_new

GitClear reduces "changed lines to review" across all sizes of pull requests, across the 12,638 samples analyzed to date

The sources of these pull requests were split roughly 50/50 between "GitClear customers opted into sharing Industry Stats," and "popular open-source repos" (specifically, those found within GitClear's Open Repos section).

You can find a detailed description of the database queries that were used to retrieve this data in [A4] to prove that the sample is a random set, which is expected to boost its reproducibility in possible future research.

The data shows that developers reviewing with GitClear and its "Commit Cruncher" algorithm read, on average, 22% to 29% fewer changed lines per pull request. The median difference between "Myers" and "Commit Cruncher" ranges from 27% to 31%, depending on the total magnitude of the change set.

link"Less Code Shown" Not Necessarily Advantageous

Though GitClear consistently presents fewer changed lines to review, this reduction comes with two important caveats. The first is that "less changed lines to review" doesn't necessarily translate to "less time to review code." For example, if the smaller pull requests were presented on a page that loaded 40% more slowly, the change benefit could be nullified.

The second (perhaps more realistic?) caveat is that "reviewing code faster" may be expected to reduce the likelihood that potential bugs are detected.

The first prospective caveat is refuted by existing studies in this field. Two such studies find that "the patch size affects the duration in most of the analyzed cases" [11] and "the patch size negatively affects all outcomes of code review that we consider as an indication of effectiveness - duration, comments left, reviewer activity" [12]

Analyzing the possibility that "faster code reviews mean less issues detected" is the primary subject of the following sections.

linkInterview-Based Comparison Method: Human Observation Data

While the raw line change counts strongly suggest that less time will be required to review code on GitClear, we sought to design an experimental method that would allow us to answer additional questions:

1) Do the reductions in "lines to review" translate to a corresponding reduction in "time to review"?

2) If so, does reviewing the pull request in less time have an observable negative or positive impact on the percentage of bugs that are discovered?

linkHow to correctly assess code review duration?

We will define the review duration as the period of time a reviewer needs to review all the files inside a pull request. The duration will not measure the time spent leaving comments or navigating the UI. This metric stands to be a subset of the definitions of review durations previously cited - them measuring the review duration as the count of "how many days the code review process lasted, from the day that the source code is available to be reviewed to the day that it received the last approval of a reviewer". [12]

Evaluating review duration is inherently difficult to assess, due to the distinct approaches or preferences teams and developers have put in place when conducting a code review. The duration and the quality of a review are tightly linked - with the former impacting the latter, and deciding whether it results in a superficial or an in-depth code review. Thus, the noise generated by the reviewer's style and previous experience made that simply recording the duration of a review would not lead to accurate results. An additional metric is required to enable an apples-to-apples comparison of the duration of reviews of similar levels of quality and thoroughness.

The next obstacle is that there is no clearly defined and ready-to-be-tracked metric for review quality and thoroughness. Most of the time, it is inferred based on the expected results ensuing the review. A study by Bacchelli and Bird [3] provides answers on the developers' motivations for conducting code reviews. Further underlying the complexity of undertaking a holistic approach to measuring review performance based on how satisfactory were the outcomes in contrast to the initial motivations.

open_in_new

Developers’ motivations for code review - Bacchelli and Bird (2013)

Yet, the same study highlights a crucial insight revealing a common challenge everyone faces when striving for a fruitful code review [3]:

"Understanding is the main challenge when doing code reviews".

Testimonials from the survey participants further support this idea:

“the most difficult thing when doing a code review is understanding the reason for the change;”

"understanding the code takes most of the reviewing time."

“in a successful code review submission, the author is sure that his peers understand and approve the change.”

Bacchelli and Bird's study shares the survey results on developers' sentiment for the level of code understanding required to accomplish the multiple aspects of the code review process.

open_in_new

Developers’ responses in surveys of the amount of code understanding for code review outcomes - Bacchelli and Bird (2013)

Time saved when a developer spends trying to understand code changes directly impacts the code review flow. Shortening the time window a developer needs to be comfortable with the code inside the pull request frees up time and mental space for additional, equally important parts of the process. Sitting at the core of almost every part of the review process, the "understanding" part of the process became a great candidate to measure alongside review duration when doing a fair comparison between GitClear and other review tools.

With this in mind, we've chosen to analyze 2 complementary metrics during the investigation:

linkInterview Methodology

To evaluate the impact of GitClear's PR Review Tool, live review sessions were conducted with developers possessing different levels of experience.

The candidates for the test were chosen randomly from the web platform CodeMentor. Choosing an environment where developers were initially unfamiliar with GitClear offered the opportunity to record how users who were new to the platform acclimated to it. No developer surveyed had prior experience with the GitClear platform.

A candidate review session comprised 2 pull request reviews. Each candidate was asked to review 1 pull request with GitClear and 1 (different) pull request with GitHub. Each pull request review was followed by a quiz to assess the candidate's understanding of the code they had reviewed.

linkPull Request Candidates

Pull requests were chosen from open-source repositories, and grouped in buckets based on the programming language used. Criteria for selecting programming languages were based on the popularity of the languages across the industry [8], [9]. Thus, the initial list contained pull requests written in JavaScript+TypeScript, Python, and C#.

Two separate pull requests were used per session to eliminate the unfair advantage the developer would get if they were asked to review the same pull request twice. Criteria for pull request eligibility were as follows:

linkCode Review Evaluation Metrics

Each review session resulted in a pair of data points: "pull request review duration" and "quiz result accuracy."

To measure the review duration, we notified the reviewer that there was no time limit imposed on the review duration and then instructed them to indicate when they were ready to start, and subsequently finish, the review.

The code understanding quiz presented 5 to 9 multiple-choice questions extracted from the contents of each pull request. The difficulty of the questions increased incrementally. Initial questions evaluated surface-level knowledge of the pull request, later questions focused on finer details of the logic inside the code.

The quiz results were scored by using a method involving an adapted elimination testing to verify partial and intermediate answers to the proposed questions. Each question could result in a value between 1 and -1, with each correct selected answer positively contributing to the overall score, whereas each incorrect answer would negatively contribute. The "final question score" is the net sum of the answers.

Subsequently, the question results were normalized and translated to percentages.

linkCode Interview Results

As of the initial publish date of this research (June 6, 2024), 48 participants had been interviewed and represented within the data tables below.

linkCode Review Duration

To understand the impact of GitClear's diff viewer compared to Github's diff viewer, we collated the data from 48 review sessions, segmented by programming language, and calculated the net and percent difference of average and median review duration, [A5].

open_in_new

Code Review Duration Aggregated Results GitClear vs GitHub

Here's how the aggregated data looks in graph form for net and percentile results, with the yellow bars illustrating the absolute difference between the two data points:

open_in_new Median Net Code Review Duration Results GitClear vs GitHub	open_in_new Median % Code Review Duration Results GitClear vs GitHub
open_in_new Average Net Code Review Duration Results GitClear vs GitHub	open_in_new Average % Code Review Duration Results GitClear vs GitHub

A general trend of review time reduction can be observed across all the investigated programming languages with GitClear's diff viewer. The net decrease in review duration is further reflected as significant percentile differences compared to GitHub's diff viewer, and consequently standard diff viewers.

The most notable difference was for the pull request 25610 , with a 42% decrease (13.16 average minutes with GitClear vs 22.76 average minutes with GitHub) in favor of GitClear reviewers.

linkCode Understanding Accuracy

The assessment of code understanding was averaged across all review sessions, with the figure below aggregating the accuracy results of GitClear and GitHub reviewers.

open_in_new

Question Accuracy Results for GitClear and GitHub Reviewers

Question accuracy percentage difference does not fluctuate more than 5%, averaging 1.12% in favor of GitClear when evaluated across the entire pool results, [A5].

Moreover, the raw data of the evaluation metrics for each individual session [A1] was plotted using a scatter chart, comparing question accuracy scores against the code review duration, as seen in the figure below. The code review duration decrease is visually outlined by the increased frequency of blue (GitClear) dots on the left side of the chart.

open_in_new

Code Review Sessions' evaluation metrics - Review Accuracy and Review Duration

linkInterpretation

There is still more data to be parsed but the initial results point to a very promising direction for GitClear's alternative take on the review process. Developers are quick to adapt to the new principles GitClear works on, due to having a common ground with the standard reviewing practices but at the same time filling in the gaps of the sought-after quality of life changes.

The 22% average code review duration decrease for GitClear's diff viewer validated our initial hypothesis and predictably correlated with the reduction of lines of code that needed to be reviewed. While the duration of the review was shorter, the quiz accuracy results were consistently similar for both GitClear and GitHub users, further cementing that no information was lost at the expense of speed. Furthermore, the 22% review reduction aftermath is a consequence of the initial reaction to GitClear's review tool, we expect the average reduction to increase as developers get more familiar with the tool.

In particular, pull request changes with a big percentage of moved code (25610 ) performed the best, resulting in a reduction of over 40% in review duration. This consolidates this operation as one of the much-needed features that diff viewers were lacking. Being able to know that parts of code were moved and unchanged relieves the brain power needed to validate that operation and frees up space to focus on other areas.

Finally, we believe that individual contributors, as well as companies, can benefit from transitioning to GitClear's diff viewer, helping them to significantly reduce the time investment in code review with little to no transition cost. Making a judgment on the recent code review statistics present in the field (SmartBear - 2021, Codegrip - 2022), we can get a better grasp of the impact such a tool can have on the industry. 22% of developers report participating in tool-based code reviews daily and 19% weekly is in line with a majority of company reports scoring between 2 to 5 hours per day invested into code reviews, as seen in the figure below. For that time interval, if developers could reduce the time spent during code reviews by at least 22% that would result in anywhere between 30 to 60 minutes per week or 26 to 52 hours per year per developer.

open_in_new

Code Review Trends in 2022 - Codegrip

linkOn Durability of Status Quo

One question raised by this research: how has diff viewing evolved to be more homogenous than any of "developer IDE," "git platform," or "system OS"? Compared to the innumerable programming languages that have come and gone in the years since Myers algorithm was created, how did so many products converge on a solution developed 40 years ago? Two possibilities seem most plausible.

The first answer is that most developers don’t even recognize that it’s possible to represent a diff without Myers. Since diffs have looked the same since their career began, nobody thinks to go looking for other options.

The second reason is that Myers is a much “cleaner” algorithm than any successor would be. Choosing Myers offers an instantly available, multi-generation tested means to show a diff. And when it comes to reviewing a diff, getting every line right (every time) is incredibly important.

While Commit Cruncher shows significant improvement over Myers in this research, it relies upon a set of iteratively tuned heuristics. None of the large git platforms can afford to imperil user trust as they iterate on a more granular representation of what changed within a commit. Much like all source control providers herded to git once it was proven reliable at enterprise-scale, no single company is likely to evolve until they have strong incentives, including proof that the new algorithm, and the heuristics imbued therein, can be trusted to consistently interpret git diffs of any size and content.

linkConclusion

Despite that standard diff viewers are the commanding choice for conducting code reviews for the majority of the developers, they demonstrate limitations due to their simplistic approach of displaying changes as binary "added" and "deleted" operations.

GitClear's diff viewer provides a more nuanced overview of the changes inside a pull request by changing the definition of "added" and "deleted" lines and making use of 5 additional code operations to provide semantic context and abstract away LoC noise. With a different diff algorithm, GitClear reliably shows fewer lines of code to the reviewer, concentrating their focus on the meaningful changes that are happening inside a pull request.

In this paper, we've analyzed GitClear's impact on the duration of code reviews, resulting in a decrease of 22% on average compared to GitHub's diff viewer. Additionally, we've measured the reviewers' level of code understanding to enable a fair comparison of different review styles and ensure the validity of the recorded data. No significant discrepancy has been observed in favor of either platform for the resulting level of code understanding, with an overall percentile average of under 1.2% in favor of GitClear.

linkCitations

linkAppendix

The data used to build this research is included below:

linkA1 - Raw data from review sessions

linkA2 - Pull request candidates

linkJavascript/Typescript

https://github.com/facebook/react/pull/25610/files

https://github.com/facebook/react/pull/25265/files

linkC#

https://github.com/microsoft/PowerToys/pull/22409/files

https://github.com/microsoft/PowerToys/pull/28884/files

linkPython

https://github.com/pandas-dev/pandas/pull/43504/files

https://github.com/django/django/pull/14776/files

linkA3 - Question Forms

linkJavascript/Typescript

https://forms.gle/MuJxASunD19PGng76

https://forms.gle/tc8mox286kFBFaoC7

linkC#

https://forms.gle/guMGFMgopnQTdWXf7

https://forms.gle/wffUHrgKBBW5Ca2p9

linkPython

https://forms.gle/TqL8hBNgJKk4gZC7A

https://forms.gle/QPsCTPwvx1vFFe6r9

linkA4 - PR Data points

The methodology used to procure this data is described in the supplementary Git Diff Line Count Data Generation Methodology Document.

linkA5 - Aggregated Results