Quote: "To test this theory, we added another question this year about the rework rate for an application: "For the primary application or service you work on, approximately how many deployments in the last six months were not planned but were performed to address a user-facing bug in the application?" Our data analysis confirmed our hypothesis that rework rate and change failure rate are related."

"Change failure rate and rework rate are used when we describe software delivery stability. This factor measures the likelihood deployments unintentionally lead to immediate, additional work."

Page 13

open_in_new

Page 14: "Throughput opposes stability"

Meaningless comparison between highest and lowest made-up clusters

Most organizations are applying "moderate-to-significant" increase in "AI prioritization"

Page 19: "We did find that respondents working in larger organizations report less reliance on AI in their daily work than respondents working in smaller organizations, which is consistent with prior literature indicating larger firms more slowly adapt to technological change because of their higher organizational complexities and coordination costs.5

Page 19: At the individual level, we found that 75.9% of respondents are relying, at least in part, on AI in one or more of their daily professional responsibilities

Page 20: Tasks AI is most used for: 1) code writing 2) summarizing 3) code explanation

Page 22: Mode says: "AI Slightly increased my productivity"

Page 24: Trust in quality of AI? Mode says "Somewhat," 2nd place "A little"

link Section: Exploring the downstream impact of AI

Page 32: If AI adoption increases by 25%, "Time doing valuable work" projected to decrease 2.6%

"Productivity, for example, is likely to increase by approximately 2.1% when an individual’s AI adoption is increased by 25% (see Figure 7). This might seem small, but this is at the individual-level. Imagine this pattern extended across tens of developers, or even tens of thousands of developers."

Page 37: If AI adoption increases by 25%, what are the general impacts on code?

Generally: Less complexity & tech debt, more code quality & especially, documentation quality

How was this benefit determined? "Our goal here is to understand if these aspects seem to vary as a function of adopting AI..."

They seem to be calculating the correlation between organizations w/ "more AI adoption" and organization that claim "low code quality" or "high technical debt"?

Not hard to imagine a variety of ways that "an organization that adopted AI" may vary from an organization that doesn't (including one they mention elsewhere: "startup" vs "big business," which these stats could equally be said to measure?)

The model that the team provides to illustrate how the "If AI increased by 25%" is provided on page 116 of the models section. It does not contradict the assumption that the "Capability" of "AI Adoption" could covary with the characteristics of the company, such as its size.

Page 39: "AI is hurting delivery performance"

If AI adoption increases by 25%... Delivery Stability decreases by 7.2%

Delivery throughput decreases by 1.5%

"Contrary to our expectations, our findings indicate that AI adoption is negatively impacting software delivery performance. We see that the effect on delivery throughput is small, but likely negative (an estimated 1.5% reduction for every 25% increase in AI adoption). The negative impact on delivery stability is larger (an estimated 7.2% reduction for every 25% increase in AI adoption). This data is visualized in Figure 10."

"We hypothesize that the fundamental paradigm shift that AI has produced in terms of respondent productivity and code generation speed may have caused the field to forget one of DORA’s most basic principles—the importance of small batch sizes. That is, since AI allows respondents to produce a much greater amount of code in the same amount of time, it is possible, even likely, that changelists are growing in size. DORA has consistently shown that larger changes are slower and more prone to creating instability"

Page 41: "High-performing teams and organizations use AI, but products don’t seem to benefit."

"Drawing a connection from these outcomes to an individual adopting AI is difficult and noisy. Sometimes it feels like we’re trying to analyze the impact of what you had for lunch today on how well your organization performs this year."

Page 42: If AI adoption increases, how do "Organizational," "Team," and "Product" performance change?

Page 44: Speculation about how "More time using AI" might start by being a net detriment, and progress to being a net benefit

link Section: Platform Engineering

Page 47: Introduction to Platform Engineering

Page 49:

"A key factor in the success is to approach platform engineering with user-centeredness (users in the context of an internal developer platform are developers), developer independence, and a product mindset."

"In this year’s report, we sought to test the relationship between platforms and software delivery and operational performance. We found some positive results. Internal developer platform users had 8% higher levels of individual productivity and 10% higher levels of team performance. Additionally, an organization's software delivery and operations performance increases 6% when using a platform. However, these gains do not come without some drawbacks. Throughput and change stability saw decreases of 8% and 14%, respectively, which was a surprising result."

Graph estimating higher developer productivity when using an "internal developer platform"

Page 50: Internal Developer Platform changes organizational performance based on age

Page 51: "At both the team and individual level we see a 5% improvement in productivity when users of the platform are able to complete their tasks without involving an enabling team. This finding points back to one of the key principles of platform engineering, focusing on enabling selfservice workflows."

Page 53: The unexpected downside

"In the case of throughput, we saw approximately an 8% decrease when compared to those who don’t use a platform. We have hypotheses about what might be the underlying cause."

"When considering the stability of the changes to applications being developed and operated when using an internal developer platform, we observed a surprising 14% decrease in change stability. This indicates that the change failure rate and rate of rework are significantly increased when a platform is being used."

link Section: Developer Experience

Page 57: Developer Experience

Page 60: "Focus on building for your user and you will create delightful products": a graph

"For example, a recent survey showed that 93% of workers reported that it’s important to have a job where they feel the work they do is meaningful. In a similar vein, another survey found that on average, respondents were willing to relinquish 23% of their entire future earnings if it meant they could have a job that was always meaningful."

Page 63: Documentation quality increases with User-centricity

Page 67: Building AI for end users creates stability in priorities, but not stability in delivery.

"We find that teams that have shifted [to adding AI-powered experiences into your service or application] have a significant 10% decrease in software delivery stability relative to teams who have not. Here is a visualization depicting the challenge."

"What can organizations do? Our findings show the negative effects of unstable priorities are resistant to having good leaders, good documentation, and a user-centered approach to software development. This leads us to believe that, aside from creating stability, there’s not much organizations can do to avoid burnout aside from finding ways to (1) stabilize priorities and (2) shield employees from having their day-to-day be impacted by the constant shift in priorities."

Page 69: Leading transformations

Transformational leadership: A recommended model for leading developers

"This year, we saw that transformational leadership leads to a boost in employee productivity. We see that increasing transformational leadership by 25% leads to a 9% increase in employee productivity."

"Our research found a statistically significant relationship between the above qualities of leadership and IT performance in 2017. High-performing teams had leaders with strong scores across all five characteristics and lowperforming teams had the lowest scores. Additionally, we saw that there’s a strong correlation between transformative leadership and Employee Net Promoter Score (eNPS), the likelihood to recommend working at a company."

Graph of estimated impact of "transformational leadership"

Page 74: "Become a data-informed organization"

"The value in the metrics lies in their ability to tell you if you are improving. The four key metrics should be used at the application and service levels, and not at the organization or line-of-business level. The metrics should be used to visualize your efforts in continuous improvement and not to compare teams — and certainly not to compare individuals.

"As our research progresses and evolves, we encourage you to think beyond the four keys. It has become clear that user feedback metrics are as important as the four key metrics. We believe this is because most teams have devised workable solutions for improving speed and stability. As a result, the benefits gained by speed and stability are diminished as higher performance becomes ubiquitous."

"We believe metrics are a requirement for excellence. Metrics facilitate decision making. The more metrics you collect, quantitative and qualitative, the better and more informed decisions you can make. People will always have opinions on the value of the data or the meaning of the data, but using data as the basis by which to make a decision is often preferable to relying on opinion or intuition."

Page 75: "Be all-in on cloud or stay in the data center"