AI in the Software Lifecycle

80 published findings · November 2026

AI in the Software Lifecycle

Teams that adopt AI agents produce more code. Whether that becomes faster and safer delivery depends on what the organization builds around the model: the checks, the review path, the permissions and the cost controls.

Where gains and costs land
High versus low AI-adoption periods in the same organizations
  1. WriteTasks completed+34%
  2. ReviewMedian review time+442%
  3. DeployDeployments per week−12%
  4. OperateIncidents per pull request+243%
Source: Faros AI, AI Engineering Report 2026, about 22,000 developers. Vendor telemetry; the comparison is correlational.

Findings

Most of the figures below come from vendor telemetry and from companies reporting on themselves. Those sources are unaudited and tend to publish good results, so controlled trials and independent studies carry more weight where they exist.

AI adds code faster than review and operations can absorb it.

Teams that adopt AI tools close more tasks and merge more pull requests. The same teams wait longer for review and see more incidents. In Faros AI's telemetry on about 22,000 developers, high-adoption periods had pull requests 51% larger, median review time about five times higher, about three times the incidents per pull request, and 11.7% fewer deployments a week.

Change in high versus low AI-adoption periods
Same organizations, about 22,000 developers on 4,000+ teams
OutputLoad on people and productionDelivery
Source: Faros AI, AI Engineering Report 2026. Vendor telemetry. Metric names as labeled by Faros. Faros does not define its adoption thresholds, and the comparison is correlational.

Developers' own impressions do not pick this up. In METR's randomized trial, experienced developers took 19% longer with AI tools on their own repositories and believed afterward that they had been 20% faster. A follow-up with newer tools pointed toward a modest speedup, which METR treats as unreliable because many developers declined to do tasks without AI.

Not every organization pays this cost. Salesforce reports merged pull requests per developer up 79% with incidents down 5%. DORA's 2025 survey of nearly 5,000 practitioners found that AI magnifies an organization's existing strengths and weaknesses, and that the quality of its internal platform correlates with the value it gets from AI.

Expected, felt and measured change in task time
METR randomized trial, 16 experienced developers, 246 tasks
Source: METR, 2025, with the confidence interval from METR's February 2026 update. Negative means faster.

Agents are dependable where a test or a metric can verify the result.

Agents do best on work that something other than a person can verify. Airbnb migrated about 3,500 test files in six weeks against an estimate of a year and a half. In several large migrations at Google, AI wrote 70 to 80% of the changes that landed. In work like this the test suite decides whether a change is correct.

Incident response shows the same dependence on verification. On an open benchmark of 500 failures, agents named a correct service 76% of the time and the exact cause 21%. In SREGym's live failure scenarios, fixes succeeded more often than diagnoses were right, and a fix still worked 22 to 62% of the time after a wrong diagnosis. A fix can be verified by watching the system recover, but a diagnosis still needs a person to confirm it.

Incident agents: partial answers versus the exact cause
500 failure cases, average of 11 frontier models
Partial creditExact cause
Source: Fang et al., OpenRCA 2.0, arXiv preprint, June 2026. Academic benchmark, work in progress.

Model choice, caching and pricing tier drive cost more than vendor price changes.

On the same 111 coding tasks, models within about two points of each other on accuracy differed more than fivefold in cost per task. The same model costs twelve times more in its fastest tier than in batch, and the same open-weight model up to twelve times more at one host than another. Cache reads cost a tenth or less of fresh input and make up 72 to 97% of an agent's tokens. By comparison, the largest single-release changes to a vendor's list price were a threefold cut and a doubling.

Token spend is also a small part of engineering cost. In Jellyfish's data, AI tools were about $17 of the $685 average cost of a merged pull request. Jellyfish divides engineering cost by merged pull requests, so the figure falls whenever pull requests multiply.

How much cost varies with each factor
Ratio of highest to lowest cost, log scale
A choice the organization makesA list-price change by one vendor
Sources: SWE-rebench leaderboard (May to July 2026 tasks); OpenRouter endpoint listings; Anthropic, OpenAI and Google pricing pages as of November 2026. A lighter segment shows a range. *The tools also differed by about 10 points in accuracy. †Batch pricing is not usable for interactive work.

Decisions

Decision 1

Measure review time, incidents and deployments alongside output

Developers report speedups that trials do not find, and output rises whether or not delivery improves.

Self-reported speedups are unreliable. The median technical worker in one survey reported a threefold speedup. Inside Anthropic, staff polls put the uplift near twelve times and structured interviews near two. "Share of code written by AI" runs from 22% to nearly 100% depending on who counts and how.

Pull requests and lines of code rise with adoption whether or not delivery improves. Review time, incidents per change and deployment frequency are the measures that moved the wrong way in the findings above, so they belong next to output from the start, with a baseline taken before rollout.

Decision 2

Start agents on work that tests can verify

The clearest wins are migrations with automated validation behind them.

The published successes are campaigns with automated validation behind them: test migrations at Airbnb, code migrations at Google, and more than 2.5 million maintenance changes at Spotify, most merged without a person through tooling that largely predates LLM agents. The first question about a proposed use is what will check the output. Work that only a reviewer can check adds to the review queue, and work with no check at all needs one built before the agent starts. A check keeps its value when the model changes.

The check also has to be strict. About half of the agent changes that passed a project's tests would not have been merged by its maintainers, who also turned down a third of the human-written fixes.

Decision 3

Plan around reviewer capacity

Review time is about five times higher in high-adoption periods.

Most of the added volume ends up waiting for review. Of 600 unmerged agent pull requests in one sample, 38% were closed with no reviewer engaging, more than twice the share that failed CI or tests. Open-source projects reached the limit first: curl ended its bug bounty, and tldraw and Godot restricted outside contributions.

Review works like any queue. Once incoming work gets close to what reviewers can absorb, waits grow much faster than volume.

Assumed team: 30 pull requests a week at baseline, 30 reviewer hours a week, a 40-hour working week. Single shared queue with random arrivals (an M/M/1 model).

Reviewer utilization
Average time in review
Versus baseline volume

An illustration with assumed values, to show the shape of the effect. It treats automatic checks as always right. In practice some of what they clear is wrong, which is the tradeoff described below.

The first way to relieve the queue is to send fewer changes to people. Spotify merges routine maintenance automatically, and the companies Jellyfish calls leading merge about 10% of pull requests with no human review, against 0.7% at the median. How far to open that path depends, for each class of change, on how good its automated checks are and what a bad merge costs. Coinbase says nearly all its code is written with AI and still has cryptographers review core cryptography line by line.

The second is to cut reviewer time per change. At Atlassian, an AI reviewer running across more than 1,900 repositories went with a 30.8% shorter median pull request cycle, by the company's own analysis. It works as a first pass, since in 300 open-source projects 16.6% of AI reviewers' suggestions were adopted against 56.5% of human reviewers'.

Decision 4

Limit retries unless a test can pick the right attempt

Five attempts solve 14 points more tasks than one, at about five times the cost.

Giving an agent more attempts raises its success rate. On 111 fresh coding tasks the top model solved 64.5% in one attempt and 78.4% in five, at about five times the cost. That gain only counts when something can tell which attempt is right. A migration with tests has that, and Airbnb retried its hardest files 50 to 100 times before finishing the last 3% by hand. In feature work the selector is a reviewer, and every extra attempt is more to read. Stripe's agents, which produce more than 1,300 merged pull requests a week, get two CI rounds before the branch goes back to an engineer.

Retries also use CI compute that the token bill does not show. A cap for each type of work makes sense, with more attempts allowed when a check picks the result than when a reviewer has to.

Decision 5

Give incident agents context before autonomy, and automate reversible actions first

Agents name a correct service 76% of the time and the exact cause 21%.

What an incident agent is given matters more than which model runs. Microsoft's RCACopilot scored 0.766 on root-cause category with collected diagnostics and similar past incidents. GPT-4 with the same diagnostics and no past incidents scored 0.026. On another benchmark, removing source-code access lowered accuracy and raised the rate of invented causes for every model. The first investment is what the agent can read: incident history, diagnostics and code. The second is an evaluation set built from the organization's own incidents with the noise left in, since Datadog's evaluations predicted production only after it added irrelevant signals to them.

What the agent may do unapproved is a separate decision, and production systems are cautious. incident.io proposes fixes only as draft pull requests. Azure's SRE agent blocks delete and key-vault commands in every mode. The results above suggest an order for widening authority: investigation first, then actions that can be undone and whose effect shows in a metric, and last anything that depends on the diagnosis being right or cannot be reversed.

Two parts of this are not settled. Withholding uncertain answers, as Meta's system does, trades reach for precision, and the right setting depends on what a wrong answer costs a responder compared with no answer. And no one has published how often automated remediation makes an incident worse, so that rate has to be measured locally as authority widens.

Decision 6

Control cost with model defaults, caching and budgets

Models of similar accuracy differ fivefold in cost per task.

The large levers are the default model, when to route to a cheaper one, how much of each prompt is cached, and what can run in batch. Orchestration belongs on the list: Anthropic's documentation says agent teams use about seven times the tokens of a single session.

Capability at a fixed level gets about 13 times cheaper a year, and bills do not follow, because teams move to the newest model and use more tokens per task. Meta reportedly put AI usage into performance reviews, saw token leaderboards appear, and switched to cost budgets.

Token cost should be weighed against engineering cost. At $17 of a $685 pull request, a cheaper model whose changes take reviewers longer costs more than it saves.

Decision 7

Evaluate models on your own code, and invest in what survives a model change

Public benchmarks are contaminated or frozen, and the same setup varies 2 to 6 points between runs.

Public benchmarks are not a sound basis for choosing a model. OpenAI found 59.4% of the hard SWE-bench Verified tasks its models kept failing were flawed and stopped reporting the benchmark, then found about 30% of the public tasks in its recommended replacement broken. The same agent configuration varies by 2 to 6 points between runs.

A small set of tasks drawn from the organization's own merged changes, run several times and graded on whether the change would be merged, answers what leaderboards cannot. It is enough to catch a bad harness or a large regression, though not to separate two models a few points apart.

Models and prices turn over within months, so the durable investments are the parts that survive a model change: checks, context, permissions, the evaluation set and cost tracking. Building or buying the agent harness is a tradeoff between fit and upkeep. An in-house harness can be shaped to internal tools and checks, and has to be kept current with every model generation.

Decision 8

Long-term effects on code quality and skills are not yet known

Complexity added in the first months stays, and the studies stop at about six months.

The longer-run evidence is thin and mostly unfavorable. After open-source projects adopted Cursor, the jump in output was gone by the third month while code complexity (up 42%) and static-analysis warnings (up 30%) stayed. Repositories that commit agent rules saw about half the rise in complexity, though those teams may simply be more disciplined. Nothing published follows a codebase past about six months of heavy agent use.

The evidence on skills is mixed. New hires who used AI daily reached their tenth pull request in 49 days against 91. Junior developers learning a new library with AI scored 17 points lower on comprehension, most of all on debugging. How much that matters depends on the team: senior engineers who know the codebase have less at stake than juniors being grown into on-call.

The low-cost step is to track complexity and duplication from the first month, so the trend is visible before it is expensive.

Decision 9

Restrict agents by risk, and manage tool access centrally

Most blanket bans were lifted, and secrets leak through tool configuration files.

The blanket bans of 2023 were mostly temporary. Samsung banned generative AI tools in one division that year and rolled out ChatGPT Enterprise and Codex to most staff in 2026. The restrictions in place now are narrow: a category of work at Coinbase, a group of people at Goldman Sachs, a specific tool such as the OpenClaw agent, blocked at Kakao and elsewhere after a serious vulnerability.

The risks behind them are specific. Public commits co-authored by one coding agent leaked secrets at about twice the baseline rate, and more than 24,000 secrets turned up in public MCP configuration files. A prompt injection in a public GitHub issue led an agent to copy private repository data into a public pull request.

Stripe runs one central server exposing about 500 internal tools and gives each agent a small default subset. A single registry with scoped access keeps credentials out of per-developer configuration files.

What to measure first

These figures are missing from the published evidence. An organization can collect each of them itself, and each one affects a decision above.

  1. Reviewer minutes per agent change. Studies report waits and relative changes. METR's rough estimate is 42 minutes of fixing per agent pull request. This sets what automatic checking is worth and what a change costs in full.
  2. Change failure rate for agent-written changes, by class of change. Published comparisons conflict: incidents per pull request rose threefold in one dataset, and one agent's open-source changes were reverted half as often as human ones. The rate for each class decides how wide automatic merging can go.
  3. How often an agent's action in production made an incident worse. There is no public base rate. It sets the pace for widening what an incident agent may do.
  4. Whether an incident agent's confidence is calibrated. Several products gate their output on confidence and none publishes calibration data. Withholding by confidence means little until this is known.
  5. Total cost per merged change. Tokens, CI compute, reviewer time and rework together. Published figures cover tokens alone or divide all engineering cost by pull requests.

All evidence

Every finding used on this page, with its source. Filter by area or producer, or search.

AreaFindingFigureSource
Restrictions on AI tools, 2023 to 2026 (12 cases)
WhenWhoWhat was restrictedScopeStated driverSource

About the evidence

Scope
Published measurements of AI tools and agents in software engineering, 2023 to November 2026: controlled trials, field studies, company telemetry, benchmarks, incident reports, surveys and published policies. Opinion pieces and forecasts are excluded.
Type
How the evidence was produced. Trials and field studies support causal claims. Telemetry and surveys show association. Case studies describe one organization.
Producer
Independent (non-profits and research groups), Academic, Vendor (a company selling AI tools, analytics or services related to the finding), Company (an organization reporting on its own engineering), Government, or Press (reporting where the underlying source is not available).
Strength
Primary means the figure was read in the original report, paper, filing or data file. Secondary means it comes from named press coverage because the original was not available.
Figures
Numbers are reported as published. Where a source's own figures conflict, the page uses the value from its tables and notes the conflict. Prices are list prices in November 2026.