Designing for a Collaborator That Is Sometimes Wrong: The Evidence Behind Human-AI Interaction Design

Sixteen experienced open-source developers worked through 246 real issues in repositories they had contributed to for an average of five years. Each issue was randomly assigned to allow or forbid AI tools. Beforehand, the developers forecast that AI would cut their completion time by 24%; afterwards, they estimated it had cut it by 20%. The measured effect was the opposite: with AI allowed, tasks took 19% longer (Becker et al., 2025, METR, arXiv:2507.09089).

Two years earlier, 95 freelance developers built an HTTP server in JavaScript and the group with GitHub Copilot finished 55.8% faster (Peng et al., 2023, arXiv:2302.06590). Both randomized studies measure different values of the same equation, which Eric Horvitz wrote down in 1999 for an email add-in that guessed when you wanted to schedule a meeting.

Why this matters: Every AI feature is a collaborator that is right some fraction of the time. Whether it saves its user time is settled less by model accuracy than by how cheaply the user can triage, verify and recover from the wrong fraction. Those costs are set by interaction design, and the research measuring them is more mature, and more sobering, than most product teams assume.

TL;DR

• Horvitz's 1999 framework reduces "should the system act?" to a threshold \(p^*\) set by four utilities; making wrong guesses cheaper lowers it as surely as a better model raises \(p\).
• Microsoft's 2019 synthesis condensed 150+ recommendations into 18 guidelines; 49 practitioners testing 20 products found 277 violations against 313 applications.
• In GitHub's 2022 study of 2,047 matched users, acceptance rate (27%) best predicted perceived productivity.
• In the CUPS study, verifying suggestions was the largest single activity, 22.4% of session time.
• A display filter could hide 25% of shown Copilot suggestions while guaranteeing 95% of the hidden ones would have been rejected.
• In the worked example below, a 30%-acceptance assistant loses time once a 2% escaped-defect rate is priced in; about 1.6% is enough to erase every second saved.
• Anthropic reports Claude Code users approve 93% of permission prompts; sandboxing cut prompts by 84%. A confirmation almost always approved is no longer a control.

At a Glance

flowchart LR
    M["Model output, right with probability p"] --> D{"Display policy"}
    D -->|"hide"| H["No cost, no benefit"]
    D -->|"show"| T["User triage"]
    T -->|"reject"| R["Lost glance time"]
    T -->|"accept"| V["Verification cost"]
    V -->|"defect escapes"| E["Downstream cost C"]
    V -->|"correct"| S["Time saved"]

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef purple fill:#6d28d9,stroke:#a78bfa,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    classDef slate fill:#334155,stroke:#64748b,stroke-width:1px,color:#e2e8f0

    class M blue
    class D purple
    class T,V,R amber
    class E rose
    class S emerald
    class H slate

Every branch but one costs the user time. The design problem is to make the green branch outweigh the amber and red ones, and most of the levers act on the costs rather than on \(p\).

From Automation Bias to Mixed Initiative

The field started with autopilots and alarm panels. In 1997 Raja Parasuraman and Victor Riley named the four outcomes (Parasuraman & Riley, 1997, Human Factors 39(2)): use; misuse, or over-reliance on automation beyond its competence; disuse, neglecting automation that would help, typically after false alarms taught the operator to ignore it; and abuse, automating whatever can be automated without regard to human performance. Two years later Linda Skitka, Kathleen Mosier and Mark Burdick split over-reliance into errors of omission, missing a problem because the aid did not flag it, and errors of commission, following an aid's wrong directive (Skitka et al., 1999, IJHCS 51(5)).

The same year, Horvitz published "Principles of Mixed-Initiative User Interfaces" at CHI '99 (Horvitz, 1999). He listed twelve critical factors, among them considering uncertainty about the user's goals, timing services to the user's attention, employing dialog to resolve key uncertainties, and minimising the cost of poor guesses, and gave them a decision-theoretic core.

timeline
    title Designing for fallible automation
    1997 : Parasuraman and Riley name use, misuse, disuse and abuse
    1999 : Horvitz frames mixed initiative as expected utility with a threshold
         : Skitka, Mosier and Burdick measure omission and commission errors
    2010 : Parasuraman and Manzey tie complacency and automation bias to attention
    2019 : Amershi et al. publish 18 guidelines for human-AI interaction
         : Google PAIR releases the People + AI Guidebook
    2021 : Bansal et al. find explanations raise acceptance of wrong advice
         : Bucinca et al. show cognitive forcing works but is disliked
    2022 : Ziegler et al. make acceptance rate the Copilot productivity proxy
    2024 : Mozannar et al. measure verification as the largest share of coding time
    2025 : METR trial finds experienced developers 19 percent slower with AI
         : Anthropic reports sandboxing cut permission prompts by 84 percent
    2026 : Anthropic reports users approve 93 percent of permission prompts

In 2010 Parasuraman and Dietrich Manzey reviewed the accumulated evidence and argued that complacency and automation bias are two faces of one attentional process: when automation is usually right, attention moves away from checking it, and expertise and training do not eliminate the effect (Parasuraman & Manzey, 2010, Human Factors 52(3)). Over-reliance is not a user flaw; it is the rational steady state of someone whose checks have almost never found anything.

In 2019 Saleema Amershi and colleagues at Microsoft consolidated more than 150 recommendations into 18 guidelines grouped by phase, including, for when the system is wrong, "support efficient dismissal", "support efficient correction" and "scope services when in doubt" (Amershi et al., 2019, CHI). They note that 8 of the 18 map onto Horvitz's 1999 principles. Google's People + AI Research group published the People + AI Guidebook on 8 May 2019, six chapters ending with "Errors + Graceful Failure" (Google PAIR, 2019).

[IMAGE: A horizontal strip mapping the 18 Amershi et al. guidelines into four phases (Initially, During interaction, When wrong, Over time), with the eight that trace back to Horvitz 1999 outlined in a second colour. Caption: "Twenty years apart, the same ideas: nearly half of the 2019 guidelines restate 1999 mixed-initiative principles."]

How the Expected-Value Model of Assistance Works

Horvitz's threshold

Horvitz's LookOut system faced a binary choice whenever a user opened an email: offer to schedule a meeting, or stay quiet. Let \(G\) be the event that the user wants the service, \(E\) the observed evidence, and \(A\) the action of offering it. The expected utilities of acting and not acting are

Four utilities describe four outcomes: helping, interrupting, missing, and correctly staying silent. Setting the expectations equal and solving for the probability gives the indifference point:

Read the numerator as the cost of a false alarm, how much worse it is to act unwanted than to stay silent, and the first bracket of the denominator as the benefit of a hit. The threshold is the false-alarm cost as a fraction of false-alarm cost plus hit benefit, and the system should act when \(p(G \mid E) > p^*\).

That formula carries the argument of this post. There are two ways to make an assistant useful more often: raise \(p\), a modelling problem, or lower \(p^*\), a design problem. Lowering \(p^*\) means making the false alarm cheaper (fast dismissal, unobtrusive rendering, a timeout) or the hit more valuable (less effort to accept and verify). Horvitz also added dialog as a third option, producing two thresholds with a band between them where asking beats acting and waiting: the formal origin of the clarifying question.

Rewriting the utilities in seconds

For a code or text suggestion the utilities can be measured in time. Per suggestion shown, let \(a\) be the probability the user accepts it, \(T_s\) the time to produce the content unaided, \(t_v\) the time to verify and adjust an accepted suggestion, \(t_r\) the time lost reading and dismissing a rejected one, \(q\) the probability that an accepted suggestion carries a defect the user did not catch, and \(C\) the average downstream cost of such a defect. The expected time saved per suggestion shown is

and setting it to zero gives the break-even acceptance rate

This is Horvitz's \(p^*\) with \(t_r\) as the false-alarm cost and \(T_s - t_v - qC\) as the hit benefit. Three consequences follow. First, \(t_r\) is paid on every rejection, so at a 27% acceptance rate the toll lands on nearly three suggestions in four. Second, \(qC\) is subtracted from the benefit of acceptance, so a more persuasive assistant with the same defect rate does more damage. Third, as \(T_s - t_v - qC\) approaches zero, \(a^*\) approaches one and no realistic acceptance rate makes the feature worth showing. That is the regime in which checking costs as much as doing.

What acceptance rate measures, and what it hides

GitHub matched 2,047 survey responses to four weeks of IDE telemetry and asked which usage measure best predicted developers' perceived productivity. The winner was the fraction of shown completions accepted, which stood at 27% (Ziegler et al., 2022, MAPS '22, arXiv:2205.06537). The authors called it suitable for "coarse-grained monitoring". In the equation it is one factor of six, silent about \(t_v\), and it rewards the property that inflates \(qC\): plausibility. Hussein Mozannar, Gagan Bansal, Adam Fourney and Horvitz himself went after the hidden terms. In the CUPS study, 21 programmers completed tasks with Copilot, then replayed their recordings and labelled each segment with one of twelve states such as "thinking/verifying suggestion", "waiting for suggestion" and "deferring thought for later" (Mozannar et al., 2024, CHI, arXiv:2210.14306). Of 1,024 suggestions shown, 34.0% were accepted. Verifying suggestions took 22.4% of session time, the largest single state; waiting took 4.2%. Half the participants spent more than 47% of their session in states that exist only because the assistant does. The authors also found considerable verification and editing after acceptance: an acceptance marks the start of a cost, not the end.

[IMAGE: A stacked horizontal bar of one average CUPS session segmented by state, with "Thinking/verifying suggestion" at 22.4% and "Waiting for suggestion" at 4.2% highlighted and a bracket over all assistant-specific states. Caption: "Where the time goes when programming with an assistant: verification is the largest single activity."]

Two modes, two cost structures

Shraddha Barke, Michael James and Nadia Polikarpova observed 20 programmers across four languages and found the interaction bimodal (Barke et al., 2023, OOPSLA, arXiv:2206.15000). In acceleration mode the programmer knows what comes next and uses the assistant to get there faster; interactions are quick and must not break flow. In exploration mode the programmer is unsure how to proceed and uses the assistant to see options, with explicit prompting and more extensive validation.

These are different parameter regimes: acceleration has small \(T_s\) and \(t_v\) and is highly sensitive to \(t_r\); exploration has large \(T_s\) and \(t_v\) and tolerates \(t_r\). Barke and colleagues concluded that acceleration wants only short, high-confidence suggestions, while exploration wants ways to compare alternatives and validate through tests.

Showing is an action

Sometimes the best suggestion is none. Mozannar and colleagues learned from telemetry to predict whether a Copilot suggestion would be accepted, and used the prediction to decide whether to display it (Mozannar et al., 2023, arXiv:2306.04930). They could hide 25% of shown suggestions while guaranteeing that 95% of the hidden ones would have been rejected, and skip generating 13% of those entirely, saving latency as well as attention. That is Horvitz's threshold, implemented as a filter in front of a modern model.

Seeing It in Motion

The CUPS states form a loop in which acceptance is a transition, not an end state.

stateDiagram-v2
    state "Writing code" as W
    state "Prompt crafting" as P
    state "Waiting for suggestion" as Wt
    state "Verifying suggestion" as V
    state "Editing accepted code" as Ed
    state "Deferred verification" as Df
    [*] --> W
    W --> P : needs help
    P --> Wt : trigger
    W --> Wt : inline trigger
    Wt --> V : suggestion shown
    V --> W : reject
    V --> Ed : accept and fix
    V --> Df : accept unchecked
    Df --> V : return later
    Ed --> W : done
    Df --> [*] : never returns

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    class W,P blue
    class Wt,V,Ed amber
    class Df rose

The rose state is accepted code nobody has verified yet. CUPS participants who deferred usually returned to verify, but every path from that state to the end of a session is where \(q\) comes from.

Agents share the structure, with a suggestion replaced by an action and a bug by a deleted directory. The design that emerged in 2025 and 2026 reserves human attention for a few decisions.

sequenceDiagram
    participant U as User
    participant A as Agent
    participant S as Sandbox policy
    participant C as Action classifier
    A->>S: Proposed shell command
    S-->>A: Inside boundary, run without asking
    A->>S: Command reaching outside workspace
    S->>C: Escalate for review
    C->>C: Fast yes or no filter
    Note over C: Flagged items get a second reasoning pass
    C-->>A: Blocked, agent tries another path
    C->>U: Rare case needs a human
    U-->>A: Approve or deny with attention

Anthropic's account of Claude Code follows this shape. Its October 2025 engineering post reports that filesystem and network sandboxing "safely reduces permission prompts by 84%" in internal usage, explicitly to fight approval fatigue (Anthropic, 2025). Its March 2026 post on auto mode opens with the statistic that users approve 93% of permission prompts, and describes a two-stage classifier that replaces most of the rest (Anthropic, 2026). In equation terms the sandbox does not raise \(p\) at all; it shrinks \(C\) inside the boundary, driving the false-alarm cost of acting toward zero and \(p^*\) with it.

[IMAGE: A two-panel plot of Horvitz's expected-utility lines against p. Left: steep false-alarm cost, threshold p near 0.5. Right: after one-keystroke dismissal and a sandbox, the "act" line's left intercept rises and p slides to about 0.25. Caption: "Lowering the cost of a wrong guess moves the threshold as much as a better model would."]

By the Numbers

Sources: Amershi et al., 2019; Ziegler et al., 2022; Peng et al., 2023; Mozannar et al., 2024; Mozannar et al., 2023; Becker et al., 2025; Ancker et al., 2017; Anthropic engineering posts of October 2025 and March 2026, vendor-reported and not independently replicated; the 17% figure rests on only 52 examples.

Acceptance rates cluster between a quarter and a third, so the rejection toll applies to most suggestions, and the two RCTs disagree in sign.

A Concrete Example

A team must decide whether an inline code assistant's suggestions are worth showing. Every input below is an assumption chosen for illustration, anchored to measured values where possible; substitute your own telemetry.

Step 1: parameters. Acceptance \(a = 0.30\) (between Ziegler's 27% and CUPS's 34%). Unaided time \(T_s = 20\) s for a two-line completion. Verification \(t_v = 6\) s. Rejection toll \(t_r = 2\) s. Escaped-defect probability \(q = 0.02\). Defect cost \(C = 600\) s, ten minutes of later debugging.

Step 2: 100 shown suggestions.

• Gains from accepted suggestions: \(30 \times (20 - 6) = 420\) s.
• Toll on rejections: \(70 \times 2 = 140\) s.
• Escaped defects: \(30 \times 0.02 = 0.6\) defects, costing \(0.6 \times 600 = 360\) s.

Net: \(420 - 140 - 360 = -80\) s, or \(-0.8\) s per suggestion. The gain is felt at acceptance; the defect cost arrives later and is blamed on the bug.

Step 3: thresholds. The hit benefit is \(T_s - t_v - qC = 20 - 6 - 12 = 2\) s, so

At 30% acceptance the assistant is far below break-even. Holding \(a = 0.30\) and solving \(0.30 \times (14 - 600q) = 0.70 \times 2\) gives \(q = 2.8 / 180 \approx 0.016\): an escaped-defect rate of about 1.6% per accepted suggestion erases every second saved.

Step 4: add a display filter. Use the published operating point: hide 25 of 100 suggestions, of which 95% (23.75) would have been rejected and 1.25 accepted. Now 75 are shown, 28.75 accepted, 46.25 rejected.

• Gains: \(28.75 \times 14 = 402.5\) s.
• Toll: \(46.25 \times 2 = 92.5\) s.
• Defects: \(28.75 \times 0.02 \times 600 = 345\) s.

Net: \(-35\) s. Filtering removes the toll on bad suggestions but not the defects in accepted ones.

Step 5: make verification sharper. The editor runs the affected tests and renders the suggestion as a diff with any failing assertion highlighted. Verification rises to 8 s, but the escaped-defect rate halves to 1%.

• Gains: \(28.75 \times (20 - 8) = 345\) s.
• Toll: \(92.5\) s.
• Defects: \(28.75 \times 0.01 \times 600 = 172.5\) s.

Net: \(+80\) s per 100 opportunities, and break-even acceptance falls to \(2 / (2 + 20 - 8 - 6) = 0.25\). Spending more time on verification flipped the sign; a team optimising time-to-accept would have rejected the change.

[IMAGE: A waterfall chart for 100 suggestion opportunities across three scenarios (baseline, filtered, filtered plus test-backed verification), each decomposed into gains, rejection toll and defect cost, ending at -80 s, -35 s and +80 s. Caption: "The sign of an assistant's value flips on the defect term, and the defect term is set by how verification is designed."]

Where It Breaks

Explanations raise acceptance, not accuracy

The intuitive fix for over-reliance is explanation. Gagan Bansal and colleagues tested it on three datasets with an AI roughly as accurate as the humans (Bansal et al., 2021, CHI). AI assistance produced complementary performance, but explanations did not add to it; they increased the chance that people accepted the recommendation regardless of its correctness. In the equation, an explanation raises \(a\) on right and wrong suggestions alike, so \(q\) rises with it. Persuasion is not verification.

Friction works, and users dislike it

Zana Buçinca, Maja Malaya and Krzysztof Gajos tried the opposite: cognitive forcing designs such as asking for the user's own decision first, showing the AI's suggestion only on demand, or making the user wait (Buçinca et al., 2021, CSCW, arXiv:2102.09692). With 199 participants, cognitive forcing significantly reduced over-reliance compared with simple explainable-AI designs. The designs that reduced it most received the least favourable ratings, and the benefit went mostly to people who enjoy effortful thinking. The intervention that trades higher \(t_v\) for lower \(q\) is the one users switch off.

Confirmations decay into reflexes

In a study of 112 primary-care clinicians, the likelihood of accepting a clinical reminder dropped by 30% for each additional reminder per encounter, and a third of reminders were repeats for the same patient within the year (Ancker et al., 2017, BMC Med Inform Decis Mak). Agent permission prompts are alerts by another name, and a 93% approval rate describes a prompt that nearly always returns "yes". The rare prompt that deserved a "no" is processed by the same reflex as the thousand before it.

[IMAGE: Line chart of relative alert acceptance against alerts per encounter, falling 30% per additional alert, overlaid with a flat bar marking the 93% prompt-approval rate. Caption: "More confirmations, less attention per confirmation."]

Suppose, as an illustrative assumption, that 1 in 200 agent actions is genuinely harmful. Prompting on everything yields 1,000 prompts for 5 hazards, caught at whatever rate a fatigued reflex delivers. On Anthropic's published figures, its classifier pipeline would block about 4 of the 995 benign actions (0.4%) and, if its 17% miss rate on real overeager actions carried over, let through roughly 0.85 of the 5. Whether that beats a human is an empirical question the vendor has framed in its favour; what is clear is that a thousand attentive human judgements are not on offer.

Perception runs the wrong way

The METR gap, 20% faster by self-report against 19% slower as measured, breaks the feedback loop product teams rely on. Surveys and acceptance rate both sample the moment of acceptance, while the costs arrive later. Tuning to perceived productivity tunes the part of the equation that is always positive.

Latency is a term, not a backdrop

Nielsen's limits put the feeling of instantaneity near 0.1 s, uninterrupted flow near 1 s, and sustained attention near 10 s (Nielsen, 1993). CUPS measured 4.2% of session time spent waiting. Streaming shortens time to first token but invites acting on half an answer before its qualification arrives, which quietly raises \(q\).

The competence boundary

The model assumes the user can verify. Outside the user's competence, \(t_v\) is undefined and \(q\) approaches the model's raw error rate. The one lever left is refusing to present the output as finished: "scope services when in doubt".

Alternative Designs

No design dominates. Threshold display is wrong in exploration mode, where users want the low-probability options, and cognitive forcing, the best-evidenced fix for over-reliance, is the least likely to survive a product review.

How It Is Used in Practice

Acceptance rate is best used as Ziegler's paper suggests: a coarse health signal, not an objective. A team shipping the Step 5 change, which lengthens verification to cut defects, needs metrics that can see \(q\): reverted suggestions, defects traced to accepted completions, and time from acceptance to the next edit of the same lines.

Amershi's guidelines were built for heuristic evaluation: assign each reviewer a feature, find applications and violations of each guideline, rate each on a five-point scale. PAIR's errors chapter asks the question this post turns into algebra: can users move forward after an AI failure?

[IMAGE: Dashboard mock with four tiles: acceptance rate, median seconds from acceptance to next edit of the same lines, 7-day revert rate of accepted suggestions, and net seconds per suggestion from the equation. Caption: "Acceptance rate is one tile of four."]

For agents, the pattern visible in Claude Code by September 2026, sandbox by default, classifier for what the sandbox cannot decide, human for the residue, is Horvitz's three-way choice among acting, asking and waiting, rebuilt as infrastructure. Its most transferable detail: a blocked action costs one retry, not the session, which is "support efficient dismissal" applied to the machine.

Insights Worth Remembering

1. The formula is from 1999. Horvitz's \(p^*\) already says an assistant should act when the probability of being wanted exceeds false-alarm cost over false-alarm cost plus hit benefit.

2. Design moves the threshold; modelling moves the probability. Cheap dismissal, sandboxes and undo lower \(p^*\) without touching the model.

3. Acceptance marks the start of a cost. A metric that stops counting at the accept keystroke misses the most expensive part.

4. Persuasiveness and defect rate compound. Escaped defects scale with accepted suggestions, so anything that raises acceptance without raising correctness, explanations included, makes the assistant worse.

5. The sign can flip on a small number. In the worked example a 1.6% escaped-defect rate erased all savings. Few teams know their \(q\).

6. More time verifying can mean less time overall. Two extra seconds of test-backed checking turned a loss into a gain; optimising time-to-accept would have rejected it.

7. A prompt approved 93% of the time is not a control. Spend human attention on rare, irreversible actions.

8. Perceived and measured productivity can differ in sign. Self-reports measure experience, not the equation.

Open Questions

Why do the two RCTs disagree in sign? Measured: a well-specified greenfield task ran 55.8% faster for freelancers; mature-repository issues ran 19% slower for their own maintainers. Candidates are expert familiarity (small \(T_s\)), high review standards (large \(t_v\)), and different tools two years apart; no experiment has isolated which dominates.

Can \(q\) be measured in production? Revert rates conflate style changes with defect fixes and miss late defects. As of September 2026, no published method attributes downstream defects to accepted suggestions at scale with known error.

Does fatigue return under classifier-mediated agents? Alarm-fatigue research predicts rarer prompts get more attention; for agent approvals this is unmeasured outside the vendor, and users may assume the classifier already checked.

Is there an explanation that reduces over-reliance without friction? Highlighting uncertain spans or surfacing counter-evidence are candidates; evidence is thin.

Can an interface infer acceleration versus exploration reliably? The modes imply different thresholds; whether a policy can switch between them without surprising users is untested.

Sources and Further Reading

1. Horvitz, E. (1999). "Principles of Mixed-Initiative User Interfaces." Proceedings of CHI '99, 159-166. doi:10.1145/302979.303030
2. Parasuraman, R., & Riley, V. (1997). "Humans and Automation: Use, Misuse, Disuse, Abuse." Human Factors, 39(2), 230-253. doi:10.1518/001872097778543886
3. Skitka, L. J., Mosier, K. L., & Burdick, M. (1999). "Does Automation Bias Decision-Making?" International Journal of Human-Computer Studies, 51(5), 991-1006. doi:10.1006/ijhc.1999.0252
4. Parasuraman, R., & Manzey, D. H. (2010). "Complacency and Bias in Human Use of Automation: An Attentional Integration." Human Factors, 52(3), 381-410. doi:10.1177/0018720810376055
5. Amershi, S., Weld, D., Vorvoreanu, M., et al. (2019). "Guidelines for Human-AI Interaction." Proceedings of CHI 2019. doi:10.1145/3290605.3300233
6. Google People + AI Research (2019). People + AI Guidebook, published 8 May 2019. pair.withgoogle.com/guidebook
7. Ziegler, A., Kalliamvakou, E., Simister, S., et al. (2022). "Productivity Assessment of Neural Code Completion." MAPS '22. arXiv:2205.06537
8. Barke, S., James, M. B., & Polikarpova, N. (2023). "Grounded Copilot: How Programmers Interact with Code-Generating Models." Proc. ACM Program. Lang., 7(OOPSLA1), 85-111. doi:10.1145/3586030
9. Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2024). "Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming." Proceedings of CHI 2024. arXiv:2210.14306
10. Mozannar, H., Bansal, G., Fourney, A., & Horvitz, E. (2023). "When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming." arXiv:2306.04930
11. Peng, S., Kalliamvakou, E., Cihon, P., & Demirer, M. (2023). "The Impact of AI on Developer Productivity: Evidence from GitHub Copilot." arXiv:2302.06590
12. Becker, J., Rush, N., Barnes, B., & Rein, D. (2025). "Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity." METR. arXiv:2507.09089
13. Bansal, G., Wu, T., Zhou, J., et al. (2021). "Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance." Proceedings of CHI 2021. doi:10.1145/3411764.3445717
14. Buçinca, Z., Malaya, M. B., & Gajos, K. Z. (2021). "To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making." Proc. ACM Hum.-Comput. Interact., 5(CSCW1). arXiv:2102.09692
15. Ancker, J. S., Edwards, A., Nosal, S., et al. (2017). "Effects of Workload, Work Complexity, and Repeated Alerts on Alert Fatigue in a Clinical Decision Support System." BMC Medical Informatics and Decision Making, 17, 36. doi:10.1186/s12911-017-0430-8
16. Nielsen, J. (1993). "Response Times: The 3 Important Limits." Nielsen Norman Group. nngroup.com
17. Anthropic (2025, 20 October). "Beyond Permission Prompts: Making Claude Code More Secure and Autonomous." anthropic.com
18. Anthropic (2026, 25 March). "How We Built Claude Code Auto Mode: A Safer Way to Skip Permissions." anthropic.com

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.