Hidden Technical Debt, a Decade On: What Continuous Delivery for ML Actually Fixed

When Google's ML Test Score team interviewed the maintainers of 36 production machine learning systems, none of their 28 tests was implemented by more than 80% of teams, and the least implemented was the check for training/serving skew, the one "responsible for production issues across a wide swath of teams" (Breck et al., 2017, The ML Test Score, IEEE Big Data). The same year, Google's TFX paper reported what fixing that check was worth on one product: Google Play found features "always missing from the logs, but always present in training", and removing the skew raised the app install rate on the store's main landing page by 2% (Baylor et al., 2017, TFX, KDD).

The fixes were worth money, but the ones built first were the ones that could become infrastructure. Pipelines, registries, canaries and rollbacks are now commodity. The debts that need someone to decide what "good" means (which slices matter, how many labels a gate needs) sit about where Sculley and his colleagues left them in 2015. LLM applications moved that balance into prompts, hosted endpoints and evaluation sets, where it is harder to see.

Why this matters: Most teams can now ship a model the way they ship a binary. Far fewer can say whether the gate that approved it could have caught the regression it exists to catch. Knowing which debts continuous delivery retired, and which it automated around, separates a safe pipeline from one that looks safe.

TL;DR

• Continuous delivery retired the 2015 anti-patterns (glue code, pipeline jungles) and much configuration debt, and almost none of the feedback-loop or undeclared-consumer debt.
• The ML Test Score takes the minimum of four section scores. In the worked example a pipeline averaging 4.4 scores 3.0, because Infrastructure, the part CD tooling automates, leads while Monitoring lags.
• The rubric gives a full point for automating a test regardless of its power. A gate comparing accuracy on 2,000 reused labels against a 0.5-point tolerance blocks an unchanged model about 19% of the time.
• Distribution-free guarantees are expensive: one (ε = 0.01, 1 − δ = 0.9999) accuracy estimate needs more than 46,000 labels, and 32 adaptive commits at ε = 0.01 need 156,955 (Renggli et al., 2019, ease.ml/ci).
• For LLM applications the model became a dependency that changes without a commit: GPT-4's accuracy on a prime/composite task fell from 84.0% to 51.1% between the March and June 2023 versions (Chen, Zaharia & Zou, 2023).
• The unsolved core is the specification: people refine their grading criteria while grading (Shankar et al., 2024, Who Validates the Validators?).

At a Glance

flowchart LR
    subgraph Change["Three axes of change"]
        C1["Code and config"]
        C2["Training data"]
        C3["Model, prompt, provider"]
    end
    CI["Deterministic CI"]
    CT["Pipeline run and CT"]
    G["Statistical gate"]
    PD["Canary and shadow"]
    MON["Monitoring"]
    J["Judgement debt"]

    C1 --> CI
    C2 --> CT
    CI --> CT
    CT --> G
    C3 --> G
    G --> PD
    PD --> MON
    MON -->|"retrain trigger"| CT
    J -.->|"what to test, thresholds, slices"| G
    J -.->|"what to alert on"| MON

    classDef blue fill:#1e40af,stroke:#3b82f6,stroke-width:1px,color:#fff
    classDef emerald fill:#047857,stroke:#34d399,stroke-width:1px,color:#fff
    classDef amber fill:#b45309,stroke:#fbbf24,stroke-width:1px,color:#fff
    classDef rose fill:#be123c,stroke:#fb7185,stroke-width:1px,color:#fff
    class C1,C2,C3 blue
    class CI,CT,PD emerald
    class G,MON amber
    class J rose

Green stages are what tooling turned into infrastructure. The amber stages exist everywhere too, but their correctness depends on the rose input: human decisions about what to measure and how much evidence counts.

Before the Pipelines

In 2014 a group of Google engineers called a workshop paper "Machine Learning: The High Interest Credit Card of Technical Debt." The expanded version, "Hidden Technical Debt in Machine Learning Systems," appeared at NIPS 2015 and became the founding text of MLOps (Sculley et al., 2015). ML systems, it argued, "have all of the maintenance problems of traditional code plus an additional set of ML-specific issues."

Its best-known figure, a small box of "ML code" among much larger ones, is often quoted as a measurement. It is not one. Its one number on the point is hedged: a mature system "might end up being (at most) 5% machine learning code and (at least) 95% glue code." That is an argument about where effort goes, not a survey result.

timeline
    title From debt catalogue to eval gates
    2014 : High Interest Credit Card workshop paper names ML technical debt
    2015 : Hidden Technical Debt in ML Systems at NIPS
         : CACE principle, pipeline jungles, configuration debt
    2017 : ML Test Score rubric scores 36 Google teams
         : TFX platform cuts time to production from months to weeks
    2019 : CD4ML adds data and model as versioned axes
         : ease.ml/ci puts statistical guarantees on ML CI
    2022 : Shankar et al. interview 18 ML engineers
    2023 : Chen et al. measure GPT-4 behaviour drift
         : Narayanan and Kapoor dispute it as behaviour, not capability
    2024 : SPADE mines prompt histories for assertions
         : EvalGen study names criteria drift
    2025 : Batch-invariant kernels make temperature-zero inference reproducible

The categories map onto what came next: entanglement (CACE, "Changing Anything Changes Everything"), correction cascades and undeclared consumers, unstable data dependencies where "even 'improvements' to input signals may have arbitrary detrimental effects," the glue code and pipeline jungle anti-patterns, and configuration debt.

[IMAGE: A two-column ledger of the 2015 debt categories with a status bar each: green for retired by tooling (glue code, pipeline jungles, configuration, reproducibility), amber for partial (skew, data dependencies), red for untouched (undeclared consumers, feedback loops). Caption: "The debts that look like code are paid. The ones that look like judgement are still on the books."]

In 2017 the ML Test Score turned the catalogue into 28 scoreable assertions and TFX turned it into a platform. By 2019 Thoughtworks had defined Continuous Delivery for Machine Learning (CD4ML), noting that ML applications "are subject to change in three axis: the code itself, the model, and the data" (Sato, Wider & Windheuser, 2019). A Microsoft survey of 551 engineers found Sculley's entanglement again, in models that "affect one another during training and tuning" (Amershi et al., 2019, ICSE-SEIP).

How Continuous Delivery for ML Actually Works

Training is compilation, and the source includes the data

The ML Test Score supplies the organising analogy: "consider ML training as analogous to compilation, where the source is both code and training data," so a trained model "needs production practices like a binary does, such as debuggability, rollbacks and monitoring."

Most CD machinery follows: build the model once, store it immutably, promote rather than rebuild, roll back by pointer, and version and review data and configuration as source.

The analogy also shows where CD stops. A compiler is deterministic; training "is often not reproducible in practice" (Infra 1). So the build is verified not by diffing outputs but by measuring behaviour, which makes verification a statistical estimate.

The ML Test Score as a scoring function

The rubric has four sections of seven tests: Data, Model, Infrastructure, Monitoring. "Half a point is awarded for executing the test manually, with the results documented and distributed"; "a full point is awarded if there is a system in place to run that test automatically on a repeated basis." Then "the final ML Test Score is computed by taking the minimum of the scores aggregated for each of the 4 sections." With \(s_{k,i} \in \{0, \tfrac{1}{2}, 1\}\) the score of test \(i\) in section \(k\):

The minimum gives the rubric its force. Under a mean, a point is a point wherever it lands. Under a minimum, the marginal value of improving section \(k\), with section total \(T_k = \sum_i s_{k,i}\), is

(strictly a subgradient; ties split it). Automating a fourth canary check in a system with weak monitoring moves the score by exactly zero, a cost the authors accepted because "all four sections are important." Their bands: \((2,3]\) is "reasonably tested"; \((3,5]\) is "strong levels of automated testing and monitoring, appropriate for mission-critical systems"; above 5 is "exceptional."

What the function cannot see is power. A test scores 1 if it is automated and repeated, whether or not it can detect the failure it names.

Levels of automation, and what stays manual

Google Cloud's architecture guide defines three levels (Google Cloud, MLOps: Continuous delivery and automation pipelines in machine learning, last reviewed 2024-08-28). At level 0 "every step is manual" and "a new model version is deployed only a couple of times per year." Level 1 automates continuous training (CT) with data and model validation. Level 2 adds CI/CD for the pipeline itself: registry, feature store, metadata store, orchestrator.

Even at level 2, "the data analysis step is still a manual process" and "the model analysis step is also a manual process." And the 2024 revision says it "applies primarily to predictive AI systems," not LLM applications.

[IMAGE: Three swimlanes, Level 0, 1, 2, each with six boxes (data analysis, prep, training, evaluation, validation, serving); automated boxes filled, manual outlined. In Level 2, data and model analysis stay outlined. Caption: "Automation grows around the analysis steps, not through them."]

CD4ML made the three axes first-class, each versioned with its own tool (DVC, MLflow, GoCD in its sample application), with schema checks, contract tests and "Threshold Tests or ratcheting" against a baseline. It warned that "if we always compute metrics against the same dataset, we can start overfitting," and kept manual promotion stages so release stays "a business decision rather than a technical one."

The gate is an estimator, and estimators have sample sizes

Every validation step ends in a comparison. From TFX: "We evaluate prediction quality by comparing the model quality against a fixed threshold as well as against a baseline model... Any new model failing any of these checks is not pushed to serving." That is a hypothesis test whether or not anyone calls it one.

ease.ml/ci is the clearest treatment of its cost. For a variable with range \(r_v\), tolerance \(\epsilon\) and failure probability \(\delta\), Hoeffding's inequality gives

CI makes this worse because it is adaptive: the developer sees pass or fail and picks the next commit accordingly, so the test set leaks. The paper bounds this by counting histories. A developer whose next model depends only on past pass/fail bits can produce at most \(2^H\) histories, so

The \(2^H\) sits inside a logarithm, which is why the bound is usable at all. The paper's numbers: for \(n > 0.8 \pm 0.05\) with \(H = 32\) and \(\delta = 0.0001\), \(N = 6{,}279\); at \(\epsilon = 0.01\) it "blows up to 156,955." Its optimisations for common conditions "lower the number of labels required by up to two orders of magnitude."

Hoeffding bounds are worst-case. The practical alternative exploits the fact that candidate and baseline are scored on the same examples. Let \(p_{10}\) be the share the baseline gets right and the candidate wrong, \(p_{01}\) the reverse. The paired estimate is \(\hat\Delta = p_{01} - p_{10}\), with

Only discordant examples contribute variance, so two models that disagree on 6% of cases compare far more tightly than two independent accuracy estimates near 90%. Most gates, classical or LLM, were built without it.

What LLM applications changed

The model became an unstable data dependency. Sculley warned about signals where "updates to the input signal may be made at any time," and recommended "a versioned copy." A hosted model is exactly this, at the centre of the system. Chen, Zaharia and Zou found that "the behavior of the 'same' LLM service can change substantially in a relatively short amount of time." The rubric's Monitor 1, "Dependency changes result in notification," was one test in 28; for an LLM application the others depend on it.

Prompts became configuration debt with a diff history. SPADE analysed "19 LLM pipelines collected from LangChain users," each with three to 11 prompt versions (Shankar et al., 2024, SPADE). About 35% of the categorised deltas were structural and 65% content-based, each a requirement discovered in production and written into configuration instead of a test. SPADE turns deltas back into assertions, and "reduces the number of assertions by 14% and decreases false failures by 21%" against simpler baselines.

The test oracle became a model. LLM graders "simply inherit all the problems of the LLMs they evaluate, requiring further human validation." In a study with nine practitioners, grading outputs helped people "define that very criteria" they were grading against; the authors named this criteria drift and concluded that "it is impossible to completely determine evaluation criteria prior to human judging of LLM outputs" (Shankar et al., 2024).

Even determinism failed for a while. One prompt sampled 1,000 times at temperature 0 on Qwen3-235B produced 80 unique completions, the most common "occuring 78 times"; with batch-invariant kernels, "all of our 1000 completions are identical" (He and Thinking Machines Lab, 2025). A team calling a hosted endpoint does not control the kernels.

Seeing It in Motion

In an LLM application, a prompt change and a silent provider change are both deploys, but only one opens a pull request.

sequenceDiagram
    participant Dev as Developer
    participant CI as CI runner
    participant Eval as Eval harness
    participant Judge as Pinned judge
    participant Reg as Release registry
    participant Prod as Production

    Dev->>CI: PR edits prompt template v7 to v8
    CI->>CI: Schema, parsing and exact-match checks
    CI->>Eval: Run paired suite, baseline v7 vs candidate v8
    Eval->>Judge: Grade discordant outputs
    Judge-->>Eval: Pass or fail per case
    Eval-->>CI: Delta estimate with standard error
    CI->>Reg: Record manifest of prompt, model id, judge id
    Reg->>Prod: Canary at small traffic share
    Note over Prod: Provider updates the model behind an alias
    Prod-->>Eval: Nightly run on main detects shift with no diff

Whether the artefact is a trained model or a prompt-plus-model manifest, its lifecycle is the same state machine.

stateDiagram-v2
    [*] --> Candidate
    Candidate --> Rejected : deterministic checks fail
    Candidate --> Evaluated : checks pass
    Evaluated --> Rejected : gate says regression
    Evaluated --> Canary : gate says no regression
    Canary --> RolledBack : online metrics degrade
    Canary --> Serving : canary healthy
    Serving --> RolledBack : alert or incident
    Serving --> Superseded : newer artefact promoted
    RolledBack --> [*]
    Superseded --> [*]
    Rejected --> [*]

[IMAGE: The state machine with each transition labelled by its governing rubric item: Evaluated to Canary, Infra 4; Canary to Serving, Infra 6; Serving to RolledBack, Infra 7 and Monitor 7. Caption: "The Infrastructure section describes this state machine. The Monitoring section decides when the backward edges fire."]

By the Numbers

Sources: Sculley et al. (NIPS 2015); Breck et al. (2017); Baylor et al. (KDD 2017); Renggli et al. (arXiv:1903.00278); Shankar et al. (arXiv:2209.09125); Chen et al. (arXiv:2307.09009); SPADE (arXiv:2401.03038); He (2025). The 5% figure is an upper bound offered as argument; the 2% gain is one product's result.

Read the evidence column first: the most quoted number is an estimate, the most rigorous are derivations. No large controlled study shows that adopting CD for ML reduces incidents, and that absence is itself a finding.

[IMAGE: Log-scale bars of labels for an accuracy gate: 2,000 (typical holdout), 6,279 (32 adaptive commits, ε = 0.05), 14,000 (worked example at 1% false blocks), 46,000 (one Hoeffding estimate, ε = 0.01), 156,955 (32 adaptive commits, ε = 0.01). Caption: "The holdout most teams have and the holdout their tolerance implies are often an order of magnitude apart."]

A Concrete Example

A subscription business retrains a churn model weekly through an orchestrated pipeline. The team scores itself and audits the gate it trusts most.

Step 1: score each section (0 not done, ½ manual and documented, 1 automated):

Step 2: aggregate. The mean is \((4.5 + 4.0 + 6.0 + 3.0)/4 = 4.375\), which would read "appropriate for mission-critical systems." The rubric's score is \(\min(4.5, 4.0, 6.0, 3.0) = 3.0\): "reasonably tested."

Step 3: find the cheapest band change. Automating the skew check (Monitor 3, 0 to 1) lifts Monitoring to 4.0, and \(S = \min(4.5, 4.0, 6.0, 4.0) = 4.0\), band \((3,5]\). One test moves the system up a band; five more infrastructure checks would move it nowhere.

Step 4: audit the gate that scored a full point. Infra 4 is automated: each candidate is compared with production on a frozen holdout of \(N = 2{,}000\) labels and blocked if its accuracy is more than 0.5 points lower. It has run on 32 candidates, and engineers saw every verdict.

Step 5: what a distribution-free guarantee needs. Splitting the difference \(n - o\) so each term is estimated to \(\epsilon/2\) at \(\delta/2\) gives \(N = 2\ln(2/\delta)/\epsilon^{2}\). At \(\delta = 0.05\), \(\epsilon = 0.005\): \(\ln 40 = 3.689\), so \(N = 7.378/0.000025 \approx 295{,}000\). Inverting, 2,000 labels support \(\epsilon = \sqrt{7.378/2000} = 0.061\), about ±6 points. For 32 adaptive uses, \(\ln(2 \cdot 2^{32}/\delta) = 0.693 + 22.181 + 2.996 = 25.87\), so \(\epsilon = \sqrt{51.74/2000} = 0.161\), ±16 points. Loose, but the order of magnitude is the lesson.

Step 6: what the paired estimate delivers. Take a truly unchanged candidate that disagrees with production on 6.5% of the holdout, split evenly, \(p_{10} = p_{01} = 0.0325\):

The chance the gate blocks it is \(\Phi(-0.5/0.57) = \Phi(-0.877) \approx 0.19\). Over 32 weeks that is about 6 expected false blocks, and \(1 - 0.81^{32} \approx 0.999\) odds of at least one. A real 1.5-point regression is blocked with probability \(\Phi((1.5 - 0.5)/0.57) = \Phi(1.75) \approx 0.96\). It catches large drops and blocks unchanged models one week in five.

Step 7: size the fix. A 1% false-block rate at the same tolerance needs \(0.5/\text{SE} \ge 2.326\), so SE ≤ 0.215 points and \(N \ge 0.065/0.00215^{2} \approx 14{,}000\) labels. Or widen the tolerance to about 1.2 points, or route borderline results to a human.

Step 8: the observation. Infra 4 earned its full point honestly, and is still a weak test. After a few false blocks the likely response, which Shankar et al. document for alerts as "fatigue and silencing of alerts," is to loosen the threshold until it stops firing. None of this shows in the score.

[IMAGE: Two normal curves of measured accuracy change, centred at 0 (unchanged) and −1.5 (regression), SD 0.57, with the gate at −0.5. Shade 19% of the unchanged curve rose (false blocks) and 4% of the regression curve amber (missed regressions). Caption: "A 2,000-label gate: good at catching a 1.5-point drop, bad at leaving unchanged models alone."]

Where It Breaks

The rubric measures presence, not power

The worked example is the general case. TFX's authors explained why loose thresholds seemed enough: "most bugs cause dramatic changes to model quality metrics that can be caught by using loose thresholds. However, there is a strong selection bias here, since more subtle issues may not have drawn our attention."

Test sets wear out, and golden files get blindly updated

A holdout used as a merge gate is used adaptively, and the ML Test Score has no test for it. LLM golden sets are worse: small, hand-curated, and the thing prompts get tuned against. The field also disagrees with itself here. The ML Test Score warns against "golden tests... difficult to maintain over time without blindly updating the golden file," while LLM testing is built on them. Breck targeted golden model outputs under training nondeterminism, not fixed inputs and criteria, but blind updating is exactly what LLM teams hit, and criteria drift guarantees updates.

The eval set is data, and data has bugs

Chen et al.'s first version reported GPT-4's prime identification falling from 97.6% to 2.4%. Narayanan and Kapoor noted the evaluation "only evaluated primality testing on prime numbers"; on composites, "the March version of GPT-4 almost always guesses that the number is prime, and the June version almost always guesses that it is composite," which is "consistent with the behavior of the models changing over time. None of it suggests a degradation in capability" (Narayanan & Kapoor, 2023). The revised paper's mixed set gives 84.0% to 51.1%; both sides agreed drift breaks applications. The lesson is data testing debt in the test set itself: a one-class eval set turned a calibration flip into an apparent 95-point collapse, and no pipeline would have flagged it.

Alert fatigue is arithmetic

False-positive alerts were "the most commonly discussed pain point" in Shankar et al.'s interviews (Shankar et al., 2022). Illustratively, 40 features each tuned to a 1% daily false-alarm rate give \(1 - 0.99^{40} \approx 33\%\) odds of a false page per day, about 94% over a seven-day shift. None of the interviewees "reported having solved the Goldilocks ML alert problem."

Some debts are outside the pipeline's reach

Undeclared consumers, correction cascades and feedback loops belong to the organisation and the world, not the repository. Monitor 1's remedy is human: "Make sure that your team is subscribed to and reads announcement lists for all dependencies." In 2026 that now includes model providers' release notes.

Practitioners do not all want the discipline

Some of Shankar et al.'s participants "felt that code review wasn't too useful, but they did it to adhere to software best practices," and chose retraining cadences for operational convenience. The rubric treats reviewed specs as foundational; some experienced engineers see ritual. Best practice and what experienced teams find worth doing are not the same list.

Alternative Designs

[IMAGE: A 2x2 grid, eval cost per run against power at small regressions: absolute threshold cheap and weak, blessing and paired judge eval (with a judge-variance error bar) in the middle, ease.ml/ci costly and strong, online A/B off-grid as high latency. Caption: "Most pipelines sit in the cheap, weak corner without having chosen to."]

Mature teams run several in sequence (Shankar et al. describe "dev, canary, staging, shadow, A/B" stages). The decision worth making explicitly is which stage may block. If the offline gate lacks power, the canary is the real gate.

How It Is Used in Practice

Google Play on TFX. A ranking model trained on "hundreds of billions of examples," warm-started from the previous model because "training from scratch can take several days to converge." Validation against production prevented "accidentally pushing partially-trained models to serving because of system failures": a pipeline bug, exactly the kind CD handles well.

Staged deployment and fallbacks. Common patterns among Shankar et al.'s 18 engineers: daily retraining so "model performance would not suffer for longer than a day," keeping "old versions as fallback models," one-to-two-week on-call rotations, and SLOs for pipelines (95% accuracy for an image classifier), which make "broken" a definition instead of an argument.

LLM applications. The emerging pattern reuses the pipeline with new contents: a manifest pinning prompt, model identifier, decoding parameters and judge; deterministic checks on every change and paired suites on behavioural ones; nightly runs to catch provider drift. SPADE's authors report it deployed in LangSmith for "over 2000 pipelines," a self-reported figure.

Insights Worth Remembering

1. Continuous delivery paid down the debts that look like code. Glue code and irreproducible builds had deterministic fixes. Slices, thresholds and offline-online correlation need decisions, which tooling automated around.

2. Scoring by the minimum is the rubric's best idea. One monitoring test moved the example up a band; five infrastructure tests would not.

3. An automated gate and a meaningful gate are different things. The rubric gives the same point to a gate with 19% false blocks as to one with 1%. Compute a gate's standard error before trusting it.

4. Every verdict an engineer sees spends the test set. The adaptive bound grows with \(H \ln 2\): cheap, not free, and almost no pipeline counts its holdout's uses.

5. For LLM applications the model is an unstable data dependency. Sculley's "versioned copy" is model pinning, and the staleness he warned it carries is the forced migration when a pinned version is retired.

6. Prompt histories are undeclared requirements. About two-thirds of SPADE's categorised edits were content changes, each a test written as configuration.

7. The evaluation set is the specification, and the specification drifts. A CI system that treats the golden set as fixed will be wrong in a new way each quarter.

Open Questions

Does adopting CD for ML reduce incidents, and by how much? The evidence is a rubric, one product's A/B result, interviews and a survey. No controlled study links maturity to incident rates. Widely believed; not measured.

Can statistical power be scored? A rubric crediting gates by measured false-block and miss rates would close the presence-versus-power gap. Whether teams would accept a score that sometimes falls when they automate something is open.

How should a test set's reuse budget be managed? ease.ml/ci shows a new-test-set alarm is straightforward in theory. How often teams can afford fresh labels, especially when labels arrive weeks late, has not been studied systematically.

Can LLM evaluation criteria be stabilised? Criteria drift was shown with nine participants. Whether it shrinks as products mature is unknown; if not, criteria may need versioning alongside prompts, with history re-graded.

Do judge scores predict user outcomes? The rubric's Model 2, offline-online correlation, scored 0 in the example and is rarely reported in public. For LLM judges the question is almost entirely unmeasured.

Sources and Further Reading

1. Sculley, D., Holt, G., Golovin, D., et al. (2015). "Hidden Technical Debt in Machine Learning Systems." NIPS 2015. Paper
2. Breck, E., Cai, S., Nielsen, E., Salib, M., & Sculley, D. (2017). "The ML Test Score: A Rubric for ML Production Readiness and Technical Debt Reduction." IEEE Big Data 2017. PDF
3. Baylor, D., Breck, E., Cheng, H.-T., et al. (2017). "TFX: A TensorFlow-Based Production-Scale Machine Learning Platform." KDD 2017. DOI:10.1145/3097983.3098021
4. Sato, D., Wider, A., & Windheuser, C. (2019). "Continuous Delivery for Machine Learning." martinfowler.com. Article
5. Google Cloud. "MLOps: Continuous delivery and automation pipelines in machine learning." Last reviewed 2024-08-28. Guide
6. Renggli, C., Karlaš, B., Ding, B., et al. (2019). "Continuous Integration of Machine Learning Models with ease.ml/ci." SysML 2019. arXiv:1903.00278
7. Amershi, S., Begel, A., Bird, C., et al. (2019). "Software Engineering for Machine Learning: A Case Study." ICSE-SEIP 2019. PDF
8. Shankar, S., Garcia, R., Hellerstein, J. M., & Parameswaran, A. G. (2022). "Operationalizing Machine Learning: An Interview Study." arXiv:2209.09125
9. Chen, L., Zaharia, M., & Zou, J. (2023). "How Is ChatGPT's Behavior Changing over Time?" arXiv:2307.09009
10. Narayanan, A., & Kapoor, S. (2023). "Is GPT-4 getting worse over time?" Essay
11. Shankar, S., Li, H., Asawa, P., et al. (2024). "SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines." arXiv:2401.03038
12. Shankar, S., Zamfirescu-Pereira, J. D., Hartmann, B., Parameswaran, A. G., & Arawjo, I. (2024). "Who Validates the Validators?" UIST 2024. arXiv:2404.12272
13. He, H., & Thinking Machines Lab. (2025). "Defeating Nondeterminism in LLM Inference." Post

Free to read, no ads, no sign-up. If it was useful you can buy me a coffee.