Deployment-Specific Benefit and Harm in Clinical Artificial Intelligence: Decision-Analytic Derivation and Multi-Institutional Evaluation of a Deployment Threshold (Preprint)

<sec> <title>BACKGROUND</title> <p>Clinical artificial intelligence operates in a health care environment heterogeneous in biology, clinical practice, and documentation, so its benefit-harm balance is deployment-specific: an intervention can help in one setting and harm in another when the determinant of response is not routinely observed.</p> </sec> <sec> <title>OBJECTIVE</title> <p>To derive the deployment threshold at which a sign-changing AI intervention becomes beneficial, using a retraining-free correction to clinical retrieval embeddings as a worked case, and to determine whether the evidence needed to apply that threshold exists in the published literature or must be measured locally.</p> </sec> <sec> <title>METHODS</title> <p>We specified a three-arm decision-analytic model - no action, universal deployment, and screen-and-treat - for interventions with sign-changing effects and instantiated it on a retraining-free correction to clinical retrieval embeddings. Empirical effect groups were defined by measured response rather than nominal mechanistic tier. Screening accuracy was derived from between-condition variability and tested by local resampling. Transportability was evaluated on emergency-department notes and discharge summaries from two US academic centers. A structural sensitivity analysis allowed the downstream value of benefit and harm to differ.</p> </sec> <sec> <title>RESULTS</title> <p>Measured response defined six benefited and seven harmed configurations at the base-case regularisation. The universal-deployment threshold was p* = 0.50; the exact and zero-cost forms differ by 2.8 × 10⁻⁵, so strategy cost does not move it, and across six regularisation settings the threshold ranged from 0.32 to 0.70. Published descriptions did not identify the response-defined state: none of 55 published systems had been evaluated under the intervention, so response prevalence remained unidentified without classification assumptions. Local screening reduced the minimum required prior probability from 0.210 with one scored condition to 0.071 with six. The nominal mechanistic tier agreed with response in all 52 institutional section-level cells, but one benchmark-harmed configuration benefited in all four institutional conditions. In the joint fit-and-evaluation resampling protocol, specificity rose from 0.328 at 10 documents to 0.974 at 75 documents.</p> </sec> <sec> <title>CONCLUSIONS</title> <p>The deployment threshold is analytically identifiable, but the state needed to apply it cannot be recovered reliably from model identity or the published literature. Deployment validity is therefore local in the setting examined: local measurement is needed when the response-defining state cannot otherwise be established reliably before deployment, and an unsuitable measured state requires local action rather than plug-and-play transfer. The data do not identify any single adaptation method as universally necessary.</p> </sec>