Healthcare AI is trained and benchmarked on claims and records assembled for billing, then judged against more of the same. Circles Datasets are different: whole episodes of care recorded prospectively, across sites, with outcomes months and years later, each label traceable to the clinician or patient who entered it.
A model is only as good as the evidence it is tested on. Of 1,357 cleared AI devices reviewed in 2026, 34 were linked to prospective trials and 3 were evaluated on patient outcomes. A Circles Dataset gives a developer, or the health system buying the model, outcome-based evidence that it works, and keeps working.
When a model’s output is challenged, the question is what it was tested against. A test set assembled from brokered data cannot show who recorded each label or when. In a Circles Dataset every label carries its author and time, so performance claims can be defended record by record. Among 950 AI devices studied, 6.3% were recalled, 43% of those within a year of clearance.
FDA guidance asks that test data be independent of training data, sequestered from development and drawn from different sites, with a plan to monitor performance after deployment. The EU AI Act requires training, validation and testing data that are relevant, sufficiently representative, and, as far as possible, free of errors and complete. Circles Datasets are built to those requirements.
Each runs on the same four pillars: the Observational Protocol, the Circle, the closed system and the technical stack. The detail sits in the linked white papers.
Prospective, multi-site Cases with outcomes the model has never seen, cut to the population and decision the model serves.
Compare predictions with outcomes as they are recorded.
FDA asked in 2025 how AI performance should be measured in the real world, noting reliance on retrospective testing and static benchmarks. Benchmarks are fixed; clinical practice, populations and models change. A Circle keeps recording, so a deployed model can be re-measured against new outcomes each quarter, and drift is visible before it reaches patients. Verification of results without moving records is in pilot.
Reserve Cases from any training license, so an evaluation set stays independent of every model it tests, as FDA guidance requires.
Where training data is licensed, every record shows its source and the patient’s consent terms, unlike data scraped or brokered.
Circle Members who design and record the protocol can review model outputs and adjudicate disputed cases, with their judgments recorded and attributed.
Sources: FDA (2026); Abulibdeh et al., PLOS Digital Health (2026); Lee et al., JAMA Health Forum (2025); Regulation (EU) 2024/1689, as amended by Regulation (EU) 2026/1744.
Those records were created to document visits and pay claims. They lack a baseline, rarely record outcomes after discharge, and code suspicions as diagnoses. Many models were trained on the same kinds of records, so testing against them measures agreement with familiar data, not accuracy for patients.
Yes, but never the same Cases for both. Cases licensed for training are excluded from evaluation sets, so an evaluation stays independent of the model it tests.
Licensed datasets are de-identified under HIPAA. Verification of model results where the data resides, so that no record moves, is in pilot.
RegenMed uses AI to research clinical questions when drafting protocols; clinical experts finalize every protocol, and no model enters data into a Case. RegenMed supplies the ground truth that models are tested against.