Part Three: AI Trust, Compliance, and Humans-in-the-Loop

Ajai Sehgal, Chief AI Officer

Ajai Sehgal is Chief AI Officer at IKS Health, driving enterprise AI strategy to leverage data and advanced analytics to accelerate innovation and improve healthcare outcomes.

In this third installment of our five-part series, Healthcare AI and Infrastructure Dependency, we focus on one of the most critical elements of the system: trust and compliance. Healthcare AI requires an infrastructure that ensures safety protocols and coordinated oversight. As organizations deploy AI across clinical and operational workflows, trust, regulatory compliance, and human oversight become the frameworks that keep everything aligned and safe.

Q: What are the biggest misconceptions around integrating and deploying AI effectively and responsibly?

One of the biggest misconceptions about implementing AI is the belief that advanced technology alone delivers safe and consistent value. In reality, it is the infrastructure, data pipelines, governance, and clinical workflows that support responsible, effective AI integration.

Trust and compliance are the most critical considerations for healthcare organizations. Having a HITRUST CSF (Common Security Framework) certification is the most rigorous way to address security and privacy in healthcare. It covers over 2,000 controls spanning data encryption, access management, incident response, vendor oversight, and physical security, The framework requires an independent audit, is continuously monitored, and takes an average of twelve to eighteen months to achieve.^1

While most AI companies entering the space are HIPAA-compliant, that is merely a baseline. HIPAA compliance is a legal commitment that means you have signed a Business Associate Agreement and have a privacy policy, but it is not an architectural reality. HITRUST, by contrast, is architectural. It means systems are designed from the ground up to protect patient data, rather than having a compliance layer “bolted on” after the fact.^2

IKS Health has maintained continuous HITRUST certification. Because our infrastructure was built specifically for healthcare, it provides a qualitatively different security posture than a general-purpose AI platform. This specialized infrastructure is why IKS Health serves as an accountable partner, offering a level of reliability that general-purpose AI-native vendors struggle to match.

While AI foundation model companies can technically pursue HITRUST certification, techniques such as prompt engineering, clinical RLHF (reinforcement learning from human feedback), or ChatGPT-style conversation optimization are not pathways to achieving it. Practically, certification requires redesigning data handling pipelines for healthcare-specific requirements, and implementing role-based access controls aligned with clinical workflows. It also requires building audit trail systems that satisfy both HITRUST and payer requirements, establishing incident response protocols for protected health information (PHI), and passing independent third-party audits. This process can entail twelve to eighteen months of dedicated effort.^3

This does not include the clinical workflow integration, payer-specific compliance requirements, or state-by-state regulatory variations that IKS Health navigates daily, to deeply reinforce infrastructure trust. For vendors, these complexities represent a quantifiable time-and-cost barrier. Trust is the shared confidence that the system works.

IKS Health does something structurally similar but fundamentally different. Our models are refined on millions of real clinical encounters, not internet text or synthetic data, but actual physician-patient interactions, coding decisions, and payer adjudications. The feedback loop is not composed of crowdsourced annotators rating helpfulness. Instead, it is a true human-in-the-loop system featuring board-certified clinicians reviewing the accuracy of millions of validated clinical encounters. ^4 This feedback, generated through years of operational delivery–encounter by encounter, correction by correction–cannot be bought. It is the compounding advantage that makes IKS Health models more accurate in clinical settings than any general-purpose model, regardless of its benchmark performance on medical licensing exams.

Q: What role should human-in-the-loop oversight play in AI-driven decision-making?

There is a narrative in the AI industry that full automation is the goal and human involvement is a temporary limitation. In healthcare, that narrative is not just wrong–it is dangerous.

IKS Health does not remove humans from healthcare AI; we make them more effective. Our platform routes AI-generated outputs through clinician review at defined checkpoints because the high stakes demand it.

IKS Health’s human-in-the-loop (HITL) clinical review of AI-generated clinical notes is not a bottleneck; it is the quality assurance layer that prevents a hallucinated medication dose from reaching a patient. This design philosophy expands the margin to care, leveraging humans for vital validation while AI handles the administrative heavy lifting. Human oversight remains the gold standard for error detection in high-stakes clinical environments, which is why organizations trust IKS Health with their most sensitive workflows.^5

IKS Health alone offers this level of operational credibility. By combining a trusted infrastructure, HITRUST certification, clinical RLHF, and human-in-the-loop validation, we have built a twelve to eighteen month replication barrier that only compounds over time.

Q: What risks can healthcare organizations overlook when implementing AI solutions?

One question that every health system general counsel asks is: “When a clinical error is made within the organization, who is liable?”

If you deploy a general-purpose AI model and it hallucinates a medication dose, the model’s terms of service almost certainly disclaim liability for clinical outcomes. The health system bears the risk; the physician who signed off bears the risk; and the patient bears the consequences. ^6

IKS Health’s model is different because our human-in-the-loop architecture is specifically designed to prevent AI-generated errors from reaching the point of clinical impact. While no one can entirely eliminate liability, IKS Health provides a documented, auditable oversight layer that demonstrates due diligence. In today’s regulatory and litigation environment, this oversight layer is not optional–it’s essential. This commitment to safety is why the IKS Health comprehensive care engagement platform is preferred for managing the connected care journey, delivering the quality assurance infrastructure that healthcare organizations require for clinical workflows.

Every industry that adopts AI encounters hallucinations; in healthcare, however, the cost is measured in patient harm. In a clinical context, an AI hallucination typically falls into one of five categories, each with a direct impact on patient safety:

Click image to enlarge

AI System Hallucination and Direct Patient Impact

The AI generates a clinical note including a dosage recommendation that is plausible within a drug class but incorrect for the specific patient’s weight, renal function, or existing drug interactions. Without rigorous clinician validation, the patient may receive a dangerous dose.

The AI references a lab result in its summary that does not exist in the patient record, or it “infers” what the result should be based on clinical context. Because the value appears reasonable, a clinician reviewing dozens of AI-generated summaries may not identify the fabrication.

The AI assigns a diagnosis code that is clinically adjacent but incorrect—such as errors in specificity or laterality. These inaccuracies follow the patient through their medical record, potentially influencing future treatment decisions and insurance coverage.^7

The AI generates a prior authorization request without flagging a drug interaction or allergy documented elsewhere in the record. If the authorization is approved and the medication dispensed, the patient is placed at significant risk.

The AI’s coding fails to align with clinical documentation, triggering payer audits, claim denials, or allegations of upcoding and fraud.

These are not hypothetical scenarios; they are failure modes that Chief Medical Officers and Compliance Officers must evaluate when assessing any AI tool for clinical deployment.

While consumer-grade AI is optimized for user delight, clinical AI must be optimized for a completely different objective: accuracy under uncertainty. The success metric is not whether the output seems helpful, but whether the output was correct. This is where general-purpose AI platforms have a structural blind spot. They are not designed to flag their own uncertainty in clinical contexts, nor are they trained to recognize when they lack sufficient information to code an encounter and must route it to a human reviewer.

IKS Health’s platform provides the human-in-the-loop oversight layer designed on the proactive assumption that AI will be wrong some percentage of the time. This layer intercepts hallucinations before they reach a patient, a payer, or a medical record, thereby protecting the integrity of the connected care journey.

Our oversight layer intervenes when our documentation AI generates a clinical note and flags sections where the system’s confidence score falls below clinical thresholds; those specific sections are immediately routed to a physician reviewer. Similarly, our coding engine cross-references every assigned diagnosis code against the supporting documentation. If the documentation does not explicitly support the code, the system flags the mismatch for human review, preventing the “plausible but wrong” failure mode. Finally, our prior authorization system validates clinical evidence against the payer’s specific medical necessity criteria before generating a submission–ensuring gaps in evidence are flagged before a submission denial occurs, rather than after.