FLARE Series (3/6): Vendor Assessment aka Trust Is Good, Validation Is Better

In our last article, we explored the first FLARE step: evaluating the intended use and determining the overall risk of an AI-based feature. If the outcome of that analysis points to a low-risk feature, say, a tool that highlights keywords or flags duplicate literature entries, then the next logical question is:

Who built this tool, and can we trust them?

That’s where Step 2 of the FLARE framework comes in! Vendor Assessment: your structured way to evaluate whether a provider is qualified, capable, and transparent enough to be part of your regulated environment.

In AI We Trust? Not Without This Step.

The 21st century may go down as the century of AI. Everyone is building “smart tools.” But not everyone is documenting how they work, or validating them to the standards we expect in the MedTech industry.

Most supplier qualification processes in medical device companies are built around generic vendor hygiene:

• Legal entity?

• VAT number?

• Quality certificates?

But that’s far from enough: When you’re working with AI that supports regulatory submissions, you need deeper insight. You need evidence of validation, risk awareness, and proof that the vendor actually understands the regulatory expectations in MedTech.

That’s why FLARE introduces the Vendor Assessment, which generates what we call a Confidence Index. It's a practical way to separate serious providers from the ones just jumping on the AI bandwagon.

Key Takeaways: A Structured Approach to Building Confidence

To support this step, we’ve developed a detailed questionnaire, a checklist covering 12 key factors across two phases:

• Initial Validation

• Revalidation

Each factor is scored from 0 to 5, giving a total possible score of 60 points for self-developed AI tools.

Want to get the full supplier evaluation checklist and the Confidence Index template? Reach out to us — we’re happy to share it.

Here’s a snapshot of what’s inside:

Initial Validation: Is This Vendor Qualified?

1. Vendor Selection & Qualification:  Is the vendor experienced in regulated environments and compliant with standards like ISO 13485?

2. Supplier Quality Agreements: Are roles, responsibilities, deliverables, and update protocols clearly defined?

3. Validation Documentation:  Can the vendor provide cross-validation results, performance metrics, or dataset justifications?

4. Regulatory & Standards Compliance:  Do they follow standards like IEC 62304 or Good Machine Learning Practice (GMLP) and can they prove it?

5. Risk-Based Approach:  Is there a documented risk management process aligned with regulatory expectations?

6. Testing & Cross-Validation:  Do they test against real-world use cases? What validation strategies are in place?

7. Explainability & Transparency:  Can they explain how their AI works? Or is it a black box?

Revalidation: How Well Is the AI Maintained?

1. Control of Updates & Patches:  Are software updates traceable and assessed for impact?

2. Post-Market Data Sharing : Does the vendor collaborate on real-world data collection and ongoing performance monitoring?

3. Defined Revalidation Triggers:  Are there clear triggers (like version changes) that prompt revalidation?

4. Integration with Customer QMS:  Can the tool be validated in your environment before deployment?

5. Audit & Oversight Capabilities:  Can you (or your auditor) review the vendor’s documentation and processes?

So Far, So Good. But What Should You Do Next?

• Start with reviewing your current vendor qualification templates:  Do they include AI-specific aspects like transparency, testing, and post-market validation?

• Use the FLARE Vendor Assessment checklist:  Apply it to all current or upcoming AI-based tools in your pipeline.

• Integrate Vendor Assessment into your procurement process:  Even for support tools, trust should be earned and documented.

• Don’t assume. Ask!  If the vendor can’t explain how their AI works, or how it’s tested, that’s your signal to dig deeper.

Final Thought: Turn Trust Into Traceability!

AI tools are only as trustworthy as the people and processes behind them. With our FLARE Vendor Assessment, you’re not just trusting a promise, you’re auditing a process.

Because in MedTech, trust isn’t just a feeling. It’s a documented decision!

Want the full checklist and Confidence Index template?

Reach out to us! We’re happy to share it.

Stay tuned for the next article in this series: Risk Analysis per Stakeholder Requirement, coming soon!