Test Method Validation (TMV)

How to Conduct TMV for Medical Device Industry

Mai Nguyen-Misra and George Omae

Medtronic

1 Test Method Validation (TMV) | Mar 2016 |

Presenter Bios
Mai Misra, Ph.D.
Sr. Principal Quality Engineer, Medtronic
Ph.D., Polymer Science;
M.S., Physics (Solid State); M.S., Chemical Engineering;
B.S., Applied Mathematics; B.S., Chemical Engineering
Dr. Mai Misra, a chemical engineer by training, has diverse
education & work experience. Mai received her PhD from
University of Akron in Polymers Science in 1995. She also
has a BS degree in Chemical Engineering and Applied
Mathematics as well as a MS in Chemical Engineering and a
MS in Solids State Physics. She has worked for over 20 years
in process and product development for various
applications serving diverse industries: US EPA
(Reformulated Gasoline and Alternative Fuels), H.B. Fuller
(Packaging, Window, Footwear etc), Boston Scientific
(Cardiovascular Drug-Eluting Stents), and Medtronic (CRDM
Pacing Leads). Mai is a co-inventor of 4 patents and has
authored many peer-reviewed papers. She has been in the
Medical Device industry for 9 years. Mai has been in the
current role as a SME on TMV for the validation group (part
of Neuromodulation Ops Quality) since 2014.
2 Test Method Validation (TMV) | Mar 2016 |
George Omae, M.S.
ASQ CSSB, CQE
Sr Engineering Manager, Medtronic
B.S. Mechanical Engineering, Nairobi University
M.S. Manufacturing Systems Engineering, Univ. of St. Thomas
M.S. Applied Statistics, Rochester Institute of Technology
MBA, Carlson School of Management, University of Minnesota
George Omae, a mechanical engineer by training, has over 19
years experience in the Medical device industry and has
worked in roles of increasing responsibility in Quality, Process
Development and R&D. He currently manages a test
development group. He has had significant experience
developing processes and products for 510k and PMA while
working at Guidant, Boston Scientific, American Medical
Systems and Medtronic. As a Six Sigma Black Belt, he has
trained and mentored engineers and other technical
professionals in the application of quality engineering tools. He
is also an experienced trainer for Test Method Validation,
Process Validation and Sampling Plans. George is also an
adjunct instructor at the St Cloud State University graduate
school (Medical Technology Quality (M.S.))
Course Objective & Approach

Objectives:
Provide an overview of what is required to plan and execute
TMV activities
Provide guidance on how to document TMV records to be in
compliance with prescribed validation requirements
Share best practices

Approach:
Presentation,
Case Study,
Audience Interaction

3 Test Method Validation (TMV) | Mar 2016 |

Course Flow
1 2

Section 2
Case study (Variables TMV)
1. TMV Planning
Section 1: TMV Overview 2. Executing & analyzing
3. Documenting TMV
Presenter: Mai Nguyen-Misra 4. Maintaining the validated state

Presenter: George Omae

3
Lessons Learned
Mai Nguyen-Misra

4 Test Method Validation (TMV) | Mar 2016 |

 Section 1

Test Method Validation (TMV)

Overview

5 Test Method Validation (TMV) | Mar 2016 |

TMV Purpose and Benefits

Purpose:
To demonstrate that a test or inspection method is suitable for
its intended purpose and is capable of producing valid results.

Benefits:
Assures consistence & compliance in the output
Ensures requirements are accurately tested throughout the
Product Lifecycle
Builds basis for evidence of requirements lifecycle traceability

6 Test Method Validation (TMV) | Mar 2016 |

Definitions

Test Method: A definitive procedure that produces a test

result. It includes any sample preparation, measurements,
observations and calculations.

Test Method Validation (TMV): The process by which a test

method has been demonstrated with objective evidence to be
suitable for its intended purpose and capable of producing
valid results.

7 Test Method Validation (TMV) | Mar 2016 |

 TMV Overview (inputs & output) Risk-based

8 Test Method Validation (TMV) | Mar 2016 |

TMV Types

Fluke A Fluke B

Analytical

QMSWI16303

Pharma

PHAP1191

9 Test Method Validation (TMV) | Mar 2016 |

 Group Discussion Sharing best practices
What types of TMV does your organization commonly
work with?

Does your organization create separate work instruction

for each type of TMV?

10 Test Method Validation (TMV) | Mar 2016 |

Factors that influence TM Variability
Influencing factors to consider for control
Cause Effect

Workpiece Instrument
Standard/Reference
(Part) (Gage)
Processing
Traceability variability Design

Stability Cleanliness
Maintenance
Stability
Measurement
System
Ergonomics (e.g. lighting) Skill Variability/Error
Vibration Procedure

Person
Environment
(Appraiser)

11 Test Method Validation (TMV) | Mar 2016 |

TMV Process Flow
The governing procedure defines the process for demonstrating that a test or
inspection method is suitable for its intended purpose and is capable of
producing valid results.

12 Test Method Validation (TMV) | Mar 2016 |

Process Activities: Lifecycle Approach
4 TM Phases:
I Good science/engineering
II Preliminary Test Method
III Characterized Test Method
IV Validated or Verified Test Method

TMs at Phase I,II,III require full

documentation of characterization elements

Phase IV TMs require validation to either No requirement to go

Level 1 or Level 2 based on the Risk through these sequentially
Identification or document evidence of
prior phase completion

13 Test Method Validation (TMV) | Mar 2016 |

TM Minimum Requirements

14 Test Method Validation (TMV) | Mar 2016 |

Process Inputs (pre-requisites) of TMV
Big 6!
Pre-requisites Tool/Source Comment
E.g. Lead body tensile 7 lbf,
1 Requirements Product/print spec DC resistance 107,
Length: 45cm 2cm

Per QMS1848 E.g. IQ NRP1086-35723 (protocol),

2 Equip qualified
E.g. MEDN-3285 (IQ) NRP1086-36375 (report),
Tooling/Fixtures OQ NRP1086-35724 (protocol),
released Inspection report NRP1086-36376 (report)

3
Software validated E.g. PCPWI8362
Critical/Major/High risk
Harm/Risk E.g. QMS1799, need TMV,
4
classification DFMEA/PFMEA/PCRW Provides validation level
& sample sizes
Documented test QMS16001 Example of content in
5
method MEDN-0002 QMS16001

E.g. MEDN-31261
6 Training documented Train to TM, store in Saba/other system
MEDN-0093

15 Test Method Validation (TMV) | Mar 2016 |

TMV Level (Risk-based)
Determine TMV level based on risk:

Level 1 TMV Moderate RL

Gathering & documenting evidence to demonstrate TM capability
Documentation : Report only

Level 2 TMV High RL

Level 1 + additional study data to demonstrate TM capability
Documentation: Protocol and Report
16 Test Method Validation (TMV) | Mar 2016 |
Maintaining a Validated State
Active TMs only

Control Plans are required for Active Test Methods

17 Test Method Validation (TMV) | Mar 2016 |

Controls and Monitoring
Document:
Type of controls, including frequency & criteria
Required general controls (e.g., training, equipment calibration &
maintenance)
TM-specific controls where applicable (e.g., periodic re-training,
quality control sample checks)

Type & Location of evidence of implementation

Document controls and actions to be taken if

controls fail in TMV report

Conduct Periodic Reviews

18 Test Method Validation (TMV) | Mar 2016 |

Maintaining a validated state (contd)

To Manage /Control Changes:

Assess impact of changes Consider impact of cumulative
effects of all changes since validation
Some examples of
Re-validate or provide justification changes
TM changes: TM
parameters,
Document decision and/or results Equipment/Tooling, SW
TM Intended Use
Requirement Changes:
Specifications, Risk level

19 Test Method Validation (TMV) | Mar 2016 |

 Group Discussion Sharing best practices

What is your process for Maintaining Validated

State?

20 Test Method Validation (TMV) | Mar 2016 |

 Section 2

Variables Test Method Validation

21 Test Method Validation (TMV) | Mar 2016 |

Section Agenda
KeyModule
Elements
2:Covered:
TMV - Variables

Evidence required to demonstrate completion of pre-

requisites

How to design the validation study (sample size, sample

part selection, appraiser selection, validation study
steps)

Case Study: Manual DC Resistance Test Method

22 Test Method Validation (TMV) | Mar 2016 |

Variables TMV Overview

Population

Inference

Sample

Full verification (100%

Acceptance
23
sampling L2a
Test Method Validation (TMV) | Mar 2016 |
inspection) L2b Process control L2c
Module 2: TMV Performance Characteristic Requirements
Key differences

24 Test Method Validation (TMV) | Mar 2016 |

 Variables Test Method
Case Study: Manual DC Resistance

25 Test Method Validation (TMV) | Mar 2016 |

Case Study - Introduction
Twist-Lock DC Resistance Test Method
A test engineer is getting ready to conduct 100% incoming
inspection and design verification testing. One of the
requirements (specification) is to test the manual DC
resistance of an improved cable used for initial evaluation for
patients that have overactive bladder. The resistance per
conductor must be between 0 - 10. The risk level is high for
this requirement.

Verify Evaluation Measure this resistance

System
26 Test Method Validation (TMV) | Mar 2016 |
Case Study - Introduction
TMV Questions:
1. What is the minimum validation level required?
2. What is the %Resolution?
3. What is the %Tolerance?
4. What is required to maintain the validated state of the TM?

Verify Evaluation Measure this resistance

System
27 Test Method Validation (TMV) | Mar 2016 |
Case Study - Select Validation Level

Start

Phase IV Level 2 TMV

(100% Inspection) High Risk
(Variables nondestructive)

28 Test Method Validation (TMV) | Mar 2016 |

Case Study - Conduct TMV
Start

29 Test Method Validation (TMV) | Mar 2016 |

Case Study - Plan the study

Key planning items Requirement

Pre-requisites: Ensure all are released and controlled
Number of appraisers: 2
Number of parts: 10 Sample size
Number of trials x3 per appraiser
Part selection: Select from typical/nominal processing conditions or
surrogates with rationale
Other
Label & randomize: Label & create randomization scheme using Minitab
considerations
Measurement location: Identify location of measurement

30 Test Method Validation (TMV) | Mar 2016 |

Case Study - Plan the study

MXXXX-XXX Report Template used

Question:
What validation
level is selected
for 100%
inspection?

31 Test Method Validation (TMV) | Mar 2016 |

Case Study - Plan the study

Brief Description
of the TM and
Requirements

32 Test Method Validation (TMV) | Mar 2016 |

Case Study - Plan the study

Sample size
&
justification

Equip info

33 Test Method Validation (TMV) | Mar 2016 |

Case Study - Execute protocol

Analyze &
Verify Pre- Execute Document
Plan Study Determine
requisites Protocol results
Acceptability

Identify:
Validation execution
roles

Document:
Step-by-step procedure for
executing
validation/verification.
Include role performing
each step.
34 Test Method Validation (TMV) | Mar 2016 |
Case Study - Execute protocol

Randomize
Data
Collection

Blind Data
Collection

35 Test Method Validation (TMV) | Mar 2016 |

Case Study - Analyze

Analyze &
Verify Pre- Execute Document
Plan Study Determine
requisites Protocol results
Acceptability

Show your calculations

Resolution

36 Test Method Validation (TMV) | Mar 2016 |

Case Study - Analyze

% Tolerance

%Tolerance =
4.27

37 Test Method Validation (TMV) | Mar 2016 |

Case Study - Document

Analyze &
Verify Pre- Execute Document
Plan Study Determine
requisites Protocol results
Acceptability

Do:
State how the data was
analyzed
Present results with
enough detail to show
how acceptance criteria
were met
State where evidence is
located (attaching in
Appendices preferred)

38 Test Method Validation (TMV) | Mar 2016 |

Case Study - Document

Cite references used

for the validation
Must have been
mentioned in report

List supporting data

attached to report
Must be mentioned in
the report

39 Test Method Validation (TMV) | Mar 2016 |

Case Study - Maintain the Validated State
Influencing factors to consider for control
Cause Effect

Workpiece Instrument Calibration & Preventative

Standard/Reference
(Part) (Gage) maintenance schedule
Elastic
Traceability deformation Design

Stability Cleanliness Maintenance

Measurement
System
Ergonomics (e.g. lighting) Skill Variability/Error
Vibration Procedure

Person e.g. Annual recertification

Environment
(Appraiser)

40 Test Method Validation (TMV) | Mar 2016 |

Case Study - Document
Use SWIPE model to
help document
controls

Describe general
controls

Describe TM
specific controls

State where change impact

assessment is captured when
changes occur
41 Test Method Validation (TMV) | Mar 2016 |
Raw Data Attachment

For Legibility:
Use the Snipping Tool or similar to capture
and attach evidence in your TMV record

42 Test Method Validation (TMV) | Mar 2016 |

Case Study - Document (raw data)

Legible raw data

Clear sample
identification
Randomization
demonstrated

Equipment used
identified

43 Test Method Validation (TMV) | Mar 2016 |

 Lessons Learned

44 Test Method Validation (TMV) | Mar 2016 |

Lessons Learned
Key Challenges:
Inconsistent record quality

Solutions:
Provide templates to ensure consistency and
completeness
Training and proficiency assessment

45 Test Method Validation (TMV) | Mar 2016 |

TMV Templates
Ensure requirements were met
Provide consistent format to tell the story

Help Ensure TMV Records Are:

Compliant ensure Author, Approver or someone else can present/defend
the work
Clear/Concise - make it easy for readers to review evidence and understand
your thinking; include rationale; legible
Complete verify all requirements addressed; attach (training, raw data) or
reference (pre-requisite reports) evidence, include data analysis records
Correct - templates can not replace the need for Authors and Approvers to
carefully read and check the work or to think critically

46 Test Method Validation (TMV) | Mar 2016 |

TMV Documentation Tips - Protocols
General Considerations
Good planning and a clear and complete protocol is key to successful validation

Use tables, flow charts, graphs, photos to create clear & concise records

Samples
Represent intended use (e.g., at nominal, bracket specification limit(s))

Explain creation of surrogate samples and why they are representative of

actual samples

Avoid Ambiguity
Include equations with units

Explain how data will be collected in compliance with GDP; consider creating
47
data collection forms
Test Method Validation (TMV) | Mar 2016 |
TMV Documentation Tips Execution
During Execution:
Follow the protocol
If deviation occurs: STOP, document deviation, investigate, document
root cause & action plan, obtain approval, then PROCEED
When collect data:
Individual: Signature, name/Employee ID
Attributable
Test Method: TM Title/Number
Equipment: Equip ID
Clear, readable, concise
Legibility
Changes must not hide/obscure original record

Contemporaneous Recorded & signed on date performed

Original Original record or verified copy.
Recorded as observed. No reprocessing. Meaning
Accurate
maintained through life of data
48 Test Method Validation (TMV) | Mar 2016 |
 Group Discussion Sharing Best Practices
Raw Data Integrity

Electronic Data and Audit Trails What do you do?

49 Test Method Validation (TMV) | Mar 2016 |

Key Points for Conducting Successful TMV

Say what you do

Written procedures

Do what you say

Follow procedures

Record what you did

Keep records

Check the results

Did it pass or fail

Act on the difference

Procedures for deviations/departures

50 Test Method Validation (TMV) | Mar 2016 |

 Wrap-up and Final Questions

51 Test Method Validation (TMV) | Mar 2016 |

 Group Discussion Sharing Best Practices
Closing Thoughts

What are the key challenges (in validation areas) has your
organization encountered?

and what have you put in place to effectively deal with those
challenges?

TMV Training: How do you ensure your TMV authors/approvers

are properly trained on the TMV process?

Finally - Any other best practices to share?

52 Test Method Validation (TMV) | Mar 2016 |

 TMV TOOLS AND RESOURCES

53 Test Method Validation (TMV) | Mar 2016 |

 References

Measurement System Analysis (1995), 4th Ed,

Automotive Industry Group, Southfield, MN
o ANOVA Method for Level 2b & 2c (variables)
o Range Method for Level 2a (variables)
ASTM E2782-11e1 Standard Guide for
Measurement System Analysis (MSA)

54 Test Method Validation (TMV) | Mar 2016 |