NordVal

c/o National Veterinary Institute P.O.Box 8156, Dep. N !! Oslo Nor"ay

Protocol #or t$e %alidation o# alternati%e &icro'iolo(ical &et$ods

)ontents

Scope Definitions Phases of validation Test characteristics Qualitative methods Method comparison study Collaboratory study Quantitative methods Method comparison study Collaboratory study #eferences

Page 2 2 5 5 5 5 1 1! 1! 1"

12$ !$2

*cope
This &ord'al protocol describes the technical procedures for validation of alternative methods for microbiological analyses of food( )ater( animal faeces( feed( and food environmental samples in the &ordic countries$

De#initions
+eneral ter&s, *lternative method *n alternative method is a method of analysis that demonstrates or estimates( for a given category of products( the same analyte as is measured using the corresponding reference method$ *nalyte The analyte is the component demonstrated or measured by the method of analysis$ +t may be the microorganism( its components or products$ Collaboratory study Study of the performance of the alternative method using common samples in several laboratories and controlled by the e,pert laboratory$ -,pert laboratory .aboratory having /ualified staff and s0ills to perform the method comparison study and organise the collaborative study$ The availability of an e,perienced statistician is essential for the analysis of the results$ Method comparison study Study performed by the e,pert laboratory of the alternative method against the reference method$ &egative deviation The alternative method presents a negative deviation if it gives a negative result )hen the reference method gives a positive result$ Positive deviation The alternative method presents a positive deviation if it gives a positive result )hen the reference method gives a negative result$ Qualitative method * /ualitative method is a method of analysis )hose response is either the presence or absence of the analyte in a certain amount of sample$

Quantitative method * /uantitative method is a method of analysis )hose response is the amount of the analyte measured either directly or indirectly in a certain amount of sample$ #elative accuracy The relative accuracy is the degree of correspondence bet)een the response obtained by the alternative method and the reference method on artificially inoculated samples$ #elative detection level Documentation must be provided that the relative detection level of the alternative method is e/ual to that of the reference method$ #elative selectivity The relative selectivity is a measure of a1 the inclusivity2 detection of the target microorganism from a )ide range of strains( and b1 the e,clusivity2 the lac0 of interference from a relevant range of non3target microorganisms$ #elative sensitivity The relative sensitivity is the ability of the alternative method to detect the analyte compared to the reference method$ #elative specificity The relative specificity is the ability of the alternative method not to detect the target microorganism )hen it is not detected by the reference method$ #eference method * reference method is a method )hich is internationally recognised and accepted 4e$g$ &M5.( +S6( C-& and *6*C methods( and certain national standards of e/uivalent standing1$ 'alidation of an alternative method The validation of an alternative method is the procedure to demonstrate if the results obtained by the alternative method are comparable to those obtained using the reference methods$

*tatistical ter&s, 6utlier *n outlier is an e,treme value )hich normally appears randomly in less than 17 of tests( but more fre/uently( if abnormal situations occur$ Statistical test procedures can be used to /uantify this probability$ #elative standard deviation4#SD1 The relative standard deviation is a measure of the precision of the method by demonstrating the #SD for repeatability and reproducibility$ #epeatability The repeatability is the closeness of agreement bet)een successive and independent results obtained by the same method on identical test material under the same conditions 4 apparatus( operator( laboratory and short intervals of time 1$ #epeatability limit 4r1 The repeatability limit is the value less than or e/ual to )hich the absolute difference bet)een t)o tests results obtained under repeatability conditions is e,pected to be )ith a probability of 857$ &ote2 +f the difference bet)een 2 results e,ceeds r( the results should be considered as suspect$ #eproducibility The reproducibility is the closeness of agreement bet)een single test results on identical test material using the same method and obtained by operators in different laboratories using different e/uipment$ #eproducibility limit 4#1 The reproducibility limit is the value less than or e/ual to )hich the absolute difference bet)een t)o test results obtained under reproducibility conditions is e,pected to be )ith a probability of 857$ &ote2 +f the difference bet)een 2 results from different laboratories e,ceeds #( the results must be considered suspect$

P$ases o# %alidation
The validation of /ualitative and /uantitative methods comprises t)o phases2 A: A method comparison study of the alternative method against a reference method (performed by an expert laboratory). B: A collaboratory study of the alternative method (organised by an expert laboratory).

-est c$aracteristics
The test characteristics for alternative methods are sho)n in the follo)ing table2

Qualitative methods
1. Relative selectivity 2. Relative accuracy 3. Relative detection level 4. Relative sensitivity #. Relative specificity

Quantitative methods
1. Relative selectivity . Relative accuracy !. Repeatability ". Reproducibility #. Relative standard deviation

Part 1 .ualitati%e &et$ods

/. 01-2OD )O0P/3I*ON *-4D5 1 3 #elative selectivity The relative selectivity is a measure of a1 the inclusivity2 detection of the target microorganism from a )ide range of strains( and b1 the e,clusivity2 the lac0 of interference from a relevant range of non3target microorganisms$ a1 inclusivity Select 5 pure cultures of target microorganisms relevant to the alternative method and the food matrices$

9rom each test strain a gro)th medium is inoculated )ith a level of 1 to 1 times greater than the detection level$ 9or both the alternative method and the reference method the complete protocol is used$ #esults 0icroor(anis& Strain 1 Strain 2 Strain ! etc$ T6T*. b1 e,clusivity Select ! pure cultures of non3target microorganisms chosen from both the strains 0no)n to cause interference )ith the target microorganism and from strains naturally present in the food matrices$ 9rom each test strain a gro)th medium is inoculated )ith a level similar to the greatest level of contamination e,pected to occur in the food matrices$ 9or both the alternative method and the reference method the complete protocol is used$ #esults 0icroor(anis& Strain 1 Strain 2 Strain ! etc$ T6T*. /lternati%e &et$od 3e#erence &et$od /lternati%e 6/7 &et$od 3e#erence 6/7 &et$od

&ote2 Criteria for the selection of test strains for measurement of selectivity see *nne, : in +S6 1;1" 2 2 ! 3 Protocol for the validation of alternative methods$ 2 3 #elative accuracy The relative accuracy is the degree of correspondence bet)een the response obtained by the alternative method and the reference method on identical samples$

*s far as possible naturally contaminated samples should be used$ <o)ever( in the &ordic countries such samples are seldom available in sufficient numbers for e,tensive validations$ Therefore( artificial contamination of samples is permitted$ *s a general rule three inoculation levels are used2 = negative control( 131 cells per 25 g sample and 1 31 cells per 25 g sample usually 2 strains relevant for the matri, are selected$ +f validation is re/uested for all food matrices( at least 5 relevant food matrices are selected from the &ord'al matri,3lists$ Samples from a food matri, should be representative by ta0ing samples from several of the sub3matrices indicated in &'3D6C2 9$ Matrices for feed should include meat3 and bonemeal and fishmeal$ *s regards Salmonella 2 relevant strains for these matrices should be used$ Matrices for faeces should include faeces from poultry( s)ine and cattle unless a specific matri, is chosen$ 9or other categories of matrices a relevant number of matrices are selected$ 9or each matri, at least ; samples are analysed to give appro,imately ! positive and ! negative results by the reference method$ -ach sample is analysed once by the reference method and once by the alternative method$ +dentical samples should be used as far as possible$ <o)ever( )hen different pre3enrichments are used( duplication of samples is necessary$ &ote2 See guidance in *nne, D in -& +S6 1;1" 22 ! 3 Protocol for the validation of alternative methods$

Tabulate the data from the accuracy measurement after screening$ 3esponses /lternati%e positi%e 8 /lternati%e ne(ati%e 8 &et$od /69 &et$od /79 3e#erence &et$od positi%e 86/ 9 >?> positive agreement 4P*1 >?3 negative deviation 4&D1 3e#erence &et$od ne(ati%e 87/ 9 3?> positive deviation 4PD1 3?3 negative agreement 4&*1

Relative accuracy % =

( PA + NA ) 100
PA + NA + PD + ND

Tabulate the data from the accuracy measurement after confirmation$

3esponses /lternati%e &et$od positi%e 8 /69 /lternati%e &et$od ne(ati%e 8 /79

3e#erence &et$od positi%e 86/ 9 >?> positive agreement 4P*1 >?3 false negatives 49&11

3e#erence &et$od ne(ati%e 87/ 9 3?> true positives 4TP1 3?3 negative agreement 4&*1 > false positives 49P1

Relative accuracy % =

( PA + NA + FP ) 100
PA + NA + TP + FN + FP

Positive deviation 4PD1 The alternative method presents a positive deviation if it gives a positive result )hen the reference method gives a negative result$ * positive deviation must be confirmed$ * positive deviation 4PD1 becomes a false positive 49P1 result )hen the true result can be proven as being negative$ * positive deviation is considered as a true positive 4TP1 result )hen the true result can be proven as being positive$ &egative deviation 4&D1 The alternative method presents a negative deviation if it gives a negative result )hen the reference method gives a positive result$ * negative deviation becomes as a false negative 49&1 )hen the true result can be proven as being positive$ +t is generally assumed that the reference method gives the true result$ <o)ever( e,perience has sho)n that poor accuracy can be due to poor performance of either the alternative method or the reference method$ ! 3#elative detection level Documentation must be provided that the relative detection level of the alternative method is e/ual to that of the reference method$ " 3 #elative sensitivity The relative sensitivity is the ability of the alternative method to detect the analyte compared to the reference method$

"

The calculation is made by using the data obtained in the accuracy measurement after confirmation$

Relative sensitivity % =

( PA + TP ) 100
PA + FN

+nterpretation2 :enerally( values above 857 are considered acceptable$ 5 3 #elative specificity The relative specificity is the ability of the alternative method not to detect the target microorganism )hen it is not detected by the reference method$
NA 100 NA + FP

Relative s eci!icity % =

* summary of all calculation tests for relative accuracy( relative sensitivity and relative specificity should be tabulated as follo)s2 *fter screening2
$atrices %A &A &' %' (um Relative Accuracy A) (*) (%A,&A) x 1-%A,&A,%',&' %A , &' Relative sensitivity (+ (*) %A x 1-%A,&' &A , %' Relative specificity (% (*) &A x 1-%',&A

& .ood cat. 1 .ood cat. .ood cat. ! .ood cat. " .ood cat. # /0/A1

& = P* >&* > PD > &D

10

*fter confirmation2
$atrices %A &A .& /% .% (um Relative Accuracy A) (*) (%A,&A,.%)x 1-& Relative sensitivity (+ (*) (%A,/%)x 1-%A,.& Relative specificity (% (*) &A x1-&A,.%

& .ood cat. 1 .ood cat. .ood cat. ! .ood cat. " .ood cat. # /0/A1

& = P* >&* > 9& > TP > 9P

B. )O::/BO3/-O35 *-4D5 The aim of the collaboratory study is to determine the variability of the results obtained by the alternative method in different laboratories using identical samples$ The e,pert laboratory shall include the alternative method and the reference method in its study$ #esults from at least @ laboratories )ith valid results must be available for the calculations$ 4Therefore( it is advisable to select 1 312 labs1$ +n case of very e,pensive instruments results from 5 laboratories are considered acceptable$ 6ne relevant food matri, is selected from the &ord'al matri, list to prepare the test samples$ *rtificial inoculation is used at ! levels 46 = negative control( 131 cells per 25 g and 1 31 cells per 25 g1$ Duplicates are prepared for each laboratory to be analysed by the alternative method$ +f the alternative method comprises more than one protocol a relevant food matri, for each protocol should be selected for the study$ The e,pert laboratory shall determine )hich data from the participating laboratories are suitable for the calculation of the precision data$ &ote2 :uidelines for the e,pert laboratory conducting the collaboratory study see2 *nne, < and . in -& +S6 1;1" 22 ! A Protocol for the validation of alternative methods$

11

)alculation. Tabulate the data obtained )ith each method as follo)s2 Positi%e results 'y t$e expert la'oratory .aboratories . #eference method *lternative method . = negative control .1 = 131 cells per 25 g .2 = 1 31 cells per 25 g ?2 ?2 Contamination level .1 ?2 ?2 .2 ?2 ?2

Positi%e results 'y t$e alternati%e &et$od #ro& t$e colla'orati%e study .aboratories . .aboratory 1 .aboratory 2 .aboratory ! -tc$ etc Total ?2 ?2 ?2 ?2 9Pa Contamination level .1 ?2 ?2 ?2 ?2 TPb .2 ?2 ?2 ?2 ?2 TPc

a 9alse positive by the alternative method b True positive at level 1 by alternative method c True positive at level 2 by the alternative method 9or level . and each method( calculate the percentage specificity SP

411

$P %

1#

FP N#

& 100%

12

Bhere2 NFP is the total number of all . testsC is the number of false positive$

9or each positive contamination level .1 > .2 and for each method( calculate the percentage of sensitivity S-

421
Bhere2 N+ TP

$' %

TP N(

& 100%

is the total number of all .1 or .2 tests respectivelyC is the number of true positive$

9or each level of contamination and the totality of the results( compare the alternative method and the reference method in order to calculate the relative accuracy in the study carried out by the e,pert laboratory$ -ach pair of results from a sample measured by the alternative and the reference method shall be reported as follo)s2 *lternative method > > 3 Total P* 9& &> #eference method 3 TP &*> 9P &3 & Total

Calculate the relative accuracy *C e,pressed in percentage A) % Bhere2 N PA *PA ( NA ( FP+ N & 100%

is the number of tested samples 4for the level .1 or all levels1C is the number of positive agreementC

NA + FPis the number of negative agreement$ Interpretation. Compare *C( S- and SP )ith their relative counterparts obtained )ithin the comparative study$

13

Part ; .uantitati%e &et$ods /. 01-2OD )O0P/3I*ON *-4D5

1 3 #elative selectivity The relative selectivity is a measure of a1 the inclusivity2 detection of the target microorganism from a )ide range of strains( and b1 the e,clusivity2 the lac0 of interference from a relevant range of non3target microorganisms$ a1 inclusivity Select at least ! pure cultures of target microorganisms relevant to the alternative method and the food matrices$ These strains shall be representative of the most common strains present in the &ordic countries$ 9rom each test strain an appropriate gro)th medium is cultured overnight before the alternative medium and the reference medium are inoculated$ 9or both the alternative medium and the reference medium the complete protocol is used #esults 0icroor(anis& Strain 1 Strain 2 Strain ! etc$ T6T*. /lternati%e 6/7 &et$od 3e#erence 6/7 &et$od

b1 e,clusivity Select at least 2 pure cultures of non3target microorganisms chosen from both the strains 0no)n to cause interference )ith the target microorganism and from strains naturally present in the food matrices$ These strains shall be representative of the most common strains in the &ordic countries$ 9rom each test strain an appropriate gro)th medium is cultured overnight before the alternative medium and the reference medium are inoculated$ 9or

14

both the alternative methods and the reference method the complete protocol is used$ #esults 0icroor(anis& Strain 1 Strain 2 Strain ! etc$ T6T*. /lternati%e &et$od 3e#erence &et$od

&ote2 Criteria for the selection of test strains for measurement of selectivity see *nne, : in -& +S6 1;1" 2 2 ! 3 Protocol for the validation of alternative methods$

2$ #elative accuracy The relative accuracy is the degree of correspondence bet)een the response obtained by the alternative method and the reference method on artificially inoculated identical samples or on naturally contaminated samples$ Documentation must be provided that the relative accuracy of the alternative method is e/uivalent to that of the reference method$ &ord'al )ill evaluate information provided by the manufacturer based on results from e,pert laboratories obtained by using international accepted validation protocols for /uantitative methods 4*9&6#( *6*C and +S61$

B. )O::/BO3/-IV1 *-4D5
* collaborative study establishes the follo)ing values2 1$ Standard deviation of repeatability 4sr1 2$ Standard deviation of reproducibility 4S#1 !$ #elative standard deviation of both 4#SD r( #SD #1 The aim of the collaborative study is to determine the variability of the results obtained by the alternative method in different laboratories using identical samples by calculating the repeatability and the reproducibility$

15

6ne relevant food matri, is selected to prepare the test samples$ *rtificial inoculation is used at " levels 4from to 1 " per 1 g1$ 9or each laboratory duplicate samples are prepared and analysed by the alternative method and by the reference method$ #esults from at least @ laboratories )ith valid results must be available for the calculations$ 4Therefore( it is advisable to select 1 312 labs1$ +n case of very e,pensive instruments results from 5 laboratories are considered acceptable$ +f the alternative method comprises more than one protocol a relevant food matri, for each protocol should be selected for the study$ &ote2 :uidelines for the e,pert laboratory conducting the collaborative study see2 *nne, < in -& +S6 1;1" 22 ! 3 Protocol for the validation of alternative methods$ Calculation Tabulate the data obtained )ith each method for statistical analyses for the calculation of standard3deviation of repeatability 4sr1( relative standard3 deviation of repeatability 4#SD r1( repeatability limit 4r1( standard3deviation of reproducibility4S#1( relative standard3deviation of reproducibility 4#SD #1 and reproducibility:limit 4#1$ &ord'al )ill evaluate information provided by the manufacturer based on results for the above mentioned characteristics obtained by using international accepted protocols 4*9&6#( *6*C( +S61$

3e#erences,
*9&6# A 2 22 #e/uirements relating to preliminary and collaborative studies conducted by an e,pert laboratory$ *6*C A 18882 Methods Committee :uidelines for 'alidation of Qualitative and Quantitative Microbiological Methods$ +S6 & 5"! A 2 22 Statistics of *nalytical Data A Protocol for the -stablishment of Precision Characteristics of Microbiological Quantitative methods by +nterlaboratory Studies$ +S6 1;1" A 2 Methods$ !2 Protocol for the 'alidation of *lternative Microbiological