You are on page 1of 9

Articles

Multicentre evaluation of multidisciplinary team meeting


agreement on diagnosis in diffuse parenchymal lung disease:
a case-cohort study
Simon L F Walsh, Athol U Wells, Sujal R Desai, Venerino Poletti, Sara Piciucchi, Alessandra Dubini, Hilario Nunes, Dominique Valeyre,
Pierre Y Brillet, Marianne Kambouchner, António Morais, José M Pereira, Conceição Souto Moura, Jan C Grutters, Daniel A van den Heuvel,
Hendrik W van Es, Matthijs F van Oosterhout, Cornelis A Seldenrijk, Elisabeth Bendstrup, Finn Rasmussen, Line B Madsen, Bibek Gooptu,
Sabine Pomplun, Hiroyuki Taniguchi, Junya Fukuoka, Takeshi Johkoh, Andrew G Nicholson, Charlie Sayer, Lilian Edmunds, Joseph Jacob,
Maria A Kokosi, Jeffrey L Myers, Kevin R Flaherty, David M Hansell

Summary
Background Diffuse parenchymal lung disease represents a diverse and challenging group of pulmonary disorders. Lancet Respir Med 2016;
A consistent diagnostic approach to diffuse parenchymal lung disease is crucial if clinical trial data are to be applied 4: 557–65
to individual patients. We aimed to evaluate inter-multidisciplinary team agreement for the diagnosis of diffuse Published Online
May 11, 2016
parenchymal lung disease.
http://dx.doi.org/10.1016/
S2213-2600(16)30033-9
Methods We did a multicentre evaluation of clinical data of patients who presented to the interstitial lung disease See Comment page 529
unit of the Royal Brompton and Harefield NHS Foundation Trust (London, UK; host institution) and required Department of Radiology
multidisciplinary team meeting (MDTM) characterisation between March 1, 2010, and Aug 31, 2010. Only patients (S L F Walsh MD), King’s College
whose baseline clinical, radiological, and, if biopsy was taken, pathological data were undertaken at the host Hospital NHS Foundation
institution were included. Seven MDTMs, consisting of at least one clinician, radiologist, and pathologist, from Trust, London, UK
(S R Desai MD); Interstitial Lung
seven countries (Denmark, France, Italy, Japan, Netherlands, Portugal, and the UK) evaluated cases of diffuse Disease Unit (Prof A U Wells MD,
parenchymal lung disease in a two-stage process between Jan 1, and Oct 15, 2015. First, the clinician, radiologist, M A Kokosi MD) and
and pathologist (if lung biopsy was completed) independently evaluated each case, selected up to five differential Department of Radiology
diagnoses from a choice of diffuse lung diseases, and chose likelihoods (censored at 5% and summing to 100% in (Prof D M Hansell FRSM), Royal
Brompton and Harefield NHS
each case) for each of their differential diagnoses, without inter-disciplinary consultation. Second, these specialists Foundation Trust, London, UK
convened at an MDTM and reviewed all data, selected up to five differential diagnoses, and chose diagnosis (J Jacob FRCR); Department of
likelihoods. We compared inter-observer and inter-MDTM agreements on patient first-choice diagnoses using Diseases of the Thorax
Cohen’s kappa coefficient (κ). We then estimated inter-observer and inter-MDTM agreement on the probability of (Prof V Poletti MD) and
Department of Radiology
diagnosis using weighted kappa coefficient (κw). We compared inter-observer and inter-MDTM confidence of (S Piciucchi MD), GB Morgagni
patient first-choice diagnosis. Finally, we evaluated the prognostic significance of a first-choice diagnosis of Hospital, Forlì, Italy;
idiopathic pulmonary fibrosis (IPF) versus not IPF for MDTMs, clinicians, and radiologists, using univariate Cox Department of Surgical
regression analysis. Pathology, Morgagni
Pierantoni Hospital, Forlì, Italy
(A Dubini MD); Université Paris,
Findings 70 patients were included in the final study cohort. Clinicians, radiologists, pathologists, and the MDTMs Sorbonne Paris Cité, EA2363
assigned their patient diagnoses between Jan 1, and Oct 15, 2015. IPF made up 88 (18%) of all 490 MDTM first- Réponses cellulaires et
fonctionnelles à l’hypoxie,
choice diagnoses. Inter-MDTM agreement for first-choice diagnoses overall was moderate (κ=0·50). Inter-MDTM
Bobigny, France
agreement on diagnostic likelihoods was good for IPF (κw=0·71 [IQR 0·64–0·77]) and connective tissue disease- (Prof H Nunes MD,
related interstitial lung disease (κw=0·73 [0·68–0·78]); moderate for non-specific interstitial pneumonia (NSIP; Prof D Valeyre MD,
κw=0·42 [0·37–0·49]); and fair for hypersensitivity pneumonitis (κw=0·29 [0·24–0·40]). High-confidence diagnoses Prof P Y Brillet MD); Assistance
Publique-Hôpitaux de Paris,
(>65% likelihood) of IPF were given in 68 (77%) of 88 cases by MDTMs, 62 (65%) of 96 cases by clinicians, and in
Service de Pneumologie
57 (66%) of 86 cases by radiologists. Greater prognostic separation was shown for an MDTM diagnosis of IPF than (Prof H Nunes, Prof D Valeyre),
compared with individual clinician’s diagnosis of this disease in five of seven MDTMs, and radiologist’s diagnosis of Service de Radiologie
IPF in four of seven MDTMs. (Prof P Y Brillet), and Service
d’Anatomie Pathologique
(M Kambouchner MD), Hôpital
Interpretation Agreement between MDTMs for diagnosis in diffuse lung disease is acceptable and good for a diagnosis Avicenne, Bobigny, France;
of IPF, as validated by the non-significant greater prognostic separation of an IPF diagnosis made by MDTMs than Serviço de Pneumologia
the separation of a diagnosis made by individual clinicians or radiologists. Furthermore, MDTMs made the diagnosis (Prof A Morais MD) and
Pathology Department
of IPF with higher confidence and more frequently than did clinicians or radiologists. This difference is of particular
(C Souto Moura MD), Centro
importance, because accurate and consistent diagnoses of IPF are needed if clinical outcomes are to be optimised. Hospitalar São João, Porto,
Inter-multidisciplinary team agreement for a diagnosis of hypersensitivity pneumonitis is low, highlighting an urgent Portugal (J M Pereira MD);
need for standardised diagnostic guidelines for this disease. Faculdade de Medicina,
Universidade do Porto, Porto,
Portugal (Prof A Morais);
Funding National Institute of Health Research, Imperial College London.

www.thelancet.com/respiratory Vol 4 July 2016 557


Articles

ILD Center of Excellence


St Antonius Hospital, Division Research in context
Heart and Lungs, University,
Medical Centre Utrecht, Evidence before this study 2013 ATS/ERS classification of idiopathic interstitial
Netherlands We searched PubMed on Jan 17, 2014, using the search terms pneumonias and the 2011 ATS/ERS/Japanese Respiratory
(Prof J C Grutters MD); “diffuse parenchymal lung disease”, “idiopathic pulmonary Society/Latin American Thoracic Association guidelines for the
Department of Radiology
fibrosis”, “idiopathic interstitial pneumonias”, “interobserver diagnosis and management of idiopathic pulmonary fibrosis.
(D A van den Heuvel MD,
H W van Es MD) and agreement”, “diagnosis”, and “multidisciplinary team” for Our study shows that diagnostic agreement between MDTMs is
Department of Pathology articles written in English and published between Jan 1, 2000, non-significant compared with inter-observer agreement
(M F van Oosterhout MD, and Jan 1, 2014. We extended our search to include studies between clinicians, radiologists, and pathologists in the setting
C A Seldenrijk MD), St Antonius
published up to Dec 1, 2015, during the writing of the of diffuse parenchymal lung disease. Specifically in idiopathic
Hospital, Nieuwegein,
Netherlands; Department of manuscript. We identified seven key publications that were pulmonary fibrosis, MDTMs have a higher level of agreement
Respiratory Diseases and pertinent to our study. Of these, we identified four studies of on diagnoses, assign diagnoses with higher confidence more
Allergy (E Bendstrup PhD) and observer agreement in the setting of diffuse parenchymal lung frequently, and provide diagnoses that have non-significant
Department of Radiology
(F Rasmussen MD) and
disease. All of these seven studies, however, pre-dated the greater prognostic separation than do clinicians or radiologists
Department of Pathology latest American Thoracic Society and the European Respiratory in most cases.
(L B Madsen PhD), Aarhus Society (ATS/ERS) update for the classification of idiopathic
University Hospital, Aarhus, Implications of all the available evidence
interstitial pneumonias, and not all studies had evaluated
Denmark; Division of Asthma, The differences in diagnostic certainty between MDTMs and
Allergy and Lung Biology,
diagnostic agreement between multidisciplinary teams, but
individual clinicians is of particular importance, because
King’s College London, London, rather focused on diagnostic agreement between individual
accurate and consistent diagnoses of idiopathic pulmonary
UK (B Gooptu PhD); observers.
Department of Cellular fibrosis are needed if clinical outcomes are to be optimised. By
Pathology, University College Added value of this study contrast, inter-multidisciplinary team agreement for a
Hospital London, London, UK Our study is the first evaluation, to our knowledge, of inter- diagnosis of hypersensitivity pneumonitis is low, highlighting
(S Pomplun FRCPath);
multidisciplinary team agreement for diagnosis in the setting an urgent need for standardised diagnostic guidelines for
Department of Respiratory
Medicine and Allergy, Tosei of diffuse parenchymal lung disease since the updated this disease.
General Hospital, Aichi, Japan
(H Taniguchi MD); Department
of Pathology, Nagasaki
University Graduate School of Introduction and the availability of novel antifibrotic IPF drugs
Biomedical Sciences, Nagasaki, Diffuse parenchymal lung disease represents a diverse (pirfenidone1 and nintedanib2), all of which might affect
Japan (Prof J Fukuoka MD);
and challenging group of pulmonary disorders with diagnostic decisions. Furthermore, many of these studies
Department of Radiology,
Kinki Central Hospital of varied prognoses and different management options. focused on individual observers rather than agreement
Mutual Aid Association of A consistent diagnostic approach to these diseases is between multidisciplinary teams.4–6,9,11 In this study, we
Public School Teachers, Itami, essential if clinical trial data are to be reliably applied to aimed to evaluate the level of inter-multidisciplinary team
Japan (T Johkoh MD); National
individual patients. With the 2014 licensing of two new diagnostic agreement between seven international
Heart and Lung Institute,
Imperial College London, antifibrotic idiopathic pulmonary fibrosis (IPF) drugs centres for diagnosis of diffuse parenchymal lung disease.
London, UK (pirfenidone1 and nintedanib2), accurate and consistent
(Prof A G Nicholson DM, diagnosis of IPF is of particular importance to achieve Methods
L Edmunds FRCPath); Brighton
clinical benefits for patients. In 2002, a joint statement by Study design, patients, and multidisciplinary team
and Sussex University
Hospitals Trust, Brighton, UK the American Thoracic Society (ATS) and the European selection
(C Sayer FRCR); and Department Respiratory Society (ERS) on the classification of For this case-cohort study we selected consecutive
of Pathology (Prof J L Myers MD) idiopathic interstitial pneumonias advocated a multi- patients who presented to the interstitial lung disease
and Department of Internal
Medicine, Division of
disciplinary diagnostic approach, involving integration of unit of the Royal Brompton and Harefield NHS
Pulmonary and Critical Care clinical, radiological, and, in cases for which lung biopsy Foundation Trust (London, UK; host institution) and
Medicine (Prof K R Flaherty MD), material is available, pathological data.3 This approach patients with challenging diagnosis had MDTM
University of Michigan, Ann has been emphasised by several studies4–7 in the past characterisation, between March 1, 2010, and
Arbor, MI, USA
12 years and was restated in the 2013 ATS/ERS update8 on Aug 31, 2010. Only patients who had all their clinical
Correspondence to:
idiopathic interstitial pneumonia classification. Although investigations (serology, high-resolution CT, and, when
Dr Simon L F Walsh, Department
of Radiology, King’s College this recommendation specifically applies to idiopathic required, surgical lung biopsy) completed at the host
Hospital NHS Foundation Trust, interstitial pneumonia, a multidisciplinary approach has institution were included. Seven multidisciplinary
London SE5 9RS, UK been widely used as the diagnostic gold standard for teams from seven different countries (Denmark,
slfwalsh@gmail.com
diffuse parenchymal lung disease in general.4,6 Several France, Italy, Japan, Netherlands, Portugal, and the
studies4–6,9 have evaluated inter-observer agreement for UK), each with specialist expertise in the diagnosis and
diagnosis in the setting of diffuse parenchymal lung management of diffuse parenchymal lung disease,
disease. However, most of these studies pre-date the were invited and agreed to participate in the study. The
2013 ATS/ERS update,8 the 2011 joint ATS/ERS/Japanese only prerequisite for participation in the study was that
Respiratory Society/Latin American Thoracic Association each multidisciplinary team had to have a regular
statement10 on the diagnosis and management of IPF, multidisciplinary meeting (about once a week) for

558 www.thelancet.com/respiratory Vol 4 July 2016


Articles

diffuse parenchymal lung disease in place with versus individual specialists, we compared the mortality
consistent attendance by at least one clinician, of each team’s diagnosis of IPF. We compared diagnoses
radiologist, and pathologist. For the retrospective by separating the entire cohort into a binary IPF diagnosis
examination of clinically indicated data, the institutional category (IPF [if first-choice diagnosis for a patient is
ethics review board of Royal Brompton and Harefield IPF] and not [if first-choice diagnosis is of another cause])
NHS Foundation Trust waived the need for informed for each MDTM, clinician, and radiologist, based on
patient consent. assigned diagnoses. Pathologist diagnoses are not shown
due to small number of cases (appendix). We calculated
Evaluation of cases the survival period for each patient from the date of
The evaluation of every case took place in two stages referral to the host institution to the minimum of date of
between Jan 1, and Oct 15, 2015. At first, clinicians, death, date the patient was last known to be alive, or
radiologists, and pathologists had to review the cases June 1, 2015 (end of the study period). We obtained vital
independently without interspecialty consultation. status for every patient on June 1, 2015, by evaluating
Clinicians had access to all the presenting clinical their electronic patient record.
information (age, sex, smoking history, history of
established connective tissue disease, symptoms Statistical analysis
[including those suggestive of connective tissue disease], We used Cohen’s kappa coefficient (κ) to evaluate
autoantibody profile, exposure history, drugs at inter-observer and inter-MDTM agreement for patient’s
presentation, bronchoalveolar lavage result [if done], first-diagnosis. We used Cohen’s weighted-kappa
and angiotensin converting enzyme concentration [if coefficient (κw) to evaluate inter-observer agreement and
assessed]), pulmonary function tests, and high-resolution inter-MDTM agreement for an estimation of the probability
CT (no access to the original high-resolution CT report or of each diagnosis. We converted the percentage diagnostic
the pathology result). Radiologists and pathologists had likelihood given for each diagnosis to a five point scale
access to only the age, sex, and smoking history for the (0–4), representing clinically useful probabilities (0 for
patient, and the high-resolution CT (radiologist) or condition not included in the differential diagnosis; 1 for
digitalised surgical lung biopsy slides (pathologist) taken low probability [5–25%]; 2 for intermediate probability
at presentation. Pathologists had access to all pathology [30–65%]; 3 for high probability [70–95%]; and 4 for
data that were available in the form of digitalised slides pathognomonic [100%]). For example, if the differential
(in .svs format), which were viewed using Aperio diagnoses given by an MDTM were IPF
ImageScope (version 12.3) viewing software. This digital (65% diagnostic likelihood), non-specific interstitial
viewing application has all the imaging functionality pneumonia (NSIP; 25% diagnostic likelihood), and
normally available to pathologists in routine clinical hypersensitivity pneumonitis (10% diagnostic likelihood),
practice and the host institution uses it to evaluate cases the probability grades would be 2 for IPF, 1 for NSIP, and
referred from outside institutions for additional opinions. 1 for hypersensitivity pneumonitis. We calculated weighted-
For each patient, observers (clinicians, radiologists, kappa coefficient values between paired observers (for
and pathologists) had to select up to five differential statements of inter-observer agreement), and between
diagnoses and provide a diagnostic likelihood (censored paired multidisciplinary teams (for statements of inter-
at 5% increments and summing to 100% in each case) MDTM agreement), which were expressed as median
from a drop-down menu of diffuse lung diseases (IQR) values for all unique combinations of pairs (21 for
(appendix). The only stipulation was that diagnoses were seven observers or seven multidisciplinary teams). See Online for appendix
considered in the context of the current 2013 version of Weighting of the kappa coefficient allowed us to quantify
the ATS/ERS classification and terminology for idiopathic the extent of disagreement, by assigning greater emphasis
interstitial pneumonias.8 to large differences between scores. Weighted-kappa
Once the cases had been reviewed independently, the coefficients were categorised as poor (0<κw≤0·20), fair
clinician, radiologist, and pathologist convened as a (0·20<κw≤0·40), moderate (0·40<κw≤0·60), good
multidisciplinary team to review the cases together and (0·60<κw≤0·80), and excellent (0·80<κw≤1·00). This
provide up to five diagnoses with diagnostic likelihoods approach has been used in previous investigations9,11 of
(also censored at 5% increments and summing to inter-observer agreement for diagnosis in diffuse lung
100% in each case). All clinical information supplied in diseases.
the first stage, including pulmonary function tests, Additionally, for each patient the first-choice diagnosis
high-resolution CT at presentation and digitalised was considered low confidence (diagnostic likelihood
surgical lung biopsy slides were available to the <70%) or high confidence (diagnostic likelihood ≥70%).
multidisciplinary team. These categories were based on the diagnostic likelihood
In this study we aimed to evaluate inter- categories used to assess the clinical probability of
multidisciplinary team agreement for the diagnosis of pulmonary embolism in the PIOPED study.12
diffuse parenchymal lung disease. To validate the We used univariate Cox regression analysis to identify
diagnosis made by the MDTMs at the seven centres associations between mortality and MDTM, clinician,

www.thelancet.com/respiratory Vol 4 July 2016 559


Articles

and radiologist diagnoses in terms of IPF versus not IPF. Role of the funding source
Reported hazard ratios (HRs) are for diagnosis of IPF The funders of the study had no role in study design,
versus not IPF. We tested the assumptions of proportional data collection, data analysis, data interpretation, or
hazards by visual inspection of the log–log plot of writing of the report. The corresponding author had full
survival, comparison of the Kaplan-Meier observed access to all of the data in the study and had final
survival curves with the Cox predicted curves for the responsibility for the decision to submit for publication.
same variable, and graphical and formal analysis of
Schoenfeld residuals (analysis not shown). Results are Results
reported as HRs, 95% CIs, and p values, and are We identified 113 consecutive new patient referrals, who
graphically shown as Kaplan-Meier survival curves. We required local MDTM characterisation, from the clinical
completed statistical analyses using Stata (version 12; database of the host institute between March 1, 2010, and
StataCorp, College Station, Texas). Aug 31, 2010. We excluded 43 (38%) of 113 referrals on the
basis that their initial work-up high-resolution CT scan
(29 patients), lung function (four patients), or surgical
Patients (n=70)
lung biopsy (ten patients) were completed by the referring
Age (years) 60·9 (15·5) institution (appendix). The remaining 70 (62%) of
Sex 113 patient referrals were included as the final study
Men 24 (34%) cohort. Basic patient demographics are in table 1. 13 (19%)
Women 46 (66%) of 70 patients had an established diagnosis of a connective
Smoking tissue disease (seven patients had systemic sclerosis,
Never 38 (54%) three had rheumatoid arthritis, two had Sjögren’s
Previously smoked 24 (34%) syndrome, and one had mixed connective tissue disease)
Current 8 (11%) at the time of presentation to the host institution, and
Established connective tissue disease history 13 (19%) 22 (31%) of 70 cases underwent surgical lung biopsy at the
Biopsy taken host institution (table 1). In patients in whom a surgical
Yes 22 (31%) lung biopsy was not done (table 1), a confident diagnosis
No 48 (69%) had been made by the host institution without the need
DLco (% predicted) 44·8 (14·5%) for a surgical lung biopsy sample. Vital status was known
FEV1 (% predicted) 73·0 (20·5%) for all patients at the end of the study period (June 1, 2015).
FVC (% predicted) 79·0 (19·6%) 70 patients resulted in the assignment of 490 first-
choice MDTM diagnoses (70 patients evaluated by
Where appropriate, data are mean (SD) and n (%). DLco=diffusing capacity of the
lungs for carbon monoxide. FEV1=forced expiratory volume in 1 s. FVC=forced
seven MDTMs). First-choice diagnoses are shown in
vital capacity. table 2. The NSIP and organising pneumonia overlap with
interstitial lung disease category was combined with the
Table 1: Patient demographics and characteristics
NSIP category. The four most prevalent first-choice
diagnoses (connective tissue disease-related interstitial
lung disease, IPF, hypersensitivity pneumonitis, and
Number of first-choice NSIP) were the focus of subsequent analyses. The
diagnoses (n=490) remaining diagnosis categories whose frequency was less
Connective tissue disease-related interstitial 146 (30%) than 10% of the total number of first-choice diagnoses
lung disease (other, sarcoidosis, drug-related interstitial lung
Idiopathic pulmonary fibrosis 88 (18%) disease, occupational lung disease, pleuroparenchymal
Hypersensitivity pneumonitis 46 (9%) fibroelastosis, and organising pneumonia) were combined
Idiopathic non-specific interstitial pneumonia 42 (9%) into an others diagnosis category. The final diagnosis
Unclassifiable interstitial lung disease 38 (8%) categories were connective tissue disease-related
Other 37 (8%) interstitial lung disease, IPF, idiopathic NSIP,
Sarcoidosis 20 (4%) hypersensitivity pneumonitis, and others.
Drug-related interstitial lung disease 18 (4%) Inter-observer agreement (between clinicians,
Smoking-related interstitial lung disease 16 (3%) radiologists, and pathologists) and inter-MDTM
Occupational lung disease 14 (3%) agreement for first-choice diagnosis are listed in table 3.
Pleuroparenchymal fibroelastosis 10 (2%) Overall inter-MDTM agreement for first-choice diagnosis
Organising pneumonia or non-specific 8 (2%) in patients was moderate (table 3). Inter-MDTM
interstitial pneumonia overlapping disease agreement for a first-choice diagnosis of IPF or of
Organising pneumonia 7 (1%) connective tissue disease-related interstitial lung disease
was good, but for a first-choice diagnosis of idiopathic
Table 2: First-choice diagnoses given by seven multidisciplinary teams
NSIP or hypersensitivity pneumonitis was fair (table 3).
for 70 cases of diffuse lung disease
In the subgroup analysis of patients in whom a surgical

560 www.thelancet.com/respiratory Vol 4 July 2016


Articles

lung biopsy was not done (table 1), overall inter-MDTM


Clinicians (κ) Radiologists (κ) Pathologists (κ) MDTM (κ)
agreement for diagnosis was moderate (table 3).
Inter-MDTM agreement for first-choice diagnoses of IPF, Total No Total No Total No Total No
(n=70) biopsy (n=70) biopsy (n=70) biopsy (n=70) biopsy
connective tissue disease-related interstitial lung disease, (n=48) (n=48) (n=48) (n=48)
and hypersensitivity pneumonitis in these patients were
Overall total 0·45 0·50 0·33 0·31 0·31 ¨ 0·50 0·57
all greater than the first-choice diagnoses of all patients
Idiopathic pulmonary 0·59 0·71 0·46 0·42 0·46 ¨ 0·60 0·70
(table 3). fibrosis total
Inter-MDTM agreement on the diagnostic likelihood of Non-specific interstitial 0·19 0·19 0·25 0·25 0·23 ¨ 0·25 0·25
IPF and connective tissue disease-related interstitial lung pneumonia total
disease was good, but moderate for idiopathic NSIP, and Connective tissue 0·57 0·62 0·10 0·11 0·22 ¨ 0·64 0·73
fair for hypersensitivity pneumonitis diagnoses (table 4). disease-related interstitial
lung disease total
Subgroup analysis of inter-MDTM agreement on the
Hypersensitivity 0·25 0·38 0·27 0·22 0·20 ¨ 0·24 0·31
estimation of diagnostic likelihood of IPF in patients pneumonitis total
without lung biopsy was good (κw=0·78, IQR 0·74–0·83).
Overall inter-observer agreement between clinicians MDTM=multidisciplinary team meeting.
for first-choice diagnosis was moderate and was fair Table 3: Unweighted kappa values (κ) for clinicians, radiologists, pathologists, and inter-
between radiologists and between pathologists (table 3). multidisciplinary team meeting agreement on individual diagnoses of diffuse parenchymal lung disease
Agreement between clinicians on the probability of a
diagnosis of IPF or connective tissue disease-related
interstitial lung disease was higher than agreement on Clinicians (κw) Radiologists (κw) Pathologists (κw) MDTM (κw)
the probability of a diagnosis of idiopathic NSIP or Idiopathic pulmonary 0·72 (0·67–0·76) 0·60 (0·46–0·66) 0·58 (0·45–0·66) 0·71 (0·64–0·77)
hypersensitivity pneumonitis (table 4). Agreement fibrosis
between radiologists or pathologists was highest for the Connective tissue 0·76 (0·70–0·78) 0·17 (0·08–0·31) 0·21 (0·06–0·36) 0·73 (0·68–0·78)
probability of a diagnosis of IPF compared with disease-related
interstitial lung disease
agreement on the probability of a diagnosis of connective
Non-specific interstitial 0·31 (0·27–0·41) 0·32 (0·26–0·41) 0·30 (0·00–0·53) 0·42 (0·37–0·49)
tissue disease-related interstitial lung disease, idiopathic
pneumonia
NSIP, or hypersensitivity pneumonitis (table 4).
Hypersensitivity 0·42 (0·30–0·47) 0·35 (0·29–0·43) 0·26 (0·10–0·45) 0·29 (0·24–0·40)
At the time of patient selection, 13 (19%) of 70 patients pneumonitis
had an established diagnosis of a connective tissue
disease, meaning any diagnosis of interstitial lung Data are median (IQR). MDTM=multidisciplinary team meeting.

disease in these cases would most likely be classified as Table 4: Weighted kappa values (κw) for estimation of diagnostic likelihood for individual diagnoses of
connective tissue disease-related interstitial lung disease. diffuse parenchymal lung disease
To investigate whether in these 13 cases there was high
agreement for this diagnosis resulted in an increase in
agreement on non-connective tissue disease diagnoses diagnoses were made with high confidence. In this
and, particularly, affected agreement on a diagnosis of subgroup, median prevalence of first-choice diagnoses
IPF, we did a post-hoc subgroup analysis in the 57 (81%) made with high confidence were 60·4% (37·5–75·0) by
of 70 cases who did not have an established diagnosis of clinicians, 66·6% (39·6–83·3) by radiologists, and 68·7%
connective tissue disease. In this analysis, although (IQR 52·8–87·5) by the multidisciplinary teams.
inter-MDTM agreement for a first-choice diagnosis of For the diagnosis of IPF, supportive non-significant
connective tissue disease-related interstitial lung disease increases in diagnostic confidence by MDTMs (68 [77%]
decreased (from κ=0·57 to κ=0·42), no significant change of 88) were shown, compared with clinicians (62 [65%] of
was observed in inter-MDTM agreement for a first- 96) or radiologists (57 [66%] of 86; p=0·23). In the
choice diagnosis of IPF (κ=0·58), idiopathic NSIP 22 (31%) of 70 cases that underwent surgical lung biopsy
(κ=0·24), or hypersensitivity pneumonitis (κ=0·23; (therefore a total of 154 diagnoses by seven pathologists),
appendix). 15 (10%) of 154 cases were assigned a first-choice
347 (70%) of 490 first-choice multidisciplinary team diagnosis of IPF, of which 12 (8%) were assigned with
diagnoses were made with high confidence (appendix). high confidence.
Median prevalence of first-choice multidisciplinary team We reviewed the 15 cases who the pathologists gave a
diagnoses made with high confidence was 67·1% first-choice diagnosis of IPF to ascertain whether in
(IQR 54·3–88·8). Median prevalence of first-choice patients who had had a surgical lung biopsy, the final
diagnoses made with high confidence was 58·9% MDTM diagnosis was usually IPF. In six of 15 cases,
(IQR 52·9–71·4) by clinicians, 68·6% (35·7–85·7) by despite the pathologist giving a first-choice diagnosis of
radiologists, and 72·7% (59·1–81·8) by pathologists IPF, the final MDTM first-choice diagnosis was not IPF.
(appendix). In the subgroup analysis of the 48 (69%) of Furthermore, in only two of 15 cases was IPF not already
70 patients who did not undergo a surgical lung biopsy, suggested by either the clinician or radiologist in the
237 (71%) of 336 first-choice multidisciplinary team MDTM (appendix).

www.thelancet.com/respiratory Vol 4 July 2016 561


Articles

In the univariate Cox regression analysis, the Discussion


multidisciplinary distinction between IPF and other We have shown that an acceptable level (based on κ>0·40
diagnoses (not IPF) had non-significant increases is deemed clinically acceptable) of diagnostic agreement
toward greater prognostic separation (as judged by exists between multidisciplinary teams in the setting of
HR p values; all seven MDTMs had a significant diffuse parenchymal lung disease. Additionally, we
separation) than the clinician distinction (in showed that this agreement was validated by the non-
five of seven teams) or the radiologist distinction (in significant increases towards greater prognostic
four of seven teams; table 5). The figure shows the separation of an IPF diagnosis made by multidisciplinary
Kaplan-Meier survival curves for the categorisation of teams than by individual clinicians or radiologists.
first-choice diagnosis as IPF or not IPF in cases (by at Furthermore, MDTMs make the diagnosis of IPF with
least four of seven MDTMs, four of seven clinicians, high confidence more frequently than do clinicians or
and four of seven radiologists). The same analysis for radiologists.
pathologists’ diagnosis of IPF was not significant for Since the publication of the ATS/ERS 2002 consensus
five of seven pathologists, probably because of the statement on the classification of idiopathic interstitial
small subgroup size (22 patients) and low prevalence pneumonias, multidisciplinary evaluation of diffuse
of IPF in this subgroup (in 15 of 154 diagnoses parenchymal lung disease has been widely used as the
pathologists’ first-choice diagnoses were IPF; diagnostic gold standard.3,8 This diagnostic approach has
appendix). been partly investigated in several settings. Flaherty and
colleagues4 examined the formulation of diagnosis in a
cohort of diffuse parenchymal lung diseases against
Clinicians (HR, 95% CI, Radiologists (HR, 95% CI, MDTM (HR, 95% CI, p value)
p value) p value) inter-observer agreement and diagnostic confidence in
one multidisciplinary team (consisting of three clinicians,
Team 1 2·09 (0·90–4·86, p=0·085) 2·80 (1·17–6·73, p=0·021) 2·67 (1·21–6·02, p=0·016)
two radiologists, and two pathologists) and showed that
Team 2 2·95 (1·33–6·59, p=0·008) 4·08 (1·84–9·04, p=0·001) 3·44 (1·54–7·68, p=0·003)
diagnostic agreement between observers improved with
Team 3 3·75 (1·65–8·51, p=0·002) 2·78 (1·11–6·97, p=0·030) 5·30 (2·26–12·41, p<0·001)
successive integration of clinical, radiological, and
Team 4 3·34 (1·38–8·00, p=0·007) 4·49 (1·71–12·29, p=0·003) 3·99 (1·49–10·66, p=0·006)
pathological data. In a second study,6 Flaherty and
Team 5 2·03 (0·87–4·69, p=0·100) 2·58 (1·08–6·21, p=0·033) 2·61 (1·12–6·06, p=0·025)
colleagues expanded on these findings by showing
Team 6 4·14 (1·72–9·97, p=0·002) 2·11 (0·91–4·89, p=0·082) 3·36 (1·40–8·07, p=0·007)
higher levels of agreement between academic physicians,
Team 7 2·96 (1·43–6·55, p=0·007) 1·28 (0·53–3·06, p=0·583) 2·43 (1·09–5·41, p=0·030)
radiologists, and pathologists for diagnosis in diffuse
Results for the multidisciplinary team meetings (MDTMs), clinicians, and radiologists are based on the whole patient lung disease than with their community counterparts.
cohort (n=70). HR=hazard ratio. Later in 2008, Thomeer and colleagues5 reported in
Table 5: Univariate Cox regression analysis for mortality according to clinician, radiologist, and MDTM
a cohort of patients included in an IPF trial, a high
diagnoses of idiopathic pulmonary fibrosis versus not idiopathic pulmonary fibrosis accuracy for a clinical diagnosis of this disease made
by six respiratory physicians from different European
centres. A limitation of these studies is that
100 not all evaluated the agreement between different
multidisciplinary teams for diagnoses,4 one focused
specifically on the diagnosis of IPF5 and all studies pre-
Cumulative survival proportion (%)

80 Non-IPF survival date the latest 2013 ATS/ERS idiopathic interstitial


pneumonia update.4–6,8,9 To our knowledge this study is
60 the first to evaluate inter-MDTM agreement for diagnosis
in diffuse lung disease since the 2013 ATS/ERS update
on the classification of idiopathic interstitial pneumonias.
40
With the 2014 licensing of two new drugs1,2 for
IPF survival
treatment of IPF in mild to moderate disease, early and
20 MDTMs accurate diagnosis of IPF is crucial. Our study showed
Clinicians that inter-MDTM agreement for the diagnosis of IPF is
Radiologists
0
good, with clinicians having only marginally lower levels
0 500 1000 1500 2000 of agreement than MDTMs for this diagnosis. On
Follow-up time (days) subgroup analysis in patients without surgical lung
biopsy, inter-MDTM agreement and inter-observer
Figure: Kaplan-Meier of survival differences between patients assigned a diagnosis of idiopathic pulmonary
fibrosis and those assigned other diagnoses (not idiopathic pulmonary fibrosis) agreement between clinicians for the diagnosis of IPF
The cohort of 70 patients was separated into cases for which at least four of seven multidisciplinary team improved and were, again, almost the same. As evidence-
meetings (MDTMs) assigned a first-choice diagnosis of idiopathic pulmonary fibrosis (IPF; blue curve; hazard based diagnostic criteria for IPF are now clearly defined
ratio 6·26, 95% CI 2·72–14·33, p<0·0001), at least four of seven clinicians assigned a first-choice diagnosis of IPF
and are relatively easily applied to many patients with
(red curve; 4·43, 1·94–10·01, p<0·0001), and at least four of seven radiologists assigned a first-choice diagnosis of
IPF (green curve; 3·76, 1·61–8·75, p=0·002); or into other diagnoses (not IPF). For number of patients at risk see this disease, the near parity of agreement between
appendix. MDTMs and between clinicians is not surprising.8,10 For

562 www.thelancet.com/respiratory Vol 4 July 2016


Articles

example, in a patient with classic appearances for usual our methodology has allowed us to show that there is good
interstitial pneumonia on high-resolution CT, a rapidly agreement on the likelihood of IPF and, as previously
progressive disease course, and no identifiable triggers, stated, reflects consistent application of established
multidisciplinary discussion is unlikely to alter a diagnostic guidelines for this disease. By contrast, MDTM
clinician’s initial impression of IPF. Our findings suggest agreement on the likelihood of hypersensitivity
that formal multidisciplinary input might not be pneumonitis was low, reinforcing the view that MDTMs
necessary in every case of suspected IPF if expert clinical were unclear on how this diagnosis is made.
evaluation is available, which could be of particular Two separate observations from our study warrant
relevance to centres with restricted access to appropriate further discussion. First, 13 patients had an established
radiology or pathology expertise.13 diagnosis of a connective tissue disease at presentation.
A strength of our findings is that the greater agreement However, after MDT evaluation, a connective tissue
on multidisciplinary diagnosis is mirrored by the disease diagnosis was constructed in an additional
non-significant improvement in prognostic separation eight cases or more by five of the seven MDTMs based on
of a multidisciplinary distinction between IPF and other presenting clinical symptoms and serology. Separation of
diagnoses compared with the clinicians’ or radiologists’ patients with idiopathic interstitial pneumonia from
distinction. To show this, we selected patients from those with connective tissue disease-related interstitial
2010 data when they first presented to the host institution, lung disease can be challenging—some patients present
to allow an analysis of 5-year survival. An added with subtle clinical features or serological abnormalities,
advantage of this approach was that these patients were suggesting an autoimmune process, but do not meet
referred to the Royal Brompton and Harefield NHS established criteria for a specific connective tissue
Foundation Trust in a pre-antifibrotic drug era, therefore disease.20–22 In 2015, an ERS/ATS task force was formed to
mortality differences between patients with IPF and establish consensus on how to classify these patients,
those without this disease were not confounded by and a set of diagnostic criteria has been suggested.23–26
antifibrotic therapy. The fact that non-significant After removal of patients with an established diagnosis of
improvements are present but inconclusive might reflect connective tissue disease, our subgroup analysis showed
powering limitations. Additionally, this inconsistency acceptable levels of agreement on connective tissue
might indicate that multidisciplinary discussion adds disease-related interstitial lung disease (by contrast
prognostic value in the subgroup of patients in which with hypersensitivity pneumonitis), underlying the
there is significant diagnostic uncertainty. importance of reaching an international consensus on
By contrast with those with IPF, many patients with disease definitions. Furthermore, the high frequency of
non-IPF interstitial lung diseases are not covered by connective tissue disease-related interstitial lung disease
evidence-based diagnostic guidelines, therefore their diagnoses made in our study highlights the importance
diagnosis is driven by clinical reasoning and analysis of of formal rheumatology input within the MDTM, which
all available data in a multidisciplinary setting. In these might include face-to-face rheumatological consultation
situations, a level of disagreement between MDTMs is with patients suspected of having an undiagnosed
predictable and borne out by the poor level of inter-MDTM connective tissue disease.
agreement in the current study for diagnoses of NSIP Second, dynamic exchanges of clinical, radiological,
and hypersensitivity pneumonitis.8,14–16 The diagnosis of and pathological information between experts in a
hypersensitivity pneumonitis is well known to be multidisciplinary process has previously been suggested
challenging because it relies on an array of data, which to result in higher-confidence diagnoses.4 However, in
are not definitive when considered in isolation.15 our study the proportion of high-confidence diagnoses
Furthermore, at least on high-resolution CT, patterns of (≥70%) assigned after MDTM evaluation did not increase
NSIP, usual interstitial pneumonia, or organising compared with the proportion of high-confidence
pneumonia might be the only expressions of this diagnoses assigned by the individual components of the
disease.16–19 The low MDTM agreement for the diagnosis MDTM. In most MDTMs, high-confidence diagnoses
of hypersensitivity pneumonitis in our study highlights were more frequently assigned by radiologists and
an urgent need for international consensus on what pathologists than were MDTM diagnoses or diagnoses
hypersensitivity pneumonitis actually is. assigned by clinicians. In our study, radiologists and
Our use of the weighted kappa to investigate inter- pathologists did not have access to clinical information,
MDTM and inter-observer agreement on diagnostic so their interpretation was based almost entirely on
probabilities is similar to other studies9 of diagnostic pattern recognition, which might conceivably result in
agreement, but warrants particular consideration. We used more confident, but not necessarily more accurate,
the weighted kappa converting the diagnostic likelihoods diagnoses. Although one benefit of the multidisciplinary
to a five-point probability scale, which enabled examination process is that diagnoses might be challenged and must
of the range of diagnostic likelihoods from minimal be publicly defended, it is possible that extra discussion
likelihood to pathognomonic. Since exclusion of IPF in creates additional difficulties in some cases that initially
diagnosis is as important as making a diagnosis of IPF, seem straightforward when evaluated by individuals in

www.thelancet.com/respiratory Vol 4 July 2016 563


Articles

isolation. However, we must highlight that in the specific Pharmaceuticals Japan Ltd, Ashai Kasei Pharmaceutical Corporation,
case of IPF, MDTMs made the diagnosis of IPF with Astellas Pharmaceutical Incorporated, AstraZeneca, Bayer, Boehringer
Ingelheim, and Chugai Pharmaceuticals. JF has received personal fees
high confidence more frequently than did clinicians or from Astellas Pharmaceutical Incorporated, Pathology Institute
radiologists. Corporation, Chugai Pharmaceuticals, and Sakura Finetek Japan. AGN
Our methodology has some limitations. Since patients has received personal fees from Sanofi, Intermune, Boehringer
were selected from a pre-antifibrotic drug era, IPF was Ingelheim, and Actelion. KRF has received personal fees from
Boehringer Ingelheim, Genentech, Ikaria, Immuneworks, Veracyte,
possibly not as prevalent in our study cohort as it would Roche, Gilead, Biogen, Afferent, Aeolus, and Pharmakea; and has
be now at most referral centres, which expend more time received grants from Boehringer Ingelheim, Genentech, Roche, and
in their evaluation of patients for approval for antifibrotic Afferent. DMH has received personal fees from Boehringer Ingelheim,
treatment. However, as discussed, the inclusion of Sanofi, AstraZeneca, Roche, and GlaxoSmithKline. The other authors
declare no competing interests.
patients who had not had antifibrotic drugs allowed us
to evaluate the veracity of MDTM diagnosis for IPF Acknowledgments
We thank the National Institute of Health Research Respiratory Disease
against the outcomes. Second, unlike a real-world Biomedical Unit (Royal Brompton and Harefield NHS Foundation Trust,
multidisciplinary process, no observers had face-to-face London, UK) and Imperial College London (London, UK). DMH is the
consultations with patients and therefore did not have recipient of a National Institute of Health Research Senior Investigator
the benefit of obtaining a clinical history or doing Award.

a physical examination first-hand. In difficult to References


1 King TE Jr, Bradford WZ, Castro-Bernardini S, et al. A phase 3 trial
characterise patients, a clinician’s impression might be of pirfenidone in patients with idiopathic pulmonary fibrosis.
influenced by direct contact with the patient in ways that N Engl J Med 2014; 370: 2083–92.
are not easily and objectively quantified. Equally 2 Richeldi L, du Bois RM, Raghu G, et al. Efficacy and safety of
nintedanib in idiopathic pulmonary fibrosis. N Engl J Med 2014;
important, but also difficult to assess, is the effect of 370: 2071–82.
individual personalities on MDTM diagnosis. Arguably, a 3 American Thoracic Society, European Respiratory Society.
consensus diagnosis at multidisciplinary evaluation American Thoracic Society/European Respiratory Society
sometimes might reflect the strongest voice in the room international multidisciplinary consensus classification of the
idiopathic interstitial pneumonias. This joint statement of the
rather than represent true agreement between American Thoracic Society (ATS), and the European Respiratory
participants. Lastly, the wide range of proportions of Society (ERS) was adopted by the ATS board of directors, June 2001
and by the ERS Executive Committee, June 2001.
high-confidence diagnoses reported between the Am J Respir Crit Care Med 2002; 165: 277–304.
different MDTMs in our study might reflect cultural 4 Flaherty KR, King TE Jr, Raghu G, et al. Idiopathic interstitial
influences on the dynamics of multidisciplinary pneumonia: what is the effect of a multidisciplinary approach to
diagnosis? Am J Respir Crit Care Med 2004; 170: 904–10.
evaluation. These less tangible influences could be the
5 Thomeer M, Demedts M, Behr J, et al. Multidisciplinary
focus of further studies. interobserver agreement in the diagnosis of idiopathic pulmonary
In conclusion, our study showed that diagnostic fibrosis. Eur Respir J 2008; 31: 585–91.
agreement between MDTMs is higher than inter- 6 Flaherty KR, Andrei AC, King TE Jr, et al. Idiopathic interstitial
pneumonia: do community and academic physicians agree on
observer agreement between clinicians, radiologists, and diagnosis? Am J Respir Crit Care Med 2007; 175: 1054–60.
pathologists in the setting of diffuse parenchymal lung 7 Tomassetti S, Piciucchi S, Tantalocco P, Dubini A, Poletti V.
disease. In particular, inter-MDTM agreement for a The multidisciplinary approach in the diagnosis of idiopathic
pulmonary fibrosis: a patient case-based review. Eur Respir Rev 2015;
diagnosis of IPF is good and validated by the non- 24: 69–77.
significant greater prognostic separation of an IPF 8 Travis WD, Costabel U, Hansell DM, et al. An official American
diagnosis made by multidisciplinary teams than by Thoracic Society/European Respiratory Society statement: update of
the international multidisciplinary classification of the idiopathic
individual specialists. We showed low levels of inter- interstitial pneumonias. Am J Respir Crit Care Med 2013;
MDTM agreement for NSIP and hypersensitivity 188: 733–48.
pneumonitis, which for hypersensitivity pneumonitis 9 Aziz ZA, Wells AU, Hansell DM, et al. HRCT diagnosis of diffuse
parenchymal lung disease: inter-observer variation. Thorax 2004;
might be in relation to the absence of evidence-based 59: 506–11.
guidelines to diagnose this disease. 10 Raghu G, Collard HR, Egan JJ, et al. An official ATS/ERS/JRS/ALAT
Contributors statement: idiopathic pulmonary fibrosis: evidence-based guidelines
SLFW, AUW, SRD, and DMH had the study concept and did data for diagnosis and management. Am J Respir Crit Care Med 2011;
183: 788–824.
analysis, and manuscript writing. SLFW, AUW, SRD, AGN, CS, LE, JJ,
MAK, and DMH did data collection. AGN and LE did evaluation of 11 Walsh SL, Calandriello L, Sverzellati N, Wells AU, Hansell DM,
UIP Observer Consort. Interobserver agreement for the ATS/ERS/
digitalised pathology data. JLM and KRF did manuscript review and
JRS/ALAT criteria for a UIP pattern on CT. Thorax 2016; 71: 45–51.
editing. All remaining authors made up the seven multidisciplinary
12 PIOPED Investigators. Value of the ventilation/perfusion scan in
teams from Denmark (EB, FR, LBM), France (HN, DV, PYB, MK), Italy
acute pulmonary embolism. Results of the prospective investigation
(VP, SP, AD), Japan (HT, JF, TJ), Netherlands (JCG, DAvdH, HWvE, of pulmonary embolism diagnosis (PIOPED). JAMA 1990;
MFvO, CAS), Portugal (AM, JMP, CSM), and the UK (SRD, BG, SP). 263: 2753–59.
Declaration of interests 13 Jo HE, Corte TJ, Moodley Y, et al. Evaluating the interstitial lung
AUW has received personal fees from Intermune, Roche, Bayer, and disease multidisciplinary meeting: a survey of expert centres.
Gilead. HN is an investigator for clinical trials by Intermune, Roche, BMC Pulm Med 2016; 16: 22.
Boehringer Ingelheim, Sanofi, and Centocor. DV has received personal 14 Travis WD, Hunninghake G, King TE Jr, et al. Idiopathic nonspecific
fees from Roche, Intermune and Boehringer Ingelheim. HT has interstitial pneumonia: report of an American Thoracic Society
project. Am J Respir Crit Care Med 2008; 177: 1338–47.
received personal fees from Abbott Japan Co Ltd, Actelion

564 www.thelancet.com/respiratory Vol 4 July 2016


Articles

15 Lacasse Y, Selman M, Costabel U, et al. Clinical diagnosis of 22 Mittoo S, Gelber AC, Christopher-Stine L, Horton MR, Lechtzin N,
hypersensitivity pneumonitis. Am J Respir Crit Care Med 2003; Danoff SK. Ascertainment of collagen vascular disease in patients
168: 952–58. presenting with interstitial lung disease. Respir Med 2009;
16 Silva CI, Muller NL, Lynch DA, et al. Chronic hypersensitivity 103: 1152–58.
pneumonitis: differentiation from idiopathic pulmonary fibrosis 23 Fischer A, West SG, Swigris JJ, Brown KK, du Bois RM.
and nonspecific interstitial pneumonia by using thin-section CT. Connective tissue disease-associated interstitial lung disease:
Radiology 2008; 246: 288–97. a call for clarification. Chest 2010; 138: 251–56.
17 Lacasse Y, Girard M, Cormier Y. Recent advances in hypersensitivity 24 Vij R, Noth I, Strek ME. Autoimmune-featured interstitial lung
pneumonitis. Chest 2012; 142: 208–17. disease: a distinct entity. Chest 2011; 140: 1292–99.
18 Churg A, Sin DD, Everett D, Brown K, Cool C. Pathologic patterns 25 Kinder BW, Collard HR, Koth L, et al. Idiopathic nonspecific
and survival in chronic hypersensitivity pneumonitis. interstitial pneumonia: lung manifestation of undifferentiated
Am J Surg Pathol 2009; 33: 1765–70. connective tissue disease? Am J Respir Crit Care Med 2007;
19 Silva CI, Churg A, Muller NL. Hypersensitivity pneumonitis: 176: 691–97.
spectrum of high-resolution CT and pathologic findings. 26 Fischer A, Antoniou KM, Brown KK, et al, ERS/ATS Task Force on
AJR Am J Roentgenol 2007; 188: 334–44. Undifferentiated Forms of CTD-ILD. An official European
20 Fischer A, du Bois R. Interstitial lung disease in connective tissue Respiratory Society/American Thoracic Society research statement:
disorders. Lancet 2012; 380: 689–98. interstitial pneumonia with autoimmune features. Eur Respir J
21 Tzelepis GE, Toya SP, Moutsopoulos HM. Occult connective tissue 2015; 46: 976–87.
diseases mimicking idiopathic interstitial pneumonias. Eur Respir J
2008; 31: 11–20.

www.thelancet.com/respiratory Vol 4 July 2016 565

You might also like