You are on page 1of 99

EUROPEAN ORGANISATION

FOR THE SAFETY OF AIR NAVIGATION

EUROCONTROL/CANSO
Standard of Excellence
2012 Measurement
Overall Status of ANSPs

Edition Number : 1.0


Edition Date : 16/11/2012
Status : Released Issue
Intended for : Restricted audience

EUROPEAN AIR TRAFFIC MANAGEMENT


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

DOCUMENT CHARACTERISTICS

TITLE

EUROCONTROL/CANSO Standard of
Excellence 2012 Measurement

Overall Status of ANSPs

Publications Reference:
ISBN Number:
Document Identifier Edition Number: 1.0
Edition Date: 16/11/2012
Abstract
This Document summarises the findings of the 2012 EUROCONTROL/CANSO Standard of
Excellence Measurement.

Keywords
ANSP Excellence Survey Findings
CANSO EUR Region EUROCONTROL

Authors

Contact(s) Person Tel Unit


Ivana Bušić +32 2 729 4632 DNM/COO/NOM/SAF

STATUS, AUDIENCE AND ACCESSIBILITY


Status Intended for Accessible via
Working Draft  General Public  Intranet 
Draft  EATM Stakeholders  Extranet 
Proposed Issue  Restricted Audience  Internet (www.eurocontrol.int) 
Released Issue  This document is restricted to the participating ANSPs,
EUROCONTROL Safety Team members and CANSO

Released Issue Edition: 1.0 ii


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

DOCUMENT APPROVAL
The following table identifies all management authorities who have successively approved
the present issue of this document.

AUTHORITY NAME AND SIGNATURE DATE

Senior Safety Expert 16-11-2012


Radu Cioponea

SoE Measurement Project Manager 16-11-2012


Ivana Bušić

Head of Safety Unit 16-11-2012


Antonio Licu

Head of Network Operations


16-11-2012
Management Division
Nicola Cooper

Chief Operating Officer DNM 16-11-2012


Joe Sultana

Director Network Management 16-11-2012


Jacques Dopagne

Safety Director NATS


Co-chairman CANSO Global Safety 16-11-2012
Standing Committee
Dr David Harrison

Executive General Manager


Safety & Assurance Business Group
Airservices Australia 16-11-2012
Chairman of CANSO Global Safety
Standing Committee Stephen Angus

Released Issue Edition: 1.0 iii


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

DOCUMENT CHANGE RECORD


The following table records the complete history of the successive editions of the present
document.

EDITION EDITION PAGES


REASON FOR CHANGE
NUMBER DATE AFFECTED

0.01 03/09/2012 First Draft ALL

0.1 11/10/2012 Proposed Issue ALL

1.0 16/11/2012 Released Issue ALL

COPYRIGHT NOTICE AND DISCLAIMER


© European Organisation for the Safety of Air Navigation (EUROCONTROL)
This document is published by the EUROCONTROL DNM/COO/NOM/Safety in the interest of improving ATM safety. This
document is restricted.

It may be copied in whole or in part providing that the copyright notice and disclaimer are included and permission is granted by
EUROCONTROL DNM/COO/NOM/Safety. The information contained in this document may not be modified without prior written
permission from the EUROCONTROL DNM/COO/NOM/Safety.

The views expressed herein do not necessarily reflect the official views or policy of EUROCONTROL, which makes no warranty,
either implied or express, for the information contained in this document, neither does it assume any legal liability or
responsibility for the accuracy, completeness or usefulness of this information.

Printer by EUROCONTROL, 96, rue de la Fusée, B-1130 Brussels, Belgium, Tel: + 32 2 729 3956, Fax: + 32 2 729 9108.

Released Issue Edition: 1.0 iv


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Contents
DOCUMENT CHARACTERISTICS.............................................................................ii

DOCUMENT APPROVAL ..........................................................................................iii

DOCUMENT CHANGE RECORD..............................................................................iv

Executive Summary ..................................................................................................1

1. Introduction.............................................................................................11
1.1 Background..............................................................................................................11
1.2 Survey Objectives....................................................................................................12
1.3 Study Areas .............................................................................................................13
1.4 Maturity Score and Maturity Level ...........................................................................13

2. Implementation of A Single european sky ...........................................15


2.1 Geographical Area...................................................................................................15
2.2 ECAC Regional Overview........................................................................................16
2.3 Survey Findings – ECAC States..............................................................................21

3. CANSO Survey Results..........................................................................53


3.1 Geographical Area...................................................................................................53
3.2 CANSO Objectives ..................................................................................................53
3.3 Survey Findings – CANSO Overview ......................................................................54
3.4 SURVEY Findings CANSO Members .....................................................................56

4. Conclusions ............................................................................................75

APPENDICES ...........................................................................................................77

Appendix 1 Reference Material .........................................................................78

Appendix 2 Methodology...................................................................................84

Appendix 3 Question Mappings ........................................................................88

Appendix 4 Performance Level Definitions......................................................89

Appendix 5 ANSP Participation.........................................................................90

Appendix 6 Glossary..........................................................................................92

Released Issue Edition: 1.0 v


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Executive Summary
Forty three of the main ANSPs1 from 44 States within the ECAC Region submitted
questionnaire responses in time for inclusion in this report, compared with only 40 States in
2011. San Marino was not included in the survey as that State is covered by Italy, but the
Maastricht Upper Air Control Centre (MUAC) is included, making 44 ANSPs covered from
the ECAC area.
This is the first year that CANSO members worldwide have been included in the survey
report; previously CANSO members have been included but only those within the ECAC
Area have been included in the survey report. Non-ECAC CANSO members were previously
surveyed on an individual basis, without performing a group analysis. Results for
participating CANSO members for the 2012 survey are reported on collectively in a separate
section of the survey report. The CANSO Section of the report includes the 38 ANSPs within
the ECAC Area who are CANSO Members plus 13 other members from outside of the ECAC
Area.
The survey followed the same methodology as in 2010 and 2011, looking into eleven Study
Areas, according to the Standard of Excellence (SoE) model, which is explained in Part 1
(Introduction) of this report. The questionnaire responses were followed up by face-to-face
interviews where all Study Areas were discussed in detail, or by telephone interviews which
focused only on the following Study Areas (in order to optimise the time for the telephone
interviews):
• Study Area 2 (Organisational and Individual Safety Responsibilities)
• Study Area 6 (Risk Management), and
• Study Area 8 (Safety Assurance).
In addition, some ANSPs provided updates during telephone interviews on Study Areas not
included in the three targeted areas. Information derived from those sources has also been
included in the overall and individual ANSP reports. Five ANSPs participated in in-depth
face-to-face interviews.
The data used for the analyses published in this report were accurate at the time of the
publication (16 November 2012). Any changes in the ANSPs’ data that may have occurred
after this date could not be included in the data sample. ANSPs participating in the survey
conducted by EASA under the framework of the SES legislation may arrive at different
values than those reported during this exercise. All potential changes will be taken into
account for the next year’s baseline.

ECAC
The average Maturity Score for each participating organisation within the ECAC region is
shown in Figure E-1. A return result of zero indicates that there was no return for that
particular year and not that the Maturity Score is zero.

1
In some instances there is only one ANSP in each State, but in others there are a number. This survey focuses on the main
ANSP in each State.
Released Issue Edition: 1.0 1
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

ANSP Average Maturity

100

90

80

70
Maturity (%)

60

50

40

30

2010 2011 2012


20

10

0
1 6 11 16 21 26 31 36 41

State Rank

Figure E-1: ECAC ANSP Average Maturity

The range of Maturity Scores, for ECAC participating States, seen in each Study Area is
presented below:

Mean
Study Area Minimum Maximum
2012
SA1 – Safety Culture 33.0 69.9 88.5
SA2 – Organisational and Individual Safety Responsibilities 32.1 71.2 90.4
SA3 – Timely Compliance with International Obligations 33.8 73.1 91.2
SA4 - Safety Achievement 33.7 71.6 89.0
SA5 - Competency 32.9 69.9 88.6
SA6 – Risk Management 33.8 70.3 87.8
SA7 – Safety Interfaces 30.6 70.0 89.7
SA8 – Safety Assurance 33.1 71.1 89.6
SA9 – Safety Performance Monitoring 33.2 70.6 88.3
SA10 – Organisational Safety Surveys and SMS Audits 32.8 69.9 87.6
SA11 – Adoption and Sharing of Best Practices 33.4 70.5 90.1
Table: E-1: Range of Maturity Scores
Table E-1 shows the maximum and minimum scores in each Study Area. Since 2011 there
has been a good increase at the lower end of scores, as well as some lowering of the scoring
at the top end, but the mean has stayed fairly static.
In a significant number of cases it appears that the lowering of the scores is due to an overall
better understanding of the survey requirements by the respondees, helped by the guidance
provided in the Questionnaire Clarification Document issued with the 2012 Questionnaire.

Released Issue Edition: 1.0 2


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Maturity Levels Distribution


2012

2% 7%

Level 1
40%
51% Level 2
Level 3
Level 4

Figure E-2: ECAC Maturity Levels Distribution


Figure E-2 shows the distribution by Maturity Levels for all participating ANSPs. This is an
improvement from last year, as further shown in Figure E-3 and the improvement continues
the trend from 2011 that goes in the direction of achieving the informal objective set by the
SAFREP TF to raise all ECAC ANSPs to at least Level 3 by the end of the SES First
Reference Period.
Level 5 is not shown in these graphs as no ANSP has yet achieved that Level.

Maturity Levels Distribution


100%

80%
Level 4
60%
Level 3
40% Level 2

20% Level 1

0%
2010 2011 2012

Figure E-3: ECAC Maturity Levels Distribution


Forty three of the 44 ECAC ANSPs invited to participate in the survey returned a
questionnaire. This represents a return rate of 98%. All ANSPs that responded by returning a
questionnaire were invited to undergo either a telephone or a face-to-face interview. The
majority agreed to participate in an interview or answer supplementary written questions2.
One of the ECAC ANSPs that submitted a questionnaire opted not to be interviewed.
The average Maturity Profile remains broadly static, although as in the 2011 survey individual
ANSPs have moved up and down relative to each other; the most significant changes being
in the 80% to 100% range of Maturity Scores. As can be seen, there is a small in increase in
the number of ANSPs at Level 3, and a small decrease in those at levels 1 and 2. Therefore,
despite the Questionnaire Clarification Document having an impact of reducing Maturity
2
A system of written supplementary questions was introduced this year to shorten the telephone interview process.
Released Issue Edition: 1.0 3
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Level responses in a number of areas, and ANSPs now having a better understanding of the
requirements, the overall average has improved slightly since the 2011 survey.
To go beyond Level 3 takes considerable application and will cost organisations in terms of
human and financial resources. Moreover, not all ANSPs may be able to reach the
“Continuous Improvement” level as they may not be able to afford the cost and resource
implications. While all ANSPs should strive for excellence3, it may not be possible for all
ANSPs to achieve and maintain levels beyond Level 3 in every single Study Area. It is not
realistic to expect that every ANSP will be developing “best practice” in every area.
Furthermore, “best practice” in one year may become common practice shortly afterwards.
What level is appropriate is therefore a decision that must be made by each individual ANSP
depending on a number of factors such as: resources, complexity of service provided, size of
the ANSP, responsibilities (e.g. in European SES States their position within a FAB) and
internal/external requirements.
The difficulties reported last year by some ANSPs with regards to sharing of data with FAB
partners were not reported this year. In most cases, this is due to further progress in
harmonising processes and procedures between partners.

Since the 2011 survey, the enforced UN-mandated no-fly zone over Libya has been lifted
meaning that ANSPs controlling the Mediterranean airspace have been able to return to
normal operations.

ANSPs’ Weakest Areas

ANSPs' Weakest Areas (Maturity Scores) In 2012


15 14
Number of ANSPs

10 9

5
3 3 3
2
1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area

Figure: E-4: Weakest Areas – Maturity Scores


As can be seen from Figure E-4, Study Area 7: Safety Interfaces, is the weakest Study Area
experienced by most ANSPs when comparing overall Maturity Scores. However, when
comparing data against the proposed aim to reach Level 3 for all Study Areas, a different
picture emerges. As can be seen in Figure E-5, there are a number of ANSPs who are at
Level 1 or Level 2 within other Study Areas.

3
CEOs of the ANSPs should strive to go beyond the minimum level of competency and push their organisation to beyond the
level of Implementing and achieve the level of Managing and Measuring […] and even the level of Continuous Improvement
[…] ATM Safety Framework Maturity Survey Methodology, 31st August 2009
Released Issue Edition: 1.0 4
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

12

10
Number of ANSPs
8

6
Level 1
4
Level 2
2

Study Area

Figure E-5: Weakest Study Areas: Levels 1 and 2


Figure E-5 shows those Study Areas where ANSPs are not yet meeting the objectives. The
reasons for this are more fully explained in the relevant areas in this report for the respective
Study Areas. However, significantly:
• Study Area 1: Safety Culture. A positive safety culture is emerging within ANSP
organisations, though in some ANSPs it is still immature, with the majority of ANSPs
being at Level 3 or 4. However none have yet reached Level 5 and there are still
eleven below Level 3 with some yet to commence the journey into establishing a
positive and proactive Safety Culture. Most ANSPs now recognise the importance of
Safety Culture and the need to measure it. They now understand that they need to
undergo Safety Culture surveys to establish where they are in developing a positive
and proactive safety culture in their organisation. A significant number of ANSPs have
chosen to undertake Safety Culture Surveys through EUROCONTROL and more are
planned before the end of 2012. It is therefore anticipated that the number of ANSPs
still below Level 3 in this Study Area will reduce further in the next year.
• Study Area 9: Safety Performance Monitoring. There has been some regression in
this Study Area with a few ANSPs at Level 3 in 2011 reassessing their performance
and marking themselves at Level 2 this year. Most ANSPs had some form of
monitoring in place, but the majority do not have a Quantitative system, nor do they
undertake internal comparative analysis. While this is not always the case, a low
level in this Study Area should have an immediate impact on most other areas, as
reaching Levels 4/5 require quantitative measurements and comparative analyses.
Many ANSPs do not see the need to routinely share safety data with the general
public, though a lessening of the resistance to do this was detected. There are still
twelve ANSPs below Level 3 and more work is required to emphasise the importance
of having a quantitative monitoring system in place.
• Study Area 11: Adoption and Sharing of Best Practices. There has been little
movement in this area in 2012. It is clear that there is misunderstanding of what
constitutes “Best Practice” and how it should be assessed in the context of this Study
Area. There are eight ANSPs shown below Level 3. It is believed that there should
be more ANSPs below that Level as some refused to mark themselves below Level 4
even though they could not show that they were using best practice on the basis that
they did not believe that it was possible to develop best practices in some of the study
areas. Greater explanation of the survey requirements in this area is required.
Released Issue Edition: 1.0 5
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Similarly to the previous paragraph, not being able to show the use of best practice
for any particular study objective should prevent an ANSP from scoring above Level
3, which means that an overall low score in SA11 should be reflected in general
levels that do not often exceed Level 3.

ANSPs Strongest Areas

ANSPs' Strongest Areas In 2012


25
22

20
Number of ANSPs

15

10
7
6
5 4
2
1 1
0 0 0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area

Figure: E-6: Strongest Areas - Maturity Scores

Overall, the data in Figure E-6 shows that Study Area 3: Timely Compliance with
International Obligations is the strongest Study Area experienced by ANSPs. It should be
noted, however, that this relates to the highest Maturity Score across all Study Areas.
When comparing data against the requirement to reach Level 3 for all Study Areas, a
different picture emerges. As can be seen in Figure E-7 there are a number of ANSPs who
are at Level 4 or Level 5 in other Study Areas.

25

20
Number of ANSPs

15

Level 4
10
Level 5

0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area

Figure: E-7: Strongest Areas – Levels 4 and 5


For instance, it can be seen that Study Area 5 is particularly strong in terms of those ANSPs
at Levels 4 and 5. This is because the Justification and Evidence submitted with the

Released Issue Edition: 1.0 6


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

questionnaires indicates that most organisations in this Study Area have well developed
training and competency processes in place.
Study Area 6 is also strong in terms of ANSPs at Levels 4 and 5 due to many ANSPs already
having risk management plans implemented or have a plan in place to implement a risk
management system.

GLOBAL CANSO
The average Maturity Score for participating CANSO Members are shown in Figure E-8. A
return result of zero indicates that there was no return for that particular year and not that the
Maturity Score is zero.

ANSP Average Maturity

100

90

80

70

60
Maturity (%)

50

40

30
2010 2011 2012
20

10

0
1 6 11 16 21 26 31 36 41 46 51

State Rank

Figure E-8: CANSO Average Maturity Score


It can also be seen that the profile in this graph is quite different from the similar graph for the
ECAC only area (Figure E-1). This is due to some non-ECAC CANSO Members having very
low maturity scores.
The range of Maturity Scores, for CANSO participating States, seen in each Study Area are
presented below in Table E-2.
Mean
Study Area Minimum Maximum
2012
SA1 – Safety Culture 3.6 64.7 98.1
SA2 – Organisational and Individual Safety Responsibilities 4.1 65.9 98.6
SA3 – Timely Compliance with International Obligations 4.4 67.8 99.0
SA4 – Safety Achievement 3.3 66.5 98.3
SA5 – Competency 3.8 64.8 98.1
SA6 – Risk Management 4.0 65.3 98.2
SA7 – Safety Interfaces 2.4 65.2 98.4
SA8 – Safety Assurance 3.9 66.2 97.5
SA9 – Safety Performance Monitoring 4.1 65.7 97.6
SA10 – Organisational Safety Surveys and SMS Audits 3.7 65.0 97.4
SA11 – Adoption and Sharing of Best Practices 4.1 65.3 98.0
Table E-2: Range of Maturity Scores for CANSO Members

Released Issue Edition: 1.0 7


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

The number of non-ECAC CANSO Members that participated in the survey, but were not
included in the ECAC Report, has tended to change year on year4. In 2010, 4 states
participated; 2011: 8 states; 2012: 13 states. Over the three years, only 4 states participated
in each year. As a result, trends are more difficult to detect. Currently the mean average
within the grouping for 2012 is 65.66%. The maximum maturity is an overall 98.1%, which is
higher than the ECAC area maximum due to additional ANSPs with very mature SMS being
included. The minimum average is 3.76%, which is much lower than the similar figure for the
ECAC area. This is due some participating CANSO ANSPs outside of the ECAC area being
at the very early stages of SMS implementation.
When including the CANSO Members outside of the ECAC area 505 organisations
participated in this part of the survey. Not all of these additional ANSPs participated in the
follow-up interview process, but those that did (11 ANSPs) gave very detailed responses to
interview questions.

Maturity Levels Distribution


2012

6%
20% Level 1
42%
32% Level 2
Level 3
Level 4

Figure E-9: CANSO Average Maturity

Figure E-9 shows the distribution by levels for all CANSO ANSPs participating in the 2012
survey, with 42% of ANSPs reaching Maturity Level 3. Because the thirteen additional
CANSO ANSPs lie outside of the ECAC Area and the majority of those are only just
beginning to initiate implementation of an SMS they are still at Levels 1 and 2. CANSO has
the same proposed objective of raising all their ANSP Members to Level 3 as soon as
feasible.

4
Although 4 non-EUR CANSO ANSPs participated in the 2010 survey and 8 participated in the 2011 survey, the report for
those years did not include the findings of these entities.
5
One ECAC ANSP that is also a member of CANSO did not return a questionnaire in 2012 and therefore could not be included
in this report.

Released Issue Edition: 1.0 8


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2012 Maturity Levels Distribution


100%

80%
Level 4
60%
Level 3
40% Level 2
Level 1
20%

0%
2012

Figure E-10: CANSO Maturity Levels Distribution

Figures E-10 indicates the distribution of Maturity Levels of CANSO members within the
2012 survey. 24 of the participating CANSO ANSPs are at Level 3 or above.

Released Issue Edition: 1.0 9


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Intentionally Blank

Released Issue Edition: 1.0 10


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

1. INTRODUCTION

1.1 Background

An agreement has been reached between CANSO and EUROCONTROL whereby


EUROCONTROL provides full support for the safety performance measurements of its
member ANSPs, in particular the measurements based on the Standard of Excellence
(SoE) methodology jointly developed by EUROCONTROL and CANSO through 2007-2009.
In addition the Network Manager will rely on this methodology and measurement to gather
safety knowledge from ANSPs and to avoid that a separate dataflow on leading indicators is
established. This methodology was used in 2010 for the first time and furthermore was
chosen by the EU as the basis for the Effectiveness of Safety Management, which is one of
its three safety Key Performance Indicators, to be used within the framework of the Single
European Sky.
The EUROCONTROL/CANSO Standard of Excellence Measurements are expected to
contribute to establishing a global level for safety, based on the highest standards, as used
in places such as Europe, Northern America, Asia Pacific. The survey will provide vital
information for the European Safety Programme, as well as the required information for
CANSO and its ANSPs in order to collectively maintain and improve their Safety
Management Systems (SMS).
For the 2012 survey a number of non-ECAC CANSO members have therefore been included
in the measurement6. This report is divided into two parts, the ECAC area, which includes 38
CANSO members; and CANSO, which includes an additional 13 ANSPs from around the
world. The ECAC area covers those ANSPs involved with the Single European Sky.
Some focal points requested written questions so, to ensure that ANSPs were comfortable
with the follow-up telephone interviews and were prepared for the answers that they wished
to provide, a system of written “supplementary questions” was used in this year’s survey. The
supplementary questions were sent out in advance of the telephone interview.
Single European Sky. Within the Single European Sky States Commission Regulation (EU)
691/2010 [See Appendix 1] on the performance scheme for air navigation services and
network functions implements Article 11 of the Framework regulation (EC 549/2004) [See
Appendix 1]. It also defines the Key Performance Indicators (KPIs) for the four key
performance areas: capacity, cost-efficiency, environment and safety. For the safety
performance assessment the Regulation defines three safety KPIs:
a) The first safety KPI shall be the effectiveness of safety management for air navigation
services providers and national supervisory authorities respectively, as measured by
a methodology based on the ATM Safety Framework Maturity Survey.
b) The second national/FAB safety KPI for the first reference period shall be the
application of the severity classification below based on the Risk Analysis Tool
methodology to the reporting of, as a minimum, three categories of occurrences:
Separation Minima Infringements, Runway Incursions and ATM-specific occurrences
at all Air Traffic Control Centres and at airports. Member States may decide not to
apply the method at airports with less than 50 000 commercial air transport
movements per year.
c) The third European Union-wide safety key performance indicator shall be reporting of
the just culture.
The Regulation does not require targets to be established for these safety KPIs for the first
reference period (2012-2014). During this time the Commission will use the data collected to

6
Although 4 non-EUR CANSO ANSPs participated in the 2010 survey and 8 participated in the 2011 survey, the report for
those years did not include the findings of these entities.
Released Issue Edition: 1.0 11
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

validate these KPIs and assess them to ensure that safety risk is adequately identified,
mitigated and managed. On this basis, the Commission shall adopt new safety KPIs for the
second reference period if necessary. Moreover, it is the intention of the Commission to use
the data collected during the first reference period to establish the performance targets for
the following reference period.
The first safety performance indicator required in the Regulation, an ATM Safety Framework
Maturity Survey, has been developed by EUROCONTROL [See Appendix 1] in collaboration
with CANSO. It is a natural extension to similar assessments which date back to 2002.
This report summarises the findings of a EUROCONTROL/CANSO Standard of Excellence
(SoE) Measurement conducted during the second half of 2012 and builds on the data
collected in the 2010 and the 2011 EUROCONTROL Safety Framework Maturity Surveys.
The data used for the analyses published in this report were accurate at the time of the
publication (16 November 2012). Any changes in the ANSPs’ data that may have occurred
after this date could not be included in the data sample. ANSPs participating in the survey
conducted by EASA under the framework of the SES legislation may arrive at different
values than those reported during this exercise. All potential changes will be taken into
account for the next year’s baseline.
Non-SES ANSPs. For non-SES States the ICAO Safety Management Manual (Doc 9859
AN/474) provides guidance on how to develop and implement a Safety Management System
(SMS).

1.2 Survey Objectives


The EUROCONTROL/CANSO Standard of Excellence Measurement establishes the extent
of progress made by ANSPs with respect to the introduction of ATM safety management
systems and how the SMS framework relates to safety in operations and engineering. It
seeks to:
• Determine the level of SMS implementation within the industry;
• Determine the extent to which learning is transferred across the industry;
• Establish a path along which ANSPs can focus their activities for continuous
improvement.
The Standard of Excellence model against which ANSPs are measured supports the clear
message promoted by the ICAO Safety Management Manual that achievement of the
highest level of SMS maturity is a long term process that must proceed in a very deliberate
step-wise manner.
The Standard of Excellence is a performance measurement, not a compliance measurement
and consists of a system enabler (Safety Culture) and a framework of four components and
10 elements. The structure is presented in Figure 1-1, below:

Released Issue Edition: 1.0 12


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Figure 1-1: SMS Excellence Model

1.3 Study Areas


The maturity of an ANSP’s safety management system is assessed in eleven separate Study
Areas, which are themselves divided into a total of 26 Study Objectives.

Study Areas
SA1 – Safety Culture
SA2 – Organisational and Individual Safety Responsibilities
SA3 – Timely Compliance with International Obligations
SA4 – Safety Achievement
SA5 – Competency
SA6 – Risk Management
SA7 – Safety Interfaces
SA8 – Safety Assurance
SA9 – Safety Performance Monitoring
SA10 – Organisational Safety Surveys and SMS Audits
SA11 – Adoption and Sharing of Best Practices
Table 1-1: Study Areas

1.4 Maturity Score and Maturity Level


ANSPs are asked to assess the maturity of their safety management system against each
Study Objective. They do so by selecting one of five possible responses (A – E) where:
• A is defined as ‘Initiating’ – equivalent to a Level 1;
• B is defined as ‘Planning/Initial Implementation’ – equivalent to a Level 2;
Released Issue Edition: 1.0 13
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

• C is defined as ‘Implementing’ – equivalent to a Level 3;


• D is defined as ‘Managing & Measuring’ – equivalent to a Level 4; and
• E is defined as ‘Continuous Improvement’ – equivalent to a Level 5. (See Appendix 2:
Methodology and Appendix 3: Question Mappings.
Not all ANSPs may be able to reach the “Continuous Improvement” level as they may not be
able to afford the cost and resource implications. While all ANSPs should strive for
excellence7, it may not be possible for all ANSPs to achieve and maintain levels beyond
Level 3 in every single Study Area. It is not realistic to expect that every ANSP to be
developing “best practice” in every area. Furthermore, “best practice” in one year may
become common practice shortly afterwards. What level is appropriate is therefore a
decision that must be made by each individual ANSP depending on a number of factors such
as: resources, complexity of service provided, the size of the ANSP, responsibilities (e.g. in
European SES States their position within a FAB) and internal/external requirements.
By combining the Maturity Score assigned by organisations against each Study Objective
with the Study Objective Mappings detailed in Appendix 3, a Maturity Score is derived. An
overall Maturity Score (the average for score over all Study Objectives) is also calculated.
Note that a Maturity Level (Level 1 – 5) is also assigned to each Study Area. This is defined
as the lowest response (A – E) provided by an ANSP for each Study Objective in a Study
Area. An overall Maturity Level for the organisation is similarly defined as the lowest
response to any Study Objective.
Study Area Performance Level Categories are shown at Appendix 4.
To summarise:
• (A – E) is the possible response on the questionnaire;
• Maturity Level (1 – 5) is equivalent to the lowest response (A – E);
• Maturity Score (0% - 100%) is derived from the Maturity Level and Study Objective
Question Mappings shown at Appendix 3.
• Performance Level Categories are shown at Appendix 4.

7
CEOs of the ANSPs should strive to go beyond the minimum level of competency and push their organisation to beyond the
level of Impelemnting andhieve the level of Managing and Measuring […] and even the level of Continuous Improvement […]
ATM Safety Framework Maturity Survey Methodology, 31st August 2009
Released Issue Edition: 1.0 14
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2. IMPLEMENTATION OF A SINGLE EUROPEAN


SKY

2.1 Geographical Area


Within Europe there are several groupings of States that are involved in improving European
Aviation Safety and some countries belong to more than one grouping. The organisation that
covers the most States is the European Civil Aviation Conference (ECAC) in which there are
several groupings such as the European Common Aviation Area (ECAA), which commits all
its partners to continue harmonising with EU legislation. This should result in equal high
standards in term of safety and ATM rules as well as fair competition across Europe.
EUROCONTROL has been tasked by ECAC with implementing its Strategy and in doing this
EUROCONTROL works closely with the European Commission (EC) and the European
Aviation Safety Agency (EASA) to implement a Single European Sky (SES).
In addition to EU Member States and the other State groupings mentioned above bilateral
agreements have been signed by the EC with Switzerland (1999), Morocco (2006) Georgia
(2010) and Jordan (2010), and although they operate under different legal frameworks and
the transposition and implementation within the SES (acquis) countries is done separately,
no distinction is made within this report.
Maastricht Upper Area Control (MUAC) is a provider of ATC services on behalf of four EU
Member States, namely Netherlands, Belgium, Luxembourg and Germany and is treated as
a “state” within the context of this report.
There are 44 States8, including MUAC, participating directly in the implementation of the SES
or including achievement of a Single European Sky within national plans. In December 2011
ECAC’s Directors General of Civil Aviation endorsed a policy statement: “ECAC’s Strategy
for the Future” that includes Aviation Safety, the statement says in respect of Safety:
“ECAC will focus on following up the safety decisions taken at the 2010 ICAO Assembly and
the High Level Safety Conference earlier that year, in particular working on a pan-European
basis with the European Commission and EASA to:
a) assist ECAC Member States in developing a harmonised European approach to specific
safety-related issues, including in particular in the development of State Safety Plans and
in the identification of the most serious risks facing its different Member States;
b) help promote across Europe a safety culture of continuous improvement and shared best
practice, working with ECAC Member States, with the European Commission and EASA,
and with the ICAO EUR-NAT Regional Office.
European expertise in international airspace matters rests with EUROCONTROL. ECAC will
continue to play a supporting role in this field, working in concert with EUROCONTROL and
the EC to promote European positions in ICAO and other international aviation fora, and to
consolidate a pan-European approach within ECAC Member States. A common approach to
SESAR and NextGen interoperability standards, common airspace policies benefiting all
ECAC Member States, and the achievement of the Single European Sky goals on a pan-
European basis are the airspace objectives which ECAC will seek to help deliver.”
A breakdown of ECAC and the various groupings it includes is at Figure 2.1.

8
ECAC has 44 members, but San Marino, although a member of ECAC, does not have its own ATM system and therefore does
not participate in SES implementation.
Released Issue Edition: 1.0 15
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Figure 2-1: Geographical scope of the ECAC Member States

Key: The EU (darkest blue + darkest red), ECAA (darkest blue + medium blue + medium
red), Eurocontrol (all blue), ECAC (all non-gray)
ECAC, EUROCONTROL, ECAA, EU
ECAC, EUROCONTROL, ECAA
ECAC, EUROCONTROL
ECAC, ECAA, EU
ECAC, ECAA
ECAC Only

2.2 ECAC Regional Overview


A total of 44 ANSPs from the ECAC Regional States, including the Maastricht Upper Air
Control Centre, were invited to participate in the survey. In some instances there is only one
ANSP within the State; in others there will be a number. This survey focuses on the main
ANSP in each State.
Of the 44 ANSPs invited, 43 responded by completing and returning the questionnaire
(compared with 40 in 2011).
Figure 2-2 shows that the profile of average maturity across all Study Areas in 2012 is
broadly similar to that in 2011 (a return result of zero indicates that there was no return for
that particular year and not that the Maturity Score is zero). There is, however, clearly a
reduction in the 80-100%, range. This is thought to be caused by ANSPs taking a more
realistic position in selecting Maturity Level requirements and to be in part due to the
guidance provided by the Questionnaire Clarification Document issued with the
Questionnaire this year and the need to follow SES legislation.

Released Issue Edition: 1.0 16


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

ANSP Average Maturity

100

90

80

70

60
Maturity (%)

50

40

30
2010 2011 2012
20

10

0
1 6 11 16 21 26 31 36 41

State Rank

Figure 2-2: Average Maturity


By comparing the Maturity Level (i.e. the lowest response to any Study Objective) we can
see in Figure 2-3 (Profile of Maturity Levels) that progress is being made. In 2010 more
ANSPs were at Level 2 than any other category and in 2011 Level 3 was the dominant Level;
this year Level 3 is again the dominant category having improved from the 2011 figure (with
an increase of 1.3% in ANSPs being at this Level). For such a change to have taken place,
ANSPs have to be able to confirm that their safety management systems have satisfied all of
the requirements from the previous level. They are moving from the point where they are
defined as “Implementing” to one where they are “Measuring and Monitoring” performance.
There is however still reluctance on the part of some ANSPs to accept that all elements of a
Study Area must be completed before the next higher category can be selected. At least two
ANSPs interviewees would not accept a lower grade as they considered that it was
impossible to set “Best Practice” in some Study Areas, such as Study Area 2. However, this
was not borne out by other ANSPs who said that they had developed best practice in internal
communication and in developing training packages.

25

20
Number of ANSPs

15

10

2010 2011 2012


5

0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-3: Profile of Maturity Levels


Results must be looked at both in terms of overall score and in terms of level. The former will
give a high-level picture about the general status of the organisation’s SMS; this allows a
direct comparison with other organisations on the scale. It shows whether, overall, the

Released Issue Edition: 1.0 17


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

organisation is mostly managing performance or it is still in the process of implementing the


mandatory regulations and achieving the proposed minimum level of maturity. However, this
could hide particular problems in certain areas as the methodology averages the scores by
Study Area and there is a significant smoothing effect. In order to identify whether an
organisation still has a significant problem in at least one area, the level view is more
appropriate. By combining the two, a more complete picture can be built. Thus, an
organisation with a high Maturity Level and high Maturity Score would show a consistent
approach to all objectives, while an organisation with a relatively high score but a low level
(see ANSPs No 11, No 39, No 40 and No 43 in Figure 2-4) would prove that it may have left
behind certain objectives, concentrating on others.

ANSP Average Maturity

100 5

90 Score Level

80 4

70

60 3
Maturity (%)

50

40 2

30

20 1

10

0 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43

ANSP

Figure 2-4: Level-Score for all participating ECAC ANSPs


In Figure 2-2: Average Maturity, ANSPs are ranked according to their average Maturity Score
in each year so it is not possible to see the progress that is being made by individual
organisations. Figure 2-5 ranks organisations according to their Maturity Score in 2012, and
shows the corresponding Maturity Score in 2010 and 2011 so that the changes for individual
organisations are evident. Note: ANSPs with a score of Zero for any year indicates a nil
return for that year.

ANSP Average Maturity

100

90

80

70
Maturity (%)

60

50

40

30
2010 2011 2012
20

10

0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

State

Figure 2-5: Comparison of Maturity Score in 2012 vs 2011 and 2010 for each ANSP

Released Issue Edition: 1.0 18


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Since the 2011 survey, 22 ANSPs increased their overall Maturity Score, with the best
performing ANSP reporting an 18.89% point increase, and another achieving a 17.43% point
increase. 17 ANSPs fell back against their 2011 Maturity Score, with the lowest two being by
14.72% points and 14.99% points respectively. This is due to a better understanding of what
is required to meet each Maturity Level, especially Levels 4 and 5. In a large majority of
cases the Maturity Score change is of the order of ±3 or 4% points.
Figure 2-6 presents an alternative view of how the Maturity Scores of individual ANSPs have
changed. In this case ANSPs are ordered according to the ratio of Maturity Scores; the 2012
Score divided by the 2011 Score. In the centre of the diagram, a group of ANSPs show either
modest or no change (ratios close to one), whilst the extremes indicate ANSPs where
significant improvements (or decreases) can be found.

Maturity Score Comparisons


1.8

1.6 Maturity Improved


Ratio of Maturity (2012/2011)

1.4 A Ratio of 1
indicates No Change
1.2

1.0

0.8 Maturity Decreased


0.6
2011/2012 Comparison
0.4
2010/2012 Comparison
0.2 No Data for 2010/2011
0.0
0 5 10 15 20 25 30 35 40 45
State

Figure 2-6: Ratio of Maturity Scores (2012/2011 and 2012/2010)

As in 2011, some ANSPs in this survey were still becoming familiar with the requirements of
the revised question set. As such, the largest decrease is unlikely to reflect any real changes
within the ANSP, it is rather the result of a re-evaluation of the response provided in 2010
and 2011. In addition, for the 2012 survey, guidance was provided in the form of a
“Questionnaire Clarification Document” which was sent out with the questionnaire. This
enabled ANSPs to have a better understanding of the requirements for each Maturity Level
and critically review their assessment.
The largest increase stems from a number of organisational improvements in areas such as
culture, reporting and investigation and interfaces, which have significantly improved that
organisation’s response to a number of Study Objectives. For example, the increasing
number of ANSPs undertaking safety culture surveys is having a direct impact on Study Area
1. Improvements being made to the reporting culture and the sharing of lessons learned is
having a direct effect on marking in Study Areas 1, 2 and 8, where improvements in reporting
have an immediate positive impact on the safety culture and move the implementation of the
SMS forward.

Released Issue Edition: 1.0 19


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Reduced
Increased Score, 17, 38%
Score, 21, 48%

No Change, 6,
14%

Figure 2-7: Changes in Overall Maturity Scores (2012/2011)


Figure 2-7 clearly shows the number of Maturity Score increases/decreases/no change
compared between 2012 and 2011. Note, however, that where an ANSP did not participate
in the 2011 survey, the score for 2012 has been compared with that resulting from the 2010
survey.

Reduced Score,
Increased Score, 18, 41%
24, 55%

No Change, 2,
4%

Figure 2-8: Changes in Average Maturity Scores (2012/2010)


Figure 2-8 shows the number of Maturity Score increases/decreases/no change compared
between the 2012 and 2010 surveys. Where an ANSP did not participate in the 2010 survey,
the score for 2012 has been compared with that resulting from the 2011 survey.
Overall, since the 2010 survey a number of ANSPs now have a greater understanding of the
maturity level requirements, which has meant that improvement in reaching higher levels of
maturity has been fairly conservative and, in a number of cases, maturity scores have
decreased. It should be noted that a few ANSPs found it difficult to justify some of the
responses they had provided but were reluctant to reduce their scores as they did not want
to be seen to be worse than the previous year and seemed not to understand that,
particularly at Levels 4 and 5, things can change from year to year.
Table 2-1 shows the number of times ANSPs have selected a certain Maturity Score for
questions in each of the Study Areas. As can be seen, Maturity Score A and B are more
prominent in Study Areas 9 (Safety Performance Monitoring), where ANSPs answered 19
times a category A or B, which represents 15% of all answers in this Study Area. Study
Area 6 (Risk Management) received a total of 6 responses in category B, which represents
14% of all responses; Study Area 11 (Adoption and Sharing of Best Practices), received a
total of 17 responses in category A and B, which represents 13% of all received responses.

Released Issue Edition: 1.0 20


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Study Areas
Answers SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
A 0 0 0 0 0 0 1 0 3 0 2
B 13 10 5 4 2 6 5 3 16 4 15
C 47 43 17 20 5 4 19 40 39 12 32
D 61 69 37 55 22 21 45 41 55 20 57
E 8 50 27 50 14 12 16 45 16 7 23
A&B Total 13 10 5 4 2 6 6 3 19 4 17
A&B Total
% 10% 6% 6% 3% 5% 14% 7% 2% 15% 9% 13%
Table 2-1: Questionnaire Responses by Study Area
The global economic downturn continues to affect some organisations. As last year some of
those interviewed reported that plans that had been discussed previously (e.g. the conduct of
Safety Culture Surveys, and revisions to the Safety Manual) have not yet been implemented.
However, that having been said, there were signs that plans are now being implemented and
there is some recruitment in support areas. Some ANSPs commented that there had not
been sufficient time between surveys for any real progress to be made. Six (including one
ANSP who did not respond to the 2012 questionnaire) ANSPs recorded no change since the
2011 survey, with a number of other ANSPs recording change in only a few Study Areas.
In 2011 some ANSPs had a change of opinion regarding the introduction of FABs with
negative opinions surfacing. This year no strong views were expressed, but some of the
smaller ANSPs stated that they were getting help from FAB partners, particularly in the form
of developing procedures and best practice. The reluctance of some commercially orientated
ANSPs to “give away” what they see as their intellectual property was not apparent this year
and FAB partners appear to be working well together, though in some cases it was clear that
support was provided on a commercial basis.
Since the 2011 survey, the enforced UN-mandated no-fly zone over Libya has been lifted
meaning that ANSPs controlling the Mediterranean airspace have been able to return to
normal operations.

2.3 Survey Findings – ECAC States 9

The range of Maturity Scores in each Study Area varies from 30.6% to 91.2%, as shown in
Figure 2-9. The interquartile range is typically around 10%, indicating that ANSPs are
actually closely grouped in their assessment of Safety Maturity.

9
This includes all ECAC Members that are also Members of CANSO and MUAC.

Released Issue Edition: 1.0 21


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Range of Maturity

100

90 90.4 91.2 89.7 89.6 90.1


88.5 89.0 88.6 87.8 88.3 87.6
80
74.5 76.0 75.0
72.5 72.8 72.9 72.6 73.3 72.8 72.7 73.5
70
64.7 66.2
63.2 63.8 63.3 63.3 63.8 62.8 62.6 64.0
60 61.7
Maturity (%)

50

40
33.0 32.1 33.8 33.7 32.9 33.8 33.1 33.2 32.8 33.4
30 30.6

20

10

0
1 2 3 4 5 6 7 8 9 10 11

Study Area

Figure 2-9: Range of Maturity


The range of Maturity Scores, for ECAC participating States, seen in each Study Area is
presented below in Table 2-2:

2012 2011 Difference


Study Area Min Mean Max Min Mean Max Min Max Mean
SA1 – Safety Culture 33.0 69.9 88.5 23.9 69.9 91.8 9.1 -3.3 -0.1
SA2 – Organisational and Individual
Safety Responsibilities 32.1 71.2 90.4 24.5 71.3 94.0 7.7 -3.6 0.0
SA3 – Timely Compliance with
International Obligations 33.8 73.1 91.2 24.5 73.4 97.1 9.3 -5.9 -0.3
SA4 – Safety Achievement 33.7 71.6 89.0 24.7 71.8 93.0 9.0 -4.0 -0.2
SA5 – Competency 32.9 69.9 88.6 24.2 70.0 91.8 8.7 -3.3 -0.1
SA6 – Risk Management 33.8 70.3 87.8 23.8 70.4 91.8 10.1 -4.0 -0.1
SA7 – Safety Interfaces 30.6 70.0 89.7 20.2 69.9 92.9 10.3 -3.2 0.1
SA8 – Safety Assurance 33.1 71.1 89.6 22.8 71.1 92.4 10.4 -2.8 0.0
SA9 – Safety Performance
Monitoring 33.2 70.6 88.3 22.6 70.7 91.3 10.6 -3.0 0.0
SA10 – Organisational Safety
Surveys and SMS Audits 32.8 69.9 87.6 24.4 70.1 91.1 8.3 -3.4 -0.2
SA11 – Adoption and Sharing of
Best Practices 33.4 70.5 90.1 23.9 69.9 91.8 9.5 -1.6 0.6
Table 2-2: Difference Comparison 2011 to 2012

It can be seen from Table 2-2 that in all Study Areas there has been a good increase in the
minimum scores submitted whilst there have been small reductions in the maximum scores
in most of the Study Areas. However, the average scores have remained fairly static, with
only a small drop in the Mean Score, showing that the improvements at the lower end are
more or less cancelled out by the small drop in scores at the higher end. It is believed that
the lowering of some scores at the top end is caused by ANSPs taking a more realistic view
of what they have achieved, which has come about by better explanation of the requirements
for each Study Area coupled with the fact that are now set out in SES regulations and are
now national requirements.

Released Issue Edition: 1.0 22


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Mean Mean
Study Area Min Max Difference
2011 2010
SA1 – Safety Culture 23.9 69.9 91.8 68.19 1.7
SA2 – Organisational and Individual Safety Responsibilities 24.5 71.3 94.0 69.64 1.6
SA3 – Timely Compliance with International Obligations 24.5 73.4 97.1 72.39 1.0
SA4 - Safety Achievement 24.7 71.8 93.0 69.85 2.0
SA5 - Competency 24.2 70.0 91.8 68.29 1.7
SA6 – Risk Management 23.8 70.4 91.8 68.70 1.7
SA7 – Safety Interfaces 20.2 69.9 92.9 68.13 1.8
SA8 – Safety Assurance 22.8 71.1 92.4 69.30 1.8
SA9 – Safety Performance Monitoring 22.6 70.7 91.3 68.69 2.0
SA10 – Organisational Safety Surveys and SMS Audits 24.4 70.1 91.1 68.26 1.9
SA11 – Adoption and Sharing of Best Practices 23.9 69.9 91.8 68.92 1.0
Table 2-3: Minimum to Maximum Scores for 2012 & Mean Comparison between 2012 and 2010

Table 2-3 shows the Minimum and Maximum scores in 2012 for each Study Area and
demonstrates that there has been an overall steady improvement since 2010 despite there
being some reductions in scores in the individual Study Area scores.

ANSP Response Category by Study Objective in 2012


100%

90%

80%
Percentage of Responses

70%

60%

50%

40%

30%

20%

10%

0%
SA10.1

SA11.1

SA11.2

SA11.3
SA1.1

SA1.2

SA1.3

SA2.1

SA2.2

SA2.3

SA2.4

SA3.1

SA3.2

SA4.1

SA4.2

SA4.3

SA5.1

SA6.1

SA7.1

SA7.2

SA8.1

SA8.2

SA8.3

SA9.1

SA9.2

SA9.3

Study Objectives A B C D E

Figure 2-10: Percentage of Answers in Each Maturity Category

As illustrated in Figure 2-10, 26% of ANSPs reported a Maturity Level of 1 or 2 in response to


Study Area 9.3 (A general public knowledge of the ANSP’s performance through routine
publication of achieved safety levels and trends). Some ANSPs are still reluctant to make
safety data available to the general public. In addition, 26% of ANSPs reported a Maturity
Level 2 for Study Area 1.2 (Regular measurement of safety culture and an improvement
programme). Although a number of ANSPs have completed a Safety Culture Survey (with 3
ANSPs currently embarking on their third survey, there are still some ANSPs who have not
yet started this journey. Conversely in Study Area 8 (Safety Reporting, Investigation and
Improvement), there is only one ANSP in each Study objective area with a response at
Maturity Level 2. With regard to Study Area 8.2 (An organisation-wide means to record and
disseminate lessons learned) it is clear that the importance of lessons learned is recognised.
However, during the interviews a number of ANSPs, whilst having a process through which
lessons learned are disseminated, there is some lack of establishing the extent to which
these lessons are indeed learned. In addition, some ANSPs have no formal processes in
place for ensuring lesson learning across the organisation.

Released Issue Edition: 1.0 23


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Gap to Next Maturity Level


18

16

14

Number at Minimum Level


12

10

0
33.42

37.81

44.37

59.97

60.83

63.12

65.87

67.75 4
69.48

70.78

72.96
3

74.24

76.66

77.34

80.02

81.03
2

82.24
Maturity

82.88

84.56

86.66
1

87.09

88.97
Level

Figure 2-11: Gaps to the next maturity level (ECAC ANSPs)


Forty three of the 44 main ANSPs in the ECAC Area that were invited to participate in the
survey, plus the Maastricht Upper Air Control Centre, returned a questionnaire. Of those all
except one were also interviewed.
Overall, organisations responded to the Study Objectives with a Maturity Level of 1 or 2
eighty nine times (7.96% of responses), compared with 278 times (24.87% of responses) for
Maturity Level 3 and 751 times (67.17% of responses) for Maturity Levels 4 and 5. Note that
this does not equate directly to the numbers shown in Figure 2-11, which illustrates the
lowest Maturity Level to any Study Objective given by an ANSP, and the number of times
they gave that response.
One ANSP reported being at Level 2 seventeen times out of a possible twenty-six question
responses. Significantly, this ANSP scores Level 2 for all eleven Study Areas. However,
despite this scoring the ANSP concerned has made significant progress over the past two
years. Twenty three ANSPs (52% of participating ANSPs) report being at Level 3 (22
ANSPs) or higher (there is one ANSP with an overall Maturity Level of 4) in all eleven Study
Areas. The strongest Study Area for these 23 ANSPs is Study Area 3 (Timely Compliance
with International Obligations), where 13 of these ANSPs are at Level 3 or higher.
The average Maturity Profile remained broadly static, although as in the 2011 survey
individual ANSPs have moved up and down relative to each other; the most significant
changes being in the 80% to 100% range of Maturity Scores. There is an increase in the
number of ANSPs at Level 3.

2.3.1 Key Findings

Seventeen ANSPs have been classified at a lower Maturity Score than in 2011, whilst twenty
one have seen their Maturity Score increase.
In previous surveys ANSPs have asked for more thorough and clear guidance which would
allow for a better and more consistent interpretation of the maturity requirements. Some were
being very precise and literal in their interpretation, whilst others were using looser
interpretations and, perhaps, over-stating their actual maturity. For the 2012 survey guidance
was provided in the form of a “Questionnaire Clarification Document” which was sent out with
the questionnaire. There was a mixed response with regard to the extent to which this
document was used when completing the 2012 questionnaire; some ANSPs commented that
the document had been very useful, whilst others had not noticed it attached to the
questionnaire and said that they had not read it, whilst some claimed not to have received it.
Released Issue Edition: 1.0 24
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

In 2011 the survey reported that there was evidence of tension building up within FABs, with
smaller organisations feeling that their larger partners are being too domineering. Similar
evidence was not detected this year. Some of the smaller ANSPs were getting support from
their FAB partners, albeit mostly on a commercial basis.
Some ANSPs still see the State judicial system, and the media, as barriers to developing a
Just Culture, although they are willing to fight to protect their staff if they have to. They
consider that it is the NSA/Regulator’s responsibility to negotiate with the Judicial Authorities.
It was noticeable that more ANSP’s CEOs are personally involved in Safety decisions and
personally approve matters, in some instances that includes the responses to the SoE
measurement questionnaire. This is thought to be due to the increased emphasis on Safety
Culture and, in SES States, the need to follow EU legislation.
This year a system of Supplementary Questions was introduced for some ANSPs (this
system was first used in the BlueMed survey with good results). The Supplementary
Questions were based on the question set provided by EUROCONTROL for use in all
interviews. This method of getting additional information was welcomed as it gave
interviewees more time to consider the response and overcame any verbal language
problems. One ANSP commented that at first he thought that it would be an added burden,
but having completed the supplementary questions, undergone the telephone interview and
seen the resulting report thought that it was very worthwhile.
Some interviewees questioned the need to complete the SoE Measurement and then
complete the EASA questionnaire later in the year, but by far the majority considered that the
exercise prepared them for the second survey and were very happy to have gone through
the exercise.
There were still a number of ANSPs who questioned the need to develop Best Practices in
all Study Areas, in fact a number refused to downgrade their score from E to D even though
they could not identify a Best Practice that they had developed, but considered that in all
other respects they met the requirements for E.
What is meant by the term “Best Practice” is misunderstood by many of those who
participated in the survey and should be clarified. It appeared that some ANSPs did not
appreciate the need to have a system to measure their SMS performance before selecting a
level beyond Level C.
Those ANSPs that underwent face-to-face interviews appreciated the face-to-face interaction
as being a very useful exchange, which gave the opportunity to discuss the Study Areas in
more detail. ANSPs considered that face-to-face interviews made it easier to clarify any
issues and also enabled more people from an ANSP to participate in discussions.
A number of interviewees commented that they considered only seven months between
surveys was too short a time for organisations to show any real improvement in their scoring
and requested that in future the gap should be larger. It was explained this survey was
brought forward exceptionally to align the process with the requirements of the Single
European Sky Legislation and the gap would revert to one year in future.

2.3.2 Study Area 1 – Development of a Positive and Proactive Safety Culture

Objectives:

1.1 A positive and pro-active just, flexible, and informed safety culture (the shared
beliefs, assumptions, and values regarding safety) that supports reporting and
learning led by management.
1.2 Regular measurement of safety culture and an improvement programme.
1.3 An open climate for reporting and investigation of occurrences.

The maturity profile for Study Area 1 ranges from a minimum of 33.0% to a maximum of
88.5%, with the average being 69.9% (compared with 23.9%, 91.8% and 69.9% in 2011).
Released Issue Edition: 1.0 25
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Figure 2-12 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 1 for 2012 and 2011.

25
21
20

Number of ANSPs
15 14
13 13
11 11 2011

10 2012

5
1
0 0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-12: SA1 – Numbers of ANSPs at each Maturity Level

It can be seen from Figure 2-12 that there are no ANSP’s who consider that they are still at
Level 1 in this Study Area and for 2012 the majority of ANSPs have reached at least Level 3,
though none have yet achieved Level 5. This indicates that a positive Safety Culture is
developing, but there is still work to do. The majority of ANSPs at Levels 3 and 4 have
improvement plans in place and those at Level 3 are beginning to measure safety culture,
however there are a number of ANSPs at Level 2, and some at Level 3, who do not know
how to measure Safety Culture. On the positive side several ANSPs have completed Safety
Culture Surveys in 2012 and more are planned for later in the year or early 2013. In addition,
some ANSPs are planning for their second or third survey. There has been some movement
between Levels 3 and 4 with some ANSPs improving their score and a few falling back from
Level 4 to Level 3, but overall the signs for this Study Area are positive.
Study Area 1 was only discussed in depth in the few face-to-face interviews undertaken so
there is only limited information on the causes for movement in this Area. However, it was
noticeable how many CEOs/DGs now take a personal interest in ensuring that Safety Culture
is being developed within their organisations and support the measures being taken by their
staff. One ANSP explained that they had developed a process for ensuring that their CEO
was fully involved with Safety Culture and SMS development and claimed this as a Best
Practice that they had shared with others.
The instances of ANSPs undertaking, or planning to undertake, Safety Culture Surveys
appears to be increasing. One ANSP had found resistance from senior management, but a
change in CEO brought a changed attitude and the ANSP is now undertaking a Safety
Culture Survey. Another significant fact to driving a change in attitude to measuring Safety
Culture is the fact that younger staff are now moving into positions of responsibility and see
the benefits of improving safety culture.
In summary, whilst ANSPs recognise the importance of measuring Safety Culture only a few
had any idea on how to achieve that and were therefore struggling to progress beyond Level
3 in this Study Area. Most recognised that they would need to enlist external assistance to
achieve a viable measurement.

Released Issue Edition: 1.0 26


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
20 SA1: Development

Movement Comparison
15 of a Positive and
10 Proactive Safety Culture
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.55 5.70 19.78
Dec 11/12 -0.82 -5.25 -14.01
Inc 10/11 0.27 6.56 18.96
Dec 10/11 -0.27 -6.18 -17.86

Figure 2-13: Minimum / Average / Maximum Change in Study Area 1


As can be seen from Figure 2-13, there are differences in the extent to which ANSPs have
either improved or decreased their Maturity Score in Study Area 1. In 2012, the average
percentage point increase by ANSPs is 5.70%. Conversely, a number of ANSPs have
decreased their score since 2011. The average percentage point decrease is minus 5.25%.
As can be seen there is less deviation between 2011 and 2012 than between the 2010 and
2011 scores. The minimum increases and decreases are much less than 1% for all years.
The minus percentage maximum deviation of 17.86% in those years is probably caused by
ANSPs overmarking themselves during the first year of the revised survey methodology. For
2011/2012 the Maximum decrease has fallen to minus 14.01%.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-14: Maturity Score % Point Movement Comparisons


Figure 2-14 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the Maturity Score
percentage point movement for each ANSP comparing 2011/2012 with 2010/2011
movements. Notably:
• Sixteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Eighteen ANSPs have had a decrease in Maturity Score since the start of the current
Maturity questionnaire in 2010. Six of these ANSPs experienced a decrease in
Maturity Score year on year;
• Twenty one ANSPs have increased their Maturity Score in this Study Area since
2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
Study Area requirements.

Released Issue Edition: 1.0 27


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2.3.3 Study Area 2 – Organisational and Individual Safety Responsibilities

Objectives:

2.1 An approved, clearly documented, and recognised system for the management of
safety. Management structure, responsibilities, accountabilities and authorities are
clearly defined and documented.
2.2 A clearly defined safety management function that is independent of line
management.
2.3 An integrated safety planning process is adopted by the organisation with
published and measurable safety goals and objectives which are accountable to the
executive.
2.4 Clear understanding and acceptance of safety management responsibilities by all
staff and contractors.

The maturity profile for this Study Area ranges from a minimum of 32.1% to a maximum of
90.4%, with the average being 71.2% (compared with 24.5%, 94% and 71.3% in 2011).
Figure 2-15 (Number of ANSPs at each Maturity Level) shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 2.

25

20
20
Number of ANSPs

17 17

15
2011
10
10 2012

6 6
5
5 3

0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-15: SA2 – Number of ANSPs at each Maturity Level


This Study Area is well developed with only six ANSPs below the target Level 3. All ANSPs
have some form of formal SMS, with someone appointed to be responsible for Safety and
safety responsibilities and accountabilities set out in their Safety Management Manual.
Figure 2-15 shows that three ANSPs claim to have reached Level 5; however two of those
could not demonstrate that they had developed best practice in this Study Area and believed
that that it was not feasible to set an industry best practice in this area and as they met all
other aspects of Level E marked themselves as such. In 2011 five ANSPs believed that they
had achieved Level 5, but two of those reassessed their score in 2012. There were ANSPs
who claimed to have developed Best Practice in this area for one of the sub-areas and
marked that at Level 5 though their overall marking for Study Area 2 did not warrant Level 5.
As in 2011 all organisations report that they have had formal Safety Management Systems
(SMS) in place for a number of years, and that regular updates take place to ensure that they
remain current and relevant. The complexity of the system is directly related to the size of the
ANSP. The larger organisations have very formal systems with clear management
demarcation. Smaller units are less formally structured as there are only three or four staff
and they all know each other; a rigid organisational structure is therefore not considered
necessary. However, irrespective of the size and structure of the SMS most systems seemed
to be formally embedded within the overall organisations.

Released Issue Edition: 1.0 28


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

In 2011 by far the greatest number of ANSPs (20) were at Level 3. This year there has been
a significant improvement with seventeen ANSPs now saying that they are at Level 4.
A number of ANSPs have undertaken major reviews of their safety organisation and safety
responsibilities. Some of the less mature ones have sought help from more mature partners
in an effort to improve the systems that they have in place. Fairly frequent reviews of safety
responsibilities are taking place and many safety managers now have direct access to their
CEOs. Safety Culture is seen as a driver for ensuring an effective SMS.
Quite a few ANSPs combine their safety and quality organisations and believe that the
quality side helps the safety organisation through such things as ISO 9001 certification by
ensuring that the correct processes are in place. Furthermore, demands for continuous
process monitoring and measuring from the quality side is solid grounds to scoring above
Level 3 in areas under the remit of the quality process.
The majority of ANSPs now have a formal safety plan in place that is reviewed annually.
Safety Responsibilities are often set out in the Safety Management Manual and are also
available for all staff to view on the Intranet. In SES States Safety Policy is based on
European Regulations. However, the safety plan is often encapsulated in overall
organisational plans that include quality, safety and security as some CEOs/DGs believe
there should only be one overall strategic plan. Departmental activity plans are then
developed from the overall plan. Performance is measured against meeting safety targets.
Only a few ANSPs have a formal review process in place to specifically identify changes
within their organisation that could affect company documentation. Mainly this is dealt with on
an ad hoc basis, or is the subject of discussion at safety related meetings. However most
ANSPs have some form of regular review of the SMS through frequent audits that results in
updates being made. Agreement on documentation updates is often undertaken through
safety committees.
ANSPs use a variety of methods to ensure that staff are kept up to date with safety
information through committees, safety leaflets/bulletins, use of the intranet, safety seminars
etc. The impression from interviews undertaken is that in most ANSPs staff are aware of
safety responsibilities, they are kept up to date with safety related information and discussion
on safety-related matters is encouraged by most ANSPs within the ECAC Area. There is little
doubt that SES legislation has helped to focus the minds of ANSPs’ senior management and
made them pay more attention to safety requirements than hitherto. This means that most
ANSPs now have well documented organisational and individual responsibilities with safety
managers having access to the highest organisational level.
A number of ANSPs said that when formulating SMS training and assessment they were
developing and using Best Practice the Best Practices developed were then shared with FAB
partners. For instance some are now involving staff generally to help improve their training
development and others are setting Technical Boards that have an input into assessment
and training requirements.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
16.

Released Issue Edition: 1.0 29


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
20 SA2: Organisational &

Movement Comparison
15 Individual Safety
10 Responsibilities
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.55 6.17 20.05
Dec 11/12 -0.27 -4.72 -16.21
Inc 10/11 0.27 6.41 17.58
Dec 10/11 -0.55 -6.36 -16.76

Figure 2-16: Minimum / Average / Maximum Change in Study Area 2


As can be seen from Figure 2-16, there are differences in the extent to which ANSPs have
either improved or decreased their Maturity Score in Study Area 2. In 2012, the average
percentage point increase by ANSPs is 6.17%. Conversely, a number of ANSPs have
decreased their score since 2011. The average percentage point decrease is minus 4.72%.
As in Study Area 1, there is less deviation between 2011 and 2012 than between the 2010
and 2011 scores. The minimum increases and decreases are much less than 1% for all
years. The minus percentage point maximum deviation of 16.76% in the maximum score for
those years is probably caused by ANSPs overmarking themselves during the first year of
the revised survey methodology.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

-5.00

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-17: % Movement Comparisons


Figure 2-17 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Eighteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Seventeen ANSPs have had a decrease in Maturity Score since the start of the
current Maturity questionnaire in 2010. Five of these ANSPs experienced a decrease
in Maturity Score year on year;
• Twenty ANSPs have increased their Maturity Score in this Study Area since 2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

Released Issue Edition: 1.0 30


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2.3.4 Study Area 3 – Timely Compliance with International Obligations

Objectives:

3.1 A formal SMS that meets all applicable safety requirements.


3.2 An organisation that strives to go beyond compliance, takes into account the need
to ensure, in a timely manner, that there are no inconsistencies with European or
national requirements or international safety standards.

The maturity profile for Study Area 3 ranges from a minimum of 33.8% to a maximum of
91.2%, with the average being 73.6% (compared with 24.5%, 97.1% and 73.4% in 2011).
Figure 2-18 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 3.

20 19 19
18
16
Number of ANSPs

14
12
12
10
10 9 2011
8
8 2012

6
4
4
2
2 1
0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-18: SA3 – Number of ANSPs at each Maturity Level


There are now no ANSPs at Level 1. The numbers at Levels 4 and 5 have remained fairly
static whilst there has been some movement up and down between Levels 2 and 3.
As in 2011 this Area has shown the strongest growth in terms of maturity scores. Although
the maximum maturity profile has reduced from 97.1% to 91.2%, the minimum has risen
9.3% from 24.5% to 33.8%. The fall in the maximum Maturity Level is due to a more realistic
marking at Level 5. All ANSPs have at least a basic SMS in place and report that they are
continually monitoring changing requirements within the SES Legislation and looking for
ways to improve their performance in this Area.
Study Area 3 was only discussed in depth in the few face-to-face interviews undertaken so
there is only limited information in this Area. However the Justification and Evidence provided
in the Questionnaires and, where applicable, the interviews suggest that all ANSPs are well
aware of their international obligations and are taking them seriously. However in some
States EU legislation has to be translated into the local language before new regulations
become law and this can be a slow process resulting in a delay before the national ANSP
can implement the changes.
Measurement of compliance with international regulations is normally undertaken by audits.
Some ANSPs claimed that they were involved in “Best Practice” development by working
with EUROCONTROL, the EU and ICAO on developing such things as RAT, SES legislation
and SARPs.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
19.

Released Issue Edition: 1.0 31


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
20 SA3: Timely Compliance
with International

Movement Comparison
15
10
Organisations
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 1.47 7.12 20.59
Dec 11/12 -0.98 -6.08 -16.67
Inc 10/11 1.47 7.07 16.18
Dec 10/11 -0.49 -5.80 -17.65

Figure 2-19: Minimum / Average / Maximum Change in Study Area 3


As can be seen from Figure 2-19, there are differences in the extent to which ANSPs have
either improved or decreased their Maturity Score in Study Area 3. In 2012, the average
percentage point increase by ANSPs is 7.12%. Conversely, a number of ANSPs have
decreased their score since 2011. The average percentage point decrease is minus 6.08%.
As in other Study Areas, there is less deviation between 2011 and 2012 than between the
2010 and 2011 scores. The minimum increases and decreases are 0.5 to 1.5% for all years.
The minus percentage point maximum deviation of 16.67% in the maximum score for
2011/2012 is probably caused by ANSPs overmarking themselves during the first year of the
revised survey. However there is a good increase in the maximum scores for 2012, which
has increased by 4.41% over the 2010/11 maximum scores.

25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

-5.00

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-20: % Movement Comparisons


Figure 2-20 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Fifteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Fourteen ANSPs have had a decrease in Maturity Score since the start of the current
Maturity questionnaire in 2010. Seven of these ANSPs experienced a decrease in
Maturity Score year on year;

Released Issue Edition: 1.0 32


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

• Seventeen ANSPs have increased their Maturity Score in this Study Area since 2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

2.3.5 Study Area 4 – Safety Standards and Procedures

Objectives:

4.1 Clearly defined and documented safety standards and procedures.


4.2 Staff know about the safety and safety management requirements and standards,
which are regularly reviewed, assessed, and maintained.
4.3 Emergency/contingency response procedures and an emergency/contingency
response plan that documents the orderly and efficient transition from normal to
emergency operations and return to normal operations.

The maturity profile for Study Area 4 ranges from a minimum of 33.7% to a maximum of
89%, with the average being 71.6% (compared with 24.7%, 93% and 71.8% in 2011). Figure
2-21 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 4.

25 23

20
20
Number of ANSPs

15
12
11 2011

10 2012
7
5
5 3 3

0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-21: SA4 – Number of ANSPs at each Maturity Level


Study Area 4 was only discussed in depth in the few face-to-face interviews undertaken so
there is only limited information in this Area. However, it is clear from Figure 2-21 that the
majority of ECAC ANSPs consider safety documentation to be important and have
completed documentation of the SMS. Systems have redundant capability and emergency
response plans are in place. ANSPs do not undertake live rehearsals of their contingency
plans for safety and economic reasons but all major organisations have some form of
desktop simulation where contingency plans are practised. However the practice of
emergency procedures seems to be undertaken in an ad hoc manner in many ANSPs and
are not very frequent, particularly in the smaller organisations. Some Organisations believe
that they are using Best Practice established by EUROCONTROL and ICAO in this area.
The statistics show that there has been strong growth with twenty three ANSPs now marking
themselves at Level 4. The majority of ANSPs are either at Level 3 or 4. Those at Level 5
have reduced by one due to an ANSP taking a more realistic approach to the requirement of
that Level for this Study Area.
Overall a significant improvement over 2011 with the average maturity profile in this area
now standing at 71.6%.

Released Issue Edition: 1.0 33


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
22.

25
20 SA4: Safety Standards
& Procedures

Movement Comparison
15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.33 5.67 18.67
Dec 11/12 -1.00 -5.56 -17.00
Inc 10/11 0.33 6.41 19.33
Dec 10/11 -0.33 -6.41 -15.33

Figure 2-22: Minimum / Average / Maximum Change in Study Area 4


As can be seen from Figure 2-22, there is little difference in the changes in the comparisons
shown for this Study Area. Although there is some up and down movement, the overall
picture remains fairly static for both comparisons.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-5.00 44

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-23: % Movement Comparisons


Figure 2-23 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Sixteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Fifteen ANSPs have had a decrease in Maturity Score since the start of the current
Maturity questionnaire in 2010. Four of these ANSPs experienced a decrease in
Maturity Score year on year;
• Twenty two ANSPs have increased their Maturity Score in this Study Area since
2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

Released Issue Edition: 1.0 34


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2.3.6 Study Area 5 – Competency

Objectives:

5.1 Staff, and contractors (where appropriate) are trained, competent in safety and
safety management, and where required, licensed.

The maturity profile for Study Area 5 ranges from a minimum of 32.9% to a maximum of
88.6% with the average being 69.9% (compared with 24.2%, 91.8% and 70% in 2011).
Figure 2-24 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 5.

25
22

20
Number of ANSPs

17

15 14 14

2011

10 8 2012

5
5
2 2
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-24: SA5 – Number of ANSPs at each Maturity Level


Figure 2-24 shows that there has been a shift in the numbers in this area. There has been an
improvement of four ANSPs moving from Level 3 to Level 4 plus an additional ANSP being at
that level. Level 5 has remained static, though overall the maximum profile figure has fallen
slightly.
Study Area 5 was only discussed in depth in the few face-to-face interviews undertaken so
there is only limited information in this Area. Justification and Evidence submitted with the
questionnaires indicates that most organisations have well developed training and
competency processes in place and many are using IANS facilities for training. ANSPs have
developed training plans and these are often divided into safety training for operational and
engineering staff and a safety training plan for non-ATC staff.
Several ANSPs have procedures for measuring the effectiveness of both internal and
external training. Most of the organisations believe that they are using best practices in their
training and competency checks.
The percentage movement from 2010 and between 2011 an 2012 is shown in Figure 2-25.

Released Issue Edition: 1.0 35


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
20 SA5: Competency

Movement Comparison
15
10
5
0
-5
-10
-15
-20
-25
Minimum Average Maximum
Inc 11/12 0.27 5.68 20.11
Dec 11/12 -0.82 -5.18 -14.13
Inc 10/11 0.27 6.53 18.48
Dec 10/11 -0.27 -6.21 -18.21

Figure 2-25: Minimum / Average / Maximum Change in Study Area 5


As can be seen from Figure 2-25, there are differences in the extent to which ANSPs have
either improved or decreased their Maturity Score in Study Area 5. In 2012, the average
percentage point increase by ANSPs is 5.68%. Conversely, a number of ANSPs have
decreased their score since 2011. The average percentage point decrease is minus 5.18%.
As can be seen there is (as with other Study Areas) less deviation between 2011 and 2012
than between the 2010 and 2011 scores. The minimum increases and decreases are less
than 1% for all years. The minus percentage point maximum for decreases has improved
from minus 18.21 in 2010/2011 to minus 14.13% for 2011/2012.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

-5.00

-10.00

-15.00

-20.00 Movement 2010/11


Movement 2011/12
-25.00

Figure 2-26: % Movement Comparisons


Figure 2-26 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Sixteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Eighteen ANSPs have had a decrease in Maturity Score since the start of the current
Maturity questionnaire in 2010. Six of these ANSPs experienced a decrease in
Maturity Score year on year;
• Twenty one ANSPs have increased their Maturity Score in this Study Area since 2011
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

Released Issue Edition: 1.0 36


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2.3.7 Study Area 6 – Risk Management

Objectives:

6.1 A continuing risk management process that identifies, assesses, classifies, and
controls all identified safety risks within the organisation, including potential future
risks.
The maturity profile for Study Area 6 ranges from a minimum of 33.8% to a maximum of
87.8%, with the average being 70.3% (compared with 23.8%, 91.8 and 70.4% in 2011).
Figure 2-27 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 6.

25
21
20 19
Number of ANSPs

15 13
12
2011

10 2012
7
6
5 4
2
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-27: SA6 – Number of ANSPs at each Maturity Level


Although there has been some movement since the past survey overall this Area has
remained fairly static for this year and the situation seems to be very much as depicted in the
2011 report. All ANSPs understand the principles of risk management and many have plans
already implemented or have a plan in place to implement a system. The majority of
organisations could show that they had a safety risk system embedded within the
organisational structure and they were measuring the risks.
The complexity of ANSPs’ risk assessment systems very much depends on the size and
complexity of the ANSP itself. Most ANSPs have a formal interface with their NSA, but once
again depending on the size of the ANSP the interface may be close or more at arms length.
However, the NSA in all States are informed about upcoming changes and their outcomes,
and in some States, the NSA conducts periodic reviews of implemented changes. The larger
ANSPs tend to have more formal systems with clear procedures of when to involve the NSA
in risk management. With some of the smaller organisations the process is more ad hoc.
Some ANSPs reported that the manpower situation within NSAs has improved slightly and
this is thought to be as a result of EU legislation.
Where there is a lack of regulatory requirement with regard to acceptable levels of risk, the
EUROCONTROL guidelines are used.
Staff are increasingly being brought into the risk assessment process and views are sought
by management of where risks are thought to lie. External stakeholders (e.g. airlines,
airports, local flying community etc.) are also involved by some ANSPs. A few organisations
are using a Risk Picture system based on IRP to develop a Hazard Log.
Future safety risks are being identified through the combination of different processes such
as trend analysis, safety assessment, monitoring & investigations procedures etc.

Released Issue Edition: 1.0 37


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

The Risk Analyses Tool (RAT) seems to be increasingly used, although a few ANSPs said
that it did not meet their needs precisely, because they believed that the repeatability
element of the tool is not yet evolved sufficiently. Some believe that the tool is more of a
Severity Assessment Tool, due to the amount of subjectivity involved. Some organisations
have developed their own system to determine risk, though the impression from the
interviews is that most of the locally developed systems follow similar principles to RAT.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
28.

25
20 SA6: Risk Management
Movement Comparison

15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.30 6.14 19.21
Dec 11/12 -0.91 -5.70 -15.85
Inc 10/11 0.91 6.59 17.99
Dec 10/11 -1.22 -6.05 -14.94

Figure 2-28: Minimum / Average / Maximum Change in Study Area 6


As can be seen from Figure 2-28, there is little difference in the changes in the comparisons
shown for this Study Area. Although there is some up and down movement, the overall
picture remains fairly static for both comparisons.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

-5.00

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-29: % Movement Comparisons


Figure 2-29 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Sixteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Seventeen ANSPs have had a decrease in Maturity Score since the start of the
current Maturity questionnaire in 2010. Five of these ANSPs experienced a decrease
in Maturity Score year on year;
Released Issue Edition: 1.0 38
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

• Twenty one ANSPs have increased their Maturity Score in this Study Area since
2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

2.3.8 Study Area 7 – Safety Interfaces

Objectives:

7.1 Effectively managed safety-related internal interfaces (e.g. quality management


system, security, and environment).
7.2 The effective management of external interfaces with a safety impact (e.g., MIL,
airspace users, airports). Formalised processes and procedures dealing with
external agreements, services, and supplies (e.g., cross-border Letters of
Agreement).
The maturity profile of Study Area 7 ranges from a minimum of 30.6% to a maximum of
89.7%, with the average being 70.0% (compared with 20.2%, 92.9% and 69.9% in 2011).
Figure 2-30: shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 7.

30
25
25 23
Number of ANSPs

20

15 2011
12
2012
9
10

4
5 3 3
2 2
1
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-30: SA7 – Number of ANSPs at each Maturity Level


This Study Area was only discussed in depth in the few face-to-face interviews undertaken
so there is only limited information in this Area. However the indications are that there has
been little movement since 2011. Figure 2-30 shows that there is one less ANSP at Level 1
and an increase of two at Level 3. The number at Level 3 has risen to twelve due to
additional ANSPs participating this year. One ANSP has moved from Level 4 to Level 5.
A number of ANSPs now have combined SMS and QMS units and this seems to have
helped interfaces, particularly where units have been ISO 9001 certified. The smaller ANSPs
tend not to have formal interfaces between departments though they do have formal
agreements with outside partners such as the military and neighbouring service providers.
The larger more established ANSPs have formal agreements between internal departments
(e.g. OPS and Engineering) and with outside suppliers which often includes such services as
MET.
As described in Section 2.3.13 (Figure 2-45), Study Area 7 is the weakest area in terms of
Maturity Score for 14 ANSPs. However, as shown in Figure 2-30, the majority of ANSPs (38)
are at Maturity Level 3 or higher, indicating that they at least meet the required regulatory
standard.

Released Issue Edition: 1.0 39


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

The percentage point movement from 2010 and between 2011 an 2012 is shown in
Figure 2-31.

25
20 SA7: Safety Interfaces

Movement Comparison
15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.40 6.59 18.65
Dec 11/12 -0.79 -5.65 -16.27
Inc 10/11 0.40 7.04 18.65
Dec 10/11 -0.79 -5.58 -16.27

Figure 2-31: Minimum / Average / Maximum Change in Study Area 7


As can be seen from Figure 2-31 there is no differences in the changes in the comparisons
shown for this Study Area. Although there is some up and down movement, the overall
picture remains fairly static for both comparisons.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-5.00 44

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-32: % Movement Comparisons


Figure 2-32 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Sixteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Seventeen ANSPs have had a decrease in Maturity Score since the start of the
current Maturity questionnaire in 2010. Six of these ANSPs experienced a decrease
in Maturity Score year on year;
• Twenty one ANSPs have increased their Maturity Score in this Study Area since 2011
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

Released Issue Edition: 1.0 40


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2.3.9 Study Area 8 – Safety Reporting, Investigation and Improvement

Objectives:

8.1 A continuing organisation-wide process to report and investigate safety


occurrences and risks.
8.2 An organisation-wide means to record and disseminate lessons learned.
8.3 Appropriate safety information and knowledge is shared with Industry
stakeholders. Information disclosure is compliant with agreed publication and
confidentiality policies/agreements.
The maturity profile of Study Area 8 ranges from a minimum of 33.1% to a maximum of
89.6%, with the average being 71.1% (compared with 22.8%, 92.4% and 71.1% in 2011).
Figure 2-33: shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 8.

30
25
25
22
Number of ANSPs

20

15 13 2011
2012
10 8
5
4
5 3 3
1
0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-33: SA8 – Number of ANSPs at each Maturity Level


As can be seen from Figure 2-33, Study Area 8 has moved forward with no ANSP at Level 1
and only three at Level 2. The majority have reached Level 3 and beyond with five more
ANSPs at Level 4 this year and an additional one at Level 5.
All ANSPs have some form of reporting and investigation systems with the majority having a
well defined system with clear processes that meet the regulatory requirement and the needs
of their own organisations. Some ANSPs also have automatic monitoring in place, but other
ANSPs report that it is difficult to get this form of monitoring accepted by staff.
The SMS processes are working well and measurement of some kind is in place. However
one ANSP believed that measuring in this area is difficult as measuring incidents is not an
ideal way to measure performance as there are too many imponderables. The greater
emphasis on Safety Culture has also helped to improve occurrence reporting. The
improvement of Just Culture will also impact because in a few ANSPs there is still a feeling
that those that report may get penalised.
The larger ANSPs tend to have specific remedial action plans that are fed into individual
departmental action plans.
The dissemination of lessons learned varies greatly. Smaller ANSPs tend to disseminate
through weekly meetings and by word of mouth. Larger organisations have more formal
means with regular publications, workshops etc. The majority of ANSPs now use their
Intranet to publicise lessons learned and include them in training. Several ANSPs report that
they are reviewing their systems for disseminating data in order to make them more
systematic.

Released Issue Edition: 1.0 41


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

The majority of ANSPs understand the importance of disseminating data as far as possible
both within the organisation and to others. During the interviews, the Hindsight magazine
was quoted as being an excellent example of spreading information between ANSPs and it is
clear that lessons learned are shared within the FAB organisations.
In the majority of ANSPs, while corrective measures are reviewed to ensure they have been
implemented correctly, there is limited activity to determine to what extent the lessons have
actually been learned. Whilst there may be some obvious metrics such as whether there is
any re-occurrence of incidents, there is little in the way of a formal review. While there is no
doubt about means being put in place to support these objectives, there is a clear need in a
number of ANSPs to ensure the quality of the system through measurements of the
reporting, investigation and lesson dissemination processes.
Some ANSPs have processes where anyone within the organisation who identifies a
problem in the system, or a document etc. can write a problem report. The reports are then
followed up by the Safety Unit/Department.
There is still reluctance in some ANSPs to share data fully with the general public and the
majority stated that they only share outside of their own organisations when required legally
to do so.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
34.

25
SA8: Safety Reporting,
20
Investigation &
Movement Comparison

15
Improvement
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.56 5.96 18.54
Dec 11/12 -0.56 -4.40 -14.04
Inc 10/11 0.56 6.65 20.51
Dec 10/11 -0.56 -6.11 -16.85

Figure 2-34: Minimum / Average / Maximum Change in Study Area 8


As can be seen from Figure 2-34, there are differences in the extent to which ANSPs have
either improved or decreased their Maturity Score in Study Area 8. In 2012, the average
percentage point increase by ANSPs is 5.96% whilst in 2011 it was 6.65%. A number of
ANSPs have decreased their score since 2011 and the maximum percentage point
movement has fallen back from 20.51% in 2011 to 18.54% in 2012. Conversely, the average
percentage point decrease is minus 4.40% in 2012 compared with minus 6.11% in 2011. The
minimum increases and decreases are less than 1% for all years.

Released Issue Edition: 1.0 42


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-35: % Movement Comparisons


Figure 2-35 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Seventeen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Seventeen ANSPs have had a decrease in Maturity Score since the start of the
current Maturity questionnaire in 2010. Five of these ANSPs experienced a decrease
in Maturity Score year on year;
• Twenty ANSPs have increased their Maturity Score in this Study Area since 2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

2.3.10 Study Area 9 – Safety Performance Monitoring

Objectives:

9.1 An established and active monitoring system that uses and tracks suitable safety
indicators and associated targets (e.g., lagging and leading indicators).
9.2 Methods to measure safety performance, which is compared within and between
ANSPs.
9.3 A general public knowledgeable of the ANSP’s performance through routine
publication of achieved safety levels and trends. (Information disclosure is
compliant with the requirements of ICAO Annex 13, Attachment E).

The maturity profile of Study Area 9 ranges from a minimum of 33.2% to a maximum of
88.3%, with the average being 70.6% (compared with 22.6%, 91.3% and 70.7% in 2011).
Figure 2-36 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 9.

Released Issue Edition: 1.0 43


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
21
20
20

Number of ANSPs
15
11 11 2011
10
10 2012
7

5
2 2
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-36: SA9 – Number of ANSPs at each Maturity Level


No ANSP claims to have achieved Level 5 in this Area. One less ANSP than in 2011
considered that they had achieved Level 3, but three ANSPs reassessed their scoring last
year and considered that they were still at Level 2.
This Study Area was not examined with most ANSPs during the interviews this year. Though
during face-to-face interviews and where inconsistencies in questionnaire responses were
noted some questions were asked. Most ANSPs had some form of monitoring in place, but
this is often based on incident reporting and the majority did not have a Quantitative system
in place, nor do they undertake internal comparative analysis.
Although some of the larger ANSPs willingly shared safety level and trend data with the
general public, utilising their public websites for this activity, many did not see the need to
share data. Some commented that they would give out information on request but preferred
to only share data that they were required to do so. On the whole the larger organisations
seem more willing to share than the smaller and also publish figures in their annual reports.
Some used the Maturity Survey data to show progress. Sharing of data between FAB
partners appears to be common.
It did appear from those ANSPs that were questioned that there is a slight lessening of the
resistance to making more information available to the general public in some States, but
other ANSPs will not release information that they are not required to due to a fear of
possible litigation.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
37.

Released Issue Edition: 1.0 44


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
20 SA9: Safety Performance
Monitoring

Movement Comparison
15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.27 6.09 19.57
Dec 11/12 -0.54 -4.59 -14.40
Inc 10/11 0.54 6.10 20.38
Dec 10/11 -0.82 -6.63 -15.76

Figure 2-37: Minimum / Average / Maximum Change in Study Area 9


As can be seen from Figure 2-37, there are differences in the extent to which ANSPs have
either improved or decreased their Maturity Score in Study Area 8. In 2012, the average
percentage point increase by ANSPs is 6.09% whilst in 2011 it was 6.1% showing little
movement. A number of ANSPs have decreased their score since 2011 and the maximum
percentage point movement has fallen back from 20.38% in 2011 to 19.57% in 2012.
Conversely, the average percentage point decrease is minus 14.40% in 2012 compared with
minus 15.76% in 2011. The minimum increases and decreases are less than 1% for all
years.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44

-5.00

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-38: % Movement Comparisons


Figure 2-38 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Seventeen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Seventeen ANSPs have had a decrease in Maturity Score since the start of the
current Maturity questionnaire in 2010. Five of these ANSPs experienced a decrease
in Maturity Score year on year;
• Twenty ANSPs have increased their Maturity Score in this Study Area since 2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

Released Issue Edition: 1.0 45


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2.3.11 Study Area 10 – Operational Safety Surveys and SMS Audits

Objectives:
10.1 Internal and independent (external) operational safety surveys and SMS audits.

The maturity profile of Study Area 10 ranges from a minimum of 32.8% to a maximum of
87.6%, with the average being 69.9% (compared with 24.4%, 91.1% and 71.2% in 2011).
Figure 2-39 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 10.

25 23

20
20
Number of ANSPs

15
12
2011
10
10 2012
7
5
5 4
3

0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-39: SA10 – Number of ANSPs at each Maturity Level


As the figures in Figure 2-39 indicate that there has been little movement in this area, as
overall figures remain much as in 2011.
This Study Area was only discussed with a few ANSPs during this year’s survey. From the
few remarks made and the Justification text contained in the questionnaires it would seem
that safety surveys and audits are undertaken. Internal surveys are often used to assess how
much staff understand the SMS processes and to ensure that safety procedures are being
followed. It is usual practice within the SES States to be audited by their NSA.
The percentage movement from 2010 and between 2011 an 2012 is shown in Figure 2-40.

25
20 SA10: Operational
Safety Surveys
Movement Comparison

15
& SMS Audits
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.29 5.73 19.83
Dec 11/12 -1.15 -5.46 -15.80
Inc 10/11 0.57 6.24 17.53
Dec 10/11 -1.72 -6.85 -14.37

Figure 2-40: Minimum / Average / Maximum Change in Study Area 10

Released Issue Edition: 1.0 46


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

As can be seen from Figure 2-40, there is little differences in the changes in the comparisons
shown for this Study Area. Although there is some up and down movement, the overall
picture remains fairly static for both comparisons. The average decrease in percentage point
movement has fallen from minus 6.85% in 2011 to minus 5.46% in 2012. The maximum
increased movement has risen slightly from 17.53% in 2011 to 19.83% in 2012. The
minimum decrease in movement has fallen from minus 1.72% in 2011 to minus 1.15 in 2012.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00

-10.00

-15.00 Movement 2010/11


Movement 2011/12
-20.00

Figure 2-41: % Movement Comparisons


Figure 2-41 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Sixteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Eighteen ANSPs have had a decrease in Maturity Score since the start of the current
Maturity questionnaire in 2010. Five of these ANSPs experienced a decrease in
Maturity Score year on year;
• Twenty one ANSPs have increased their Maturity Score in this Study Area since
2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

2.3.12 Study Area 11 – Adoption and Sharing of Best Practices

Objectives:

11.1 A structured approach exists to promote safety, its standing within the
organisation and lessons learned through application of the SMS.
11.2 A structured approach to gather information on operational safety and SMS
best practises from the industry.
11.3 Sharing of safety and SMS-related best practises with industry stakeholders.

The maturity profile of Study Area 11 ranges from a minimum of 33.4% to a maximum of
90.1%, with the average being 70.5% (compared with 26.5%, 91.8% and 69.9% in 2011).
Figure 2-42 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 11.

Released Issue Edition: 1.0 47


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

20 19
18
16
16 15 15

Number of ANSPs
14
12
10 2011

8 2012
6
6 5
4
4
2
2 1 1
0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 2-42: SA11 – Number of ANSPs at each Maturity Level


There has been no change in the higher Maturity Levels in this Study Area from 2011.
However, the number of ANSPs reaching the minimum requirement level has increased by
three.
Although this Area was only examined in face-to-face interviews during the interviews this
year it is clear from comments made when discussing other Study Areas that, despite the
increase in maturity level for some ANSPs, there is a general misunderstanding what is
required. As an example, one ANSP commented that “We are a single Unit – best practice is
a single practice” The interviewee then said that the organisation did not have a system for
producing or collecting best practice but “kept an eye out for other ANSPs doing something
in a best practice way that would improve their own practice.”
Although the Questionnaire Clarification Document provides an explanation of best practice a
number of ANSPs stated that a clear definition is required in the context of this
measurement. However there is some indication that the sharing of data and exchange of
views on Best Practice is occurring particularly at international meetings such as the
EUROCONTROL Safety Team, regional ICAO meetings and FAB meetings.
This Study Area appears to be gradually improving, though there is a suspicion that some
ANSPs are over marking themselves, particularly in respect of the development of “Best
Practice”. Sometimes it is difficult to know if a “Good Practice” shared with others actually
becomes a “Best Practice” as there is no evidence to support this.
The percentage point movement from 2010 and between 2011 an 2012 is shown in
Figure 2-43.

Released Issue Edition: 1.0 48


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
20 SA11: Adoption & Sharing

Movement Comparison
15 of Best Practices
10
5
0
-5
-10
-15
-20
-25
Minimum Average Maximum
Inc 11/12 0.82 5.70 19.31
Dec 11/12 -0.17 -4.19 -13.83
Inc 10/11 0.46 7.00 17.75
Dec 10/11 -0.33 -5.40 -19.59

Figure 2-43: Minimum / Average / Maximum Change in Study Area 11

As can be seen from Figure 2-43, there is some difference in the changes in the extent to
which scores have decreased between 2010/2011 and 2011/2012. The maximum number of
percentage points between 2011/2012 is 5.7 percentage points better than between
2010/2011. The average decrease in percentage point movement has fallen from minus
5.4% in 2011 to minus 4.19% in 2012. The maximum increased percentage point movement
has risen slightly from 17.75% in 2011 to 19.31% in 2012. The minimum decreases in
movement are all below 1%.
25.00

20.00

15.00

10.00

5.00

0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-5.00 44

-10.00

-15.00

-20.00 Movement 2010/11


Movement 2011/12
-25.00

Figure 2-44: % Movement Comparisons


Figure 2-44 provides detail with regard to how ANSPs have progressed during the
introduction of the revised questionnaire in 2010. This graph indicates the percentage point
movement for each ANSP comparing 2011/2012 with 2010/2011 movements. Notably:
• Eighteen ANSPs have reduced their Maturity Score in this Study Area since 2011;
• Nineteen ANSPs have had a decrease in Maturity Score since the start of the current
Maturity questionnaire in 2010. Eight of these ANSPs experienced a decrease in
Maturity Score year on year;
• Twenty five ANSPs have increased their Maturity Score in this Study Area since
2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.

Released Issue Edition: 1.0 49


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

The above average increases in movement for this area may be partly explained by the
number of ANSPs that now assess themselves at Level 4.

2.3.13 ANSPs Weakest Areas

ANSPs' Weakest Areas (Maturity Scores) In 2012


15 14
Number of ANSPs

10 9

5
3 3 3
2
1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area

Figure: 2-45: Weakest Areas - Maturity Scores

Overall, the data in Figure 2-45 shows that Study Area 7: Safety Interfaces is the weakest
Study Area experienced by most ANSPs. This means that of all eleven study areas in the
survey, fourteen ANSPs recorded the lowest score of their measurement in SA7, while nine
other had SA9 as their lowest score and seven ANSPs had their lowest score in SA1.
The SAFREP Task Force proposed that all ECAC ANSPs should be at Level 3 for all Study
Areas by the end of the SES First Reference Period.
When comparing data against the objective to reach Level 3 for all Study Areas, a different
picture emerges. As can be seen in Figure 2-46 there are a number of ANSPs who are at
Level 1 or Level 2 within a number of Study Areas. SA1 (Safety Culture) and SA9
(Performance Monitoring) appear to be particularly weak. This is usually due to having one
question answered at a low level in those areas, but overall they do not necessarily arrive at
the lowest score due to the influence of other questions in the survey and the effect of the
weighting matrix (interdependencies between the questions).

Released Issue Edition: 1.0 50


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

12

10

Number of ANSPs
8

6
Level 1
4
Level 2
2

Study Area

Figure 2-46: Weakest Study Areas: Levels 1 and 2

Figure 2-46 shows those Study Areas where ANSPs are not yet meeting the regulatory
requirements. The reasons for this are more fully explained in the relevant areas in this
report for the respective Study Areas. However, significantly:
• Study Area 1: Safety Culture. A positive safety culture is emerging within ANSP
organisations, though in some ANSPs it is still immature, with the majority of ANSPs
being at Level 3 or 4. However none have yet reached Level 5 and there are still
eleven below Level 3 with some yet to commence the journey into establishing a
positive and proactive Safety Culture. Most ANSPs now recognise the importance of
Safety Culture and the need to measure it. They now understand that they need to
undergo Safety Culture surveys to establish where they are in developing a positive
and proactive safety culture in their organisation. A significant number of ANSPs have
chosen to undertake Safety Culture Surveys through EUROCONTROL and more are
planned before the end of 2012. It is therefore anticipated that the number of ANSPs
still below Level 3 in this Study Area will reduce further in the next year
• Study Area 9: Safety Performance Monitoring. There has been some regression in
this Study Area with some ANSPs at Level 3 in 2011 reassessing their performance
and marking themselves at Level 2 this year. Most ANSPs had some form of
monitoring in place, but the majority do not have a Quantitative system, nor do they
undertake internal comparative analysis. Many ANSPs do not see the need to
routinely share safety data with the general public, though a lessening of the
resistance to do this was detected. There are still 12 ANSPs below Level 3 and more
work is required to emphasis the importance of having a quantitative monitoring
system in place.
• Study Area 11: Adoption and Sharing of Best Practices. There has been little
movement in this area in 2012. It is clear that there is misunderstanding of what
constitutes “Best Practice” and how it should be assessed in the context of this Study
Area. There are eight ANSPs shown below Level 3. It is believed that there should be
more ANSPs below Level 4 as some refused to mark themselves below Level 4 even
though they could not show that they were using best practice on the basis that they
did not believe that it was possible to develop best practices in some of the study
areas. Greater explanation of the survey requirements in this area is required.
However there is an indication that ANSPs are exchanging data at international
meetings and this may be that the best way to educate ANSPs on what constitutes
“Best Practice” is in that type of fora.

Released Issue Edition: 1.0 51


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

2.3.14 ANSPs Strongest Areas

ANSPs' Strongest Areas In 2012


25
22

20
Number of ANSPs

15

10
7
6
5 4
2
1 1
0 0 0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area

Figure: 2-47: Strongest Areas - Maturity Scores

Overall, the data in Figure 2-47 shows that Study Area 3: Timely Compliance with
International Obligations is the strongest Study Area experience by ANSPs. This means that
of all eleven study areas in the survey, 22 ANSPs recorded the highest score of their
measurement in SA3, while seven others had SA4 as their highest score and six ANSPs had
their highest score in SA6
When comparing data against the objective to reach Level 3 for all Study Areas, a different
picture emerges. As can be seen in Figure 2-48 there are a number of ANSPs who are at
Level 4 or Level 5 in certain Study Areas. Of particular strength seem to be SA5 and SA6.

25

20
Number of ANSPs

15

Level 4
10
Level 5

0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area

Figure: 2-48: Strongest Areas – Levels 4 and 5


For instance, it can be seen that Study Area 5 is particularly strong in terms of those ANSPs
at Levels 4 and 5. This is because the Justification and Evidence submitted with the
questionnaires indicates that most organisations in this Study Area have well developed
training and competency processes in place and ANSPs are claiming development of Best
Practices in this area. Study Area 6 is also strong in terms of ANSPs at Levels 4 and 5
because many ANSPs already have risk management plans implemented or have a plan in
place to implement a risk management system. In other areas Level 4 is justified by ANSPs
using Best Practices shared within FABs.

Released Issue Edition: 1.0 52


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3. CANSO SURVEY RESULTS


This is the first year that CANSO Members have been included in the survey as a separate
section of the report. Previously CANSO Members have been included within the ECAC
Area only. Within the 44 ECAC States 38 ANSPs are CANSO Members, all but one of those
returned a questionnaire, and 13 additional non-ECAC ANSPs participated in the survey and
are included in this section.

3.1 Geographical Area


CANSO Members are worldwide and are collectively responsible for over 85% of the world's
airspace.

Figure 3-1: The CANSO Membership Area

3.2 CANSO Objectives


CANSO10 is an international association of ANSPs that control more than 65% of the world’s
Air Traffic. CANSO provides a forum for discussion of Air Traffic Management related issues,
where all aviation stakeholders unite to develop and exchange ideas in support of global Air
Navigation Services. CANSO represents the views and interests of Members at the relevant
international institutions.
As the global association of ANSPs, CANSO represents its members by coordinating joint
positions and speaking out on industry issues. CANSO acts as the global ANSP voice on
both regulatory and industry issues and coordinates closely with representatives of both
sides; ICAO, IATA, ACI and the many regional bodies.
CANSO Members are keen to improve the overall performance of the global ATM network.
This is achieved through close cooperation between Members in the different fields of ANSP
business operations. In order to support the development and promotion of best practices,
CANSO unites ANSP experts to stimulate exchanges of information and the development of

10
The first four paragraphs of this section are taken directly from the CANSO Website.
Released Issue Edition: 1.0 53
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

global policies. Regular meetings, workshops and conferences ensure that all ANSPs are
fully aware of current industry trends and developments. In addition CANSO coordinates
industry positions with customer representatives and other aviation stakeholders to ensure
global standards are developed and adopted.
The SoE Measurement aims to find how well CANSO Members are meeting the elements of
the CANSO SoE model.

3.3 Survey Findings – CANSO Overview


This Section contains the results of 3711 ECAC Members that are also Members of CANSO
plus 13 additional CANSO Members from around the world who participated in the 2012
survey. The majority of the additional ANSPs agreed to participate in the interview or answer
supplementary written questions, only two of the additional 13 ANSPs did not acknowledge
email requests.
Figure 3-2 shows the profile of average maturity of each participating CANSO ANSP across
all Study Areas in 2012. The level part of the graph on the left hand side showing zero is
created because only a few of the 13 CANSO ANSPs from outside of the ECAC area
participated in the 2010 and/or 2011 surveys. Those that did not participate in the survey in
2010 or in 2011 are shown as zero. It can be seen from the 2012 (green) line that there is a
much less steep climb in the graph once the additional CANSO ANSPs are included. This is
caused by there being far more ANSPs at Level 1 than in previous years due to the status of
some of the additional CANSO Members.

ANSP Average Maturity

100

90

80

70

60
Maturity (%)

50

40

30

2010 2011 2012


20

10

0
1 6 11 16 21 26 31 36 41 46 51

State Rank

Figure 3.2: CANSO Average Maturity


It can also be seen that the profile in this graph is quite different from the similar graph for the
ECAC only area (Figure 2-2). This is due to some non-ECAC CANSO Members having very
low maturity scores.

11
There are 38 Members of ECAC who are also Members of CANSO, but one ECAC ANSP did not return a questionnaire in
2012.
Released Issue Edition: 1.0 54
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Maturity Levels Distribution


2012

6%
20% Level 1
42%
32% Level 2
Level 3
Level 4

Figure 3-3: Maturity Levels Distribution for 2012


Figures 3-3 shows the distribution by levels for all CANSO ANSPs participating in the 2012
survey, with 42% of ANSPs reaching Maturity Level 3. Because the thirteen additional
CANSO ANSPs lie outside of the ECAC Area and the majority of those are only just
beginning to initiate implementation of an SMS they are still at Levels 1 and 2. CANSO has
the same proposed objective as ECAC of raising all their ANSP Members to Level 3 as soon
as feasible.
Figure 3.4 demonstrates the impact that the additional CANSO ANSPs have had on previous
year’s statistics. For 2012 there are an additional two ANSPs at Level 1 since 2011 and an
additional eight since 2010. The number at Level 2 has reduced by one and Levels 3 and 4
have increased by four and one respectively. There are no ANSPs at Level 5.

25

20
2010
Number of ANSPs

15 2011

2012
10

0
Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-4: Profile of CANSO Maturity Levels


Figure 3.5 looks at the results both in terms of overall score (Blue columns) and in terms of
level (Red diamonds). The former will give a high-level picture about the general status of the
organisation’s SMS, this allows a direct comparison with other organisations on the scale. It
shows whether overall the organisation is mostly managing performance or it is still in the
process of implementing the mandatory regulations and achieving the minimum standards of
maturity. However, this could hide particular problems in certain areas as the methodology
averages the scores by Study Area and there is a significant smoothing effect. In order to
identify whether an organisation still has a significant problem in at least one area, the level
view is more appropriate. By combining the two, a more complete picture can be build. Thus,
an organisation with high level and high score would show a consistent approach to all

Released Issue Edition: 1.0 55


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

objectives, while an organisation with a relatively high score but a low level would prove that
it may have left behind certain objectives, concentrating on others.

ANSP Average Maturity

100 5

90 Score Level

80 4

70

60 3
Maturity (%)

50

40 2

30

20 1

10

0 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

ANSP

Figure 3-5: Level-Score for all participating CANSO ANSPs


It should be noted that an organisation’s overall Maturity Level is defined to be the lowest
response to any Study Objective in any Study Area.

3.4 SURVEY Findings CANSO Members


The figures in this Section include 37 ECAC CANSO Members plus the 13 ANSPs from
outside of the ECAC area who participated in the survey. One Member did not submit a
questionnaire.
Mean
Study Area Minimum Maximum
2012
SA1 – Safety Culture 3.6 64.7 98.1
SA2 – Organisational and Individual Safety Responsibilities 4.1 65.9 98.6
SA3 – Timely Compliance with International Obligations 4.4 67.8 99.0
SA4 – Safety Achievement 3.3 66.5 98.3
SA5 – Competency 3.8 64.8 98.1
SA6 – Risk Management 4.0 65.3 98.2
SA7 – Safety Interfaces 2.4 65.2 98.4
SA8 – Safety Assurance 3.9 66.2 97.5
SA9 – Safety Performance Monitoring 4.1 65.7 97.6
SA10 – Organisational Safety Surveys and SMS Audits 3.7 65.0 97.4
SA11 – Adoption and Sharing of Best Practices 4.1 65.3 98.0
Table 3-1: Range of Maturity Scores for CANSO Members
The number of non-ECAC CANSO Members participating in the survey has tended to
change year on year. In 2010, four ANSPs participated; 2011: eight ANSPs and in 2012
thirteen ANSPs. Over the three years, only four ANSPs participated in each year. As a result,
trends are more difficult to detect. Currently the mean average within the grouping for 2012 is
65.66%. The maximum maturity is an overall 98.1%, which is higher than the ECAC area
maximum due to additional ANSPs with very mature SMS being included. The minimum
average is 3.76%, which is much lower than the similar figure for the ECAC area. This is due
to the number of participating CANSO ANSPs outside of the ECAC area that are at the very
early stages of SMS implementation.
Released Issue Edition: 1.0 56
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

When including the CANSO Members outside of the ECAC area 50 organisations
participated in this part of the survey. Not all of these additional ANSPs participated in the
follow-up interview process, but those that did (11 ANSPs) gave very detailed responses to
interview questions.

Range of Maturity

100 98.6 99.0 98.3 98.4


98.1 98.1 98.2 97.5 97.6 97.4 98.0

90

80
73.5 74.3 73.8 73.0
71.4 71.6 72.3 72.2 72.1 71.1 71.9
70

60 60.7 60.0 60.1 60.4 60.0 59.6


Maturity (%)

58.9 59.1 59.0 59.2


57.2
50

40

30

20

10
3.6 4.1 4.4 3.3 3.8 4.0 3.9 4.1 3.7 4.1
2.4
0
1 2 3 4 5 6 7 8 9 10 11
Min - Max limits
Study Area
Interquartile range

Figure 3-6: Range of Maturity


The range of Maturity Scores in each Study Area varies from 2.4% up to 99%, as shown in
Figure 3.6. The interquartile range is typically around 12%, indicating that ANSPs are
actually closely grouped in their assessment of Safety Maturity. Study Areas 3 and 7 are non
typical with an interquartile spread of over 15% and shows that the Area of Safety Interfaces
and Compliance with International Obligations are potentially weak Areas.
As illustrated in Figure 3-7, 36% of ANSPs reported a Maturity Level of 1 or 2 in response to
Study Area 1.2 (which calls for regular measurement of safety culture and an improvement
programme). This number is considerably more than for any other single Study Objective.
Conversely the Study Area with the least ANSPs at Levels 1 or 2 is Study Area 7 (requiring
safety-related internal interfaces to be effectively managed) which shows 11% at these
levels. Although this could be considered as in opposition to the interquartile range in Figure
3-6 it shows that although, overall, Study Area 7 (from a Maturity Score perspective) is a
weak area, the actual numbers of ANSPs at Levels 1 and 2 are low in comparison to other
Study Areas.

Released Issue Edition: 1.0 57


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

ANSP Response Category by Study Objective in 2012


100%

90%

80%
Percentage of Responses

70%

60%

50%

40%

30%

20%

10%

0%
SA1.1

SA1.2

SA1.3

SA2.1

SA2.2

SA2.3

SA2.4

SA3.1

SA3.2

SA4.1

SA4.2

SA4.3

SA5.1

SA6.1

SA7.1

SA7.2

SA8.1

SA8.2

SA8.3

SA9.1

SA9.2

SA9.3

SA10.1

SA11.1

SA11.2

SA11.3
Study Objectives A B C D E

Figure 3-7: Percentage of Answers in Each Maturity Category


Overall (Figure 3-8), organisations responded to the Study Objectives with a Maturity Level
of 1 or 2 in 211 times, compared with 316 times for Maturity Level 3 and 773 times for
Maturity Levels 4 and 5. Note that this does not equate directly to the numbers shown in
Table 3-2 which shows the lowest Maturity Level for any Study Objective given by an ANSP,
and the number of times they gave that response.

Gap to Next Maturity Level

25

20
Number at Minimum Level

15

10

0
3.76

19.92

33.42

37.81

43.76

44.37

59.97

60.83

63.29

67.62

4
68.83

70.24

72.54

73.59

75.07

3
75.97

76.87

80.02

81.03

2
82.24

82.88

84.56

86.66

87.36

1
Maturity
92.93

Level

Figure 3-8: Gaps to the next maturity level (CANSO ANSPs)

Released Issue Edition: 1.0 58


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

CANSO ANSPs Study Areas


Answers SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
A 7 7 2 5 4 3 5 7 11 2 14
B 25 25 12 16 4 7 6 10 19 3 17
C 47 48 18 21 4 7 28 46 41 17 39
D 59 62 35 50 24 20 43 39 56 19 50
E 12 58 33 58 14 13 18 48 23 9 30
A&B Total 32 32 14 21 8 10 11 17 30 5 31
A&B Total
% 21% 16% 14% 14% 16% 20% 11% 11% 20% 10% 21%
Table 3-2: Questionnaire Responses by Study Area
Table 3-2 shows the number of times participating CANSO ANSPs have selected a Maturity
Score for questions in the Study Areas. As can be seen, answers A and B are more
prominent in Study Area 1 (Safety Culture) and Study Area 11 (Adoption and Sharing of Best
Practices), where there is a total of 31 and 32 such answers (21% of overall responses from
CANSO ANSPs). Study Area 6 (Risk Management) and Study Area 9 (Safety Performance
Monitoring) also result in a significant percentage (20%) of answers A and B, indicating
weaknesses in those areas.
Specifically, there are 4 ANSPs who, overall, are at Level 1 or 2 in each Study Area.
Conversely, there are 24 ANSPs who are at Level 3 or higher. Those at Levels 1 and 2 are in
the very early stage of SMS implementation.

3.4.1 Key Findings12


Whilst the CANSO Members who had participated in the survey previously through their
membership of ECAC were familiar with the requirements of the study those from outside of
the ECAC area were less familiar. A few had answered the survey in 2010 (4 ANSPs) and
2011 (8 ANSPs), but were not included in the EUROCONTROL Reports for the respective
years, and for the rest the modus operandi of the survey was a new experience. Only four of
the non-EUR CANSO Members had participated in all three surveys. More time had to be
spent by the team on those ANSPs participating for the first time as there was
misunderstanding of the requirements and the impression was that they had not really
received sufficient briefing on the survey objectives.
A high percentage of the CANSO ANSPs included in the ECAC survey have achieved an
overall Level of 3 or better. As will be seen from the graphs and tables the fact that so many
of the new ANSPs are in the very early stages of implementing an SMS has a large effect on
the overall maturity scores for the CANSO Members.
Eleven of the new participants willingly participated in the follow-up interviews and gave
comprehensive responses to questions. Two of the ANSPs did not respond to email or
telephone requests for interviews and it is considered this is because of the very early stages
of SMS development that they are at. In one instance the organisation is undergoing a
fundamental realignment of Management and position within the State and is probably not in
a position to say more than they did in the Questionnaire.
The more developed SMS are within ANSPs that have been developing their SMS for at
least 5 years, some longer. The other ANSPs are only at the implementaion stage at best,
most have a rudimentary SMS with responsibilities already set out in an SMM, but some are
only just getting to grips with the ICAO requirements for an SMS.

5. Only Key Findings pertaining to the additional CANSO ANSPs that are not members of ECAC are included here.
Released Issue Edition: 1.0 59
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3.4.2 Study Area 1 – Development of a Positive and Proactive Safety Culture


Objectives:
1.1 A positive and pro-active just, flexible, and informed safety culture (the shared
beliefs, assumptions, and values regarding safety) that supports reporting and
learning led by management.
1.2 Regular measurement of safety culture and an improvement programme.
1.3 An open climate for reporting and investigation of occurrences.
The maturity profile for Study Area 1 ranges from a minimum of 3.6% to a maximum of
98.1%, with the average being 64.7%. Figure 3-9 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 1.

25

20
20
Number of ANSPs

15 14
11
10

5 4

1
0

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-9: Study Area 1 – Numbers of ANSPs at each Maturity Level


It can be seen from Figure 3-9 that for 2012 the majority of ANSPs have reached Level 3 or
better. Introduction of some of the additional CANSO ANSPs who are only just
understanding the requirements of Safety Culture mean that there are eighteen ANSPs
below Level 3. Overall there seems to be a positive Safety Culture developing, but there is
still work to do particularly for those ANSPs who are only just initiating an SMS. However, a
willingness to cultivate a Safety Culture was detected amongst those ANSPs that are only
just beginning to implement SMS.
This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. Whilst the majority of
ANSPs within ECAC are well aware of the need to develop a positive and proactive safety
culture and are reasonably advanced in not only developing one but are also well aware of
the need to measure the safety culture within their organisations, outside of ECAC the
situation is quite different. Most ANSPs are aware of the requirements of an SMS and the
need to develop a safety culture and many have programmes of workshops, seminars and
training courses to instil a safety culture, few have any idea how to measure the culture. The
exceptions are those ANSPs who have been developing their SMS for 5 years or more.
Outside of the ECAC Area one ANSP has achieved Level 5 and can demonstrate setting
best practice. This ANSP has individuals across the organisation who are proactively and
constantly striving to improve their approach to systematic safety management. They are
supported by measurement and review processes and organisational management. The
ANSP has also conducted a number of safety climate surveys. In addition, a just culture
policy is in place that identifies acceptable and unacceptable behaviour, and has been
communicated to all operational personnel. One ANSP is at Level 4, three are at Level 3 and
the remaining eight are at Levels 1 or 2.

Released Issue Edition: 1.0 60


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3.4.3 Study Area 2 – Organisational and Individual Safety Responsibilities

Objectives:

2.1 An approved, clearly documented, and recognised system for the management of
safety. Management structure, responsibilities, accountabilities and authorities
are clearly defined and documented.
2.2 A clearly defined safety management function that is independent of line
management.
2.3 An integrated safety planning process is adopted by the organisation with
published and measurable safety goals and objectives which are accountable to
the executive.
2.4 Clear understanding and acceptance of safety management responsibilities by all
staff and contractors.
The maturity profile for Study Area 2 ranges from a minimum of 4.1% to a maximum of
98.6%, with the average being 65.9%. Figure 3-10 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 2.

18
16
16 15
Number of ANSPs

14
12
10 9

8
6
6
4
4
2
0

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-10: Study Area 2 – Number of ANSPs at each Maturity Level


As with the ECAC area this Study Area is well developed with thirty seven ANSPs at target
Level 3 or better. Introduction of the additional ANSPs who are at the early stages of SMS
implementation means that thirteen ANSPs remain below the proposed target Level.
However all ANSPs have at least an ad-hoc system in place and most have nominated
somebody specifically responsible for safety. Figure 3-10 shows that six ANSPs claim to
have fully developed systems and are sharing Best Practice. Overall the ANSPs who were
participating in the survey for the first time were aware of what was required in this area, but
there were different levels of implementation due to internal management structures, lack of
manpower and cultural issues.
Within those ANSPs that have had an SMS established for a number of years and have
achieved at least Level 3, the SMS is robust, but for those ANSPs that are at Level 1 or Level
2 there is no formal process in place to assess the extent to which safety policy and SMS
processes are understood within the organisations. A few of the ANSPs at Level 1 and 2
have performed some form of gap analysis and most have at least rudimentary SMS
implementation plans. Most of these ANSPs are in the process of conducting training to
enable all personnel to understand the Safety policy and SMS processes.
There appears to be no formal process in most of these ANSPs to understand the likely
safety impact of the actions of operational personnel. However, most claim to have
discussions, meetings and interactions with staff to discuss these matters and to ensure that
staff understands the consequences of their actions.

Released Issue Edition: 1.0 61


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Outside of the ECAC Area three ANSPs has achieved Level 5, there are no ANSPs at Level
4, two are at Level 3 and the remaining eight are at Levels 1 or 2.

3.4.4 Study Area 3 – Timely Compliance with International Obligations

Objectives:

3.1 A formal SMS that meets all applicable safety requirements.


3.2 An organisation that strives to go beyond compliance, takes into account the
need to ensure, in a timely manner, that there are no inconsistencies with
European or national requirements or international safety standards.

The maturity profile for Study Area 3 ranges from a minimum of 4.4% to a maximum of
99.0%, with the average being 67.8%. Figure 3-11 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 3.

20 19
18
16
Number of ANSPs

14
12 11
10
10 9
8
6
4
2 1
0

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-11: SA3 – Number of ANSPs at each Maturity Level


Within the ECAC area there are no ANSPs at Level 1 so the one shown in Figure 3-11 at
Level 1 is from the additional CANSO ANSPs and six of those at Level 2 also. The chart
shows that forty ANSPs meet safety regulatory requirements and understand and meet
regional and international safety standards.
This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. This is a strong area for
both ECAC and non-ECAC Members in that the majority of ANSPs recognise the need to
comply with national and international safety standards, although there are varying degrees
of compliance with the requirements of this Study Area. Three of the non-ECAC ANSPs say
they are at Level 5, two at Level 4 and one at Level 3. The remaining seven are at Levels 1
and 2. One of the most common problems with timely compliance with international
regulations is that in some States regulations have to be taken into national law and also
translated into the national language.

Released Issue Edition: 1.0 62


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3.4.5 Study Area 4 – Safety Standards and Procedures

Objectives:

4.1 Clearly defined and documented safety standards and procedures.


4.2 Staff know about the safety and safety management requirements and standards,
which are regularly reviewed, assessed, and maintained.
4.3 Emergency/contingency response procedures and an emergency/contingency
response plan that documents the orderly and efficient transition from normal to
emergency operations and return to normal operations.

The maturity profile for Study Area 4 ranges from a minimum of 3.3% to a maximum of
98.3%, with the average being 66.8%. Figure 3-12 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 4.

25 23

20
Number of ANSPs

15

10
10
7 7

5 3

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-12: SA4 – Number of ANSPs at each Maturity Level


This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. From Figure 3-12 it is
clear that the majority of CANSO ANSPs consider safety standards and procedures to be
important and many have completed documentation of the SMS. There are only ten ANSPs
below Level 3 and a good percentage of those have some documentation and processes to
maintain safety management procedures exists.
Most systems within the ECAC area CANSO Member’s have redundant capability and some
form of emergency response plans are in place for the majority of ANSPs. The statistics
show that forty ANSPs mark themselves at Level 3 and above and with the ten ANSPs still
either initiating or beginning to implement their SMS this appears to be a strong growth area
for both the ECAC area and the wider CANSO area.
Three of the non-ECAC ANSPs say they are at Level 5, two at Level 4 and one at Level 3.
The remaining seven are at Levels 1 and 2. Most of those ANSPs at Levels 1 and 2 have
some processes and procedures in place though they have not been formally approved and
are still in development. Two do not have redundant capability but most have some form of
contingency plan in place.

Released Issue Edition: 1.0 63


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3.4.6 Study Area 5 – Competency

Objectives:

5.1 Staff, and contractors (where appropriate) are trained, competent in safety and
safety management, and where required, licensed.

The maturity profile for Study Area 5 ranges from a minimum of 3.8% to a maximum of
98.1%, with the average being 64.8%. Figure 3-13 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 5.

30

25 24
Number of ANSPs

20

15 14

10

5 4 4 4

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-13: Study Area 5 – Number of ANSPs at each Maturity Level


This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. Figure 3-13 shows that
thirty eight ANSPs have well developed competency with fourteen ANSPs claiming fully
developed systems that are continuously improving their competency methods, training
plans, risk mitigation and are also setting Best Practice.
Of the twelve ANSPs below Level 4 three are at the implementation stage whilst nine are still
initiating or planning implementation of their systems. Nonetheless, where this was
discussed, it seemed that those still initiating their SMS recognise the need for a method to
measure competency.
One of the non-ECAC ANSPs say they are at Level 5, five are at Level 4 and one at Level 3.
The remaining six are at Levels 1 and 2. Those at Levels 1 and 2 have training plans in place
and some have internal competency assessment processes. However much of the
Justification and evidence in the Study Area indicates that those ANSPs just beginning the
SMS implementation process only have ad hoc processes in place.

3.4.7 Study Area 6 – Risk Management

Objectives:

6.1 A continuing risk management process that identifies, assesses, classifies, and
controls all identified safety risks within the organisation, including potential
future risks.

The maturity profile for Study Area 6 ranges from a minimum of 4.0% to a maximum of
98.2%, with the average being 65.3%. Figure 3-14 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 6.

Released Issue Edition: 1.0 64


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25

20
20

Number of ANSPs
15 13

10
7 7

5 3

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-14: Study Area 6 – Number of ANSPs at each Maturity Level


As can be seen from Figure 3-14 thirty three ANSPs are at Level 4 or 5 and a further seven
at Level 3. These all have well developed systems that meet requirement, though the ones
at Level 3 are not yet mature. The remaining ten ANSPs either have nothing in place or what
they have is very rudimentary. Resources, both financial and human seem to be a problem
within some ANSPs. Whilst most of the CANSO ANSPs who are also members of ECAC are
separated from their NSA some of those outside of ECAC are still closely linked to their
Regulator and although functionally separated are not organisationally separated. One ANSP
is in fact an entirely military organisation.
Those ANSPs at Level 1 do not have any risk management in place nor do they have a
process for corrective action plans. Those at Level 2 have some form of risk management.
One ANSP reported they did not involve their Regulator unless there was an accident.
The Risk Analysis Tool (RAT) is widely used within the ECAC area but not so outside of it.
One ANSP from outside of ECAC commented that it believed the repeatability element of the
tool is not yet evolved sufficiently. The ANSP thought that the tool is more of a Severity
Assessment Tool, due to the amount of subjectivity involved and therefore would not
consider using it within their organisation. Another non-ECAC ANSP commented “A risk
analysis tool is used, however it is less elaborate than EUROCONTROL’s RAT and the tool
is best suited to the ANSP’s size and needs.”
Two of the non-ECAC ANSPs say they are at Level 5, two at Level 4 and three at Level 3.
The remaining six are at Levels 1 and 2. Most of these ANSPs are very aware of the
importance of risk management and most, but not all, have some form of risk management in
place. Some of the ANSPs at Level 2 have systems in place, but staff have not been fully
trained and in one instance the system, although documented, is not being used correctly.

3.4.8 Study Area 7 – Safety Interfaces

Objectives:

7.1 Effectively managed safety-related internal interfaces (e.g. quality management


system, security, and environment).
7.2 The effective management of external interfaces with a safety impact (e.g., MIL,
airspace users, airports). Formalised processes and procedures dealing with
external agreements, services, and supplies (e.g., cross-border Letters of
Agreement).

The maturity profile for Study Area 7 ranges from a minimum of 2.4% to a maximum of
98.4%, with the average being 65.4%. Figure 3-15 shows the number of ANSPs at each
Released Issue Edition: 1.0 65
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Maturity Level when measured against the objectives of Study Area 7.

25
21
20

Number of ANSPs
16
15

10
6
5 4
3

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-15: SA7 – Number of ANSPs at each Maturity Level


This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. From Figure 3-15 it can
be concluded that overall Safety Interfaces within CANSO are managed well. Only nine out
of the fifty ANSPs report that they are below Level 3. However the majority of ANSPs that
claim to be at Levels 4 and 5 are within the ECAC area.
One of the non-ECAC ANSPs say they are at Level 5, one at Level 4 and five at Level 3.
The remaining six are at Levels 1 and 2. It can therefore be conclude, with the two
exceptions, that interfaces outside of the European Region are less satisfactory than within.
Generally the ANSPs have interfaces in place and some are formal, however, the majority
say that LOAs, MOUs etc. are mostly informal, though some have partially formalised
interfaces. It would appear that this is a Study Area where improvements could easily be
achieved.

3.4.9 Study Area 8 – Safety Reporting, Investigation and Improvement

Objectives:

8.1 A continuing organisation-wide process to report and investigate safety


occurrences and risks.
8.2 An organisation-wide means to record and disseminate lessons learned.
8.3 Appropriate safety information and knowledge is shared with Industry
stakeholders. Information disclosure is compliant with agreed publication and
confidentiality policies/agreements.

The maturity profile for Study Area 8 ranges from a minimum of 3.9% to a maximum of
97.5%, with the average being 66.2%. Figure 3-16 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 8.

Released Issue Edition: 1.0 66


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

25
22

20

Number of ANSPs
15 13

10
7
5
5 3

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-16: SA8 – Number of ANSPs at each Maturity Level


As with the ECAC area CANSO ANSPs mostly have some form of reporting and
investigation systems with the majority having a well defined system with clear processes
that meet the regulatory requirement and the needs of their own organisations. Those
ANSPs that fall into Level 1 and 2 categories have some form of reporting system, though
not all have effective investigation systems. The greater emphasis on Safety Culture appears
to have helped to improve occurrence reporting as could the improvement of Just Culture.
Whilst the majority of the CANSO ANSPs understand the importance of disseminating data
as far as possible, both within their organisation and to others, some are reluctant to do so.
To some the occurrence information is confidential and others lack the mechanisms to
disseminate. There are varying degrees of confidentiality, in at least one ANSP occurrence
data is known only to those involved, whilst others are willing to share the information
internally and externally, but lack the mechanisms to do so. There are also a few States
where releasing data could cause repercussions with the judicial authorities, so the
occurrence data is deemed confidential and kept within the organisation.
There is still reluctance in some ANSPs to share data fully and a few ANSPs have stated that
their organisation lacks a clear policy on sharing of data, though the majority stated that they
share information with CANSO and at other international fora when it is appropriate to do so.
None of the non-ECAC ANSPs say they are at Level 5, two are at Level 4 and four at Level
3, the remaining seven are at Levels 1 and 2. Whilst most of these ANSPs, like those within
ECAC, have some form of reporting system, outside of Europe it is far less formalised.
Lessons learned are shared within the organisations, but most of the ANSPs at Level 1 and 2
do not have a formal system in place for sharing outside of their own organisation. There do
not appear to be many national legal restrictions on sharing data, but processes are not in
place, or in the very early stage of development, to do so.

Released Issue Edition: 1.0 67


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3.4.10 Study Area 9 – Safety Performance Monitoring

Objectives:

9.1 An established and active monitoring system that uses and tracks suitable safety
indicators and associated targets (e.g., lagging and leading indicators).
9.2 Methods to measure safety performance, which is compared within and between
ANSPs.
9.3 A general public knowledgeable of the ANSP’s performance through routine
publication of achieved safety levels and trends. (Information disclosure is
compliant with the requirements of ICAO Annex 13, Attachment E).
The maturity profile for SA 9 ranges from a minimum of 4.1% to a maximum of 97.6%, with
the average being 65.7%. Figure 3-17 shows the number of ANSPs at each Maturity Level
when measured against the objectives of Study Area 9.

25

20
20
Number of ANSPs

15
12 12

10

5
5
1
0

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-17: Study Area 9 – Number of ANSPs at each Maturity Level


This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. From Figure 3-17 it can
be seen that one ANSP believes that they have reached Level 5. Within Europe no ANSP
has claimed to be at this Maturity Level. One of the non-ECAC ANSPs say they are at Level
5, two are at Level 4 and two at Level 3, the remaining eight are at Levels 1 and 2.
It can be seen that this is not a strong area for those CANSO Members outside of ECAC.
With the exception of incident reports there is little performance monitoring apparent. Most of
those ANSPs at Level 1 and 2 do not have any systematic way to measure safety
performance and safety data is not normally shared with the general public. A few ANSPs
say they share data through their Annual Report, but it is not clear which data that covers.
Some blame lack of national legislation on the subject, whilst others say they will provide
data when required to do so. This is an area where improvements can be achieved by
ANSPs outside of the ECAC area.

3.4.11 Study Area 10 – Operational Safety Surveys and SMS Audits

Objectives:

10.1 Internal and independent (external) operational safety surveys and SMS audits.

The maturity profile for Study Area 10 ranges from a minimum of 3.7% to a maximum of
97.4%, with the average being 65%. Figure 3-18 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 10.
Released Issue Edition: 1.0 68
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

20 19
18 17
16

Number of ANSPs
14
12
10 9
8
6
4 3
2
2
0

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-18: Study Area 10 – Number of ANSPs at each Maturity Level


This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. From Figure 3-18 it can
be seen that the majority of the fifty ANSPs that responded to the survey are at Level 3 or
above.
Two of the non-ECAC ANSPs say they are at Level 5, two are at Level 4 and six at Level 3,
the remaining three are at Levels 1 and 2. It can therefore be seen that this is a much
stronger area for the non-ECAC CANSO Members with the majority having audit and safety
survey systems in place. The one ANSP at Level 2 has an audit system for Operations, but it
lacks any form of follow-up process and the organisation lacks development plans. Of the
two ANSPs at Level 1, one has nothing in place at all and the other conducts some audits on
an ad hoc basis, but they are not undertaken in a systematic manner.

3.4.12 Study Area 11 – Adoption and Sharing of Best Practices

Objectives:

11.1 A structured approach exists to promote safety, its standing within the
organisation and lessons learned through application of the SMS.
11.2 A structured approach to gather information on operational safety and SMS
best practises from the industry.
11.3 Sharing of safety and SMS-related best practises with industry stakeholders.

The maturity profile for Study Area 11 ranges from a minimum of 4.1% to a maximum of
98.0%, with the average being 65.6%. Figure 3-19 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 11.

Released Issue Edition: 1.0 69


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

20 19
18
16 15

Number of ANSPs
14
12
10
8 7 7
6
4
2
2
0

Level 1 Level 2 Level 3 Level 4 Level 5

Figure 3-19: Study Area 11 – Number of ANSPs at each Maturity Level


This Study Area was not discussed in any great depth during follow-up interviews. From
Justification and Evidence provided in the completed questionnaires and from the few
ANSPs where this was discussed some conclusions can be made. From Figure 3-19 it can
be seen that there are still 14 of the fifty ANSPs that responded to the survey below Level 3.
Most of those ANSPs recognise the importance of Safety Promotion and some have ad hoc
processes in place, these consist of seminars, irregular safety meetings etc. A few have
plans to put more structured systems in place, but most of those below Level 3 struggle with
the concept of Best Practice.
One of the non-ECAC ANSPs say they are at Level 5 and have developed a unique best
practice tool that allows timely analysis of air occurrences and incidents in 3-D with voice.
Two non-ECAC ANSPs are at Level 4 and two at Level 3, the remaining eight are at Levels 1
and 2. This is clearly a weak area for the non-ECAC ANSPs. Only one ANSP appears to
have a systematic way of collecting Best Practice, one or two claim that they collect Best
Practice whilst attending international meetings and seminars and some say they share
practices with aviation stakeholders. Most of those below Level 3 do not seem to understand
the principles of Best Practice and it would appear that clear guidance is required. This is
another area where improvements can easily be made by the non-ECAC ANSPs. Provision
of further guidance, particularly on Best Practices, would assist those ANSPs below Level 3
to improve their scores.

Released Issue Edition: 1.0 70


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3.4.13 ANSPs Weakest Areas

ANSPs' Weakest Areas (Maturity Scores) in 2012

20

16
Number of ANSPs

15

11

10
8

5
5
3 3
2
1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11

Study Area

Figure: 3-20: Weakest Areas - Maturity Scores

As in the ECAC results, overall the CANSO data in Figure 3-20 shows that Study Area 7:
Safety Interfaces, is the weakest Study Area experienced by most ANSPs. This means that
of all eleven study areas in the survey, sixteen ANSPs recorded the lowest score of their
measurement in SA7, while eleven other had SA10 as their lowest score and eight ANSPs
had their lowest score in SA1.
CANSO supports the SAFREP Task Force proposal that all ECAC ANSPs should be at Level
3 and considers that all CANSO Members should achieve the same level as soon as
feasible.
When comparing data against the requirement to reach Level 3 for all Study Areas, a
different picture emerges. As can be seen in Figure 3-21 there are a number of ANSPs who
are at Level 1 or Level 2 within a number of Study Areas with Study Area 7 being one of the
better ones. SA1 (Safety Culture) and SA9 (Performance Monitoring) appear to be
particularly weak. This is usually due to having one question answered at a low level in those
areas, but overall they do not necessarily arrive at the lowest score due to the influence of
other questions in the survey and the effect of the weighting matrix (interdependencies
between the questions).

Released Issue Edition: 1.0 71


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

14

12

10

Number of ANSPs
8

Level 1
6
Level 2

0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area

Figure 3-21: Weakest Study Areas: Levels 1 and 2


Figure 3-21 shows those Study Areas where ANSPs are not yet meeting the Objectives. The
reasons for this are more fully explained in the relevant areas in this report for the respective
Study Areas. However, significantly:
• Study Area 1: Safety Culture. Some of the additional CANSO ANSPs from outside of
the ECAC area are only just beginning to understand the requirements of Safety
Culture and there are eighteen ANSPs below Level 3. However, overall there seems
to be a positive Safety Culture developing, but there is still work to do particularly for
those ANSPs who are only just initiating an SMS, though a willingness to cultivate a
Safety Culture was detected amongst those ANSPs. One of the additional ANSPs
has achieved Level 5.
• Study Area 9: Safety Performance Monitoring. This is not a strong area for those
CANSO Members outside of ECAC. With the exception of incident reports there is
little performance monitoring apparent. Most of those ANSPs at Level 1 and 2 do not
have any systematic way to measure safety performance and safety data is not
normally shared with the general public. A few ANSPs say they share data through
their Annual Report, but it is not clear which data that covers. Some blame lack of
national legislation on the subject, whilst others say they will provide data when
required to do so. This is an area where improvements can be achieved by ANSPs
outside of the ECAC area.
• Study Area 11: Adoption and Sharing of Best Practices. This is clearly a weak area
for the n-ECAC ANSPs. None have any systematic way of collecting Best Practice,
one or two claim that they collect Best Practice whilst attending international meetings
and seminars and some say they share practices with aviation stakeholders. Most of
those below Level 3 do not seem to understand the principles of Best Practice and it
would appear that clear guidance is required. This is another area where
improvements can easily be made by the non-ECAC ANSPs. Provision of further
guidance, particularly on Best Practices, would assist those ANSPs below Level 3 to
improve their scores.

Released Issue Edition: 1.0 72


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

3.4.14 ANSPs Strongest Areas

ANSPs' Strongest Areas (MaturityScore) in 2012

30
26
25
Number of ANSPs

20

15

10
7
6 6
5
1 1 1 1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11

Study Area

Figure: 3.22: Strongest Areas - Maturity Scores


Overall, the data in Figure 3.22 shows that Study Area 3: Timely Compliance with
International Obligations is the strongest Study Area experience by ANSPs. This means that
of all eleven study areas in the survey, 26 ANSPs recorded the highest score of their
measurement in SA3, while seven others had SA4 as their highest score and six ANSPs had
their highest scores in SA7 and SA 8.
When comparing data against the requirement to reach Level 3 for all Study Areas, a
different picture emerges. As can be seen in Figure 3-23 there are a number of ANSPs who
are at Level 4 or Level 5 in certain Study Areas. Of particular strength seem to be SA5 and
SA 6

25

20
Number of ANSPs

15

Level 4
10
Level 5

0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area

Figure: 3-23: Strongest Areas – Levels 4 and 5


For instance it can be seen that Study Area 5 is a strong area with thirty eight ANSPs who
have well developed competency with fourteen ANSPs claiming fully developed systems that
are continuously improving their competency methods, training plans, risk mitigation and are
also setting Best Practice.
Study Area 6 is also a strong area with thirty three ANSPs at Level 4 or 5, all have well
developed systems that meet requirements.
Released Issue Edition: 1.0 73
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Intentionally Blank

Released Issue Edition: 1.0 74


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

4. CONCLUSIONS
1. Overall the maturity level of ANSPs in the ECAC Region has increased slightly since
the 2011 survey.
2. The introduction of the additional CANSO ANSPs has demonstrated that ECAC
ANSPs are in the main further advanced in terms of SMS implementation than
ANSPs outside of the ECAC Area, where many ANSPs are only in the early stages of
SMS implementation.
3. Relationships within FABs seem to have improved. In 2011 the survey reported that
there was evidence of tension building up within FABs, with smaller organisations
feeling that their larger partners are being too domineering. Similar evidence was not
detected this year anymore. A number of small ANSPs seem to appreciate the help
they can get from the more mature ANSPs, even if this help is often provided on a
commercial basis.
4. It was noticeable that more ANSPs CEO/DGs are personally involved in Safety
decisions. This is thought to be due to the increased emphasis on Safety Culture and,
in SES States, the requirement to follow EU legislation.
5. As reported in the 2011 survey report, based on the overall Maturity Scores, the
weakest areas for ANSPs is Study Area 7 (Safety Interfaces). This is for the ECAC
and CANSO areas. The data provides a different picture when looking at the
achieved Level. Significantly, ANSPs need to improve their progress in Study Area 1
(Safety Culture), Study Area 9 (Safety Performance Monitoring) and Study Area 11
(Adoption and Sharing of Best Practices). For CANSO ANSPs, improvements are
required across all areas.
6. The strongest area for ANSPs based on the overall Maturity Scores is Study Area 3
(Timely Compliance with International Obligations). When looking at the extent to
which ANSPs have reached Level 4 or Level 5, within the ECAC area, Study Area 4
(Safety Achievement) Study Area 5 (Competency) and Study Area 6 (Risk
Management) all produce good results. For the CANSO area, good results have been
achieved in Study Area 3, Study Area 4, Study Area 5, Study Area 6, and Study
Area 10 (Organisational Safety Surveys and SMS Audits).
7. The amount of Justification and Evidence in the questionnaires has improved, some
ANSPs providing comprehensive details. A reason for this was due to the additional
clarification provided by EUROCONTROL on how to answer the questionnaire.
8. The Clarification Document provided by EUROCONTROL was used by some ANSPs,
but not all. Where the guidance document was used extensively, ANSPs answered
more realistically – in some cases considerably changing their scores. This has had a
marked effect on the overall results in this year’s survey. Some ANSPs believe that
further guidance is required with regard to what is a good response (e.g. through
providing some sample answers). Furthermore, some ANSPs question the possibility
of developing best practice for some areas and would welcome further clarification in
this respect (with examples).
9. Among the additional CANSO ANSPs the more developed SMS are within ANSPs
that have been developing it for at least 5 years, some longer. The other ANSPs are
only at the implementaion stage at best.
10. Those ANSPs that underwent face-to-face interviews appreciated the interaction as
being a very useful exchange, which gave the opportunity to discuss the Study Areas
in more detail. ANSPs considered that these interviews made it easier to clarify any
issues and also enabled more people from an ANSP to participate in discussions.

Released Issue Edition: 1.0 75


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Intentionally Blank

Released Issue Edition: 1.0 76


APPENDICES

Appendix 1 Reference Material .........................................................................78

Appendix 2 Methodology...................................................................................84

Appendix 3 Question Mappings ........................................................................88

Appendix 4 Performance Level Definitions......................................................89

Appendix 5 ANSP Participation.........................................................................90

Appendix 6 Glossary..........................................................................................92

77
2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Appendix 1 Reference Material

The following References and legislation were used during the survey for purposes of
clarification for survey participants, both European and non-European. They are valid at the
date of publication, but as the legislative environment is dynamic and changes constantly, it
may not be accurate when used at a later date.
These References should be used for clarification purposes only and do not form a
formal legislative Document.

ICAO Requirements
The ICAO Safety Management Manual (Doc 9859 AN/474) provides States with guidance on
how to develop a regulatory framework and provides supporting guidance material for the
implementation of Safety Management Systems (SMS) by service providers.
The following International Standards and Recommended Practices (SARPs) contained in
the ICAO Annexes mentioned below pertain to questions in the EUROCONTROL/CANSO
Standard of Excellence Measurement questionnaire where appropriate:
• Annex 1 — Personnel Licensing;
• Annex 6 — Operation of Aircraft;
• Annex 8 — Airworthiness of Aircraft;
• Annex 11 — Air Traffic Services;
• Annex 13 — Aircraft Accident and Incident Investigation;
• Annex 14 — Aerodromes.
The ICAO safety management SARPs are contained in Annexes 1; 6, Parts I and III; 8; 11;
13 and 14. These Annexes address the activities of approved training organisations,
international aircraft operators, approved maintenance organisations, organisations
responsible for type design and/or manufacture of aircraft, air traffic service providers and
certified aerodromes. In the case of Annex 1, the safety management SARPs are limited
exclusively to approved training organisations that are exposed to safety risks during the
provision of their services.
The safety management SARPs are aimed at two audience groups: States and service
providers. In the context of the ICAO Safety Management Manual, the term “service provider”
refers to any organisation providing aviation services. The term thus encompasses approved
training organisations that are exposed to safety risks during the provision of their services,
aircraft operators, approved maintenance organisations, organisations responsible for type
design and/or manufacture of aircraft, air traffic service providers and certified aerodromes,
as applicable.
The ICAO safety management SARPs address the following three distinct requirements:
a) the State safety programme (SSP), including the acceptable level of safety (ALoS) of
an SSP;
b) safety management systems (SMS), including the safety performance of an SMS;
and
c) management accountability vis-à-vis the management of safety during the provision
of services.

Released Issue Edition: 1.0 78


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

The ICAO safety management SARPs introduce the notion of acceptable level of safety
(ALoS) as the way of expressing the minimum degree of safety that has been established by
the State and must be assured by an SSP, and the notion of safety performance as the way
of measuring the safety performance of a service provider and its SMS.

SES Legislation (European ANSPs Only)


The following lists and briefly summarises the SES Legislation in force on 1 December 2011.

Commission Regulation (EC) No 549/2004 laying down the Framework for the Creation
of the Single European Sky
The Framework Regulation, supported by three other Regulations (Air Navigation Service
Provision, Airspace and Interoperability) is designed to create a European Airspace
conceived and managed as a single continuum (the Single European Sky - SES) to optimise
the safety and efficiency of the European Air Traffic Management Network (EATMN).
Commission Regulation (EC) No 550/2004 on the Provision of Air Navigation Services
in the Single European Sky
This Regulation sets out an authorisation system, compliance review mechanism and revised
payment arrangements for the provision of air navigation services within the community.
Commission Regulation (EC) No 551/2004 on the Organisation and use of Airspace in
the Single European Sky
This Regulation sets out a mechanism to establish a single coherent Community airspace
with common design, planning and management procedures.
Commission Regulation (EC) No 552/2004 on the Interoperability of the European Air
Traffic Management Network
This Regulation is designed to achieve interoperability between the Community's air
navigation service providers and the creation of an internal market in equipment, systems
and associated services.
Commission Regulation (EC) No 1070/2009 Amending Regulations (EC) No 549/2004,
(EC) No 550/2004, (EC) No 551/2004 and (EC) No 552/2004 in order to Improve the
Performance and Sustainability of the European Aviation System
This Regulation amends Regulations (EC) No 549/2004 (The Framework Regulation), (EC)
No 550/2004 (Air Navigation Service Provision), (EC) No 551/2004 (Airspace) and (EC) No
552/2004 (Interoperability).
Commission Regulation (EC) No 2096/2005 laying down Common Requirements for
the Provision of Air Navigation Services
This Regulation lays down in detail the Common Requirements (CRs) for air navigation
service provision (includes ATS. Met Services, AIS and CNS/ATM provision). (Amended by
Commission Regulation (EC) No 1315/20077, Commission Regulation (EC) No 482/20088,
Commission Regulation (EC) No 668/20089) and Commission Regulation (EU) No 691/2010.
Commission Regulation (EC) No 2150/2005 laying down Common Rules for the
Flexible use of Airspace (FUA)
This Regulation reinforces and harmonises the application with the SES of the Concept of
FUA.
Commission Regulation (EC) No 730/2006 on Airspace Classification and Access of
Flights Operated under Visual Flight Rules above Flight Level 195

Released Issue Edition: 1.0 79


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

This Regulation establishes a harmonised airspace classification (Class C) to be applied by


Member States above flight level 195 (19,500 ft) and lays down harmonised requirements for
access of flights operated under Visual Flight Rules (VFR) to this airspace.
Directive 2006/23/EC on a Community Air Traffic Controller Licence
The objective of this Directive is the harmonisation of air traffic controller training and
licensing standards and mutual recognition of licences between Member States.
Commission Regulation (EC) No 1033/2006 laying down the Requirements on
Procedures for Flight Plans in the Pre-flight Phase for the Single European Sky
This Regulation lays down the requirements on procedures for flight plans in the pre-flight
phase in order to ensure consistency of flight plans, repetitive flight plans and associated
update messages between operators, pilots and air traffic services units through the
Integrated Initial Flight Plan Processing System.
Commission Regulation (EC) No 1032/2006 laying down Requirements for Automatic
Systems for the Exchange of Flight Data for the purpose of Notification, Co-ordination
and Transfer of Flights between Air Traffic Control Units
This Regulation lays down requirements for the automatic exchange of flight data for the
purpose of notification, co-ordination and transfer of flights between air traffic control units
and for the purpose of civil-military co-ordination.
Commission Regulation (EC) No 1794/2006 Draft Commission Regulation laying down
a Common Charging Scheme for Air Navigation Services
This Regulation lays down the necessary measures for the development of a charging
scheme for air navigation services which is consistent with the Eurocontrol Route Charges
System and shall apply to air traffic service providers designated under Article 8 of the
Service Provision Regulation (550/2004)2.
Commission Regulation (EC) No 633/2007 laying down Requirements for the
Application of a Flight Message Transfer Protocol used for the purpose of Notification,
Co-ordination and Transfer of Flights between Air Traffic Control Units
This Regulation defines the peer to peer communication protocol - the FMTP - for the
automatic system information exchanges between air traffic control units and for civil-military
co-ordination.
Commission Regulation (EC) No 1265/2007 laying down Requirements on Air-Ground
Voice Channel Spacing for the Single European Sky (8.33 kHz)
Eurocontrol is mandated to develop a draft interoperability Implementing Rule for the
deployment of air-ground voice communications based on reduced 8.33 kHz channel
spacing.
Commission Regulation (EC) No 1315/2007 on Safety Oversight in Air Traffic
Management and Amending Regulation (EC) No 2096/2005
This Regulation establishes a safety oversight function to be carried out by National
Supervisory Authorities concerning air navigation services, air traffic flow management and
airspace management by adopting the provisions of ESARR1 (ATM safety oversight).
Commission Regulation (EC) No 219/2007 on the Establishment of a Joint Undertaking
to Develop the New Generation European Air Traffic Management System (SESAR)
This Regulation establishes the Joint Undertaking to develop the new generation of
European air traffic management.
Commission Regulation (EC) No 482/2008 Establishing a Software Safety Assurance
System to be Implemented by Air Navigation Service Providers and Amending Annex
II to Regulation (EC) No 2096/2005

Released Issue Edition: 1.0 80


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

This Regulation lays down the requirements for the definition and implementation of a
software safety assurance system by air traffic service providers.
Commission Regulation (EC) No 668/2008 amending Annexes II to V of Regulation
(EC) No 2096/2005 laying down common requirements for the provision of air
navigation services, as regards working methods and operating procedures
This Regulation amends the text of Annexes II to V of the Common Requirements
Regulation
Commission Regulation (EC) No 29/2009 laying down requirements on data link
services for the single European sky
This Regulation lays down requirements for the coordinated introduction of data link services
based on air-ground point-to-point data communications as defined in Article 2(5).
Commission Regulation (EC) No 262/2009 laying down requirements for the
coordinated allocation and use of Mode S interrogator codes for the single European
sky.
This Regulation lays down requirements for the coordinated allocation and use of Mode S
interrogator codes for the purposes of the safe and efficient operation of air traffic
surveillance and civil-military coordination.
Commission Regulation (EU) No 73/2010 laying down requirements on the quality of
aeronautical data and aeronautical information for the the the single European sky.
This Regulation lays down requirements for the quality of aeronautical data and aeronautical
information (ADQ) in terms of accuracy, resolution and integrity.
Commission Regulation (EU) No 255/2010 laying down common rules on air traffic
flow management.
This Regulation lays down the requirements for air traffic flow management (ATFM) in order
to optimise the available capacity of the European air traffic management network and
enhance ATFM processes.
Commission Regulation (EU) No 691/2010 laying down a performance scheme for air
navigation services and network functions and amending Regulation (EC) No
2096/2005 laying down common requirements for the provision of air navigation
services.
This Regulation lays down the necessary measures to improve the overall performance of air
navigation services and network functions for general air traffic within the ICAO EUR and AFI
regions where Member States are responsible for the provision of air navigation services with
a view to meeting the requirements of all airspace users.
Commission Regulation (EU) No 1191/2010 amending Regulation (EC) No 1794/2006
laying down a common charging scheme for air naviagtion services.
This Regulation amends the Charging Regulation (EC) No 1794/200616 in line with the
Performance Scheme IR and the introduction of determined costs from 2012.
Commission Regulation (EU) No 176/2011 on the information to be provided before the
establishment and modification of a functional airspace block.
This Regulation lays down the requirements for the information to be provided by the
Member States concerned to the Commission, EASA and other Member States and
interested parties before the establishment and moification of a FAB.

Released Issue Edition: 1.0 81


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Commission Regulation (EU) No 677/2011 laying down detailed rules for the
implementation of air traffic management (ATM) network functions and amending
Regulation (EU) No 691/2010.
This Regulation lays down detailed rules for the implementation of air traffic management
(ATM) network functions in accordance with Article 6 of Regulation (EC) No 551/2004 in
order to allow optimum use of airspace in the single European sky and ensure that airspace
users can operate preferred trajectories, while allowing maximum access to airspaces and
air navigation services.
Commission Implementing Regulation (EU) No 1216/2011 amending Commission
Regulation (EU) No 691/2010 laying down a performance scheme for air navigation
services and network functions.
This Regulation amends the text of CR 691/2010 the Performance Regulation.

The Tables below are extracted from EASA’s Notice of Proposed Amendment (NPA)
No 2011-18 and show the Mapping between the SES Management Objectives and the
Survey Study Areas.

Released Issue Edition: 1.0 82


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Released Issue Edition: 1.0 83


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Appendix 2 Methodology

A2.1 Overview
The methodology is described in detail in ATM Safety Framework Maturity Survey –
Methodology for ANSPs (The document was distributed with the survey Questionnaires), and
summarised in Figure A2-1
.

Figure A2-1: Survey Methodology

Released Issue Edition: 1.0 84


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

A2.2 Study Areas


The study areas for ANSPs were developed by EUROCONTROL in consultation with the
main stakeholder groups and industry bodies. From this the questionnaire was derived,
tested and refined during a pilot survey in 2009. In the summer of 2010 EUROCONTROL
distributed the questionnaire to all of the ANSPs in the ICAO EUR Region (with ICAO
support for States outside the SES region).

ANSPs were asked a total of 26 questions grouped into 11 Study Areas, see Table A2-1
below.

ANSP Study Areas


SA1 – Safety Culture
SA2 – Organisational and Individual Safety Responsibilities
SA3 – Timely Compliance with International Obligations
SA4 - Safety Achievement
SA5 – Competency
SA6 – Risk Management
SA7 – Safety Interfaces
SA8 – Safety Assurance
SA9 – Safety Performance Monitoring
SA10 – Organisational Safety Surveys and SMS Audits
SA11 – Adoption and Sharing of Best Practices
Table A2-1: Study Areas

A2.3 Capability Maturity Model


For each question there are five possible answers, from Initiating to Continuous
Improvement, based on a Capability Maturity Model, see Figure A2-2: . An organisation
should achieve/match all of the characteristics at one maturity category before they can
consider moving up to the next. Even if many aspects of a higher category are applicable, a
lower level of maturity should be selected if the higher category is not fully satisfied.

Figure A2-2: Maturity Categories

Once the completed questionnaires are received, the organisation’s nominated focal points
are contacted to discuss the results and seek supporting evidence. In a number of cases the
responses to specific questions were revised up or down based on the interview, and a
revised questionnaire submitted for final analysis.

Since the methodology relies on a self-assessment, the SAFREP TF is seeking to increase


the robustness of the maturity measurements by:

• Encouraging peer surveys under the umbrella of EUROCONTROL;

Released Issue Edition: 1.0 85


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

• Visiting 5-10 organisations to conduct a short face-to-face detailed review of the


questionnaire. States would be selected at random, although it is hoped that every
State could be visited during the first reference period; and
• Taking into account the existing data (e.g. LSSIP) and audit reports (e.g. IUSOAP,
ESIMS).

The site visits reports and a comparison of the survey findings against data already held by
EUROCONTROL have been incorporated into a separate confidential report.

A2.4 Numerical Analysis


EUROCONTROL, in discussion with its stakeholders, has weighted each question (0 – 5)
according to its relevance to each Study Area. The responses provided by ANSPs on their
questionnaires are also assigned a numerical value (0 - 4 corresponding to categories A - E).

Mathematically, the maturity score is calculated from the questionnaire responses and
weighting factors as follows:
ni , j
100∑ rk , j ,i ⋅ wk , j
S i, j = k =1
ni , j

4∑ wk , j
k =1
Where:

Si, j is the maturity score for State i in Study Area j.


rk, j, i is the numeric value of the response of State i to question k in Study Area j
wk, j is the weight factor of question k to Study Area j
ni, j is the number of questions in Study Area j for which non-nil responses were
provided by the State i.

An overall score for each State is then also estimated by taking the average of the scores
over all Study Areas.

A2.5 Telephone Interview Procedure

Preparation

• Contact the interviewee by e-mail or telephone to agree a time and date for the
interview;
• Confirm the appointment in UCT and local time;
• Read the available background material.

Interview

• At the appointed time, make the call;


• Confirm that the interviewee is free to talk and establish how long they have
available;
• Introduce yourself and how the interview fits in with the overall aims of the project;
• Ask the interviewee to describe their organisation and their role within it;

Released Issue Edition: 1.0 86


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

• Work through each response in the questionnaire seeking evidence to support the
maturity score chosen;
- Ask the interviewee to explain why the particular category was chosen;
- Use open questions;
- Let the interviewee do most of the talking;
- Use the prompts in the ANSP Interview Questions to challenge reasoning;
- Explore issues that aid or hinder progress;
- If necessary seek additional confirmation by supplementary e-mail;
- Clarify anything that you do not understand or which seems to conflict with
earlier statements;
- Try to understand whether you are being given an honest opinion or one
from which they are unwilling to move;
- Do not push too hard otherwise the interviewee will dry up and give you
nothing;
- If what you are being told agrees with the level of maturity selected, move
on;
- Record all relevant details and private notes in your project logbook;
• Confirm any actions that have been agreed (e.g. to provide information, change
maturity levels etc.);
• Ask whether there is anything you can pass on to EUROCONTROL on their
behalf;
• Confirm that the interviewee has your contact details just in case they decide they
want to provide more information;
• Thank the interviewee for their time and end the call.

Post Interview

• Update the Status Report;


• Undertake any actions you have agreed to;
• Monitor actions the interviewee has agreed to undertake;
• Write up the interview into the Repository;
• Inform the Project Manager if a new Questionnaire has been created with revised
scores.

Released Issue Edition: 1.0 87


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Appendix 3 Question Mappings

A3.1 Mapping of ANSP Questions to Study Areas

Study Areas
Questions SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
SA1-1 5 5 5 3 5 4 4 5 4 4 4
SA1-2 5 4 0 1 5 1 0 0 0 4 4
SA1-3 4 3 1 2 4 2 0 5 5 3 3
SA2-1 4 4 4 4 4 4 4 5 4 4 3
SA2-2 2 5 5 3 2 3 1 2 2 2 3
SA2-3 5 5 3 3 5 4 3 4 4 4 4
SA2-4 2 5 5 3 2 3 1 2 2 2 3
SA3-1 1 2 5 2 1 2 1 2 1 2 2
SA3-2 1 3 5 3 1 1 1 2 2 2 2
SA4-1 2 5 2 5 2 2 2 2 2 2 3
SA4-2 3 3 1 4 3 4 1 4 4 4 4
SA4-3 3 2 0 3 3 4 3 3 4 4 5
SA5-1 4 5 3 2 5 4 2 4 4 4 3
SA6-1 4 5 3 4 4 5 4 4 4 4 3
SA7-1 5 5 0 4 5 5 5 4 4 5 3
SA7-2 3 3 2 3 3 4 5 3 3 2 2
SA8-1 5 3 1 2 5 0 0 5 5 2 3
SA8-2 5 2 0 3 5 4 2 5 4 4 5
SA8-3 3 3 2 3 3 4 4 4 3 2 2
SA9-1 2 3 2 2 2 2 2 4 5 5 2
SA9-2 4 2 0 2 4 4 2 3 5 3 2
SA9-3 4 2 0 3 4 4 2 4 5 4 2
SA10-1 4 4 2 4 4 4 4 4 4 5 5
SA11-1 4 3 0 3 4 3 3 2 4 4 5
SA11-2 4 3 0 3 4 3 3 2 4 4 5
SA11-3 3 2 0 1 3 2 4 5 4 2 4

Those questions highlighted in Tan have a weighting of four or five and hence have a major
contribution to the maturity score for the Study Area. States wishing to improve their overall
maturity in a particular area would be advised to concentrate on improving their response to
these questions.
As there are a significant number of questions with high weightings across many Study
Areas, this will effectively smooth out the maturity scores in each area

Released Issue Edition: 1.0 88


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Appendix 4 Performance Level Definitions

Released Issue Edition: 1.0 89


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Appendix 5 ANSP Participation

The following table provides a summary of the participation as at 28th September 2012.

ANSP PARTICIPATION
State Questionnaires Interview Report
Score
Issued Returned Completed Completed
Albania YES YES YES YES
Argentina YES YES NO YES
Armenia YES YES YES YES
Australia YES YES YES YES
Austria YES YES YES YES
Azerbaijan YES YES YES YES
Belgium YES YES YES YES
Bosnia & Herzegovina YES YES YES YES
Bulgaria YES YES YES YES
Canada YES YES YES YES
Croatia YES YES YES YES
Curacao YES YES NO YES
Cyprus* YES YES YES YES
Czech Republic YES YES YES YES
Denmark YES YES YES YES
Estonia YES YES YES YES
Finland YES YES YES YES
France YES YES YES YES
Macedonia* (FYROM) YES YES YES YES
Georgia YES YES YES YES
Germany YES YES YES YES
Greece* YES YES YES YES
Hungary YES YES YES YES
Iceland* YES YES YES YES
India YES YES YES YES
Ireland YES YES YES YES
Italy YES YES YES YES
Jordan YES YES YES YES
Latvia YES YES YES YES
Lithuania* YES YES YES YES
Luxembourg YES YES YES YES
Maastricht UAC YES YES YES YES
Malta YES YES YES YES
Moldova YES YES YES YES
Monaco YES YES NO YES
Montenegro YES YES YES YES
Mozambique YES YES YES YES
Netherlands YES YES YES YES
New Zealand YES YES YES YES

Released Issue Edition: 1.0 90


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Norway YES YES YES YES


Poland YES YES YES YES
Portugal YES YES YES YES
Romania YES YES YES YES
Serbia YES YES YES YES
Singapore YES YES YES YES
Slovak Republic YES YES YES YES
Slovenia YES YES YES YES
South Africa YES YES YES YES
Spain YES YES YES YES
Sweden YES YES YES YES
Switzerland YES YES YES YES
Tanzania YES YES YES YES
Turkey YES YES YES YES
Uganda YES YES YES YES
Ukraine YES NO NO NO
United Kingdom YES YES YES YES
US YES YES YES YES

Notes:

1. Green Coloured name indicates CANSO Member.


2. * indicates a detailed face-to-face interview.
3. EUROCONTROL distributed questionnaires to all ECAC States and CANSO
distributed questionnaires to Member ANSPs outside of the ECAC area.

Released Issue Edition: 1.0 91


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Appendix 6 Glossary

Acronym or Term Meaning


ACC Area Control Centre
AirProx Report to authorities by pilot or ATCO when separation standards have been
compromised
AIS Aeronautical Information Service
ANSP (or ASP) Air Navigation Services Provider
APP Approach
AST Annual Summary Template
ATCO Air Traffic Control Officer
ATM Air Traffic Management
ATS Air Traffic Service
ATSEP Air Traffic Safety Electronic Personnel
CAA Civil Aviation Authority
CAAct Civil Aviation Act
CAD Civil Aviation Department
CANSO Civil Air Navigation Services Organisation
CRs Single European Sky Regulations of the European Community
DG Director General
EAD European Aeronautical Information Service Database
EASA European Aviation Safety Agency
EC European Commission
EMAC European Civil Aviation Conference
ECCAIRS European Co-ordination Centre for Aviation Incident Reporting Systems
ESARR EUROCONTROL Safety Regulatory Requirement
ESIMs ESARR Implementation Monitoring and Support (ESIMS) Programme.
ESP European Safety Programme for ATM
EU European Union
FAB Functional Airspace Blocks
FAA Federal Aviation Authority
FIR Flight Information Region
FoI Freedom of Information Act
IANS Institute of Air Navigation Services
ICAO International Civil Aviation Organisation
IFR Instrument Flight Rules
ISIS Implementation of the Single European Sky In South East Europe
IUSOAP ICAO Universal Safety Oversight Audit Programme
LSSIP Local Single Sky ImPlementation
‘Just Culture’ A culture in which front line operators or others are not punished for actions, omissions
or decisions taken by them that are commensurate with their experience and training,
but where gross negligence, wilful violations and destructive acts are not tolerated.’
KPI Key Performance Indicator
MoT Ministry of Transport
NSA National Aviation Safety Authority

Released Issue Edition: 1.0 92


2012 EUROCONTROL/CANSO Standard of Excellence Measurement

Acronym or Term Meaning


OAT Operational Air Traffic
OII Office for Incident Investigation
OJT On the Job Trainer/Training
PC Provisional Council
Regulator Regulator, often the National Civil Aviation Authority.
SAFREP TF Safety Data Reporting and Data Flow Task Force
SASI Support to ANSP Safety Management System Implementation
SES Single European Sky
SISG Safety Improvement Sub Group
SLA Supplementary Letter of Agreement
SMM Safety Management Manual
SMS Safety Management System
SMU Safety Management Unit
SRC Safety Regulation Commission
TCA Terminal Control Areas
TLS Target Level of Safety
TOKAI Tool Kit for ATM Occurrence Investigation
VFR Visual Flight Rules

Released Issue Edition: 1.0 93

You might also like