Professional Documents
Culture Documents
EUROCONTROL/CANSO
Standard of Excellence
2012 Measurement
Overall Status of ANSPs
DOCUMENT CHARACTERISTICS
TITLE
EUROCONTROL/CANSO Standard of
Excellence 2012 Measurement
Publications Reference:
ISBN Number:
Document Identifier Edition Number: 1.0
Edition Date: 16/11/2012
Abstract
This Document summarises the findings of the 2012 EUROCONTROL/CANSO Standard of
Excellence Measurement.
Keywords
ANSP Excellence Survey Findings
CANSO EUR Region EUROCONTROL
Authors
DOCUMENT APPROVAL
The following table identifies all management authorities who have successively approved
the present issue of this document.
It may be copied in whole or in part providing that the copyright notice and disclaimer are included and permission is granted by
EUROCONTROL DNM/COO/NOM/Safety. The information contained in this document may not be modified without prior written
permission from the EUROCONTROL DNM/COO/NOM/Safety.
The views expressed herein do not necessarily reflect the official views or policy of EUROCONTROL, which makes no warranty,
either implied or express, for the information contained in this document, neither does it assume any legal liability or
responsibility for the accuracy, completeness or usefulness of this information.
Printer by EUROCONTROL, 96, rue de la Fusée, B-1130 Brussels, Belgium, Tel: + 32 2 729 3956, Fax: + 32 2 729 9108.
Contents
DOCUMENT CHARACTERISTICS.............................................................................ii
1. Introduction.............................................................................................11
1.1 Background..............................................................................................................11
1.2 Survey Objectives....................................................................................................12
1.3 Study Areas .............................................................................................................13
1.4 Maturity Score and Maturity Level ...........................................................................13
4. Conclusions ............................................................................................75
APPENDICES ...........................................................................................................77
Appendix 2 Methodology...................................................................................84
Appendix 6 Glossary..........................................................................................92
Executive Summary
Forty three of the main ANSPs1 from 44 States within the ECAC Region submitted
questionnaire responses in time for inclusion in this report, compared with only 40 States in
2011. San Marino was not included in the survey as that State is covered by Italy, but the
Maastricht Upper Air Control Centre (MUAC) is included, making 44 ANSPs covered from
the ECAC area.
This is the first year that CANSO members worldwide have been included in the survey
report; previously CANSO members have been included but only those within the ECAC
Area have been included in the survey report. Non-ECAC CANSO members were previously
surveyed on an individual basis, without performing a group analysis. Results for
participating CANSO members for the 2012 survey are reported on collectively in a separate
section of the survey report. The CANSO Section of the report includes the 38 ANSPs within
the ECAC Area who are CANSO Members plus 13 other members from outside of the ECAC
Area.
The survey followed the same methodology as in 2010 and 2011, looking into eleven Study
Areas, according to the Standard of Excellence (SoE) model, which is explained in Part 1
(Introduction) of this report. The questionnaire responses were followed up by face-to-face
interviews where all Study Areas were discussed in detail, or by telephone interviews which
focused only on the following Study Areas (in order to optimise the time for the telephone
interviews):
• Study Area 2 (Organisational and Individual Safety Responsibilities)
• Study Area 6 (Risk Management), and
• Study Area 8 (Safety Assurance).
In addition, some ANSPs provided updates during telephone interviews on Study Areas not
included in the three targeted areas. Information derived from those sources has also been
included in the overall and individual ANSP reports. Five ANSPs participated in in-depth
face-to-face interviews.
The data used for the analyses published in this report were accurate at the time of the
publication (16 November 2012). Any changes in the ANSPs’ data that may have occurred
after this date could not be included in the data sample. ANSPs participating in the survey
conducted by EASA under the framework of the SES legislation may arrive at different
values than those reported during this exercise. All potential changes will be taken into
account for the next year’s baseline.
ECAC
The average Maturity Score for each participating organisation within the ECAC region is
shown in Figure E-1. A return result of zero indicates that there was no return for that
particular year and not that the Maturity Score is zero.
1
In some instances there is only one ANSP in each State, but in others there are a number. This survey focuses on the main
ANSP in each State.
Released Issue Edition: 1.0 1
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
100
90
80
70
Maturity (%)
60
50
40
30
10
0
1 6 11 16 21 26 31 36 41
State Rank
The range of Maturity Scores, for ECAC participating States, seen in each Study Area is
presented below:
Mean
Study Area Minimum Maximum
2012
SA1 – Safety Culture 33.0 69.9 88.5
SA2 – Organisational and Individual Safety Responsibilities 32.1 71.2 90.4
SA3 – Timely Compliance with International Obligations 33.8 73.1 91.2
SA4 - Safety Achievement 33.7 71.6 89.0
SA5 - Competency 32.9 69.9 88.6
SA6 – Risk Management 33.8 70.3 87.8
SA7 – Safety Interfaces 30.6 70.0 89.7
SA8 – Safety Assurance 33.1 71.1 89.6
SA9 – Safety Performance Monitoring 33.2 70.6 88.3
SA10 – Organisational Safety Surveys and SMS Audits 32.8 69.9 87.6
SA11 – Adoption and Sharing of Best Practices 33.4 70.5 90.1
Table: E-1: Range of Maturity Scores
Table E-1 shows the maximum and minimum scores in each Study Area. Since 2011 there
has been a good increase at the lower end of scores, as well as some lowering of the scoring
at the top end, but the mean has stayed fairly static.
In a significant number of cases it appears that the lowering of the scores is due to an overall
better understanding of the survey requirements by the respondees, helped by the guidance
provided in the Questionnaire Clarification Document issued with the 2012 Questionnaire.
2% 7%
Level 1
40%
51% Level 2
Level 3
Level 4
80%
Level 4
60%
Level 3
40% Level 2
20% Level 1
0%
2010 2011 2012
Level responses in a number of areas, and ANSPs now having a better understanding of the
requirements, the overall average has improved slightly since the 2011 survey.
To go beyond Level 3 takes considerable application and will cost organisations in terms of
human and financial resources. Moreover, not all ANSPs may be able to reach the
“Continuous Improvement” level as they may not be able to afford the cost and resource
implications. While all ANSPs should strive for excellence3, it may not be possible for all
ANSPs to achieve and maintain levels beyond Level 3 in every single Study Area. It is not
realistic to expect that every ANSP will be developing “best practice” in every area.
Furthermore, “best practice” in one year may become common practice shortly afterwards.
What level is appropriate is therefore a decision that must be made by each individual ANSP
depending on a number of factors such as: resources, complexity of service provided, size of
the ANSP, responsibilities (e.g. in European SES States their position within a FAB) and
internal/external requirements.
The difficulties reported last year by some ANSPs with regards to sharing of data with FAB
partners were not reported this year. In most cases, this is due to further progress in
harmonising processes and procedures between partners.
Since the 2011 survey, the enforced UN-mandated no-fly zone over Libya has been lifted
meaning that ANSPs controlling the Mediterranean airspace have been able to return to
normal operations.
10 9
5
3 3 3
2
1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area
3
CEOs of the ANSPs should strive to go beyond the minimum level of competency and push their organisation to beyond the
level of Implementing and achieve the level of Managing and Measuring […] and even the level of Continuous Improvement
[…] ATM Safety Framework Maturity Survey Methodology, 31st August 2009
Released Issue Edition: 1.0 4
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
12
10
Number of ANSPs
8
6
Level 1
4
Level 2
2
Study Area
Similarly to the previous paragraph, not being able to show the use of best practice
for any particular study objective should prevent an ANSP from scoring above Level
3, which means that an overall low score in SA11 should be reflected in general
levels that do not often exceed Level 3.
20
Number of ANSPs
15
10
7
6
5 4
2
1 1
0 0 0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area
Overall, the data in Figure E-6 shows that Study Area 3: Timely Compliance with
International Obligations is the strongest Study Area experienced by ANSPs. It should be
noted, however, that this relates to the highest Maturity Score across all Study Areas.
When comparing data against the requirement to reach Level 3 for all Study Areas, a
different picture emerges. As can be seen in Figure E-7 there are a number of ANSPs who
are at Level 4 or Level 5 in other Study Areas.
25
20
Number of ANSPs
15
Level 4
10
Level 5
0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area
questionnaires indicates that most organisations in this Study Area have well developed
training and competency processes in place.
Study Area 6 is also strong in terms of ANSPs at Levels 4 and 5 due to many ANSPs already
having risk management plans implemented or have a plan in place to implement a risk
management system.
GLOBAL CANSO
The average Maturity Score for participating CANSO Members are shown in Figure E-8. A
return result of zero indicates that there was no return for that particular year and not that the
Maturity Score is zero.
100
90
80
70
60
Maturity (%)
50
40
30
2010 2011 2012
20
10
0
1 6 11 16 21 26 31 36 41 46 51
State Rank
The number of non-ECAC CANSO Members that participated in the survey, but were not
included in the ECAC Report, has tended to change year on year4. In 2010, 4 states
participated; 2011: 8 states; 2012: 13 states. Over the three years, only 4 states participated
in each year. As a result, trends are more difficult to detect. Currently the mean average
within the grouping for 2012 is 65.66%. The maximum maturity is an overall 98.1%, which is
higher than the ECAC area maximum due to additional ANSPs with very mature SMS being
included. The minimum average is 3.76%, which is much lower than the similar figure for the
ECAC area. This is due some participating CANSO ANSPs outside of the ECAC area being
at the very early stages of SMS implementation.
When including the CANSO Members outside of the ECAC area 505 organisations
participated in this part of the survey. Not all of these additional ANSPs participated in the
follow-up interview process, but those that did (11 ANSPs) gave very detailed responses to
interview questions.
6%
20% Level 1
42%
32% Level 2
Level 3
Level 4
Figure E-9 shows the distribution by levels for all CANSO ANSPs participating in the 2012
survey, with 42% of ANSPs reaching Maturity Level 3. Because the thirteen additional
CANSO ANSPs lie outside of the ECAC Area and the majority of those are only just
beginning to initiate implementation of an SMS they are still at Levels 1 and 2. CANSO has
the same proposed objective of raising all their ANSP Members to Level 3 as soon as
feasible.
4
Although 4 non-EUR CANSO ANSPs participated in the 2010 survey and 8 participated in the 2011 survey, the report for
those years did not include the findings of these entities.
5
One ECAC ANSP that is also a member of CANSO did not return a questionnaire in 2012 and therefore could not be included
in this report.
80%
Level 4
60%
Level 3
40% Level 2
Level 1
20%
0%
2012
Figures E-10 indicates the distribution of Maturity Levels of CANSO members within the
2012 survey. 24 of the participating CANSO ANSPs are at Level 3 or above.
Intentionally Blank
1. INTRODUCTION
1.1 Background
6
Although 4 non-EUR CANSO ANSPs participated in the 2010 survey and 8 participated in the 2011 survey, the report for
those years did not include the findings of these entities.
Released Issue Edition: 1.0 11
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
validate these KPIs and assess them to ensure that safety risk is adequately identified,
mitigated and managed. On this basis, the Commission shall adopt new safety KPIs for the
second reference period if necessary. Moreover, it is the intention of the Commission to use
the data collected during the first reference period to establish the performance targets for
the following reference period.
The first safety performance indicator required in the Regulation, an ATM Safety Framework
Maturity Survey, has been developed by EUROCONTROL [See Appendix 1] in collaboration
with CANSO. It is a natural extension to similar assessments which date back to 2002.
This report summarises the findings of a EUROCONTROL/CANSO Standard of Excellence
(SoE) Measurement conducted during the second half of 2012 and builds on the data
collected in the 2010 and the 2011 EUROCONTROL Safety Framework Maturity Surveys.
The data used for the analyses published in this report were accurate at the time of the
publication (16 November 2012). Any changes in the ANSPs’ data that may have occurred
after this date could not be included in the data sample. ANSPs participating in the survey
conducted by EASA under the framework of the SES legislation may arrive at different
values than those reported during this exercise. All potential changes will be taken into
account for the next year’s baseline.
Non-SES ANSPs. For non-SES States the ICAO Safety Management Manual (Doc 9859
AN/474) provides guidance on how to develop and implement a Safety Management System
(SMS).
Study Areas
SA1 – Safety Culture
SA2 – Organisational and Individual Safety Responsibilities
SA3 – Timely Compliance with International Obligations
SA4 – Safety Achievement
SA5 – Competency
SA6 – Risk Management
SA7 – Safety Interfaces
SA8 – Safety Assurance
SA9 – Safety Performance Monitoring
SA10 – Organisational Safety Surveys and SMS Audits
SA11 – Adoption and Sharing of Best Practices
Table 1-1: Study Areas
7
CEOs of the ANSPs should strive to go beyond the minimum level of competency and push their organisation to beyond the
level of Impelemnting andhieve the level of Managing and Measuring […] and even the level of Continuous Improvement […]
ATM Safety Framework Maturity Survey Methodology, 31st August 2009
Released Issue Edition: 1.0 14
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
8
ECAC has 44 members, but San Marino, although a member of ECAC, does not have its own ATM system and therefore does
not participate in SES implementation.
Released Issue Edition: 1.0 15
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
Key: The EU (darkest blue + darkest red), ECAA (darkest blue + medium blue + medium
red), Eurocontrol (all blue), ECAC (all non-gray)
ECAC, EUROCONTROL, ECAA, EU
ECAC, EUROCONTROL, ECAA
ECAC, EUROCONTROL
ECAC, ECAA, EU
ECAC, ECAA
ECAC Only
100
90
80
70
60
Maturity (%)
50
40
30
2010 2011 2012
20
10
0
1 6 11 16 21 26 31 36 41
State Rank
25
20
Number of ANSPs
15
10
0
Level 1 Level 2 Level 3 Level 4 Level 5
100 5
90 Score Level
80 4
70
60 3
Maturity (%)
50
40 2
30
20 1
10
0 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43
ANSP
100
90
80
70
Maturity (%)
60
50
40
30
2010 2011 2012
20
10
0
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
State
Figure 2-5: Comparison of Maturity Score in 2012 vs 2011 and 2010 for each ANSP
Since the 2011 survey, 22 ANSPs increased their overall Maturity Score, with the best
performing ANSP reporting an 18.89% point increase, and another achieving a 17.43% point
increase. 17 ANSPs fell back against their 2011 Maturity Score, with the lowest two being by
14.72% points and 14.99% points respectively. This is due to a better understanding of what
is required to meet each Maturity Level, especially Levels 4 and 5. In a large majority of
cases the Maturity Score change is of the order of ±3 or 4% points.
Figure 2-6 presents an alternative view of how the Maturity Scores of individual ANSPs have
changed. In this case ANSPs are ordered according to the ratio of Maturity Scores; the 2012
Score divided by the 2011 Score. In the centre of the diagram, a group of ANSPs show either
modest or no change (ratios close to one), whilst the extremes indicate ANSPs where
significant improvements (or decreases) can be found.
1.4 A Ratio of 1
indicates No Change
1.2
1.0
As in 2011, some ANSPs in this survey were still becoming familiar with the requirements of
the revised question set. As such, the largest decrease is unlikely to reflect any real changes
within the ANSP, it is rather the result of a re-evaluation of the response provided in 2010
and 2011. In addition, for the 2012 survey, guidance was provided in the form of a
“Questionnaire Clarification Document” which was sent out with the questionnaire. This
enabled ANSPs to have a better understanding of the requirements for each Maturity Level
and critically review their assessment.
The largest increase stems from a number of organisational improvements in areas such as
culture, reporting and investigation and interfaces, which have significantly improved that
organisation’s response to a number of Study Objectives. For example, the increasing
number of ANSPs undertaking safety culture surveys is having a direct impact on Study Area
1. Improvements being made to the reporting culture and the sharing of lessons learned is
having a direct effect on marking in Study Areas 1, 2 and 8, where improvements in reporting
have an immediate positive impact on the safety culture and move the implementation of the
SMS forward.
Reduced
Increased Score, 17, 38%
Score, 21, 48%
No Change, 6,
14%
Reduced Score,
Increased Score, 18, 41%
24, 55%
No Change, 2,
4%
Study Areas
Answers SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
A 0 0 0 0 0 0 1 0 3 0 2
B 13 10 5 4 2 6 5 3 16 4 15
C 47 43 17 20 5 4 19 40 39 12 32
D 61 69 37 55 22 21 45 41 55 20 57
E 8 50 27 50 14 12 16 45 16 7 23
A&B Total 13 10 5 4 2 6 6 3 19 4 17
A&B Total
% 10% 6% 6% 3% 5% 14% 7% 2% 15% 9% 13%
Table 2-1: Questionnaire Responses by Study Area
The global economic downturn continues to affect some organisations. As last year some of
those interviewed reported that plans that had been discussed previously (e.g. the conduct of
Safety Culture Surveys, and revisions to the Safety Manual) have not yet been implemented.
However, that having been said, there were signs that plans are now being implemented and
there is some recruitment in support areas. Some ANSPs commented that there had not
been sufficient time between surveys for any real progress to be made. Six (including one
ANSP who did not respond to the 2012 questionnaire) ANSPs recorded no change since the
2011 survey, with a number of other ANSPs recording change in only a few Study Areas.
In 2011 some ANSPs had a change of opinion regarding the introduction of FABs with
negative opinions surfacing. This year no strong views were expressed, but some of the
smaller ANSPs stated that they were getting help from FAB partners, particularly in the form
of developing procedures and best practice. The reluctance of some commercially orientated
ANSPs to “give away” what they see as their intellectual property was not apparent this year
and FAB partners appear to be working well together, though in some cases it was clear that
support was provided on a commercial basis.
Since the 2011 survey, the enforced UN-mandated no-fly zone over Libya has been lifted
meaning that ANSPs controlling the Mediterranean airspace have been able to return to
normal operations.
The range of Maturity Scores in each Study Area varies from 30.6% to 91.2%, as shown in
Figure 2-9. The interquartile range is typically around 10%, indicating that ANSPs are
actually closely grouped in their assessment of Safety Maturity.
9
This includes all ECAC Members that are also Members of CANSO and MUAC.
Range of Maturity
100
50
40
33.0 32.1 33.8 33.7 32.9 33.8 33.1 33.2 32.8 33.4
30 30.6
20
10
0
1 2 3 4 5 6 7 8 9 10 11
Study Area
It can be seen from Table 2-2 that in all Study Areas there has been a good increase in the
minimum scores submitted whilst there have been small reductions in the maximum scores
in most of the Study Areas. However, the average scores have remained fairly static, with
only a small drop in the Mean Score, showing that the improvements at the lower end are
more or less cancelled out by the small drop in scores at the higher end. It is believed that
the lowering of some scores at the top end is caused by ANSPs taking a more realistic view
of what they have achieved, which has come about by better explanation of the requirements
for each Study Area coupled with the fact that are now set out in SES regulations and are
now national requirements.
Mean Mean
Study Area Min Max Difference
2011 2010
SA1 – Safety Culture 23.9 69.9 91.8 68.19 1.7
SA2 – Organisational and Individual Safety Responsibilities 24.5 71.3 94.0 69.64 1.6
SA3 – Timely Compliance with International Obligations 24.5 73.4 97.1 72.39 1.0
SA4 - Safety Achievement 24.7 71.8 93.0 69.85 2.0
SA5 - Competency 24.2 70.0 91.8 68.29 1.7
SA6 – Risk Management 23.8 70.4 91.8 68.70 1.7
SA7 – Safety Interfaces 20.2 69.9 92.9 68.13 1.8
SA8 – Safety Assurance 22.8 71.1 92.4 69.30 1.8
SA9 – Safety Performance Monitoring 22.6 70.7 91.3 68.69 2.0
SA10 – Organisational Safety Surveys and SMS Audits 24.4 70.1 91.1 68.26 1.9
SA11 – Adoption and Sharing of Best Practices 23.9 69.9 91.8 68.92 1.0
Table 2-3: Minimum to Maximum Scores for 2012 & Mean Comparison between 2012 and 2010
Table 2-3 shows the Minimum and Maximum scores in 2012 for each Study Area and
demonstrates that there has been an overall steady improvement since 2010 despite there
being some reductions in scores in the individual Study Area scores.
90%
80%
Percentage of Responses
70%
60%
50%
40%
30%
20%
10%
0%
SA10.1
SA11.1
SA11.2
SA11.3
SA1.1
SA1.2
SA1.3
SA2.1
SA2.2
SA2.3
SA2.4
SA3.1
SA3.2
SA4.1
SA4.2
SA4.3
SA5.1
SA6.1
SA7.1
SA7.2
SA8.1
SA8.2
SA8.3
SA9.1
SA9.2
SA9.3
Study Objectives A B C D E
16
14
10
0
33.42
37.81
44.37
59.97
60.83
63.12
65.87
67.75 4
69.48
70.78
72.96
3
74.24
76.66
77.34
80.02
81.03
2
82.24
Maturity
82.88
84.56
86.66
1
87.09
88.97
Level
Seventeen ANSPs have been classified at a lower Maturity Score than in 2011, whilst twenty
one have seen their Maturity Score increase.
In previous surveys ANSPs have asked for more thorough and clear guidance which would
allow for a better and more consistent interpretation of the maturity requirements. Some were
being very precise and literal in their interpretation, whilst others were using looser
interpretations and, perhaps, over-stating their actual maturity. For the 2012 survey guidance
was provided in the form of a “Questionnaire Clarification Document” which was sent out with
the questionnaire. There was a mixed response with regard to the extent to which this
document was used when completing the 2012 questionnaire; some ANSPs commented that
the document had been very useful, whilst others had not noticed it attached to the
questionnaire and said that they had not read it, whilst some claimed not to have received it.
Released Issue Edition: 1.0 24
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
In 2011 the survey reported that there was evidence of tension building up within FABs, with
smaller organisations feeling that their larger partners are being too domineering. Similar
evidence was not detected this year. Some of the smaller ANSPs were getting support from
their FAB partners, albeit mostly on a commercial basis.
Some ANSPs still see the State judicial system, and the media, as barriers to developing a
Just Culture, although they are willing to fight to protect their staff if they have to. They
consider that it is the NSA/Regulator’s responsibility to negotiate with the Judicial Authorities.
It was noticeable that more ANSP’s CEOs are personally involved in Safety decisions and
personally approve matters, in some instances that includes the responses to the SoE
measurement questionnaire. This is thought to be due to the increased emphasis on Safety
Culture and, in SES States, the need to follow EU legislation.
This year a system of Supplementary Questions was introduced for some ANSPs (this
system was first used in the BlueMed survey with good results). The Supplementary
Questions were based on the question set provided by EUROCONTROL for use in all
interviews. This method of getting additional information was welcomed as it gave
interviewees more time to consider the response and overcame any verbal language
problems. One ANSP commented that at first he thought that it would be an added burden,
but having completed the supplementary questions, undergone the telephone interview and
seen the resulting report thought that it was very worthwhile.
Some interviewees questioned the need to complete the SoE Measurement and then
complete the EASA questionnaire later in the year, but by far the majority considered that the
exercise prepared them for the second survey and were very happy to have gone through
the exercise.
There were still a number of ANSPs who questioned the need to develop Best Practices in
all Study Areas, in fact a number refused to downgrade their score from E to D even though
they could not identify a Best Practice that they had developed, but considered that in all
other respects they met the requirements for E.
What is meant by the term “Best Practice” is misunderstood by many of those who
participated in the survey and should be clarified. It appeared that some ANSPs did not
appreciate the need to have a system to measure their SMS performance before selecting a
level beyond Level C.
Those ANSPs that underwent face-to-face interviews appreciated the face-to-face interaction
as being a very useful exchange, which gave the opportunity to discuss the Study Areas in
more detail. ANSPs considered that face-to-face interviews made it easier to clarify any
issues and also enabled more people from an ANSP to participate in discussions.
A number of interviewees commented that they considered only seven months between
surveys was too short a time for organisations to show any real improvement in their scoring
and requested that in future the gap should be larger. It was explained this survey was
brought forward exceptionally to align the process with the requirements of the Single
European Sky Legislation and the gap would revert to one year in future.
Objectives:
1.1 A positive and pro-active just, flexible, and informed safety culture (the shared
beliefs, assumptions, and values regarding safety) that supports reporting and
learning led by management.
1.2 Regular measurement of safety culture and an improvement programme.
1.3 An open climate for reporting and investigation of occurrences.
The maturity profile for Study Area 1 ranges from a minimum of 33.0% to a maximum of
88.5%, with the average being 69.9% (compared with 23.9%, 91.8% and 69.9% in 2011).
Released Issue Edition: 1.0 25
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
Figure 2-12 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 1 for 2012 and 2011.
25
21
20
Number of ANSPs
15 14
13 13
11 11 2011
10 2012
5
1
0 0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5
It can be seen from Figure 2-12 that there are no ANSP’s who consider that they are still at
Level 1 in this Study Area and for 2012 the majority of ANSPs have reached at least Level 3,
though none have yet achieved Level 5. This indicates that a positive Safety Culture is
developing, but there is still work to do. The majority of ANSPs at Levels 3 and 4 have
improvement plans in place and those at Level 3 are beginning to measure safety culture,
however there are a number of ANSPs at Level 2, and some at Level 3, who do not know
how to measure Safety Culture. On the positive side several ANSPs have completed Safety
Culture Surveys in 2012 and more are planned for later in the year or early 2013. In addition,
some ANSPs are planning for their second or third survey. There has been some movement
between Levels 3 and 4 with some ANSPs improving their score and a few falling back from
Level 4 to Level 3, but overall the signs for this Study Area are positive.
Study Area 1 was only discussed in depth in the few face-to-face interviews undertaken so
there is only limited information on the causes for movement in this Area. However, it was
noticeable how many CEOs/DGs now take a personal interest in ensuring that Safety Culture
is being developed within their organisations and support the measures being taken by their
staff. One ANSP explained that they had developed a process for ensuring that their CEO
was fully involved with Safety Culture and SMS development and claimed this as a Best
Practice that they had shared with others.
The instances of ANSPs undertaking, or planning to undertake, Safety Culture Surveys
appears to be increasing. One ANSP had found resistance from senior management, but a
change in CEO brought a changed attitude and the ANSP is now undertaking a Safety
Culture Survey. Another significant fact to driving a change in attitude to measuring Safety
Culture is the fact that younger staff are now moving into positions of responsibility and see
the benefits of improving safety culture.
In summary, whilst ANSPs recognise the importance of measuring Safety Culture only a few
had any idea on how to achieve that and were therefore struggling to progress beyond Level
3 in this Study Area. Most recognised that they would need to enlist external assistance to
achieve a viable measurement.
25
20 SA1: Development
Movement Comparison
15 of a Positive and
10 Proactive Safety Culture
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.55 5.70 19.78
Dec 11/12 -0.82 -5.25 -14.01
Inc 10/11 0.27 6.56 18.96
Dec 10/11 -0.27 -6.18 -17.86
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
Objectives:
2.1 An approved, clearly documented, and recognised system for the management of
safety. Management structure, responsibilities, accountabilities and authorities are
clearly defined and documented.
2.2 A clearly defined safety management function that is independent of line
management.
2.3 An integrated safety planning process is adopted by the organisation with
published and measurable safety goals and objectives which are accountable to the
executive.
2.4 Clear understanding and acceptance of safety management responsibilities by all
staff and contractors.
The maturity profile for this Study Area ranges from a minimum of 32.1% to a maximum of
90.4%, with the average being 71.2% (compared with 24.5%, 94% and 71.3% in 2011).
Figure 2-15 (Number of ANSPs at each Maturity Level) shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 2.
25
20
20
Number of ANSPs
17 17
15
2011
10
10 2012
6 6
5
5 3
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5
In 2011 by far the greatest number of ANSPs (20) were at Level 3. This year there has been
a significant improvement with seventeen ANSPs now saying that they are at Level 4.
A number of ANSPs have undertaken major reviews of their safety organisation and safety
responsibilities. Some of the less mature ones have sought help from more mature partners
in an effort to improve the systems that they have in place. Fairly frequent reviews of safety
responsibilities are taking place and many safety managers now have direct access to their
CEOs. Safety Culture is seen as a driver for ensuring an effective SMS.
Quite a few ANSPs combine their safety and quality organisations and believe that the
quality side helps the safety organisation through such things as ISO 9001 certification by
ensuring that the correct processes are in place. Furthermore, demands for continuous
process monitoring and measuring from the quality side is solid grounds to scoring above
Level 3 in areas under the remit of the quality process.
The majority of ANSPs now have a formal safety plan in place that is reviewed annually.
Safety Responsibilities are often set out in the Safety Management Manual and are also
available for all staff to view on the Intranet. In SES States Safety Policy is based on
European Regulations. However, the safety plan is often encapsulated in overall
organisational plans that include quality, safety and security as some CEOs/DGs believe
there should only be one overall strategic plan. Departmental activity plans are then
developed from the overall plan. Performance is measured against meeting safety targets.
Only a few ANSPs have a formal review process in place to specifically identify changes
within their organisation that could affect company documentation. Mainly this is dealt with on
an ad hoc basis, or is the subject of discussion at safety related meetings. However most
ANSPs have some form of regular review of the SMS through frequent audits that results in
updates being made. Agreement on documentation updates is often undertaken through
safety committees.
ANSPs use a variety of methods to ensure that staff are kept up to date with safety
information through committees, safety leaflets/bulletins, use of the intranet, safety seminars
etc. The impression from interviews undertaken is that in most ANSPs staff are aware of
safety responsibilities, they are kept up to date with safety related information and discussion
on safety-related matters is encouraged by most ANSPs within the ECAC Area. There is little
doubt that SES legislation has helped to focus the minds of ANSPs’ senior management and
made them pay more attention to safety requirements than hitherto. This means that most
ANSPs now have well documented organisational and individual responsibilities with safety
managers having access to the highest organisational level.
A number of ANSPs said that when formulating SMS training and assessment they were
developing and using Best Practice the Best Practices developed were then shared with FAB
partners. For instance some are now involving staff generally to help improve their training
development and others are setting Technical Boards that have an input into assessment
and training requirements.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
16.
25
20 SA2: Organisational &
Movement Comparison
15 Individual Safety
10 Responsibilities
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.55 6.17 20.05
Dec 11/12 -0.27 -4.72 -16.21
Inc 10/11 0.27 6.41 17.58
Dec 10/11 -0.55 -6.36 -16.76
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
Objectives:
The maturity profile for Study Area 3 ranges from a minimum of 33.8% to a maximum of
91.2%, with the average being 73.6% (compared with 24.5%, 97.1% and 73.4% in 2011).
Figure 2-18 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 3.
20 19 19
18
16
Number of ANSPs
14
12
12
10
10 9 2011
8
8 2012
6
4
4
2
2 1
0
0
Level 1 Level 2 Level 3 Level 4 Level 5
25
20 SA3: Timely Compliance
with International
Movement Comparison
15
10
Organisations
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 1.47 7.12 20.59
Dec 11/12 -0.98 -6.08 -16.67
Inc 10/11 1.47 7.07 16.18
Dec 10/11 -0.49 -5.80 -17.65
25.00
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
• Seventeen ANSPs have increased their Maturity Score in this Study Area since 2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.
Objectives:
The maturity profile for Study Area 4 ranges from a minimum of 33.7% to a maximum of
89%, with the average being 71.6% (compared with 24.7%, 93% and 71.8% in 2011). Figure
2-21 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 4.
25 23
20
20
Number of ANSPs
15
12
11 2011
10 2012
7
5
5 3 3
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
22.
25
20 SA4: Safety Standards
& Procedures
Movement Comparison
15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.33 5.67 18.67
Dec 11/12 -1.00 -5.56 -17.00
Inc 10/11 0.33 6.41 19.33
Dec 10/11 -0.33 -6.41 -15.33
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-5.00 44
-10.00
Objectives:
5.1 Staff, and contractors (where appropriate) are trained, competent in safety and
safety management, and where required, licensed.
The maturity profile for Study Area 5 ranges from a minimum of 32.9% to a maximum of
88.6% with the average being 69.9% (compared with 24.2%, 91.8% and 70% in 2011).
Figure 2-24 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 5.
25
22
20
Number of ANSPs
17
15 14 14
2011
10 8 2012
5
5
2 2
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5
25
20 SA5: Competency
Movement Comparison
15
10
5
0
-5
-10
-15
-20
-25
Minimum Average Maximum
Inc 11/12 0.27 5.68 20.11
Dec 11/12 -0.82 -5.18 -14.13
Inc 10/11 0.27 6.53 18.48
Dec 10/11 -0.27 -6.21 -18.21
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
-15.00
Objectives:
6.1 A continuing risk management process that identifies, assesses, classifies, and
controls all identified safety risks within the organisation, including potential future
risks.
The maturity profile for Study Area 6 ranges from a minimum of 33.8% to a maximum of
87.8%, with the average being 70.3% (compared with 23.8%, 91.8 and 70.4% in 2011).
Figure 2-27 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 6.
25
21
20 19
Number of ANSPs
15 13
12
2011
10 2012
7
6
5 4
2
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5
The Risk Analyses Tool (RAT) seems to be increasingly used, although a few ANSPs said
that it did not meet their needs precisely, because they believed that the repeatability
element of the tool is not yet evolved sufficiently. Some believe that the tool is more of a
Severity Assessment Tool, due to the amount of subjectivity involved. Some organisations
have developed their own system to determine risk, though the impression from the
interviews is that most of the locally developed systems follow similar principles to RAT.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
28.
25
20 SA6: Risk Management
Movement Comparison
15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.30 6.14 19.21
Dec 11/12 -0.91 -5.70 -15.85
Inc 10/11 0.91 6.59 17.99
Dec 10/11 -1.22 -6.05 -14.94
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
• Twenty one ANSPs have increased their Maturity Score in this Study Area since
2011.
The reasons for the reduced scores are predominantly due to a better understanding of the
questionnaire requirements.
Objectives:
30
25
25 23
Number of ANSPs
20
15 2011
12
2012
9
10
4
5 3 3
2 2
1
0
Level 1 Level 2 Level 3 Level 4 Level 5
The percentage point movement from 2010 and between 2011 an 2012 is shown in
Figure 2-31.
25
20 SA7: Safety Interfaces
Movement Comparison
15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.40 6.59 18.65
Dec 11/12 -0.79 -5.65 -16.27
Inc 10/11 0.40 7.04 18.65
Dec 10/11 -0.79 -5.58 -16.27
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-5.00 44
-10.00
Objectives:
30
25
25
22
Number of ANSPs
20
15 13 2011
2012
10 8
5
4
5 3 3
1
0
0
Level 1 Level 2 Level 3 Level 4 Level 5
The majority of ANSPs understand the importance of disseminating data as far as possible
both within the organisation and to others. During the interviews, the Hindsight magazine
was quoted as being an excellent example of spreading information between ANSPs and it is
clear that lessons learned are shared within the FAB organisations.
In the majority of ANSPs, while corrective measures are reviewed to ensure they have been
implemented correctly, there is limited activity to determine to what extent the lessons have
actually been learned. Whilst there may be some obvious metrics such as whether there is
any re-occurrence of incidents, there is little in the way of a formal review. While there is no
doubt about means being put in place to support these objectives, there is a clear need in a
number of ANSPs to ensure the quality of the system through measurements of the
reporting, investigation and lesson dissemination processes.
Some ANSPs have processes where anyone within the organisation who identifies a
problem in the system, or a document etc. can write a problem report. The reports are then
followed up by the Safety Unit/Department.
There is still reluctance in some ANSPs to share data fully with the general public and the
majority stated that they only share outside of their own organisations when required legally
to do so.
The percentage point movement from 2010 and between 2011 an 2012 is shown in Figure 2-
34.
25
SA8: Safety Reporting,
20
Investigation &
Movement Comparison
15
Improvement
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.56 5.96 18.54
Dec 11/12 -0.56 -4.40 -14.04
Inc 10/11 0.56 6.65 20.51
Dec 10/11 -0.56 -6.11 -16.85
25.00
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
Objectives:
9.1 An established and active monitoring system that uses and tracks suitable safety
indicators and associated targets (e.g., lagging and leading indicators).
9.2 Methods to measure safety performance, which is compared within and between
ANSPs.
9.3 A general public knowledgeable of the ANSP’s performance through routine
publication of achieved safety levels and trends. (Information disclosure is
compliant with the requirements of ICAO Annex 13, Attachment E).
The maturity profile of Study Area 9 ranges from a minimum of 33.2% to a maximum of
88.3%, with the average being 70.6% (compared with 22.6%, 91.3% and 70.7% in 2011).
Figure 2-36 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 9.
25
21
20
20
Number of ANSPs
15
11 11 2011
10
10 2012
7
5
2 2
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5
25
20 SA9: Safety Performance
Monitoring
Movement Comparison
15
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.27 6.09 19.57
Dec 11/12 -0.54 -4.59 -14.40
Inc 10/11 0.54 6.10 20.38
Dec 10/11 -0.82 -6.63 -15.76
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
Objectives:
10.1 Internal and independent (external) operational safety surveys and SMS audits.
The maturity profile of Study Area 10 ranges from a minimum of 32.8% to a maximum of
87.6%, with the average being 69.9% (compared with 24.4%, 91.1% and 71.2% in 2011).
Figure 2-39 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 10.
25 23
20
20
Number of ANSPs
15
12
2011
10
10 2012
7
5
5 4
3
0 0
0
Level 1 Level 2 Level 3 Level 4 Level 5
25
20 SA10: Operational
Safety Surveys
Movement Comparison
15
& SMS Audits
10
5
0
-5
-10
-15
-20
Minimum Average Maximum
Inc 11/12 0.29 5.73 19.83
Dec 11/12 -1.15 -5.46 -15.80
Inc 10/11 0.57 6.24 17.53
Dec 10/11 -1.72 -6.85 -14.37
As can be seen from Figure 2-40, there is little differences in the changes in the comparisons
shown for this Study Area. Although there is some up and down movement, the overall
picture remains fairly static for both comparisons. The average decrease in percentage point
movement has fallen from minus 6.85% in 2011 to minus 5.46% in 2012. The maximum
increased movement has risen slightly from 17.53% in 2011 to 19.83% in 2012. The
minimum decrease in movement has fallen from minus 1.72% in 2011 to minus 1.15 in 2012.
25.00
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
-5.00
-10.00
Objectives:
11.1 A structured approach exists to promote safety, its standing within the
organisation and lessons learned through application of the SMS.
11.2 A structured approach to gather information on operational safety and SMS
best practises from the industry.
11.3 Sharing of safety and SMS-related best practises with industry stakeholders.
The maturity profile of Study Area 11 ranges from a minimum of 33.4% to a maximum of
90.1%, with the average being 70.5% (compared with 26.5%, 91.8% and 69.9% in 2011).
Figure 2-42 shows the number of ANSPs at each Maturity Level when measured against the
objectives of Study Area 11.
20 19
18
16
16 15 15
Number of ANSPs
14
12
10 2011
8 2012
6
6 5
4
4
2
2 1 1
0
Level 1 Level 2 Level 3 Level 4 Level 5
25
20 SA11: Adoption & Sharing
Movement Comparison
15 of Best Practices
10
5
0
-5
-10
-15
-20
-25
Minimum Average Maximum
Inc 11/12 0.82 5.70 19.31
Dec 11/12 -0.17 -4.19 -13.83
Inc 10/11 0.46 7.00 17.75
Dec 10/11 -0.33 -5.40 -19.59
As can be seen from Figure 2-43, there is some difference in the changes in the extent to
which scores have decreased between 2010/2011 and 2011/2012. The maximum number of
percentage points between 2011/2012 is 5.7 percentage points better than between
2010/2011. The average decrease in percentage point movement has fallen from minus
5.4% in 2011 to minus 4.19% in 2012. The maximum increased percentage point movement
has risen slightly from 17.75% in 2011 to 19.31% in 2012. The minimum decreases in
movement are all below 1%.
25.00
20.00
15.00
10.00
5.00
0.00
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
-5.00 44
-10.00
-15.00
The above average increases in movement for this area may be partly explained by the
number of ANSPs that now assess themselves at Level 4.
10 9
5
3 3 3
2
1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area
Overall, the data in Figure 2-45 shows that Study Area 7: Safety Interfaces is the weakest
Study Area experienced by most ANSPs. This means that of all eleven study areas in the
survey, fourteen ANSPs recorded the lowest score of their measurement in SA7, while nine
other had SA9 as their lowest score and seven ANSPs had their lowest score in SA1.
The SAFREP Task Force proposed that all ECAC ANSPs should be at Level 3 for all Study
Areas by the end of the SES First Reference Period.
When comparing data against the objective to reach Level 3 for all Study Areas, a different
picture emerges. As can be seen in Figure 2-46 there are a number of ANSPs who are at
Level 1 or Level 2 within a number of Study Areas. SA1 (Safety Culture) and SA9
(Performance Monitoring) appear to be particularly weak. This is usually due to having one
question answered at a low level in those areas, but overall they do not necessarily arrive at
the lowest score due to the influence of other questions in the survey and the effect of the
weighting matrix (interdependencies between the questions).
12
10
Number of ANSPs
8
6
Level 1
4
Level 2
2
Study Area
Figure 2-46 shows those Study Areas where ANSPs are not yet meeting the regulatory
requirements. The reasons for this are more fully explained in the relevant areas in this
report for the respective Study Areas. However, significantly:
• Study Area 1: Safety Culture. A positive safety culture is emerging within ANSP
organisations, though in some ANSPs it is still immature, with the majority of ANSPs
being at Level 3 or 4. However none have yet reached Level 5 and there are still
eleven below Level 3 with some yet to commence the journey into establishing a
positive and proactive Safety Culture. Most ANSPs now recognise the importance of
Safety Culture and the need to measure it. They now understand that they need to
undergo Safety Culture surveys to establish where they are in developing a positive
and proactive safety culture in their organisation. A significant number of ANSPs have
chosen to undertake Safety Culture Surveys through EUROCONTROL and more are
planned before the end of 2012. It is therefore anticipated that the number of ANSPs
still below Level 3 in this Study Area will reduce further in the next year
• Study Area 9: Safety Performance Monitoring. There has been some regression in
this Study Area with some ANSPs at Level 3 in 2011 reassessing their performance
and marking themselves at Level 2 this year. Most ANSPs had some form of
monitoring in place, but the majority do not have a Quantitative system, nor do they
undertake internal comparative analysis. Many ANSPs do not see the need to
routinely share safety data with the general public, though a lessening of the
resistance to do this was detected. There are still 12 ANSPs below Level 3 and more
work is required to emphasis the importance of having a quantitative monitoring
system in place.
• Study Area 11: Adoption and Sharing of Best Practices. There has been little
movement in this area in 2012. It is clear that there is misunderstanding of what
constitutes “Best Practice” and how it should be assessed in the context of this Study
Area. There are eight ANSPs shown below Level 3. It is believed that there should be
more ANSPs below Level 4 as some refused to mark themselves below Level 4 even
though they could not show that they were using best practice on the basis that they
did not believe that it was possible to develop best practices in some of the study
areas. Greater explanation of the survey requirements in this area is required.
However there is an indication that ANSPs are exchanging data at international
meetings and this may be that the best way to educate ANSPs on what constitutes
“Best Practice” is in that type of fora.
20
Number of ANSPs
15
10
7
6
5 4
2
1 1
0 0 0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area
Overall, the data in Figure 2-47 shows that Study Area 3: Timely Compliance with
International Obligations is the strongest Study Area experience by ANSPs. This means that
of all eleven study areas in the survey, 22 ANSPs recorded the highest score of their
measurement in SA3, while seven others had SA4 as their highest score and six ANSPs had
their highest score in SA6
When comparing data against the objective to reach Level 3 for all Study Areas, a different
picture emerges. As can be seen in Figure 2-48 there are a number of ANSPs who are at
Level 4 or Level 5 in certain Study Areas. Of particular strength seem to be SA5 and SA6.
25
20
Number of ANSPs
15
Level 4
10
Level 5
0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area
10
The first four paragraphs of this section are taken directly from the CANSO Website.
Released Issue Edition: 1.0 53
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
global policies. Regular meetings, workshops and conferences ensure that all ANSPs are
fully aware of current industry trends and developments. In addition CANSO coordinates
industry positions with customer representatives and other aviation stakeholders to ensure
global standards are developed and adopted.
The SoE Measurement aims to find how well CANSO Members are meeting the elements of
the CANSO SoE model.
100
90
80
70
60
Maturity (%)
50
40
30
10
0
1 6 11 16 21 26 31 36 41 46 51
State Rank
11
There are 38 Members of ECAC who are also Members of CANSO, but one ECAC ANSP did not return a questionnaire in
2012.
Released Issue Edition: 1.0 54
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
6%
20% Level 1
42%
32% Level 2
Level 3
Level 4
25
20
2010
Number of ANSPs
15 2011
2012
10
0
Level 1 Level 2 Level 3 Level 4 Level 5
objectives, while an organisation with a relatively high score but a low level would prove that
it may have left behind certain objectives, concentrating on others.
100 5
90 Score Level
80 4
70
60 3
Maturity (%)
50
40 2
30
20 1
10
0 0
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50
ANSP
When including the CANSO Members outside of the ECAC area 50 organisations
participated in this part of the survey. Not all of these additional ANSPs participated in the
follow-up interview process, but those that did (11 ANSPs) gave very detailed responses to
interview questions.
Range of Maturity
90
80
73.5 74.3 73.8 73.0
71.4 71.6 72.3 72.2 72.1 71.1 71.9
70
40
30
20
10
3.6 4.1 4.4 3.3 3.8 4.0 3.9 4.1 3.7 4.1
2.4
0
1 2 3 4 5 6 7 8 9 10 11
Min - Max limits
Study Area
Interquartile range
90%
80%
Percentage of Responses
70%
60%
50%
40%
30%
20%
10%
0%
SA1.1
SA1.2
SA1.3
SA2.1
SA2.2
SA2.3
SA2.4
SA3.1
SA3.2
SA4.1
SA4.2
SA4.3
SA5.1
SA6.1
SA7.1
SA7.2
SA8.1
SA8.2
SA8.3
SA9.1
SA9.2
SA9.3
SA10.1
SA11.1
SA11.2
SA11.3
Study Objectives A B C D E
25
20
Number at Minimum Level
15
10
0
3.76
19.92
33.42
37.81
43.76
44.37
59.97
60.83
63.29
67.62
4
68.83
70.24
72.54
73.59
75.07
3
75.97
76.87
80.02
81.03
2
82.24
82.88
84.56
86.66
87.36
1
Maturity
92.93
Level
5. Only Key Findings pertaining to the additional CANSO ANSPs that are not members of ECAC are included here.
Released Issue Edition: 1.0 59
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
25
20
20
Number of ANSPs
15 14
11
10
5 4
1
0
Objectives:
2.1 An approved, clearly documented, and recognised system for the management of
safety. Management structure, responsibilities, accountabilities and authorities
are clearly defined and documented.
2.2 A clearly defined safety management function that is independent of line
management.
2.3 An integrated safety planning process is adopted by the organisation with
published and measurable safety goals and objectives which are accountable to
the executive.
2.4 Clear understanding and acceptance of safety management responsibilities by all
staff and contractors.
The maturity profile for Study Area 2 ranges from a minimum of 4.1% to a maximum of
98.6%, with the average being 65.9%. Figure 3-10 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 2.
18
16
16 15
Number of ANSPs
14
12
10 9
8
6
6
4
4
2
0
Outside of the ECAC Area three ANSPs has achieved Level 5, there are no ANSPs at Level
4, two are at Level 3 and the remaining eight are at Levels 1 or 2.
Objectives:
The maturity profile for Study Area 3 ranges from a minimum of 4.4% to a maximum of
99.0%, with the average being 67.8%. Figure 3-11 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 3.
20 19
18
16
Number of ANSPs
14
12 11
10
10 9
8
6
4
2 1
0
Objectives:
The maturity profile for Study Area 4 ranges from a minimum of 3.3% to a maximum of
98.3%, with the average being 66.8%. Figure 3-12 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 4.
25 23
20
Number of ANSPs
15
10
10
7 7
5 3
Objectives:
5.1 Staff, and contractors (where appropriate) are trained, competent in safety and
safety management, and where required, licensed.
The maturity profile for Study Area 5 ranges from a minimum of 3.8% to a maximum of
98.1%, with the average being 64.8%. Figure 3-13 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 5.
30
25 24
Number of ANSPs
20
15 14
10
5 4 4 4
Objectives:
6.1 A continuing risk management process that identifies, assesses, classifies, and
controls all identified safety risks within the organisation, including potential
future risks.
The maturity profile for Study Area 6 ranges from a minimum of 4.0% to a maximum of
98.2%, with the average being 65.3%. Figure 3-14 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 6.
25
20
20
Number of ANSPs
15 13
10
7 7
5 3
Objectives:
The maturity profile for Study Area 7 ranges from a minimum of 2.4% to a maximum of
98.4%, with the average being 65.4%. Figure 3-15 shows the number of ANSPs at each
Released Issue Edition: 1.0 65
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
25
21
20
Number of ANSPs
16
15
10
6
5 4
3
Objectives:
The maturity profile for Study Area 8 ranges from a minimum of 3.9% to a maximum of
97.5%, with the average being 66.2%. Figure 3-16 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 8.
25
22
20
Number of ANSPs
15 13
10
7
5
5 3
Objectives:
9.1 An established and active monitoring system that uses and tracks suitable safety
indicators and associated targets (e.g., lagging and leading indicators).
9.2 Methods to measure safety performance, which is compared within and between
ANSPs.
9.3 A general public knowledgeable of the ANSP’s performance through routine
publication of achieved safety levels and trends. (Information disclosure is
compliant with the requirements of ICAO Annex 13, Attachment E).
The maturity profile for SA 9 ranges from a minimum of 4.1% to a maximum of 97.6%, with
the average being 65.7%. Figure 3-17 shows the number of ANSPs at each Maturity Level
when measured against the objectives of Study Area 9.
25
20
20
Number of ANSPs
15
12 12
10
5
5
1
0
Objectives:
10.1 Internal and independent (external) operational safety surveys and SMS audits.
The maturity profile for Study Area 10 ranges from a minimum of 3.7% to a maximum of
97.4%, with the average being 65%. Figure 3-18 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 10.
Released Issue Edition: 1.0 68
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
20 19
18 17
16
Number of ANSPs
14
12
10 9
8
6
4 3
2
2
0
Objectives:
11.1 A structured approach exists to promote safety, its standing within the
organisation and lessons learned through application of the SMS.
11.2 A structured approach to gather information on operational safety and SMS
best practises from the industry.
11.3 Sharing of safety and SMS-related best practises with industry stakeholders.
The maturity profile for Study Area 11 ranges from a minimum of 4.1% to a maximum of
98.0%, with the average being 65.6%. Figure 3-19 shows the number of ANSPs at each
Maturity Level when measured against the objectives of Study Area 11.
20 19
18
16 15
Number of ANSPs
14
12
10
8 7 7
6
4
2
2
0
20
16
Number of ANSPs
15
11
10
8
5
5
3 3
2
1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area
As in the ECAC results, overall the CANSO data in Figure 3-20 shows that Study Area 7:
Safety Interfaces, is the weakest Study Area experienced by most ANSPs. This means that
of all eleven study areas in the survey, sixteen ANSPs recorded the lowest score of their
measurement in SA7, while eleven other had SA10 as their lowest score and eight ANSPs
had their lowest score in SA1.
CANSO supports the SAFREP Task Force proposal that all ECAC ANSPs should be at Level
3 and considers that all CANSO Members should achieve the same level as soon as
feasible.
When comparing data against the requirement to reach Level 3 for all Study Areas, a
different picture emerges. As can be seen in Figure 3-21 there are a number of ANSPs who
are at Level 1 or Level 2 within a number of Study Areas with Study Area 7 being one of the
better ones. SA1 (Safety Culture) and SA9 (Performance Monitoring) appear to be
particularly weak. This is usually due to having one question answered at a low level in those
areas, but overall they do not necessarily arrive at the lowest score due to the influence of
other questions in the survey and the effect of the weighting matrix (interdependencies
between the questions).
14
12
10
Number of ANSPs
8
Level 1
6
Level 2
0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area
30
26
25
Number of ANSPs
20
15
10
7
6 6
5
1 1 1 1 1
0 0
0
1 2 3 4 5 6 7 8 9 10 11
Study Area
25
20
Number of ANSPs
15
Level 4
10
Level 5
0
SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
Study Area
Intentionally Blank
4. CONCLUSIONS
1. Overall the maturity level of ANSPs in the ECAC Region has increased slightly since
the 2011 survey.
2. The introduction of the additional CANSO ANSPs has demonstrated that ECAC
ANSPs are in the main further advanced in terms of SMS implementation than
ANSPs outside of the ECAC Area, where many ANSPs are only in the early stages of
SMS implementation.
3. Relationships within FABs seem to have improved. In 2011 the survey reported that
there was evidence of tension building up within FABs, with smaller organisations
feeling that their larger partners are being too domineering. Similar evidence was not
detected this year anymore. A number of small ANSPs seem to appreciate the help
they can get from the more mature ANSPs, even if this help is often provided on a
commercial basis.
4. It was noticeable that more ANSPs CEO/DGs are personally involved in Safety
decisions. This is thought to be due to the increased emphasis on Safety Culture and,
in SES States, the requirement to follow EU legislation.
5. As reported in the 2011 survey report, based on the overall Maturity Scores, the
weakest areas for ANSPs is Study Area 7 (Safety Interfaces). This is for the ECAC
and CANSO areas. The data provides a different picture when looking at the
achieved Level. Significantly, ANSPs need to improve their progress in Study Area 1
(Safety Culture), Study Area 9 (Safety Performance Monitoring) and Study Area 11
(Adoption and Sharing of Best Practices). For CANSO ANSPs, improvements are
required across all areas.
6. The strongest area for ANSPs based on the overall Maturity Scores is Study Area 3
(Timely Compliance with International Obligations). When looking at the extent to
which ANSPs have reached Level 4 or Level 5, within the ECAC area, Study Area 4
(Safety Achievement) Study Area 5 (Competency) and Study Area 6 (Risk
Management) all produce good results. For the CANSO area, good results have been
achieved in Study Area 3, Study Area 4, Study Area 5, Study Area 6, and Study
Area 10 (Organisational Safety Surveys and SMS Audits).
7. The amount of Justification and Evidence in the questionnaires has improved, some
ANSPs providing comprehensive details. A reason for this was due to the additional
clarification provided by EUROCONTROL on how to answer the questionnaire.
8. The Clarification Document provided by EUROCONTROL was used by some ANSPs,
but not all. Where the guidance document was used extensively, ANSPs answered
more realistically – in some cases considerably changing their scores. This has had a
marked effect on the overall results in this year’s survey. Some ANSPs believe that
further guidance is required with regard to what is a good response (e.g. through
providing some sample answers). Furthermore, some ANSPs question the possibility
of developing best practice for some areas and would welcome further clarification in
this respect (with examples).
9. Among the additional CANSO ANSPs the more developed SMS are within ANSPs
that have been developing it for at least 5 years, some longer. The other ANSPs are
only at the implementaion stage at best.
10. Those ANSPs that underwent face-to-face interviews appreciated the interaction as
being a very useful exchange, which gave the opportunity to discuss the Study Areas
in more detail. ANSPs considered that these interviews made it easier to clarify any
issues and also enabled more people from an ANSP to participate in discussions.
Intentionally Blank
Appendix 2 Methodology...................................................................................84
Appendix 6 Glossary..........................................................................................92
77
2012 EUROCONTROL/CANSO Standard of Excellence Measurement
The following References and legislation were used during the survey for purposes of
clarification for survey participants, both European and non-European. They are valid at the
date of publication, but as the legislative environment is dynamic and changes constantly, it
may not be accurate when used at a later date.
These References should be used for clarification purposes only and do not form a
formal legislative Document.
ICAO Requirements
The ICAO Safety Management Manual (Doc 9859 AN/474) provides States with guidance on
how to develop a regulatory framework and provides supporting guidance material for the
implementation of Safety Management Systems (SMS) by service providers.
The following International Standards and Recommended Practices (SARPs) contained in
the ICAO Annexes mentioned below pertain to questions in the EUROCONTROL/CANSO
Standard of Excellence Measurement questionnaire where appropriate:
• Annex 1 — Personnel Licensing;
• Annex 6 — Operation of Aircraft;
• Annex 8 — Airworthiness of Aircraft;
• Annex 11 — Air Traffic Services;
• Annex 13 — Aircraft Accident and Incident Investigation;
• Annex 14 — Aerodromes.
The ICAO safety management SARPs are contained in Annexes 1; 6, Parts I and III; 8; 11;
13 and 14. These Annexes address the activities of approved training organisations,
international aircraft operators, approved maintenance organisations, organisations
responsible for type design and/or manufacture of aircraft, air traffic service providers and
certified aerodromes. In the case of Annex 1, the safety management SARPs are limited
exclusively to approved training organisations that are exposed to safety risks during the
provision of their services.
The safety management SARPs are aimed at two audience groups: States and service
providers. In the context of the ICAO Safety Management Manual, the term “service provider”
refers to any organisation providing aviation services. The term thus encompasses approved
training organisations that are exposed to safety risks during the provision of their services,
aircraft operators, approved maintenance organisations, organisations responsible for type
design and/or manufacture of aircraft, air traffic service providers and certified aerodromes,
as applicable.
The ICAO safety management SARPs address the following three distinct requirements:
a) the State safety programme (SSP), including the acceptable level of safety (ALoS) of
an SSP;
b) safety management systems (SMS), including the safety performance of an SMS;
and
c) management accountability vis-à-vis the management of safety during the provision
of services.
The ICAO safety management SARPs introduce the notion of acceptable level of safety
(ALoS) as the way of expressing the minimum degree of safety that has been established by
the State and must be assured by an SSP, and the notion of safety performance as the way
of measuring the safety performance of a service provider and its SMS.
Commission Regulation (EC) No 549/2004 laying down the Framework for the Creation
of the Single European Sky
The Framework Regulation, supported by three other Regulations (Air Navigation Service
Provision, Airspace and Interoperability) is designed to create a European Airspace
conceived and managed as a single continuum (the Single European Sky - SES) to optimise
the safety and efficiency of the European Air Traffic Management Network (EATMN).
Commission Regulation (EC) No 550/2004 on the Provision of Air Navigation Services
in the Single European Sky
This Regulation sets out an authorisation system, compliance review mechanism and revised
payment arrangements for the provision of air navigation services within the community.
Commission Regulation (EC) No 551/2004 on the Organisation and use of Airspace in
the Single European Sky
This Regulation sets out a mechanism to establish a single coherent Community airspace
with common design, planning and management procedures.
Commission Regulation (EC) No 552/2004 on the Interoperability of the European Air
Traffic Management Network
This Regulation is designed to achieve interoperability between the Community's air
navigation service providers and the creation of an internal market in equipment, systems
and associated services.
Commission Regulation (EC) No 1070/2009 Amending Regulations (EC) No 549/2004,
(EC) No 550/2004, (EC) No 551/2004 and (EC) No 552/2004 in order to Improve the
Performance and Sustainability of the European Aviation System
This Regulation amends Regulations (EC) No 549/2004 (The Framework Regulation), (EC)
No 550/2004 (Air Navigation Service Provision), (EC) No 551/2004 (Airspace) and (EC) No
552/2004 (Interoperability).
Commission Regulation (EC) No 2096/2005 laying down Common Requirements for
the Provision of Air Navigation Services
This Regulation lays down in detail the Common Requirements (CRs) for air navigation
service provision (includes ATS. Met Services, AIS and CNS/ATM provision). (Amended by
Commission Regulation (EC) No 1315/20077, Commission Regulation (EC) No 482/20088,
Commission Regulation (EC) No 668/20089) and Commission Regulation (EU) No 691/2010.
Commission Regulation (EC) No 2150/2005 laying down Common Rules for the
Flexible use of Airspace (FUA)
This Regulation reinforces and harmonises the application with the SES of the Concept of
FUA.
Commission Regulation (EC) No 730/2006 on Airspace Classification and Access of
Flights Operated under Visual Flight Rules above Flight Level 195
This Regulation lays down the requirements for the definition and implementation of a
software safety assurance system by air traffic service providers.
Commission Regulation (EC) No 668/2008 amending Annexes II to V of Regulation
(EC) No 2096/2005 laying down common requirements for the provision of air
navigation services, as regards working methods and operating procedures
This Regulation amends the text of Annexes II to V of the Common Requirements
Regulation
Commission Regulation (EC) No 29/2009 laying down requirements on data link
services for the single European sky
This Regulation lays down requirements for the coordinated introduction of data link services
based on air-ground point-to-point data communications as defined in Article 2(5).
Commission Regulation (EC) No 262/2009 laying down requirements for the
coordinated allocation and use of Mode S interrogator codes for the single European
sky.
This Regulation lays down requirements for the coordinated allocation and use of Mode S
interrogator codes for the purposes of the safe and efficient operation of air traffic
surveillance and civil-military coordination.
Commission Regulation (EU) No 73/2010 laying down requirements on the quality of
aeronautical data and aeronautical information for the the the single European sky.
This Regulation lays down requirements for the quality of aeronautical data and aeronautical
information (ADQ) in terms of accuracy, resolution and integrity.
Commission Regulation (EU) No 255/2010 laying down common rules on air traffic
flow management.
This Regulation lays down the requirements for air traffic flow management (ATFM) in order
to optimise the available capacity of the European air traffic management network and
enhance ATFM processes.
Commission Regulation (EU) No 691/2010 laying down a performance scheme for air
navigation services and network functions and amending Regulation (EC) No
2096/2005 laying down common requirements for the provision of air navigation
services.
This Regulation lays down the necessary measures to improve the overall performance of air
navigation services and network functions for general air traffic within the ICAO EUR and AFI
regions where Member States are responsible for the provision of air navigation services with
a view to meeting the requirements of all airspace users.
Commission Regulation (EU) No 1191/2010 amending Regulation (EC) No 1794/2006
laying down a common charging scheme for air naviagtion services.
This Regulation amends the Charging Regulation (EC) No 1794/200616 in line with the
Performance Scheme IR and the introduction of determined costs from 2012.
Commission Regulation (EU) No 176/2011 on the information to be provided before the
establishment and modification of a functional airspace block.
This Regulation lays down the requirements for the information to be provided by the
Member States concerned to the Commission, EASA and other Member States and
interested parties before the establishment and moification of a FAB.
Commission Regulation (EU) No 677/2011 laying down detailed rules for the
implementation of air traffic management (ATM) network functions and amending
Regulation (EU) No 691/2010.
This Regulation lays down detailed rules for the implementation of air traffic management
(ATM) network functions in accordance with Article 6 of Regulation (EC) No 551/2004 in
order to allow optimum use of airspace in the single European sky and ensure that airspace
users can operate preferred trajectories, while allowing maximum access to airspaces and
air navigation services.
Commission Implementing Regulation (EU) No 1216/2011 amending Commission
Regulation (EU) No 691/2010 laying down a performance scheme for air navigation
services and network functions.
This Regulation amends the text of CR 691/2010 the Performance Regulation.
The Tables below are extracted from EASA’s Notice of Proposed Amendment (NPA)
No 2011-18 and show the Mapping between the SES Management Objectives and the
Survey Study Areas.
Appendix 2 Methodology
A2.1 Overview
The methodology is described in detail in ATM Safety Framework Maturity Survey –
Methodology for ANSPs (The document was distributed with the survey Questionnaires), and
summarised in Figure A2-1
.
ANSPs were asked a total of 26 questions grouped into 11 Study Areas, see Table A2-1
below.
Once the completed questionnaires are received, the organisation’s nominated focal points
are contacted to discuss the results and seek supporting evidence. In a number of cases the
responses to specific questions were revised up or down based on the interview, and a
revised questionnaire submitted for final analysis.
The site visits reports and a comparison of the survey findings against data already held by
EUROCONTROL have been incorporated into a separate confidential report.
Mathematically, the maturity score is calculated from the questionnaire responses and
weighting factors as follows:
ni , j
100∑ rk , j ,i ⋅ wk , j
S i, j = k =1
ni , j
4∑ wk , j
k =1
Where:
An overall score for each State is then also estimated by taking the average of the scores
over all Study Areas.
Preparation
• Contact the interviewee by e-mail or telephone to agree a time and date for the
interview;
• Confirm the appointment in UCT and local time;
• Read the available background material.
Interview
• Work through each response in the questionnaire seeking evidence to support the
maturity score chosen;
- Ask the interviewee to explain why the particular category was chosen;
- Use open questions;
- Let the interviewee do most of the talking;
- Use the prompts in the ANSP Interview Questions to challenge reasoning;
- Explore issues that aid or hinder progress;
- If necessary seek additional confirmation by supplementary e-mail;
- Clarify anything that you do not understand or which seems to conflict with
earlier statements;
- Try to understand whether you are being given an honest opinion or one
from which they are unwilling to move;
- Do not push too hard otherwise the interviewee will dry up and give you
nothing;
- If what you are being told agrees with the level of maturity selected, move
on;
- Record all relevant details and private notes in your project logbook;
• Confirm any actions that have been agreed (e.g. to provide information, change
maturity levels etc.);
• Ask whether there is anything you can pass on to EUROCONTROL on their
behalf;
• Confirm that the interviewee has your contact details just in case they decide they
want to provide more information;
• Thank the interviewee for their time and end the call.
Post Interview
Study Areas
Questions SA1 SA2 SA3 SA4 SA5 SA6 SA7 SA8 SA9 SA10 SA11
SA1-1 5 5 5 3 5 4 4 5 4 4 4
SA1-2 5 4 0 1 5 1 0 0 0 4 4
SA1-3 4 3 1 2 4 2 0 5 5 3 3
SA2-1 4 4 4 4 4 4 4 5 4 4 3
SA2-2 2 5 5 3 2 3 1 2 2 2 3
SA2-3 5 5 3 3 5 4 3 4 4 4 4
SA2-4 2 5 5 3 2 3 1 2 2 2 3
SA3-1 1 2 5 2 1 2 1 2 1 2 2
SA3-2 1 3 5 3 1 1 1 2 2 2 2
SA4-1 2 5 2 5 2 2 2 2 2 2 3
SA4-2 3 3 1 4 3 4 1 4 4 4 4
SA4-3 3 2 0 3 3 4 3 3 4 4 5
SA5-1 4 5 3 2 5 4 2 4 4 4 3
SA6-1 4 5 3 4 4 5 4 4 4 4 3
SA7-1 5 5 0 4 5 5 5 4 4 5 3
SA7-2 3 3 2 3 3 4 5 3 3 2 2
SA8-1 5 3 1 2 5 0 0 5 5 2 3
SA8-2 5 2 0 3 5 4 2 5 4 4 5
SA8-3 3 3 2 3 3 4 4 4 3 2 2
SA9-1 2 3 2 2 2 2 2 4 5 5 2
SA9-2 4 2 0 2 4 4 2 3 5 3 2
SA9-3 4 2 0 3 4 4 2 4 5 4 2
SA10-1 4 4 2 4 4 4 4 4 4 5 5
SA11-1 4 3 0 3 4 3 3 2 4 4 5
SA11-2 4 3 0 3 4 3 3 2 4 4 5
SA11-3 3 2 0 1 3 2 4 5 4 2 4
Those questions highlighted in Tan have a weighting of four or five and hence have a major
contribution to the maturity score for the Study Area. States wishing to improve their overall
maturity in a particular area would be advised to concentrate on improving their response to
these questions.
As there are a significant number of questions with high weightings across many Study
Areas, this will effectively smooth out the maturity scores in each area
The following table provides a summary of the participation as at 28th September 2012.
ANSP PARTICIPATION
State Questionnaires Interview Report
Score
Issued Returned Completed Completed
Albania YES YES YES YES
Argentina YES YES NO YES
Armenia YES YES YES YES
Australia YES YES YES YES
Austria YES YES YES YES
Azerbaijan YES YES YES YES
Belgium YES YES YES YES
Bosnia & Herzegovina YES YES YES YES
Bulgaria YES YES YES YES
Canada YES YES YES YES
Croatia YES YES YES YES
Curacao YES YES NO YES
Cyprus* YES YES YES YES
Czech Republic YES YES YES YES
Denmark YES YES YES YES
Estonia YES YES YES YES
Finland YES YES YES YES
France YES YES YES YES
Macedonia* (FYROM) YES YES YES YES
Georgia YES YES YES YES
Germany YES YES YES YES
Greece* YES YES YES YES
Hungary YES YES YES YES
Iceland* YES YES YES YES
India YES YES YES YES
Ireland YES YES YES YES
Italy YES YES YES YES
Jordan YES YES YES YES
Latvia YES YES YES YES
Lithuania* YES YES YES YES
Luxembourg YES YES YES YES
Maastricht UAC YES YES YES YES
Malta YES YES YES YES
Moldova YES YES YES YES
Monaco YES YES NO YES
Montenegro YES YES YES YES
Mozambique YES YES YES YES
Netherlands YES YES YES YES
New Zealand YES YES YES YES
Notes:
Appendix 6 Glossary