Quality Indicators

Indikator Kualiti dalam Pendidikan Tinggi

(Quality Indicators in Higher Education)

After completing this topic, you should be able to:

•Explain the meaning of quality indicators
•Describe the characteristics of quality indicators
•Learn the importance of quality indicators in HEI
•Understand the application of quality indicators in
HEI
 Quality Indicators

Performance cultures in higher education

• HEIs worldwide have undergone reform to improve

quality
•HEIs have implemented systematic and formalized
quality assurance processes to achieve greater
efficiency and accountability
•Establishment of quality models and organizations to
audit and review university performance
•Institutional and national quality models and
performance indicators are vital components to raise
the standard of HEIs
•Quantitative performance indicators are used to
provide international comparisons
 Quality Indicators

Rationale for performance indicators

• To ensure education provided by HEIs equips

students for employment and provide the country with
a highly skilled workforce that support economic
growth.

•To contribute to educational, social, and political

values.
 Quality Indicators

Purposes of performance indicators in HEIs

• To monitor own performance for comparative

purposes.
•To facilitate the assessment and evaluation of
institutional operations.
•To provide information for external quality assurance
audits.
•To provide information to the government for
accountability and reporting purposes (Rowe, 2004).
 Quality Indicators

The use of performance indicators in HEIs

• Ensure accountability for public funds

•Improve the quality of higher education provision
• Stimulate competition within and between
institutions
•Verify the quality of new institutions
•Assign institutional status
•Underwrite transfer of authority between the state
and institutions
•Facilitate international comparisons

Ref: Chalmers (2008)

 Quality Indicators

Defining performance indicators

• Simple indicators – expressed in absolute figures

and are intended to provide an unbiased description
of a situation or process.
•Performance indicators – imply a point of reference;
for example, a standard, objective, assessment, or
comparator, are relative rather than absolute in
character. Involve value judgements.
•General indicators – externally driven and are not
indicators in the strict sense; they are frequently
opinions, survey findings or general statistics.

Ref: Chalmers (2008, quoted from Hanney, Henkel & Kogan, 1997)
 Quality Indicators

Defining performance indicators

• Currently there is no common definition of
performance indicators.
•PI cannot be considered as facts, but are goal, value
and context laden, and utilized in different ways
depending on the performance model being used.
•PI are defined as measures which give information
and statistics context; permitting comparisons
between fields, over time and with commonly
accepted standards. They provide information about
the degree to which teaching and learning quality
objectives are being met within higher education
sector and institutions.
Ref: Chalmers (2008, p.3)
 Quality Indicators

Types of performance indicators

• Input
•Process
•Output
•Outcome

The types can be categorized as quantitative

indicators and qualitative indicators.

Ref: Chalmers (2008, p.3)

 Quality Indicators

Quantitative Indicators
• define as those associated with the measurement of
quantity or amount, and are expressed as numerical
values.
Input indicators
• Human, financial and physical resources in
supporting institutional programmes, activities and
services.
Output indicators
• Output reflects the quantity of outcomes produced,
including immediate measurable results, and direct
consequences of activities implemented to produce
results. Do not demonstrate quality of education, but
quantities of outcomes.

Ref: Chalmers (2008)

 Quality Indicators

Qualitative Indicators
• associated with observation based descriptions,
rather than an exact numerical measurement or
value. Relate to or involve comparisons based on
qualities of non-numerical data such as policies and
processes for assessing students’ learning, the
experience, the content of a mission statement.
Outcome Indicators
• Focus on the quality of educational program, activity
and service benefits for all stakeholders.
•Insightful, meaningful and accurate since they are
related to the objectives of higher education.
•Students are treated as customers.
Ref: Chalmers (2008, p.5)
 Quality Indicators

Qualitative Indicators

Process Indicators
• include the means used to deliver educational
programmes, activities and services within the
institutional environment.
• qualitative information on teaching and learning
such as policies and practices.
•Considered as most practical, useful and appropriate
measures of quality teaching and learning.
Ref: Chalmers (2008)
 Quality Indicators

Research conducted in Australia

Look at 13 process indicators

• Mission, Vision and Objectives
•Teaching and Learning Plans and Policies
•Teaching and Learning Indicators
•Internal and External Performance Funds for
Teaching and Learning
•Organizational Unit Review
•Curriculum Review
•Assessment and Feedback Policies
•Graduate Attribute Statement
•Student experience
Ref: Chalmers (2008)
 Quality Indicators

Research conducted in Australia

Look at 13 process indicators

• Professional Development
•Appointment and Promotion Criteria
•Review of Academic Staff – performance
•Recognition of Excellence in Teaching and
Enhancing Student Learning Experience
Ref: Chalmers (2008)
 Quality Indicators

Research in Australia – Quality Teaching

Look at four dimensions of teaching practice

• Institutional climate and systems – commitment to
the enhancement, transformation and innovation of
learning. Measure student experience and level of
satisfaction.
• Diversity – relates to ethnic, cultural and
socioeconomic diversity as well as students’ abilities,
talents and learning approaches.

Ref: Chalmers (2008)

 Quality Indicators

Research in Australia – Quality Teaching

Look at four dimensions of teaching practice

•Assessment – the assessment tasks of student
enrolled in the program of study. Look at the design,
delivery and administration, provision of feedback,
moderation, and review of assessment.
• Engagement and learning community – student
engagement, i.e. the student commitment and
engagement with their own education. Also includes
staff engagement.
Ref: Chalmers (2008)
 Quality Indicators

Higher Education in Malaysia– Challenges

• Peranan universiti dan ahli akademik (University’s

and academics’ roles)
• Perkembangan kurikulum mengikut keperluan
pasaran (Curruculum development according to
market needs)
•Penyelidikan, pembangunan dan pengkormesialan
dalam sistem inovasi kebangsaan (Research,
development and commercialization in the national
innovation system)
•Kaedah pengajaran dan pembelajaran (Teaching and
Learning Methods)
•Perluasan akses dan mobiliti pengetahuan
(Accessibility and knowledge mobility)
 Quality Indicators
Higher Education in Malaysia–
Challenges
• Globalisasi dan piawaian melalui pemeringkatan dan
penarafan (Globalization and standardization through
development and accreditation)
•Peluang guna tenaga (Resource utilization
opportunities)
 Quality Indicators

Malaysia - Current scenario

• 20 public universities
•21 polytechnics
•37 community colleges
• >400 registered private colleges
• 21 private universities and university colleges
• 11 local university branch campuses + 5 foreign university
branch campus

Areas for indicators:

• Academic staff
• Educational programs
• Student selectivity
• Educational resources
• Governance

The method used will be peer review.

 Quality Indicators

How are universities ranked?

Different ranking approaches:

• League table – each university is assigned a
specific rank. Higher ranks indicate higher quality,
lower ranks indicate lower quality.
•Quality criteria and indicators are used in this
ranking methodology to assess universities.
•Each indicator such as research impact as the
number of citations per faculty in the Thompson
Scientific Database or teaching quality as in THES
are given weight.
•This approach are applied to all universities
assessed.
 Quality Indicators

How are universities ranked?

Different ranking approaches:

•A ranking of individual disciplines or departments
instead of whole institutions.
•A multidimensional concept of university quality
instead of a ‘one-size-fits-all’ approach, taking into
account the diversity of academic institutions,
missions and goals as well as language and
cultural specifics.
•A separate measurement and presentation of
single indicators.
•A presentation of ranking results in rank groups
(top, middle, bottom groups) instead of league
tables.
 Quality Indicators

Purposes of ranking of HEI

Ranking serves several purposes:
•Responds to demands from consumers for easily
interpretable information on the standing of HEIs.
•Stimulates competition among universities.
•Provides some rationale for allocation of funds.
•Helps to differentiate among different types of
institutions and different programs and disciplines.
•Contributes to the definition of ‘quality’ of HEIs
within a particular country.

(source: Berlin Principles of Ranking of Higher Education Institutions, 2006

http://www.che.de/downloads/Berlin_Principles_IREG_534.pdf)
 Quality Indicators
Berlin Principles of Ranking of Higher Education
Institutions
Purposes and goals of rankings:
•Be one of a number of diverse approaches to the
assessment of higher education inputs, processes, and
outputs.

•Be clear about their purpose and their target groups.

•Recognize the diversity of institutions and take the different

missions and goals of institutions into account.

•Provide clarity about the range of information sources for

ranking and the messages each source generates.

•Specify the linguistic, cultural, economic, and historical

contexts of the educational systems being ranked – should be
aware of possible biases.

(source: Berlin Principles of Ranking of Higher Education Institutions, 2006)

 Quality Indicators
Berlin Principles of Ranking of Higher
Education Institutions

Design and weighting indicators:

•Be transparent regarding the methodology used
for creating the rankings.
•Choose indicators according to their relevance
and validity.
•Measure outcomes in preference to inputs
whenever possible.
•Make the weights assigned to different indicators
(if used) prominent and limit changes to them.
 Quality Indicators
Berlin Principles of Ranking of Higher
Education Institutions

Collection and processing of data:

•Pay due attention to ethical standards and the
good practice recommendations articulated in
these Principles.
•Use audited and verifiable data whenever
possible.
•Include data that are collected with proper
procedures for scientific data collection.
•Apply measures of quality assurance to ranking
processes themselves.
•Apply organizational measures that enhance the
credibility of rankings.
 Quality Indicators
Berlin Principles of Ranking of Higher
Education Institutions

Presentation of ranking results:

•Provide consumers with a clear understanding of
all of the factors used to develop a ranking, and
offer them a choice in how rankings are
displayed.

•Be compiled in a way that eliminates or reduces

errors in original data, and be organized and
published in a way that errors and faults can be
corrected.
 Quality Indicators

Purposes of Quality Indicator System

Colorado State, USA:

•Encouraging continuous improvement by
institutions in achieving high levels of
performance.
•Measuring institutional performance and
accountability.
•Determining funding recommendations and the
funding distribution for the higher education
system.
•Build public support for increased funding for
higher education.
 Quality Indicators

Some Quality Indicators

•Baccalaureate graduation rates

•Achievement scores of graduating students on
various comprehensive examinations, tests,
and /or professional specific licensure or
certification examinations
•Graduates employed or continuing their
education
•Institutional support expenditures –
administration expenditure, expenditures per
student
•Undergraduate class size
•Faculty teaching workload
 Quality Indicators

Current scenario
• Categories of institutions
• APEX university
• Research intensive
•General

•The structure of Malaysian Qualifications Agency

(MQA)
• One-stop center for institutions for registration and
accreditation of courses
 Quality Indicators

Accountability and Quality

The concept of accountability and quality assessment in

higher education is an international phenomenon
In America, many regions are moving toward ‘performance
incentive funding’.
In Europe and Australia, the central government is directly
involved in establishing quality mechanisms through:
Quality control,
Quality audit
Quality assessment
The agencies involved are like the Higher Education Quality
Council and the Higher Education Statistics Agency (HESA)
 Quality Indicators

Accountability and Quality

The focus of attention in performance indicators in the U.S.
has been cost efficiency, scientific and technical education,
management of intellectual property produced at
universities, and undergraduate education.

Less attention is paid to graduate education and research.

Categories of costs for higher education:

Education and general: faculty and staff salaries, equipment,
libraries, administrative and basic academic computing, and
certain capital or such as rent.
Cost for sponsored research
Costs of student living: room, board, clothing, laundry,
entertainment, and etc.
Cost of foregone earnings: While disengaged from the
productive work force.
The Movement in Setting Performance
Indicators in Higher Education
(U.S.A.)

1980s
Era concerned with growth in enrollments and
access was over
Emerging issues include:
Public accountability
Quality
Productivity
Undergraduate education
In 1986, all 50 states and the District of Columbia
had developed initiatives to improve the
undergraduate education
Shift from growth funding (formula funding) toward funding
outcomes, results, and performance
These efforts paralleled developments in Europe and Australia
1990s
The development of performance indicators differs from that in
1980s
From voluntary institutional improvement to a system of
mandated public accountability
By 1994, 18 states had developed indicator systems
Greater centralization of authority
Issue raised:
Will the federal government assume greater centralized control
of higher education through areas such as accreditation and
financial aids by using a set of national goals and performance
standards?
The Future of Higher Education
The White Paper 2003

Higher education must expand to meet rising skill

needs
The social gap among those entering university
remains too wide
Many of our economic competitors invest more in
higher education
Universities are struggling to employ the best
academics
Funding per student fell 36% between 1987 and
1997
Universities need stronger links with business and
industry
 Reports of Institutional Effectiveness
(EOIE) Virginia’s Public Institutions
of Higher Learning

Annual report to provide meaningful information on

the academic quality and operational efficiency of
Virginia’s public institutions
To provide evidence of institutional effectiveness –
the extent to which institutions accomplish their
missions and students achieve their educational
goals.
Structure of the Reports
(Five Points)

Institution’s mission
The mission statement sets a vision for the
institution and defines how it will get there.
Institutional profile
In-depth views of enrollment and projections of
future enrollment.
System-wide measures
Include 14 performance measures focused on
operational efficiency and factors associated with
academic quality:
Example: Classroom and laboratory space
utilization, percentage of professionally accredited
programs and etc.
Institution-specific measures
Represent unique aspects of the mission that the
college or university chose to highlight
Core competency reports
Explore student general education assessments in
written communication and technology/information
literacy.
 Performance Indicators of California Higher
Education, 2001

Describes the scope of the current set of

indicators reported by the Commission, and
highlights recent trends based on current
information related to these indicators.
This report are divided into five categories:
Population Context,
Fiscal Context,
Student Preparation,
Student Access, and
Student Outcomes
 Key Performance Indicators (KPIs)

Key Performance Indicators are quantifiable

measurements, agreed to beforehand, that reflect
the Critical Success Factors of an organization.
They defer depending on organization
KPIs must:
Reflect the organizational goals
Be key to its success
quantifiable
 Curtin’s efficiency and effectiveness
Key Performance Indicators (KPIs)

Teaching and learning

Effectiveness indicators
Quality of graduates
Quality of teaching
Student progress and achievement
Input
Efficiency indicators
Teaching and learning expenditure
Student progress and achievement
 EFFICIENCY & EFFECTIVENESS

EFFICIENCY means: saving TIME, MONEY or

EFFORT
Efficiency measures the resources used to attain a
certain level of output

EFFECTIVENESS means how well the the job

gets done. i.e. the quality of the output.
Effectiveness measures the extent to which
outcomes have been achieved
 The End of Quality
6th Quality in Higher Education International Seminar
(Birmingham, UK) May. 2001

Three Major themes:

1. Has external quality review has its day?
2. Has control of quality been usurped by the
market and by information technology?
3. Does the development of mass education
necessarily mean the end of quality?
 Transforming Quality
7th Quality in Higher Education International
Seminar (Melbourne) Oct. 2002

Three main themes:

1. To reconceptualise how higher education
engages with access, employability and funding
issues
2. What constitutes a high quality learning process
and outcomes
3. How might quality evaluation be transformed to
help improve the quality of the experience and of
the learning?
 8th Quality in Higher Education International
Seminar (Sheffield) May. 2003

Two major themes:

1. How does student feedback inform quality?To
what extent do institutions need to adopt new
procedures to make student feedback effective?
2. What does the White Paper encourage a closer
link between quality and learning?
 The First Session of the Regional Follow-up Committee for the
World Conference on Higher Education (WCHE), 2- 3
November 2000, Kuala Lumpur Malaysia

Recommendations to Member Countries:

1. Need for ongoing efforts to broaden access
taking into account the disadvantaged groups
(women and ethnic minorities)
2. Provide increased support for staff development
and research
3. More participation of women in higher education
particularly in decision making level
 Indicators of Research Quality
in Higher Education

The vast majority of discoveries have been made

in higher education environment (Dill, 1986).
Review of literature on research productivity
highlighted several indicators which include:
Productivity dollars
Productivity publications
Peer evaluation
Productivity dollars
The number of dollars generated by research was the most
often cited measurement of success
Those universities that are ranked higher, their faculty have are
adept at obtaining research grants

Productivity publications
The number of publications is frequently used as an indicator of
quality in research
The research that is published is taken as an indication of its
quality

The types of publication which determine its quality:

Journal articles, monographs, chapters, books
Quality: reputation of publication in discipline, distribution of
publication, refereed vs. non-refereed journals, invited
chapters/papers
 OUTPUT & IMPACT FACTOR

Output versus Impact factor of publications

Output refers to how prolific the the research is

producing acceptable articles/books

Impact was determined by checking citations of

the articles over a period of years
 PEER EVALUATION

Peer Evaluation
Assemble a group of peers to review the research
efforts and make a determination of the quality of
those efforts
The concerns of such approach include:
The visiting group doesn’t fully understand the
work of the unit/individual being reviewed
(especially when it is multidisciplinary)
 Curtin’s Efficiency and Effectiveness
Key Performance Indicators (KPIs)

Research and Development

Effectiveness indicators
Research Performance Index
Research Quantum
Comparison between Curtin and all Australian
Universities
Research Funding
Research Publications
Efficiency and Effectiveness
Key Performance Indicators (KPIs)

Efficiency Indicators:
Research Performance Index
Research Funding
Research Publications

Effectiveness measures the extent to which

outcomes have been achieved
Efficiency measures the resources used to attain a
certain level of output
 Publications and Number of Ph.D. Graduates
2001 76
2002 89
2003 112

Publications
2000 1,864 (319 international journals, 212 in
local journals)
2001 1,815 (303 international journals, 204 in
local journals)
2002 2,507 (496 international journals, 328
local journals)
Malaysia was ranked #63 by MASTIC (Pusat
Informasi Sains dan Teknologi Malaysia) in terms
of production referred journals
 Critical Success factors (Research)

Wide academic base and facilities

Graduate students
Screening of IRPA application and monitoring
Evaluation of research and innovation
Research culture and administrative support
Strong research networking
Research cluster development
Incentives & in-house competition as incubator
 Steps to be Taken by Institutions to
Promote Excellence in Teaching

Define what they mean by excellent in teaching

Having well-defined criteria about excellent
teaching and standards for weighting and rating of
teaching/research/service
Weigh teaching more heavily
Increase sophistication of teachers
Promote excellent teaching, not just excellent
teachers
Not treat promotion as a separate issue
(Gibbs, 1995)
Theoretical Definition for Excellent Teachers

The character of the professor

Values, personality, and social
intelligence
The knowledge of the professor
Disciplinary and pedagogical
understanding
The actions of the professor
Problem-solving behaviors
The responses of the students
Learning outcomes
 Quality Teaching in Higher Education
Flexibility in approaches to teaching and learning (including
assessment)

Good organization of subject matter and course, including relevance

and coherence of content and planned teaching/learning activities

Effective communication

Knowledge and enthusiasm for subject matter and teaching

Facilitation of learning through student interaction and active

experience

Respect for and positive attitude toward students

Critically reflective orientation to teaching including effective use of

feedback to guide learning and improve teaching

Appropriateness and fairness in assessment and grading

Reeders, E, & Marshall, H. 1996

 GOOD TEACHING: THE TOP TEN REQUIREMENTS
By Richard Leblanc, York University, Ontario , 1998.

1. Good teaching is as much about passion as it is

about reason.
2. Good teaching is about substance and treating
students as consumers of knowledge.
3. Good teaching is about listening, questioning,
being responsive, and remembering that each
student and class is different.
4. Good teaching is about not always having a
fixed agenda and being rigid, but being flexible,
fluid, experimenting, and having the confidence
to react and adjust to changing circumstances
6. Good teaching is also about style
This is very important -- good teaching is about
humor.
7. Good teaching is about caring, nurturing, and
developing minds and talents.
8. Good teaching is supported by strong and
visionary leadership, and very tangible
institutional support -- resources, personnel, and
funds.
9. Good teaching is about mentoring between
senior and junior faculty, teamwork, and being
recognized and promoted by one's peers.
10. At the end of the day, good teaching is about
having fun.
 Quality in College Teaching:
A Research Approach

Flexibility in approaches to teaching and learning

(including assessment)
Good organization of subject matter and course,
including relevance and coherence of content and
planned teaching/learning activities
Effective communication
Knowledge and enthusiasm for subject matter and
teaching
Facilitation of learning through student interaction
and active experience
Respect for and positive attitude toward students
Critically reflective orientation to teaching including
effective use of feedback to guide learning and
improve teaching
Appropriateness and fairness in assessment and
grading