Quantitative Research

Methods

Session 1:
What on earth are we trying to do
here?
Helen Petrie
Preamble
The course is not just about how to do
statistical tests and which ones to
choose
Will introduce a much broader approach
to quantitative research methods
Why do we need this?
Examples of research
disasters/problems
Rats in mazes example: left or right
turns?
Creating technology for a problem
people dont have (very prevalent in
assistive technology)
Explaining a difference that doesnt
exist? (worst case execution time
analysis - WCET)
and an example I will draw on
Do people have difficulty navigating
websites?
What characteristics of the navigational
facilities alleviate these problems?
What inconsistencies in these facilities
contribute to these difficulties?
A starting point
What is the science in
computer science
about?

What does science mean
to you?

Write down what makes
computer science a
science

What should make it a
science
The use of the scientific method

the collection of data through observation
and experimentation, and the formulation and
testing of hypotheses (Websters Dictionary)

Is this generally what we are doing in
computer science?
Why has the scientific method got to
do with Quantitative Research
Methods?
If you are interested in QNT, then you must be
interested in undertaking something related to
the scientific method (SM), although maybe
you dont realize this

Considering the whole philosophy and
practicality of the SM should help you to do it
more accurately, thoroughly and elegantly
First pass at defining the
scientific method
Science is a structure
built upon facts
(Davies, 1968)
Empiricism
Observe (lots of) facts -->
build a theory to explain
them

Galileo often cited here -
dropping objects off
Leaning Tower of Pisa
(repeated on the moon)

What are the facts???
Is this man about to step onto the
surface of the moon or a secret
NASA film set?
Are the shadows correct, where
are the stars in the sky, did the
flag flutter, where is the blast
crater, etc etc
The observations/facts can be
hotly disputed
Excellent discussion on wikipedia
of the moon landing conspiracy
including application of the
scientific method to the debate

The facts depend on many
things
Could get into a whole
discussion of whether
you see red the way I
see red
Apart from people who
have colour vision
deficiencies (and there
are a lot of them), as
long as we agree to call
a particular range of the
spectrum red, lets not
stress this one
Facts for experts, facts for
novices
More interestingly,
what an expert
/scientist sees may
differ from what a
novice sees
e.g. X-rays, K+
mesons, finger print
whorls etc etc

Scientific method v2
What you observe and how you describe it is
shaped (I wont say determined) by your
theoretical framework

So:
Observe facts within a framework -> further
develop theory via induction -> make
predictions via deduction

Small example
A number of descriptive/relational studies show
that people have difficulty navigating websites
when the navigational bars are inconsistent in
their location through a website
(by induction)
People need consistency in navigational
mechanisms
(by deduction)
People will have more difficulty and find a
website less acceptable if the navigation is
inconsistent
Logical induction
(not the same as mathematical induction)

Particular -> general/universal

All lectures I have attended are boring.
Therefore all lectures are boring.

Problem of generalisability (highly important to the SM)
I have only attended lectures by some lecturers (a
sample out of the population of lecturers) so my logic
may be flawed


Flawed logic
Notice that I deliberately did that when I gave
my example on navigation

Studies show that people have difficulty
navigating websites when the navigation bars
are inconsistent in their LOCATION
Therefore people need consistency in
NAVIGATIONAL MECHANISMS

Maybe its only LOCATION thats important

Strong theories
However, Im making a stronger theory here, that is
easier to falsify

First lets look at the next step of the example:
People will have more difficulty and find a website less
acceptable if the navigation is inconsistent

Deduction: general to specific
Nitty-gritty of the SM, making a good, falsifiable test of
this
Turning a theoretical hypothesis into a testable
hypothesis
Three types of empirical
research
Descriptive studies:
carefully mapping out the situation (in effect,
describing the facts)
Observing behaviour, ethnographic research
Generally not enough of this in the social sciences
(because they are so busy testing their theories),
so we lack information about how people behave
(Carrolls psychology of tasks)
Are we developing software that people dont
need?
Three types of studies
Relational/correlational studies
Looking for relationships between things,
even if we dont have a theory to explain
them
fishing expedition research - looking for
what affects what, trying to find the
components for a theory
Three types of research
What Rosnow and Rosenthal (gurus of
research methods in psychology) call
experimental research
but Id rather call causal research - as its not
always really experimental
where you try and pin down the nature of the
relationships, the theory behind the
observations/facts
Test a hypothesis
Usually requires a series of studies, not just one
experiment
Creating a testable hypothesis
We tend to start from a general, vague question
Need to turn this into something specific and
appropriate
Often have two things we need to specify:
Independent variable (the aspect of the
environment that we are interested in)
Dependent variable (the behaviour that we are
interested in)

(variable = something that changes, takes
different values, that we can alter or
measure)

Operational definitions
Called operationalizing the hypothesis -
turning the vague/theoretical concepts
into operational definitions
Independent variable - the nature of the
navigation on the website
Dependent variable - the difficulty that
people have
Operational definitions II
Not necessarily one particular operationalization of a
variable that is the best
May well need multi-operationalism (i.e. different
operational definitions for variables) in the same, or
different studies, before one is confident that one has
understood a particular phenomenon
In HCI and related areas I think we do not do this
enough - one study, one measure and we move on;
in psychology one finds many studies on the same
phenomenon with slight variations published
Non-psychologists see this as obsession, but its good
science
Operational definitions
Navigational consistency
changes in navigational bars and elements of
those bars: location, font colour, background
colour, font type, exact wording, background
decoration, grouping
changes in in-text navigation: initial colour,
underlining, visited colour
So in this one very small aspect of web design,
there are many variables
One of the problems we have is isolating
exactly what is causing the problem (true
experimental design helps here - tomorrow)
Operational definitions
Difficulty that people have
Objective measures - time taken to complete
tasks, errors made
But need to consider two aspects:
will the size of the difference be noticeable?
[equipment + power calculations]
Might I get ceiling or floor effects (i.e. everyone
can do the task error free/everyone finds
something incredibly difficult)
Subjective measures - ask people to rate how
acceptable a website is (and what exactly are
you going to ask people to rate?)

Operational hypothesis
People will take longer to complete tasks, make
more errors, and give lower ratings of
acceptability on a website with a navigation
bar that varies in its location from screen to
screen in comparison to one in which the
navigation bar appears in a consistent
position on all screens
[I have multi-operationalized the dependent
variable, but have a narrow, single
operational definition for the independent
variable - tomorrow you will see why]
H0 vs H1
I have stated the alternate hypothesis - that there will be
a difference (known as H1)
I have stated it as a directional alternate hypothesis -
that Im predicting that one condition (level of the
independent variable, arrangement of the world) will
produced higher task times and errors, lower
acceptability ratings
Sometimes one is predicting a difference, but cannot
predict which direction it will take (a non-directional
alternate hypothesis) - this makes a lot of difference
in the statistical tests one conducts, and Im sure
Paul will take that up in his part of the course
H0 vs H1
The null hypothesis is the prediction that
there will not be any difference - that
navigational consistency will not have
any effect on times/errors/acceptability
ratings
In doing your statistical tests, you are
actually trying to reject the null
hypothesis
Fallacy of rejecting H0
If you do reject H0, you still might not have identified
exactly what in the situation that is causing the
difference
A problem much discussed in research methods, the
fallacy of rejecting H0
Paul Meehl, one of my heros, argued that unless you do
a very tight experiment, your chances of falling into
this fallacy is about 50% - so you might as well toss a
coin
Penguin research video
Do you need an operational
definitions/hypotheses?
A question that Im often asked by students - do
I need hypotheses for my research?
Depends a lot on whether you are doing
descriptive/relational/causal research
If causal - absolutely
If the others, it certainly helps to set out what
are your variables (theoretical/operational
definitions), the
phenomenon/question/hypothesis you are
investigating
Might not be able to formalize it to a precise H1
Allows you to make a simulation
of what you will find
Really useful to mock up the data you will
produce in a study, the levels of the
independent and dependent variables, the
numbers etc
Will it be statistically analysable (may need to
consult a statistician, but much easier for
them to advise you)
Will it really answer your hypothesis/question?

Examples
Rats: just whether they turned left or right did
not produce the right kind of data that
discriminated enough between the conditions
to answer the question posed (this was
obvious to a person with some statistical
training)
Navigation: is the question really about the best
location for the bar or whether the bar is
consistently in the same position (perhaps
you need to answer the first before the
second - a very common outcome of planning
variables and hypotheses
Reading for this session
Chalmers, A.F. (1999). What is this thing called
science? 3rd Edition. Open University Press.

Rosnow, R.L. and Rosenthal, R. (2005). Beginning
behavioral research: a conceptual primer. 5th
Edition. Pearson Prentice Hall.

Rosenthal, R. and Rosnow, R.L. (1991). Essentials of
behavioral research. 2nd Edition. McGraw Hill.