Tutorial

Experimental Research
Experiments are, thus, particularly suited to research questions that investigate causal relations
between variables. Gibbins and Salterio (1996, p. 24) suggest four guidelines for good experimental
research in accounting:
1. A clear statement of the problem, its importance and the contribution to knowledge that its
solution will make.
2. A clear statement of the theory that underlies the process, in particular the theory that drives
the behavior and the impact of context on theory.
3. A sound experimental design – since fatal flaws can be introduced by inappropriate or
inadequate designs.
4. Recognition of the importance of external validity. If this means that we need more realistic
experimental settings, then we need correspondingly richer theoretical explanations of
resultant behavior.
The role of theory in real judgement settings
In order to take advantage of the potential strength of experimental settings in testing causal
relations, a well-developed theoretical framework is essential. Once the theoretical framework has
been established, the researcher can decide which variables should be manipulated and measured,
and which controlled.
Theory and context
With the explosion of experimental research in auditing in the mid-1970s, the use of experienced
subjects or expert practitioners as participants was thought to be essential to provide the necessary
external validity to the research. As a result, most journals have expressed a preference for the use
of practitioners in any experimental research. This has imposed a serious constraint on the
implementation of experimental work, particularly over the last ten years, because it has become
increasingly difficult to secure the participation of Big 4 practitioners, say, for experimental
research. but fortunately, for experimental work outside the auditing sphere there is increasing
recognition that surrogate participants may have the necessary skill base to participate.
Survey methods
Survey methods are often criticized as being the ‘poor man’s experiment’ because of their inability
to assign subjects randomly to treatments, and their consequent inability to rule out rival
hypotheses. Brownell (1995, p. 31), among others, recognizes the internal validity threats, but he
suggests that survey studies can be designed to minimize such threats while optimizing their
external validity benefits. He emphasizes the need for good theory in order to underpin the
specification of causal relationships.

Surveys can be conducted via mail, telephone, e-mail, internet or face-to-face interview. There is
still very little literature in the accounting domain with respect to e-mail or internet-based research.
The dominant methods remain the mail survey and face-to-face interview, each with its own
advantages and disadvantages.
Factors of mail survey methods Decline:
 A growing interest in alternative forms of research which can provide richer data sources.
 The increasing difficulty of having mail survey studies published in major refereed journals.
 Doubts about the usefulness of survey research in accounting.
 the difficulty of achieving adequate levels of response to mail surveys,
Difficulties of achieving adequate levels of response to mail surveys:

• low target populations (average only 207);

1
 • low numbers of respondents (average only 146);
• few studies using follow-up procedures to increase the sample size;
• absence of analysis of non-response bias;
• absence of studies using both subjective and objective measures of performance;
• absence of the use of sampling procedures; and,
• failure to collect both factual and psychological data within the same study, making it
impossible to link practices with behavioral variables.

Design and planning issues

A number of fundamental questions need to be answered at the design stage.
1. What sort of survey are we contemplating?
Mail questionnaires allow a large enough sample to reduce sampling error to acceptable levels, at
considerably lesser costs than either telephone or face-to-face interviews.
2. What sort of respondent are we targeting?
It will make a great deal of difference at the planning stage, depending on whether we are targeting
the population in general or a very specific proportion
3. What questions do we want answers to?
4. What response categories are we contemplating?
For example, are we asking for opinions, judgements or knowledge? Are we setting questions
which are closed (requiring yes/no, Likert-scale responses or tick-a-box type answers) or open
(allowing a considered narrative response)? We need to address these issues early on or they can
come back to haunt us.
5. What sequence of questions should we pursue?
short and easy questions should be used at the beginning, leading to the meatiest questions in the
middle of the survey, followed by relatively shorter and easier questions towards the end in order
to encourage completion of the whole survey document
6. What layout of the survey instrument?
Most authors concur that the survey should not be too long, but that, more importantly, it should
be made interesting and relevant to the target audience. Long questionnaires are more cost-
effective, but only if they are actually returned! The optimum length depends on the format of the
survey instrument (e.g., the desire to leave white space, or the requirement to provide gaps for
narrative inputs), but should not normally be greater than four pages
7. How do we select the sample?
There are several key considerations: Do we know the size of the population and its constituent
items? In many accounting research projects the answer to this question is no. As a result scientific
methods of sample selection are hindered, and If we have a known population, we will probably
have a readily available sampling frame. There are mathematical formulae for calculating the

2
required sample size to deliver the necessary statistical accuracy of estimates. We should be able
to specify the tests we intend to perform, and the number of ways that the data will be split; we can
identify all the cells of the analysis and we would like at least ten (20 is better) items in each to
give us confidence in conducting the intended statistical tests. If we cannot adequately resolve order
issues at the pilot stage of the survey, then the requirement for multiple versions of the final
instrument will expand the required sample size by the same multiple. For example, we do not want
to be in the position where we want to test for gender effects, say, with data on individuals, and
then find we have too few females in the sample (which happens more often than it should with
accounting data). Similarly, when testing for industry effects with company data, we may have too
few retail representatives, though with some countries (e.g., Australia and New Zealand) the
populations themselves may not be big enough to yield the samples required to conduct tests of all
desirable relationships.
Online surveys

Electronic interview as those conducted via the internet and highlight a whole new set of
issues to be addressed:
 Software problems replace the technical problems so often
 The richness of responses may be limited

Interview methods
The structured interview. This is the format which most closely resembles that of the self-
completion mail questionnaire
The semi-structured interview. This format allows a series of questions to be asked, but in no
fixed order.
The unstructured interview. This format commences with a series of topics for discussion, rather
than specific questions to be asked.
Several areas of concern may arise, which may cause the outcomes of the interview process
to be questioned:
 Poorly worded questions may cause confusion or misunderstandings among interviewees.
 Memory problems among interviewees can make instant responses unreliable.
 Questions may be asked in an inconsistent manner or a particular interviewer may behave
differently over time and between respondents.
 Problems may arise in the recording and processing of responses.
 Non-response bias can arise, as it does with mail surveys, but refusals are compounded by
absenteeism.