Where Exactly Are the Difficulties in Reasoning Logically about

Code? Experimentation with an Online System

Michelle Cook, Jason O. Hallstrom Tim Schwab, Yu-Shan Sun,
Megan Fowler Comp. & Elec. Eng. and CS Murali Sitaraman
College of Education Florida Atlantic University School of Computing
Clemson University Boca Raton, FL 33431, USA Clemson University
Clemson, SC 29634, USA jhallstrom@fau.edu Clemson, SC 29634, USA
{mcook, mefowle}@clemson.edu {tschwab,yushans,murali}@clemson.edu
Joseph E. Hollingsworth
Computer Science
Indiana University Southeast
New Albany IN 47150, USA
jholly@ius.edu

ABSTRACT about Code? Experimentation with an Online System. In

CS students can typically reason about what a piece of code
does on specific inputs. While this is a useful starting point,
graduates must also be able to logically analyze, comprehend, and
predict the behavior of their code in more general terms, no matter
what the inputs are. Results of data collection and analysis from
an online educational system show it can help to pinpoint the
difficulties in doing this for individual students and groups, and to
partition the groups in terms of their difficulties so that
instructional interventions may be better targeted. Unlike
traditional debugging, this online system helps reveal difficulties
in reasoning in more general terms because it is equipped with a
verification engine.
CCS CONCEPTS
• Applied computing → Computer-assisted instruction
• Theory of computation → Program reasoning
• Social and professional topics → Computational thinking
KEYWORDS
Activities, correctness, logic, reasoning, online system, tool
ACM Reference Format:
Michelle Cook, Megan Fowler, Jason O. Hallstrom, Joseph E.
Hollingsworth, Tim Schwab, Yu-Shan Sun, and Murali Sitaraman.
2018. Where Exactly Are the Difficulties in Reasoning Logically
_______________________________
Permission to make digital or hard copies of all or part of this work for personal or
classroom use is granted without fee provided that copies are not made or distributed
for profit or commercial advantage and that copies bear this notice and the full
citation on the first page. Copyrights for components of this work owned by others
than ACM must be honored. Abstracting with credit is permitted. To copy otherwise,
or republish, to post on servers or to redistribute to lists, requires prior specific
permission and/or a fee. Request permissions from Permissions@acm.org.
ITiCSE '18, July 2–4, 2018, Larnaca, Cyprus
© 2018 Association for Computing Machinery.
ACM ISBN 978-1-4503-5707-4/18/07...$15.00
https://doi.org/10.1145/3197091.3197133
Proceedings of 23rd Annual ACM Conference on Innovation and
Technology in Computer Science Education (ITiCSE’18). ACM,
New York, NY, USA, 6 pages.
https://doi.org/10.1145/3197091.3197133
1 INTRODUCTION
The best programmers can often state in abstract terms what a
piece of code does, i.e., predict the behavior of a piece of code,
without actually running the code on specific inputs. Clearly, this
reasoning is more difficult than being able to say what the code
will do on particular inputs, though that is a good starting point.
The long-term goal of this research—and a central goal of
undergraduate computing education—is to understand the exact
difficulties students face, and to help students reason rigorously
and logically—about the behavioral correctness on all valid
inputs—for the software they write and maintain. Educators have
much experience in achieving this objective manually on a per-
student basis, an effort that is time intensive and is often not
possible to offer for large classes of students given typical resource
shortages. The objective of this research and experimentation is
to understand where automation may be beneficial.
At the time of submission, we have developed versions of an
online system with different interfaces, and associated reasoning
activities that range from traditional constructs to ones involving
objects and calls to operations with pre- and post-conditions. Over
a period of two years, two institutions (with a few more joining
this term), three different courses, nine sections, five instructors,
and over 400 students have been involved in the experiments. The
system has been used to identify specific difficulties and provide
feedback and has been used with and without accompanying
instruction on reasoning. The benefits have been assessed by
evaluating performance objectively through data collected by the
system, attitudinal surveys, pre- and post-reasoning exercises,
exam questions, and think-alouds with a subset of students. We
have done significant automated and subjective analysis, and

39
 ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
more analysis and tool improvements are in progress. To make
the presentation of this paper meaningful within the page limits,
we restrict the focus of this paper to one experiment using the
simplest possible examples and discuss some findings in that
context to show the potential of the approach.
Spohrer and Soloway have noted the difficulties novice students
face in reasoning about compositions of statements, even as these
students seem to understand individual programming constructs
[[29]]—so we have begun our experimentation at this level of
complexity. For example, over 50% of students deem the following
code to be correct in an assessment exercise in a junior-level
software engineering course in multiple offerings:
int max (int i, int j) {
max = i + j;
if (i > j) { max = max – j; }
if (j > i) { max = max – i; }
return max; }
The error in the code is subtle. It is not correct when i = j (even
ignoring overflow/underflow errors). When the exercise is given
on a quiz or on an exam, and a student deems it correct, a plausible
conclusion is that the student overlooked the case when i = j.
Alternatively, when a student deems the code incorrect, a
plausible conclusion is that the student understood the error. Such
educator conclusions are suspect unless students are asked to
show detailed steps for their answers, and those steps are
evaluated carefully to give feedback [[5]]. Even assuming students
have a proper notion of correctness (which itself is questionable
[[20]]), what if the students who deemed the code incorrect (the
expected answer) did so because they did not understand the
effect of this sequence of assignment statements?
max = i + j;
max = max – j;
While the larger objective is to improve the logical reasoning
abilities of learners, the system and experiment described in this
paper aim to understand the specific obstacles individuals or
groups of students face in generalizing their reasoning from
specific inputs. The question is important because unless we
understand these specific obstacles, it is hard to provide effective
help. When class sizes are large, some automation is critical.
This paper is focused on learning obstacles associated with code
composition; the obstacles are language-independent and logical
in nature. To achieve the objectives, we have built a novel online
system that is supported by the freely available RESOLVE
verification engine to help beginners reason about code [[9], [28]].
Unlike debugging, the verification engine makes it possible to
learn to reason about code without assuming particular input
values. It offers a class of learner activities and directed logical
feedback not otherwise possible with standard IDEs. Since correct
answers do not have to be computed to check learner answers, the
engine forms the basis for a powerful pedagogical agent and
allows learner activities to be created easily.
2 ONLINE REASONING SYSTEM
Working memory load, also known as cognitive load [[31]], is
implicated as a major impediment to constructivist learning [[8]].
40
M. Cook et al.
The reasoning system and experiment design have been guided
by related prior research [[35]], which discusses methods to
decrease intrinsic cognitive load (e.g., a sequence of simple to
complex tasks), decrease extrinsic cognitive load (e.g.,
segmentation, contiguity, and signaling), and promote germane
cognitive load (e.g., expectancy-driven models and sub-goaling).
The system allows educators to create custom activities that
address obstacles specific to their students. It is flexible enough to
accommodate activities targeting a variety of concepts [[15], [32]]
and misconceptions (e.g., [[19]]) at varying levels of Bloom’s
taxonomy [[7]]. Due to space limitations, this paper focuses on
activities under the category of “correct evaluation of simple, self-
referential and compound assignment operators” in [[21]], which
includes activities for each “learning objective.”
The left side of the user interface shown in Figure 1 contains an
activity and reference material—scaffolding for a beginner. For
example, it may include a list of all available relational operators,
or note that “:=” is the assignment operator. (We reserve “=” for
logical equality, and this distinction appears to pose no problems
for our students.) The right side of the interface contains the
activity and a button used to check correctness.
We note that while Java was the programming language used in
the course, the slightly different language used in the presented
code appeared to pose no problems at all for students, as indicated
by their post surveys. The reasoning does require an
understanding of two “non-programming” or logical ideas
described next and explained easily.
Figure 1: Simple First Reasoning Activity Using Constants
The student’s task is to trace the given code and complete the
Confirm assertion (where “=” means equals, as in logic). The
student can only change /* expression */ in the assertion,
which should not be confused with an assert statement; nothing
is executed. The system is robust enough to detect and prevent
trivial answers, such as “Confirm K = K”. When an answer is
wrong, either the activity is repeated or a follow-up activity is
given to pinpoint obstacles in the student’s reasoning. While
learners can run code on specific inputs and check answers with
test-based systems (e.g., [[21],[33]]), the present system allows
reasoning over all inputs with the aid of a verification engine.
The example activity below contains a composition of self-
referential assignments and asks students to state the values of I
and J in terms of the input values. These initial input values are
remembered to be #I and #J, respectively, at the line marked
Remember. (This information appears in the Reference Material
 Where Exactly Are the Difficulties in Reasoning Logically ...
on the screen; it is not shown here. Note that it is also possible to
use extra variables to hold initial values and not use Remember.)
Figure 2: Reasoning Activity 3
Instead of stating concrete values of I and J at the end of the code
for specific inputs, students now have to state the values
symbolically, in terms of the initial values, namely #I and #J. We
want students to understand and be able to explain that the code
swaps the values of I and J (ignoring overflows, for now). The
(expected) correct answers are given below.
Confirm I = #J; -- Can only edit the RHS
Confirm J = #I; -- Can only edit the RHS
The reasoning system will attempt to confirm the answers using
the verification engine. In case of legitimate claims, such as the
ones below, it will agree that the stated, logically equivalent
answers are indeed correct.
Confirm I = #I + #J - #I;
Confirm J = #I + #J - #J;
As noted earlier, the system is robust enough to reject trivial
answers, such as the ones below, though they are logically correct.
Confirm I = I;
Confirm J = J;
Syntactically meaningless answers are also rejected:
Confirm I = I +-;
Confirm J = $J;
We discuss Activity 3 in some detail to motivate the ideas. To
pinpoint the learning obstacles, follow-on activities are given. For
example, follow-on Activity 3a (not shown) has shorter code and
eliminates the last assignment statement, leaving only two
statements to be traced. If the student cannot answer this
simplified version, Activity 3b is given. It is the same as Activity
3a, except that it asks the student to assert the values after each
statement. This mirrors a typical step-by-step approach to identify
where, specifically, the student is encountering an obstacle. Figure
3 shows a screenshot that captures Activity 3b, a student response,
and the system response, highlighting (in red) which part of the
answer is incorrect.
The example illustrates several points. Once an obstacle has been
identified for a specific student or group of students, the instructor
may guide students directly, or the system can guide them. This
guidance is not a part of the experiment in this paper.
Figure 3: Answers and Visual Feedback for Activity 3b
41
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
The reasoning system has been designed to provide feedback that
is immediate [[4], [24]] and reduces cognitive load [[25]]. Such
feedback prevents learners from floundering and can lead to the
same level of learning in less time. If corrected immediately,
students may not encode incorrect “knowledge” into memory
[[3]].
3 EXPERIMENTAL SETUP
The experiment was conducted in laboratory sections associated
with a second-year course on software development that
emphasizes software engineering principles; it is taught in Java.
Course details may be found in [[12], [16]]. The course description
includes specification and reasoning among the topics covered. It
is required for all CS majors. No formal instruction was given on
reasoning before the experiment, so the tool was the first
introduction to the topic, including use of Remember and
Confirm constructs. Students were allowed to ask the TAs for
help, though few took advantage. The students were not time-
constrained. They took from 20 minutes to an hour to complete
all the activities.
The experiment was conducted in the fall of 2016, and benefitted
from a piloting of the ideas the year before. There were five
laboratory sections for the course, each with 20-25 students. There
was a browser issue in the first lab section, so we did not use any
of the data from that section in our analysis. Eighty-eight students
in the other four lab sections participated.
4 QUANTITATIVE DATA ANALYSIS
Reasoning activities for this experiment were organized as
follows. The first two were introductory activities that
familiarized students with basic concepts. Of the next eight, three
involved assignment statements, three involved if-then
statements, and two involved while loops. We discuss in detail the
analysis of student responses for Activity 3. No demographic
analysis is reported because the sample sizes are small, and in
these small samples, we did not notice any appreciable
differences.
4.1 Student Performance on Activity 3
Activity 3 (Figure 2) consisted of three follow-up activities (3a,
3aa, and 3b), as shown in Figure 4. Once a syntactically correct
answer was entered, those who answered it correctly progressed
to the next activity (along the edge marked with a C). Others went
to Activity 3a (along the edge marked with an I)
Figure 4: Students Answering 3 Correctly and Incorrectly
Eighty-eight students attempted Activity 3, with 31 correctly
entering the assertions I = #J and J = #I. Sixty-five percent (or
57 out of 88 students) provided incorrect answers for one or both
 ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
of the assertions, and the system moved these students to Activity
3a. Qualitative analysis suggests possible reasons.
Activity 3a’s code is shorter. It is identical to Activity 3’s code,
except that the third assignment statement from Activity 3 is
eliminated. Correct answers for Activity 3a are: I = #I + #J and
J = #I. That is, J has the swapped value, but I does not.
By making the problem simpler and focusing on the first two
assignments, we begin the pinpointing process. We keep track of
how many students correctly answer this sub-problem, and then
begin to gauge the level of complexity where different students
begin to encounter insurmountable learning obstacles. Figure 5
indicates that 49% of the 57 students who attempted Activity 3a
answered it correctly. The system then moved these students to
Activity 3aa, and those who answered incorrectly to Activity 3b.
Figure 5: Further Pinpointing with Activity 3a
One additional obstacle pinpointing strategy incorporated into
the system is to guide step-by-step reasoning using Confirm cues
for values of variables at intermediate points. Activity 3b (Figure
3) comprises the same first two assignment statements as Activity
3 and Activity 3a, but now a Confirm is used to cue the student
for the value of the affected variable after each assignment
statement. Students who failed to correctly answer Activity 3a are
guided in Activity 3b to tackle the problem at an even finer-
grained level. The correct answer for Activity 3b’s first Confirm
is I = #I + #J, and for the second Confirm is J = #I.
Figure 6 shows the results of the 29 students who attempted
Activity 3b. They repeated Activity 3b until they succeeded. Fifty-
two percent (15/29) took one attempt to determine the correct
assertions, while 17% (5/29) took two attempts, and so on. Forty-
eight percent (14/29) of students took two or more tries at 3b, or
approximately 15% of the original 88 students. At this point, these
15% saw a version of this problem a minimum of four times (in
Activity 3, 3a, and at least two times in 3b). This observation leads
us to believe the following: There is a significant percentage of
students who fail to understand, at a fundamental level, the
computation presented in these activities. To tease out the
difficulties of this particular population, simple manipulation of
integer variables appears to be sufficient.
Figure 6: Most Struggling Students: Activity 3b Breakdown
42
M. Cook et al.
Recall that students who correctly answered Activity 3a were
moved to Activity 3aa (Figure 7, shown with correct student
responses in green). Activity 3aa is identical to Activity 3a, except
that the third assignment statement has been reintroduced,
making 3aa’s executable code identical to the original Activity 3
(Figure 2). The only difference between Activities 3 and 3aa is the
inclusion of step-by-step Confirm assertions. The motivating idea
behind 3aa is to observe how many of the students who
demonstrated understanding of 3a (a simplified version of
Activity 3) could now demonstrate an understanding of the entire
three-assignment sequence found in the original Activity 3. The
correct expressions for 3aa’s first two Confirms are exactly the
same as for Activity 3a; the last Confirm should be I = #J.
Figure 7: Activity 3aa Code
The analysis of the data for 3aa (Figure 8) shows that 50% (14/28)
of students were able to answer correctly in one attempt.
However, the other 50% took two or more tries at 3aa, indicating
that they had difficulty understanding the fundamental constructs
of addition/subtraction, assignment, and control flow. The two
groups of students–the 14 students from 3aa who took two or
more tries, and the 14 students from 3b who took two or more
tries–collectively account for 31% (28/88) of the original 88
students who attempted the sequence of activities associated with
Activity 3. Importantly, many CS1 and CS2 instructors have
observed overall course failure rates at this level.
Figure 8: Subset with Some Obstacles: Activity 3aa
4.2 Activity 6 Analysis with If-Then Statement
Activity 6 (Figure 9) is focused on finding the minimum of three
numbers. Unlike Activity 3, it has no follow-up activities. The
system repeats the activity until the student successfully
determines the correct answer: K <= #I, K <= #J, and K <= #K.
 Where Exactly Are the Difficulties in Reasoning Logically ...
Figure 9: If-then-else Activity 6
Activity 6 contains if-then branching and assignment, but no self-
referential assignments. Figure 10 shows the difficulty this
presented to students. Surprisingly, 59% of students required two
or more tries at this problem, with 18% requiring more than five
tries. Only the number of attempts are shown because there were
no follow-up activities. Using our experience we have designed
follow-up activities (not shown in this paper) to tease out learning
obstacles for reasoning about if-then and if-then-else branching.
Figure 10: Activity 6 Breakdown
4.3 Summary Analysis of Activities 1-8
Figure 11 illustrates the flow through the first four activities.
Eighty-eight students attempted them (and possibly follow-up
activities). The figure only shows the percentages of students
successfully completing the initial version of each activity.
Figure 11: Student Performance on Activities 1–4
Figure 12 shows the number of attempts for activities 5–8, with
no follow-up, for 85 students. Once we understand the best
approaches (e.g., using cueing), they will be further refined.
To see whether students can learn to generalize their reasoning to
cover all inputs, we gave a paper-and-pencil final exam question
about assignment compositions six weeks after too usage. Most
students did well, with nearly 80% of the students earning full
credit on this reasoning question.
43
ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
Figure 12: Attempts on Activities 5–8, with No Follow-Up
5 QUALITATIVE ANALYSIS
A more full-fledged qualitative analysis, necessary for validity, is
in progress using the methodology such as in [[18]]. This
subsection contains a summary of the limited initial analysis. The
analysis has benefitted from the excellent discourse in [[14]]. It is
based on the think-alouds of eight participants, to gain insight
into the cognitive process being used during the activities [[13]].
Of the eight, four were women, two were African American, and
one was Hispanic. One student had a learning disability.
Since the think-alouds were retrospective, participants were cued
to recall by completing a portion of the activities (Activities 1-3)
again. The interviews were recorded and analyzed using the
constant comparative method [[30]]; student responses were
coded to develop categories. The findings indicate the online
reasoning system helped most students pinpoint their learning
obstacles.
A subset of students who made errors on Activity 3 did seem to
understand the reasoning process, but made algebraic errors only
because they insisted on doing it “in their heads.” Only two of the
eight students, when offered, used a sheet of paper. For this group,
a step-by-step process would have been helpful.
Another subset of students had a more fundamental
misconception. Their obstacle to reasoning was not revealed in
the automated analysis. These students decided that the second
statement that assigned a value to J was not relevant in their
analysis of the value of I at the end of the code (though the third
statement uses J). In other words, to reason about the value of I,
these students crossed out the second statement and transformed
the 3-statement code in Figure 2 to the following:
I := I + J;
I := I – J;
For most of the students, the simplification of the task was
advantageous in working through their obstacles. Their
comments included that it was “less to think about” or “easier to
process.” Almost half of the students had this particular
misunderstanding and were eventually able to work through it.
One student relied exclusively on guessing and checking and
could not progress beyond Activity 2 within the timeframe of the
meeting.
6 DISCUSSION AND CONCLUSIONS
While we do not have space to report a significant number of
attitudinal findings, students generally either stated that the
activities were interesting and made them think, or were
 ITiCSE’18, July 2–4, 2018, Larnaca, Cyprus
beneficial for learning. To give an idea of some of the inputs from
students, we discuss a follow-up replication study experiment
conducted at Florida Atlantic University (a Hispanic-serving
institution) in the spring of 2017. The user interface employed a
natural language-like syntax, though the underlying reasoning
activities were the same, and the tool usage was preceded by
instruction. A total of 19 students, 10 of whom were female and
10 of whom were members of a minority group, completed an
attitudinal survey: 16 found the interface easy to use (3 neutral),
13 (6 neutral) found the learning experience useful, and 10 (8
neutral) found it engaging.
The quantitative data analysis in Section 4 illustrates the benefits
of the system and instructional approach, and will provide a basis
for classifying obstacles, distinguishing, for example, students
who have trouble reasoning about a single statement from those
who have trouble only with statement compositions. This is
accomplished at a fine-grained level, so help can be targeted.
Subsequent analysis differentiates students with difficulties with
if-then or if-then-else constructs versus students with difficulties
only when multiple such statements are composed. A junior-level
software engineering course (not discussed in this paper) involves
component-based projects [[10], [27]] and includes activities with
objects and operation calls with pre- and post-conditions to
understand specific difficulties students have with design-by-
contract. Examining students’ understanding on particular inputs
is unlikely to reveal such distinctions among individuals and
groups.
Once specific difficulties are pinpointed, the reasons need to be
understood. In some cases, system, scaffolding, or activity
redesign may be helpful. In some others, tutoring along the lines
of efforts in [[1], [6], [21], [23], [34]] might be useful. When
tutoring—manual or automated—is combined with logical
reasoning, more effective learning can result.
ACKNOWLEDGMENTS
We thank Emeritus Professor Bruce Weide at Ohio State for his
contributions to this paper. We also acknowledge US NSF grants
DUE-1161916, DUE-1609718, and DUE-1611714.
REFERENCES
[1] V. Aleven, B. M. McLaren, and J. Sewall. 2009. Scaling Up Programming by
Demonstration for Intelligent Tutoring Systems Development: An Open-
Access Web Site for Middle School Mathematics Learning. IEEE Trans. on
Learning Technologies 2, 64-78.
[2] B. S. Alqadi and J. I. Maletic. 2017. An Empirical Study of Debugging Patterns
Among Novices Programmers. In Procs. SIGCSE ‘17, ACM, NY, USA, 15-20.
[3] J. R. Anderson, A. T. Corbett, K. R. Koedinger, and R. Pelletier. 1995. Cognitive
Tutors: Lessons Learned, The Journal of the Learning Sciences 4, 167-207.
[4] R. Azevedo and R. M. Bernard. 1995. A Meta-Analysis of the Effects of
Feedback in Computer-Based Instruction. Journal of Educational Computing
Research 13, 2, 111 – 127.
[5] N. Brown and A. Altadmri. 2014. Investigating novice programming mistakes:
educator beliefs vs. student data. In Procs. ICER '14. ACM, NY, USA, 43-50.
[6] P. Bhattacharya, L. Tao, B. W. Kai Qian, and E. K. Palmer. 2011. A Cloud-based
Cyberlearning Environment for Introductory Computing Programming
Education. In Procs. 11th ICALT, 12-13.
[7] B. S. Bloom, et al. (Eds.) 1956. Taxonomy of Educational Objectives. The
Classification of Educational Goals, Handbook I: Cognitive Domain. David
McKay Company, Inc., NY, USA.
[8] M. P. Cook. 2006. Visual representations in science education: The influence of
prior knowledge and cognitive load theory on instructional design principles.
Science Education 90, 6, 1073-1091.
44
M. Cook et al.
[9] C. T. Cook, H. Harton, H. Smith, and M. Sitaraman. 2012. Specification
Engineering and Modular Verification Using a Web-Integrated Verifying
Compiler. Proc. 34th ICSE, IEEE/ACM, NY, USA, 1379-1382.
[10] C. T. Cook, S. Drachova, Y-S. Sun, M. Sitaraman, J. Carver, and J. E.
Hollingsworth. 2013. Specification and Reasoning in SE Projects Using a Web-
IDE. In Procs. 26th CSEE&T, IEEE, NY, USA.
[11] D. De Bock, J. Deprej, W. van Duren, M. Roelens, and L. Verschaffel. 2011.
Abstract or Concrete Examples in Learning Mathematics? A Replication an
Elaboration of Kaminsky, Stoutsky, and Heckler’s Study. Journal for Research
in Mathematics Education 42, 2, 109-126.
[12] S. V. Drachova, J. O. Hallstrom, J. E. Hollingsworth, J. Krone, R. Pak, and M.
Sitaraman. 2015. Teaching Mathematical Reasoning Principles for Software
Correctness and Its Assessment. ACM Trans. Comput. Education 15, 3, 1-22.
[13] K. A. Ericsson and H. A. Simon. 1998. How to study thinking in everyday life:
Contrasting think-aloud protocols with descriptions and explanations of
thinking. Mind, Culture, and Activity 5, 3, 178-186.
[14] S. Fitzgerald, B. Simon, and L. Thomas. 2005. Strategies that students use to
trace code: an analysis based in grounded theory. ICER '05 ACM, NY, 69-80.
[15] K. Goldman, et al. 2008. Identifying important and difficult concepts in
introductory computing courses using a Delphi process,” In Procs. 39th
SIGCSE ’08, ACM, 256-260.
[16] J. O. Hallstrom, C. Hochrine, J. Sorber, and M. Sitaraman. 2014. An ACM 2013
exemplar course integrating fundamentals, languages, and software
engineering. In Procs. 45th SIGCSE '14. ACM, NY, USA, 211-216.
[17] P. B. Henderson. 2003. Mathematical Reasoning in Software Engineering
Education. Communications of the ACM 46, 9, 45-50.
[18] C. D. Hundhausen, J. L. Brown, S. Farley, and D. Skarpas. 2006. A methodology
for analyzing the temporal evolution of novice programs based on semantic
components. In Procs. ICER '06. ACM, NY, USA, 59-71.
[19] T. Kohn. 2017. Variable Evaluation: An Exploration of Novice Programmers'
Understanding and Common Misconceptions. In Procs. ACM SIGCSE '17, NY,
USA, 345-350.
[20] Y. B-D. Kolikant. 2005. Students' alternative standards for correctness. In Procs.
First ICER '05. ACM, NY, USA, 37-43.
[21] R. Kumar, C. P. Rose, Y. Wang, M. Joshi, and A. Robinson. 2007. Tutorial
Dialog as Adaptive Collaborative Learning Support. In Procs. Conference on
Artificial Intelligence in Education: Building Technology Rich Contexts That
Work, IOS Press, 383-390.
[22] C. M. Lewis. 2014. Exploring variation in students' correct traces of linear
recursion. In Procs. ICER '14. ACM, NY, USA, 67-74.
[23] C. Li, Z. Dong, R. Untch, Chasteen, and N. Reale. 2011. PeerSpace—An Online
Collaborative Learning Environment for Computer Science Students. In Procs.
11th ICALT, 409-411.
[24] K. Martinez and M. Eisenhart. 2004. L.E.A.P. – Literature Review of Best
Practices in College Physics and Best Practices for Women in College Physics,
January 2004.
[25] R. Moreno. 2004. Decreasing Cognitive Load for Novice Students: Effects of
Explanatory versus Corrective Feedback in Discovery-Based Multimedia.
Instructional Science 32, 99-113.
[26] T. Price, Y. Dong, and D. Lipovac. 2017. iSnap: Towards Intelligent Tutoring in
Novice Programming Environments. SIGCSE '17, ACM, NY, 483-488.
[27] A. L. Santos. 2015. Collaborative course project for practicing component-
based software engineering, In Procs. 15th Koli Calling Conference on
Computing Education Research, ACM, NY, 142-146.
[28] M. Sitaraman, et al. 2011. Building a Push-Button RESOLVE Verifier: Progress
and Challenges. Springer Formal Aspects of Computing 23, 607-626.
[29] J. C. Spohrer, and E. Soloway. 1986. Novice mistakes: are the folk wisdoms
correct? Communications of the ACM 29, 7, 624-632.
[30] A. L. Strauss and J. Corbin. 1998. Basics of qualitative research: Techniques and
procedures for developing grounded theory. Sage Publications, Thousand
Oaks, CA.
[31] J. Sweller, P. Ayres, and S. Kalyuga, 2011. Cognitive load theory. Springer, NY,.
[32] A. E. Tew. 2010. Assessing fundamental introductory computing concept
knowledge in a language independent manner, Ph.D. Dissertation, Georgia
Institute of Technology, 146 pages.
[33] Web-CAT is an advanced automated grading system that can grade students
on how well they test their own code. http://web-cat.org/home
[34] J. B. Wiggins, et al. 2015. JavaTutor: An Intelligent Tutoring System that
Adapts to Cognitive and Affective States during Computer Programming. In
Procs. SIGCSE ’15, ACM, NY, USA, 599-599.
[35] P. Wouters, F. Paas, and J. J. G. van Merriënboer. 2008. How to Optimize
Learning from Animated Models: A Review of Guidelines Based on Cognitive
Load. Review of Educational Research.