Nifwl Seirff

Honours: literature review : first draft

Anti Compiler: an Educational Tool in First Year Programming

Kymberly Fergusson

lit-review.html
School of Computer Science and Software Engineering
Monash University

PIC
Bachelor of Computre Science Honours (4300)
Clayton Campus

Literature Review 2002

AntiCompiler: an Educational Tool for First Year Programming

by

Kymberly Fergusson 11922176
Supervisor: Dr. Linda McIver
Abstract
Learning a new programming language is a difficult challenge, especially for those who have never programmed before. Various things impede progress and cause frustration for the novices.

It is therefore important that in teaching a first programming language which problems are minimised, through a well designed language, using tools to help novices develop understanding of the language, and techniques to educate about common pitfalls.

However, as many programming languages taught to novice programmers have obscure and terse syntaxes, it would be useful to develop a tool that converted this into a more descriptive, natural language.

Semantic errors are extremely for beginner programmers to locate and solve. This could be helped by running a program that automatically detects and highlights common errors, simplifying the debugging cycle.

This project aims to descover if it would be feasible for a conversion program translating a more complex language to a more familiar language, to automatically detext common semantic errors.

This review analyses exisiting research in the areas of language design, educational techniques for first time programmers, common errors made by these programmers and various tools and languages currently available.

Contents

Chapter 1
Introduction

Programming is a difficult task, especially for students who do not understand the workings of a computer, let alone have the background knowledge for linking programming concepts and learning a new language.

There has been extensive research into how to help novice programmers learn, and how to develop better strategies and languages to teach beginner programmers. There are many aspects that have been considered and found important: the psychology of how people learn; language design; error detection and recovery methods, error reporting techniques, and the various types of errors made.

Each of these aspects influence the way students learn. Many introductory programming courses teach students languages that were specifically designed by experts with little or no regard for the accessibility of the language to beginners, let alone considering the psychology of how these novices learn new concepts.

Beginner programmers, concentrating on learning programming constructs, often completely different to their prior knowledge, as well as trying to learn debugging techniques, find programming an extremely difficult task, especially when combined with a language with a terse non-natural syntax (such as C (KR88)). A language with a more natural syntax would be more accessible to beginners, even if used to represent a translation of their program.

There are two main types of errors in programming - syntax or language specific errors, and semantic or logic errors. Syntax errors are often discovered by compilers, but semantic errors most often are not. Students frequently get bogged down in the language syntax while trying to analyse their code for semantic errors. This produces frustration and can lead to students giving up, regarding programming as an impossible achievement.

Semantic errors that beginners produce are extremely common, and may be able to be detected automatically. If there existed a tool that could automatically identify and highlight such errors, it may simplify the debugging cycle, and result in a better understanding of the code and errors made.

1.1 Outline of Review

The review is divided into sections, each considers the various aspects of the project and related research in those areas. The psychology of how novice learn programming, is covered in Chapter 2.

Chapter 3 analyses the types and locations of mistakes that novice programmers commonly make and analyses some results of empirical and observational studies conducted on both novice and experienced programmers. This Chapter also discusses some existing error categorisation schemes.

An overview of the various languages, tools and teaching methods used in teaching novices programmers is provided in Chapter 4. The conclusion of this chapter focuses on methods, languages and tools that may help with error resolutions, also discussing the common methods that novices use when debugging programs.

Chapter 5 discusses why further research into these areas is needed, and analyses where the AntiCompiler fits into the existing research and combines useful features.

The review ends with some concluding remarks, recapping the previous Chapters.

Chapter 2
How Novices Learn

This Chapter focuses on the cognitive processes needed to learn new concepts and programming languages as well as techniques that help the learning process. Some types of learners are discussed, and helpful guidance techniques for both of these types are proposed. Some thoughts on how languages are learnt and how various aspects hinder the learning process are considered at the end of the Chapter.

2.1 Requirements For Learning

There are many requirements for learning, including a safe physical environment, prior experience and enough positive continuing experience to continue trying to solve problems and learn new concepts. McIver (McI02), comments that if novices have had serious problems in the past, say with computer crashing consistently, using a computer with the fear that it will crash again will pose a serious barrier to learning new concepts.

2.2 Linking New Concepts

Mayer (May86) explained that in order for someone to learn something new, it needs something to map to in long term memory, some existing concept to link to, or the learner will end up knowing the new concept only by rote. This process is called assimilation. Figure 2.1 graphically represents this process, showing that a new idea is received in short term memory, which then queries long term memory to see if there is a concept to link to, and only if there is existing knowledge will meaningful learning occur, as opposed to rote learning.


 PIC
Figure 2.1: The Linking of a New Concept.

2.2.1 Rephrasing

On technique that has been helpful in teaching in all fields is encouraging students to rephrase the question, or a paragraph in their own words. This cements their understanding of the concepts involved and can often bring misconceptions to the fore. Mayer (May86) noted that elaboration often helps link new concepts to long term memory.

2.2.2 Instruction

Several studies have found that with some instruction, especially when an impasse is reached, novices can link new information more effectively (PHH+89HS98). Perkins et.al. (PHH+89) proposed that this seemed to indicate that the beginners may feel un-confident, due to the lack of existing knowledge which would normally provide them with avenues of testing procedures, and may result in them giving up, or continually tinkering. Some guidance often helps ‘Stoppers’ past the impasse, ‘Movers’ to continue a more promising avenue or recall past tests.

2.2.3 Conflicting Information

Information that conflicts with existing knowledge often gets disregarded or altered to fit the existing conceptual knowledge (McI02). This process is called cognitive dissonance (Cor94), and can causes much difficulty in learning conflicting concepts. One example of this is where in computing, multiplication is represented with an asterisk instead of a small ‘x’.

Another difficulty arises when two concepts are only subtly different, encouraging the learner to believe they are the same.

Both of these situations can cause novice programmers distress, and lead to a feeling of helplessness, not being able to deal with the way the computer or language works.

2.3 Types of Learners

According to Perkins et.al. (PHH+89) there are two main types of learners: Stoppers and Movers.

Stoppers are the more common of the two types, and generally ‘give up’ when they reach an impasse, not looking for ways to solve the problem.
Movers are the opposite to Stoppers in that they continually try new solutions, often introducing errors and repeating solutions, never thinking clearly about which solution seemed to work better.

It was discovered that with some instruction and guidance, both groups often were encouraged to find a solution. Perkins et.al. (PHH+89) proposed that the students lacked the ‘mental model’ of the language which was required in order to break the problem down. Pane and Myers (Pan00) found supporting evidence that a representation of a solution to a programming problem in a programming language, often did not match the ‘mental model’ or natural language representation of that solution.

2.4 Learning Languages

Many empirical studies have been done to compare whether there are any cognitive differences between understanding programming concepts in a natural language versus a more obscure programming language. Dyck and Mayer (DM85) found no notable difference in errors and misconceptions when comparing the understanding of problems and representation of solutions of relatively experienced BASIC programmers to natural language representations of non-programmers, strongly suggesting that there are no differences in cognitive processes. This indicates that many of the problems that novice programmers have may be due to the unfamiliarity and complexity of the programming language they are using.

This evidence has been supported by a myriad of studies, one of which compared two languages and found commonalities in the errors and error rates (EL92). Other studies have shown that the closer a language resembles the programmers natural language and existing knowledge, the easier it is to learn and fewer errors are made (BS85May87SBE89Mur93MC96AE99McI00Pan00PRM01). Some of these studies also found that when a novice reached an impasse they frequently used a natural language representation to solve the problem(BS85AE99).

2.5 Methods to Find a Solution

As was mentioned in the previous section, many novices will represent solutions in their own natural language easier than in a programming language and will fall back on those representations when their programming knowledge is inadequate. (BS85).

A study of experienced programmers found that even they are more comfortable representing problems and solutions in a natural language than in the programming language, with 3 categories of method finding behaviour being observed:

However, these programmers typically followed these methods and reached a solution reliably were experienced. Novices find problem solving to be much harder as they don’t have the required knowledge to instinctively try a possible solution (GO86McI02). This is a problem especially where languages force the programmer to represent their solutions unnaturally. (Pan00)

2.6 Summary

Learning new concepts is a multi-faceted challenge. The novice must feel secure in their environment and have some existing knowledge that new knowledge can be linked to. Programming languages use the same cognitive processes as normal learning, so techniques that work for natural language problem solving should work for learning programming concepts and language syntax. Rephrasing concepts and receiving guidance may encourage those that are stuck to find a new way to link the information with knowledge stored in long-term memory, rather than learning by rote or giving up. Experienced programmers rely on problem solving techniques that the novice would find difficult as they would also be dealing with new concepts and syntax, making the process much more laborious.

The next Chapter will discuss mistakes made by novices, contrasting findings of various studies, and comparing with studies of mistakes made by experienced programmers, in addition, some existing categorisation methods for errors will be discussed and compared.

Chapter 3
Mistakes Made By Novices

Novices make a lot of errors, it is how much of the learning is achieved. Gugerty and Olson (GO86) found that experienced programmers found existing errors in programs much faster and more reliably than novices, and did not introduce any where near as many errors as the novices did in the course of the debugging task set.

The mistakes that novices make are very common, and typically similar within language paradigms. In order to discuss common errors, categorisation of the errors made needs to be analysed.

3.1 Categorisation of Errors

Several schemes for the categorisation of errors have been developed, the most common being the simple division between syntactic or language specific errors and semantic or logic errors. (End75McI00). Both of these areas have been sub-divided further, but as most compilers find and report syntax errors, semantic errors are more interesting and difficult to detect. Pennington (Pen87) provided a categorisation of errors that spanned both semantic and syntactic mistakes:

Operations - reading/assigning.
Control Flow - order of actions.
Data Flow - transformations of data objects throughout the program .
State - the state of the rest of the program when a specific place is reached.
Function - main goals and subgoals of a program and the role of procedures.

It was shown that the majority of the errors were in the categories State and Function, the next most common errors being Data Flow, and the least common categories Control Flow and Operations.

3.1.1 Syntax Errors

Endres (End75) found that only 15% of the errors made by experienced programmers in the DOS system (VS release 28), leaving the majority to be logic or understanding errors. This was supported by the empirical study looking at specific types of errors when novices were using LOGO and SOLO (EL92). It was found that 43.3% of the errors made were syntax errors and could be pre-empted by the development environment.This study also showed that novices do make more syntax errors than experienced programmers, however, these errors are usually dealt with by the compiler, leaving the semantic errors to be detected by hand amidst confusing concepts and language syntax.

In both cases, semantic errors comprise the majority of ‘bugs’ in programs, which is also supported by Murnane (Mur93). As they are not generally detected by a compiler, these are the errors that can cause novices the most grief.

3.1.2 Semantic Errors

Spohrer, Soloway and Pope (SSP85) classified semantic errors into two types:

Goal Drop Out is where a goal is forgotten when a merging of goals occur.
Goal Fragmentation is where a goal is broken into sub-goals and is evaluated out of order.

This classification scheme is useful as it illustrates the cognitive processes behind the errors, why the novices made them. Both errors occur when novices merge goals together, which results in a more complex structure, resulting in steps being done out of order, or being forgotten.

An alternative and more focused classification scheme was developed by Pea (Pea86), again providing insights into why the novices made the errors.

Parallelism
In natural language, a loop condition is continually tested at every step taken through the repeating set of actions. This means it is being done in parallel with the steps inside the loop. Another form of parallelism is where multiple steps are completed at the same time - again, this is often how repeated actions are represented in natural language. Consider the following example:
       while (watching tv)
       {
            eat dinner;
            drink coffee;
            read paper;
       }

The person completing these actions would normally be doing the three steps inside the loop at the same time (in parallel), and if at any time the tv is turned off, they would stop, no matter where they are in the loop. This is an example of a very common mistake that novices make.

Intentionality This is where the novice attributes forward looking capabilities to the computer. For example, if there were a condition that changed the state of watching tv further along in the program, outside the loop, the novice most likely would assume that the program could look ahead and check that condition when inside the loop.
Egocentrism The final category attributes intelligence to the computer, where the novice assumes that computer knew what they meant. An example is where the novice had meant to initialise a variable, but believed that the computer could correctly initialise the variable, possibly due to them giving it a descriptive name. Sometimes this may occur with procedures - where the novice calls a procedure SUM and thinks the computer can work out what to do from the name alone.

Other categorisation techniques have been discussed in papers, Du Boulay (DB89) proposes a different three categories:

Misapplication of analogy - an example would be believing a variable is like a box, and is able to store multiple items.
Over-generalisation - what works for one thing, works for another, an example: not requiring semi-colons at the end of comments in C means all statements don’t require them.
Inexpert handling of complexity in general, and interaction in particular - subsections of the program are incorrectly interleaved.

These various categorisation schemes serve as a varied illustration of the misconceptions novices and even experienced programmers have of programming concepts and the language constructs. However, they do not lend themselves to automatically detecting specific semantic errors in programs.

3.2 Common Mistakes - Experimental Findings

Studies have concentrated on problem areas, focusing on common novice mistakes. Several important programming constructs have been found to be notoriously difficult for novices to understand and represent correctly.

Loops
As was mentioned in the analysis of the categorisation scheme developed by Pea (Pea86) above, natural language has a completely different interpretation of repeating tasks than does a step-by-step or imperative programming language such as C (SSP85KR88). Many other studies have verified the findings, noting that novices typically assume continuous testing of loop conditions (BS85DB89Pan00).

Soloway, Bonar and Ehrlich completed an empirical study of two types of looping constructs: READ then PROCESS and PROCESS then READ. They found that the first case was correctly implemented by novice, intermediate and advanced programming students on average nearly twice as often as the second method. The first method is a more natural representation of the looping construct found in PASCAL (WJ75), and the second representation, the actual looping construct of that language.

Conditions
The study by McQuire and Eastman (ME98) concentrated on database queries using negation, and found there were a multitude of misconceptions about precedence and scope of the NOT construct. They found that the more complex the statement containing disjunctions, conjunctions and prepositions, the higher the ambiguity of the statement.

The AND construct is equally ambiguous across it’s use in natural language, mathematics and programming languages. For example, a search for cats AND dogs on the Internet will yield resources that contain references to both cats and dogs, instead of returning references to cats as well as references to dogs. Several studies have found that the AND conditional causes many problems (BS85Pan00).

Assignment
A more specific problem for languages that use = for assignment of a value to a variable and == as a comparison checking for equality (DB89). Sometime this error may simple by a typing mistake, but it isn’t detected by the compiler, and is therefore counted as a semantic error.
Not Initialising Variables
Many novices assume that the computer knows what values to initialise variables, or the novice initialises them incorrectly, sometime multiple times (JS86Pea86).
Wrong Ordering or Merged Goals
An extremely common error is programming steps in the wrong order (SSP85,  DB89). For example putting a loop counter outside of a loop, which would mean the loop would never finish. This often occurs when programmers ‘merge goals’, creating a combined, complicated and often wrong process of stepping through the operations in a program (SSP85JS86).

3.3 Summary

There has been much research into various types of common errors that novices make, with several different categorisation schemes used to describe the mistakes. Most of the categorisation methods illustrate why the novices made the error, but would be difficult to use as methods to automatically detect errors.

The next Chapter will describe some of the existing tools, techniques and languages that have been developed to help novice programmers, and various educational methods to aid the understanding of difficult constructs and languages that are taught.

Chapter 4
Tools and Techniques

There have been many tools and techniques developed to help novices learn, especially for learning difficult languages and constructs. This Chapter will discuss both desirable and undesirable features for a beginner programming language, and discuss some of the C programming language’s shortcomings. Following that, some tools developed and teaching methods used will be analysed, and finally how each of these aids can be used to help error resolution.

4.1 Languages

It has been found by numerous studies that many existing programming languages that have obscure and terse syntaxes are much harder to learn than those who have more descriptive and natural representations. (Mur93MC99PRM01BS85) Even though natural language errors are made, they are usually easier to recover from (AE99).

4.1.1 Current Languages

Most languages were developed by experts, focusing on speed, or properties specific to their domain, hardly ever considering the beginner programmer (Mur93Pan00). Most of the languages uses in first year programming courses are popular languages used in industry: C, Scheme, Ada, C++ (Lev95Con94).

4.1.2 Good Design

Conway (Con94) developed some criterion for selecting a beginner programming language:

These goals, though laudable, are typically not supported in languages available today. Many new languages have been specifically developed to aid novice programmers, GRAIL(McI02), SOLO(EL92), LOGO, MOOSE (AE99).

MOOSE was designed to be as close to the users’ natural language as possible and was used inside a MUD (Multi User Dungeon) environment by children. This study concluded that a language that was close to the natural language was highly recommended.

GRAIL was designed with the general goals to facilitate learning, and maximise readability (McI02). More specifically:

These goals are eminently more achievable when creating a language from scratch, and can be applied to a meta-language. These goals were also supported by Hsi et.al. (HS98) and Pane et.al. (Pan00).

4.1.3 Bad Design

Many languages are designed with no thought to usability and learnability, typically containing the following aspects (Moy92Mod93Mur93Pan00)

These bad aspects leave novices floundering, not able to concentrate on concepts as they are trying to cope with a badly designed, difficult to learn language.

4.2 Tools

Environments supporting programming have always been popular, especially the Graphical User Interface (GUI) with syntax highlighting, and easy access to help on various functions. Some developers have taken this further and provided tools that have built in syntax checkers (don’t have to compile to find the errors) (EL92). Du Boulay and Matthew (DBM84) built compilers that check for both syntactic and semantic errors, reporting problems with verbose comments. KyMir (BKMT94), a Russian program that uses marginal notes to debug programs without I/O was developed to reduce wasted time by teachers and students.

Brusilovsky bru:93 developed an intelligent tutoring system which integrated a visual tool to step through programs, an interactive development platform and an intelligent programming environment. He commented that this approach can be used for other languages, as it is simply built around the language.

4.3 Teaching Methods

There are many teaching methods, specifically designed for novices. Brusilovsky (Bru93) comment that teaching support for the programming language should contain the following elements:

Some methods for teaching a language have focused on:

It is important to chose reliable and comprehensive tools to support the novices’ language development. The main aim of a beginner programming course, is not to learn a difficult language, but to learn the concepts of programming (Lev95).

4.4 Error Resolution and Prevention

Using a natural and verbose language helps to reduce the number of errors made as the programmers can understand it without having to wade through confusing syntax. (AE99). Obviously accurate and descriptive error reporting techniques are a must for beginner programmers, as these would cut down the amount of time and frustration spent debugging. Currently most compilers and run-time environments do not check for semantic errors. Visual environments that can highlight errors as the novice types, and even auto-correct some errors can dramatically reduce the number of errors made. (BKMT94AE99) The tutor must be aware of common errors and explain them out to novices, before they get frustrated and give up, this allows the novice to look out for those errors, and have strategies ready for dealing with them.

4.5 Summary

Current languages are typically obscure and hard to learn, not made any easier by the tool-set that supports them. Good language need to be based on the beginner’s existing knowledge and natural language, be simple and consistent and have excellent error detection and reporting capabilities. Some of the tools that are available are of an excellent standard, but they do not support the most commonly taught and used languages. The languages that are used are poorly supported by teaching tools, and typically have very negative aspects which hinder the learning of the novice.

The next Chapter will briefly discuss where the AntiCompiler fits in, followed by some concluding remarks.

Chapter 5
Why The AntiCompiler?

As one of the most important factors in the ability of a novice to learn a new language is the requirement that the language be as close to natural language as possible, it is fairly surprising that the most commonly taught language is C, with it’s obscure and terse syntax and poor error reporting. C compilers have improved over the years and can detect and warn of syntax errors, albeit confusing at times.

It would be useful to be able to translate C programs into a more natural representation, allowing the novice to look for logic errors much more easily. It may help the novice learn the concepts more thoroughly and allow a better mapping between the solution they develop in natural language the the program they write to implement that solution.

A semantic error checker would be a boon for many programmers, novices and advanced programmers alike, as it could cut down the time spent searching for steps that are done out of order, or initialised variables, the annoying logic errors that are always so hard to find, especially for the beginner.

It remains to be seen if a reliable semantic error detection method could be designed, but that aside, simply by translating a confusing C program to a more natural representation, the novice may be able to find their errors quicker and with less frustration.

Chapter 6
Concluding Remarks

A huge amount of research has been conducted, investigating what types of errors novices make, the frequency of such errors, as well as aspects of accessible language design.

More research is needed into how much difficult languages impede the learning process. Especially as the main teaching languages are decided by what is most popular in industry, rather than those designed to aid the novice in learning the concepts of programming without having to fight with an obscure and frustrating language.

It may not be necessary to change which language, if a good set of supporting tools are provided with the difficult language, providing excellent error reporting, enforcing good program design, and providing a supportive and interactive development environment. Research into the achievability of these goals and the impact they make make on the learning process is required.

If instead of frightening novice programmers, they felt comfortable enough with the language and the learning environment, maybe better programmers would result.

Bibliography

   Bruckman A. and E. Edwards. Should we leverage natural-language knowledge? An analysis of user errors in a natural-language-style programming language. In CHI’99 Papers, 15-20 May 1999.

   P. Brusilovsky, A. Kouchnirenko, P. Miller, and I. Tomek. Teaching programming to novices: a review of approaches and tools. In T. Ottman and I. Tomek, editors, ED-MEDIA’94 - World Conference on Educational Multimedia and Hypermedia, pages 103-110, 1994.

   P. Brusilovsky. Towards and Intelligent Environment for Learning Introductory Programming, pages 114-124. Springer-Verlag, 1993.

   J. Bonar and E. Soloway. Preprogramming knowledge: A major source of misconception in novice programmers. Human Computer Interaction, 1(2):133-161, 1985.

   D. Conway. Criteria and consideration in the selection of a first programming language. Technical Report no 93/192, Monash University, December 1994.

   R.J. Corsini. Encyclopedia of Psychology. Wiley, 1994.

   B. Du Boulay. Some Difficulties of Learning to Program, pages 283-299. Lawrence Erlbaum Associates, 1989.

   B. Du Boulay and I. Matthew. Fatal error in pass zero: how not to confuse the novices. Behaviour and Information Technology, 3(2):109-118, 1984.

   J. Dyck and R. Mayer. Basic versus natural language: Is there one underlying comprehension process? In CHI’85 Proceedings, Comference on Human Factors in Computing Systems, April 1985.

   M. Eisenstadt and M. Lewis. Errors in an Interactive Programming Environment: Causes and Cures, chapter 5. Lawrence Erlbaum Associates, 1992.

   A. Endres. An analysis of errors and their causes in system programs. IEEE Transactions on Software Engineering, SE-1(2), June 1975.

   L. Gugerty and G. Olson. Debugging by skilled and novice programmers. In CHI’86, pages 171-174, April 1986.

   S. Hsi and E. Soloway. Learner centered design. SigCHI Bulletin, 30(4):53-55, 1998.

   S.A. Joni and E. Soloway. But my program runs! discourse rules for novice programmers. Educational Computing Research, 2(1):95-128, 1986.

   B. Kernighan and D. Ritchie. The C Programming Language, 2nd Ed. Prentice Hall, 1988.

   S.P. Levy. Computer language usage in cs1: Survey results. SIGCSE Bulletin, 27(3):21-26, 1995.

   R. Mayer. The Psychology of How Novices Learn Computer Programming, pages 129-159. Baywood Publishing Co. Inc., 1986.

   R. Mayer. Cognitive Aspects of Learning and Using a Programming Language, pages 61-79. MIT Press, 1987.

   L. McIver and D. Conway. Seven deadly sins of introductory programming language design. In Software Engineering: Education and Practice. IEEE Computer Society Press, 1996.

   L. McIver and D. Conway. Grail: A zero’th programming language. In G Cummings, T. Okamoto, and L. Gomex, editors, 7th International Conference on Computers in Education ICCE’99, volume 2, pages 43-50. IOS Press, November 1999.

   L. McIver. The effects of programming language on error rates of novice programmers. In 12th Annual Meeting of the Psychology of Programming Interest Group, April 2000.

   L. McIver. Syntactic and Semantic Issues in Introductory Programming Education. PhD thesis, Computer Science and Software Engineering, January 2002.

   A. McQuire and C. Eastman. The ambiguity of negation in natural language queries to information retrieval systems. Journal of the American Society for Information Science, 49(8):686-692, 1998.

   R. P. Mody. C in education and software engineering. SIGSCE Bulletin, 24(3):45-56, September 1993.

   P.J. Moylan. The case against c. Technical Report EE9240, Centre for Industrial Control Science, Department of Electrical and Computer Engineering, The University of Newcastle, NSW 2308 Australia, July 1992.

   J. Murnane. The psychology of computer languages for introductory programming courses. New Ideas in Psychology, 11(2):213-228, 1993.

   B. Pane, J. Myers. The influence of the psychology of programming on a language design: Project status report. 12th Annual Meeting of the Psychology of Programming Interest Group, 10-13 April 2000. PPIG 2000.

   R. D. Pea. Language-independent conceptual “bugs” in novice programming. Educational Computing Research, 2(1):25-36, 1986.

   N. Pennington. Comprehension Strategies in Programming, pages 100-112. Ablex Publishing Corporation, New Jersey, 1987.

   D.N. Perkins, C. Hancock, R. Hobbs, F. Martin, and R Simmons. Conditions of Learning in Novice Programmers. Lawrence Erlbaum Associates, 1989.

   J. F. Pane, C. Ratanamahatana, and B. A. Myers. Studying the language and structure in non-programmers’ solutions to programming problems. International Journal of Human-Computer Studies, 54:237-264, 2001.

   E. Soloway, J. Bonar, and K. Ehrlich. Cognitive Strategies and Looping Constructs: An Empirical Study, pages 191-207. Lawrence Erlbaum Associates, 1989.

   J. C. Spohrer, E. Soloway, and E. Pope. Where the bugs are. In CHI’85 Conference on Human Factors in Computing Systems, pages 261-279, April 1985.

   N. Wirth and K. Jensen. Pascal User Manual and Report. Springer-Verlag, 1975.