Separating Programming Sheep from Non-Programming Goats

This article is wrong. Programming is about taking a basic set of instructions, and making something complex out of it. You can’t ask questions about basic instructions, and if a student answers it wrong assume they can’t program.

It does take time for the logical parts of the brain to develop. If you are trying to teach someone to program who can’t grasp the logic, don’t assume that the’ll never ba able to program. You just have to help them develop the logical part of the brain.

Lastly programming requiers problem solving ability. Just becouse a student is not good at this, does not mean they will never be good at it. Things like project euler can quickly get someone better at problem solving. If not, than more gradual approches are availible.

This article seems to discurage people who can’t program “in the early stage” to continue programming, they will never get better. That is BS. Everyone can get better at programming. Just becouse you fail your first test does not meant you’ll fail all of them.

I just have to wonder:

If we accept the premise that assignment + MeaningVsRules is both the first hurdle and the failure point for so many student, could we instead just spend a little extra time dealing with those points for those students who need it?

The question does seem biased towards some programming experience. If the idea was to identify an ability to somehow detect rules consistently, when they really want to exclude any previous familiarity with the language, they should have picked symbols which aren’t so C-ish. After all, they say the test takers have to detect a consistent connection, not know C. And that = sign in both the question and the answers, is also complicating the matters.

I wonder how similar the answers would have been if the questions were written like:

} a10 %
} b
20 %
a*b %

The rules are the same rules. Just replacing some unknown symbols with other unknown symbols.

Plus, I don’t like the idea that they think they can tell what represents a consistent outlook, and what isn’t. It’s perfectly possible that someone new will see a consistent rule which the researchers didn’t. It could be a seemingly silly connection, one that a programmer will never make, but it may still be consistent. Someone who doesn’t know the rules can come up with very wacky, yet totally consistent, ones.

The language, as mentioned in the comments above, is actually something I don’t have a problem with. The values are new values, even if they’re the original values. The values were just assigned, so they’re new.

On the technical side, the original value may even have been the default uninitiated memory. Without exact knowledge of the language, the compiler, and the context (maybe a and b have been defined before, and this is a re-definition? Maybe something else? Remember, this isn’t officially C, and doesn’t necessarily comply with any specific C standards) the values may very well be new.

And on the semantic side the variables didn’t have a value before they were assigned, so this is the new value. Otherwise you justify sentences like “My new car? Oh, this isn’t my new car, I just got my driver’s license and bought it yesterday, so it’s the first car I ever owned.”

Quibbling about the language is missing the point. I think part of what this test seeks to identify is the ability to determine these things heuristically, without necessarily knowing how syntax translates into semantics. That’s mostly what you end up doing in production code - you don’t write clever academic programs from scratch, you use libraries and frameworks and other people’s code that has either minimal or no documentation, and try to make sense of what it means and how to use it. If you can’t do that, and you need to have everything spoon-fed to you, then as the test states, you have no business being a programmer.

Remember that the test is looking for consistency, not necessarily correctness. Even if someone got the syntax wrong (I’m not sure how), they’re looking for the formation of a mental model, which is a required element of problem solving, which is what programming is really about. If you can’t form a consist mental model on your own, you can’t code. When you write real production code, you’re most often working in an unfamiliar problem domain and you have to do the same thing - solve a problem you don’t understand!

So it’s using C syntax, or C# syntax, or the syntax of any of the other 50 languages which is exactly the same. So what? It’s very close to the algebraic syntax you ought to have learned in grade 9 or earlier, much closer than a language like VB or Pascal.

Blame the teachers if you want, but there are lousy teachers in every discipline and you don’t see the double hump in their classes. What’s so special about programming?

Interesting discussion.

A few observations from my 23 years of programming and 3 years of teaching high school programming.

Sexism: In my prior life as a programmer/manager, about half my team was female. Quality of programming was indistinguishable between the sexes. In some teams the best programmer was female; in others it was male; often it depended on the business knowledge of the programmer rather than on the mechanics of programming.

Language to teach: I opted to teach Java as my introductory programming language. It provides a safe language for beginners: they can’t accidentally do something that trashes their system. It can also be written and run on any of the major high school platforms: PC, Mac or Linux. And yes, all three platforms are represented in my student population. And no, knowledge of multiple platforms does NOT mean that the student is a better programmer!

Beyond that, Java allows all the basic strategies of programming: procedural approach, sequential processing, compiling (OK, so it’s byte code and not a “true” compile. They’re still exposed to the thought process of translating written code into something machine readable.), logic statements, loops, recursion, data types, arrays, sorting, trees, linked lists, file IO, searching through documentation for the code they need. I also require documentation (the bane of many “true” programmers), requirements gathering, testing (string, unit full application), and good design. For my advanced class I also get into project life cycle and maintenance by taking a single application through several cycles of improvement.

Weeding out: My first couple lectures are completely from hand-outs, and the first assignments are based on that information and the class discussion. I then review the programs in class using student code (from that class or another, with the victim’s name removed) so we can see examples of good and bad design and code approaches. I also put my own code up for review, including bad code, so we can go through trouble shooting and debugging. That is the most useful teaching technique I’ve found based on the students’ “ah ha!” response. After several weeks and assignments I talk with all the students about continuing in programming or finding another area of interest. Most don’t need the conversation.

BTW, my favorite first program is to write the directions for making a peanut butter and jelly sandwich. Then I’ll pick one and misinterpret as many of the directions as I can to illustrate the stupidity of the computer, the importance of clear instructions.

English language: Having come from the corporate world of application programming and working with the user community, I emphasize to the students and to our English teachers the importance of clear writing skills. Writing is the most important part of working with users: meeting minutes, email, user documents - if they are unclear to the users, you are just wasting your time and theirs, and your project is doomed. Unless you work for someone who has those skills, but you won’t ever be in charge of many successful projects. But maybe you only ever want to be a bit twiddler, and those of us in management appreciate the bit twiddlers who work for us. (Is that food for flaming, or what?!)

Quality of teaching: There is no doubt that there are good teachers and there are bad teachers. And while there is pressure on elementary teachers, and to a certain extent on high school teachers, to use better methods and techniques, I don’t see the same urgency in the college and higher levels for teachers to keep up with current teaching techniques. In many cases they have there PhD so know what is best for us, and if we can’t get it, well, then we just don’t belong in the field! I’ve been fortunate to have had good or excellent teachers. Not that there are things I wouldn’t have changed! And for the money, there are few better college classes than evening classes at community colleges in the DC area: the students are pros in their fields, looking to improve their skills or expand their knowledge base (no trade mark infringement intended) by taking evening classes. They know what they want and are very demanding, so the best teachers are assigned to their classes.

And yes, I will play with the test and see how it does in my classes. And I’ll watch out for follow up studies, because this has said, “There is something interesting here.”

Well, this has certainly gone on much longer than I intended, but there were a lot of points in the discussion that I wanted to respond to based on my experience.

I find it curios that only one poster commented on the imperative nature of this test.

It would be interesting to see what results would follow if they’d choose a purley functional model of programming.

I can even imagine that the dichotomy isn’t between ability to program or not but rather affinity towards functional programming v. affinity towards imperative programming.

I don’t believe coding is pretty much decided from the start. There is of course the gifted programmer, who rarely has any problems. The one that doesn’t seem to understand anything, and the one struggles at first, but eventually becomes better because they don’t actually understand what’s going on at first.

I personally think using a language like java cripples the programmer with all its features. I started out in C++, passed the first course of programming with a C, then struggled with the second semester and got a D. I retook, and still only ended up with a B, then took a advanced course of C++ and got a A.

I actually enjoyed learning new things, even though they were pretty difficult. However, when I transferred, and the school that I go to now teaches java, everything seems oversimplified, and they don’t like to go into technical details, but like to talk in abstract terms.

I believe the test you described was kind of off-topic. Someone can easily read that statement from right to left and get the opposite answer ( it’s not impossible to make a language’s assignment operator in the opposite direction).

In general I would have to say that a programmer’s ability is partially decided by their initial state, but it’s not impossible to become a good programmer though experience.

You say this test is “intended for students who have never looked at a line of code in their lives”. How, then would they even know that “a=b” is changing the value of a or b at all?! If they learned algebra, they’d think that you’re saying “a and b are equal”. And how are they supposed to know what the hell “int” means? That test is bogus.

In the foolish belief that someone might actually care about reading this comment, I find this article extremely sad.

You have identified the very thing that needs to be taught in order for someone to program. You have determined a way to test whether it has been learned or not. It is beautiful. The logical next step is to figure out a way to teach that. Why do you stop there and assume the rest is pointless? Why not dig deeper and figure out how to teach that particular point?

Programming is no more difficult than reading. At one time, only the special few, the priests and scholars, were able to read. As time progressed, our teaching methods improved, and more and more people learned to read, until today, even little kids learn to read. The techniques to teach programming will likewise improve, until eventually we are able to teach it to anyone. You have made an excellent advance in this area: why did you stop there?

As woman, someone who did poorly in algebra, and never programmed (well I had BASIC in grade school, but it was just copying pre-existing programs), some of the comments here are why I initially stayed out of the computer science field. Fortunately I met some men in my department who were just really encouraging and supportive so here I am in my first year of starting programming, and I discovered that I love it. I should have known, considering that 2 years before I decided that I wasn’t happy about my lack of mathematical abilities and decided to teach myself and ended up loving it despite my performance in high school. What’s interesting is that the book I used was from the “Art of Problem Solving” series which is designed for gifted students and students taking part in math competitions. However it teaches math in a way that I love and enjoy. I wish I had it during high school. Anyway, I found computer programming to be the same. I love it, and when I find resources that explain it to me in a way that I understand I excel. I also am very fortunate to have a professor that believes that everyone can get it and hard work determines your success. I also find it interesting that when there are women in my CS class, they are usually of South Asian and Middle Eastern descent. They don’t seem to have any confidence issues, and don’t seem to buy into the theory that because they are women they are disadvantaged. I’ve learned a lot from them and try to think as they do.

(Warning: Gross generalizations ahead!) There is a real preoccupation with superior/inferior and smart/dumb in Western culture. People are believed to be inherently superior or inferior and there’s not much they can do about that. Contrast that with the Asian model which posits that your superiority in a subject is a result of how much effort you put into it.

I think the conclusion of the study represents the split in these two schools of thoughts. The authors of the study seem to accept the Western model, so they conclude that the students who don’t form a consistent model can never be taught, rather than conclude that those who don’t form a consistent model should be taught how to form a consistent model.

I would suggest using the assessment not to weed out people, but as a placement for different Intro to CS courses. Most universities should have Intro to Programming courses that separate those who have had some programming experience and those who have absolutely none. I know this is not feasible for many departments but I think to assume that all students step into the the Intro CS course with an equal footing is quite wrong. In my experience this has never been the case, and those with more experience tend to drive the level of the course upward and make it miserable for those with now programming ability.

I have to say I’m pretty glad I don’t have the authors of these articles as my prof. I’d probably be pretty disadvantaged.

I can tell the programmers from reading the comments.

You’re not reading the rules, or blindly ignoring them. Sure you can teach them to pass the initial test, but that’s useless – it invalidates the test.

I could go further, but if you don’t get it, you won’t get it. Why waste the time? I respect the rule.

Since no one has mentioned it yet, it is important to note that this 2006 unpublished study has been retracted. The retraction can be found by the original authors in the paper “Mental models, Consistency and Programming Aptitude” (ACE 2008). They conclude that:

Two years ago we appeared to have discovered an exciting and enigmatic new predictor of success in a fi rst programming course. We now report that after six experiments, involving more than 500 students at six institutions in three countries, the predictive e ffect of our test has failed to live up to that early promise.

So that’s that.

1 Like

Interesting article, I enjoyed it.

Extending on what you said, if the a=b test is for assignment and sequence, I would suggest that coding a solution for the Knight’s Tour

would be a good test for recursion / iteration. To me this begs the question, what would be the concurrency test?

An intelligent, long-form, highly referenced treatment of the subject is in the book “making software” by O’reilly. It’s chapter 7, “Why Is It So Hard to Learn to Program?” by Mark Guzdial.

44% of students formed a consistent mental model of how assignment works (even if incorrect!)
39% students never formed a consistent model of how assignment works.
8% of students didn’t give a damn and left the answers blank.

Oddly, approximately 9% of computer science students are unable even to perform basic tasks such a comprising a portion of 100%.

The authors of the paper have rejected their own thesis in the meanwhile!

Please update your article!

“We cannot claim to be separating the programming goats from the non-programming sheep”
[Source: http://crpit.com/confpapers/CRPITV78Bornat.pdf ]

Indeed, as of July 11, 2014 author has now retracted the paper in full. The retraction is available here: http://www.eis.mdx.ac.uk/staffpages/r_bornat/papers/camel_hump_retraction.pdf

2 Likes

Thanks for that, quoting the conclusion of that retraction:

There wasn’t and still isn’t an aptitude test for programming based on Dehnadi’s work. It still appears to be true that novices who answer ‘consistently’ in the test are more likely to pass a programming course. Current work, by others, begins to suggest reasons for the phenomenon and open future research avenues.

1 Like

So, what was the right answer? I don’t know how to program, but after I read this test,

Read the following statements and tick the box next to the correct answer.
int a = 10;
int b = 20;
a = b;
The new values of a and b are:
a = 20 b = 10

Because, we have a=b, so you can change numbers vice versa. This is my logic from the statement I see.

I must say that I take offense at the title of this article, which implies that greater intelligence is a barrier to accomplishment in the field of computer programming. We raise goats, and I can assure you that goats are smarter, easier to train, don’t smell half as bad, and are better looking that sheep. Sheep, on the other hand, are so dim that they can be used as a light-sink. They’re so stupid that describing them as being “as dumb as a box of rocks” is an insult to the rocks. Sheep smell so bad that skunks avoid their pasture. The only thing more stupid than a sheep is another sheep. Sheep will not only bite the hand that feeds them, they’ll trample that hand and whatever it’s attached to in a panicked attempt to get away from the feed, which they believe might be out to attack them. Goats, on the other hand, are cheerful, playful, affectionate, and…OK, pointlessly and brutally violent, but only to other goats. If ever I was to attempt to teach a ruminant to program I’d choose a goat for my experiemental subject - although they’d probably want to eat the keyboard and stand on top of the computer.

:slight_smile:

1 Like