Separating Programming Sheep from Non-Programming Goats

I have not seen it mentioned, but the US Gov has known about this since the 60’s. They still use the same test for programmers, the ADPE ( Automated Data Processing Exam ). It consists of four sections, word association, numerical sequencing, memory, and dimensional correlation.

The failure rate for the tech training was around 80-90% as well.

I think the test should be in some meta-language pseudo-code because intelligent people good at algebra and math but have never touched a programming language in their lives doesn’t have to know that in something called C programming language “=” is an assignment operation. Maybe something like.

BEGIN PROGRAM
x - 10
y - 20
x - b
END PROGRAM

What are the values of x and y at the end of the PROGRAM.

People are going to lynch me for this, but those inventing pseudo-language are a bit silly. There’s already a perfect language for this: COBOL

Granted, my COBOL is incredibly rusty, so I probably can’t write code that compiles here, and it would be best to leave the division separators off, or, in fact, the whole data division, but:

MOVE 10 TO A
MOVE 20 TO B
MOVE B TO A

With a note that ‘MOVE leaves the first value intact’ if needed. (Or make it ‘COPY’ instead.)

In fact, this is basically what COBOL was designed for…having non-programmer readable programming language, and the reason it’s such a joke is that, as this article points out, that doesn’t actually work. Certain people simple cannot parse programming languages, no matter how simple the languages are.

It’s not a matter of intelligence, I’ve known very very smart people who passed programming course by ‘faking it’ by copying and pasting code they knew worked, and I’ve known pretty dumb-in-other-respects people who could write code just fine.

‘Sequence’ is a big one that programmers who have not tried to teach simply won’t believe, but there are people who simply do not order statements in their mind when evaluating them. They know to do it, they just look at blocks of code as a gestalt, and see a=10, and that’s the right answer. This isn’t due to intelligence or even knowledge level, it’s something like dyslexia. Just like people with dyslexia can manually go letter by letter to get a word, they can manually run them in order, but that’s very hard when loops or recursion or functions are involved.

There are, indeed, entire groups of people who cannot be programmers, will never be programmers, and really should be encouraged into some other line of work at the start of their programming education. Just like someone with dyslexia probably shouldn’t be a copy-editor.

Now, if someone wants to star in their own made-for-TV movie about how they overcame hardship to do what they loved to do, that’s one thing, but honestly, most of the people just heard their was a lot of money in computers, and should be firmly encouraged into law or something else where they aren’t going to always be struggling just to understand what’s going on.

Hrm. No preview. Well, let’s hope this is right.

Mr. Bauman,
I agree with what you and your wife say, however, I don’t think you realize why it’s not happening: Departments need numbers of students to get funding, building space, faculty, staff, etc. So they suck in every student they can and help them as much as they feel like whether they have a shot or not.

I think it’s reticent of one of the biggest problems with bureaucracy: It puts money where it’s needed in strange ways that have nothing to do with the the industry and everything to do with the number of people and their ability to fill out useless forms. And it rewards bad spending!

It’s possible that not every school has this glaring problem and that some may actually deeply care about the students well being and education, but I’d bet most are more like this.

Also, to other commenters: You’re not reading the study. Their conclusion was that the group that fairs the best is the group who finds a consistent model (whether it’s C like or not) and sticks with it. They also found that not much changes after they understand the syntax.
And if you really want a familiar syntax you should use an algebraic syntax:
a=5
b=6
a=b

Unfortunately, this still doesn’t make sense as algebra is formed without statements. This is more similar to formal proof language where you setup some variables early on (except you don’t assign numbers, obviously).

However, back to your idea Mr. Bauman. I think the way to do this would be private industry placement exams taken between high school and your sophomore year of college (it doesn’t make much difference if you take the wrong major in your freshman year, especially if you’re in the right college – as in college inside a university system).
So students could pay for exams that would say if they have aptitude for a given field. You could add a general aptitude test to give them some direction, but most students would likely be able to guess based on their personal history.

Charge $50 for a test like this and give nice results. I imagine you could get quite a few students on board! You might need to hire overpriced PhD’s to “back” your tests though, in which case you can charge $150 for the exams.

I think it’s interesting that so many respondents seem to see the mere possibility of a test that may reliably indicate programming talent or the lack thereof as being somehow threatening or pejorative. Have we really come to the place where an individual is lessened somehow if they don’t possess all talents? I guess I should really reexamine my value to society since I’m a pretty good programmer but I can’t carry a tune to save my life?

One of the things that intrigued me about the study is that it reminded me of discussions I’ve had with my wife. She is a top-notch educator with 30+ years experience and she has often mentioned that she thinks our institutions of higher learning should have an effective way to detect and discourage education majors who have absolutely no talent for the vocation. Her reasoning is that it is a disservice to the college students to allow them to spend so much of their time and money pursuing something they will never excel at and that it is also a disservice to the future students who wind up in their classrooms. I think the same reasoning can be applied to CS majors who have no talent for the trade. They suffer because they don’t measure up in their jobs and those of us who come after them suffer when we spend our time cleaning up their mistakes.

I think it’s important to strive for excellence in all areas but there are some areas where individuals will never achieve excellence because those individuals are not well suited for that particular type of endeavor. That doesn’t mean we should prohibit them from pursuing those goals if they’re set on doing so but it might be helpful to them to know that it’s likely to be an uphill climb; we can leave certainly leave the final decision to them.

My take on it is that the Teacher is inexperienced. I’ve not seen that sort of results in the night classes I taught for Harnell college in King City, CA and surrounding areas. At least after the first 3 or 4 years of teaching.

I believe that most “intro to computer” classes are taught by graduate students. As a group they tend to assume that Everybody already knows the basics. By the time they grasp how to teach the novice, if they ever do, they’ve moved on and a new inexperienced teacher takes their place.

Bob Durtschi

I think all the comments to the effect that “a = b is bad syntax” miss the point entirely. The test isn’t about whether this represents good syntax or not - it’s about whether the student will do well in CS or not. You can argue all you want about this syntax, but if this test is a good predictor, that’s all that counts. If it’s bad syntax, then apparently an intuitive understanding of this very bad syntax is a good predictor of success in CS - the fact that it’s bad syntax is entirely off the point.

For the record, I never thought it was bad syntax. Any math person understands things like “let x = 15. Then x+5 is 20. Now let x=20. Then x+5 is 25”. Nobody gets confused and says “Wait! You just said x=15 which means it can’t be 20!”. On the other hand “x=20+5. Then x=25. Oh, wait, now x=30.” is confusing. It just depends on whether you’re using “=” as a declarative statement about the permanent and necessary value of x or whether you’re using it as a verb to consider x to have a value of 15 for the moment. Both are used in standard math without confusion so I never really understood people’s confusion about the verb sort of usage in languages. Some languages us “let” or “set” to clarify, which I guess makes it a bit clearer that the verb form is being used, but it seems a bit unnecessary to me.

I like the guy’s comments about meaninglessness. I remember trying to teach my mom a bit of programming. If I would write down “age=20” she would immediately pipe in “no, that’s not my age!”. I’d try to explain that it’s just a name but no amount of explaining would do. Finally I’d change it to “x=20” in an attempt to divorce it from the meaning she was assigning it. At that point she’d ask what “x” was. I’d have to tell her we were using it to represent her age and we’d be back to “but I’m not 20!”. She wasn’t able to accept that “x” or “age” is meaningless in the context of the program and had to know what it “stood for” at which point all sorts of real life connotations would flood in making it impossible for her to understand what was going on.

This problem has puzzled me for years. I learned programming myself, with able tutoring from a friend. In turn, I’ve tried to teach others, some smarter than me, some not. There were some that picked up the concepts and then built on them. Others had great difficulty with the basic principles - and even when they’d gotten them, could not seem to carry forward with what they’d just learned.

I’ve always wondered at this difference. Looking back at my own history, I’ve also been curious why certain populations thrived in the school environment but never seemed to really grasp complex subjects - and vice-versa, some of the best minds felt totally out of place and suffered during their school career.

This essay really opened my eyes, and explained so much of why I felt out of place in school:
http://www.reciprocality.org/Reciprocality/r0/Day1.html
(if the comments don’t allow links, google the words
mappers,packers,software and look for the link “Thinking about Thinking”.

I don’t attach a value judgement to mapper vs. packer, each have their place; and of course no one is all one or the other, it’s a continuum. Still, it’s a useful mental model when thinking about the issues this blog post has raised.

I have the advantage that I know Richard Bornat, the author of this paper. A lot of the comments on him here make completely false assumptions. In particular, he wrote his own programming text book many years ago which was quite well-known in its time, and he was an expert in functional programming, which he taught. So the idea that he’s fixated on imperative programming, or just some dumb person trying to teach from a prescribed textbook is wrong, wrong, wrong.

Richard Bornat was Head of Department at the university Computer Science department where I teach when I joined it as a new lecturer (he’s now moved on to another university for various reasons). He taught introductory programming, and I taught it for several years after him.

Every year we used to get into fits of depression - and I’m seeing it with the new guys who are teaching intro programming now - just WHY can’t the students get it? We’ve tried everything - functional first, objects first, back to imperative first, use an IDE, don’t use an IDE, use a language they use in industry, use a simple teaching language etc etc - and the SAME pattern still happens, a huge proportion of those on the intro programming course either fail or ought to fail (sometimes we just have to be lenient on the passing requirements since if we failed too many we might lose our jobs).

What’s more, if you look at the big conferences on teaching Computer Science, you’ll find people saying the same thing the world over. No-one seems to have hit on a way of teaching intro programming so that most people who take the course pick it up and end up competent in it. If there was such a way, and Bornat and me and everyone else who teaches intro programming are just bad teachers, wouldn’t that way have become apparent by now?

Bornat’s also a political lefty, the sort of person who naturally recoils from the idea that there are innate born abilities, with people naturally divided into categories. He’s certainly not saying this because it suits his ideology. No, he’s saying it because it reflects his long experience of trying to teach programming. It seems there really is a divide between those who “get” programming and those who don’t.

The test he’s using here is acknowledged to be really simplistic, an observation which might be taken further, but with statistics so clear it’s obvious something is happening. If you read the paper, it’s clear it’s not a test on whether you’ve previously encountered C-style assignment statements as many of the detractors have suggested (did they actually read it before commenting?)

Two interpretations, not necessarily consistent:

1 - Only one person even hinted at the fact that those who best perceive abstraction would vary their answers. Test takers presumably aim for the most correct answers. There is no sound reason to assign a particular function to the ‘=’ operator (its resemblance to algebraic and programming usage is irrelevant and misleading). Thus one would have to be a gambler (most programmers are not) or a sheep to consistently assign it the same function. An actual rational actor would define the most likely interpretations - say, it either shifts the value right to left or vice-versa - and alternate between them, shooting for the highest possible score. The sheep, however, finds one answer that works, and sticks to it. Later, they will be told which one is correct and stick to that. They make good byte pushers because they uphold the banner of standards and conformity and ensure that everything is always “ready for the enterprise.” Goats tend to wander off and explore flights of fancy (variables can not be assigned as values to other variables, so the value of the variable a is now the string “b”.) Requires more intelligence and a more thorough insight into abstraction and the arbitrariness of formal systems, but they never get any work done.

2 - OK, so those who were consistent in their (incorrect) model of ‘=’ over the first few WEEKS of this class did better than those who perceived the need for consistent interpretation, but changed their interpretation after they presumably learned (or got an idea of) how ‘=’ works in nearly all commonly used programming languages?

I’m with the people who vote that the teachers sound pretty bad. IMHO the entire metaphor for dealing with concepts on computers is changing, and more control is being given to users.

I have found some programmers to have fluid powers of conceptualization and good powers of communication. Others are fairly rigid. Luckily, there are programming languages and jobs for all different types of people.

I, on the other hand, am in the “oh jeez, that makes my pupils dilate and I must sleep immediately” category.

This test was refuted by a trio of Australian computer science researchers. They found no correlation between the “consistent” group and the group that passed their intro-to-CS course.

While I understand the motivation, this test is just badly designed.

After reading the paper I find that the results simply needed a lot of guesswork to interpret, even without considering that they had problems mapping the results from the first administration to those from the second administration, i.e, some subjects couldn’t remember the pseudonames they had used for the first administration.

Besides, the Java model of assignment is simply not intuitive.
Maybe if the background of the experimenters where in ML, Python, Haskell or [insert name of well-designed language here], the results would be interpreted differently (not to mention the test would be formulated differently). Or at least maybe they would have the sense to administer the test properly so they could then actually interpret the results.

Perhaps they were on a deadline to publish something. I might be offbase here but Saeed Dehnadi is a graduate student at Middlesex. This might be his first crack at the problem. Don’t take it as gospel.

His mentor’s (Richard Bornat), background is in Java so that might contribute slightly to the bias in the test. Even though he appears to be proficient at it.

Also, and I’m nitpicking now, Saeed’s webpage has no title. This might be oversight or might be choice, but it suggests one of two possibilities, either he pays less attention to detail than he should or he lacks creativity. Admittedly, it could also mean he has little time for such things given all the incredible code he writes, you know, with lots of formal content.

Okay, I was being sarcastic, but I agree with Andrew Shebanow above, the last comment on page 16 of the paper is extremely arrogant and elitist. I mean, how would they stack up next to a software engineer at Googleplex?

Philosophy is similar in many respects. Many people can argue, but few understand the rules of logic to form a coherent argument.

In philosophy, much like programming, you have to have the intellectual integrity to succumb to a logical argument. An argument is composed of 1: givens(axioms) 2: Logic. In order to refute an argument you have to 1: disagree with the Axioms or 2 Find fault in the logic. If you agree with the axioms and cannot find fault in the logic, you must always agree with the conclusion. No matter what. Its hard to grok at first, much like understanding that the computer has no clue what you MEANT to do. It will only do what you SAID to do.

Philosophy students have a similar problem understanding an axiom vs. logical operand in their first year.

Highly illogical paper. Who says we want to be taught?

I find it surprising that only two commentators mentioned that Intro to CS is usually a weeder course. Do they really want to teach the kids who don’t know any programming, or have they set up the course (and this test) as a filter to find those who don’t need an Intro to CS class.

I had a teacher who was once in charge of grading the AP computer science test back when they still used Visual Basic. He ran statistics between correct answers on the questions and between the actual score the student got.

He found a few questions that, had they asked only those, could have scored students with something like 80-90% accuracy. The question with the highest correlation was this (sorry, I only know VB.NET syntax if this is off):

What is the value of a Boolean x after the following line of code?

x = (x = false)

  1. x is always true
  2. x is always false
  3. x flips
  4. x stays the same
  5. None of the above

Basically, if a test-taker got this question right, s/he succeeded on the exam, and if s/he got this question wrong, they didn’t do well.

I think it’s a similar number for mathematics. Some people just can’t get it.

I went to a tech college for computers and remember all the moaning in algebra class. It's worthless and a shouldn't be required computers come with a calculator. When am I ever going to need this junk.

These where the same people who where lucky to do very poorly in the programming classes.

before i started learning to program, i would have been in the 8% who refused to answer. i would have been appalled at this test, would have stopped execution of the task right at seeing the string “int”.
my logic just refuses to go further if there are unclear details. and maybe that’s why only assembly could lure me into trying to learn programming. no other language could. i’m serious. :slight_smile:
so, what does the paper predict about the 8%? i don’t see mention of that?
now, i’m still just learning, started it recently, can’t say if i’m going to be a good programmer or not (i love assembly but i need to learn some other languages obviously to actually make serious programs), but now i’m at least able and willing to answer this test question :slight_smile: (in the way a c++ programmer would.)
so it isn’t true a (GOOD) course can’t change some students. i don’t like the generalization drawn from this test. but i do agree that you need some abilities to become a good programmer. i used to help classmates with math… they were really dumb, i can’t imagine them learning programming. not saying there is a very strong correlation between math and programming skills, friend hates math but can do cool stuff in programming.
oh and i said GOOD course! i did have a programming course in elementary school and absolutely didn’t learn anything from it. for the same reason i would have been in the 8%. we were just told to copy the program lines and then press F5 or whatever to run the program (it was qbasic and pascal in the old times). no chance for me there. and i felt stupid too, i saw some guys were playing with the computers in the class in a cool way, and i had no clue, further discouraging me, preventing me from looking at programming, for about 10 years! so yes social circumstances and teaching quality do matter too…

felix, you’re right, you need to give time for practice (reading with little practice won’t ever teach you to program). that got me thinking… the paper on this subject could have gone wrong in one aspect: how much time did they give for students to get into programming / get comfortable with it? then lets assume the ones who’d been able to answer are the ones who’d already had some experience in programming.
(the other aspect is the quality of the teaching :slight_smile: )

but, of course, one thing might eliminate this possibility: they did accept the solutions that were consistent even if not giving the “correct” solution. that doesn’t require knowing any language, just having a strong enough sense of logic.