The Wrong Level of Abstraction

I found your post on the same page as this link:
http://www.daemonology.net/blog/2009-06-11-cryptographic-right-answers.html
…on Hacker News (http://news.ycombinator.com) - coincidence? :slight_smile:

Regardless of 3rd party implementation, understand the underlying technology/theory first.

Whilst sometimes the fun is taken out of somebody having already written a good abstraction library, the reality is, is that it’s likely your issue has already been encountered and solved many times before.

When building some projects I often feel like I’m playing with Lego. Referencing this library here and that one there. All with their handy API methods like ‘EncryptStringForBrowser()’.

I’m just gluing the libraries together to create the right stack that my business model can use to solve the business problem.

I’m never working at any level of abstraction other than abstraction I have written myself. I cannot rely on third party code in projects as valuable as the ones I’m using - unless I review all of it. And that will me take much more time than rewriting this from scratch.

@Taylor

I tend to agree with Jesse, although I think coders ‘roll their own’ because they don’t know enough about ‘wheels’: they’ll just try to solve their problems as they understand it (in their limited way)

jQuery is probably overkill for most tasks. Hundreds of functions that have to be reparsed by the browser on every page load - in the case of Javascript, writing to jQuery incurs a performance hit. Using a JS library is not analog to using a C library.

The only problem arises when one million websites build their security using EncryptStringForBrowser() and then someone finds an exploit on that function call.

Since no code is completely safe, it is in fact beneficial to have many languages, and many ways in those languages, to achieve the same goals. This prevents any one layer of abstraction from bringing down the whole series of tubes.

We have faced a few problem of this nature at work. Working at the wrong level of abstraction is usually a problem that can be attributed to novice users of a programming language. It takes a while to fully understand a language and much more time to find out, try and test libraries which are already available in that language. I know of novice programmers of Python writing code in Python as if they were coding in C++.

Good post.

It is not always the high-level abstractions you need to examine, it is often the lower levels too.

Do I really need slick low-profile mag-alloy wheels on my wagon or would a round wooden one do? Do I really need to optimise the hell out of this library fragment I am writing or do I just make it work perfectly?

Fear is the mind killer - don’t be afraid of just doing it right.

Paul

You need Clippy to pop up with a message saying this:

“You look like you’re being interrupted by a pop-up paperclip with delusions of Clue! Would you like to get a real development environment?”

And when the user clicks Yes, download and install Emacs…

Oh, and to those “programmers” who roll their own code instead of relying on tested, peer-reviewed libraries used (and debugged) by thousands of people… please post with the name of the company you work for and the products you sell. I need to make sure I never accidentally deal with you, for security’s sake.

@Mecki

I’m never working at any level of abstraction other than abstraction I have written myself.

So you do all of your coding in assembly?

One big reason people roll their own is both the advantage and disadvantage of third party libraries: somebody else wrote it.

In many cases, this is an advantage because the code has been written by somebody that understands the problem to be solved. The code has undergone testing and has proved to be reliable.

Then there’s the other side of the coin. How do I know I can trust other people’s code? I have seen some really awful code in 3rd party libraries. How do I know the code is processor and resource-efficient? Bug-free? Secure? Safe? Not filled with back doors or phone-homes? What side effects does it have? Is it supported? What happens if I find a bug?

I remember installing Drupal a couple of years ago, based on its popularity. I couldn’t believe the number of errors generated when starting it up the first time. When I examined the source code, it was some of the worst PHP code I had ever seen. It didn’t do any error checking and in many cases, errors were being ignored or suppressed. Hopefully, Drupal has improved since then, but you won’t find me using it any time soon.

An email parsing library I had been using was extremely memory-intensive because it used recursion inefficiently, resulting in multiple copies of email data being created. With emails containing large attachments, the library crashed with memory errors.

When I write the code myself, I know what I am getting.

@Gary has it right. Use other peoples’ code when it doesn’t suck. When it does suck, make your own. If you don’t know if it sucks, find someone who does.

I sometimes find myself paralyzed for fear of re-inventing a wheel. I’m afraid I’ll unknowingly “roll my own” implementation of something that already exists in the .NET Framework.

It means even the simplest tasks have to be started with research of what might already be there, how it works, and how well it may or may not work for my particular situation.

It’s like putting on the brakes and pulling off the road to search for a faster route. It doesn’t always pay off.

“The only problem arises when one million websites build their security using EncryptStringForBrowser() and then someone finds an exploit on that function call.”

A good counter, but there’s a counter to that too - if someone finds an exploit in widespread library code, it is known about quickly* and fixed quickly**. All that is generally required to fix an exploit that you may not even understand is a patch or replacement that someone else provides.

  • whether that be because the person who discovers it has the integrity to refrain from abusing it, or because those without that integrity cause problems that are noticed and traced back to the flaw.

** Even if that ‘fix’ is to advise people not to use it for the time-being.

LOL @ one of the re-captcha words being ‘orange’ :slight_smile:

I have exactly one objection to this (and it doesn’t apply all the time). In general, the more abstract a system is, the less efficient it ends up being.

For example, ASM vs anything else; at worst, you can always make the ASM as good as the other option and if you know what you are doing, you can often make it better. (This doesn’t always happen though).

The other end is languages like C++ or (IIRC) Lisp where well written code can get processed at “compile time” down thought the layers of abstraction until it’s just as good as the lower level option. The trade off here is that the language and libraries require much more work to develop and the tools take longer to run.

@Gary:

“When I write the code myself, I know what I am getting.”

When you use open source, you know what you’re getting.
Back in the day, when VB6 libraries were compiled from C++ (which the author rarely made available even if a VB6 dev could read C++), you had a point. Nowadays, there are libraries for everything written in your native development environment, that you can include as .Net source code in your projects. If you don’t have time to review and evaluate the source for these libraries, you certainly don’t have time to re-write them.

Modern libraries are written using popular and proven design patterns (sometimes all of them at once it seems!), in your native language, with source code provided and oftentimes free of charge. Maybe you would have done something differently, but I’m just not as confident in the likelihood of out-thinking the internet community. I’ll keep using CodeSmith, Google Charts, CodeSharp, ActiveRecord, NetTiers, etc. My apps are practically generated for me, and I have no problem admitting they’re better than I could have written, minus the business-specific knowledge I’ve gathered working for my organization.

“painfully clawing our way up the evolutionary tree of abstraction…from assembly language”

Which was a pretty high branch compared to machine language…which at least didn’t require timing memory drum spinups and flipping toggles. Which were a vastly better IDE than soldering irons and wire strippers I’m sure.

This post is a handy reminder that probably 99% of the world’s projects – even in C – is a long way from bare metal. Everything we do is standing on the shoulders of giants.

I will just rewrite this part of the post:

"Q: How do I write this in JavaScript?

jQuery is still JavaScript, GWT is actually one abstraction level up.

Someone told me one day: “There is no problem in IT that can’t be solved by adding an extra layer of abstraction”, and I must admit he hasn’t been proven wrong since…

The best encryption is no encryption

That might just be the most overly soundbitey misrepresentation of fact I red today.

You might have meant to say something like “The best way to make sure noone will read your message is to make sure no one can access the message”. This does not preculde any need to make it (cryptographically) hard to read the message in case access was gained forcefully [1].

For the record, no encryption is actually the worst encryption.

[1] see http://www.vaserv.com/ today: you’ll find that 100k websites have been compromised (in this case the (virtual) servers had been completely deleted, but they could just as easily have lurked around and harvest all CC, e.g., details on every site, lest it were decently protected by cryptography)
[2] For linguists, pedantists: Except in the strictest sense that it cannot be bad encryption if it isn’t encryption in the first place. Either way, you could safely say that no encryption does not encrypt anything very well.

I’d have to agree with Gary. The problem with using a black box is not knowing what lies therein. If the source is provided so you can review it yourself, and it is used widely (such as JQuery), then I would say the concerns are mostly alleviated. I would still be skeptical of using a lesser-known black box for security requirements.

Btw, I just noticed you have an official CAPTCHA now, no more orange. Nice!