Rate Limiting and Velocity Checking

No dogs in restaurants - because they cannot be controlled
Only small groups of students - because large groups cause problems
Only 1 forum post per 30 seconds - because people who do more are usually spammers

Dogs can be well behaved in restaurants, but they tend not to be, and other customers may complain even if they are well behaved (I would…)

Large groups of customers are a good thing when they do not discourage other customer from entering the store, and they are all buying goods, large groups of students tend to be unruly and put off other customers and tend not to buy much… shoplifting is not (generally) a problem with students (or if it is then a lone student will shoplift as well)

Enthusiastic forum posters are a good thing, but someone who posts many times a minute tend not to be contributing anything useful, or is a spambot

I wonder how many searches a minutes you have to submit to get the captcha on Google? I suspect it is more than is humanly possible?

@everling
Is there any other effective search engine with as simple an interface?

Try scroogle:
http://www.scroogle.org/

We actually have a limit on the number of http requests per hour on an IP. It might sound crazy but I monitor how often it happens (the site sends me an email with relevant info anytime someone receives the message, and it gives them a nice little email link to complain if they want to).

So far it’s caused very few false positives and defeated an army of problems, especially from the perspective of website statistics.

@Mike: Just as crazy is having as many unallocated / future use blocks as they do, despite claiming that IPv4 address space is nearly used up.

Listed as unallocated:
1/8
2/8
14/8 (as of Feb 2008)
23/8
27/8
31/8
36/8
37/8
39/8
42/8
46/8
49/8
50/8
100/8
101/8
102/8
103/8
104/8
105/8
106/8
107/8
175/8
176/8
177/8
179/8
180/8
181/8
182/8
183/8
185/8
233/8

Listed as future use:
240/8
241/8
242/8
243/8
244/8
245/8
246/8
247/8
248/8
249/8
250/8
251/8
252/8
253/8
254/8
255/8

That’s a total of 47 of 256 /8 blocks. That around 18.36% of IPs in the IPv4 system that aren’t allocated… not counting the 16 /8 blocks reserved for Multicast, the 127/8 block reserved for a single IP (127.0.0.1), or counting any unallocated blocks in the CIDR networks.

233/8 was a typo, it should have been 223/8

Nice post, these real world problem turned into more abstract design discussion posts are good!

re: Your experience with craigslist

There’s no rate limiter like a cease and desist letter.

For my website or web app, definitely.
But for my clients, wait. There are many things in this. This world is full of unscrupulous people and your clients are a subset of that set. There’s a lot of overlap. Plus there’s ignorance.

Two really well designed web pages that do snappy work can take multiple football-fields of server farms, across the globe - just TWO web pages of a very elementary web app we are familiar with.

There are people in this world who pay you for making websites and they look at this 2-page wonder and say - Hey! that has just two pages, how difficult could it get!
And if you were to tell them that Wall Street is going crazy over those two pages, they’d laugh at you. But it’s they who need to be laughed at, no, pitied.
Most of my customers want websites and they have very able answerers/advisers who tell them that a web page is well a page - if it does things, it is a page plus some bells and whistles - how much could that be?

With such a customer, you need to give them exactly that - a web page with bells and whistles. When their site gets hacked badly, and they come running at you, ask them that given that the world is full of thieves, would they blame the carpenter or the brick-layer or the gardener or the electrician or the plumber for their home being burgled?
That’s the only language they understand - that of experience.
Tell them that they’d have to take extra precautions for that and call in some experts. Tell them that you have that expertise and that you can do that too, but that costs because they never told you they wanted an unbreakable site - they asked for a page plus bells and whistles.
No kidding. Don’t ever give more than needed to such clients. Let them learn the hard way. I’ve tried being perfect on Day One. The result is that I have been perfect on day one - but not my clients - even about their only work - checking and paying. Ignorance about programs and technology is found everywhere. It is in the air you breathe :wink: .
The people to show off your good code are other developers who have scruples and knowledge.
and you’ve got to keep looking for the signs that tell these from those - eventually, it is case-by-case, as are most things in this world, actually.

/rant

Jeff, it is not shoplifting or keeping an eye on them.

Teenagers as a group tend to be a pain in the ass. They all want to show off how cool they are. So they bother other customers or mess up your store. It doesn’t mean they’re going to steal from you.

Not letting a dog in a restaurant is simply because dogs CAN be annoying if they don’t behave. The only solution is to ban all dogs, because it’s hard to decide which dog will behave. It’s not because they dislike dogs, they want their customers to be satisfied and a constantly barking dog won’t do that.

Very cool post, very well written! I would have thought to read something like this in a magazine.

Right, just to be clear, I haven’t read ALL the comments, but would like to say that, in terms of determining the type of person behind the IP address, Google icould/i use the user info it gathers as you surf, or any other type of info it has on you. And even though you may have a dynamic IP, you still (probably) only have one or two or three GMAIL accounts being used from that computer.

So in essence, it would probably make more sense to look at the behaviour of the users trying to access whatever it is you’re limiting and then applying restrictions to the computers (MAC addy?) to those who are abusing the service, or who seem likely to do so. Of course this is not perfect, but could work better than simply creating a bottleneck by applying restrictions to everyone.

On the origin of ridiculous signs… this one’s pretty funny:

:dog: Dogs are welcome in this motel. We never had a dog smoke in bed and set fire to the blankets. We never had a dog who stole our towels or played the TV too loud. We never had a dog that got drunk and broke up the furniture or punched holes in the walls. So, if your dog can vouch for you, you are welcome too.

3 must be the limit that 1 store holder can keep an eye on at any given time. So what we need is more advanced charging systems where all items have RFID, which are read when the person goes through a reader. The products might be in pockets, puches, bags or a trolly. Whatever the case - nobody leaves without paying for whatever rung up.

Not a hard system.

How does this equate to your problem of a spammer? When faced with a problem in the current system, come up with a better system.

Have people report a question OR reply as spam. People are much better at identifying spam than any computer/system you can put in place. If enough people flag a question, reply or user as spam, their account and all posts can be put on hold until reviewed. Like graffiti getting painted over - if spam is taken down quickly every time it is put up, the spammers lose interest and stop doing it as much.

Can’t this process be open to abuse? Yes - but once reviewed as safe you can raise the bar for the question/answer/uer to be marked as spam again. Similarly, if a particular user is marking too many things as Spam within a short timespan, their account can just as easily be marked spam worthy.

That’s what I would do anyway.

For those interested in the child rights issue raised here, I urge you to consult the UN convention on the rights of the child:

http://www.unhchr.ch/html/menu3/b/k2crc.htm

This convention has been ratified by 193 countries including every member of the UN bar Somalia and - that bastion of freedom and democracy - the United States. Obviously, it’s arguable whether this particular practise is covered by the UNCRC; although article 15 states Parties recognize the rights of the child to freedom of association and to freedom of peaceful assembly., that probably doesn’t cover shop premises which are, after all, private property - IANAL.

However, I think it’s safe to assume that the intent of the CRC, and of those nations that ratified it, was to end discrimination against children, which this certainly feels like. Of course, certain laws may involve discriminating against children (in the interests of health, for example) but I am not aware of any law regarding the number of people in a shop.

Saying that the sign is there because the owner has previously had problems with children is no defence. Any form of discrimination can be ‘justified’ by reference to an individual example. That does not excuse tarring an entire race, gender, or generation with such an ugly brush.

@Bobby Jack

Well if that UN convention makes it impossible to limit the number of students in a store, the US made a very wise decision not to sign it.

@Bobby Jack

Actually, young or not, I think the Only 3 students in the store at a time, please. is probably less offensive than the far more accurate Only three loud, obnoxious, shoplifting dickheads in the store at a time please… Funny that you should mention the right of assembly, though, because I’ve met a very large number of perfectly likeable teenagers that simply make asses of themselves when in groups with their peers. Strangely, that sense of anonymity translates very well to web design, when users feel free to do whatever they like when they do not feel like they have identified themselves. Jeff draws an excellent parallel here.

For a good view of what life would be like if children had legal rights equal to those of their parents, read Serpent’s Teeth by Spider Robinson.

I think one can remove the limit on IP-basis, because it only annoys the regular users and does not limit the bad users at all.

The good users are annoyed once more than a single user is behind the same router, which is certainly common, think universities, schools or companies.

On the bad side, I have myself implemented a program that circumvents such filters and limitations on IP-basis in a few hour in python. Just grab a proxylist, some async communication and URLs to call (or a pattern to construct those) and you are done circumventing the IP-based filter. Thats no magic, thats basic internet usage with some programming experience.

The flood control/rate limiting I implemented for my app has two parameters:

  1. The number of seconds which must elapse between subsequent times the action is attempted

  2. The number of times the action can be taken within this timeframe before rate limiting starts

Therefore, I can set the flood control on ‘login attempts’ to allow 6 at once, and then 1 per 45 seconds after that, per IP.

If forum search interfaces did this, it would prevent the inevitable ‘please wait 30 seconds’ warning after refining your first search phrase.

Re. google robot warning:

The warning DOES actually pop up inadvertently when using cybersearch + the awesome bar in Firefox 3; it happens something like several times a week. Though that is probably due to bad implementation … every character you type in the URL bar is being transmitted to google and searched. Can get annoying…

I’m horrible for not reading through all the other comments, (if there are more people like me it means you’re not reading my comment)

BUT

if you are limiting password tries (as in the twitter case) you should not then lock the account out completely. (as this can then be used by evil-doers to lock people’s accounts.

Denying your good users their service with even needing a DOS attack…

just thought you didn’t mention this.

I know the point of your article was not to discuss the motivation for the signs. BUT…

I always got the impression that the reason for them was not because they would shoplift - it’s because they detract from the experience for other customers. I’m thinking about one deli near me in particular:

#1 - they travel in packs. They get off for lunch at the same time, and 15 of them will show up at the deli at the same time. So if you are unlucky to show up just after they do, you have a 20 minute wait, and you’re going to leave and go somewhere else to buy your sandwich.

#2 - they’re obnoxious. Criticize me if you want, I see them all the time, and they are loud, rude, and profane. I remember what I was like at that age and I was pretty similar. I don’t fault them, I just don’t want to be around 20 of them at once.

My deli tried the 3 at a time thing (even having a bouncer outside during lunch hour). But since, I’ve noticed that they’ve changed to simply having a separate sandwich line for students. This works great - they don’t prevent other people from getting served promptly, and they are far enough away from the other (higher-paying, I might add) customers so as to be less of nuisance.