Rate Limiting and Velocity Checking

I remember running into it at school last year (last school year, 07) in networking class, some time during first semester.

If you read the linked post, I did that Google scraping in 2004. I’m pretty sure the Google rate limiting CAPTCHA wasn’t on the scene back then.

Okay, so now I’m in a position where I (1) don’t want to seem like I’m harping on this thing about children’s rights when it’s not the point, and (2) don’t want to seem like I’m posting an opinion and then running away when people actually challenge it. Since I don’t feel like turning this blog into my personal soapbox, I’ll say my piece here and welcome anyone to email me if they want to continue the discussion: ijmaxwell AT gee-mail.

I could defend the black analogy, since I don’t think that whether people will continue to be X in the future has anything to do with whether it’s okay to discriminate against X. I mean, if hypothetically black people would cease to be black someday, would that mean it was okay to keep them out of your store after all?

Unfortunately, I’m having trouble coming up with a real-life example of a protected group that people grow out of (though religious affiliation is one that people can enter and exit as they please), so I’m limited to these hypothetical situations here.
But then, even if we’re only forbidden to discriminate against permanent attributes, why would an only 3 black students sign be a problem? Again, this is strictly more permissive than the rule this store actually has.

More importantly, they can’t vote and therefore can’t make something like this illegal in the future when they can vote (either because by then they’ll agree with it or they’ll have forgotten about it)

Yes, it’s true that this seems to happen. It’s kind of unfortunate, really, how short memories are… However, I can vote, I still don’t agree with it, and I still haven’t forgotten about it. My sixteen-year-old self had some dumb ideas, but so did my twenty-two year old self and many of my family members’ current, all-grown-up selves, and I think my dumb sixteen-year-old self should have been respected at least as much as all the other dumb people in the world. I think it’s unfortunate that the world spends so much time talking about special children’s rights while apologizing for the violations of actual normal rights that they deal with on a daily basis.

I was shocked how little comprehensive information was out there on rate limiting and velocity checking for software developers, because they are your first and most important line of defense against a broad spectrum of possible attacks.

I agree with this. I’ve spend some time thinking about this and for non-trivial systems (e.g. distributed front-ends for load balancing, remote authentication services, standard authentication protocols) this is a hard problem and I’ve yet to find a good discussion of the issues much less real advice.

If you read the linked post, I did that Google scraping in 2004. I’m pretty sure the Google rate limiting CAPTCHA wasn’t on the scene back then.

Oh whoops, I just looked at the post date. Thanks!

Off topic: This picture is a good example of bad door design. Why do they have a vertical bar when the want people to pull it? Is that not against people’s natural instincts?

You guys blabbing on about student rights and discrimination are completely missing Jeff’s point. That sign on the convenience store is most likely not there simply because the owner hates students - it is there because the owner has had problems with large groups of students in the store before. The owner is now taking steps to control it. The fact that these types of signs show up all over the place means that students are generally causing a problem.

Instead of worrying about rights, worry about preventing the students from causing enough problems that such actions from store owners are necessary. Once a solution to that problem is found, there will be no need for these signs.

Jeff, I dont think the examples you’ve given are good enough.

  • Most ATMs only allow you to withdraw $300 cash maximum in one day.
    Free email accounts typically limit how many emails can be sent per day.
  • Internet providers limit individual download and upload speeds to ensure they aren’t overselling their bandwidth.
  • There’s a maximum on how many Xbox Live Points you can add to your account per day. (All 500+ Rock Band songs aren’t going to download themselves, after all.)
  • Google limiting too many requests from one IP.

All of the above have NOTHING to do with stealing things – which is ALL that shoplifting is about. All of the above are about safeguarding OTHER fellow users of the system - because these are defined by the scalability of the system.

Shoplifting has NOTHING to do with scalability. Stealing even one ear-bud is a PROBLEM.

See, I can, unwittingly, with no real mal intensions create an interaction that goes beyond what a system allows.

Are you suggesting that an average teenager does not know that they are stealing when they shoplift?

OK, now to the real problem given that shoplifting is a real problem, how do shopowners limit that liability?

  • Encourage buying - loyalty points? you’ll get the 4th pencil for free!
  • Talk to shoplifters - Tell them how it badly impacts your business.
  • You probably opened the store at a wrong place - or started the wrong business!
  • There must be better ways!
1 Like

I think it is important if you’re going to be limiting questions to fail fast. I should be able to type in a whole question and then be told I have to wait 7 more minutes before I can post it. There should be a timer on the page counting down or it shouldn’t let me start to compose in the first place.

I should be able to type in a whole question and then be told I have to wait 7 more minutes before I can post it

Does Google tell you how many more queries you need to issue before you get CAPTCHA-blocked?

Putting out a sign isn’t always a smart idea. Better for the limits to be invisible to 99% of users.

Re. the signs themselves, I’ve seen those for a long time around here, and I’m not sure they say anything about the moral character of teenagers (or even students). They do say something about the moral character of adults. If the sign said Only 3 black people at a time in the store, you would probably respond very differently. Or even Only 3 black students, even though that sign would be strictly more permissive than Only 3 students.

Then again, maybe this really is about schooling (which people can control), rather than age (which they can’t). I wonder if a group of kids in the store could offer the defense that they’re all dropouts.

Does Google tell you how many more queries you need to issue before you get CAPTCHA-blocked?

No, but your limits seem much lower and your audience much more technical. The limits would be invisible unless you had already posted a question in the last 10 minutes. I can imagine clicking post and then having to wait 7 minutes being a big turn off. I’ll be interested to see your statistics on how frequently the limit is being hit.

They do say something about the moral character of adults. If the sign said Only 3 black people at a time in the store, you would probably respond very differently. Or even Only 3 black students, even though that sign would be strictly more permissive than Only 3 students.

Right, and the analog is with an IP Address. How do you tell what kind of person is behind that IP Address, or if it’s even the same person? No way to know!

So when you block for actions by IP (by far the most common technique) you are accepting the inherent ambiguity of judging someone by actions from their IP.

Timely entry today, Jeff. We were about to launch a site with no limits at all. We thought about it at the start, said we’d get to it later 'cuz we had bigger problems. We forgot all about it.

Until now.

I can imagine clicking post and then having to wait 7 minutes being a big turn off.

Trust me, I’ve been working intimately with Stack Overflow for the last six months, and you ABSOLUTELY DO NOT want new users posting more than one question every 10 minutes.

We should have done this months ago, for reasons that have nothing to do with human “look how fast I can type a CAPTCHA!” (sigh) attacks. The attack mitigation is just a nice bonus!

1st page!

seriously tho, Jeff: were you ever discriminated like this as a teenager?

These signs started appearing in the UK about 25 years ago along with a general increase in uncontrollable thugs rampaging about the town when school was out.

Limiting by IP is bound to be an increasingly problematic way of doing things, especially as further oppressive laws pass. Sweden is ready to pass a law allowing monitoring of all private internet traffic. Encryption circular/onion routing services like TOR are bound to increase in popularity, making more and more request originate from the same apparent IP.

Right, and the analog is with an IP Address. How do you tell what kind of person is behind that IP Address, or if it’s even the same person? No way to know!

This seems a lot less discriminatory because an IP is an instance rather than a class, to borrow a software design metaphor. On the other hand, some sites block entire countries because of problems. I block China on a site I run because I have no customers in China and only get attacks, something on the order of a few hundred a day.

Jon, that is incredibly discriminatory. I used to work at a retail place near a factory, and all the workers, largely poor and Latino got off at the same time. Despite the problems, if we did the same thing to them, we would have (rightfully) been looking at a lawsuit. I personally didn’t mind them at all, I hated serving old people because they split meals, complained constantly and were terrible tippers, but nobody would dare ban old people. Young people are the last group it’s acceptable to discriminate against. Offtopic, but it burns me being a former young person and all.

I’ve seen those signs that limit ammount of school kids in school in UK and New Zealand too but this is very boring anyway.