So You'd Like to Send Some Email (Through Code)

blog
42

@ruchitgarg

Using a google apps account with your domain doesnt mean it will pass. google does not sign your emails. more likely to end up in spam because google will use its own mx.google smtp servers to send emails for yourdomain.com unless ofcourse you send to people with a google address i guess.

I ran a verification by emailing check-auth@verifier.port25.com from my google apps account and got this:

The Port25 Solutions, Inc. team

==========================================================
Summary of Results
==========================================================
SPF check:          neutral
DomainKeys check:   neutral
DKIM check:         neutral
Sender-ID check:    neutral
SpamAssassin check: ham


----
"neutral"
       the authentication method completed without errors, but was unable
       to reach either a positive or a negative result about the message.
43

You harp on about Twitter and you criticize email for being a poor communication modality? Give me a break, Jeff.

44

This is my first post on this blog, and I hate to start off sounding like a hater, but I get the sense that Jeff Atwood doesn’t appreciate email because he doesn’t have what some people call a “real job.” He spends his days sitting around at home with his cats and blogging (to be fair, it appears he did have a “real job” for three years, so I guess he has a little experience). Right now, in Jeff’s world, it makes sense to get rid of email. But we don’t all live in that world. In railing against email, he is definitely not putting himself in the shoes of us “working folks.”

Here’s what’s in my inbox:

  • a message to me from a visual designer (i.e. artist) telling me where the jpgs are (I need to know this so I can put them on the site).

Okay, so obviously the designer is not going to twitter or FB this. So why don’t they IM me? Well, for one thing, once I close the IM window the information is gone, so I have to dig through some IM logs to find it. Here’s another reason: if I’m logged off, she CAN’T send it, so what is she supposed to do, wait around and not move on to other tasks until I return to my desk? She (understandably) wants to fire off a message and forget about it. Finally, here’s another reason (probably the most important one). Follow along with this: I put the wrong assets on the site, I push the site live, the client yells at us or fires us for putting the wrong assets up, and then I blame the designer for not sending me the right info (I personally wouldn’t do this, but many would). If she’s not logging that IM conversation, then it’s my word against hers. But if she has an email that she can point to, then I get in trouble, not her. And since we use Exchange, our emails are archived on a server that only IT has access to, so even if she has a tech-savvy friend outside the company, if she didn’t send it then there’s no way she can fake an email.

Oh, and another thing – the people CCed on this email are carefully selected to be the exact people that need to know this info. Me, two producers, the QA person, technical project manager, another designer. Account managers not CCed – they don’t deal with this stuff, ever. Random other programmers and team members not working on this project are not CCed. If she sent an IM, two producers and the tech project manager and maybe QA are all going to be going to her desk or IMming her and saying “Hey, did you send that image to Jason?” And she’s going to be saying “yes” like 3 or 4 times. And again, more importantly, when the producer forgets to tell the client that the image is live, and then later comes to the visual designer and says “you never told me it was ready”, this designer can point to the email and say “Yes I did, here it is – I CCed you.” Believe me, this happens all the time, even on the less backstabbing accounts.

Anyway I could go on but the TL/DR is that it’s really about accountability. Email is used nowadays in much the same way that good-old on-paper MAIL used to be used. If somebody wants to send you an official communication, they send you an email. It’s no accident that when you register for “e-billing” by your utility company, there’s no option to register for “facebook billing” or IM billing or twitter billing. Email is for official business. I guess Jeff must not have much official business to conduct, which makes sense, since from what I understand he’s mostly about blogging – which is fine, that’s what you do, but don’t make the mistake of assuming everyone’s life is just like yours.

45

Great tutorial but I have found that moving all my email functionality to something like gmail proves to be a simpler solution.

46

The last phrase made it all! :))

47

Sorry Jeff but you got this one wrong.

None of this code belongs in your applications code. DKIM, rDNS, SFP, etc. should all be handled by your SMTP server, not your application. Your application is nothing more than an email client, like Outlook, Thunderbird, etc. Outlook isn’t adding any DKIM info and individual users don’t worry about DNS records, that is all taken care of by the SMTP server.

And I would say SPF (what you call Sender ID) is just as if not more important than DKIM. It is not Microsoft centric at all, almost all mail systems and spam filters check SPF records. DKIM is almost exclusively used/enforced by Yahoo.

48

I agree with TonyB. This is not a responsibility of your app. It just does not belong there. SMTP servers are build to handle such matters so you don’t have to.

Every serious company has an own SMTP server and an admin that manages it. And you have a lot of choices online if you don’t have you own SMTP.
What will you do if you mail gets greylisted, bounced etc… Will you mimic the mail server and try again later? Will you roll out your own mail server? Tha application should just pass the mail with basic info to the SMTP server. It is important however that you have a mean of processing the rejected or undeliverable mails. But this is commonly achieved by adding custom header entries so that the mail server is in a way aware of the sender.

49

Oh and about e-mail. It is by far the most reliable and wide spread medium of communication over the web. And It is not bad at all. I has one very big advantage. Everybody knows what it is and how to use it. Furthermore I can easily have my mails locally or on a server and have them archived.

Good luck archiving all other means of communication. And yes, to me it is very important that I can look back to see what my coworker wanted me to do one year back.

50

So email has it’s flaws, I agree with that. (Calling it ‘broken’ is a little overstated, though).

But advocating Twitter as an alternative? Seriously! The only reason I won’t call twitter ‘broken’ is that it was never whole to begin with.

51

That’s interesting, if Google’s App Engine (which has built-in mail API) provides all this stuff.

52

I have to concur with TonyB. Your application should be a mail client. One of the problems that most apps that send mail directly fail to deal with properly is greylisting. A real SMTP server knows how to handle the SMTP responses, many of the libraries do not.

53

/agree with all the people mentioning this all belongs on the smtp server, not the code level.

While the things you listed in your blog post are all good to do, they really are the minimum. Lots of people have mentioned more things to take into consideration.

One thing i haven’t seen mentioned yet is proper use of mime in email messages. And this is something that you would control at the client level.

Basically if you are going to be sending HTML in your email, you need to provide a plain text version as well, don’t just send the HTML.

If you browse the asp.net forums you will see hundreds of posts where people are suggesting to do something like MailMessage.IsBodyHtml = true;

That is incorrect, the body should be text/plain and the html version should be added as an alternate view.

When you do this, the recievers email client will automatically pick the right version, and the recievers email server will be less likely to reject your message to begin with.

@Meyerweb in regards to how ptr relates to spf records.

spf records are basically a listing of ip addresses that are authorized to send email from a domain. When you see the ptr token in the spf record it is basically a short cut for saying “the ip address indicated in the ptr record is allowed to send email for this domain”

54

@Erick Stubbs:

Including ptr in an SPF record doesn’t seem like a good idea to me.

Simply put, if I have control over my own DNS, I can add a PTR record of 1.2.3.4 to mxa.mail.google.com (or whatever Google’s first SMTP server is).

It’s A records that are generally safe, as they map DNS names to IPs. The A record for mydomain.com is under my control, unless the root server for .com gets tricked or the receiver has had their DNS hacked (and then it’s their problem, not mine).

55

I haven’t tried this to send mail to external parties as I work on internal ERP (Accounting) systems, but primarily you really just have to change how you think about it.

On everything I design email is an “optional”, “Under ideal circumstances” tool. For example we have an internal licensing application that we use to generate licenses for our customers. I could notify the licensing department using email - but what happens if it goes wrong? Instead I write to an actual table and have an application sit over it.

Lets face it - email is and will always probably be a crapshoot. It just wasn’t designed to be terribly reliable and the problem with spam has only compounded that.

56

I just want to say, this blog post inspired me to add an email error-reporting feature to one of my tools my support department uses. Thank you for that.

57

I love the post but the whole sentiment about email being the cockroach of communications is one I hadn’t considered!! Our business ( http://www.yreceipts.com ) depends on automated emails! We send email receipts from retail shops as an alternative to paper and I’d be very interested to hear of other options for doing this if people have any. Thanks.

58

Of the items you listed…you missed the most basic one…and the one most frequently screwed up: include a Date header.

It’s required by RFC2822, and it’s absence is a key indicator of spam.

59

I wrote up something similar with a hands on Ubuntu slant.

http://www.lowtek.ca/roo/2009/earning-trust-for-your-email-server/

Others in the comments have mentioned Yahoo, but that was a particularly problematic step for me. I did succeed - but it took some perseverance. Yahoo hosts Rogers email, meaning that many local people were having spam issues with my server.

60

It’s a good article, except that in the Gmail section you incorrectly conflate “Reverse PTR” and “SPF” - these are two separate things. Gmail and port25.com aren’t actually reporting on whether Reverse PTR checked out.

You’re right to say that SPF is a nice-to-have these days, and DKIM is more important. Lots of other articles on the web say the reverse, primarily because a few years ago, SPF appeared to have taken the lead.

SPF is actually badly thought out - it’s debatable as to whether it is even a good idea to use it. Certainly you should never reject any email that fails an SPF check, because there are just too many normal, legitimate situations in which it may report a failure.

SPF makes the false assumption that if the recipient server isn’t getting the mail delivered directly from the sender’s MTA with no hops in between, then the mail is illegitimate. It’s not hard to see where this fails: if the recipient has set up his mail account to redirect mail to another, that adds an intermediate server that the sender has no control over. If the final receiving server does an SPF check, it sees that the mail has come from an unrecognised server, which doesn’t match the sender’s server. This happens even if the mail has not been altered in any way by that intermediate server. The sender has no way of anticipating or avoiding this fail condition, except either to a) not use SPF at all or b) use SPF with “?all” at the end, ensuring mails that fail the check are not penalised.

On the other hand, DKIM acknowledges that mail may pass through more than one server on its way to the destination, and even goes so far as to check that it hasn’t been tampered with on its way. There’s far fewer situations where DKIM is likely to give a false fail, though there are a couple of edge cases. One is where an intermediate server or proxy (such as a virus scanner) adds a footer to the bottom of mails. Another is where a mailing list server alters some of the headers of the mail before sending the mail out. In the first, you could argue that such tampering by intermediate servers /should/ cause a DKIM failure. In the second, there are other options for mailing lists: they could verify and remove the DKIM header, possibly even re-signing the message with its own DKIM header matching the new Sender address and headers.

61

Another point that could be made is that not all these measures are about getting past spam filters.

Having a reverse PTR that’s not faked definitely will help you, and not just with email. That’s the most basic thing you should get working.

DKIM and SPF can prove that you are the domain you claim to be, but they don’t prove in their entirety that you’re not a spammer. There’s nothing that says a spammer can’t have an IP address and domain name, and due to incompetant or criminal hosting companies they often do. In terms of beating spam filters, passing something like DKIM or SPF is going to be one of a number of factors that a filter may use in their decision, and content-based spam filters are certainly not becoming obsolete any time soon.

It’s a simple point, but worth saying; there isn’t a magic bullet to spam prevention not getting past spam filters.

What DKIM and SPF can do, however, is cut down on a particular type of spam, where a spammer falsely poses as some reputable company.