[top]

Crimes against email

Email is pretty good. Why is everyone trying to kill it?

The other day a new parser I wrote broke on some "=20" junk characters stuck on the end of otherwise readable lines. Of course it was a Microsoft/quoted-printable issue:

X-Mailer: Microsoft Office Outlook 11
Content-Transfer-Encoding: quoted-printable

RFC 1521 says that whitespace is fine in quoted-printable, except when it occurs at the end of a line, when it will be replaced by '=' and a hex code:

This rule is necessary because some MTAs (Message Transport Agents, programs which transport messages from one user to another, or perform a part of such transfers) are known to pad lines of text with SPACEs, and others are known to remove "white space" characters from the end of a line. Therefore, when decoding a Quoted-Printable body, any trailing white space on a line must be deleted, as it will necessarily have been added by intermediate transport agents.

As an encoding quoted-printable is pretty innocuous. It bundles up non-ASCII characters using one or two reserved characters in an attempt to maintain maximum readability. But changing the encoding of a common character like a space depending on its position within a line is lunacy, and based on an act of bad faith. A content encoding scheme should not be trying to correct the occasional MTA problems of 1993. Encoding should simply be encoding. 1993 and its MTAs are long gone. Quoted-printable and its accompanying frass are with us still.

Welcome to 2010. I'm still writing mail parsers.

This spam squeaked through spam assassin:

Date: Sun, 9 Aug 2009 08:03:12 +0200
From: "Citi Account Services" <citicards@info.citibank.com>
X-Mailer: The Bat! (v2.10.01) Business
To: chris@cabstand.com
Subject: your account has been blocked

I'm a pretty open-minded guy, but none of my friends use The Bat! and I don't think Citibank is going to break that streak.

Two good excerpts from its wikipedia entry:

The Bat! is a shareware e-mail client for the Microsoft Windows operating system, developed by RitLabs, a company based in Chişinău, Moldova.

A number of Internet service providers, sites and organizations claim that The Bat! is a spamming tool and, on that basis, block messages containing "The Bat!" in the X-Mailer header. Many spam messages do have the X-Mailer header field set to The Bat!, but this is because it is one of the default settings in the Advanced Mass Sender program, which is frequently used for sending spam mail.

With a straight face this wiki page is saying that the one legitimate software product that originated in Moldova just happened to be victimized by the default settings in a notorious spam tool.

After years of reading headers, obviously Occam's Razor suggests that The Bat is a spam tool, pure and simple. But why do so many spams come through with The Bat in the first place? If you built a spam tool, wouldn't you set the default to be something people in the first world actually used? What if The Bat actually is a real client? Is it a case of the other crabs pulling the escaping crab down, spammers conspiring to smear a real product? Or are spam developers so blinkered that they believe The Bat to be a reasonable default setting?

Update: 4 months on, with more ensuing indignant updates to the wiki page, the last option appears to be the best answer. "The Bat" must be a valid client and a valid setting for spammers. Right? My mistake was assuming first-world email users to be spam targets. Instead there must be a vast pool of emerging-market email users hesitantly running The Bat and being hoodwinked by faked "The Bat" headers as they buy fake watches and diplomas. In my snide opening I recapitulated Pauline Kael's famous Nixon quote. Just because I don't know any user of "The Bat" doesn't mean they don't exist. In fact, to the spammers, it is I and everyone I know who don't exist.

The other day a spam got through that was crafted better than 90% of the legitimate emails I get.

Received: from 5ac77cfc.bb.sky.com (5ac77cfc.bb.sky.com [90.199.124.252])
        by zonker.stanford.edu (8.13.4/8.13.6) with SMTP id n6EJ1bL5074066;
        Tue, 14 Jul 2009 12:01:39 -0700 (PDT)
        (envelope-from edwardcjrk@elestigma.com.ar)
Message-ID: <29794259.6561751670159.JavaMail.vmail@service2.colo.trueswitch.com>
Date: Tue, 14 Jul 2009 12:01:39 -0800  (EDT)
From: "timothy" <edwardcjrk@elestigma.com.ar&rt;
Reply-to: edwardcjrk@elestigma.com.ar
To: fancylad@zonker.stanford.edu
Subject: Re: yo bud
Content-Type: text/plain

The envelope "From" matches the body "From" and "Reply-to". The body "To" matches the envelope address it was delivered to. Small things, but most marketers mess them up. Also it's text only, with line breaks. (Thus immediately better formed than any yahoo email.) Now it gets good:

yo mate, ok I`ll give you my trick but if you give it someone else I`ll fuckin kill you :)
you know in roulette you can bet on blacks or reds. If you bet $1 on black and it goes
black you win $1 but if it goes red you loose your $1.
So I found a way you can win everytime:

bet $1 on black if it goes black you win $1

now again bet $1 on black, if it goes red bet $3 on black, if it goes red again bet $8 on
black, if red again bet $20 on black, red again bet $52 on black (always multiple you
previous lost bet around 2.5), if now is black you win $52 so you have $104 and you bet:

$1 + $3 + $8 + $20 + $52 = $84 So you just won $20 :)

now when you won you start with $1 on blacks again etc etc.  its always bound to go black
eventually (it`s 50/50) so that way you eventually always win. But there`s a catch. If you
start winning too much (like $1000 a day) casino will finally notice something and can ban
you. I was banned once on royal casino. So don`t be too greedy and don`t win more then $200
a day and you can do it for years. I think bigger casinos know that trick so I play for
real
money on smaller ones, right now I play on lucky june casino: rabbit-jackpot.net for more
then 3 months, I win $50-$200 a day and my account still works. You`ll find roulette there
when you log in go to
"specialty games" - "american roulette". And don`t you dare talling about it anyone else,
if too many people knows about it casinos will finally found a way to block that trick. If
you have any questions just drop me a line here or on skype.

c ya

That's a lot of cogent message before the URL is buried in the final paragraph. I actually spent a minute or two thinking this was a real email I'd received by mistake.

----- Original Message -----
From: "nikki_howell70" <fancylad@chappie.stanford.edu>
To: <edwardcjrk@elestigma.com.ar>
Sent: Thursday, July 09, 2009 3:06 PM
Subject: Please send me the system

> Hi timothy.
>
> Please tell me when you will send me your roulette trick?
> You promised you`ll send it few weeks ago :(
>
> Thanks in advance.
>

To top it off there's a fake "original message" appended at the bottom, with the "From" matching my email and a plausible date and time. (The subjects don't match, though-- one uncharacteristically sloppy piece.)

The betting system described is a modified Martingale scheme that will make you go broke with 100% certainty over time. Sorry bud.

I think it's great that the LAPD even has periodic email alerts, but whatever system they're using is clunky enough so that occasionally this happens:

Subject: [Enter Subject Here]
Date: Sat, 16 May 2009 13:45:39 -0700
From: E-Police Newsletter <"Northeast Division"@lapdonline.org>
X-Mailer: PHPMailer [version 1.73]
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable
Lines: 168

(What follows is 93 lines of css, some canned footers and no message.)

Why does an institution like the LAPD need to send markup in its emails at all?

When I was working at an online clothing store we spent a lot of time making sure our outbound flyer and receipt emails had impeccable html components, with logos, product shots and special fonts built in. Corporations have to put on a pretty face and use complicated markup to increase credibility and show off their attention to detail, since consumers have so many choices available. For an online-only corporation it's especially acute, because you only have a limited exposure to the customer, and there are fewer ways to signal your legitimacy and polish.

A public entity like the LAPD is the very opposite. They are a literal polished presence in their constituents' lives already with an ironclad legitimacy. They already show their attention to detail in thousands of ways, like the act of sending information out to their constituents. There is no possible consumer alternative. There is no need to waste resources on costly, noise-increasing peacocking.

Email incubated for 25 years without a consumer face at all. During that time its protocols and the internet matured together, so that now email is a stable, ubiquitous way to communicate in written text. Public entities should be reaping the harvest of this accumulated engineering to send zero-cost simple, readable text emails.

When they do send out a valid email, the LAPD doesn't use the markup for anything that demands markup. It's just a way to put boldface headings and bloated css in an email. It's just something somebody sold them, something somebody thought they had to do, and now police budget and efforts are being wasted on some terrible piece of software that does a poor job at sending text to a blind alias-- a job that's been well-supported by conventional email since at least the mid-1970s.

Update: In July the LA Times discovered that the LAPD's online crime map omitted 40% of crimes. The chief "blamed the problems on the private vendors hired to develop the site."

Taxing email to end spam is a terrible idea sent from 1996 to murder us all, but Prospect Magazine and The NY Times are already victims so maybe it's succeeding.

From a practical point of view, such a tax is feasible. Whether you're using a browser or a client-based email system, every email sent must have both a sender address and a recipient address--each in the form "someone@somewhere." This makes all emails easily identifiable by ISPs, through which most private internet traffic is routed. As they already impose a monthly charge on users, it would be simple for ISPs to pay the tax and pass it on in the monthly bill to their users.

Most spam is sent with a forged "From" address that has nothing to do with either the content of the message or the unwitting machine that was used to send it.

The days of spam being an "email" in any meaningful sense are long over. Spam is just a delivery for web ads. I randomly checked 50 messages in my raw spam folder and only 3 were not just links. One was probably a test as it had nothing in it. One had only a phone number to call. The one that was trying to invite an email from the victim had the reply address buried in the message. The "From" was a dummy.

This is the top of a typical spam:

Received: from bd7b51bb.virtua.com.br (bd7b51bb.virtua.com.br [189.123.81.187] 
        by bab.cabstand.com (8.13.8/8.13.8) with ESMTP id n2KL3OVN034997
        for <postmaster@cabstand.com&rt;; Fri, 20 Mar 2009 14:03:25 -0700 (PDT)
        (envelope-from sulphursmt424@filersatv.com)
Message-ID: <000d01c9a9e1$d125f030$6400a8c0@sulphursmt424>
From: "Bud Walter" <sulphursmt424@filersatv.com>
To: <postmaster@cabstand.com>

We could try to tax or restrict the random IP address in Brazil that was hijacked to send it, but breaking the protocols like that usually backfires. What's certain is that Bud Walter had nothing to do with this. Filersatv.com is just a valid domain name picked at random.

Protecting yourself from spam without breaking email is easier than ever. Step done: Install a decent filter (like SpamAssassin.) If you still hate the few false negatives, run everything through a whitelist of your contacts. Procmail plus a text file works fine. Every so often check the "unknowns" folder for false negatives mixed in with people that should be on your whitelist.

If you want strangers to be able to send you email, you have to expect that a few strangers might send you email.

"'We started out by saying to ourselves, 'What might e-mail look like if it had been invented today?'" said Lars Rasmussen, who worked on Wave in Australia with his brother and just three other Google employees.

If Google had invented (or bought) email today, I would expect it to be loaded down with brightly colored crap and quietly dismissed about three months later, along with OpenSocial, Orkut, Dodgeball, Froogle and whatever else.

Email has survived for 40 years by having nothing in common with a Google press release.

If you want to "combine elements of e-mail, instant messaging, wikis and photo sharing in an effort to make online communication more dynamic" have a great time, just keep it away from email.

(Quotes from the LA Times story)

It's pretty standard for companies to mangle the text/plain (i.e. "default") portion of their outbound emails, but Citibank's credit card statements are special:

 Dear CHRISTOPHER PEIFFER:

 Your Citibank statement is now available at <a
 href="http://www.citicards.com
 ">www.citicards.com</a>.  This notification is part of
 the All-Electronic Program you enrolled in to receive your
 statements online only instead of in the mail.

 To ensure that you receive monthly statement notifications
 via e-mail, please keep your contact information
 current.  If you're planning to change your e-mail address,
 sign-on to <a
 href="http://www.citicards.com
 ">www.citicards.com</a>, go to the Manage My Account
 menu, and choose Update Personal Profile to edit your Email
 Profile.  To change your postal address, just use the same
 menu and choose Address & Phone Change.

 If you use your work e-mail address, keep in mind some
 employers may block receipt of employees' personal e-mail.
 Please update your e-mail address at <a
 href="http://www.citicards.com
 ">www.citicards.com</a> -see instructions above.

 We hope you continue to enjoy the many benefits of the
 All-Electronic Program.

 Sincerely,
 S. Larson
 Customer Service

They word-wrapped at 60 columns and put markup into plain text (but only anchor markup.) How did they even get the quotes at the end of the URLs to separate and wrap to a new line? At least they classed it up with the inexplicable one-space left margin.

All fixable in two seconds if they had even once bothered to check the basic case of text emails in a text client.

First lines of an email I got from a friend the other day:

X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10

Then later that day from someone else at his company:

X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10
X-Barracuda-RBL-Trusted-Forwarder: 10.10.0.10

Up from seven barracudas to nine, now we're making progress.

If you run your company with a security policy that forces you to spam-filter outbound email, you'd think you'd want to suppress that information, or at least suppress duplicate headers.

If you have yahoo messenger running, you can elect to receive pop-up alerts telling you that a new message has come for you on yahoo mail. The alert rolls up in the bottom right of the screen: a little envelope icon and the subject of the message, which is clickable.

In every webmail system ever made, clicking on the subject of an email opens that message. One would expect the clickable subject in this alert to be no different. But instead it takes you to a "welcome" page, which is mostly devoted to a snack-cracker advertisement and a few wire service articles. There is a link for "Inbox." From the inbox you have to hunt for your new message, which you open by clicking on its subject.

This alert violates the user's expectation about mail software and forces them through two extra screens that are clearly just there to show ads.

If I'm logged in to yahoo messenger and receiving yahoo mails, I don't need a "welcome" to the yahoo experience. There's a pretty good chance I know that yahoo has news and ads.

This is an email I got from ticketmaster yesterday after ordering tickets from a third party site:

Date: Sun, 13 Jul 2008 11:05:17 -0700
From: "chris@cabstand.com" 
To: chris@cabstand.com
Subject: ticketFast from [blah]
User-Agent: Mutt/1.4i

All an email really needs is a (meaningful) From, To and some text. Ticketmaster decided that they're above having to identify where this email is actually from and put my own email address in both the From and To. It's a pretty cheap way to annoy your customers and try to foil basic whitelist schemes. (Got me on both counts.)

Look at the User-Agent header. It's set to Mutt/1.4i. Not only is Ticketmaster faking the From, but they've chosen to fake a "User-Agent" on an obviously program-generated email and they've chosen mutt, of all possible clients. I use mutt, but how many other people in the US do? 15?As far as I know there's no mutt backend library for generating bulk emails. Is ticketmaster's whitelist-worming software that good that after setting the From to my email address, they set the User-Agent to the probable mailer I use? Paranoid men should not read headers too closely.

The email's opening lines are perversely reassuring:

[-- Attachment #1 --]
[-- Type: text/plain, Encoding: 7bit, Size: 1.8K --]

****************************************************************************************************
                              YOUR TICKETS ARE ATTACHED TO THIS EMAIL!
****************************************************************************************************

99 character lines in plain text? I guess ticketmaster is not tracking my thoughts too well after all.

I just bought something from Camper's online store. They sent me a confirmation email.

Date: Wed, 9 Jul 2008 08:48:20 -0700 (PDT)
To: chris@cabstand.com
From: "noreply@camper.com" 
X-Mailer: SAP R/3 Internet Mail Gateway 4.6D14
Subject: Camper e-shop:  Confirmation Camper

[-- Attachment #1: Camper e-shop:  Confirmation Camper --]
[-- Type: text/PLAIN, Encoding: 7bit, Size: 0K --]


[-- Attachment #2: Camper e-shop:  Confirmation Camper --]
[-- Type: application/PDF, Encoding: base64, Size: 56K --]

[-- application/PDF is unsupported (use 'v' to view this part) --]

There's no text at all, not even html. All the content is in the form of a neatly rendered custom pdf. It took them 56k and an attachment to send 1455 bytes of message.

The pdf itself is all in italicized courier typeface, so it arguably looks worse than any client would have rendered it.

This is doubtless the natural progression of the European trend of terrible, all-flash shopping websites. I guess I should be thankful they didn't send me a flash object with background music and animated cars that sprout all over the screen when you try to do anything, because that's what's currently on their website.

Here is the receipt I just received after using paypal to make a payment:

Date: Tue, 08 Jul 2008 23:59:39 -0700
Subject: Receipt for Your Payment to [blah]
X-MaxCode-Template: email-receipt-xclick-payment
To: Christopher Peiffer 
From: "service@paypal.com" 
X-Email-Type-Id: PP120
X-XPT-XSL-Name: email_pimp/default/en_US/xclick/ReceiptXClickPayment.xsl

Dear Christopher Peiffer,

Leave aside how unprofessional and messy it is to send three separate ad-hoc mail headers (the ones prefixed with "X-") all the way to the customer. If you're going to do that, at least spare me from the wit of whatever developer thought it was hilariously awesome to name some script "email_pimp" and stick it right in my face when all I did was give your business money.

This is what you get when you send a picture from a T-Mobile Blackberry with gmail enabled:

[-- Attachment #1 --]
[-- Type: text/plain, Encoding: 7bit, Size: 0.1K --]
Content-Type: text/plain


Sent via BlackBerry from T-Mobile
[-- Attachment #2: =?Windows-1252?B?SU1HMDAwMDguanBn?= --]
[-- Type: image/jpeg, Encoding: base64, Size: 640K --]

That horrible looking filename is something encoded in the Windows-1252 character set. Why do something like that on a filename, which should be as simple as possible? Why risk so much, especially with a partisan, poorly-supported character set like Windows-1252?

Predictably it breaks a MIME parser that's worked for at least ten years.

For years Yahoo mail used to send out a decently formatted mail. There was an even an option in the preferences where you could set how many characters you wanted your lines to wrap at. The default was something insane like 30, but if you set it to 72 you could actually get a normal wrapped line.

Now, not only is that option gone from the preferences, but Yahoo mail does not wrap at all. No lines breaks are inserted, so everything comes through as one big awkward line of text. Even if you explicitly say "no rich text" and the mail goes out with no html attachment, the text portion is unwrapped.

For 40 years, the only real requirement for an email has been To, From, and a few lines of text with line breaks. Yahoo had that feature and decided to remove it.