Posted on 2008-05-22 19:00:00, modified on 2008-06-03 19:00:00
Tags: SMTP, Spam, Email
BATV stands for Bounce Address Tag Validation and is a method to prevent backscatter from spam runs. It works by modifying (danger! technical content ahead!) the Envelope From address in an SMTP session from [email protected] to [email protected]. If this email is undeliverable, it will be send back to [email protected] instead of to [email protected] and your mail host knows that this is a valid undeliverable message.
So what has Postgrey to do with this? Postgrey is a greylisting server. It is (danger! technical content ahead!) forcing email deliveries from addresses and hosts which are not yet known to be retried later. Why? Earlier this century, emails sent by viruses and spam-hosts weren't smart enough to understand this and the email with the malicious payload was not accepted by your mailhost.
Yes, but what has greylisting to do with it? Greylisting delays every email from / email to / sending host combination it hasn't seen before. So if BATV changes the email from address every day, the first email from that user will be delayed every day. Every day! So Postgrey needs to be taught what the real email address is. Luckely BATV keeps this information in the from address: prvs=tag-value=[email protected]. Small patch, and it works.
And now the tricky stuff: Not every read the documentation properly, and the two following formats have been seen:
Brilliant! They swapped it around! So my four line patch becomes an eight line patch.[email protected] [email protected]
Anyway, the patch is available and submitted to the Postgrey author.
Note: Please note that I've made a little change to the patch to pick the second field (as the standard suggests) instead of the wrong standard. Not that it ever should come to there, but it's a "just in case" thing.
Posted on 2007-11-29 09:00:00
Tags: Email, SMTP
About two years ago we re-implemented greylisting, sender verification and RBL blacklisting on our MTAs. As a result, we had a very steep drop in unsolicited email, but also very interesting mailbounces from applications which didn't set a proper envelope-from address. It taught us that we needed two things: a whitelist service and a blocked-email reporting service.
The blocked-email reporting service was easy, just parse the Postfix log files once a day and check all the NOQUEUE entries. Then match the temporary failed deliveries against the successful deliveries and the leftover is a failed delivery.
The whitelist service is not too difficult, thanks to the check_policy_service feature which gives the black/whitelist daemon the sending MTA, and the envelope-from and envelope-to email addresses. A lookup against the black/whitelist database and all goes fine.
It is not really one lookup, it is a set of seven lookups:
Recently we were getting strange errors in the blocked-email reporting service, talking about Internal server configuration errors which were happening on our primary MTA. Not that the email got blocked by it, it just was delivered to our backup MTA. But still it was something which didn't make sense. We found out very soon that it was caused by a slow black/whitelist daemon. Not really a slow black/whitelist daemon, more a slow database. And not really a slow database, just a busy database.
A busy database? All it is doing is euhmmm... let's see... 70 queries per second?!?!?! A quick look at the Cacti graphs showed me that over the last three weeks mail servers went from an average of two messages per second to an average of six messages per second (message delivery attempts that is), and since every message has seven database queries it kind of shot through the roof and the black/whitelisting daemon we had couldn't handle it fast enough.
Time for a redesign. We have about 1500 entries in the black/whitelist database, and a hit-rate of about zero (most of the rejecting of email is done via sender verification and via DNS-RBLs). With such a low hit-rate, it shouldn't be a bad thing to locally cache a copy of it and refresh it every 15 minutes. That will save about 5400 database queries during that time period.
Three hours later, some rewriting of the black/whitelist daemon and the database server is happy again with the two queries per 15 minutes now. And the mail? It kept flowing...
Posted on 2006-04-20 16:06:44, modified on 2006-04-20 16:15:25
Tags: Networking, Postmaster, SMTP, Email
Todays idiot mail server of the week award goes to webcontrolcenter.com!
All seems to go fine, until you try to send email as postmaster:
[~] edwin@k7>telnet mail22.webcontrolcenter.com 25 Trying 216.119.106.23... Connected to mail22.webcontrolcenter.com. Escape character is '^]'. 220 mail22.webcontrolcenter.com ehlo edwin.adsl.barnet.com.au 250-MAIL22 Hello [203.222.131.252] 250-SIZE 31457280 250-AUTH LOGIN CRAM-MD5 250 OK mail from:<> 250 OK <> Sender ok rset 250 OK mail from:<[email protected]> 250 OK <[email protected]> Sender ok rset 250 OK mail from:<[email protected]> 550 Sender is not allowed. Connection closed by foreign host.
And they rudely close the TCP session for you too!
Posted on 2006-03-29 13:10:58, modified on 2006-03-29 13:13:05
Tags: Networking, SMTP, Email
I've just enabled sender verification on our mailservers.
What does that mean you might wonder, and how is that different from grey-listing.
With sender verification, postfix checks if the address in the MAIL FROM command gets accepted by the MX servers of that address. It tries to do it realtime, but if it takes too long (>6 seconds) it will temporary fail the SMTP session and waits until the sending MTA retries.
Grey-listing on the other just temporary fails the first delivery attempt and waits until the sending MTA retries.
Keep in mind that two different things are checked here:
One day I'll be brave enough to do also SPF checking (Allowed delivery of email for that domain by that MTA) and everything is as open as it will be, or as closed as it will be.
Posted on 2005-05-25 10:28:14, modified on 2006-01-09 16:29:23
Tags: Networking, Spam, SMTP, Email
If you think this is bad: (mavetju.org isn't served by 200.121.183.223)
Received: from mavetju.org ([200.121.183.223])
by imta02sl.mx.bigpond.com with ESMTP
id <[email protected]>;
Tue, 24 May 2005 23:20:49 +0000
message-id: <[email protected]>
Wait until you see this:
Return-Path: [email protected] Received: from APlessis-Bouchard-152-1-59-216.w82-121.abo.wanadoo.fr (APlessis-Bouchard-152-1-59-216.w82-121.abo.wanadoo.fr [82.121.170.216]) by mx1.midcoast.com.au (8.13.1/8.13.1) with SMTP id j4N6sWvS003077 for <[email protected]>; Mon, 23 May 2005 16:54:47 +1000 Received: from mail3.barnet.com.au by APlessis-Bouchard-152-1-59-216.w82-121.abo.wanadoo.fr (8.9.3/8.9.3) with ESMTP id PCEIP7onXFNw for <[email protected]>; Mon, 23 May 2005 14:41:04 -0700 Received: from (root@localhost) by mail3.barnet.com.au (8.12.8/8.12.8/Submit) id 1GaCy2wErDj5Ks for <[email protected]>; Mon, 23 May 2005 14:41:04 -0700 Date: Mon, 23 May 2005 14:41:04 -0700 From: Edwin Groothuis <[email protected]> Reply-To: Edwin Groothuis <[email protected]> Message-ID: <[email protected]>
What do the headers says?
Why is this worsening? It is because the email actually looks, for the untrained eye and a lot of automatic header-parser programs, like it was coming from mail3.barnet.com.au:
In the first example, everybody who knows a little bit about SMTP headers first checks if 200.121.183.223 is somewhat related to 16wardell.com.au.
In the second example, you have two more lines to parse. I admit that the syntax of the second-last line isn't proper (it is missing the hostname/ip address between brackets in the from field), but for the rest looks pretty good.
What is still wrong with it?
Could this have been prevented if mx1.midcoast.com.au would have done SPF checks? Yes. The SPF tests would have failed on every received line with a hostname.
Posted on 2004-10-25 23:06:15, modified on 2006-01-09 16:29:22
Tags: Networking, SMTP, Email
Some mailers are dumb (see previous article), but some people who configure them are dumb too! After a mass mailing, I got the following errors:
If it is a temporarily error, give me a 4xx error, not a 5xx error!<[email protected]>: host mailhub.totel.ru[213.242.36.74] said: 550 sorry, user is temporarily unavailable (#5.1.1 - chkusr) (in reply to RCPT TO command).
Yes. But I'm in Australia. Wrong hemisphere mate!<[email protected]>: host chaos.fxp.org[209.251.159.150] said: 554 <mailout2.barnet.com.au[218.185.88.16]>: Client host rejected: No China Spam (in reply to RCPT TO command)
Can you please check my SPF records? Thanks!<[email protected]> host mailmaster.MysticWALL.COM[210.249.95.190] said: 550 <mailout2.barnet.com.au[218.185.88.16]>: Client host rejected: We don't accept mail from spammers (in reply to RCPT TO command)
At least they checked the content, but it would be nice if they could get me the score from the spam-scanner so I could see if I was maxed out or that it was just with my toes over the line.<[email protected]>: host mx.argosnet.com[213.30.158.180] said: 554 5.7.1 The message was categorized as SPAM. (in reply to end of DATA command)
Posted on 2004-10-25 16:39:16, modified on 2006-01-09 16:29:22
Tags: Computers, Networking, Email, Rant, SMTP
Recently we implemented so called greylisting on our mail servers. This means that all incoming SMTP sessions with the following new combinations of sending mail server IP address, sender email address and recipient email address gets temporary rejected (SMTP error code 450, meaning: try again later).
From RFC2821: 4yz: Transient Negative Completion reply
The command was not accepted, and the requested action did not occur. However, the error condition is temporary and the action may be requested again. The sender should return to the beginning of the command sequence (if any). It is difficult to assign a meaning to "transient" when two different sites (receiver- and sender-SMTP agents) must agree on the interpretation. Each reply in this category might have a different time value, but the SMTP client is encouraged to try again. A rule of thumb to determine whether a reply fits into the 4yz or the 5yz category (see below) is that replies are 4yz if they can be successful if repeated without any change in command form or in properties of the sender or receiver (that is, the command is repeated identically and the receiver does not put up a new implementation.)
This saves us about 99% of the incoming spam and viruses and is a relief for our mailboxes and the email virusscanners.
Now the bad news, there are some very brain-dead SMTP servers on the internet...
And guess what? They all run on MS Windows. Who had expected that?
Here is the list of them:
MailMax from SmartMax Software Inc.. When receiving an 450, they bounce the mail back to the sender. And this is the error message they are getting:
The 'To' address [email protected] was rejected by the remote server.Which mailserver was it talking to? What was the full error message? What was the error code? And why do you say it's a permanent error while it was a transient error? BRAIN DEAD!This is permanent error, and the message will not be retried any further.
And they claim on their website:
IMPORTANT: Codes that start with 4 and 5 are the ones that tell you that your message won't be sent until you find and fix the problem.You! Yes you! You should fix the problem, and not the other side, or the MailMax mail server.
Update: With their latest version *version 5.5), at least the error messages are better:
Sorry, your message from [email protected]> to <[email protected]> could not be delivered. The specific error is: 450 <[email protected]>: Recipient address rejected: BarNet Engineered Transit Delay -- 39 seconds This is permanent error, and the message will not be retried any further.
Still it's a 'permanent' error, but at least it's visible for the person the email was returned to that they interpreted it wrongly.
Another Update
Sorry, your message from <[email protected]> to <[email protected]> could not be delivered. The specific error is: 450 <[email protected]>: Recipient address rejected: BarNet Engineered Transit Delay -- 45 seconds 2 attempts will be made to re-send your e-mail. Each attempt will be 3 hours apart.
That's much better! Everybody upgrade to the latest version!
CapeSoft Mailer by CapeSoft. It also immediately bounces the email without retrying. BRAINDEAD!
Bigpond.com by Telstra
This is all the attempt of Telstra (Australian ISP) to handle SMTP sessions with a 450 status:
Oct 29 16:17:56 mag postfix/smtpd[10870]: NOQUEUE: reject: RCPT from gizmo06ps.bigpond.com[144.140.71.41]: 450 <[email protected]>: Recipient address rejected: BarNet Engineered Transit Delay -- 45 seconds; from=<[email protected]>, to=<[email protected]> proto=SMTP helo=<gizmo06ps.bigpond.com>
That's all: one attempt. And the sender doesn't get an "Your email has failed" message. BRAINDEAD!
InterMail from Openwave Systems Inc..
It doesn't retry at all. (experienced with ozemail.com.au)