For the last year or so, I’ve been running my own mail server. This provides me with a number of advantages:

• I have no storage limits (other than the physical limits of my hardware).
• I have complete control over the mail server configuration.
• I’m not at the mercy of my ISP’s e-mail server issues and delays if the mail queues get full
• I can use my home system to try out new features and functionality.

It’s been working well. A few newsletters stopped arriving after I installed the Exchange Server intelligent message filter (IMF) and there is no way to whitelist addresses or subjects with the IMF that I’m aware of but, together with the realtime block lists that I use, it generally does a good job of trapping spam with only a few false positives.

I’ve been wondering for a while why there always seemed to be a problem with e-mailing AOL users but it’s such a rare occurrence that I didn’t get too hung up on it. Then I needed to send something over to someone and it bounced back (actually, AOL just refused to accept a connection from my mail server), so I started to look into the problem more closely.

I checked out AOL’s postmaster best practice guidelines and there didn’t seem to be a problem at first. I use a combination of Microsoft Exchange Server 2003 and Entourage 2004 so the mail should be RFC-compliant; I verified the connecting IP address (using AOL’s own tools); I checked that I have a valid reverse DNS entry (again, using AOL’s own tools); my mail server is not operating as an open relay; there were no links (let alone invalid ones) in the bounced messages and I have a static IP address on my business ADSL connection.

Having checked all of AOL’s technical guidelines for whitelisting I applied to join the AOL whitelist using AOL’s spam feedback form. AOL’s postmaster replied with the following:

AOL does not accept email sent directly from dynamic IPs. If you have a static IP, please contact your ISP and have them send us an updated list of dynamic and static IP ranges. If you are on a dynamic IP, please send through your ISP’s mail server.

It seems that the problem is my IP address. AOL’s senders’ FAQ states that:

Customers with residential IP addresses should use the provider’s SMTP servers and should not be sending email directly to another ISP’s SMTP servers.

I do have a static IP address and a business account (i.e. it is non-residential) but that’s not the point – AOL has the audacity to prevent anyone mailing them from an IP address that they have recorded on their database as residential! Admittedly my configuration is not normal but why should AOL dictate how I provide my e-mail service? I do understand that much of the world’s spam will originate from zombie-infected PCs on people’s home networks (i.e. residential) but I have an SPF record implemented in order to verify that e-mail purporting to originate from my mail server really is from my server. In any case, AOL’s reason for blocking me was nothing to do with authenticating my e-mail server but was purely based on their assertion that I have a residential IP address – something that doesn’t seem to bother any other mail hosting provider.

Not really wanting to get into the situation where my ISP says it’s AOL’s problem and AOL says it’s up to my ISP, I decided to work around the problem through reconfiguring Exchange Server:

1. Firstly, I changed the cost on my existing SMTP connector (set to use DNS to route e-mail and using an address space of *) from 1 to 2.
2. Next, I created a new SMTP connector for mail to be forwarded to my ISP’s relay, gave this a lower cost (1) and added aol.com to the address space.

Now e-mail for anyone@aol.com will be sent using my ISPs servers and all other external e-mail will go directly based on the MX records that are specified for the recipient’s domain. Amset IT solutions have a page on their website which explains the configuration in detail.

I needed a quick way of testing the message flow, so I sent a test message via my mail server to my e-mail address at work. Checking the headers on receipt showed that it had gone straight from my server to my employer’s e-mail gateway. Next, I added the domain name for my work e-mail address to the address spaces on the connector for e-mail to be routed via my ISP and repeated the test. Again, checking the headers verified that the message had indeed passed through my ISP’s relay. Finally, I took my employer’s domain name out of the address space on the new connector and verified that e-mail was directly routed once more.

It’s not difficult but it is a further complication in my mail server configuration, just to satisfy the requirements of one (admittedly large) ISP …and just one more reason for me to cringe when I hear that someone is an AOL subscriber.

A couple of months back, I wrote a post about controlling spam using the Microsoft Exchange Intelligent Message Filter. Whilst it has to be said that the IMF has been effective in reducing my spam volumes (with very low false positives – strangely enough my blog posts are the ones it has most trouble with) it’s still not catching all of the unsolicited commercial e-mail (UCE) that I receive, so this week I resorted to another spam control – real time DNS block lists.

Various lists exist with details of known spam relays and the one I’m using is from the Spamhaus project. Actually I’m using two of their lists – the Spamhaus block list (SBL) and the Spamhaus exploits block list (XBL), both of which are free for non-commercial use – I may add other services later.

Setting up the block lists within Microsoft Exchange Server was reasonably straightforward, following advice from Daniel Petri (further information can be found in Microsoft knowledge base article 823866). I then tested the service as recommended at Crynwr Software’s spam blocking resources page. After initial problems testing the service as my mail was being routed via my ISP’s relays (but I could see the conversation when I telnetted to Crynwr’s servers) I switched to DNS-based routing and received a satisfactory response to the e-mail tests – most importantly showing the following text in the SMTP conversation:

550 5.7.1 knownspamserveripaddress has been blocked by Spamhaus
Terminating conversation

So, that’s another tool in my anti-spam arsenal. The UCE levels appear to be tailing off now… hopefully I’m not dropping too much “real e-mail”. One day I hope to be able to say (in the style of John C Dvorak) “I get no spam”.

It may just be a co-incidence, but since I switched my e-mail from my ISP’s servers to my own server a few months back, I’ve been seeing a huge increase in the amount of unsolicited commercial e-mail (UCE) – commonly known as spam – in my mailbox.

At the time of writing, statistics from MessageLabs show a decline in the volumes of spam over the last 12 months (although they still indicate that 58.39% of all e-mails sent were spam). Postini’s statistics suggest that 73% of e-mail is spam.

If you think those statistics are bad, according to Microsoft, Bill Gates receives 4 million spam messages a day, making him probably the most spammed man in the world (it’s no surprise then that he is rumoured to have his own mail server at Microsoft).

Any effective strategy for dealing with UCE (specifically for Exchange Server 2003, but the generic advice is the same for all mail servers) needs to operate a multiple levels within the e-mail transport (these are defined on the Message Delivery Settings under Global Settings in Exchange System Manager but need to be imposed using the properties for each SMTP virtual server):

• Server-level accept/deny lists can be used to always accept, or always deny, messages from certain domains. The trouble with this method of trapping e-mail is that I occasionally receive non-delivery reports (NDRs) for messages that were allegedly sent from markwilson.co.uk but that actually never came near my servers, so without a real-time DNS lookup mechanism to verify the sender’s domain (such as Sender ID), these are of limited use.
• Connection filtering using real-time block lists (RBLs) is the next level of protection, using a DNS query against a RBL provider’s servers, such as the SpamHaus project.
• Sender filtering can be used to drop any messages that claim to come from a particular e-mail address, optionally archiving them.
• Recipient filtering is a method of rejecting certain e-mail addresses (e.g. for people who have left the organisation, or for non-existent addresses). One option is to filter messages for recipients who are not in the directory; however this can leave an organisation open to a directory harvest attack as the server gives different responses for valid and invalid addresses. To avoid such attacks, a “tarpit” (see Microsoft knowledge base article 842851) can be employed, to delay responses to bad addresses by a few seconds, slowing down any directory harvest attacks significantly (it would normally be possible to harvest all four-character address combinations within a few minutes – with a 5 second tarpit delay this is increased to a couple of months – and most addresses have much longer aliases than 4 characters).
• Finally, the intelligent message filter (IMF – previously a separate download but now included with Exchange Server service pack 2) employs a Microsoft-proprietary algorithm (SmartScreen) to scan each message and mark it with a spam confidence level (SCL), which is then used to process the mail accordingly at the gateway or mailbox level.

Each of these tools filters out less obvious types of UCE with increasing levels of cost in terms of server resource. Whilst the junk e-mail filters in Outlook 2003/2007 and Entourage 2004, which are also based on SmartScreen but doesn’t use the SCL mechanism, are pretty good at filtering messages, they are far from perfect (in my experience, Outlook seems to be better at this than Entourage). Activating the IMF on my server has provided an additional level of filtering which has greatly reduced the volume of UCE making it through as far as my mailbox.

The IMF uses 11 SCL ratings, set as an attribute in the message header:

• -1 is used for messages submitted internally with an authenticated connection – eliminating false positives for internal e-mail.
• 0 is used for messages that are marked as not spam.
• 1-9 are used to highlight varying levels of probability that a message is spam (9 being the most likely).

Within Exchange, the SCL value can be used to filter UCE on gateway servers as well as with a lower level SCL used by the information store to move messages to the user’s junk e-mail folder – therefore allowing for the most obvious UCE to be trapped at the gateway (least chance of false positives) and for users to retrieve any messages in the mid-range that are incorrectly marked as junk. The gateway blocking action is also configurable – with options for archival, deletion (without NDR), no action, or rejection.

Archived messages will be saved (by default) to %programfiles%\Exchsrvr\Mailroot\vsi 1\UCEArchive. Each message is archived as an .EML file, which can be viewed with a text editor. To resubmit a message for delivery it can simply be moved to the corresponding %programfiles%\Exchsrvr\Mailroot\vsi 1\Pickup folder. Obviously, viewing individual messages in a text file is time-consuming and the IMF Archive Manager is a great tool for managing IMF-archived messages.

The SCL at which to block messages for a particular organisation will vary according to the profile of e-mail sent to/from the organisation – I have my SCL level for gateway blocking set to 7 with archiving enabled and so far I have only had one false positive – but clearly for organisations receiving more e-mail than I do, this will be a bigger issue! At the store level (set to move messages with an SCL greater than 4) things are not working quite so well but that is to be expected as in the grey area between good and bad mail, some legitimate (good) messages will inevitably get marked with the same SCL as the (bad) UCE. It’s worth noting that marking a sender as safe in Outlook will only override the SCL at the mailbox-level – it has no effect at the gateway.

To assist in judging the SCL levels to use for filtering, it is possible to expose the SCL in Outlook and in Outlook Web Access (OWA). Also useful may be (temporarily) enabling diagnostic logging on the MSExchangeTransport\SMTP Protocol for a server, such that SMTP events are logged. Performance monitor counters from the MSExchange Intelligent Message Filter object can also be used to log the amount of spam filtered or acted upon, the relative SCL levels and overall IMF performance. Based on the performance monitor data, the IMF gateway blocking configuration can be reduced from no action to archive, and then finally (once confident that the levels are correct) to delete, as the appropriate SCL levels are determined.

It’s also possible to mark the SCL on archived messages by creating an new registry key called ContentFilter at HKEY_LOCAL_MACHINE\Software\Microsoft\Exchange\ and a corresponding DWORD value named ArchiveSCL set to 1. A string value named ArchiveDir can also be used to change the archive folder. Both of these settings are detailed in the Microsoft Exchange Server TechCenter along with details for applying the IMF to trusted (authenticated) connections and increasing the size limit for the rule used to process spam at mailbox level (allowing more blocked and safe senders).

Suggested further reading
IMF release notes (Microsoft knowledge base article 867633).
Microsoft Exchange Team Blog.