We had been testing extensively on our test environment a series of major changes to how anti-spam filtering is handled on our server. We are now confident that series of updates is ready for the production environment. This email will be long. But we seek that you take the time to read it in full and in detail, seeking to clarify with us when in doubt. This will be the first major update since we started implementing server-wide anti-virus scanning last year for all of our clients, and we are confident that it can dramatically reduce spam volumes for you and your clients. As this will impact in some way how emails are now filtered, you are encouraged to contact your clients and explain to them the series of improvements as well.
We are rolling out this to all servers over the whole of next week. No downtime is expected, and no reboots of the server is required to bring about the new functionality.
1. Server wide Anti-spam Filtering
This update will add server-wide anti-spam filtering with a combination of technologies, most of which will be explained further in section 2. With this introduction, all emails will be filtered with Spamassassin on the MTA level upon receipt. As this is now enabled for all emails and thus all domains, all users will have their domain level Spamassassin disabled to prevent wastage of resources from double-scanning with little improvements to efficacy.
2. Introduction of new key anti-spam technologies
Two new highly accurate anti-spam technologies is introduced, together with a series of Spamassassin rules. They are Razor, DCC, as well as a series of Spamassassin rulesets from SARE.
Razor - Vipul's Razor is a distributed, collaborative, spam detection and filtering network. Through user contribution, Razor establishes a distributed and constantly updating catalogue of spam in propagation that is consulted by email clients to filter out known spam. Detection is done with statistical and randomized signatures that efficiently spot mutating spam content. User input is validated through reputation assignments based on consensus on report and revoke assertions which in turn is used for computing confidence values associated with individual signatures. This method is highly effective and has a high accuracy rate.
DCC - As of mid-2004, the DCC or Distributed Checksum Clearinghouse is a system of millions of users, tens of thousands of clients and more than 250 servers collecting and counting checksums related to more than 150 million mail messages on week days. The counts can be used by SMTP servers and mail user agents to detect and reject or filter spam or unsolicited bulk mail. DCC servers exchange or "flood" common checksums. The checksums include values that are constant across common variations in bulk messages, including "personalizations." This method is highly effective and has a high accuracy rate.
3. Per account customization
We had configured it to allow every user to disable any of the 4 types of filtering right from their Cpanel: Filetypes, Antivirus, Spamassassin, and RBL. This brings an unpredented level of fine-grain control for each and every user. They can access this by going to their Cpanel and clicking on "Server Wide Spam and Virus Protection". By default, it is enabled for all domains and accounts.
We can also manually blacklist and whitelist both incoming and outgoing by domains, for anti-virus, spamassassin, filetype filtering, and do IP blacklisting of errant mailservers. This improves substantially on the current settings.
4. Compliance with Standards
In accordance with RFC standards, the mail server now verifies each incoming email based on their reply to the HELO command. This includes forging of hostnames, empty HELO replies and using our hostname in the HELO reply. Spammers tend to use forged hostname in order to throw off attempts to catch them. This would now result in the emails being rejected at the server level.
5. Better logging
We have now added the logging of email subjects as well as the folder of the script executing the mail server. As such, this would improve dratically the level of detail logged as well as provide faster debugging and identification of all spammers.
6. What hasn't changed.
We will continue to use the same RBLs and anti-virus filtering as before. This has not changed althought it is now configurable on a per-user basis. Previous protections such as the blocking of dictionary attacks still remains. In fact, this new changes add on to the previous changes or improve on previous functionality without the removal of any previous features.
Thank you. Let us know if you have any questions.