REVERSE SPAM FILTERING
Winning Without Fighting

	MetaNote 1   This is a work in progress. MetaNote 2   The copyright notice above refers to my writing. I would be delighted if people adopt this strategy or terminology (“greenlist” etc.), and help refine it. MetaNote 3   This page was mentioned in the 22 August 2002 issue of The Naked PC Newsletter, which has over 100,000 subscribers! Keywords (for bots)    spam fighting, junk email, UCE, UBE, filter, filtering, procmail, negative filter, reverse filter, inverse filter, The Art of War

 

On This Page

Winning-Without-Fighting Strategy   Fighting and Not Winning  Email Deflexion Flowchart  Keys to Implementation My Implementation Dissecting the Flowchart   See Also

 

Winning-Without-Fighting Strategy

Therefore, one hundred victories in one        hundred battles is not the most skillful. Subduing the other’s military without battle        is the most skillful.

The Art of War by Sun Tzu, Chapter 3

 

After years of thinking about, writing about, and filtering messages, I've decided that the best strategy for me is to not filter spam, but instead to filter non-spam and let the dregs, which are often spam, fall through my filters and land in catchall mailboxes. I then periodically open each catchall box, sort it by spam score, and visually scan it looking for non-spam. Since the non spam messages, if there are any, bubble to the top of the sorted-by-spam-score box, I need to look carefully at only the top of these catchall mailboxes. When I find a non-spam message in a catchall box, I “bounce forward” it to one of my magnet-updater email addresses so that my non-spam filters will catch this type of message in the future.

 

Fighting and Not Winning

For an example of a strategy that seems antithetical to the wisdom of The Art of War, see Paul Graham's article Filters that Fight Back. For a discussion about this article and some other strategies, see the 10-August-2003 thread Paul Graham: Filters that Fight Back at slashdot.org. An especially insightful comment is Re: And now by the mysterious Zeinfeld, a former experimental physicist.

 

Email Deflexion Flowchart

Below is a flowchart that illustrates my strategy. Note that this flowchart leaves out a lot of the SMTP level filtering that is essential nowadays. For example, nowadays it is common for an SMTP server to:

• reject messages to unknown recipients;
• check if the combination of the IP address, sender, and recipient is familiar and, if it's not, embargo or temporarily reject the message. This is called greylisting;
• use other SMTP level anti-spam techniques, some of which are described in Stopping e-mail abuse at Wikipedia.

My focus below is on aspects of the mail flow that are needed for my reverse spam filtering strategy. For example, recipient splitting (exploding) is essential so that each user can use his personal bluelist and personal greenlist to pre-filter his mail before expensive or error-prone direct spam filtering -- such as content-based filtering, Bayesian filtering, etc. -- is done.

 

||   
||   
\/   

||   
\/   

=====>
yes  

|| no
\/   

=====>
yes  

|| no
\/   

||   
\/   

||   
\/   

||   
||   
\/   

||   
\/   

=====>
yes  

|| no
\/   

||   
||   
\/   

=====>
yes  

||   
||   
||   
||   
||   
|| no
||   
\/   

||   
\/   

=====>
yes  

||   
|| no
||   
\/   

=====>
yes  

|| no
\/   

=====>
yes  

|| no
\/   

||   
\/   

=====>
yes  

||   
||   
||   
\/   

=========================>
no  

 

---
¹ The word “box” can be interpreted to mean a traditional mailbox, a bucket, or a label. Labels, which are also called keywords, can be used to construct virtual mailboxes (aka virtual folders or smart folders).

² When a message is moved to the green box, its sender will be automatically greenlisted (for example, by a script that is regularly run against the green box).

Tip:  Do not put your own email addresses in your greenlist because spammers often forge the From header with your address as a way to sneak into your green box! Another reason to keep your addresses out of your greenlist is that you can see how spammy your messages are (because they will get analyzed and tagged with a spam score).

³ LOAF is a GPL'd distributed-social-network filter that is a private way to greenlist your correspondents and limelist the correspondents of your correspondents (aka your “2nd degree correspondents”).

SUPPORT INFINITE INK!

all plans include shell access, pine, & procmail Current promotion: 77¢/mo for 1st year

 

 

Keys to Implementation

The keys to implementing this strategy are

• good message filtering software (at all layers: MTA, MDA, and in the mail client)
  
  
• good tool for analyzing and assigning a spam score/probability to the fall-through (yellow and red) messages
  
  
• good mail client that makes it easy to
  • process multiple incoming mailboxes
  • sort (order) a mailbox by spam score (for example the yellow or red box)
  • “bounce forward” (aka resend or redirect) a message to magnet-updater addresses
    
    
• easy way to train Bayes filters; one possibility is Jem Berkes' webfilt - Flexible web interface to server-side spam filters
  
  
• good program for checking multiple incoming mailboxes for new messages (I'm collecting a list here)
  
  
• automatic way to update bluelist and greenlist filter recipes (magnets)
  
  Tip:  Do not put your own email addresses in your greenlist because spammers often forge the From header with your address as a way to sneak into your green box!
  
  
• Message transfer agent (MTA) that munges your incoming messages and puts the original envelope addresses in headers such as X-Original-From, X-Original-To, or X-Rcpt-To. This header is used to determine which messages should be deflected to the magnet-updater program. For more about the original envelope recipient address and problems with Bcc'd messages, see this section of my Procmail Quick Start. If you know of a mail-hosting provider that has set up their MTA to do this, please let me know and I'll link to them.
  
  
• good mailbox-naming scheme that makes it easy to organize, archive, find, sort (order), and search mailboxes. I discuss my naming style in my Procmail Quick Start.

 

My Implementation

I implement this strategy using . . .

• Procmail to snag viruses and sort solicited-bulk (blue) & trusted-non-bulk-sender (green) incoming messages as they arrive at the mail server.
  
  
• The latest SpamAssassin (which at the moment is 3.0.2) to tag most of my “catchall” (yellow and red) messages with a spam score. Note that you will get significantly better results if you use the most up-to-date SpamAssassin.
  
  One of the keys to my deflexion strategy is to have the spam score (_HITS_) inserted into the beginning of the Subject header. This makes it easy to order messages by score because I can simply do a sort-by-subject in my mail client. I recommend that you surround the Subject tag with braces (squiggly brackets) because 1) these tagged Subjects will then sort below almost all other Subjects when the Subjects are sorted using an ASCII sort (because  {  is ASCII character 123 out of 127); and 2) a popular alternative, square brackets, will not work in an IMAP sort because the IMAP sort and thread specification says to ignore text that is in square brackets during a sort. If you want to muck around with the format of the _HITS_ token, see Keith C. Ivey's message Re: adding SPAM hits score to headers in the SA mailing list.
  
  Below are the especially useful setting in my SA 2.64 user_prefs file (comments are preceded by #). The first two settings are essential to my deflexion strategy.

# Site-wide settings go in ...etc/mail/spamassassin/local.cf (location depends on installation)
# User-specific settings override site-wide settings and go in ~/.spamassassin/user_prefs
   
# To insert the spam score at the beginning of the Subject, include the following two settings.
# This is an essential part of my deflexion strategy.
rewrite_subject  1
subject_tag      {* _HITS_ *}
 			
# Because automatic spam/non-spam separation is not as simple as black/white or yes/no,
# I want the "spamminess" to be put in the X-Spam-Level header and Subject header 
# (see above 2 settings) of every message that gets a score (hits) of 1.00 or more.
# [Note that this is a much lower threshold than what most people use because I use
#  the scores to stratify my messages (rather than to try to say 'YES this is spam.')]
required_hits    1.00
   
# I need the spam-level "stars" (default character is *) for my Procmail recipes
spam_level_stars 1
  
# I use R (for Red) as the spam-level character because it is easier than * to match in Procmail and
# because spammers are more likely to forge SA headers using the default spam-level character, which is *
add_header all Level _STARS(R)_
# NOTE: add_header was introduced in 2.60. If you use SA 2.55 or earlier, use the following instead:
## spam_level_char  R

 
# I (unfortunately) read only English so I use the following settings
ok_languages     en
ok_locales       en
  

# Since I use Pine 4.61 as my mail client and Dallman Ross's Virus Snaggers (vsnag), 
# I do not need to worry about malicious attachments. And I need each 
# message to be close to its original state for the message processing that
# I do using Procmail, IMAP, and "bounce forward"
report_safe      0 
# NOTE: report_safe is available in SA version 2.50 and later (and you need 2.51 or
#  later to be able to set its value to 2). If you are using 2.4x or earlier, use
#  defang_mime instead (but you really should upgrade!)


   
# NOTE: use_bayes is available in SA version 2.50 and later  
# I do not use SA's Bayesian analysis because with my deflexion system, almost every 
# message that SA sees is spam and thus SA cannot automatically learn much about my 
# non-spam messages (and I don't want to spend time using sa-learn to train SA; I'd
# rather spend that time training my procmail-invoked greenlists and bluelists!)
use_bayes        0

# NOTE: trusted_networks is available in SA version 2.60 and later
# Some of my email addresses are hosted on other systems and automatically
# redirected or forwarded to this system. To save SA from checking those 
# system's IP addresses, I tell SA to trust them. 
## trusted_networks ip.add.re.ss[/mask]

# I do not skip RBL checks because they improve the accuracy of SA's scores. 
# NOTE 1: SA 2.60-rc4 and later are much better at timing out if a Real-time
# Block List (RBL) is having problems or is dead. NOTE 2: SA does not use RBLs
#  to "block" mail, it uses them to help determine a message's score.
skip_rbl_checks  0

# I use the following so that I (& my users) will be able to see that a message
# was processed by SA running at spam.deflexion.com. I prepend 'www.' because
# some mail clients, for example pine, will then turn it into a link.  
add_header all Checker-Version SpamAssassin _VERSION_ (_SUBVERSION_) on www.spam.deflexion.com
  
  
# If you are thinking about changing some of the default scores that SA gives, I recommend
# that you first read Matt Kettler's SA-rules-howto.txt and note that in order for 
# user rules to work with spamd 2.55 & later, local.cf must contain allow_user_rules
# (if you run spamassassin rather than spamd, custom user rules will work by default)
## put custom user rules here
  
# For info about the following rule, see Re: habeas problems in the SA discussion group
# If you use Bayes, I suggest you read the thread bayes should ignore habeas headers?
score HABEAS_SWE -0.1

• 	The above SpamAssassin user_prefs variables work SA 2.64. Some of these variables do not exist in earlier or later versions of SA so make sure that you read the documentation for your version of SA. For example, In SA 3.x, The rewrite_subject and subject_tag configuration options were deprecated and are now removed. Instead, use rewrite_header Subject [your desired setting]

   
  
  To see if your user_prefs file is syntactically valid, type

  spamassassin --lint

  and make sure that this invokes the spamassassin that is processing your messages! To check which spamassassin the above command invokes, type

  which spamassassin

  To check what version this is, type

  spamassassin --version

  To see what tests SA performs, for example to check if network tests are happening, type

  spamassassin --lint --debug

  
  
• Procmail to filter SA-tagged messages into yellow, red, and infrared (to-be-devnulled) mailboxes. The Procmail recipes that I use for this are in my Procmail Quick Start in the Using SpamAssassin section.
  
  
• Mozilla, Mulberry, and Pine to check, process, and optionally snarf multiple mailboxes that reside on the mail server (using IMAP, the Internet Message Access Protocol)
  
  
• A magnet-updater script that I run on my mail server to automatically update my blue and green lists.
  
  
• Procmail recipes that detect almost any mailing-list message and delivers it to a mailbox with a name that contains the local part (left-hand side) of the mailing-list address.

But, you can implement this strategy using many different tools. For example, if your mail client can do filtering and can be set up to do spam-scoring (possibly via a plug-in), you can use it to do the scoring, filtering, and processing of your mailboxes. Another option is to find a mail hosting service that gives their users the option to use server-based filtering and spam-detection tools. My IMAP Service Providers page includes many such providers.

 

 

Dissecting the Flowchart

(To Be Written)

explanation of my strategy (why i do things this way and in this order); the imperfect metaphor I'm using (light:prism :: incoming-messages:message-filtering software); details about how I implement and adapt it using Procmail, SpamAssassin, Mozilla, Pine, Mulberry; my magnet-updater script

 

 

 

 

See Also

Definitions and Terminology

• A magnet is a filtering recipe (aka rule) that deflects a message into a particular mailbox. To learn how the term magnet is used in the world of POPFile, which is an excellent email-classification tool, see What is a magnet? in the POPFile Wiki and FAQ.
• Internet Mail Consortium: Unsolicited Bulk Email (UBE): Definitions and Problems -- also discusses the term Unsolicited Commercial Email (UCE)
• Imperial College Free On-Line Dictionary of Computing (FOLDOC): spam (in mixed case or all lower-case letters)
• Hormel Foods Corporation: SPAM (in all upper case letters) -- also see Hormel's very cool “position on the relationship between UCE and our trademark SPAM”
• Procmail mailing list: Discussion of the terms greenlist and whitelist in Re: Creating a "Don't Kill" file... and Re: Certified Mail Delivery agent, both by David W. Tamkin

 

Spam in the News

• MessageLabs > News And Events > News Articles
• WeCanStopSpam.org: SpamNewsSources -- this is a Wiki that anyone can edit
• Spamotomy.com -- top page includes a section called “Spam in the News”
• Dougal Campbell: The War on Spam
• popfile.sourceforge.net: POPFile and John Graham-Cumming in the media (Internet, radio, TV or print) -- the items on this page include a lot of good general spam news and information
• Deflexion.com: Deflexion & Reflexion from Infinite Ink -- links and commentary in the blog style, many items are related to spam and email in general
• Schpider.com: Spam News

 

Strategies for Reverse Spam Filtering

• About.com: How to negative filter spam
• About.com: The Spam Series, especially Spam 7: Filtering Out Spam
• Gerald Oskoboiny: Whitelist-based spam filtering
• Nic's Rejecting spam with a procmail accept list
• Caffeine: squashing spam
• Pete Beim's [Unofficial] Eudora FAQs & Links: Filtering SPAM By Checking Who The Message Is Addressed To (Concept By Anthony Greene)
• Prof. Timo Salmi:  Foiling Spam with an Email Password System
• Procmail mailing list:  Re: jump-through-my-hoops autoresponders by David W. Tamkin -- this is related to Timo's strategy above

 

Tools for Reverse Spam Filtering

• On 2004 March 19, Maciej Ceglowski announced that he and Joshua Schachter are working on LOAF, a GPL'd distributed social network filter that seems like it will be a robust and private way to greenlist your correspondents and limelist the correspondents of your correspondents. Version 0.1 works with Procmail and Pine.
• Taughannock Networks (taugh.com): Technical responses to spam (pdf format) by John R. Levine, June 2003
• Infinite Ink (ii.com): Procmail Quick Start -- this steps you through setting up Procmail to filter solicited bulk email into mailboxes named --list-name; also see the section Avoiding False Positives With Greenlists and Bluelists
• Infinite Ink (ii.com): Power Pine, especially the sections on Setting Local and Remote Folder Collections and Creating an IMAP-Accessible Personal Address Book -- I use IMAP-accessible sent mailboxes and address books, along with a script called makegreenlist, to maintain my "greenlist"
• Postfix SpamAssassin Procmail by Robert Bushman
• Sieve: A Mail Filtering Language - informal coordination point and RFC3028
• Mulberry and Sieve
• Jason R. Mastaler's Tagged Message Delivery Agent (TMDA) -- TMDA's Whitelist-centric Strategy: “Deny everything that is not explicitly allowed”
• John Conover's Confirmed Mail Delivery (CMD) -- “If the server/gateway is handling a message from an e-mail address that is in the "whitelist", it is delivered with no intervention. However, if the e-mail address is not in the "whitelist", the message is filed, and a request sent back to the sender asking for a reply confirmation; if the sender replies, then the original message is delivered , and the sender's e-mail address added to the "whitelist".”
• Catherine A. Hampton's SpamBouncer -- can be set up to use a LEGITLIST and NOBOUNCE, which are like my bluelist (Solicited Bulk Email) and greenlist
• Molten Lava Productions: Spambam
• SpamAssassin -- the README includes a section called Automatic Whitelist System
• Active Spam Killer (ASK) -- challenge-response system with a whitelist
• UnixReview.com: Comparison of Client Methods to Block Spam by Robert Haskins
• Sendmail's Content Filter API (milter) “provides third-party programs to access mail messages as they are being processed by the Mail Transfer Agent (MTA), allowing them to examine and modify message content and meta- information.” Also see Milter community web site and Sendmail Milters in Python.
• well connected trust network - “For Eudora, this means that well connected can order your mailbox by trust metric, learning automatically the people you know well and who's emails are more important. Spam sinks like a rock to the lowest level in your inbox, and when you sit down at your computer, even if you only have a few minutes, you see the emails you most care about.”
• Bayesian Nets, Latent Semantics, Despamming and other speculations in Tim Oren's weblog

 

Miscellaneous

• From Stargazers to Starships by David P. Stern: The Many Colors of Sunlight -- this is related to the imperfect metaphor I'm using (light:prism :: incoming-messages:message-filtering software)
• The Joy of Visual Perception by Peter K. Kaiser: 1931 CIE Chromaticity Diagram

 

MetaLinks

• dmoz.org: Computers: Internet: Abuse: Spam
• Google Web Directory > Computers > Internet > Abuse > Spam
• I am collecting Spam-related links at del.icio.us/Deflexion.com/Messaging/Spam.