Well, that's what I get for taking a vacation ;o)

Anyhow Chuck's problem was resolved rather quickly -- the errors about the spam folders were due to LP not filtering his spam messages correctly. And the filtering was working correctly because his user_prefs file was the default file that I bundle in my zip archive that has a place where you MUST insert your proper cpanel username. Without that username being put in the user_prefs file, SpamAssassin will refuse to run, which means LP cannot filter messages properly.

By the way, anyone wondering where the last few messages went, Mitch, one of the lead moderators/admins of the forums decided to merge a number of messages into a central place and created a new message thread where we could just discuss the failing SpamAssassin engine issue on various servers.

The first few pages go back a couple of months, but catch up to recent messages soon enough:
http://www.lunarforums.com/index.php/topic,42723.0.html

Hello Ian,

Thank you very much for creating spam assassin.

I've set SA up and the script seems to be working well. When I run the cgi script I get the following message:

sa-trainer.cgi version 3.04 by Ian Douglas, iandouglas.com, Copyright 2004-2007
Some Rights Reserved under a Creative Commons "Attribution Non-commercial" license
Support for this script available here

Autodetected mail storage as Maildir; you could speed up this script slightly if you configure $mail_format in the script to "Maildir"

Training SpamAssassin for mydomain.com:
WARNING: Could not find spambox for admin@mydomain.com, cannot scan SPAM
Checking /home/xt88002/mail/mydomain.com/alan/.spam/cur/ to learn SPAM: Learned tokens from 0 message(s) (0 message(s) examined)
WARNING: Could not find spambox for ana@mydomain.com, cannot scan SPAM
Checking /home/xt88002/mail/mydomain.com/cameron/.spam/cur/ to learn SPAM: Learned tokens from 0 message(s) (0 message(s) examined)
WARNING: Could not find spambox for enquiries@mydomain.com, cannot scan SPAM
WARNING: Could not find spambox for globalham@mydomain.com, cannot scan SPAM

Checking Global Email-based Hambox for HAM messages:
Checking /home/xt88002/mail/mydomain.com/globalham/cur/ to learn HAM: Learned tokens from 0 message(s) (0 message(s) examined)

Number of HAM messages scanned over time: 877
Number of SPAM messages scanned over time: 15404

re-scan mailboxes

(I have replaced my domain name with "mydomain.com")

1) Does this look correct?

2) How do I configure $mail_format in the script?

3) How do I access the Spam folders using Thunderbird? Do I have to set up a Spam IMAP account for each Email account I have? I've set up the global address for spam, but how do I see the spam folders? I don't wish to use webmail.

4) I have received a Spam that SA doesn't recognise as Spam. How do I tell SA that it's Spam? I don't see anywhere in your instructions, or this thread, how to deal with spam that not recognised as spam.

I would just like to add, that the Spam that's coming through doesn't appear to be being checked by SA. Am I incorrect? Is this sufficient (or too much?) infomation for you:

From - Wed Sep 05 21:49:38 2007
X-Account-Key: account4
X-UIDL: UID485-1182998340
X-Mozilla-Status: 0000
X-Mozilla-Status2: 00000000
X-Mozilla-Keys:                                                                                 
Return-path: <jai@alltheunknown.com>
Envelope-to: babbittbuchwald@mydomain.com  <--I have Changed my domain name here
Delivery-date: Wed, 05 Sep 2007 02:43:49 -0700
Received: from 61-225-150-170.dynamic.hinet.net ([61.225.150.170])
   by atlas.lunarpages.com with esmtp (Exim 4.66)
   (envelope-from <jai@alltheunknown.com>)
   id 1ISrOO-0004ah-NU
   for babbittbuchwald@mydomain.com; Wed, 05 Sep 2007 02:43:48 -0700
Received: from [61.225.150.170] by mailstore1.secureserver.net; Wed, 36 Aug 2007 17:42:41 +0800
Message-ID: <0107ffa4$0107fe78$aa96e13d@jai>
From: "Alissa Sadler" <jai@alltheunknown.com>
To: <babbittbuchwald@mydomain.com> <--I have Changed my domain name here
Subject: Save $369.05 adobe acrobat 8 $79
Date: Wed, 36 Aug 2007 17:42:41 +0800
MIME-Version: 1.0
Content-Type: text/plain;
   format=flowed;
   charset="windows-1250"
Content-Transfer-Encoding: 7bit
X-Priority: 3
X-MSMail-Priority: Normal
X-Mailer: Microsoft Outlook Express 6.00.2800.1106
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106
X-Spam-Status: No, score=
X-Spam-Score:
X-Spam-Bar:
X-Spam-Flag: NO


Thank you kindly. This is a lot of fun, by the way. Spam sucks!

Not sure how relative this guide is after the cPanel 11 upgrades that changed the way Spam Assassin works a lot.  So with that said, I'm going to go ahead and lock this thread till we can get some more information on this issue to you.  Thanks!

Alright, we are open for business yet again.    Thread unlocked!

The output from the script is a good indication of whether or not it's working -- can you post the output of the script so I can help determine if things are running okay?


This is the most common setup that LP likely has -- you as a user set up your Email client to download all incoming mail via POP3 so it's not stored on the server any longer. However, since the SpamAssassin training script runs on the server, any messages you want to train as ham/spam also need to be copied back onto the server. This is why the training script requires that you set up an IMAP account so you can drag messages from your POP3 Inbox back to your IMAP 'spam' folder, and to do the same with moving non-spam messages to your IMAP 'ham' folder. The difference is that after scanning, you would drag the IMAP 'ham' messages back to your Inbox (or wherever you sort them from there), and you would simply delete the spam messages from the IMAP 'spam' folder.



1. How to configure the script to scan the 'ham' folder in, for example, sa@mydomain.com?

Since the default "global ham" mailbox is set for "globalham" you will need to find the line that says and change it to "sa" like this: This way, all of your 'ham' (non-spam) messages can simply forwarded to that mailbox by all of your domain users, and they will be trained as non-spam the next time you run your script.


2. How to configure the script to scan the 'spam' folder in sa@mydomain.com?

By default, the script will want to scan individual spam mailboxes. You will need to make the following changes to allow for a different scenario:

- change the line that says so the "Y" is set to "N", like this:
- uncomment the line that says so it looks like this: and replace "mydomain.com" with your actual domain. This is an undocumented 'feature' of using the script to configure the folder names -- the folder names are actually disk-path relative to /home/yourusername/mail/ so in this case you're going to tell the script to scan /home/yourusername/mail/mydomain.com/sa/spam/cur/ (as a full disk path including the 'cur' portion for Maildir) for spam messages.


3. Will the script and SpamAssassin handle e-mail copied from my other domains properly? Won't it mark my addresses as spam senders?

Using the global spam folder, I add an extra command-line setting to the actual SpamAssassin training program that tells it to ignore the message headers, so it won't mark your users as spammers. So any messages copied to the ham/spam folders of your sa@mydomain.com account won't necessarily be as *accurate* when training ham/spam, but will definitely be better than nothing. Typically, you would *want* to scan the headers of all incoming mail so it can learn tokens about where the message came from, the date, subject line, etc. but using these 'global' settings make it more convenient for you and your users.

My script has the same requirement as the SpamAssassin software itself: that you have messages back on the server in order to train them. The SpamAssassin software on the LunarPages servers has no way to read the mail sitting on your hard drive that you downloaded via POP3. The most convenient way to move messages back to the server is IMAP. There are other ways to move your messages back to the server, but they are WAY more complicated, which is why I tell people to set up IMAP accounts to move the messages back to LunarPages for scanning. (As for webmail, it's simply just a browser-based interface for IMAP.)

Aside from the fact that Horde is just a web interface for IMAP, yes

But yes, I think that *copying* non-spam to your 'ham' folder is the best way to handle training ham. The other way would be to *move* it to your 'ham' folder, run the training script, then move it all back. But if you *copy* to your 'ham' folder, you can just delete the messages in the 'ham' folder when you're done. The other scenario (move, train, move back) I feel is riskier because there's a chance you forget a needed message and delete it by accident.

Personally, I *move* non-spam messages that I don't need to keep (newsletters, notices, mailing list traffic that doesn't interest me, Email notices that someone replied to this message thread, for example) into my 'ham' folder, and only *copy* over non-spam messages that I *need* to keep copies of (messages for work/contracts, notes from my wife, etc)

Spam *always* gets moved to the 'spam' folder -- no need to keep it around ever.

But personally, I only use IMAP for my Email 'cause I figure that (a) LP has the storage, (b) keeps backups, and (c) since I triple-boot into Gentoo/Ubuntu/Windows frequently throughout the day, I can still access all of my Emails no matter which OS I'm running, or even have access to all of those messages while on the road.

Though, given the CPanel 11 fiasco, I'm still tempted to just set up a Linux box here at home to be my mail server for all of my domains and just download everything here at home, run my own SpamAssassin filters, etc., and not have to rely on any third-party for my Email ... some days I really miss running my own hosting business.

Hi gadgetfan,

Your users can just forward their incoming non-spam messages as-is, either as an attachment or as an inline message, SpamAssassin will learn enough tokens to remember what kind of mail you like to get. It's also configured, via my user_prefs suggestions, to ignore other SpamAssassin headers, so pre-scored messages are fine.

Ian

Ian--

I've got a new issue, and I figured I'd mine your SA expertise before submitting a trouble ticket to LP, if that's what's required.  As of this weekend, SA has gone back to rewriting subject lines and for some reason is putting all of the SA scoring information in the X-Spam-Report header instead of at the beginning of the message body as it used to.  I've also seen a marked decrease in the amount of spam I'm seeing in my spam box.  That last part is not a problem, as long as it's only spam that I'm missing, and not legitimate mail.  I have no evidence that I've missed any legitimate mail, but then again, you often don't know that you didn't receive a message

I've gone into cpanel, disabled SpamAssassin, disabled the Spam box in SA, re-enabled both, and replaced my SpamAssassin user prefs file with the one recommended in the first message of this thread.  Some spam has started to trickle in (though not at the rate that it had been up until the weekend), but it's still rewriting the subject line.  I've even gone and changed the "rewrite_subject" line in user_prefs to 0 and it's still doing the same thing: rewriting subjects and putting scoring data in the X-Spam-Report header.  It's much easier to refile ham messages back into the inbox when they're attached as an .eml attachment, and that's no longer happening.

In my other thread on this subject (sorry for the repeat post...should have posted in this thread to begin with), Mitch said that LP hasn't made any changes that should affect how SA is running or the amount of spam I'm receiving.

Any thoughts?  Your help would be most appreciated.