A
Andy \Krazy\ Glew
Ever since I started using Outlook in 1995, I have
wasted a hell of a lot of time trying to get Outlook's
rules to run reliably. I've started a wiki page
describing my problems - and I thought that I might post
it here, seeking help.
If you want to reply, post here, but also please send
email to outlook-rule-problems-2004 AT patten-glew DOT net.
(I am posting this from my company account,
because it is a company-related problem, so I
think this is a legitimate business use. But email
works more reliably on my non-work accounts.)
Disclaimer: this is personal information and opinion,
not the position of my employer, Intel.
-- Created by Main.AndyGlew - 15 Oct 2004
---+ Outlook Mail Sorting Rules Reliability Problems
I depend on email spam filtering and rules that automatically sort mail into
different topics and priorities. However, I have not had rules that work
reliably on Outlook. (I have had great success with rules on other email
systems.)
My employer's department, circa October 2004, does not support rules.
Occasionally I have obtained minor help, but the problems seem
to be like toothpaste - it only seems to squeeze elsewhere.
I will record these in reverse chronological order,
most recent problems at topic.
See OutlookMailSortingRulesIssueTrackingTOC for a table of contents.
---++ Monday October 18, 2004
Sigh. My rules seem to have been run incompletely today.
This morning, there were 9 messages in
my server side Inbox. I ran my rules on them manually.
This afternoon, after the DPG BUM,
there were 28 rules in my server side Inbox.
Now, I only had 9+28=37 messages in my server side
Inbox, because many - around 40 - _were_ successfully
sorted by a first rule that segregated my XP mailing list email.
However, it is frustrating that the remaining rules did not get
run.
---+++ SpamBayes inteference
I have to mention the possibility of SpamBayes interfering.
Possibly SpamBayes has hidden a message while the rule should be
looking at it.
However, although Spambayes inteference may be a problem,
I am inclined to ignore this possibility because I have
had these rule problems before.
I may also install SpamBayes' proxy mode, to eliminate
SpamBayes contention in the Outlook message store.
Hmm... no, won't work. I was about to say, if I could
gtet Outlook to forward all of my email to UNIX,
and then get UNIX to process it and mail it back to me,
I might be able to get reliable rules.
But our IT people make it hard to forward out and back.
---++ October 15, 2004 - 5pm
This time I went to a conference room,
surprise disconnect,
Outlook was running, thought it was online,
but was disconnected since I did not reconnect
in the conference room.
On return to my desk, mail was in my server side Inbox,
and did not get my rules run on it - nor did SpamBayes run.
Conjecture: the not-really-connected Outlook session never
saw the message arrival; the messages were marked as arrived,
even though not processed by rules, so they were not
subject to rules at next startup or synchronization.
This "surprise" disconnect may be a common thread:
it may also explain the failures overnight,
if going into standby has the same sort of problem.
---++ October 15, 2004 - Bug: rules stopped running overnight
Yesterday, after talking to my employer's Engineering Computer Support's
Josh, I had some hope that the rules were working.
Incoming email was being spam filtered and sorted reliably.
However, I left my laptop attached to the network
overnight. Returning to work in the morning,
all of the email received overnight was sitting in
my server Inbox, unsorted and un-spam filtered.
Exiting and reentering Outlook did not cause the rules
to be run. Apparently, these emails were already classified
as having arrived, and the rules did not trigger for them.
(By the way, my current rules configuration
moves all email out of the server side Inbox,
moving it to various client side folders,
to remove any doubt as to whether mail in the server side
Inbox has been
processed by the rules or not.)
Running the rules manually worked, as usual.
Now that I am actively using the machine,
the rules seem to be working on incoming mail.
As usual, it is just mail that was received overnight
that seems to not be handled appropriately.
However, my laptop Outlook client was active all
night long... I conjecture that a power saving mode
may have affected rule processing
I.e. it may have been active from my point of view,
but it may have gone into standby.
Perhaps Outlook rule processing doesn't properly
understand standby mode and client side rules.
---+++ This is a Security Problem
Outlook's unreliable rules may constitute a security problem.
E.g. my (e-mail address removed) and (e-mail address removed) accounts do virus scanning:
I currently request them to tag and forward the possibly virus
email, because occasionally there have been false positives.
I use mail sorting rules to segregate such suspicious email.
If the rules do not work, the suspicious email appears in
folders where I may not be as careful as I normally am
in processing suspicious email that has been placed into
special segregated folders.
Hard to prove, but, can the software vendor be liable for such problems?
---++ Thursday October 14, 2004
Today noticed that the rule that was supposed to filter out
extremeprogramming mailing list email into an older folder
was not always firing; XP mail was being treated as ordinary mail,
and placed in my client side Inbox.
The rules looked something like:
* XP
* if string extremeprogramming in To:
* apply category "Sorted into Folder"
* move to XP - extremeprogramming client side folder
* client side inbox
* for all mail
* except if already has category "Sorted into Folder"
* apply category "Inbox - handled"
* move to Inbox - client side
* copy to server side folder Inbox - copied to client
Of course, the rules were running manually, correctly.
Sometimes they ran on incoming mail;
sometimes they did not.
They seemed to always run correctly on mail received
while I was not running Outlook,
i.e. when rules are run when first connected.
XP (extremeprogramming) mail was appearing in the "Inbox - client"
folder, with the "Inbox - handled" category,
but did not have the "Sorted into Subfolder" category.
Clearly, the subfolder rule was not triggering.
However, as usual it triggered correctly when
the rules were run manually.
I might be suspicious if one rule were server side
and one were client side - but both rules were clientside.
In exasperation called up Engineering Computer Support and
begged for help, even though rules are not officially supported.
Josh took a look at my rules, thought they looked reasonable.
He suggested adding the "stop processing more rules" clause
* XP
* if string extremeprogramming in To:
* apply category "Sorted into Folder"
* move to XP - extremeprogramming client side folder
* stop processing more rules
* client side inbox
* for all mail
* except if already has category "Sorted into Folder"
* apply category "Inbox - handled"
* move to Inbox - client side
* copy to server side folder Inbox - copied to client
Did this. The problem seemed to go away.
Note: I have used the stop processing more rules clause in
the past. Recently, I have started using the category tagging
as an alternative, since it seemed to be more reliable.
Now I am using both.
As we will see above, the problem just moves.
---+++ Conjecture: Mutex Problem?
It nearly always happens that rules run correctly when
run by hand. The rules often, but not always, run correctly
when run at Outlook startup - clientside rules running
on mail received while the client is disconnected.
Most of the rule problems occur when they run on the fly,
during heavy load - when there is lots of incoming mail.
Problems such as the above are the sort of thing that
happens if there are mutual exclusion violations.
E.g. imagine that the XP rule and the Inbox-clent rule
have both triggered, and both are trying to modify the
categories by a read-modify-write. If there is a race,
the XP rule categorization may be lost.
It would be strange, however, to run these rules in parallel.
Both are client side, and their semantics are supposedly
sequential, with the earlier rule taking priority over the
latter. I'm grasping at straws here.... but perhaps they
are run in parallel by default, and clauses such as
"Stop Processing More Rules" impose a partial order?
And perhaps the code that runs them in parallel is not
always correct about when it is permissible?
---++ Historical Problems and Approaches
I'm not going to reconstruct the whole sorry story,
the problems that I have had with Outlook rules over *years*.
But I'll note the high (low) points here.
---+++ Filteringt based on presence in Address Book
Before I started using SpamBayes, I was quite desperately
trying to improve my spam filter quality.
Amongst other things, I tried to filter out all email
from did not come from my employer's Exchange server,
by using the rule
* where sender is in specified (the company) Address Book.
As usual, this rule ran correctly manually,
but it did not run while I was up and running
connected full time to the server.
I asked my employer's IT TAC support for help.
They don't officially support Outlook rules,
but took a perfunctory look.
As usual, they reported "seems to work for me"
\- which is exactly my situation - the rules seem to work,
but then stop working in hard-to-isolate circumstances.
---+++ Rule Overflow
The server side storage for rules is quite small, something like 32K.
When I first moved my rules from UNIX procmail to Outlook,
I overflowed this storage.
Unfortunately, Outlook did not handle this overflow cleanly:
instead it seemed to randomly drop stuff, sometimes the first
rules, sometimes the last, and sometimes in the middle of a rule.
Hmm... I wonder if this sort of misbehavior could be exploited by a virus?
---+++ Usage Models
As noted elsewhere, the rules get run at several different times
* manually
* when mail is received on server
* when Outlook client connects to server when Outlook starts
* when Outlook client connects to server, synchronizing (periodic or
manual) when working offline
The division into server side and client side rules
affecdts which rules run at which time.
This is my current understanding:
* manually
* both server and client side rules run, in the interleaved order
specified
* when mail is received on server
* if client is connected
* both server and client side rules run, in the interleaved order
specified
* if client is not connected
* only server side rules run; client side rules run later
* when Outlook client connects to server when Outlook starts
* client side rules run on email received while disconnected
* when Outlook client connects to server, synchronizing (periodic or
manual) when working offline
* client side rules run on email received while disconnected
Because the server and client side rules run in different orders depending
on whether the client is connected, I have occasionally tried to force all
rules to be client side. However, I usually find I want one or two server
side rules. However-however, the very small limits on server storage space
nearly always mean that any rule that sorts mail into subfolders has to use
client storage, and becomes client side.
The various times that rules run have different reliabilities:
* manually
* almost always works (quality score 9/10)
* when mail is received on server
* if client is connected
* occasionally has problems (quality score 6/10)
* if client is disconnected
* often has problems (quality score 3/10)
* when Outlook client connects to server when Outlook starts
* sometimes has problems, but tends to completely work
or completely not work (quality score 7/10)
* when Outlook client connects to server, synchronizing (periodic or
manual) when working offline
* sometimes has problems, but tends to completely work
or completely not work (quality score 8/10)
These observations about quality or reliability of the different
times have influenced various attempts to define usage
models that avoid the unreliable situations.
---++++ Always Disconnected Model
The most successful Usage Model I have found, from the point
of view of Outlook rules, has been to always run disconnected.
The rules then only run when manually synchronizing with the
server.
I find if that is the only time the rules are run,
behaviour is reliable.
I practiced this usage model at a first company 2000-2002.
Unfortunately, although this usage model leads to reliable
behavior for Outlook rules, it has other problems:
1. Outlook only allows rules to be run manually or edited when
connected to an Exchange server - even when the rules
are local, using local input and outputs, and do not use
the server at all.
2. Being mainly disconnected led to annoyances such as
* not seeing newly arrived email until I synched
* sent email waiting until next synch to be sent
* Overall, quantization errors - I often found that
I had thought I had sent somebody email, but it was
sitting on my laptop waiting to send, and vice versa.
3. Manually synchronizing could take half an hour at a time,
interrupting work flow.
---++++ Always Disconnected Model - non-Outlook Rules
I addressed the problem of not being able to run or edit Outlook
rules when the client is disconnected
by rolling my own rules using Visual Basic Automation,
binding my rule system to a button.
These rules worked better than Outlook's rule system
- they always ran reliably.
They were not so convenient as Outlook's rules
(e.g. they did not track folder renames)
but they were significantly more powerful.
I used this approach at the first company in 2002.
But, they suffered bit rot. When they stopped working
after an Outlook upgrade and a change of employer,
I never tried to get them working again.
I kept hoping, vainly, that Outlook's built-in rules
would start working reliably in the new release.
---++++ Always Connected Client Model
I keep hoping that if I can just leave my client
always connected, with Outlook always running,
the Outlook rules will work reliably.
Unfortunately, as of October 2004, this is not true.
I suspect that some of the problems may be due to
the client going into power saving modes.
In the past, circa 2002, I have disabled power saving
for the clientr and gotten more reliable behavior.
I have not yet done so to see if it helps with my 2004
Outlook problems.
---+++ Non-Outlook mail approaches
I started using Outlook in 1995, when Microsoft documents
started becoming common in my email.
I continue to use UNIX utilities such as procmail
on other systems, so compare my experience there to
my bad experience with Outlook rules.
I continue to use Outlook because it *is* integrated
better with Windows, and because corporate IT mandates
Outlook based calendaring, which does not work if you use
non-Outlook email. Outlook's rules are my only major
complaint.
I'm not a UNIX bigot. I'll use Outlook. But I would really like
to have reliable rules in Outlook. Rules are so damned important to me that
I would abandon Outlook, if I could find a good solution.
I would be willing to switch away from Outlook if I could
find a solution that
* allowed me to use a single email account
* I do NOT like having two different email accounts,
one for Outlook and one for non-Outlook (UNIX) email
* allowed me to read Microsoft documents easily
in email
* allowed me to use the corporate calendaring mechanism
- Outlook Calendar
---++++ Single Email Account
I'm lazy. I do not want to use two different email accounts,
one for Outlook and one non-Outlook. I always forget to read
the other account.
Or, more specifically, I only want to use one mail reader,
and I want to have all of my accounts' email placed in
same reader. Not different instances of the same reader
- but the same reader. I am a single input queue system
I only grudgingly use different mailers to handle personal
and work email, for legal reasons. I would infinitely
prefer to handle all of my email in one mailer.
---+++++ Why not forward?
I would gladly forward all of my Outlook email to a UNIX account.
There seem to be two ways to do this:
1. All or nothing
* You can't log in to Outlook at all
* You get no Outlook calendaring account
2. Outlook rule forwarding
* Only works if your client is running. Breaks easily.
There seems to be no way to get an Outlook calendaring account,
and still arrange for all non-Calendar email to be sent
outside Outlook.
---++++ Microsoft Application Format Email
I have been fairly successful in configuring UNIX like tools
to read Microsoft documents such as PowerPoint and Excel.
My main approach has been to use CygWin on top of Windows.
First, though, I'll talk about the other approaches I have tried.
When I was still UNIX workstation bound, I regularly received
all of my email on UNIX, segrated Microsoft attachments, wrote thdem to a
shared filesystem where I could read them from a PC.
This worked pretty well, but added an annoying level of indirection
to reading such attachments.
I abandoned this approach (a) when Microsoft documents became more common,
(b) Outlook Calendaring became the corporate standard, and (c) I became
dependent on laptop and tablet PCs.
I've not had great luck with using OpenOffice on UNIX (LINUX)
to handle such documents - mainly because I personally tend to use advanced
features of Word that OpenOffice does not yet support.
Currently this is embedded subdocuments to arbitrary depth;
but once OpenOffice catches up here, there will surely be others.
I've not tried using WINE or other Windows emulation to run the true
Microsoft applications on LINUX. Maybe that'll be next... but I continue to
use Windows as the base OS largely because it has better features, such as
power management and TabletPC, than LINUX.
2000-2002 I have used LINUX on VMware on Windows to read my mail
in a UNIX environment, transferring Windows documents to a shared "virtually
networked" filesystem so that I could read Microsoft document formats using
true Windows tools. But I never made this seamless. Michael Fetterman may
have automated this more completely than I have. Moreover, I found that
VMware a hassle to manage, and did not work well on a power managed laptop.
So far, my most successful approach to handling Microsoft documents in a
UNIX-like environment has been to use Cygwin on Windows.
* I use GNU EMACS mailreading tools, mh-e (nmh) and gnus
* I use fetchmail to access mail fromj the Exchange server
* I use standard UNIX mail sorting tools, procmail, spamassassin etc.
I am somewhat embarassed to admit that I have not completely solved the last
step
* getting MIME to kick off native Microsoft apps to read such email
* and, even when this works, it is nicer to read
a Word document in Outlook's preview window
than to have to kick off another app in a different window
Also
* I have not figured out calendar handling here.
---++++ Outlook Calendaring
I think calendaring tools are great.
My employer's standard is Outlook Calendaring.
Outlook Calendaring is email based.
If you ask IT to send all of your email from
the Exchange server to a non-Outlook (UNIX, SMTP)
account, you lose Outlook Calendaring.
You can't easily arrange to forward non-Calendar
email: this requires an always connected client.
It can't be done server side.
If you have an Exchange account for Outlook calendaring,
people tend to send you email there - and then you
have the hassle of two email accounts / two mailers.
My employer's IT refuses to create special purpose Outlook Calendaring
only email accounts - e.g. an account that is named something
like "Glew, Andy, Outlook Calendaring Only, Don't Send Email."
The best way I can see to read email outside Exchange/Outlook but to use
Outlook Calendaring would be to use fetchmail, but to only fetch
non-calendar messages. Or, better, to leave calendar messages on the
Exchange server where they can be autoaccepted, but to send a copy to your
mailreader so that you can look at them.
(Outlook autoaccept has its own problems.)
---++++ Ximian Evolution
I have not yet tried using Outlook compatible open source projects such as
Ximian evolution.
1. I really would rather get Outlook rules reliable
2. Ximian isn't supported by corporate IT
3. Ximian is too Outlook compatible.
* If I have to use a GUI, I might as well use Outlook
* If I'm not using Outlook, I'd really rather
go back to a powerful interface like EMACS
4. I doubt that Ximian solves the rule problems
that I have with Outlook - although I have not
looked that deeply.
The last is my main obstacle. I don't want another wild
goose chase.
If Ximian is fully Outlook compatible, and supports Outlook's
concept of server side and client side rules, it seems likely
to have the same problems as Outlook.
What I really want is for Ximian to have a better rule system;
for the Ximian rules to run only client side, with well
defined and reliable behavior; and for these special Ximian
rules to NOT be exported to the Exchange server at all
(since I have repeatedly had problems with inconsistencies
between server and client side replicas of the rules).
I do not know if Ximian rules are like this.
Ximian *is* Open Source. I suppose I should do it myself.
Except that much of the Exchange server interaction seems
to not be Open Source.
Anyway, I have too much FUD to chase a Ximian solution.
But if anyone can tell me that it works, great!
If it's a wild goose chase, I'll invite you to dinner.
---+++ Outlook auto-accept
Outlook autoaccept has its own problems.
It only marks meeting requests fully accepted,
or reject them if a conflict;
it does not allow them to be accepted tentatively.
It has enough reliability problems of its own
that IT support and admins nearly always recommend
not to use autoaccept for personal calendars.
Even Microsoft recommends it be used only
for resources such as conference rooms.
One of the nicest thing about using
my own VBA rules is that I got a decent
autoaccept, that was able to do things such as
accepting meetings tenatively.
---+++ Version Control
Outlook does not have a version control system for
its rules. Such a system is essential
for tools like this, where small tweaks
can have problems, and where you want to isolate
these problems over a long time.
I just do it by hand: manually exporting the rules
on a regular basis to a folder that has lots of files
of different dates.
---+++ This has been happening a long time
I started using Outlook in 1995.
I've had these reliability problems with Outlook
rules since the first week.
I have spent far too much time with two company's
IT help personnel, who very seldom know more
about Outlook rules than I do.
I have reported these problems repeatedly to Microsoft.
Outlook's rule handling has improved over the years,
but the problems have never been vanquished.
---+ Disclaimer
Disclaimer: this is personal information and opinion,
not the position of my employer, Intel.
wasted a hell of a lot of time trying to get Outlook's
rules to run reliably. I've started a wiki page
describing my problems - and I thought that I might post
it here, seeking help.
If you want to reply, post here, but also please send
email to outlook-rule-problems-2004 AT patten-glew DOT net.
(I am posting this from my company account,
because it is a company-related problem, so I
think this is a legitimate business use. But email
works more reliably on my non-work accounts.)
Disclaimer: this is personal information and opinion,
not the position of my employer, Intel.
-- Created by Main.AndyGlew - 15 Oct 2004
---+ Outlook Mail Sorting Rules Reliability Problems
I depend on email spam filtering and rules that automatically sort mail into
different topics and priorities. However, I have not had rules that work
reliably on Outlook. (I have had great success with rules on other email
systems.)
My employer's department, circa October 2004, does not support rules.
Occasionally I have obtained minor help, but the problems seem
to be like toothpaste - it only seems to squeeze elsewhere.
I will record these in reverse chronological order,
most recent problems at topic.
See OutlookMailSortingRulesIssueTrackingTOC for a table of contents.
---++ Monday October 18, 2004
Sigh. My rules seem to have been run incompletely today.
This morning, there were 9 messages in
my server side Inbox. I ran my rules on them manually.
This afternoon, after the DPG BUM,
there were 28 rules in my server side Inbox.
Now, I only had 9+28=37 messages in my server side
Inbox, because many - around 40 - _were_ successfully
sorted by a first rule that segregated my XP mailing list email.
However, it is frustrating that the remaining rules did not get
run.
---+++ SpamBayes inteference
I have to mention the possibility of SpamBayes interfering.
Possibly SpamBayes has hidden a message while the rule should be
looking at it.
However, although Spambayes inteference may be a problem,
I am inclined to ignore this possibility because I have
had these rule problems before.
I may also install SpamBayes' proxy mode, to eliminate
SpamBayes contention in the Outlook message store.
Hmm... no, won't work. I was about to say, if I could
gtet Outlook to forward all of my email to UNIX,
and then get UNIX to process it and mail it back to me,
I might be able to get reliable rules.
But our IT people make it hard to forward out and back.
---++ October 15, 2004 - 5pm
This time I went to a conference room,
surprise disconnect,
Outlook was running, thought it was online,
but was disconnected since I did not reconnect
in the conference room.
On return to my desk, mail was in my server side Inbox,
and did not get my rules run on it - nor did SpamBayes run.
Conjecture: the not-really-connected Outlook session never
saw the message arrival; the messages were marked as arrived,
even though not processed by rules, so they were not
subject to rules at next startup or synchronization.
This "surprise" disconnect may be a common thread:
it may also explain the failures overnight,
if going into standby has the same sort of problem.
---++ October 15, 2004 - Bug: rules stopped running overnight
Yesterday, after talking to my employer's Engineering Computer Support's
Josh, I had some hope that the rules were working.
Incoming email was being spam filtered and sorted reliably.
However, I left my laptop attached to the network
overnight. Returning to work in the morning,
all of the email received overnight was sitting in
my server Inbox, unsorted and un-spam filtered.
Exiting and reentering Outlook did not cause the rules
to be run. Apparently, these emails were already classified
as having arrived, and the rules did not trigger for them.
(By the way, my current rules configuration
moves all email out of the server side Inbox,
moving it to various client side folders,
to remove any doubt as to whether mail in the server side
Inbox has been
processed by the rules or not.)
Running the rules manually worked, as usual.
Now that I am actively using the machine,
the rules seem to be working on incoming mail.
As usual, it is just mail that was received overnight
that seems to not be handled appropriately.
However, my laptop Outlook client was active all
night long... I conjecture that a power saving mode
may have affected rule processing
I.e. it may have been active from my point of view,
but it may have gone into standby.
Perhaps Outlook rule processing doesn't properly
understand standby mode and client side rules.
---+++ This is a Security Problem
Outlook's unreliable rules may constitute a security problem.
E.g. my (e-mail address removed) and (e-mail address removed) accounts do virus scanning:
I currently request them to tag and forward the possibly virus
email, because occasionally there have been false positives.
I use mail sorting rules to segregate such suspicious email.
If the rules do not work, the suspicious email appears in
folders where I may not be as careful as I normally am
in processing suspicious email that has been placed into
special segregated folders.
Hard to prove, but, can the software vendor be liable for such problems?
---++ Thursday October 14, 2004
Today noticed that the rule that was supposed to filter out
extremeprogramming mailing list email into an older folder
was not always firing; XP mail was being treated as ordinary mail,
and placed in my client side Inbox.
The rules looked something like:
* XP
* if string extremeprogramming in To:
* apply category "Sorted into Folder"
* move to XP - extremeprogramming client side folder
* client side inbox
* for all mail
* except if already has category "Sorted into Folder"
* apply category "Inbox - handled"
* move to Inbox - client side
* copy to server side folder Inbox - copied to client
Of course, the rules were running manually, correctly.
Sometimes they ran on incoming mail;
sometimes they did not.
They seemed to always run correctly on mail received
while I was not running Outlook,
i.e. when rules are run when first connected.
XP (extremeprogramming) mail was appearing in the "Inbox - client"
folder, with the "Inbox - handled" category,
but did not have the "Sorted into Subfolder" category.
Clearly, the subfolder rule was not triggering.
However, as usual it triggered correctly when
the rules were run manually.
I might be suspicious if one rule were server side
and one were client side - but both rules were clientside.
In exasperation called up Engineering Computer Support and
begged for help, even though rules are not officially supported.
Josh took a look at my rules, thought they looked reasonable.
He suggested adding the "stop processing more rules" clause
* XP
* if string extremeprogramming in To:
* apply category "Sorted into Folder"
* move to XP - extremeprogramming client side folder
* stop processing more rules
* client side inbox
* for all mail
* except if already has category "Sorted into Folder"
* apply category "Inbox - handled"
* move to Inbox - client side
* copy to server side folder Inbox - copied to client
Did this. The problem seemed to go away.
Note: I have used the stop processing more rules clause in
the past. Recently, I have started using the category tagging
as an alternative, since it seemed to be more reliable.
Now I am using both.
As we will see above, the problem just moves.
---+++ Conjecture: Mutex Problem?
It nearly always happens that rules run correctly when
run by hand. The rules often, but not always, run correctly
when run at Outlook startup - clientside rules running
on mail received while the client is disconnected.
Most of the rule problems occur when they run on the fly,
during heavy load - when there is lots of incoming mail.
Problems such as the above are the sort of thing that
happens if there are mutual exclusion violations.
E.g. imagine that the XP rule and the Inbox-clent rule
have both triggered, and both are trying to modify the
categories by a read-modify-write. If there is a race,
the XP rule categorization may be lost.
It would be strange, however, to run these rules in parallel.
Both are client side, and their semantics are supposedly
sequential, with the earlier rule taking priority over the
latter. I'm grasping at straws here.... but perhaps they
are run in parallel by default, and clauses such as
"Stop Processing More Rules" impose a partial order?
And perhaps the code that runs them in parallel is not
always correct about when it is permissible?
---++ Historical Problems and Approaches
I'm not going to reconstruct the whole sorry story,
the problems that I have had with Outlook rules over *years*.
But I'll note the high (low) points here.
---+++ Filteringt based on presence in Address Book
Before I started using SpamBayes, I was quite desperately
trying to improve my spam filter quality.
Amongst other things, I tried to filter out all email
from did not come from my employer's Exchange server,
by using the rule
* where sender is in specified (the company) Address Book.
As usual, this rule ran correctly manually,
but it did not run while I was up and running
connected full time to the server.
I asked my employer's IT TAC support for help.
They don't officially support Outlook rules,
but took a perfunctory look.
As usual, they reported "seems to work for me"
\- which is exactly my situation - the rules seem to work,
but then stop working in hard-to-isolate circumstances.
---+++ Rule Overflow
The server side storage for rules is quite small, something like 32K.
When I first moved my rules from UNIX procmail to Outlook,
I overflowed this storage.
Unfortunately, Outlook did not handle this overflow cleanly:
instead it seemed to randomly drop stuff, sometimes the first
rules, sometimes the last, and sometimes in the middle of a rule.
Hmm... I wonder if this sort of misbehavior could be exploited by a virus?
---+++ Usage Models
As noted elsewhere, the rules get run at several different times
* manually
* when mail is received on server
* when Outlook client connects to server when Outlook starts
* when Outlook client connects to server, synchronizing (periodic or
manual) when working offline
The division into server side and client side rules
affecdts which rules run at which time.
This is my current understanding:
* manually
* both server and client side rules run, in the interleaved order
specified
* when mail is received on server
* if client is connected
* both server and client side rules run, in the interleaved order
specified
* if client is not connected
* only server side rules run; client side rules run later
* when Outlook client connects to server when Outlook starts
* client side rules run on email received while disconnected
* when Outlook client connects to server, synchronizing (periodic or
manual) when working offline
* client side rules run on email received while disconnected
Because the server and client side rules run in different orders depending
on whether the client is connected, I have occasionally tried to force all
rules to be client side. However, I usually find I want one or two server
side rules. However-however, the very small limits on server storage space
nearly always mean that any rule that sorts mail into subfolders has to use
client storage, and becomes client side.
The various times that rules run have different reliabilities:
* manually
* almost always works (quality score 9/10)
* when mail is received on server
* if client is connected
* occasionally has problems (quality score 6/10)
* if client is disconnected
* often has problems (quality score 3/10)
* when Outlook client connects to server when Outlook starts
* sometimes has problems, but tends to completely work
or completely not work (quality score 7/10)
* when Outlook client connects to server, synchronizing (periodic or
manual) when working offline
* sometimes has problems, but tends to completely work
or completely not work (quality score 8/10)
These observations about quality or reliability of the different
times have influenced various attempts to define usage
models that avoid the unreliable situations.
---++++ Always Disconnected Model
The most successful Usage Model I have found, from the point
of view of Outlook rules, has been to always run disconnected.
The rules then only run when manually synchronizing with the
server.
I find if that is the only time the rules are run,
behaviour is reliable.
I practiced this usage model at a first company 2000-2002.
Unfortunately, although this usage model leads to reliable
behavior for Outlook rules, it has other problems:
1. Outlook only allows rules to be run manually or edited when
connected to an Exchange server - even when the rules
are local, using local input and outputs, and do not use
the server at all.
2. Being mainly disconnected led to annoyances such as
* not seeing newly arrived email until I synched
* sent email waiting until next synch to be sent
* Overall, quantization errors - I often found that
I had thought I had sent somebody email, but it was
sitting on my laptop waiting to send, and vice versa.
3. Manually synchronizing could take half an hour at a time,
interrupting work flow.
---++++ Always Disconnected Model - non-Outlook Rules
I addressed the problem of not being able to run or edit Outlook
rules when the client is disconnected
by rolling my own rules using Visual Basic Automation,
binding my rule system to a button.
These rules worked better than Outlook's rule system
- they always ran reliably.
They were not so convenient as Outlook's rules
(e.g. they did not track folder renames)
but they were significantly more powerful.
I used this approach at the first company in 2002.
But, they suffered bit rot. When they stopped working
after an Outlook upgrade and a change of employer,
I never tried to get them working again.
I kept hoping, vainly, that Outlook's built-in rules
would start working reliably in the new release.
---++++ Always Connected Client Model
I keep hoping that if I can just leave my client
always connected, with Outlook always running,
the Outlook rules will work reliably.
Unfortunately, as of October 2004, this is not true.
I suspect that some of the problems may be due to
the client going into power saving modes.
In the past, circa 2002, I have disabled power saving
for the clientr and gotten more reliable behavior.
I have not yet done so to see if it helps with my 2004
Outlook problems.
---+++ Non-Outlook mail approaches
I started using Outlook in 1995, when Microsoft documents
started becoming common in my email.
I continue to use UNIX utilities such as procmail
on other systems, so compare my experience there to
my bad experience with Outlook rules.
I continue to use Outlook because it *is* integrated
better with Windows, and because corporate IT mandates
Outlook based calendaring, which does not work if you use
non-Outlook email. Outlook's rules are my only major
complaint.
I'm not a UNIX bigot. I'll use Outlook. But I would really like
to have reliable rules in Outlook. Rules are so damned important to me that
I would abandon Outlook, if I could find a good solution.
I would be willing to switch away from Outlook if I could
find a solution that
* allowed me to use a single email account
* I do NOT like having two different email accounts,
one for Outlook and one for non-Outlook (UNIX) email
* allowed me to read Microsoft documents easily
in email
* allowed me to use the corporate calendaring mechanism
- Outlook Calendar
---++++ Single Email Account
I'm lazy. I do not want to use two different email accounts,
one for Outlook and one non-Outlook. I always forget to read
the other account.
Or, more specifically, I only want to use one mail reader,
and I want to have all of my accounts' email placed in
same reader. Not different instances of the same reader
- but the same reader. I am a single input queue system
I only grudgingly use different mailers to handle personal
and work email, for legal reasons. I would infinitely
prefer to handle all of my email in one mailer.
---+++++ Why not forward?
I would gladly forward all of my Outlook email to a UNIX account.
There seem to be two ways to do this:
1. All or nothing
* You can't log in to Outlook at all
* You get no Outlook calendaring account
2. Outlook rule forwarding
* Only works if your client is running. Breaks easily.
There seems to be no way to get an Outlook calendaring account,
and still arrange for all non-Calendar email to be sent
outside Outlook.
---++++ Microsoft Application Format Email
I have been fairly successful in configuring UNIX like tools
to read Microsoft documents such as PowerPoint and Excel.
My main approach has been to use CygWin on top of Windows.
First, though, I'll talk about the other approaches I have tried.
When I was still UNIX workstation bound, I regularly received
all of my email on UNIX, segrated Microsoft attachments, wrote thdem to a
shared filesystem where I could read them from a PC.
This worked pretty well, but added an annoying level of indirection
to reading such attachments.
I abandoned this approach (a) when Microsoft documents became more common,
(b) Outlook Calendaring became the corporate standard, and (c) I became
dependent on laptop and tablet PCs.
I've not had great luck with using OpenOffice on UNIX (LINUX)
to handle such documents - mainly because I personally tend to use advanced
features of Word that OpenOffice does not yet support.
Currently this is embedded subdocuments to arbitrary depth;
but once OpenOffice catches up here, there will surely be others.
I've not tried using WINE or other Windows emulation to run the true
Microsoft applications on LINUX. Maybe that'll be next... but I continue to
use Windows as the base OS largely because it has better features, such as
power management and TabletPC, than LINUX.
2000-2002 I have used LINUX on VMware on Windows to read my mail
in a UNIX environment, transferring Windows documents to a shared "virtually
networked" filesystem so that I could read Microsoft document formats using
true Windows tools. But I never made this seamless. Michael Fetterman may
have automated this more completely than I have. Moreover, I found that
VMware a hassle to manage, and did not work well on a power managed laptop.
So far, my most successful approach to handling Microsoft documents in a
UNIX-like environment has been to use Cygwin on Windows.
* I use GNU EMACS mailreading tools, mh-e (nmh) and gnus
* I use fetchmail to access mail fromj the Exchange server
* I use standard UNIX mail sorting tools, procmail, spamassassin etc.
I am somewhat embarassed to admit that I have not completely solved the last
step
* getting MIME to kick off native Microsoft apps to read such email
* and, even when this works, it is nicer to read
a Word document in Outlook's preview window
than to have to kick off another app in a different window
Also
* I have not figured out calendar handling here.
---++++ Outlook Calendaring
I think calendaring tools are great.
My employer's standard is Outlook Calendaring.
Outlook Calendaring is email based.
If you ask IT to send all of your email from
the Exchange server to a non-Outlook (UNIX, SMTP)
account, you lose Outlook Calendaring.
You can't easily arrange to forward non-Calendar
email: this requires an always connected client.
It can't be done server side.
If you have an Exchange account for Outlook calendaring,
people tend to send you email there - and then you
have the hassle of two email accounts / two mailers.
My employer's IT refuses to create special purpose Outlook Calendaring
only email accounts - e.g. an account that is named something
like "Glew, Andy, Outlook Calendaring Only, Don't Send Email."
The best way I can see to read email outside Exchange/Outlook but to use
Outlook Calendaring would be to use fetchmail, but to only fetch
non-calendar messages. Or, better, to leave calendar messages on the
Exchange server where they can be autoaccepted, but to send a copy to your
mailreader so that you can look at them.
(Outlook autoaccept has its own problems.)
---++++ Ximian Evolution
I have not yet tried using Outlook compatible open source projects such as
Ximian evolution.
1. I really would rather get Outlook rules reliable
2. Ximian isn't supported by corporate IT
3. Ximian is too Outlook compatible.
* If I have to use a GUI, I might as well use Outlook
* If I'm not using Outlook, I'd really rather
go back to a powerful interface like EMACS
4. I doubt that Ximian solves the rule problems
that I have with Outlook - although I have not
looked that deeply.
The last is my main obstacle. I don't want another wild
goose chase.
If Ximian is fully Outlook compatible, and supports Outlook's
concept of server side and client side rules, it seems likely
to have the same problems as Outlook.
What I really want is for Ximian to have a better rule system;
for the Ximian rules to run only client side, with well
defined and reliable behavior; and for these special Ximian
rules to NOT be exported to the Exchange server at all
(since I have repeatedly had problems with inconsistencies
between server and client side replicas of the rules).
I do not know if Ximian rules are like this.
Ximian *is* Open Source. I suppose I should do it myself.
Except that much of the Exchange server interaction seems
to not be Open Source.
Anyway, I have too much FUD to chase a Ximian solution.
But if anyone can tell me that it works, great!
If it's a wild goose chase, I'll invite you to dinner.
---+++ Outlook auto-accept
Outlook autoaccept has its own problems.
It only marks meeting requests fully accepted,
or reject them if a conflict;
it does not allow them to be accepted tentatively.
It has enough reliability problems of its own
that IT support and admins nearly always recommend
not to use autoaccept for personal calendars.
Even Microsoft recommends it be used only
for resources such as conference rooms.
One of the nicest thing about using
my own VBA rules is that I got a decent
autoaccept, that was able to do things such as
accepting meetings tenatively.
---+++ Version Control
Outlook does not have a version control system for
its rules. Such a system is essential
for tools like this, where small tweaks
can have problems, and where you want to isolate
these problems over a long time.
I just do it by hand: manually exporting the rules
on a regular basis to a folder that has lots of files
of different dates.
---+++ This has been happening a long time
I started using Outlook in 1995.
I've had these reliability problems with Outlook
rules since the first week.
I have spent far too much time with two company's
IT help personnel, who very seldom know more
about Outlook rules than I do.
I have reported these problems repeatedly to Microsoft.
Outlook's rule handling has improved over the years,
but the problems have never been vanquished.
---+ Disclaimer
Disclaimer: this is personal information and opinion,
not the position of my employer, Intel.