<pre>
Mail System Design
------------------
1. The mail store
The mail store is a complex problem. On the one hand, we
could simply store RFC822 messages in raw form.
Unfortunately, this could lead to serious problems when trying
to manipulate messages. (For example, the code to parse out
part 2.1.5 of a message would have to actuall parse MIME.) It
might be better to use a view in which MIME messages are
pre-separated into parts. Or perhaps a single message is
stored and a "part" table describes the ranges covered by
various MIME parts.
More research into this needs to be done. For the short term,
using a non-MIME mechanism for storing information we wish to
send out (attachments in bboard, the like) is sufficient. The
conversion can be done at mail send time. Received email can
be treated as text/plain for the moment, even when it is
richer.
Here's a plan for the table design:
message_body
------------
rfc822_id
header_from
header_date
header_message_id
header_content_type
... others
message
-------
message_id
rfc822_id
env_from
env_to
With each message_body being associated with a content value
from the acs_contents table (or CR revisions table.)
Notice that each message body belongs to one *or more*
messages. This is because the same message might be addressed
to more than one recipient, but still be the same message.
Because of this it is recommended that editing messages take
place via "copy on write" rather than "modify in place". This
will allow messages to be edited, but for old versions in
other locations not be be lost. (Keep the message_id the
same, but point it at a new rfc822_id for the body.)
Separating messages from message_bodies also makes it more
clear how to determine what access control and URL any message
belongs to. Each message belongs in only one place, but the
message_body may be shared.
The above design for message_body should include whatever
header fields are determined to be necessary and useful.
Difficulties arise, since for email addresses it would be nice
for this version to point at parties, but the "from" header of
messages may actually contain more than one address, and so
on. Some thought needs to be applied here.
2. Sending email
The first iteration should send email using ns_sendmail or the
Oracle UTL_SMTP stuff. ns_sendmail is preferred, since it's
much higher level than UTL_SMTP.
A simple queue should be sufficient for sending. Each message
may be queued to be sent to various people, like so:
outgoing_message_queue
----------------------
message_id
Notice that the pointer is to a message, not a message body.
The expectation is that a new message is created (to contain
the new envelope information), and that that message is
recorded in this queue.
Once the message is sent, the message is removed from this
queue (and possibly added to an audit trail? Kept in the
queue marked as done?)
The requirement to be able to delete messages from the system
immediately after deletion means that there are some other
issues here. See "open issues" below.
3. Receiving email
The simplest design for receiving email into various queues
assumes that each message actually belongs to one and only one
queue:
incoming_message_queue
----------------------
message_id
queue_name
A table can describe a set of rules (with like patterns?) to
use to determine what queue_name should be used for initial
insertion. This gives any application the capability to
periodically find all new messages that it's interested in
processing. Removal from the queue is per application
request.
An alternative method is for messages to go in all queues with
matching requirements. The same as above, only unique on the
pair instead of just message_id. This would allow messages to
be placed into multiple queues for processing by more than one
application. I suspect this is unnecessary.
A final thought is that it would be possible to simply record
message ids in this table (as in the outgoing queue.) Then
individual applications do a select over the incoming message
queue, looking for interesting messages. When they are
finished, they delete the message. Again, this is based on
the "no message will belong to more than one application"
model.
4. Manipulating messages
This is a difficult task in Tcl, since MIME is fairly
complicated. There is already a good API for Java. I suggest
that for the Tcl API, simple abilities to deal with encoding
(and later possibly decoding) messages with multipart/mixed
and multipart/alternative content types. This will require
good base64 encoding tools in Tcl.
5. Open questions
The biggest open question is garbage collection.
Paradoxically, the split into messages and message_bodys may
help with this. Since we can imagine that each application
keeps its own message_body object, the application is allowed
to delete that message_body in a reasonable way. At that
point, it's sufficient to periodically clear the message store
of any message bodies that have no associated messages!
Extension of the message class also becomes more possible.
When a new message comes in to be entered into a bboard forum,
the bboard application can create a new message object
pointing at the same message body. The new message object can
be a "bboard_message". Previously, acs_messaging would create
the message object, and bboard would have to deal with it even
if it were a "normal" message.
It is unclear at this point whether this system should
actually provide mechanisms to automatically build digests.
On the one hand, it already has scheduled tasks to do such
things. On the other hand, putting the generic mechanisms to
handle collation and the like into the system is messy. And
on top of that, there are optimizations that applications can
do. (For example, bboard could build a single message_body
for a daily digest and send it out to all subscribers, instead
of messaging building a customized message for each
subscriber, because it can't determine that they would all be
the same.)
6. Data model in short:
outgoing_message_queue
| (message_id)
|
message_body -- message --<|------------ [ extended by application ]
(rfc822_id) | (message_id)
|
incoming_message_queue
(message_id, queue_name)
A scheduled process periodically sends any messages in the
outgoing message_queue and deletes the message entry.
When messages arrive, some MTA mechanism inserts them as a
message_body, with a message containing the envelope info, and
puts that message into the appropriate
incoming_message_queues.
A scheduled process periodically deletes any message_body for
which there exists no referencing message.
Each message belongs to an individual application. That
application is responsible for removing it once it is no
longer needed (and then GC can be accomplished.)
</pre>