Email Handler

part of the ArsDigita Community System by Henry Minsky

User-accessible directory: none
Site administrator directory: not currently available
data model : sql/email-handler-create.sql
Tcl procs: /tcl/email-handler
Perl procmail script: bin/queue-message.pl

System dependencies: you will be in a world of hurt unless you have Perl DBI/DBD installed (so that a Perl script on your computer can talk to Oracle) and a mailer configured to exec a procedure when mail arrives addressed to a particular alias. See Appendix for installation tips.

The Big Picture

You can build a gateway for handling incoming email messages for your application using a Perl script called queue-message.pl. That Perl script will accept an incoming email message from the mailer and insert its contents into a queue table in the database. A procedure can be scheduled to sweep the queue at some interval to process the messages.

Using the `queue-message.pl` script

The script takes a list of command-line arguments, which tell it which database to connect to, and a classification tag for the message.

  usage: queue_message.pl db_user db_passwd tag

  Inserts the data from stdin into a queue table.

  Assumes the following table and sequence are defined in the db:

    create table incoming_email_queue (
            id          integer primary key,
            tag         varchar(256),
            content             clob,           -- the entire raw message content
                                                -- including all headers
            arrival_time        date
    );

    create sequence incoming_email_queue_sequence;

The tag field is a string tag which you can assign to a message, so that the routine which sweeps the queue can distinguish where it came from. You might use this if you had several different mail recipient aliases on your system, which all accept messages and put the into the queue.

To configure your mailer, you must add a mailer alias which invokes the script. For sendmail, this would be done in the aliases file. For qmail, you create a file in the qmail/alias directory with a name .qmail-your-alias-here.

Example: You are setting up an email handler for user feedback messages which are addressed to user-feedback@yourhost:

/var/alias/.qmail-user-feedback:
|/web/yourwebserver/packages/email-handler/bin/queue-message.pl yourdbuser yourdbpassword user_feedback

The alias above specified that incoming messages will be piped to the perl script, which will connect to the specified database, and will insert the message with the tag "user_feedback".

Some guidelines: Try to sure that the from and reply-to headers on your outgoing message are not the same as your incoming mail handler alias. This will help prevent the possibility of nasty mailer loops, in the case where messages may bounce or be returned for some reason.

Scheduled Procedures and Parsing Mail Messages

The Perl script doesn't do anything except fill the incoming_email_queue Oracle table. A scheduled procedure , process_email_queue, is launched by the email-handler at server startup to sweep the email queue table. It will dispatch on each message tag to call a procedure which you specify in the DispatchTable parameter of the email-handler package.

Setting Up A Mapping From A Tag To An Email Handler Procedure

The email-handle package parameter called DispatchTable lets you set up a mapping between tagged incoming email messages and Tcl procedures to handle them.

The DispatchTable parameter should be set to a list of key-value pairs where the key matches a tag which you used in your call to queue-message.pl and the value is the name of a Tcl procedure to run on the message contents.

When installing the email handler, you must go to the Site Map on your ACS web server and instantiate a copy of the package with the name 'email-handler'. It should be mounted at /email-handler in order to edit it's parameters, although you can probably mount it at another URL and it will still work.

Once it is mounted, you will see the DispatchTable parameter, with a default value as shown:

  DispatchTable   =>  user_feedback|email_handler_test

The example above specifies that tickets with the tag "user_feedback" will be passed to the procedure email_handler_test.

You can set DispatchTable to a list of space-separated key-value pairs, for example

user_feedback|email_handler_test ticket|handle_ticket_tracker_email bboard|handle_bboard_mail

The Tcl procedure invoked by the dispatcher is passed one argument, a string containing the raw message text including headers. It is up to you to parse or handle the message in any way you wish. After the call to your dispatch procedure, the message is deleted from the queue.

Email Handling Utilities

Some routines in /tcl/email-utils will help you parse the raw mail message contents in the db.

parse_email_message message_body: returns an ns_set mapping each mail header to its value, as well as a key named message_body which contains the message body text.
clean_up_html html_text: Returns the html_text with all HTML escape characters quoted properly.

Testing The Installation

Mount the email handler at /email-handler via the Site Map admin pages.
Add a mapping parameter to the email handler dispatch table which looks like
```
email-test|email_handler_test
```
To do this, go to the Site Map page in your ACS admin area, and click Set Parameters on the email-handler mount point. Set the parameter DispatchTable to the above value.
Test the installation by manually calling the queue-message.pl script as follows from a shell:
```
% /web/yourserver/packages/email-handler/bin/queue-message.pl yourdbuser yourdbpasswd email-test
this is some
test email message stuff
^D
```
The perl script should insert an entry into the table incoming_email_queue, which should then (within 10 minutes) get handled by the email handler test procedure and stuffed into the email_handler_test table, where it can be viewed by the test web page at /email-handler on your server.

Create a mailer alias file to accept mail for the address acs-mail-test and stick them into the database with the tag 'email-test':

For qmail,  put the following into a file named
/var/qmail/alias/.qmail-acs-mail-test

|/web/yourwebserver/packages/email-handler/bin/queue-message.pl yourdbuser yourdbpassword email-test

Send email to the address acs-mail-test@yourserver
Check that the mail appears in the /email-handler test page query within ten minutes.

Tips for Oracle 8i Achievers

Oracle 8i (8.1.5 and later) includes a Java virtual machine. You are thus able to load up a Java email parsing library that will take apart the messages in a queue very nicely (presumably more robustly than the Tcl parse_email_message). That would include goodies like parsing out MIME attachments. The ACS Webmail package uses this facility.

There is also a new Tcl standard library available from http://dev.scriptics.com/software/tcllib/ which includes MIME decoding routines. This is not loaded by default with ACS 4, but can be downloaded and installed if needed.

hqm@arsdigita.com

Appendix: Installing Perl DBI

It is necessary to install the Perl DBI module and DBD Oracle driver module. These can be obtained from the web site below:

DBD Oracle Driver

Oracle DBI package

Installation

Untar the Perl DBI and DBD Oracle packages in the packages/email-handler/perl directory

% cd perl
% tar -zxvf DBD-Oracle-1.06.tar.gz
% tar -zxvf DBI-1.14.tar.gz

 Install DBI module
% cd DBI-1.14
% perl Makefile.PL
% make
% make test
% su
# make install
# cd ..

 On Linux (and possibly Solaris), this is needed to allow the
DBD Oracle driver to find the libcntlsh.so loadable module from the
Oracle installation.

# export LD_LIBRARY_PATH=/ora8/m01/app/oracle/product/8.1.6/lib
 Install Oracle Driver module
# cd DBD-Oracle-1.06
# perl Makefile.PL
# make
# make test
# su 
# make install
# cd ..

Test the Perl DBI installation using the test-perl-dbi.pl script in the bin directory.
It may be necessary to change the location of the Perl executable at the start of the test-perl-dbi and queue-message.pl scripts. To find the location of your Perl (> 5.004) executable, you can type
```
# type perl  (if you are using bash)
# perl is hashed (/usr/local/bin/perl)
or
# where perl (if you are using csh)
```
Change the first line of the scripts to this location:
```
#!
```

Workaround for Problem I experienced with Linux: In the case that Perl DBI cannot resolve the Oracle shared library location for the driver, you may need to make a shell script to set the environment variable LD_LIBRARY_PATH: I defined a shell script called queue-message.sh which sets the environment for running an Oracle client:

#!/bin/bash

. /etc/profile

export ORACLE_BASE=/ora8/m01/app/oracle
export ORACLE_HOME=$ORACLE_BASE/product/8.1.6
export PATH=$PATH:$ORACLE_HOME/bin
export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib
export ORACLE_SID=ora8
export ORACLE_TERM=vt100
export ORA_NLS33=$ORACLE_HOME/ocommon/nls/admin/data

NLS_LANG=.UTF8
export NLS_LANG

NLS_DATE_FORMAT=YYYY-MM-DD
export NLS_DATE_FORMAT

TZ=GMT
export TZ

exec `dirname $0`/queue-message.pl $*

Installing hook from your mailer

These are instructions for calling the queue-message.pl script from a mailer when mail arrives to a designated alias or aliases. The instructions are for qmail, but configuring other mailers like sendmail to call the script is straightforward, as long as you can figure out how to get your mailer to execute a script on when an email arrives to a specific address.

For qmail, you need to create an alias file. In a standard installation this would probably be a file named /var/qmail/alias/.qmail-your-alias-address, containing a call to exec the queue-message.pl script:

|/web/yourwebserver/packages/email-handler/bin/queue-message.pl yourdbuser yourdbpassword tagname