emailrelay/doc/reference.txt

E-MailRelay Reference
=====================

Introduction
------------
This is the E-MailRelay reference guide. It contains material which is
supplementary to the user guide.

Command line usage
------------------
The "emailrelay" program supports the following command-line usage:

	emailrelay [<switch> [<switch> ...]]

where <switch> is:
# --admin (-a)
  Enables the administration interface and specifies its listening port number.

# --as-client (-q)
  Runs as a client, forwarding spooled mail to <host>: equivalent to "--log --no-syslog --no-daemon --dont-serve --forward --forward-to".

# --as-proxy (-y)
  Runs as a proxy: equivalent to "--log --close-stderr --immediate --forward-to".

# --as-server (-d)
  Runs as a server: equivalent to "--log --close-stderr --postmaster".

# --client-auth (-C)
  Enables authentication with remote server, using the given secrets file.

# --close-stderr (-e)
  Closes the standard error stream after start-up.

# --connection-timeout (-U)
  Sets the timeout (in seconds) when connecting to a remote server (default is 40).

# --debug (-g)
  Generates debug-level logging (if compiled-in).

# --domain (-D)
  Sets an override for the host's fully qualified domain name.

# --dont-listen (-X)
  Dont listen for smtp connections (usually used with --admin).

# --dont-serve (-x)
  Dont act as a server (usually used with --forward).

# --filter (-z)
  Defines a mail processor program.

# --forward (-f)
  Forwards stored mail on startup (requires --forward-to).

# --forward-to (-o)
  Specifies the remote smtp server (required by --forward and --admin).

# --help (-h)
  Displays help text and exits.

# --immediate (-m)
  Forwards each message as soon as it is received (requires --forward-to).

# --interface (-I)
  Listen on a specific interface.

# --log (-l)
  Writes log information on standard error and syslog.

# --log-time (-L)
  Adds a timestamp to the logging output.

# --no-daemon (-t)
  Does not detach from the terminal.

# --no-syslog (-n)
  Disables syslog output.

# --pid-file (-i)
  Records the daemon process-id in the given file.

# --poll (-O)
  Enables polling with the specified period (requires --forward-to).

# --port (-p)
  Specifies the smtp listening port number.

# --postmaster (-P)
  Deliver to postmaster and reject all other local mailbox addresses.

# --remote-clients (-r)
  Allows remote clients to connect.

# --response-timeout (-T)
  Sets the response timeout (in seconds) when talking to a remote server (default is 1800).

# --server-auth (-S)
  Enables authentication of remote clients, using the given secrets file.

# --spool-dir (-s)
  Specifies the spool directory (default is "/var/spool/emailrelay").

# --user (-u)
  Names the effective user to switch to when started as root (default is "daemon").

# --verbose (-v)
  Generates more verbose output (works with --help and --log).

# --verifier (-Z)
  Defines an external address verifier program.

# --version (-V)
  Displays version information and exits.

Under Windows there are a few differences. Use "--help --verbose"
to see the complete list.

If no command-line switches are supplied at all then the default
behaviour is:
* to run as a daemon, detached from the terminal
* listen on the standard SMTP port (25)
* store e-mail messages in "/usr/local/var/spool/emailrelay" or "/var/spool/emailrelay"
* reject connections from remote clients
* disable the administration interface
* generate no logging or diagnostic messages

The "--as-server" switch makes sure that logging is enabled and that
the standard error stream is closed.

To foward spooled messages to the ISP the command-line switch "--as-client"
is provided to run the program...
* in foreground, exiting when all spooled mail has been processed
* forwarding spooled mail from "/usr/local/var/spool/emailrelay" or "/var/spool/emailrelay"
* without listening on any port
* with error, warning and information messages sent to stderr
* without using syslog

Message store
-------------
Mail messages are stored as text files in the configured spool directory. Each
message is represented as an envelope file and a content file. The envelope
file contains parameters relevant to the SMTP dialogue, and the content file
contains the RFC822 headers and body text.

The filenames used in the message store have a prefix of "emailrelay", followed
by a process-id, timestamp and sequence number, followed by "envelope" or
"content". The envelope files then have an additional suffix to implement a
simple locking scheme.

The envelope suffixes are:
* ".new" -- while the envelope is first being written
* <none> -- while the message is spooled
* ".busy" -- while the message is being forwarded
* ".bad" -- if the message cannot be forwarded
* ".local" -- for copies of the envelope file for delivery to local recipients

Copies of the content file for delivery to local recipients will also have
a "local" suffix.

If a message cannot be forwarded the envelope file is given a "bad" suffix,
and the failure reason is written into the file.

SMTP issues
-----------
# Authentication:

  The AUTH extension is supported, providing the CRAM-MD5 and LOGIN mechanisms.

# Local delivery:

  In server mode (ie. with "--postmaster") recipient addresses like "postmaster",
  "postmaster@localhost" and "postmaster@fqdn" (where "fqdn" is the host's fully
  qualified domain name) are treated as the local postmaster, resulting in local
  delivery rather than mail forwarding. Recipient addresses other than "postmaster"
  addresses with no "at" sign (@) or ending in "@localhost" will be rejected at the
  time the message is submitted by the client.

  Delivery of mail to a local "postmaster" is a feature of E-MailRelay which is
  provided for completeness and for comformance to the SMTP specification. But
  note that the E-MailRelay daemon does not actually deliver mail to the postmaster
  mailbox.  All it does is create an envelope and content file in the spool
  directory with a ".local" suffix. Some external system, such as a shell
  script run from cron calling "procmail", should be used to process the
  ".local" files. An example script is provided.

# Timeouts:

  A simple client-side timeout is implemented which will abort the transaction
  if the server fails to respond to any of the client's SMTP commands within the
  given time period.

# Message loops:

  Message loops are not detected.

# Eight bit messages:

  The 8BITMIME SMTP extension is supported, however no attempt is made to
  re-encode 8-bit messages into 7-bit messages if the next-hop server
  does not.

Administration interface
------------------------
If enabled with the "--admin" command line switch, the E-MailRelay server will
provide a network interface for performing administration tasks. This is a
simple command-line interface which is compatible with "telnet":

	$ emailrelay --as-server --port 125 --forward-to localhost:25 --admin 10026
	$ telnet localhost 10026
	E-MailRelay> help
	E-MailRelay> quit

The "flush" command is used to get the E-MailRelay server to forward spooled
mail to the next SMTP server, as an alternative to running "emailrelay --as-client"
as a separate process. In proxy mode it is a way of getting the proxy
server to scan the spool-directory for new messages.

The utility program "emailrelay-poke" can be used to issue the "flush"
command to an E-MailRelay server. For example:

	$ emailrelay --as-server --port 125 --forward-to localhost:25 --admin 10026
	$ emailrelay-poke 10026
	$ echo $?

To use the "flush" command the next SMTP server address must have been defined
on the "emailrelay" command line at start-up using the "--forward-to" switch.

Also, the "list" command lists the messages in the spool directory, the "info"
command provides network status information and activity statistics, and the
"notify" command enables asynchronous event notification.

Mail processing
---------------
The "--filter" command-line switch allows you to specify a mail pre-processor
program which operates on mail messages as they pass through the E-MailRelay
system. The mail pre-processor program is run as soon as the mail message has been
stored in the spool directory, with the full path of the content file specified
on the command line.

For example, the following command will start a proxy server on port 10025 which
processes mail using the specified filter program, and then forwards the mail on
to the system's default MTA (on port 25):

	emailrelay --as-proxy localhost:smtp --port 10025 --no-syslog \
	  --filter ${HOME}/.emailrelay/filter \
	  --spool-dir ${HOME}/.emailrelay/spool

The pre-processor program should terminate with an exit code of zero to indicate
success, or a value between 1 and 99 to indicate failure. Exit codes between
100 and 107 are reserved for special processing: 100 is used to cancel all
further processing of the message, and 103 has the effect of immediately expiring
any "--poll" timer.

If the pre-processor program terminates with a non-zero exit code then the first
few thousand characters of the standard output stream are searched for a line
starting with "<<" followed by ">>". The text inbetween is taken as a failure
reason, and passed back to the SMTP client.

The pre-processor program can edit any part of the message's envelope file or
content file: E-MailRelay remembers nothing about the message while the
pre-processor is running, except the filename. But if the message is deleted
by the pre-processor then E-MailRelay will be upset, so to avoid the
error message use an exit code of 100.

If the pre-processor program creates new messages in the spool directory then
they may not be processed immediately, or they may be completely ignored. To get
E-MailRelay to pick up new messages in the spool directory use the "--poll"
switch, or run "emailrelay --as-client" from within the pre-processor program.

As an example, the following shell script examines the client's IP address and
conditionally dumps the message into "sendmail" (using the sendmail command-line
interface rather than SMTP):

	#!/bin/sh
	#
	content="${1}"
	envelope="`echo \"${content}\" | sed 's/content/envelope.new/'`"
	ip="`awk '/MailRelay-Client:/ {print $2;exit}' \"${envelope}\"`"
	if test "${ip}" = "192.168.0.2"
	then
		cat "${content}" | /usr/sbin/sendmail -t
		rm -f "${envelope}" "${content}"
		exit 100 # <= cancel further processing by emailrelay
	fi
	exit 0

The first thing this script does is convert the path of the content file which
it is given, into the corresponding envelope file. It then extracts the client's
IP address out of the envelope file using "awk". If this matches then it pipes the
message content into sendmail, deletes the message and exits with a value of 100.
The exit value of 100 tells E-MailRelay to forget the message, and not to
complain about the files disappearing.

A more complex example script which does rot-13 masking is also provided in the
distribution ("emailrelay-process.sh"). This does some simple MIME encoding, and
could be used as a template for more sophisticated message encryption.

Some points to note when writing "--filter" programs:
* The standard input and output are not used; the content filename is passed on the command line.
* Programs run with a reduced set of environment variables.
* The E-MailRelay process is completely blocked while the "--filter" program runs.
* E-MailRelay files use CR-LF line terminators, as required by the RFCs.

Address verification
--------------------
In proxy mode all addresses supplied to the SMTP commands "RCPT" and "VRFY" are
accepted by E-MailRelay as valid. In server mode (or more accurately when
"--postmater" is in effect) addresses are accepted if they contain an at sign
or if they are some sort of local "postmaster" address, as described in the SMTP
section above.

However, this behaviour can be modified by using an external verifier program,
using the "--verifier" command-line switch.

The verifier program is passed a command-line containing: (1) the full
address, (2) the user-name part of the address, (3) the host-name part, (4) the
local host's fully qualified domain name, (5) the current "MAIL" command's "FROM:"
address or the empty string for the "VRFY" command, (6) the IP address of the
client connection, (7) the authentication mechanism used by the client ("NONE"
if trusted), and (8) either the authentication name or the fourth field from
authentication secrets file if a trusted IP address. Under Windows there is an
additional command line argument: (1) is the file descriptor which must be
used instead of stdout, (2) is the full address, etc.

For valid local mailbox addresses the verifier is expected to write two lines to
the standard output -- the full name associated with the mailbox, and the
canonical mailbox name -- and then exit with a value of zero. For valid non-local
addresses the first line of output should be empty, the second line should be
copied from the first command-line argument, and the exit value should be one.
For invalid addresses the exit value should be greater than one, and anything
written to the standard output is taken as the failure reason. (Only the few
few thousand characters are read from the verifier's standard output stream;
any more is thrown away.)

In this simple example script all addresses are accepted as long as they contain
an at sign. This is equivalent to removing the "--postmaster" command-line
switch:

	#!/bin/sh
	# verifier.sh
	address="${1}"
	user="${2}"
	host="${3}"
	if test "${address}" != "${user}@${host}" ; then exit 2 ; fi
	echo "${address}"
	echo "${address}" # again
	exit 1 # accept

As another example, this verifier script accepts all recipient addresses by
default, but rejects remote addresses if the client has bypassed authentication
by connecting on a trusted IP address:

	#!/bin/sh
	# verifier.sh
	host="$3"
	local_domain="$4"
	auth_mechanism="$7"
	if test "${auth_mechanism}" = "NONE" -a "${host}" != "${local_domain}"
	then
		echo "cannot relay without authentication"
		exit 2 # reject the recipient address
	fi
	echo "${address}"
	echo "${address}" # again
	exit 1 # accept the recipient address

If this verifier script is used with a suitable "--server-auth" file then it can
be used to prevent open relay without restricting authenticated clients.

Security issues
---------------
A significant security concern is the use of external mail pre-processors and
address verifiers (using the "--filter" and "--verifier" switches), and so the
following precautions are taken:

# effective userid

  Suid privileges are revoked at start-up, switching the effective userid/groupid
  to be the real userid/groupid values. If started as "root" then the effective
  userid/groupid are taken from the "daemon" user. Special privileges are only
  reclaimed when needed to bind sockets and do file i/o. Normally this means
  temporarily switching the userid and groupid back to what they were at start-up.
  However, when writing spool files only the effective userid is changed, not the
  groupid.

# execution environment

  The mail pre-processor runs with an almost empty set of environment variables
  ("PATH" and "IFS"), and with no open file descriptors other than "stdin" and
  "stderr" open onto "/dev/null", and "stdout" open onto a pipe.

# configuration

  The mail pre-processor filename has to be configured using a full path, so there
  is no dependence on the current working directory or the PATH environment variable.

Security issues which relate to the SMTP protocol itself are beyond the scope of
this document, but RFC2821 makes the following observation: "SMTP mail is
inherently insecure in that it is feasible for even [..] casual users to [..]
create messages that will trick a [..] recipient into believing that they came
from somewhere else. [..] Real [..] security lies [..] in end-to-end methods [..]
such as those which use digital signatures."

The "Authentication" section below also relates to security.

Some other points are:
* The program runs for most of the time with a "umask" of 177, switching to 117 when creating spool files.
* Strings are dynamically allocated, so buffer overflow/truncation issues are avoided.
* By default connections to the SMTP and administrative ports will be rejected if they come from remote machines.
* No configuration parameters can be changed through the administrative interface.
* No exec(), system() or popen() calls are used other than execve() to spawn the mail pre-processor and/or address verifier.
* The submit utility is installed as set-group-id with group ownership of "daemon".

Authentication
--------------
E-MailRelay supports the ESMTP "AUTH" extension, as defined in RFC2554, on both
the server-side and client-side. The only authentication mechanisms currently
provided are the non-standard but widely used "LOGIN" mechanism, which uses
plaintext passwords, and the "CRAM-MD5" mechanism, as defined in RFC2195.

Authentication is enabled with the "--auth-client" and "--auth-server"
command-line switches. The switch parameter is the name of a "secrets" file,
containing usernames and passwords.

The secrets file has a line-based format: blank lines are ignored and the hash
character (#) is used for comments. Lines have four white-space delimited
fields: "mechanism", "client-or-server", "userid", and "secret". The "mechanism"
field must be "LOGIN" or "CRAM-MD5" (case-insensitive); the "client-or-server"
field must be "client" or "server"; the "userid" field is xtext-encoded user
identifier; and the "secret" field is the xtext-encoded "LOGIN" password or
"CRAM-MD5" key.

The "xtext" encoding scheme is defined properly in RFC1891, but basically it
says that non alphanumeric characters should be represented in hexadecimal as
"+XX".

The client-side secrets file specified with "--auth-client" is used when
E-MailRelay talks to a remove server. The file should contain at least one
"LOGIN client" or "CRAM-MD5 client" entry.

A server-side secrets file specified with "--auth-server" is used when a remote
client tries to authenticate with E-MailRelay. The file should normally contain
several "LOGIN server" or "CRAM-MD5 server" entries.

The same secrets file may be specified for both "--auth-client" and
"--auth-server" switches.

The "CRAM-MD5" keys can be generated using the "emailrelay-passwd" utility.

As an example, the following secrets file defines "jsmith" as the username to be
used when E-MailRelay authenticates with a remote SMTP server, and defines two
usernames ("user1" and "user2") which can be used by clients when they
authenticate with the E-MailRelay server:

	#
	# emailrelay secrets file
	#
	LOGIN client jsmith my+20password
	LOGIN server user1 secret
	LOGIN server user2 e+3Dmc2

A "CRAM-MD5" version would look like this:

	#
	# emailrelay secrets file
	#
	CRAM-MD5 client jsmith 688498119.2977922305.1278051807.3015243256.2216875978.2833592318.2902375592.3156808220
	CRAM-MD5 server user1 4059553961.2316091643.3282746241.1444639637.3735501773.3404060330.2760590371.1201092398
	CRAM-MD5 server user2 2798539199.3144534242.3784876256.2879973305.2327113479.216533878.2436460291.2361831919

When using the LOGIN mechanism you have to store plaintext passwords in a file
and then send them unencypted over a network. This is a bad thing. You should at
least make sure that the secrets file has tight permissions, and that the
passwords in it are not also used for anything important (such as root access).

On the server side authentication is advertised by E-MailRealy in the response
to the SMTP "EHLO" command if the "--auth-server" command-line switch is used.
Authentication by the client is mandatory unless the client's IP address is
configured as a trusted address. If the client does authenticate then the
authenticated user-id is stored with the message and then passed on to the next
SMTP server using an "AUTH=userid" parameter on the SMTP "MAIL FROM" command.

Trusted IP addresses are configured with lines in the secrets file having "NONE"
in the first field, "server" in the second field, a wildcarded IP address in
the third field, and an arbitrary keyword in the fourth field. The keyword
is passed to any external address verifier program specified by the "--verifier"
command-line switch.

For example this secrets file allows any client connecting from the 192.168.0.0
domain to connect without authentication desipte the "--auth-server" switch:

	#
	# emailrelay secrets file
	#
	NONE server 192.168.0.* localdomain
	LOGIN server user1 secret
	LOGIN server user2 e+3Dmc2

On the client side authentication is performed when E-MailRelay has connected to
a server which supports the AUTH extension with the LOGIN or CRAM-MD5 mechanism.
If client authentication is enabled (with the "--auth-client" switch) but the
server does not support the AUTH extension, or does not support the LOGIN or
CRAM-MD5 mechanism, then E-MailRelay will fail the first message and terminate
with an error message.

Note that some ISPs require separate POP/IMAP authentication before SMTP access
from a particular IP address is allowed. This type of POP-before-SMTP
authentication can be done outside the E-MailRelay system by POP/IMAP utilities
such as "fetchmail".

Application notes
-----------------
# *Mutt* [http://www.mutt.org]

  Mutt can be configured to use the E-MailRelay 'submit' program rather than
  "sendmail" with an entry in the ".muttrc" file like this:

	set sendmail="/usr/local/libexec/emailrelay-submit"

  or

	set sendmail="/usr/sbin/emailrelay-submit"

  This puts the messages in the E-MailRelay spool-directory, but note that in
  proxy mode they will be ignored unless you use "--poll". Or you could use
  "--admin" plus "flush"/"emailrelay-poke".

# *Spam Assassin* [http://spamassassin.taint.org]

  The E-MailRelay server can use Spam Assassin to mark potential spam by having
  a small shell script which calls spamassassin's "spamc" program installed
  as the E-MailRelay mail pre-processor (using the "--filter" command-line switch).

  Your "--filter" shell script would look something like this:

	#!/bin/sh
	tmp="/tmp/`basename $0`.$$.tmp"
	pre="dos2unix"
	post="unix2dos"
	${pre} < ${1} | spamc > ${tmp}
	if test $? -eq 0 ; then ${post} < ${tmp} > ${1} ; fi
	rm -f ${tmp} 2>/dev/null
	exit 0

  This just pipes the content file into "spamc", using "dos2unix" to remove
  carriage-returns. The output is saved in a temporary file, and then put back
  into the content file using "unix2dos" to restore the carriage-returns.

Files and directories
---------------------
Following a normal build from source, a "make install" puts files in the
following locations:
* /usr/local/libexec/emailrelay
* /usr/local/libexec/emailrelay-deliver.sh
* /usr/local/libexec/emailrelay-notify.sh
* /usr/local/libexec/emailrelay-poke
* /usr/local/libexec/emailrelay-process.sh
* /usr/local/libexec/emailrelay-resubmit.sh
* /usr/local/man/man1/emailrelay-passwd.1.gz
* /usr/local/man/man1/emailrelay-poke.1.gz
* /usr/local/man/man1/emailrelay-submit.1.gz
* /usr/local/man/man1/emailrelay.1.gz
* /usr/local/sbin/emailrelay
* /usr/local/sbin/emailrelay-passwd
* /usr/local/sbin/emailrelay-submit
* /usr/local/share/emailrelay/doc/NEWS
* /usr/local/share/emailrelay/doc/README
* /usr/local/share/emailrelay/doc/changelog.gz
* /usr/local/share/emailrelay/doc/changelog.html
* /usr/local/share/emailrelay/doc/developer.html
* /usr/local/share/emailrelay/doc/developer.txt
* /usr/local/share/emailrelay/doc/emailrelay.css
* /usr/local/share/emailrelay/doc/index.html
* /usr/local/share/emailrelay/doc/readme.html
* /usr/local/share/emailrelay/doc/reference.html
* /usr/local/share/emailrelay/doc/reference.txt
* /usr/local/share/emailrelay/doc/userguide.html
* /usr/local/share/emailrelay/doc/userguide.txt
* /usr/local/share/emailrelay/doc/windows.html
* /usr/local/share/emailrelay/doc/windows.txt
* /usr/local/var/spool/emailrelay/emailrelay.*.content
* /usr/local/var/spool/emailrelay/emailrelay.*.envelope

This directory structure is constrained by the GNU/"autoconf" conventions rather
than the Filesystem Hierarchy Standard (FHS).

To force FHS compliance you can use the "--enable-fhs" switch when running
"configure", as is done for the RPMs. This results in the following file
locations:
* /etc/init.d/emailrelay
* /usr/lib/emailrelay/emailrelay-poke
* /usr/sbin/emailrelay
* /usr/sbin/emailrelay-passwd
* /usr/sbin/emailrelay-submit
* /usr/share/doc/emailrelay/NEWS
* /usr/share/doc/emailrelay/README
* /usr/share/doc/emailrelay/changelog.gz
* /usr/share/doc/emailrelay/changelog.html
* /usr/share/doc/emailrelay/developer.html
* /usr/share/doc/emailrelay/developer.txt
* /usr/share/doc/emailrelay/emailrelay.css
* /usr/share/doc/emailrelay/examples/emailrelay-deliver.sh
* /usr/share/doc/emailrelay/examples/emailrelay-notify.sh
* /usr/share/doc/emailrelay/examples/emailrelay-process.sh
* /usr/share/doc/emailrelay/examples/emailrelay-resubmit.sh
* /usr/share/doc/emailrelay/index.html
* /usr/share/doc/emailrelay/readme.html
* /usr/share/doc/emailrelay/reference.html
* /usr/share/doc/emailrelay/reference.txt
* /usr/share/doc/emailrelay/userguide.html
* /usr/share/doc/emailrelay/userguide.txt
* /usr/share/doc/emailrelay/windows.html
* /usr/share/doc/emailrelay/windows.txt
* /usr/share/man/man1/emailrelay-passwd.1.gz
* /usr/share/man/man1/emailrelay-poke.1.gz
* /usr/share/man/man1/emailrelay-submit.1.gz
* /usr/share/man/man1/emailrelay.1.gz
* /var/spool/emailrelay/emailrelay.*.content
* /var/spool/emailrelay/emailrelay.*.envelope

For finer control of the directory structure the following variables
can be specified on the "configure" command-line (but note that the
"--enable-fhs" will override them):
* e_sbindir
* e_libexecdir
* e_docdir
* e_initdir
* e_spooldir
* e_man1dir
* e_examplesdir

For example, running "./configure --prefix=/usr e_spooldir=/tmp/spool" will
create the GNU-style directory structure under "/usr" rather than "/usr/local",
and create the E-MailRelay spool directory as "/tmp/spool" rather than
"/usr/local/var/spool/emailrelay".

The default spool directory path which is built into the executables and scripts
comes from "configure", via the makefiles. So after running "configure" with a
different spool directory do a "make clean" in at least "src/main" and "bin".

Even though the "--enable-fhs" switch overrides all other directory specifiers
during the build process, it is still possible to change the installation
root directory using "make install DESTDIR=<root>" or "DESTDIR=<root> make -e install".
However this will not affect the default spool directory path built into the
scripts and executables; the spool directory path will have to be explicitly
defined at run-time.

For example, building with "configure --enable-fhs ; make ; make install DESTDIR=/export"
would create a spool directory "/export/var/spool/emailrelay", but the server
run as "/export/usr/sbin/emailrelay --as-server" would expect a spool directory of
"/var/spool/emailrelay". The fix is to add "--spool-dir /export/var/spool/emailrelay"
to the emailrelay command-line.

The "emailrelay" start/stop script in "init.d" creates a pid file in "/var/run"
if the directory exists, or in "/tmp" otherwise.




Copyright (C) 2001-2003 Graeme Walker <graeme_walker@users.sourceforge.net>. All rights reserved.