Mail Queue

Courier's mail queue is a combination of certain filesystem directory structures and an set of several code modules. Each code module performs a specific, narrowly-defined, task. Code modules use FIFOs and pipes to communicate with each other.

Module diagram

Here's a somewhat crude diagram that illustrates the relationship between the modules:

 Input modules                                                 Output modules
 =============                                                 ==============

+-------------+                                               +--------------+
|  sendmail   |   -----\                               /----> | courierlocal |
+-------------+        |                               |      +--------------+
                       |                               |
+-------------+        |                               |      +--------------+
|courieresmtpd|   -----|    +------+     +----------+  |----> | courieresmtp |
+-------------+        |    |      |     |          |  |      +--------------+
                       +--->|submit| --> | courierd |--+
+-------------+        |    |      |     |          |  |      +--------------+
| courierdsn  |   -----/    +------+     +----------+  |----> |  courierdsn  |
+-------------+                                        |      +--------------+
                                                       |
                                                       |      +--------------+
                                                       \----> |  courieruucp |
                                                              +--------------+

The first set of modules are the "input modules". An input module receives the message from some unspecified external source. The "sendmail" module is designed to read the message directly from the system command line, or an external process. The sendmail module is also linked as "rmail". In the "rmail" incantation, sendmail reads mail from a UUCP gateway. The "courieresmtpd" module talks ESMTP to receive a message from a remote mail relay. The input module may or may not rewrite the headers of the E-mail message, before passing it to the submit module.

The submit module

The submit module is a uniform mechanism for adding a message to the mail queue. The permissions on submit are set so that it cannot be executed by an ordinary process, and is only accessible to the input modules. submit's tasks include:

• Verify the integrity of the message.
  
  
• Rewriting all E-mail addresses - the sender, the recipients, the addresses in the mail headers - into a canonical form.
  
  
• Run all recipient addresses through the system's alias database. Remove duplicate addresses from the recipient list.
  
  
• Run any site-defined mail filters to reject the message for all, or some recipients.
• Create a control file and a data file for the message.
  
  

Submit does a lot of work here. By design, as much time consuming work as possible is placed into the submit process. The system may receive many incoming messages at the same time, and multiple submit processes can be running concurrently, avoiding any major serializes bottlenecks where everything must be squeezed into. Only once the message is completely accepted by submit does it go into the global mail queue.

Each message in the mail queue consists of two files: control and data. The data file contains just the text of the actual message, as received and rewritten by submit. The control file contains all the associated metadata for the message. The metadata in the control files includes: the sender address and the recipient addresses; a complete record of all delivery attempts for each recipient, and a designation of which recipients have been delivered or permanently rejected; the times of any completed delivery attempts and the corresponding error messages; when another delivery attempt is scheduled, and the message's expiration time in the mail queue.

submit may create multiple control and data files if the message has a very large list of recipients, but usually there's just one. The control and data files are created in the directory $localstatedir/tmp/iiiii. "$localstatedir" is a variable that is set by Courier's configuration script, it defaults to /usr/lib/courier/var. "iiiii" represents the current system time, represented as seconds since the epoch, divided by 10,000. New "iiiii" subdirectories are created automatically. Older ones are automatically deleted.

submit does not have a permanent connection to the "courierd" process, which oversees the global mail queue. After submit creates the control and the data file, it sends a signal through a FIFO to the courierd process. If courierd process is not running, submit ignores the error. If Courier's main server processes are not running, incoming mail can continue to pile up in $localstatedir/tmp, and it will be processed when the server is started. There is a specific convention used for naming the files in $localstatedir/tmp are named. This convention allows incomplete control and data files - files for a message that has not yet been completely received by the input module - to be ignored. Any files older than 36 hours are automatically deleted.

Mail queue on disk

The mail queue consists of two directories: $localstatedir/msgs and $localstatedir/msgq. The control file is stored in $localstatedir/msgs/nnnn/Ciiiii, and the data file is $localstatedir/msgs/nnnn/Diiiiii. "iiiiii" is the inode number of the control file. Since inode numbers are always unique, the inode is a convenient way to obtain unique identifiers for each message in the mail queue. "nnnn" is the inode number hashed, the size of the hash is set by the configure script (100 is the default, so this is usually just the last two digits of the inode number).

One item of information that's stored in the control is the time of the next scheduled delivery attempt of this message, expressed as seconds this the epoch. There is also a hard link to the control file: $localstatedir/msgq/xxxx/Ciiiiii.tttttt. "tttttt" is the next scheduled delivery time of this message. "xxxx" is the time divided by 10,000. 10,000 seconds is approximately two and a half hours. Each subdirectory is created on demand. Once all delivery attempts, scheduled for the time range that's represented by each subdirectory, have been made, the empty subdirectory is deleted. If a message needs to be re-queued for another delivery attempt, later, the next scheduled delivery time is written into the control file, and it's hard link in $localstatedir/msgq is renamed.

This scheme comes into play when there is a large amount of mail backed up. When reading the mail queue, Courier doesn't need to read the contents of any directory that represents a time interval in the future. Also, Courier does not have to read the contents of all subdirectories that represent the current and previous time intervals, when it's falling behind and can't keep up with incoming mail. Courier does not cache the entire mail queue in memory. Courier needs to only cache the contents of the oldest one or two subdirectories, in order to begin working on the oldest messages in the mail queue.

Output modules

The courierd process manages the mail queue, and schedules messages for delivery. courierd only schedules the delivery attempts. The actual delivery attempts are implemented in the output modules.

courierlocal delivers mail to local mailboxes. courieresmtp delivers mail to remote mail relays, via ESMTP. courieruucp forwards message to a UUCP gateway. courierdsn is a hack that generates delivery status notifications (bounces). After a message is delivered, Courier checks if the message requested any delivery notifications (bounces or return receipts). If a delivery status notification is necessary, Courier dispatches the message to the courierdsn output module. courierdsn reads the control and the data file, and formats the delivery status notification message. The DSN is then injected into the mail queue by running submit. After submit takes the DSN, courierdsn then deletes the original message from the mail queue. If the original message did not need a DSN, courierd deletes the message by itself.

courierd

This process is the core mail server engine whose task is to make sure that messages are moving through the mail queue, smoothly. This process uses several different data structures in order to be able to achieve a nearly constant implementation complexity.

The message queue cache

"The message queue" refers to the portion of the mail queue that is cached in memory. Courier reads the mail queue, in chunks, into memory, then works with the message queue cache to schedule messages for delivery attempts, without going back to disk. The size of the message queue is computed when Courier initially starts. There is a separate "high watermark" and "low watermark" message queue size. The high watermark value is the fixed maximum size of the message queue. Courier reads the mail queue until it's read all of it, or until the message queue reached its high watermark size. As message delivery attempts conclude, the corresponding entries in the message queue are deleted. When the size of the message queue falls below the low watermark value, Courier goes back to the disks, and fills the mail queue up to the high watermark again.

Each output module has a corresponding configuration file: $sysconfdir/module.local, $sysconfdir/module.uucp, $sysconfdir/module.local, and $sysconfdir/module.dsn. $sysconfdir is set by Courier's configuration script, its default value is /usr/lib/courier/etc. Each module configuration file contains several settings. The MAXDELS value sets the maximum number of simultaneous delivery attempts that this module can handle. The default low and high watermark values are calculated based on the output modules' MAXDELS values.

The default low watermark value is the sum of all MAXDELS values. If the sum is less than 200, it is adjusted to 200.

The default high watermark value is twice the low watermark value, with a maximum of 1000 above the low watermark value.

The configuration file $sysconfdir/queuelo overrides the computed low watermark value. The configuration file $sysconfdir/queuehi overrides the computer high watermark value. These configuration files can be used to manually tune Courier's performance.

Courier reads the mail queue when it starts, and whenever the message queue falls below the low watermark value. Courier begins with the mail queue subdirectory for the oldest time interval. Courier doesn't stop reading the subdirectory when the message queue goes over the high watermark. That's because files can be listed in a directory in any arbitrary order, therefore Courier continues to read the rest of the mail queue subdirectory, to pick up messages that have an earlier scheduled delivery time than (some of) the messages in the message queue. Each time Courier finds a message with an earlier delivery time than the message with the message queue with the most latest delivery time, the message with the most latest delivery time is evicted from the message queue, to make room for the new message. Note that the files in the mail queue subdirectory have both their unique message queue id number (their inode number), and their scheduled delivery time, encoded into the filename. Therefore, can ignore the messages that are already in the message queue, and messages with a delivery time that's after the latest delivery time in the message queue (if the message queue is at the high watermark).

courierd receives a signal (via a FIFO) whenever submit receives a new message. When courierd starts, or when it receives the signal, courierd will read $localstatedir/tmp, and move any new messages it finds into $localstatedir/msgs and $localstatedir/msgq. Afterwards, the message is added to the message queue cache, subject to the same exact condition. The new message will be added if the message queue is below its high watermark size, or if another message with a later scheduled delivery time can be evicted from the message queue.

The wait queue and the pending queue

The message queue is further subdivided into two parts - the wait queue and the pending queue. At any time the message queue may contain messages whose delivery times are in the past, or in the future. A message that's added to the message queue goes into the pending queue part if its next scheduled delivery time has already been passed. Otherwise, it goes into the wait queue, and when its scheduled delivery time comes up, it goes into the pending queue.

Moving a message into the pending queue is a fairly involved process. The message's control file is read to obtain a list of the recipients that haven't been delivered to, yet. A recipient address is stored in a canonical form usually "user@domain" (or a UUCP bang-path). Each recipient address is then decomposed into three parts: driver, host, and address.

Each output module (courierlocal, courieresmtp, courieruucp) has a function that's linked directly into the courierd process. This function is called a "rewrite function". It's job is to rewrite an address into its three components: driver, host, and address. The rewrite functions come from the different output module, but they are physically present in the courierd process. A rewrite function may fail to rewrite an address, meaning that this output module simply cannot deliver mail to this particular address. For example, courierlocal's rewrite function will only accept addresses of local mailboxes. The courierd process calls the output modules' rewrite functions, until one of them returns the host and the address component, then the driver component is set to the corresponding output module. If none of the rewrite functions accept the address, the message's control file is updated to mark that recipient as undeliverable. Every recipient (except those recipients to whom the message has already been delivered) is rewritten in this fashion.

The PRIORITY setting in a module's configuration file determines the order in which the rewrite functions are called. The actual PRIORITY value is set by Courier's configuration script, and appears in the module configuration file for informative purposes only.

courierd does not place any literal interpretation on the host or the address component of a rewritten address. The output's rewrite function derives these components, and the output module (the driver) receives this information when it is time to deliver the message to the recipient, so the output module also interprets their meaning. Just as an example: the courieresmtpd output module sets the "host" value to the domain portion of the address, and "address" to be the full address, "user@domain".

However, courierd does use the host component in scheduling delivery attempts. After decomposing all the recipient addresses into their individual driver, host, and address components, courierd groups together all addresses with the same driver and host values, on the presumptions that these addresses can be handled by a single delivery attempt.

The configuration file of each output module contains a MAXRCPT setting. This is the maximum number of recipients in a single delivery attempt. courierd will not create larger groups of addresses, for a single delivery attempt. If necessary, courierd will create multiple delivery attempts for the same driver and host. Some modules (such as courieresmtpd), allow MAXRCPT to be adjusted. Some modules do not. The courierlocal must have MAXRCPT set to 1. Courier will fail to start if courierlocal's MAXRCPT is anything else.

The conclusion of the rewriting phase is a list of one or more "delivery attempts". A delivery attempt consists of a driver, a host, and one or more addresses. courierd will then attempt to start the delivery attempt.

The configuration file for each output module contains two values, MAXDELS and MAXHOST. MAXDELS is the maximum number of outstanding delivery attempts for this driver. An outstanding delivery attempt is a delivery attempt that has been sent to the output module, for processing, but which has not yet finished. MAXHOST is the maximum number of outstanding delivery attempts with the same "host" value (as returned by the output module's rewrite function). Both MAXDELS and MAXHOST can be adjusted, subject to the resource limits of the operating system. After adjusting MAXDELS, MAXHOST (and MAXRCPT, if you know what you're doing), the command courier restart must be executed for Courier to restart and reconfigure itself, appropriately.

A delivery attempt is immediately sent to the output module provided that the total number of outstanding delivery attempts is less than MAXDELS, and the total number of outstanding delivery attempts for the same host is less than MAXHOST. Otherwise, the delivery attempt is placed into the "pending delivery attempt" queue.

When the output module finishes a delivery attempt, it updates the message's control file with the status of each recipient (accepted, rejected, postponed). The delivery attempt then comes back to courierd. The first thing that happens is that courierd looks at the pending delivery attempt queue, in order to check if it's possible now to send any pending delivery attempt to the output module. The pending delivery attempt queue is sorted by the "host" value. Additionally, it is maintained in the MRU order - most recently used. When a delivery attempt comes back to courierd, any other pending delivery attempts for the same host are moved to the head of the pending delivery attempt queue, so that they come up first. The assumption is that a preference for most recently seen hosts will translate to less setup overhead in the output module. One such example is the courieresmtp output module. After delivering the message to the remote mail relay the connection is kept open for a short period of time. If courieresmtp receives another delivery attempt to the same host, the message will be sent using the existing connection, instead of creating a new one. This can result in substantial savings, if the connection used authentication or encryption.

After moving the completed host's pending deliveries to the head of the queue, the host at the tail end of the pending queue is also moved to the head of the pending delivery attempt queue, in order to prevent a "starvation" situation. With a classical most-recently-used implementation, a non-stop stream of mail to the same host can "starve out" mail to other hosts. The same host will be constantly moved to the beginning of the pending delivery attempt queue, and mail to other hosts will starve at the end of the pending queue. Therefore, the oldest pending delivery attempt is artificially raised to the head of the pending delivery attempt queue, each time another host gets up there on its own merits. The end result is a balance between a preference for most recently seen hosts and starved hosts.

After finishing with the pending delivery attempt queue, courierd then checks if the completed delivery attempt was the last outstanding delivery attempt for its message. A message to multiple recipients will usually have more than one delivery attempt. If a message has any other outstanding delivery attempts, nothing else needs to be done. If all delivery attempts has been completed, the message's control file is read to determine if all recipients have been delivered to (or rejected).

If some recipients haven't been delivered to (their delivery was postponed due to a transient delivery error), the message is simply evicted from the message queue. If the message has been completely delivered (or if it's time in the mail queue has expired), the control file is also consulted to check if the message will require a delivery status notification (a bounce, or a return receipt). If a DSN is not required, the message is deleted from the mail queue. Otherwise, a fake delivery attempt is created for the courierdsn output module. This delivery attempt gets queued up the same way as bone-fide delivery attempts. When a completed delivery attempt comes back from the courierdsn driver, courierd knows that it simply needs to be tossed away, without looking at the control file again (the courierdsn module takes care of deleting the message from the mail queue).

Delayed DSNs are handled similarly. After the last outstanding delivery attempt for a message has been completed, but the message has recipients that haven't been delivered to yet, courierd checks the control file to determine if the message was queued up long enough, and a delayed DSN has not been sent yet. If the conditions are met, a dummy delivery attempt to courierdsn is prepared, dispatched, and the control file is updated accordingly. When the courierdsn delivery attempt comes back, the message is evicted from the message queue.

Before evicting the message that has not been fully delivered, courierd calculates the next scheduled delivery attempt. The control file is updated, and the link in $localstatedir/msgq is renamed accordingly. After evicting the message from the message queue, the mail queue is rescanned if the message queue is now below the low watermark size. Unless there's a mail back-up, this message will simply go back into the message queue with its new scheduled delivery time.

Control File Format

Courier is transport neutral. All that the main scheduling and processing engine knows is that messages originate from one of several defined 'transports', such as SMTP, UUCP, or the local Unix shell command line, and that messages eventually should end up being delivered to another transport. Each defined transport mechanism has an 'input' module, which submits messages to Courier, and an 'output' module, which delivers messages from Courier (see the first half of this document for details). Courier doesn't care much about the input module, all that the input module needs to do is to run Courier's submit program to put a message into Courier's scheduling queue. However, Courier does care about the output module, which is started by Courier, and communicates with Courier via pipes in a specific protocol. Courier tells the output module which message to send, and to which addresses. The output module is responsible for fetching the message from the message queue, and attempting to deliver it. When done, the output module is responsible for updating the control file in the message queue, and informing Courier that the delivery attempt has been complete.

In the message file, all headers appear in the 'canonical' format. Different transports may use different address conventions. E-mail addresses from all transports are converted to a standard, canonical format. E-mail addresses on messages dispatched to a transport provider are rewritten according to the transport module's conventions. The 'canonical' format is the one that is used by E-mail messages that are to be delivered to local mailboxes, so that when Courier delivers a local message, certain optimizations may be possible. The headers are rewritten from the 'canonical' format when the message is sent to a non-canonical address.� This is the responsibility of the output module.

There is a corresponding canonical convention format for message envelope sender and recipients, which are rewritten to the canonical format when a message is entered into the queue.

Control file

Control records

The 's' record specifies the message envelope sender. It may be null. The 's' record is always the first record in the control file.

rxxxxxxxxxx

The 'r' record specifies the message recipient. The first 'r' record is recipient #0, the next one is recipient #1, etc...

Rxxxxxxxxxx

The 'R' record specifies the "original message recipient", as defined by RFC1891 (Delivery Status Notification). The first 'R' record is original recipient #0, the next one is original recipient #1, etc... An empty R record is used to designate that the original message recipient was not specified. The contents of the R record are in the encoded form.

Nsss

The 'N' record specifies any delivery status notifications to be sent in regards to this recipient. The contents of the N record are zero or more of the following characters: 'S' - notify upon succesfull delivery, 'F' - notify upon failed delivery, 'D' - notify upon delayed delivery, 'N' - do not send DSNs for any reason.. Empty contents can be interpreted as a default.

faddresstype; address

The "Received-From-MTA" record. This specifies what goes into this header for DSNs generated due to this message.

exxxxxxxxxxx

The envid of this message, as specified in RFC1891. If this record is missing, no ENVID was specified

Mxxxxxxxxxxx

The 'M' record contains this message's "message id."

ixxxxxxxxxxx

The 'i' record specifies the name used to authenticate the sender, if authentication was provided.

tx

'x' is either 'F' or 'H', specifying FULL or HDRS in the RET parameter that was given in the MAIL FROM command, as specified in RFC1891. If RET was not used, x is not specified.

Etttttt

Expiration time of this message. If undeliverable after that time, the message gets bounced. "tttttt" would be the time in seconds, as returned by the time() system call.

ptttttt

Expiration time of this message, for the fax module.

Wtttttt

When to send a warning message to the sender, if the message remains undeliverable at that time. "tttttt" is also in seconds.

w

The simple 'w' record indicates that a warning message has already been sent.

8

The simple '8' record indicates that the message contains 8bit data. This fact can be used by the output module for optimization.

In t xxxxxxx

'I' records contain diagnostic information that is reported for each delivery attempt to recipient #n. After each delivery attempt for recipient #n is made (whether the delivery was succesfull, unsuccessfull, or deferred), one or more I records are appended to the control file. In each I record, the recipient number immediately follows the I, followed by a single space, a diagnostic type, a space, then diagnostic text. Multiple I records may be appended for a single delivery attempt.

Diagnostic type 'P' - this record contains the address of the remote peer who was contacted for this delivery attempt.

Diagnostic type 'C' - this record contains a text describing an error that occured while trying to connect to a remote peer.

Diagnostic type 'S' - this record contains a message that was sent to the remote peer which caused an error.

Diagnostic type 'R' - one or more of these contains the possibly multiline response from the remote server. For failures, this will be an error message. For successfull deliveries, this will be the acknowledgement reply. The diagnostic type R records always follow the format of an SMTP reply message. The last line of a reply starts with three digits, followed by a space and arbitrary text. For multiline responses, the preceding lines of a reply starts with three digits, a dash, then arbitrary text.

Sn tttttt y
Fn tttttt
Dn tttttt

These control records indicate that for recipient #n, a delivery attempt succeeded - 'S', failed - 'F', or deferred - 'D', at time tttttt (in seconds, as returned by the time() system call. The 'y' parameter is optional, and can specify the following: 'r' - message relayed to a non-RFC1891-conforming relay, 'l' - message delivered locally. Absence of the 'y' parameter indicates that the message has been delivered to an RFC1891-conforming relay.

One or more S, F, or D records will always appear after I records for a given delivery attempt.

NOTE: when a message has multiple recipients, it is possible that multiple delivery attempts will conclude at about the same time. It is NOT guaranteed that all I records for a single delivery attempt will ALWAYS appear consecutively in the control file, and will IMMEDIATELY be followed by an S, F, or a D record.

Ctttttt

All delivery attempts for this message have been completed at the indicated time. Counting the number of C records tells you how many times Courier tried to deliver the message, and is used to calculate the exponential retry backoff interval.

V

Presence of this record indicates that the envelope sender address should be VERPed. Courier will always send a single copy of the message to each recipient, even if all the recipients go to the same module and host. For each recipient, a different sender will be used. If the message envelope sender is user@host, then if the recipient's address is foo@bar, the envelope sender will be set to user-foo=bar@host, on the message to the user. Please note that the VERP feature must be implemented by each individual output module. When one Courier server sends mail via ESMTP to another Courier server, only one copy of VERPed messages are sent.

vxxxxxxxxxxx

The 'v' record recordss the vhost argument given to submit, or the domain of authuser if no argument was given on the command line.

Xxxxxxxxxxxx

The 'X' record indicates that the message has been cancelled, and records the reason given for cancellation.

Axxxxxxxxxxx

The 'A' record indicates the timestamp of the next scheduled delivery attempt.

m

The simple 'm' record indicates that the message has 8-bit headers.

Lxxxxxxxxxxx

The 'L' record indicates a delay before the first delivery attempt, in the format of the queuetime configuration setting.

Uxxxxxxxxxxx

The 'U' record flags messages that use Courier's SECURITY virtual E-mail network extension.

uxxxxxxxxxxx

The 'u' record indicates the message source (local, smtp, authsmtp, etc).

T

The simple 'T' record indicates that backscatter should be suppressed.

Oxxxxxxxxxxx

'O' records indicate environment variable names and values saved by the submit process.

Queue directories

There are no files directly in $localstatedir/msgs, the only contents of $localstatedir/msgs are subdirectories, named 0, 1, 2, and so on, up to the hash value set (or defaulted to) by the configure.

The control file Cnnnnn, where nnnnn is the control file's inode number, can be found in $localstatedir/msgs/x, where 'x' is the remainder of nnnnn divided by HASHMAX.

Every control file also has a second link to it, in $localstatedir/msgq. The second link for the control file Cnnnnnn is called Cnnnnnn.tttttt, where tttttt is the next scheduled attempted delivery time, in seconds. Cnnnnnn.tttttt is located in $localstatedir/msgq/xxxxxx, where xxxxxx is tttttt without the last FOUR digits. What this does is put the control files to all messages scheduled to be delivered in the same 10000 second interval (slightly less than three hours), into the same directory. So when Courier needs to figure out which messages should be delivered next, it will not have to scan the entire message queue, just this directory. This should result in much better performance on very busy mailing list servers.

Adding messages to the queue

A) Submit is invoked with one argument - the name of the input module. This is used for header rewriting. submit opens all the shared libraries, and initializes them.

B) Submit reads one or more addresses from standard input. Each address comes on one line of text. The first address may be empty. The list of addresses ends when an empty line is read. The first address is the message envelope sender, the second and subsequent addresses are message envelope recipients.

C) The first address can be optional followed by a tab character, then optionally by F or H, representing the FULL or HDRS paramater to the MAIL FROM: command; another character; then the ENVID field as specified in RFC1891. The second and the subsequent addresses can be optionally followed by a tab character, then zero or more of the following letters: S, F, D, N; another tab character, and original recipient address, as specified by RFC1891.

D) For each address read, submit prints a reply on standard output, indicating if the address is accepted. If the envelope sender address is rejected, submit immediately terminates.� The addresses are verified as follows. The input module's shared library's rw_rewrite() function is called with mode argument set to RW_ENVSENDER or RW_ENVRECIPIENT. In addition, for recipients, submit actually attempts to locate the rewritten address's output module, by calling rw_rewritedel functions. Errors reported by any shared library function will cause this address to be rejected. If submit was invoked with command line arguments that request an address to be VRFYed and EXPNed, submit only carries out the recipient verification code. If submit was invoked with EXPNed, submit cheats, and attempts to read aliases.dat GDBM files (see below) to look up the mailing list recipients for the rewritten addresses, and prints those.

E) The reply follows the RFC822 SMTP reply format -- indicating acceptance, deferral, or permanent rejection. It can be a multiline response.

F) After the blank line that ends the list of envelope recipients, submit reads the message itself, from standard input, until end of file.

G) Somewhere along the process, the file $localstatedir/tmp/yyyyyyyy/tttttt.ppppp.hhhhhh is created. tttttt is the current system time as returned by time(). yyyyyyyy is the current system time without the last four digits. hhhhhhh is the return value of the gethostname() system call. ppppp is the process ID of the submit process. This guarantees that these filenames are unique even if multiple processes are submitting the same message. Due to the presence of the hostname, it is possible to EXPORT $localstatedir/tmp, PROVIDED THAT stat()/fstat() on the client returns the inode on the *server*. That is, a client running on a machine that has mounted this tmp directory will obtain inode numbers from the filesystem on the server exporting the directory. This allows multiple machines on the network submit mail into the server. Also, the contents of $sysconfdir/me may be used instead of gethostname(), but in this case every machine on the network must have their own distinct me! If submit is unable to create the file, it is possible that the yyyyyyyy sub directory hasn't been created yet. $localstatedir/tmp is NOT writeable by the mail group, only by the mail user, and submit only has the set-group-id bit set, not set-user-id. If submit can't create the file, it tries to fork and exec the $libexecdir/courier/submitmkdir utility, which is executable only by the mail or user group, and has the set-user-id bit set. submitmkdir merely creates the subdirectory, chmods it to group-writeable, and returns.

H) submit reads the batchsize configuration file, which specifies the maximum number of recipients per file. If the message has more recipients, submit does the following: it renames the file created in step F to Ciiiiii.0, where iiiiii is its inode number. Then, submit creates additional control files, as many as necessary to hold all the recipients, starting with Ciiiiii.1, then Ciiiiii.2, and so on.

I) The file $localstatedir/tmp/yyyyyyyy/Diiiiii is created, where iiiiii is the inode number of the file created in step F. The first line in the file will be a Received: header added by submit, showing its userid. submit then appends all remaining input to the file, which is then closed. The message follows the Received: header, and submit rewrites headers according to rewriting rules defined for the input module. Submit will not add a Received: header if its real userid is mail -- message is coming from a trusted input, which should add its own Received: header.

J) After receiving the entire message, submit consults the environment variable "MIME" to check if the message needs to be rewritten according to RFC2045. MIME will be set to: "7bit" to specify that 8bit or raw text must be converted to quoted-printable; "8bit" to specify that quoted-printable encoded text can be converted to 8bit; "none" to specify that absolutely no kind of RFC2045 rewriting must take place. If MIME is not set, the default behavior is to simply add any missing RFC2045 headers in order to specify the system's locale. If rewriting needs to be done, submit rewrites the message to Diiiiii.1, then renames the file back to Diiiiii.

K) If submit created additional control files in step G, Diiiiii is hard-linked to Miiiiii.1, then Miiiiii.2, etc...

L) The first control file is closed, and renamed to Cnnnnnn. submit tries to open $localstatedir/tmp/trigger for write only, with O_NDELAY set, write one null byte, and close it (silently ignoring any errors).

M) submit rejects messages longer than sizelimit bytes. If the environment variable SIZELIMIT is set, it overrides $sysconfdir/sizelimit.

N) submit implements sendmail-style aliasing using a GDBM database file, using $sysconfdir/aliases.dat. After calling the input module's rewrite function, with mode set to RW_ENVRECIPIENT, the end result is looked up in these files, to see if aliases are defined for this address. If not found, and the address's @domain is in locals, strip the domain, and try again. If we succeed, we replace the single recipient with its aliases.

O) Submit counts Received: headers.

P) After submit has received end of file, submit exits with 0 if the control file has been succesfully renamed, or non-0 if there was an error. Submit may also print an SMTP-style response code on standard output. Programs running submit must close the pipe they use to send the message, wait until standard output from submit is closed, saving the message in a buffer, then read submit's exit code.

Q) Additional environment variables: NOADDMSGID - if set, submit will not generate a Message-ID: header, if one is missing; NOADDDATE - if set, submit will not generate a Date: header, if one is missing.

R) Additional environment variables used only if their corresponding values are not specified as input to submit: DSNRET - contains FULL or HDRS, and DSNENVID, containing DSN envelope ID. DSNNOTIFY - contains NEVER, SUCCESS, FAIL, DELAY - specifies DSN conditions. See "C" for more details.

tmp directory cleanup, and front end processing

A) $localstatedir/tmp is read. Each time a C file is found, the inode number is divided by HASHMAX, to calculate 'x'.

B) Call rename($localstatedir/tmp/yyyyyyyy/Mnnnnnn,$localstatedir/msgs/x/Dnnnnnn). If the rename call fails, call mkdir(local/queue/x), then try the rename again. Execute the first rename call again. This automatically creates the hash directories, as needed.

C) Call rename($localstatedir/tmp/yyyyyyyy/Cnnnnnn,$localstatedir/msgs/x/Cnnnnnn), then link($localstatedir/msgs/x/Cnnnnnn, $localstatedir/msgq/y/Cnnnnnn.tttttt).

On a regular basis, $localstatedir/tmp is purged of all files more than two days old. More on that later.