In the UNIX environment, traditional mail applications have not used any network protocol at all. They have instead accessed the locally stored mailbox files directly through the file system. Typically, the inbox of each user is stored in a single file in either the /var/mail or the /var/spool/mail directory with the same name as that of the user (for example, /var/spool/mail/joe). The focus of this book is to discuss Linux-based e-mail solutions for the small office where users do not wish to log on to a central server with a terminal application in order to access their mail, so local mail storage will only be covered briefly on a need-to-know basis.

The most important protocol in Internet mailing is the Simple Mail Transfer Protocol (SMTP). Its simple purpose is to transport e-mail messages between two computers. These computers may either both be servers, or one of them may be a client machine on which the user runs the mail application—Outlook, Thunderbird, Eudora, or whatever. To collect new messages, the end user does not utilize SMTP. This is where the Post Office Protocol (POP), and the Internet Message Access Protocol, (IMAP), come in.

Some proprietary systems such as Microsoft Exchange and Lotus Notes use their own protocols to access messages, and we will not discuss them here.

POP is the older and more widely used protocol of the two. It focuses on giving the users access to their inboxes, from which the users can download the new messages to their local computers and then delete them from the server. POP servers are not meant to be used for permanent storage of messages and the POP services of some Internet providers even prohibit users from leaving messages on the server after they have been downloaded once. The chief disadvantage of POP is that it only provides access to an intermediary storage medium and the users must store their messages permanently someplace else (for example, on their local hard drives). This is not only impractical for users who want to access their e-mail messages from multiple locations, but it is also a hassle for the system administrator who has to implement a backup solution for the users' messages. POP also does not have any notion of providing multiple folders for every user; with POP a user can access his or her inbox, period.

IMAP is meant as an access method to a first-class mailstore, that is, it is designed to allow the user to store the messages permanently on the server. This solves the system administrator's backup problem and allows the user to access all messages from any place in the world (firewall restrictions aside). IMAP also has a more widespread implementation of TLS-secured connections, making IMAP safe to use in hostile environments. To improve performance and allow users to work with their mailboxes while not being connected to the mail server, most mail applications with IMAP support cache the downloaded mailboxes and messages in the local hard drive.

SMTP is a line-oriented text protocol that runs over TCP, which makes it trivial to decode SMTP transcripts and to initiate SMTP sessions using the regular Telnet client found on just about any computer. An SMTP client starts a session by connecting to port 25 on the SMTP server. After the server has greeted the client, the client must respond by saying hello, or actually HELO or EHLO, followed by the client's hostname. If the server accepts the cordial greeting, the client may begin the first mail transaction.

An SMTP mail transaction consists of three parts: a sender, one or more recipients, and the actual message contents. The sender is specified with the MAIL FROM command, each recipient with an RCPT TO command, and the start of the message contents with a DATA command. If the server accepts the message then the client may continue with additional transactions or issue the QUIT command to terminate the SMTP session.

Let us be less abstract and look at an actual SMTP session to give you a better idea. The bold face print represents what the client sends to the server.

220 mail.example.com ESMTP Postfix (2.2.3)

EHLO gw.example.net

250-mail.example.com
250-PIPELINING
250-SIZE
250-VRFY
250-ETRN
250 8BITMIME

MAIL FROM:<[email protected]> SIZE=112
250 Ok
RCPT TO:<[email protected]>
250 Ok
RCPT TO:<[email protected]>
250 Ok
RCPT TO:<[email protected]>

550 <[email protected]>: Recipient address rejected: User unknown in local recipient table

DATA
354 End data with <CR><LF>.<CR><LF>

Subject: Test mailTo: <[email protected]>Date: Sun, 15 May 2005 20:23:22 +0200 (CEST)This 
is a test message..

250 Ok: queued as B059D3C2B

QUIT
221 Bye

This example shows a host that claims to be named gw.example.net connects to an SMTP server that calls itself mail.example.com. Because the server's first response contains ESMTP, the client decides to try Enhanced SMTP (ESMTP) and greets the server with EHLO instead of HELO. The server accepts this greeting and responds with a list of the supported ESMTP extensions.

Together with the sender address, the client sends the SIZE attribute to indicate the size of the message to the server. This is allowed because the server has stated that it supports the SIZE extension. If the size specified by the client exceeds the message size limit set by the server, the message can be rejected at once rather than after the whole message has been received and the server can assess the size.

An SMTP message can obviously have more than one recipient. This has a few consequences that must be remembered while implementing a mail system and inventing policies. In the example above, the mail server accepts the first two recipients but rejects the third one. Since two recipients have been accepted by the server, the client will try to send the message contents. Here the message is accepted by the server and queued for delivery (250 Ok: queued as B059D3C2B), which means that the SMTP server has taken over the responsibility for the delivery of the message to the accepted recipients. If the message cannot be delivered, the server will send a non-delivery message (bounce) back to the sender. The server could also have chosen to reject the whole message. If so, it would have rejected it for all recipients and not delivered it at all. In other words, in response to the message contents the server must either reject the message for all recipients or accept it for all recipients.

It is vital to understand the difference between the envelope and the header. The envelope of a message consists of the information given in the MAIL FROM and RCPT TO commands, that is, the sender and recipient information that is used to deliver the message. An SMTP server pays no attention what so ever to the From, To, and Cc message headers. In our example the To header contains just a single address with no other relation to the actual recipient addresses than the domain, but that is just a coincidence. Bounces are always sent to the envelope sender address, in this case [email protected]. The sender address of bounce messages is the empty sender address, often called the null sender. However tempting it may be for some people, the null sender address must not be blocked.

So far we have not commented on the numerical codes given by the server at the beginning of each line. Each number has a specific meaning and it is important to learn the correct interpretation of the first digit.

In SMTP, error conditions can be either temporary or permanent. Both 4 and 5 are used to signal errors. A client that receives a temporary error designated by 4 should disconnect, keep the message in the queue, and retry at a later time. Typical temporary error conditions include a full mail queue disk, a server configuration error that must be resolved before messages can be accepted, or a temporary DNS lookup error. Permanent errors are indicated by the first digit being 5 and mean that the request will never be accepted, so a client will have to remove the message from the queue and send a bounce to the sender telling him or her that the message could not be delivered.

There is a lot more to SMTP than this quick introduction has covered, but this should get your started. A number of RFC documents cover SMTP related topics, but the probably most important ones are RFC 821 (Simple Mail Transfer Protocol) and RFC 822 (Standard for the format of ARPA Internet text messages).