Variable envelope return path

Last updated

Variable envelope return path (VERP) is a technique used by some electronic mailing list software to enable automatic detection and removal of undeliverable e-mail addresses. It works by using a different return path (also called "envelope sender") for each recipient of a message.

Contents

Motivation

Any long-lived mailing list eventually contains addresses that can't be reached. Addresses that were once valid can become unusable because the person receiving the mail switched to a different provider. In another scenario, the address may still exist but be abandoned, with unread mail accumulating until there is not enough room left to accept any more.

When a message is sent to a mailing list, the mailing list software re-sends it to all of the addresses on the list. The presence of invalid addresses in the list results in bounce messages being sent to the owner of the list. If the mailing list is small, the owner can read the bounce messages and manually remove the invalid addresses from the list. With a larger mailing list, this is a tedious, unpleasant job, so it is desirable to automate the process.

However, most bounce messages have historically been designed to be read by human users, not automatically handled by software. They all convey the same basic idea ("the message from X to Y could not be delivered because of reason Z") but with so many variations that it would be nearly impossible to write a program to reliably interpret the meaning of every bounce message. RFC 1894 (obsoleted by RFC 3464) defines a standard format to fix this problem, but support for the standard is far from universal. However, there are several common formats (e.g., RFC 3464, qmail's qsbmf, and Microsoft's DSN format for Exchange) that cover large proportion of bounces.

Microsoft Exchange can sometimes bounce a message without providing any indication of the address to which the original message was sent. When Exchange knows the intended recipient, but is not willing to accept email for them, it omits their address. If a message is sent to joe@example.com and the server knows that this is "Joe User", it will bounce the message saying that the message to "Joe User" could not be delivered, leaving out the joe@example.com address altogether. VERP is the only viable way to handle such bounces correctly.

How VERP solves the bounce handling problem

The hard part of bounce handling is matching up a bounce message with the undeliverable address that caused the bounce. If the mailing list software can see that a bounce resulted from an attempt to send a message to user@example.com, then it doesn't need to understand the rest of the information in the bounce. It can simply count how many messages were recently sent to user@example.com, and how many bounces resulted; and if the proportion of bounced messages is too high, the address is removed from the list.

While bounce message formats in general vary wildly, there is one aspect of a bounce message that is highly predictable: the address to which it will be sent. VERP takes full advantage of this. In a mailing list that uses VERP, a different sender address is used for each recipient.

The mailing list manager knows that it sent a message from X to Y, so if a bounce message is received at address X, it can only be because address Y was undeliverable, because nothing was sent from X to any other address. Thus the important information has been extracted from the bounce message, without any need to understand its contents, which means the person in charge of the list does not need to deal with it manually.

Origin

The first serious advocate of this solution, and the originator of the term VERP to describe it, was Daniel J. Bernstein, who first put the idea into practice in his qmail MTA and ezmlm mailing list manager. [1]

Example

Assume there is a mailing list called wikipedians@example.net and that an individual, bob@example.org has subscribed to it, but later on, Bob has left example.org, so his address is no longer valid. Consider what happens when someone sends a message to the list.

Without VERP

Without VERP, the mailing list manager might send a message with the following characteristics:

This would result in a bounce, generated by the MTA of either example.net or example.org, with the following characteristics:

The mailing list manager can't be expected to understand the contents of this bounce, and can't deduce anything from the recipient address because hundreds of other people besides Bob were also sent messages from wikipedians-owner@example.net.

With VERP

With VERP, the original message would be different:

The bounce, then, will be more useful:

From this bounce message the mailing list manager can deduce that a message to bob@example.org must have failed.

This example shows the simplest possible method of matching a VERP to a list subscriber: the entire recipient address is included within the return path, with the at sign replaced by an equals sign because a return path with two at signs would be invalid. Other encoding schemes are possible.

Software supporting VERP

Disadvantages

The use of VERP requires each message to be sent once for every recipient, instead of once to each receiving SMTP server. This is because of a limitation of SMTP, which allows multiple recipient addresses to be specified in a single transaction, but only one sender address. When there are many subscribers in the same domain, a mailing list that is not using VERP can combine multiple deliveries into a single transaction. It connects to the appropriate server for the domain, gives the single sender address, the recipient addresses, and then sends the message contents only once.

A mailing list using VERP, on the other hand, must send the entire message body repeatedly, which leads to an overall increase in bandwidth usage. This inefficiency is usually not considered a big problem, especially by qmail users, since qmail always sends messages once per recipient, even when VERP is not being used. Some packages mitigate the impact of VERP by applying it selectively, for example a mailing list manager might only use VERP on 1 in 10 mailings. This way you can gain much of VERP's tight bounce control and accurate feedback without incurring the processing and network overhead every time.

Another problem with VERP (and with any automatic bounce handling scheme) is that there are MTAs on the Internet that fail to follow basic SMTP standards. VERP depends on the recipients' MTAs following the rule that bounces are sent to the envelope sender. This has been a standard requirement since the dawn of SMTP in 1982 (see RFC 821), but still there are MTAs that get it wrong, usually by bouncing to the address in the From: header.

Systems that implement greylisting work fine with VERP if the envelope sender follows the above-mentioned format. However, some VERP implementations use message number or random key as part of VERP, which causes each post to the mailing list to be delayed unless the greylisting system treats "similar" sender addresses as being equivalent.

See also

Related Research Articles

<span class="mw-page-title-main">Email</span> Mail sent using electronic means

Electronic mail is a method of transmitting and receiving messages using electronic devices. It was conceived in the late–20th century as the digital version of, or counterpart to, mail. Email is a ubiquitous and very widely used communication medium; in current use, an email address is often treated as a basic and necessary part of many processes in business, commerce, government, education, entertainment, and other spheres of daily life in most countries.

The Simple Mail Transfer Protocol (SMTP) is an Internet standard communication protocol for electronic mail transmission. Mail servers and other message transfer agents use SMTP to send and receive mail messages. User-level email clients typically use SMTP only for sending messages to a mail server for relaying, and typically submit outgoing email to the mail server on port 587 or 465 per RFC 8314. For retrieving messages, IMAP is standard, but proprietary servers also often implement proprietary protocols, e.g., Exchange ActiveSync.

<span class="mw-page-title-main">Email client</span> Computer program used to access and manage a users email

An email client, email reader or, more formally, message user agent (MUA) or mail user agent is a computer program used to access and manage a user's email.

A mail exchanger record specifies the mail server responsible for accepting email messages on behalf of a domain name. It is a resource record in the Domain Name System (DNS). It is possible to configure several MX records, typically pointing to an array of mail servers for load balancing and redundancy.

qmail is a mail transfer agent (MTA) that runs on Unix. It was written, starting December 1995, by Daniel J. Bernstein as a more secure alternative to the popular Sendmail program. Originally license-free software, qmail's source code was later dedicated to the public domain by the author.

An email address identifies an email box to which messages are delivered. While early messaging systems used a variety of formats for addressing, today, email addresses follow a set of specific rules originally standardized by the Internet Engineering Task Force (IETF) in the 1980s, and updated by RFC 5322 and 6854. The term email address in this article refers to just the addr-spec in Section 3.4 of RFC 5322. The RFC defines address more broadly as either a mailbox or group. A mailbox value can be either a name-addr, which contains a display-name and addr-spec, or the more common addr-spec alone.

Various anti-spam techniques are used to prevent email spam.

Sender Policy Framework (SPF) is an email authentication method which ensures the sending mail server is authorized to originate mail from the email sender's domain. This authentication only applies to the email sender listed in the "envelope from" field during the initial SMTP connection. If the email is bounced, a message is sent to this address, and for downstream transmission it typically appears in the "Return-Path" header. To authenticate the email address which is actually visible to recipients on the "From:" line, other technologies such as DMARC must be used. Forgery of this address is known as email spoofing, and is often used in phishing and email spam.

Greylisting is a method of defending e-mail users against spam. A mail transfer agent (MTA) using greylisting will "temporarily reject" any email from a sender it does not recognize. If the mail is legitimate, the originating server will try again after a delay, and if sufficient time has elapsed, the email will be accepted.

A bounce message or just "bounce" is an automated message from an email system, informing the sender of a previous message that the message has not been delivered. The original message is said to have "bounced".

In email, a return receipt is an acknowledgment by the recipient's email client to the sender of receipt of an email message. What acknowledgment, if any, is sent by the recipient to the sender is dependent on the email software of the recipient.

Email authentication, or validation, is a collection of techniques aimed at providing verifiable information about the origin of email messages by validating the domain ownership of any message transfer agents (MTA) who participated in transferring and possibly modifying a message.

<span class="mw-page-title-main">Message submission agent</span>

A message submission agent (MSA), or mail submission agent, is a computer program or software agent that receives electronic mail messages from a mail user agent (MUA) and cooperates with a mail transfer agent (MTA) for delivery of the mail. It uses ESMTP, a variant of the Simple Mail Transfer Protocol (SMTP), as specified in RFC 6409.

The Sender Rewriting Scheme (SRS) is a scheme for bypassing the Sender Policy Framework's (SPF) methods of preventing forged sender addresses. Forging a sender address is also known as email spoofing.

In computing, Bounce Address Tag Validation (BATV) is a method, defined in an Internet Draft, for determining whether the bounce address specified in an E-mail message is valid. It is designed to reject backscatter, that is, bounce messages to forged return addresses.

<span class="mw-page-title-main">Callback verification</span> Technique used with SMTP to validate e-mail addresses

Callback verification, also known as callout verification or Sender Address Verification, is a technique used by SMTP software in order to validate e-mail addresses. The most common target of verification is the sender address from the message envelope. It is mostly used as an anti-spam measure.

Email forwarding generically refers to the operation of re-sending a previously delivered email to an email address to one or more different email addresses.

An email alias is simply a forwarding email address. The term alias expansion is sometimes used to indicate a specific mode of email forwarding, thereby implying a more generic meaning of the term email alias as an address that is forwarded in a simplistic fashion.

Backscatter is incorrectly automated bounce messages sent by mail servers, typically as a side effect of incoming spam.

A bounce address is an email address to which bounce messages are delivered. There are many variants of the name, none of them used universally, including return path, reverse path, envelope from, envelope sender, MAIL FROM, 5321-FROM, return address, From_, Errors-to, etc. It is not uncommon for a single document to use several of these names.

References

  1. ↑ D. J. Bernstein qmail, February 1, 1997
  2. ↑ "Handling bouncing e-mails". 17 May 2021.
  3. ↑ mlmmj mailing list manager
  4. ↑ "Email Delivery for IT Professionals. A MailChimp Guide" (PDF). Mailchimp.
  5. ↑ "Plesk VERP Support since Version 18.0.30".
  6. ↑ "Sendmail VERP ruleset - comp.mail.sendmail". Archived from the original on October 27, 2014.{{cite web}}: CS1 maint: unfit URL (link)
This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.