Google and Yahoo!, in an effort to reduce spam and phishing, both recently announced tighter security requirements for emails sent to the gmail.com and yahoo.com email domains. The major consequence of this is that anyone who manages an email service no longer has any excuses to not implement long-accepted best practices regarding email security.  Those best practices are now effectively mandatory if you wish for email emitted from your domain to reach the bulk of recipients. And it isn’t just email admins who have to worry about this. Anyone who is a customer of bulk-email services such as Constant Contact or Emma (or any other 3rd party service that might need to send email that seems to come from your domain) also has to deal with this.

The Problem At Hand

So what exactly is domain security, and what problem is it being used to address?

Simply put, “From” addresses on email are trivially simple to forge. In the absence of any sort of domain authentication protections being discussed here, anyone can craft an email that purports to come from ANY email address. The SMTP protocol itself provides no real protection against spoofing. It is up to the managers of each email domain to make sure that domain authentication protocols are being followed to guard against addresses in their domain being spoofed.

Domain Authentication Protocols

There are three main technologies that are widely available and supported for guarding against email spoofing:

• Sender Policy Framework (SPF)
• DomainKeys Identified Mail (DKIM)
• Domain-based Message Authentication, Reporting and Conformance (DMARC)

A fourth, more advanced domain security technology, Authenticated Received Chain (ARC), also exists, but it is not yet broadly supported by all email services.

Sender Policy Framework (SPF)

 

The Sender Policy Framework (SPF) is designed to verify that the IP address that a given message has come from is authorized to emit mail for the domain of the envelope sender address (as represented in the Return-Path, or in P1 identity itself if Return-Path is blank or absent). The owner of the sending domain publishes a TXT record in their DNS zone which enumerates the authorized sources.  Note that this entry can also include nested references to other SPF records.  However, the total number of DNS calls to retrieve the list of valid IP addresses cannot exceed 10 lookups (including nested lookups within INCLUDE directives). This limitation sometimes makes it impractical to add SPF information for all third-party services that need to send from a given email domain. (DKIM or authenticated relay are preferred solutions for such scenarios.)

When a mail service which supports SPF receives a message, it takes the “MAIL FROM” value from the beginning of the SMTP transaction and extracts the sending domain. DNS for that domain is queried to look for a TXT record containing SPF information. If the SMTP connection is not coming from a source listed in the SPF record, that message fails SPF and may be subject to quarantine or rejection, depending upon the settings in the final portion of the SPF record or the results obtained by subsequent DMARC checks. For example, if the SPF record ends with “-all” (indicating that a hard fail should be applied to non-matches), the message will be rejected, whereas “~all” would indicate that a soft fail mode is preferred, and the message will be passed along. The response can be further modified by DMARC results and how strictly the receiving system enforces domain authentication, as other anti-spoofing technologies can utilize the results of the SPF test.

Note that a TXT record in DNS has a maximum string length of 255 characters. However, multiple strings are allowed in a TXT record. The process which makes the SPF query is expected to concatenate multiple strings in a lookup result without inserting separators. If multiple strings are used, all but the last should end with a space.

It is important to note that SPF does have some limitations:

• SPF has a DNS lookup limit of 10 for resolving SPF records. Because of this, SPF should only be used for the principal sources of email for a given email domain.
• SPF is broken by forwards if the forwarding host does not change the envelope sender address.  This can be mitigated by using Sender Rewriting Scheme (SRS), which EXO now supports. The catch here is that using SRS can lead to SPF alignment failures in DMARC.
• Whereas SPF focuses on authenticating the validity of a message based upon the Envelope Sender address, the fact of the matter is that the Envelope Sender is not typically shown to the recipient, so it will not typically get forged, which somewhat limits SPF’s usefulness.

DomainKeys Identified Mail (DKIM)

DKIM is a mechanism for not only protecting against email spoofing but also for guarding against message tampering and manipulation. Any authorized sender for a given email address space (whether it is the owner of the address space or a third-party email provider) first crafts a public/private signing key pair and associates with that key pair a short descriptive string called a “selector”.  (Note that each authorized sending service can have its own selector, thus allowing some compartmentalization of this scheme among third-party providers.)  The public key and selector are given to the domain owner, who publishes them in a DKIM record in DNS.  When the provider sends out an email, they will then use the private key to sign the message and add the digital signature as a header in the message along with the selector string.  The receiving system will then use a DNS query to pull the public string and use that to validate the signature, thus validating the integrity of all content in the message prior to the DKIM signature header.

DKIM does have limitations:

• Note that the domain used to lookup the DKIM record is specified IN the DKIM-Signature header. It does not necessarily have to match the domain of the From address.  This situation is referred to as an alignment error. This does not invalidate the DKIM signature, but it does allow for scenarios in which malicious senders can send spoofed messages with valid DKIM signatures.
• DKIM signatures are sensitive to header and message body modification, which can be an issue with forwarding or with mailing lists. For example, messages forwarded through exchange can sometimes have headers modified in a semantically unimportant fashion, but which nevertheless invalidate the signatures. Likewise, it is not uncommon for mailing list servers to add prefixes to message subjects or to add footers, which can also invalidate the signature. A possible future workaround for these issues might be Authenticated Received Chain (ARC), but that technology is not yet widely implemented.

Domain-based Message Authentication, Reporting and Conformance (DMARC)

DMARC is an email authentication policy and reporting protocol which builds upon (and uses) SPF and DKIM. The owner of the email domain publishes a policy in DNS describing how recipient systems are expected to enforce DKIM and SPF on messages purporting to be from that domain.  DMARC policies can additionally enforce alignment of addresses (matching of P1 and P2 addresses) if desired. Furthermore, the DMARC policy can specify email addresses for reporting of failures to match policy.  This reporting feature can be particularly useful for operating DMARC in a test or evaluation mode. If either SPF or DKIM alignment checks are passed, then DMARC test is passed. Depending upon the directives found in the published DMARC DNS record, failures can result in reporting and/or actions taken on the message.

The receiving system will perform DKIM and SPF checks on the incoming message but will additionally use the From address to determine what TXT record to retrieve from DNS to pull the relevant DMARC policy. This DNS record will have a name in the form “_dmarc.<domain>.” If a valid DMARC record is retrieved, the receiving system is then expected to follow its directives regarding how to handle SPF, DKIM, or alignment failures, and provide individual message reports or periodic aggregated reports if so specified in the policy.

DMARC Dependence Upon SPF and DKIM

Quite simply, a message passes DMARC if it passes either SPF or DKIM checks. The DMARC policy simply addresses what should be done with the message if the message passes neither SPF nor DKIM.

Alignment

Alignment refers to whether or not the domain name evaluated by SPF or DKIM matches the domain name of the From: address. Keep in mind that SPF always acts upon the domain name of the Envelope Sender address (which may or may not be the same as the From: address). DKIM, depending upon how it is configured, might refer in its “d=” attribute to either domain, or yet a different one.

How Misalignment Can Be Abused By Malicious Actors

Suppose that a malicious individual has their own domain, myevil.org, and they have set up valid SPF and DKIM signing for mail originating from their servers with myevil.org email addresses used in the Envelope Sender address. They can spoof emails from a different domain, say gooddomain.org, putting addresses for that domain in the From: address of malicious messages. The recipient never sees the Envelope Sender address (unless they happen to look at the message headers for the Return-Path header), and believe that the incoming messages are from gooddomain.org.

It is up to the owners of gooddomain.org to make sure that their SPF, DKIM, and DMARC infrastructure is set up properly, and it is up to the email infrastructure of the receiving environment to make sure that alignment checks are performed on incoming email to spot and block such spoofing.

Composite Authentication (compauth)

Composite Authentication (or “compauth”) is a proprietary component of Exchange Online Protection (EOP) which is used to synthesize multiple domain authentication results, in conjunction with other signals that are not publicly documented, to add a further degree of domain authentication. In addition to DMARC failure, the absence of SPF/DKIM/DMARC validation can result in a message failing compauth. Furthermore, compauth enforces domain alignment regardless of DMARC settings. A compauth failure results in Exchange depositing the message into the Junk mail folder.

In the headers of a given message, compauth results can be seen in the Authentication-Results header, along with the results of SPF, DKIM, and DMARC. (DKIM signatures and SPF validation have their own headers, but the Authentication-Results header summarizes these.) Here is an example of such a header:

Authentication-Results: spf=pass (sender IP is 208.75.123.167)
 smtp.mailfrom=in.constantcontact.com; austin.utexas.edu; dkim=pass (signature
 was verified) header.d=auth.ccsend.com;austin.utexas.edu; dmarc=none
 action=none header.from=connect-community.org;compauth=fail reason=001

In this example, from an external source, SPF record relied on the domain of the Envelope Sender address, in.constantcontact.com, and the domain specified in the DKIM header was auth.ccsend.com, neither of which aligned with the domain of the From address, connect-community.org, resulting in a compauth failure, despite the message having passed SPF and DKIM checks, with the lack of a DMARC policy potentially contributing to the failure. (The reason code of 001 indicates spoofing of an external address.)

Here is an example of a 3rd party mailer being used to send messages on behalf of a utexas.edu address (with the details of the department obfuscated with a fake domain name):

Authentication-Results: spf=pass (sender IP is 198.207.147.238)
smtp.mailfrom=cvent-planner.com; mydept.utexas.edu; dkim=pass (signature
was verified) header.d=cvent-planner.com;mydept.utexas.edu; dmarc=none
action=none header.from=mydept.utexas.edu;compauth=fail reason=601

The reason code of 601 denotes an attempt at intra-org spoofing (meaning that the message was spoofing an address in one of our domains). Again, SPF and DKIM both passed, but there is again a lack of domain alignment (as well as the lack of a DMARC policy record).

Both of these examples fail compauth for the same reasons. Of course, only the second one is something that we can do anything about, or more specifically, the owners of the departmental domain in question can do something about it, as addressed in the next section.

Addressing Alignment Issues With 3rd-Party Mailers

In the previous example, not only does the owner of the departmental domain need to add a DMARC policy record to DNS, but they also need to properly configure the 3rd party mailer to perform DKIM signing against a domain that is in alignment with the from address. The precise details of how to do this will vary from vendor to vendor. Whatever the details that are specific to the 3rd-party vendor, the domain owner will need to be certain that the appropriate DKIM public key is published in their domain’s DNS.

Some bulk mail providers allow the reference to a public key to simply be a CNAME redirecting back to one of the vendor’s public keys. In other cases, a key pair will need to be generated (and the private key configured into the vendor’s system).