Email Encryption Options: TLS, MTA-STS, S/MIME and OME

Which method should be used to encrypt email? The limits of opportunistic TLS, setting up MTA-STS and TLS-RPT, S/MIME key management and using Purview Message Encryption.
Technical summary
Technology
STARTTLS, MTA-STS, TLS-RPT, DANE, S/MIME, Microsoft Purview Message Encryption, Exchange Online
Topic
Comparing email transport and content encryption methods
Level
Intermediate–advanced
Prerequisites
  • Access to DNS and web server management
  • Basic knowledge of SMTP and TLS
  • Exchange Online PowerShell (for OME)
In this article7 sections

When an auditor asks “are your emails encrypted?”, most teams say “yes, we have TLS”. The answer is incomplete. A TLS connection between servers doesn’t stop the message from sitting in the clear on the server and in the recipient’s mailbox; end-to-end encryption, on the other hand, makes scanning and archiving harder. This guide compares opportunistic TLS, MTA-STS and TLS-RPT, S/MIME and Microsoft Purview Message Encryption in a way that helps you decide which to choose in which scenario.

The difference between transport and content encryption

Email encryption is considered at two separate layers:

  • Transport encryption: The message is encrypted while traveling from one server to another (STARTTLS over SMTP). Each server receives the message in the clear, processes it and sends it on to the next over another encrypted connection. It protects against an attacker listening on the network; it doesn’t protect against the server administrator, a compromised mailbox or the wrong recipient.
  • Content encryption: The message itself is encrypted (S/MIME, OME). The message stays encrypted in the mailbox too and only the authorized recipient can open it. In some solutions, usage rights such as forwarding, printing or copying can also be restricted.

This distinction is decisive when interpreting compliance requirements and enterprise cybersecurity policies. If the policy says “emails containing personal data are sent encrypted” and all you have is TLS, the message sits in the clear in the recipient side’s storage. The general framework for encryption controls on data at rest and in transit is in the implementing encryption controls guide.

Opportunistic TLS and its limits

In SMTP, TLS is opportunistic by default (RFC 3207): if the other server announces the STARTTLS capability in its EHLO response, the sender switches to an encrypted connection; if not, it sends the message in plain text. Nobody is warned. There are three basic weaknesses:

  1. Downgrade attack: If an attacker on the path deletes the STARTTLS line from the EHLO response, the sending server silently falls back to plain text.
  2. The certificate isn’t validated: In opportunistic mode, servers usually don’t check the certificate’s validity or name match; someone in the middle can present their own certificate.
  3. The MX record isn’t validated: By altering the DNS response, the message can be redirected to the attacker’s server.

There are two standard ways to close these gaps: DANE (RFC 7672), which relies on DNSSEC, and MTA-STS (RFC 8461), which relies on a policy published over HTTPS. DANE requires DNSSEC both on your domain and on the domain of your MX servers. For organizations not using DNSSEC, MTA-STS is a more accessible starting point. Verify DANE support of the cloud mail service you use in the vendor’s current documentation.

Configuring MTA-STS and TLS-RPT

MTA-STS tells servers sending email to your domain: “Deliver to this domain only to these MXs, with TLS and a valid certificate.” It has three components.

1. DNS record. Announces the policy’s existence and version. Every time the policy file changes, the id value must be updated too; senders refresh their cache by looking at this value. This is the step most often skipped.

_mta-sts.example.com.  IN  TXT  "v=STSv1; id=20261110T090000;"

2. Policy file. Published at https://mta-sts.example.com/.well-known/mta-sts.txt with a valid certificate signed by a publicly trusted CA:

version: STSv1
mode: testing
mx: mail.example.com
mx: *.mail.protection.outlook.com
max_age: 604800

The mx lines must match the names in your MX records; a wildcard can only be used in the leftmost label. The Exchange Online pattern in the example is common, but verify the current name of your MX record from your own tenant. max_age is in seconds; RFC 8461 allows a value of at most 31557600 (about a year). The mode values are testing, enforce and none.

3. TLS-RPT record. The reporting mechanism in RFC 8460; sending servers report the TLS and policy errors they hit during delivery to you in JSON reports:

_smtp._tls.example.com.  IN  TXT  "v=TLSRPTv1; rua=mailto:[email protected]"

Implementation order: publish TLS-RPT, start the policy in testing mode, and once the reports show no more certificate name mismatches or unlisted MX errors, switch to enforce mode and update the id value. When changing MX provider, the order isn’t reversed: first the new MX is added to the policy file, then DNS changes. If you do it the other way around, senders who have cached the policy refuse delivery to the new server. Mail doesn’t arrive.

MTA-STS protects the inbound direction. If you want mandatory, validated TLS for email to a specific partner, it’s defined on the sending side. In Exchange Online, this can be set up with an outbound connector for the partner domain:

New-OutboundConnector -Name "Partner mandatory TLS" -ConnectorType Partner `
  -RecipientDomains partner.example -UseMXRecord $true `
  -TlsSettings DomainValidation -TlsDomain mail.partner.example

S/MIME: certificate and key management

S/MIME (current version RFC 8551) lets each user sign email with their X.509 certificate and private key, and encrypt it with the recipient’s public key. The signature proves the content hasn’t changed and the sender’s identity; encryption makes the message readable only by the recipient. Technically it’s strong. The operational load is heavy:

  • Certificate distribution: Each user needs a certificate that carries their email address in the SAN field and has the email protection usage. For external recipients, certificates from a publicly trusted CA are subject to the CA/Browser Forum’s S/MIME Baseline Requirements.
  • Recipient certificate: Encryption requires the recipient’s public certificate. With external recipients, an exchange of signed messages is usually needed first.
  • Key escrow and archive: If the encryption key is lost, old messages can’t be read. In corporate use, encryption keys should be securely escrowed, and the private keys of expired certificates should be kept. Signing keys, however, aren’t escrowed.
  • Loss of scanning: The encrypted body and attachments can’t be scanned at the gateway; malware and DLP controls shift to the endpoint.

The scenario where S/MIME fits is signed and encrypted communication with a limited number of partners you correspond with regularly and who have their own PKI. A picture we often run into in the field, however: S/MIME was enabled for all users, the key lifecycle wasn’t planned, and a few years later old messages can no longer be opened after a laptop replacement. Lose the key, lose the archive.

Microsoft Purview Message Encryption

Purview Message Encryption (OME) uses Microsoft 365’s Azure Rights Management infrastructure. Neither the user nor the recipient manages certificates. Microsoft 365 recipients open the message directly in their clients; other recipients are directed to a web portal through a notification link and authenticate with a one-time passcode or a supported account.

Common protection options are Encrypt-Only and Do Not Forward (forwarding, printing and copying restricted). The user can choose encryption themselves in Outlook, or it’s applied automatically with a mail flow rule. Test the IRM configuration before writing a rule; if licensing is off, the rule silently does nothing:

Get-IRMConfiguration | Select-Object AzureRMSLicensingEnabled, InternalLicensingEnabled
Test-IRMConfiguration -Sender [email protected] -Recipient [email protected]

New-TransportRule -Name "Outbound secure subject" -SentToScope NotInOrganization `
  -SubjectContainsWords "[secure]" -ApplyRMSTemplate "Encrypt"

The method of typing a keyword in the subject line depends on the user remembering, and it gets forgotten. A sturdier route is tying encryption to sensitive content detection: defining an encryption action for the email location with a Microsoft Purview DLP policy, or using sensitivity labels that include encryption. Advanced features such as message expiry, revocation and multiple branding templates (Advanced Message Encryption) require an additional license; verify the license coverage in Microsoft’s current service description.

The advantage of OME is that the encrypted message can remain subject to content inspection, eDiscovery and retention policies within the tenant. Its limitation shows up on the external recipient side: the portal experience feels unfamiliar, and replies are also written through the portal. A short note explaining to critical external recipients in advance how to use the portal keeps the first message from going unanswered.

A selection table by scenario

Scenario Recommended method Note
Protection against downgrade attacks on all inbound email MTA-STS + TLS-RPT DANE too, if DNSSEC is available
Contractual mandatory TLS with a specific partner Partner-specific validated TLS Both sides must configure it
Payroll, external sends containing personal data OME (automatic via DLP or label) No certificate needed at the recipient
Signed correspondence proving sender identity and integrity S/MIME signature A qualified certificate may be required for legal validity
End-to-end confidentiality with a limited number of partners who have their own PKI S/MIME encryption Scanning shifts to the endpoint
Blocking forwarding and printing of the message OME Do Not Forward or a label Doesn’t block screenshots

In most organizations, the sensible order is: first tighten the transport layer with MTA-STS and TLS-RPT, then identify sensitive data flows and tie OME to automatic rules, and open S/MIME only to limited groups with a real business need. Sender authentication is a separate layer of this structure. If DMARC enforcement isn’t complete, encryption does nothing against a forged sender. Configuring these controls together in a Microsoft 365 tenant can be handled within Microsoft 365 security, and the transport layer and sender authentication as part of an email security project. The reference for MTA-STS’s official definition is RFC 8461.

Frequently asked questions

If I publish MTA-STS, will my outbound email also go with mandatory TLS?

No. The MTA-STS policy you publish tells servers sending email to you how to deliver to your domain; in other words, it protects the inbound direction. Protection in the outbound direction depends on your own sending server reading and applying the other side’s MTA-STS policy. In services such as Exchange Online, the provider does this validation; if you use an on-premises MTA, you need to enable it separately.

Do emails encrypted with S/MIME go through DLP and malware scanning?

Gateways and cloud filters can’t decrypt end-to-end encrypted content; attachment scanning and content-based DLP rules can’t be applied to the encrypted body. Before rolling out S/MIME encryption widely, clarify endpoint protection and which users can encrypt.

How does a recipient who doesn’t use Microsoft 365 read an email encrypted with OME?

The recipient gets a notification containing a link, and the encrypted message opens in Microsoft’s web portal. Authentication is done with a one-time passcode or by signing in with a supported account. The recipient doesn’t need to install a special client or certificate.

For mandatory TLS, should you use MTA-STS or a partner-specific connector?

MTA-STS publishes a general policy to all servers sending to your domain and depends on the other side’s support. If contractually required mandatory TLS with a specific partner is needed, defining certificate-validated mandatory TLS for that domain on your own sending server is a more definite control. The two aren’t alternatives; they complement each other.

Support

Contact us for support or a discovery session on this topic.

Contact form
[email protected]

Eretek Cybersecurity

Firewall, VPN, endpoint security and threat prevention. We reduce the attack surface and measure and report the risks.

Knowledge Center

Other articles