Self-Hosted Mail Server

DNS and Deliverability Fundamentals

by John Stray on April 25th, 2026

Introduction If running a mail server is about building the system, then DNS and deliverability are about convincing the rest of the internet to trust it. This is the section where things either quietly work… or everything you built ends up in spam purgatory. No pressure. You can have a perfectly configured server, but without the right DNS re

If running a mail server is about building the system, then DNS and deliverability are about convincing the rest of the internet to trust it. This is the section where things either quietly work… or everything you built ends up in spam purgatory. No pressure.

You can have a perfectly configured server, but without the right DNS records and policies in place, your mail will be rejected, marked as spam, or silently dropped. Modern email providers are extremely strict, and for good reason—spam and abuse are constant problems.

In this section, we'll set up the core DNS records and explain how receiving servers evaluate your mail.

The Role of DNS in Email

DNS is how other mail servers find and verify your system. When someone sends you an email, their server performs a series of lookups:

  1. Find your domain's MX Record
  2. Resolve that to a hostname
  3. Resolve that hostname to aan IP address (A/AAAA record)
  4. Optionally verify reverse DNS (PTR)

Each of these steps must be correct and consistent. A typical setup looks like this:


example.com      -> MX -> mail.example.com
mail.example.com -> A  -> 1.2.3.4

MX Records (Mail Exchange)

MX records tell the world which server is responsible for receiving mail for you domain. Example:


example.com.  IN  MX  10 mail.example.com.

Key points:

  • The hostname must resolve to a valid IP address
  • Lower priority numbers are preferred (10 is higher priority than 20)
  • You can define multiple MX records for redundancy

For most setups, a single MX records is sufficient.

Personally, I use a single server setup (ie. no redundancy) as my mail server is only serving a small number of domains and addresses. If you were to run a self-hosted mail server in a business environment, redundancy probably wouldn't hurt here. I may write a guide at a later date on how to setup redundant mail servers and have them work together nicely. Stay tuned for that one.

A and AAAA Records

Your mail server hostname must resolve to an IP address:


mail.example.com.  IN  A     1.2.3.4
mail.example.com.  IN  AAAA  2001:1234:1234:1234::1

IPv6 is increasingly important, and some providers will prefer it if available. If your hosting provider or ISP has IPv6 available, i would recommend enabling it for your mail server.

SPF (Sender Policy Framework)

SPF tells receiving servers which systems are allowed to send mail on behalf of your domain. Example:


example.com.  IN  TXT  "v=spf1 mx -all"

This means:

  • "Allow servers listed in my MX records to send mail"
  • "Reject everything else"

Common mechanisms:

  • mx - allow your MX servers
  • ip4:x.x.x.x - allow a specific IP address
  • include: - allow third-party services
SPF Gotchas
  • SPF records must be a single TXT record (multiple records can break validation)
  • There is a 10 DNS lookup limit
  • ~all (softfail) vs -all (hardfail) changes enforcement

On initial setup, start with setting your SPF record to softfail. Once you are sure that email is being delivered successfully and not into a spam box, then upgrade the SPF policy to a hardfail (reject completely).

DKIM (Domain Keys Identified Mail)

DKIM

  • The message hasn't been altered
  • It was authorised by your domain

A DKIM record looks like this:


default._domainkey.example.com.  IN  TXT  "v=DKIM1; k=rsa; p=PUBLIC_KEY"

The public key is stored in DNS, while the private key stays on your server (used by Rspamd in our setup).

Why DKIM Matters

Without DKIM:

  • Your mail is more likely to be marked as spam
  • DMARC policies (covered next) won't pass properly

DMARC (Domain-based Message Authentication, Reporting & Conformance)

DMARC builds on SPF and DKIM to define a policy for how receiving servers should handle your mail. Example:


_dmarc.example.com. IN TXT "v=DMARC1; p=none; rua=mailto:postmaster@example.com"

Policies:

  • none - monitor only
  • quarantine - send suspicious mail to spam
  • reject - outright reject failing mail

Start with:

p=none

Then move to stricter policies once you’re confident everything is working.

DMARC Reporting

The rua field allows you to receive aggregate reports from other mail providers. These reports can help you:

  • Detect misconfigurations
  • Identify abuse or spoofing

DMARC Reports can be rather difficult to read. It's often easier to implement an external tool to assist with reading these reports. I currently use and suggest using: ValiMail.

 

How Receiving Servers Evaluate Your Mail

When your server sends an email, the receiving server will typically check:

  • Does the IP have a valid reverse DNS?
  • Does the hostname match?
  • Does SPF pass?
  • Does DKIM pass?
  • Does DMARC align?
  • What is the sender's reputation?

Failing one check isn't always fatal — but failing several usually is.

In our mail server setup, we will be configuring Postfix, in combination with Rspamd, to perform these same checks on email that arrives at our server from other servers.

Testing Your Setup

Once your DNS records are in place, it's important to test them. Useful tools include:

  • dig or nslookup for DNS queries
  • Online DNS testing tools: WhatsMyDNS, MXToolbox
  • Sending test emails to major providers (Gmail, Outlook, etc.)

Common Pitfalls

A few issues to watch for when things don't appear to be working:

  • Missing or incorrect PTR records
  • SPF records that exceed lookup limits or are incorrect
  • DKIM keys not matching
  • DNS changes not fully propagated
  • Using a domain with poor reputation

A Note on Reputation

Even with perfect configuration, deliverability isn't guaranteed. New servers often start with limited trust and improve over time. Factors that affect reputation include:

  • IP address history
  • Domain age
  • Sending patterns
  • Complaint Rates

Checkpoint

At this stage, you should have:

  • A valid MX record pointing to your mail server
  • A/AAAA records resolving correctly
  • Reverse DNS configured
  • SPF, DKIM and DMARC records in place

If all of these are correct, your server is in a strong position to send and receive mail reliably.

What's Next

Now that DNS and deliverability fundamentals are in place, we can start configuring the core mail transfer agent.

In the next section, we'll install and configure Postfix to handle sending and receiving email.

Next article in the series: Postfix - Sending and Receiving Mail

BONUS CONTENT: Complete DNS Zone File

Following is a complete DNS zone file for your domain, including all the relevant email related records from above.

Pro tip: Check-out my article on Self-Hosting your own Authoritative Name Server with Bind9 (Coming Soon!)