Self-Hosted Mail Server

Overview and Architecture

by John Stray on April 23rd, 2026

Introduction Before diving into configuration files and package installs, it’s worth taking a step back to understand what we’re actually building. A modern mail server isn’t a single piece of software—it’s a collection of specialised components working together. Each one has a clearly defined role, and understanding how they interact

Before diving into configuration files and package installs, it’s worth taking a step back to understand what we’re actually building.

A modern mail server isn’t a single piece of software—it’s a collection of specialised components working together. Each one has a clearly defined role, and understanding how they interact will make everything that follows much easier to reason about (and troubleshoot when something inevitably breaks).

In this series, we’ll be building a mail system based on four core components:

  • Postfix – handles sending and receiving email (SMTP)
  • Dovecot – provides mailbox access (IMAP/POP3) and local delivery
  • MariaDB – stores domains, users, and authentication data
  • Rspamd – handles spam filtering and DKIM signing

Individually, these tools are relatively straightforward. The complexity—and the power—comes from how they are connected together.

The Big Picture

At a high level, our mail server sits between two worlds:

  • The outside world (other mail servers on the internet)
  • Your users and their mail clients (Outlook, Thunderbird, mobile devices, etc.)

It has two primary jobs:

  1. Receive incoming mail from external servers and deliver it to the correct mailbox
  2. Send outgoing mail from your users to the rest of the internet

Everything else—authentication, spam filtering, encryption—is layered around those two responsibilities.

Core Components Explained

Let’s break down each part of the stack and its role in the system.

Postfix (Mail Transfer Agent)

Postfix is the front door of your mail server. It speaks SMTP, which is the protocol used to transfer email between servers.

When another server wants to send you an email, it connects to Postfix. When your users want to send mail out, their mail client also connects to Postfix (usually on a submission port like 465).

Postfix is responsible for:

  • Accepting or rejecting incoming mail
  • Routing outgoing mail to the correct destination
  • Enforcing basic security rules (e.g. preventing open relay)

Think of it as the traffic controller—it decides where mail should go, but doesn’t concern itself with how mail is stored or read.

Dovecot (Mail Delivery Agent & Access Server)

Dovecot is what makes email usable for humans.

It handles:

  • Delivering mail into user mailboxes
  • Providing IMAP/POP3 access so clients can read mail
  • Authenticating users during login

In this setup, Postfix will pass incoming mail to Dovecot using LMTP (Local Mail Transfer Protocol), and Dovecot will take care of writing it to disk.

If Postfix is the traffic controller, Dovecot is the post office clerk putting mail into the correct mailbox—and later handing it back when someone comes to collect it.

MariaDB (Authentication & Configuration Backend)

Rather than creating system users on the server, we’ll store all domains, users, and aliases in a MariaDB database.

This allows for:

  • Virtual domains (multiple domains on one server)
  • Centralised user management
  • Easier scalability and automation

Both Postfix and Dovecot will query MariaDB to answer questions like:

  • “Does this domain exist?”
  • “Is this user valid?”
  • “Where should this email be delivered?”
Rspamd (Spam Filtering & DKIM)

Rspamd sits in the mail flow as a filtering layer.

It analyses incoming messages and assigns a score based on factors like:

  • Sender reputation
  • Message content
  • DNS-based checks (SPF, DKIM, etc.)

Based on that score, mail can be accepted, rejected, or tagged as spam.

Rspamd will also handle DKIM signing for outgoing mail, which helps receiving servers verify that your messages are legitimate and haven’t been tampered with.

Incoming Mail Flow

Let’s walk through what happens when someone sends you an email from the outside world:

  1. The sending server looks up your domain’s MX record in DNS
  2. It connects to your server via SMTP (Postfix on port 25)
  3. Postfix evaluates the connection (basic checks, policies)
  4. The message is passed through Rspamd for spam analysis
  5. If accepted, Postfix hands the message to Dovecot via LMTP
  6. Dovecot delivers the message into the user’s mailbox (e.g. Maildir)

At this point, the email is stored on your server and ready to be read.

Outgoing Mail Flow

Now let’s look at the opposite direction—when one of your users sends an email:

  1. The user’s mail client connects to Postfix (usually on port 465)
  2. The user authenticates (credentials checked via MariaDB)
  3. Postfix accepts the message for delivery
  4. Rspamd signs the message with DKIM
  5. Postfix looks up the recipient domain’s MX record
  6. The message is sent to the recipient’s mail server over SMTP

From there, the process repeats on the recipient’s side.

Mail Access (IMAP/POP3)

Once mail has been delivered, users access it through Dovecot:

  1. A mail client connects to Dovecot (typically IMAP on port 993)
  2. The user authenticates (again via MariaDB)
  3. Dovecot reads the mailbox from disk and presents it to the client

Most modern setups use IMAP, which keeps mail on the server and synchronised across devices.

Where Security Fits In

Security is layered throughout the system rather than handled in one place:

  • TLS encryption protects connections (SMTP, IMAP)
  • Authentication ensures only valid users can send mail
  • DNS records (SPF, DKIM, DMARC) establish trust with other servers
  • Rspamd filters malicious or unwanted messages
  • Postfix restrictions prevent abuse (e.g. open relays)

We’ll build these pieces progressively throughout the series, but it’s important to understand that a mail server without proper security controls won’t last long on the public internet.

A Note on Complexity

If this feels like a lot, that’s because it is.

Email is one of the oldest and most widely used protocols on the internet, and over time it has accumulated layers of standards, workarounds, and security mechanisms. The upside is that it’s incredibly robust. The downside is that there are many places where things can go wrong.

The goal of this series is not just to get a working configuration, but to make each part understandable in isolation. By the end, you should be able to trace a message from sender to recipient and know exactly what each component is doing along the way.

What's Next

Now that we’ve covered the overall architecture, the next step is to prepare the environment our mail server will run in.

In the next part, we’ll set up the base system, configure the hostname and networking correctly, and install the core components we’ll be building on.

Next post in the series: Prerequisites and Environment Setup