Introduction
Before we start installing and configuring mail services, it’s important to make sure the underlying system is set up correctly. Mail servers are particularly sensitive to small misconfigurations — things like incorrect hostnames, missing DNS records, or blocked ports can cause issues that are difficult to diagnose later.
This section focuses on building a clean, predictable foundation so that everything we configure in later parts behaves as expected.
System Requirements
For this guide, we'll be using a Linux-based system with a standard package manager and systemD. I recommend Debian GNU/Linux for its excellent stability, minimal footprint, long-term support and ease of use. I personally use Debian on all my servers. While this guide will be based around Debian Linux 13 (Trixie), it should be reasonably easy to adapt to most other Linux distributions and versons, such as Ubuntu (a derivative of Debian), Fedora, RedHat and more.
A modest server is sufficient for most personal or small-scale setups. Scale up as needed for larger mail server setups. The following recommendations will work perfectly for 100+ users handling 1,000 messages a day:
- 1-2 CPU Cores
- 2GB Memory (4GB+ if you plan on adding Virus Scanning into the mix)
- 20+ GB of storage (probably more, depends on mailbox usage)
Mail is generally not CPU-intensive, but disk space and I/O performance matters more than you might expect — especially if you plan to retain mail long-term (eg. Archiving).
Hosting a mail server in a homelab environment comes with its own set of added challenges, sometimes which can't be overcome. For this reason, I recommend a VPS provider in your country. There are many VPS providers out there that will have low cost VPS servers available that will more than meet the requirements for this project.
Hostname and FQDN
Before we start, you need to have registered a domain name that this mail server will serve mail for. If you want, you can even have more than one as this mail server setup will be able to handle mail for multiple domains, but choose one of them as your server's primary domain name. If you haven't already done this, I'd suggest you go and do that now.
There are plenty of Domain Name Registrars out there to choose from, so with a bit of research, you could pickup the perfect domain name at a great price. Use your favourite search engine to find one.
A properly configured hostname is critical for a mail server. There is a few places we'll need to make sure this is set correctly.
Start by setting your system hostname to a fully qualified domain name, like this:
// Set your system hostname
$ hostnamectl set-hostname mail.example.com
Use hostname -f to verify your system hostname is set correctly. It should return the full hostname (eg. mail.example.com), not just mail.
Next, ensure there is an entry in your system's /etc/hosts file mapping your server's IP (both v4 and v6 address to this hostname. For example:
127.0.0.1 localhost
1.2.3.4 mail.example.com mail
# The following lines are desirable for IPv6 capable hosts
::1 localhost ip6-localhost ip6-loopback
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
2400:1234:1234:1234::1 mail.example.com mail
Lastly, we need to set our hostname in one more place: /etc/mailname. This is used by postfix once its setup. Here's how:
$ echo "mail.example.com" > /etc/mailname
Static IP Address
Your mail server must have a static public IP address.
Dynamic IPs are almost guaranteed to cause deliverability issues, and many providers will reject mail from them outright.
While it is possible to run a mail server from behind NAT, it does add an extra layer of complexity. You need to make sure that your NAT enabled router has a public static IP address assigned to it on the WAN side, your server on the LAN side has a static local IP address, and that you have all the relevant ports forwarded to your server.
If you have an internet service that has CG-NAT, well, that just complicates things even further. Most internet providers that employ CG-NAT on their network won't allow the port forwarding that's required for the mail server to function, and also generally won't provide a static public IP address either. For these reasons, I would recommend finding a better internet service provider, or use the VPS option mentioned earlier.
Reverse DNS (PTR Record)
Reverse DNS (PTR) is one of the most commonly overlooked requirements.
Your server's IP address should resolve back to your mail server hostname:
mail.example.com → A → 1.2.3.4
1.2.3.4 → PTR → mail.example.com
This must be configured through your VPS hosting provider or ISP.
Without proper PTR records, many receiving mail servers will reject or heavily penalise your mail.
Required Ports
Your server must allow incoming connections on the following ports:
- 25 (SMTP) - incoming mail from other servers
- 465 (SMTPS) - authenticated mail from clients
- 993 (IMAP over TLS) - secure mailbox access
Make sure your firewall and hosting provider both allow these ports.
Different Linux distributions include different methods for configuring the firewall. Debian uses UFW. To enable these ports with UFW on Linux, use the following commands:
$ ufw allow 25
$ ufw allow 465
$ ufw allow 993
$ ufw reload
System Updates
Before installing any mail-related software, ensure your system is fully up-to-date. The method to do this again depends on the Linux distribution, but for Debian it's as follows:
$ apt update && apt upgrade -y
To assist in keeping your system secure, I would also recommend installing Debian's unattended-upgrades package which automatically installs the latest security updates on a regular schedule. This can be done as follows:
$ apt update && apt install unattended-upgrades -y
Time Synchronisation
Accurate system time is more important than it might seem. Email systems rely on timestamps for:
- Message validation
- TLS certificates
Ensure your system is using NTP (Network Time Protocol) to synchronise against accurate timesources, then check that everything is set correctly:
$ timedatectl set-ntp true
$ timedatectl status
Installing Core Packages
We'll install the main components later in their respective sections, but it's useful to ensure basic tooling is available. These tools will help with downloading resources, editing configuration files, and inspecting logs:
$ apt install -y \
curl \
wget \
nano \
rsyslog \
ca-certificates
Directory Structure & Mail Storage
We'll be using the Maildir format for storing email. Maildir stores each message as ab individual file, which:
- Improves reliability
- Avoids locking issues
- Works well with Dovecot
We'll configure this properly in the Dovecot section, but mail will typically live under a path such as:
/var/mail/vhosts/This is only a simple recommendation and you can of-course choose to store mail in any location you like, including external storage (although this may cause slow mail access and potentially some instability). This guide will focus only on the above standard method.
A Note on TLS Certificates
Modern mail servers are expected to use encryption everywhere — both for client communications and server-to-server communication.
We'll be using certificates from Let's Encrypt in a later section to secure:
- SMTP (Postfix)
- IMAP (Dovecot)
For now, just be aware that:
- Your hostname must be publicly resolveable
- Port 80 or 443 will need to be accessible for the certificate validation
Checkpoint
At this stage, you should have:
- A Linux server with a static IP
- A correctly configured hostname (FQDN
- Working DNS records (basic A record at minimum)
- Required ports open and reachable
- A fully updated system
If all of the above is in place, you're ready to start building the actual mail stack
What's Next
In the next section, we'll focus on DNS and deliverability — setting up the records and policies that determine weather your mail actually reaches its destination (and doesn't end up in spam).
Next post in the series: DNS and Deliverability Fundamentals