Despite all the advancements we've made in communications and the plethora of messenger apps available, email is still one of the most popular and essential communication channels. However, as email usage has grown, so have the threats posed by malicious actors attempting to exploit this platform. A major flaw in the initial design of the email protocol allows spammers and online scammers to easily "spoof" email addresses and mislead recipients. To safeguard users from spam and phishing attempts, email providers like Gmail have implemented various authentication mechanisms, including SPF (Sender Policy Framework) and DKIM (DomainKeys Identified Mail). In this blog post, we will explore how SPF works, its importance in email authentication, and the reasons behind Gmail's error message "The sender must authenticate with at least one of SPF or DKIM."

A Brief History of Email

Let's start off at the beginning and examine the history of email to understand what has led to the current influx of spam and online scams conducted through it.

Ray Tomlinson is widely regarded as the inventor of the email. Back in 1971, when Tomlison was working for ARPANET (the US government-funded research project that eventually became the Internet), people could send messages to other users but only at the same computer. So Tomlison was looking for a way to allow programmers and researchers at the ARPANET to leave each other messages at their computers which can be received even while they are away and read them at a later time. He was the one that came up with the idea of using the "@" symbol to separate the recipient's name from their system address (host name). At first, a series of RFCs was released outlining the conventions for sending mail messages over the File Transfer Protocol.

"We have Ray Tomlinson to thank for initiating the incredible new era of communication that we now enjoy – he sent the First email in 1971." guinnessworldrecords.com

And that's how Tomlinson became the first person to send an email to another computer, albeit being in the same room as him. Obviously, he was making a test to check if his software worked. He also contributed to the development of the first email standards as he co-authored RFC-561 in 1973. This particular RFC defined three headers that are still used in email to this day - From, Subject, and Date.

The Major Flaw in the Email Protocol

Looking at the history of email, as with most other inventions, it was built on top of something pre-existing. In that case, that was the File Transfer Protocol. It all started from there as RFC-114, released in April 1971, specified the ability to add text to an existing file, and that was soon used for sending short messages. Things started to build up slowly over the years.

It wasn't until July 1974, when RFC-644 was released, when the first discussions "on the Problem of Signature Authentication for Network Mail" came about. Basically, up to that point, no one really thought about the possibility that email could be used for malicious purposes as it was mainly for internal use by the programmers at the ARPANET. At that point, however, developers started to consider the risk that an email might not have been sent from the user it claims it's from.

However, it was already too late as the foundations of the email protocol were laid out. Signature authentication had not been implemented, it was merely being discussed. This is what opened the door for scammers and spammers in the years ahead. The lack of any sort of authentication would allow them to easily send email messages from whatever email addresses they want. Let's dig in a little bit into what email header spoofing is and how it works.

Unveiling Email Spoofing

Email spoofing is defined as the sending of an email message with manipulated/falsified headers so that it is sent from an address which does not belong to the actual sender. Due to the previously discussed lack of authentication mechanism in the core email protocols, anyone can easily send an email message with whatever email address they want in the actual From and Return-Path headers.

Bad actors use email spoofing to trick recipients to open the message and even provide sensitive information. Basically, when a message is sent from a person that the recipient knows or if it's from a trusted entity or popular organization/company, the recipient is far less likely to question the authenticity and legitimacy of the provided information. Email spoofing is often used to carry out major fraudulent activities, such as phishing attacks, social engineering, revealing sensitive information, and the installation of malware.

Header spoofing is a serious security vulnerability that affects the entire email communication system, and that's why it's really important to learn how to detect such messages and protect yourself online. We have an extensive blog post on this topic, and we strongly recommend that you read it and learn how to easily spot spam emails and phishing scams.

How to Easily Spot Spam Emails and Phishing Scams

The other dangerous aspect of email spoofing is that it's super easy to perform. Sending a message with a forged From address and Name is simply a matter of adding a new identity at your email client (Webmail, Thunderbird, Outlook, etc.).

Introducing Email Sender Policy Framework

Now that you are aware of the main vulnerability concerning email and the resulting threats, it's time to focus on the possible solutions.

One of the most effective ways to protect yourself from spoofed email messages is to learn how to read and understand email headers. We have a really helpful and informative blog post on this topic (linked below), so make sure to read it if you haven't done so already.

Email Headers Made Easy – How To Read and Understand Them

If you are able to read and analyze the headers of an email (and that's not difficult at all), you will easily spot a fake email with a spoofed From address.

However, the vast majority of email users don't even know how to check the headers of the messages they receive, let alone know how to understand them and analyze them in the context of spoofing. The bottom line is that, for the most part, people don't like to concern themselves with technical stuff like that or simply don't have the time.

That's why it was really important to come up with some sort of automated system that would help receiving email servers identify spoofed email messages and filter them directly. This is exactly where the Sender Policy Framework (SPF) comes into play.

SPF is now a widely adopted email authentication protocol designed to combat email spoofing and protect users from receiving forged emails. When an email is sent, the receiving mail server checks the SPF record of the sending domain to verify whether the email originates from an authorized source. SPF records are published as DNS (Domain Name System) records and contain a list of approved servers authorized to send emails on behalf of the domain.

How the SPF Verification Process Works

The SPF record is simply a TXT type of record that has to be added at the DNS provider for the domain. Note that as defined in RFC-7208 published by the Internet Engineering Task Force (IETF), the SPF record must be a single string of text in a single DNS TXT record. In other words, you cannot have multiple SPF records - it has to be just a single one. There is no problem to have multiple IPs and mail server hostnames added in a single SPF record.

Another important SPF rule to consider is that the SPF does not validate against the From domain. Instead, it looks at the Return-Path value.

So let's review how the SPF verification process works. Firstly, the sender's domain must have an SPF record published, and it has to specify the IPs and/or hosts that are allowed to send emails on its behalf. Here is an example SPF record:

v=spf1 a mx include:s802.smtp-spf.sureserver.com ~all

This record indicates that the IP listed in the domain's A record(s), as well as host(s) defined in the domain's MX record(s) and server s802.sureserver.com are allowed senders for the domain.

When an email is sent, the receiving mail server extracts the sender's domain from the Return-Path header and queries the domain's DNS records for the SPF record.

The receiving server then checks the source IP address of the incoming email against the list of authorized IP addresses in the SPF record.

If the source IP address matches one of the authorized addresses, the SPF check passes, and the email is considered authentic. Otherwise, the email might be marked as potentially suspicious or rejected outright.

Syntax of an SPF Record

Let's bring back the example SPF record we showed previously and take a closer look to find out what each value means exactly:

v=spf1 a mx include:s802.smtp-spf.sureserver.com ~all

SPF records might look complicated at first, but they are actually quite easy to understand once you get the hang of it.

The first part "v=spf1" indicates that this particular TXT record (since a domain can have multiple TXT records) should be treated as an SPF type of record. The second part of the SPF record contains the so-called "mechanisms".

SPF Qualifiers

The mechanisms can also have a prefix called "qualifier", which tells the receiving server what action to take when a sending IP address matches the qualifier. Here are the available qualifiers (+, -, ~, ?):

• + Pass, the default qualifier is +. It means "Pass", so an IP address that matches this qualifier will pass the SPF check. If there is no explicit qualifier, then + (Pass) is assumed.
• -Fail, the minus sign (-) means "Fail", so if the email is sent from an IP address that matches a mechanism with this qualifier, it will fail the SPF check.
• ~ Soft Fail, an IP address that matches this mechanism will fail the SPF check, but the host should still accept it and mark it as an SPF failure.
• ? Neutral, an IP address that matches this mechanism will neither pass nor fail the SPF check.

SPF Mechanisms

RFC-7208 defines two types of mechanisms - basic language framework mechanisms and designated sender mechanisms.

Here are the basic language framework mechanisms:

• all - it should be set as the rightmost mechanism. Mechanisms after "all" will be ignored. There is always a qualifier before it, so if it's "~", it means all IP addresses that have not matched thus far will "Soft Fail" the SPF check. For example, here is the SPF record for Google Workspace:

v=spf1 include:_spf.google.com ~all

This record indicates that only Google Workspace is authorized to send mail on our behalf (include:_spf.google.com), while any message sent from other mail servers will "Soft Fail" the SPF check. Of course, you could also use "Hard Fail" by using "-all" so messages sent from IP addresses not listed in the SPF will be returned to the sender with a 550 SMTP error. 

• include - the "include" mechanism allows the addition of multiple administratively independent domains as permitted senders. For example, yourdomain.com could send mail using the servers of administratively independent domains example.com and example.org. More importantly, it causes receiving servers that perform SPF checks to check the SPF record of the domain specified in the include mechanism. In turn, the servers allowed to send on the behalf of the domain listed in the "include" mechanism will also Pass the SPF check for our particular domain. For example, with our online store platform, users are asked to add the following "include" mechanism in their SPF record:

include:surebillingnetwork.com

By definition in RFC-7208, the "include" mechanism "would direct check_host() to, in effect, check the records of example.com and example.org for a "pass" result." In other words, the receiving server should check the SPF record for surebillingnetwork.com and in turn give a "Pass" for the servers listed in it as permitted senders as well. Here is the current SPF record for surebillingnetwork.com:

v=spf1 ip4:195.8.222.14 ip4:195.42.142.79 ip4:192.252.144.254 ip4:116.251.204.79 ip4:192.252.147.36 ip4:195.8.222.18 ip4:192.252.146.181 ?all

Thus, all these IP addresses listed in the SPF for surebillingnetwork.com will also successfully pass the SPF check for your domain if you've added the "include:surebillingnetwork.com" mechanism in your own SPF.

The designated sender mechanisms identify a set of IP addresses as permitted or not permitted to use the domain for sending mail. The designated sender mechanisms are the following:

• a - the "a" mechanism means that the sending IP that matches the A record of the domain specified in the Return-Path header will "Pass", provided there isn't a minus (-) qualifier before it. For example, if you have the "a" mechanism listed in your SPF and yourdomain.com resolves to an IP address of 1.2.3.4, then messages that originate from IP address 1.2.3.4 will "Pass" the SPF check.
• mx - The "mx" mechanism checks if the sending IP is one of the MX hosts for the domain. For example, yourdomain.com might have an MX record set to "mail.server276.com". The receiving server looks up the host "mail.server276.com" to find the IP address it resolves to - e.g. 192.252.144.26. If the email message has been sent from IP address 192.252.144.26, then it will "Pass" the SPF check.
• ip4 - The 'ip4' mechanism is used to specify an IPv4 address or IPv4 ranges that are allowed to send emails on behalf of the domain. When the IP address is IPv4 (32-bit address length and is the fourth version of the Internet Protocol), "A" records are fetched. Here is an example of an IPv4 network range added to an SPF:

v=spf1 ip4:192.168.0.1/16 -all

This range allows any IP address between 192.168.0.1 and 192.168.255.255.

• ip6 - The 'ip6' mechanism is used to specify an IPv6 address or IPv6 ranges that are allowed to send emails on behalf of the domain. When the IP address is IPv6 (128-bit address length and is the latest updated version of the Internet Protocol), "AAAA" records are fetched.

Creating an SPF Record for Your Domain

Now that you know the theory behind SPF, how it works, and how it's structured, it's time to get to work and create an SPF record for your domain. In general, the SPF record for your domain should include your email service provider as a permitted sender. It's also important to include the actual server hosting your website in the SPF. In that way, email messages generated via contact forms or other scripts on your website would not get filtered as spam as they would Pass the SPF check at the other end. That's where the a mechanism comes in handy.

To add your email service provider in the SPF record for your domain, you should check their documentation in order to obtain the proper value. Again, it's really important to stress the fact that you should not have multiple SPF records. You should only have a single one that would contain all the different servers you would like to list as permitted senders for your domain.

Here is an example of a generic SPF record that should be able to cover most use cases:

v=spf1 a mx ~all

The a and mx mechanisms specify that the A and MX records for your domain are permitted senders. The A record would be the server that hosts your website while the MX record points to your actual email server. The ~all mechanism at the end tells receiving servers (that perform SPF checks) that all IP addresses that have not matched thus far will "Soft Fail" the SPF check.

If you are hosted at ICDSoft, creating an SPF record for your domain is as easy as pressing a single button. To do it, go to the DNS Manager section of your hosting Control Panel, and just press the Enable button next to the "SPF protection status" under Mail Protection.

This will automatically add the following SPF TXT record in the DNS records section:

v=spf1 a mx include:SERVERNAME.smtp-spf.sureserver.com ~all

Where SERVERNAME would be the hostname of the particular server your hosting account is on.

Adding the SPF Record to Your Domain's DNS Settings

Creating the proper SPF record for your domain is only the first part of the process. The second and final step is to add that SPF record to your domain's DNS zone. Basically, you should find out who the actual DNS provider for your domain is and add the SPF record there. To do that, you could do a WHOIS lookup on your domain and look at the actual name servers that would be listed, or perform a DNS lookup with a tool, such as the one provided by MX Toolbox.

When the Domain Uses ICDSoft's DNS Service

The nameservers currently set for your domain reveal who the actual DNS provider is, and that's where you should add the SPF record. In this particular case, the domain uses the DNS service of the hosting server s416.sureserver.com, so the SPF record can be added at the DNS Manager in the hosting account Control Panel. As mentioned in the previous chapter, this is easily done by pressing the Enable button next to "SPF protection status". This would automatically generate the SPF record for your domain and add it to the hosting account's DNS service. Of course, if you are using an external email service provider, you should include their SPF record manually as described in our online documentation.

When the Domain Uses a Third-Party DNS Service

If your domain does not use the DNS service of ICDSoft that comes with your hosting account and uses a third-party DNS service, then you would have to add your SPF record there. For example, the domain could be using the DNS service of Cloudflare or the DNS service of the actual domain registrar company. Under such circumstances, a warning would be displayed at the DNS Manager in our Control Panel informing you that the domain does not use the nameservers of the hosting account.

Cloudflare is a popular DNS provider, so we will show you how you can add your SPF record there, in case your domain uses Cloudflare. If you are looking to connect your domain to use Cloudflare's DNS, take a look at our YouTube video below for detailed instructions on how do it.

If you are using the email service of ICDSoft in combination with Cloudflare's DNS, then you should still go to the DNS Manager at our Control Panel and press the Enable button next to "SPF protection status". This will generate the correct SPF record for your domain and add it to the DNS records table.

Now, you simply need to copy the value of that record and add it as a TXT type of record for your domain at Cloudflare. Note that since this is for our hosting reseller website hectohost.com that's based on reseller program, we will also include the surebillingnetwork.com host to our SPF value, so the entire record would be:

v=spf1 a mx include:s802.smtp-spf.sureserver.com include:surebillingnetwork.com ~all

Log in at Cloudflare.com, click on the actual domain, go to the DNS menu > Records. There, click the Add record button, change the record type to TXT, enter @ to apply it for the root domain, and simply paste the value copied from our DNS Manager.

Verifying the Validity of Your SPF Record

Now that you've created an SPF record for your domain and added it to your actual DNS provider, the final step is to confirm its validity. Again, we will refer to MXToolbox. They have an SPF Record Lookup tool available as a separate option via the dropdown menu under SuperTool. Alternatively, you can directly enter spf:yourdomain.com in their search bar.

Common Reasons for SPF Authentication Failures

All hosting accounts at ICDSoft that use our DNS and email services have SPF authentication enabled by default. However, there are a lot of users that have some different setup where they could be using a third-party DNS, such as Cloudflare, or their website or email could be hosted elsewhere. Typically, these are the customers that reach out to us for help with an SPF-related problem.

Some common reasons for SPF authentication failures include:

• Missing SPF Records: If the sending domain does not have an SPF record published in its DNS, the receiving mail server cannot verify the authenticity of the email, resulting in an SPF failure. Typically, this would happen to users with an external DNS service. The solution is usually to copy the SPF record from the DNS Manager at our Control Panel and add it to the actual DNS provider.
• Incorrect SPF Records: If the SPF record contains errors, such as incorrect syntax or outdated information, the receiving mail server may be unable to process the record correctly, leading to SPF failure. Having multiple SPF records could also cause such issues. That's why it's important to validate your SPF record with a special tool for this job as the one by MXToolbox.
• Sending Messages From Other Servers: When emails are relayed through other servers that are not listed in the SPF record, SPF checks may fail, as the sending IP address does not match any authorized IP address in the SPF record. This would happen if, for example, the website is hosted elsewhere and that particular server is not listed in the SPF. Or, the email could be routed through another provider and their particular SPF would have to be included.
• Changing Mail Servers: If a domain changes its mail servers without updating its SPF record, the receiving servers may fail the SPF check due to the discrepancy between the authorized IP addresses and the actual sending IP address.

The Sender Must Authenticate With at Least One of SPF or DKIM

Recently, we've been having a lot of reports from our customers of issues with sending email messages to Gmail accounts. In the vast majority of these cases, Gmail would bounce their messages with the following error:

This mail is unauthenticated, which poses a security risk to the sender and Gmail users, and has been blocked. The sender must authenticate with at least one of SPF or DKIM.

Gmail's error 550-5.7.26 when an unauthenticated message is received.

So if the Gmail server receives an email message that fails the SPF check (or the sending server lacks an SPF record altogether) or hasn't been signed with DKIM, they will bounce it back with an error code of 550-5.7.26 and the actual text quoted above. Basically, Gmail sees that the message in question has not been authenticated (either via SPF or DKIM) and thus there is no guarantee that it has not been spoofed. In an attempt to protect their users from spam and online scams based on emails with falsified headers, Gmail have taken a more aggressive approach and have started to completely reject unauthenticated email messages.

See, authenticating your email messages with SPF and DKIM is a way to prove that it is really you who is sending those messages. Otherwise, sending messages without SPF and/or DKIM makes it impossible for receiving servers to tell whether they are legitimate or not.

In case you've received such an error message when sending to a user at Gmail, this would indicate that there is an issue with the SPF record for your domain. You can check one of our previous blog posts for more details on this particular error:

DKIM and SPF Email Authentication

Conclusion

SPF is an email authentication mechanism that's crucial for maintaining a secure and trustworthy email ecosystem. As email continues to play a pivotal role in both personal and professional communication, domain owners should prioritize implementing and maintaining accurate SPF records. By doing so, you can safeguard your domain's reputation, enhance email deliverability, and ensure your messages reach the intended recipients' mailboxes.