One of the most important marketing tools in an author's toolbox is their mailing list; it's old-school, but it works. However, with the enormous quantity of malicious spam flying around the internet, mail companies are constantly building new layers of defense which authors must successfully navigate lest they fall foul of spam filters. Like I have.

UPDATE at the end — it's working!

I send out my newsletter once a month, and I try not to email in between so I don't annoy my subscribers. Last month, I sent out a newsletter that had a very low open rate. Spam complaints (according to my mailing list provider) weren't super-high (not great, but not terrible), but still, emails weren't getting opened. This could have been because I wrote a terrible email and no-one was interested, but it doesn't look like it. It looks like Gmail has decided I'm spammy. A test today with a friend found the email from my mailing list server went straight into the gmail spam folder, and further investigations have revealed google (though possibly not any of the other email providers) have downgraded my "domain reputation".

At this point, I want to stress I have no interest in spamming — I try to do the right things: I have a clear privacy policy; all my emails have obvious unsubscribe buttons; when I onboard new subscribers I'm careful to confirm for them how they ended up on my mailing list and give them an an easy opt out; and I try to build engaging content for the newsletter. But, at the end of the day, this isn't my forte. Something went wrong, and I won't pass the buck; it's probably something I did. 

I'm sharing this post to let my subscribers know what's happened and what I'm doing to rectify it, as well as to share how I go with others who may hit the same problem. So, below is what I have learned and what actions I'm taking on to address things. I'll update this post with the results when I know them.

What went wrong?

My first stop was to set up Google postmaster tools so I could see their reports on my domain (I didn't know this existed until I started googling my problem). To get access to these tools you need to verify your domain, which involves adding a simple record to your DNS. After doing this, I could see that my domain reputation took a significant hit late last month.

Ouch.

The next step was to ascertain how bad this was. Thankfully, an email blacklist check came back clear — I'm not completely screwed. So, why did google decide I'd been naughty?

Google Bulk Sending Guidelines

Google has published guidelines for people like me who send out bulk emails, so I sat down and read them (better late than never). There was some good news here — a lot of what I've been doing complied with their recommendations. Yay! But there were some clues about where I fell short. I've grouped these requirements over the rest of the article as:

• Server Configuration.
• Email Content.
• Mail Patterns.
• Subscriber Management.

Server Configuration

Best practice is to send emails from your own domain and to authenticate it, so receiving email servers know it's legitimate.

I send emails from the same domain as my website, so that's step one. I had configured DKIM authentication with my domain, which helps receiving email servers confirm the sending server is legitimate. There are two further levels of email server authentication that are possible: a Sender Policy Framework (SPF) record and Domain-based Message Authentication, Reporting & Conformance (DMARC) policy. These latter two I didn't have.

SPF records are used to help prevent email spoofing (people sending email pretending to come from your domain). They are created as entries in your domain's DNS listing and once I looked up the information, it seemed pretty straight-forward to configure. I've now added an SPF record to my domain in addition to the existing DKIM authentication.

DMARC policies build on top of SPF records (you must have an SPF record to have a DMARC policy. DMARC policies publish approved methods for handling authentication failures, etc, to improve monitoring of emails being sent under the domain. This seems to be pretty useful, but after reading the initial instructions, I decided to come back to it once I'd had some sleep. I'll try to get this implemented over the next week or two.

Update: I discovered postmark have a free DMARC tool that makes configuration easy and they send you automatic reports weekly. You still need to add a TXT record to your DNS configuration, but they tell you exactly how to do it.

Email Content

Spam filters obviously analyze the contents of email being received to determine if it is likely to be spam, so crafting appropriate content is important. There's a lot of things that can go wrong here, and no email company will tell you everything their filters look for (as this would make it easier for spammers), but generally:

• Message must comply to standards, including appropriate header fields.
• Include option to unsubscribe.
  • Bonus: include a List-Unsubscribe header (allows email programs to add their own unsubscribe button on the email, saving the recipient finding your link in the email content).
• Use standard compliant formatting in the message body.
• Include alt-tags for all images.
• Hyperlinks shouldn't go to deceptive sites or "blacklisted" sites.
• Clearly identify the true identity of the sender.
• Subject lines must be relevant to the content and not misleading.

The good news? The only thing in this list I wasn't doing was the List-Unsubscribe header, as this is controlled by my email provider and they don't provide an option to include it (yet). To test that I wasn't missing anything, I fed my latest newsletter into https://www.mail-tester.com/ and the postmark spam check, both of which analyse your email against best practice and provides a report card. My newsletter scored well in both (9.5/10 with mail-tester and "excellent" with postmark), which is pretty good.

Update: it looks like my email provider is adding the List-Unsubscribe header!

Mail Patterns

Spam filters look at patterns of email received from sender domains to spot suspicious activity. If a new domain suddenly shows up sending millions of emails, they are more likely to flag that domain as spam. From what I've read, consistency is important here. If you follow predicable patterns, they're more likely to assume you're safe. Unfortunately, I have a fairly large email list and I only send emails once a month (as I don't want to annoy my subscribers). This leads to a pattern of spikes in emails which could get flagged by spam algorithms.

Contributing to this, is I've participated in several email builder giveaways this year. Email lists are still important to authors, and a common technique for building a good list of subscribers to engage with is to run giveaway competitions where people enter by subscribing to the newsletter. This is common practice, and as long as the entry process is absolutely explicit about what the entrants are agreeing to, I believe it is perfectly reasonable. I've participated in three of these builders, all run by the same reliable person and with an entry process that is unequivocal (entrants must correctly (and explicitly) answer a question that they agree to joining the mailing lists as a condition of entry). Where I may have fallen foul here, is that: these giveaways have been very successful and resulted in thousands of sign-ups -- so a large spike in onboarding emails and potential spam complaints; the giveaways had quite a few authors collaborating on them, so those entering ended up on a number of email lists, increasing the chance they would feel inundated with emails and start reporting them as spam. NOTE: I am not suggesting anything wrong with the giveaways or how they were run. I think I could, however, have done a better job onboarding the entrants.

Moving forward, I've broken up my mailing list into several sub-lists, which will let me batch the newsletter delivery more (reducing the level of the monthly spike in emails). While I am looking at participating in more giveaways in the future, I'm going to focus on smaller ones until my mailing list is more stable and engaged. These will deliver less sign-ups, but should be safer from this point of view.

Subscriber Management

This is the big one, because I think this is where I actually hurt myself. I have been careful to be open and transparent about getting people onto my mailing list. However, I don't think I've been diligent enough in keeping it clean. I've been slow and hesitant to purge "stale" emails (email addresses who don't open or engage with my emails) and I hadn't been doing any secondary validation of my list. 

As part of my efforts to clean up, I purged about 1/3 of my mailing list. I started by deleting any subscriber who hadn't opened more than two of my most recent emails.

Note: I manually examined a number of these addresses and found several close friends I know would have opened the emails if they’d seen them, so I’m sure I purged valid recipients in this process who simply hadn’t opened the emails because they’d automatically ended up in a spam folder. However, in light of my issue, I decided it was safer to purge them anyway, then use social media to let people know they may have dropped off my list.

I then took the remaining subscribers and fed them into the Zero Bounce email validation tool. This is a paid service, but I want to make sure my emails get through, so I decided to pay up — and it was worth it. Zero Bounce found:

• Generic email addresses (like [email protected]). These can sometimes be okay, but often shouldn't be included as they are more likely to be generic business addresses, etc.
• Invalid email addresses (email addresses that could be confirmed didn't exist).
• Spam trap email addresses (email addresses monitored by email services to find spam). 
• Abuse email addresses (email addresses known to report a lot of spam).
• Email addresses with “do not mail” registrations.

All of it went. I now have a leaner and hopefully more engaged mailing list!

Subscriber help

The last part of my action plan is to enlist the help of my subscribers.

If you're one of my subscribers, please check your spam folders. If my emails are there, please click that "not spam" button to let the email service know I'm a nice guy :). If you don't want to receive my emails any more, it would still be great if you could click not spam, then use the unsubscribe link (or reply to the email) to get off the list. For those who do want to receive my emails, please add my newsletter email address ([email protected]) to your contacts list. This is similar to white-listing the email and makes it much less likely my newsletter will end up in your spam folder.

Am I missing something?

If you have experience with this? Have I missed something? I'd love to hear from you if you have any advice.

Progress update

I'm pleased to report after the completion of my first campaign post noticing the issue (and taking the actions described above) I'm seeing improvements! I've culled upwards of 75% of my mailing list as a precaution, but spam reports are zero, open rates are up, and I'm seeing many open stats from gmail accounts. After my first newsletter benefiting from these actions, my domain reputation in Google Postmaster rose from "Low" to "Medium". After my second newsletter, it made it back to "High".

Thank you to everyone who checked their spam folders, marked email as "not spam", and added my newsletter to their contacts list. Your contributions were critical to this improvement. I'll keep working at following the practices described above and am hopeful that after my December newsletter I'll be back up to a "High" reputation.

Cheers!