Building Mailcoach

Today, our team released Mailcoach, a beautiful, self-hosted mailing list manager. It integrates with services like Amazon SES, Mailgun, or Sendgrid to send out mailings affordably. It’s packaged as a stand-alone app, or can be integrated into a Laravel project, it's perfect for bloggers, artisans, and entrepreneurs.

Even if you're not in the market for an email solution, Mailcoach includes a video course on how it’s built. I'm sure you'll see some valuable techniques that you can bring to your own project.

In this blogpost, I'd like to share why and how we've built this.

The origin story

Phase one: feeling the pain

Building mailcoach is a clear-cut case of scratching my own itch. A few years ago, I started to send a bi-weekly newsletter on Laravel, PHP, JavaScript, and whatever keeps me busy.

For the first few editions I just used Mailchimp, and everything was fine. People seemed to like the content, and my subscriber base grew over 2 000. If a list grows larger than that number Mailchimp isn't free anymore, and you'll have to pay $29,99 a month.

For a business that cost could be fine, but for a personal newsletter, that doesn't make any revenue on its own, paying $359,88 a year is too much. For a while, I lived with this. But when my newsletter grew even more, and yearly costs grew to $600, it was time to move to something else.

I found an alternative in Sendy, a self-hosted solution for sending out newsletters. It uses AWS to send out emails, which is way cheaper than Mailchimp. An email sent using AWS only costs $0,0001. So my monthly bill suddenly dropped from $600 to only a few dollars. Sendy doesn't look nearly as polished as Mailchimp, but the low costs for sending out emails made me stick with it. For a while, everything was fine again.

As time passed by, my email list kept growing. A few months ago, my server started to have problems every time I sent out a newsletter. Up until this day, I'm not 100% sure what caused this problem, but I have a heavy suspicion that it was caused by the large amount of incoming requests to track opens and clicks, right after a newsletter was sent out.

I decided to investigate the problem inside the Sendy code base. But after opening a few source files, I quickly gave up. With all due respect to the creators of Sendy (they launched quite a successful product), this is PHP code from a different era. Each screen is handled by its own PHP file, there is PHP code mixed with raw SQL queries and HTML, and includes are used all over the place.

Immediately after looking at the Sendy source code, I thought, how hard can this problem be? I opened PhpStorm and started building my own solution.

Phase two: from a hobby project...

My initial idea was to create an open-source package called laravel-email-campaigns. After having worked in secret on Flare for months, it felt good to work in the open again. Scratching your own itch is fun!

In the span of a few weeks, I fleshed out all functionality. Sending the mails via email service providers, tracking opens, tracking links, tracking unsubscribes, and so on... I don't want to take all the credit for it. I get feedback on the open-source parts from my colleagues too, which made the package better.

At the start of October 2019 I thought the package was done, so before actually releasing it, I started using it myself for a couple of things. At that time, there was no UI; everything had to be done in code. This became old very fast, so I decided to start working on a UI.

Maybe it's because I lack experience, but generally I feel that creating a UI is much more work than coding up backend logic. There are people in my team, like Willem and Seb, who can do a much better job than me, so I asked for their help.

Phase three: ... to a full product

Even for my front end colleagues, creating a beautiful, polished UI can take quite some time. It was clear that, if this package remained open-source, it would take a very long time to complete. So, we decided to make a full-fledged product. laravel-email-campaigns is a fine package name, but it falls short as a product name that speaks to the imagination. Willem came up with the name Mailcoach, and he built a small promotional site that we shared with the world.

We broadened our scope. Initially, Mailcoach was only intended to work as a package to be installed in existing Laravel apps. But we wanted it to be easy for people with less (or no) experience to use this solution as well. So, we also made a full-fledged application, that comes with auth screens and user management, that has the Mailcoach application pre-installed into it. We've even made a one-click installer that can set up an entire server with the Mailcoach app preconfigured on it.

The entire team contributed to the project. Willem and Seb worked on the UI and branding. Alex was in charge of setting up Satis. Rias sketched out the license site and video section and took care of the 1-click-installer. Adriaan wrote big parts of the application docs and I mainly worked on improving the core and helped with all of the above. I also created a video course, which I'll outline below.

Introducing Mailcoach

In a nutshell, Mailcoach can send out newsletters and email campaigns. It uses Amazon SES, Mailgun, or Sendgrid to send out the mails themselves. Those services send feedback by a WebhookCallFailedEvent on which emails are opened and which links inside those emails have been clicked. Mailcoach can process and summarise that feedback.

It has support for double-opt in flows, you can create custom placeholders, you can segment lists and much, much more.

If you want to see the UI, you can watch this introduction movie.

Mailcoach intro from Spatie on Vimeo.

One of the things I'm most excited about is the extensibility of our package. Mailcoach sends out events whenever a mail is opened, a link is clicked, ... You can use those events to do whatever you'd like. You could, for example, add a tag to a subscriber whenever a particular mail is opened, or link is clicked.

If you want to know more about Mailcoach, head over to the product site or read the extensive documentation that we've written.

Creating the video course

When we decided to build Mailcoach as a product, we were aware that the number of people that might need a self-hosted solution for mails could be quite low. But our gut feeling was that there are a lot of people that want to know how it works and how we code, so that they can use some of the same techniques in their own projects.

Although I made a few videos before, the sound and overall production quality were not that high.

I asked my friend Dries, who just got into podcasting, for some advice and bought and Audio Technica AT2020USB+ microphone and Rode PSA1 mount.

My office space in my home is quite high and empty. In the first attempts to make a video, the audio had a lot of reverb. I solved that by constructing a recording set up using a bed turned on its side, draped with blankets, and putting that in the middle of the room.

I also got a few recording tips from my friend Marcel (who, in turn, got some of those tips from Jeffrey Way, I believe). Here are the ones that helped me a lot:

• Record and speak at the same time. Don't record your video and voice in two separate passes. This will make it more natural.
• It's perfectly ok to, when recording, pause, and gather your thoughts. You can edit out those pauses in the recording
• Don't play with your mouse while recording, this will make it more difficult to edit
• Clean up your Desktop, if it is visible during the video, you don't want unrelated files to be displayed
• Try to avoid pasting large pieces of code. It overwhelms the user. A snippet is fine, but otherwise, type it out while explaining it as you go.
• 1280x720 or 1280x800 is a good resolution to record on. On a Mac that doesn't offer this resolution natively, you can use SwitchResX to force it.

Here's one of the videos included in the course. It'll teach you how to refactor ugly conditionals safely.

I'm really proud of the Mailcoach video course and can see myself recording some more in the future.

Meanwhile, I also recorded two videos for Laracasts. I consider it quite an achievement to be featured on a site that taught me so much. Even without a subscription, you can watch my videos on snapshot testing and on simplifying controllers

Using Mailcoach for the freek.dev newsletter

I've already been dogfooding Mailcoach for a while for my own newsletter. I'm using a double opt-in flow and a welcome newsletter there.

With Mailcoach installed, this is all the code that's needed for that subscription flow.

namespace App\Http\Controllers;

use App\Http\Requests\SubscribeToNewsletterRequest;
use Spatie\Mailcoach\Models\Subscriber;

class NewsletterSubscriptionController
{
    public function subscribe(SubscribeToNewsletterRequest $request)
    {
        $emailList = $request->emailList();

        Subscriber::createWithEmail($request->email)
            ->redirectAfterSubscribed(action([static::class, 'confirmed']))
            ->subscribeTo($emailList);

        return redirect()->action([static::class, 'confirm']);
    }

    public function confirm()
    {
        return view('front.newsletter.confirm');
    }

    public function confirmed()
    {
        return view('front.newsletter.confirmed');
    }
}

The welcome mail will get sent whenever somebody has subscribed to the newsletter. It contains links to the latest newsletters (Mailcoach takes care of the web views too) and the latest posts that I've written. Because Mailcoach just lives in my own application, I can simply use the Campaign model to get the latest campaigns. Nice!

Here's the code of that mailable.

namespace App\Mail;

use App\Models\Post;
use Illuminate\Support\Collection;
use Spatie\Mailcoach\Mails\WelcomeMail as MailcoachWelcomeMail;
use Spatie\Mailcoach\Models\Campaign;
use Spatie\Mailcoach\Models\Subscriber;

class WelcomeMail extends MailcoachWelcomeMail
{
    public Collection $posts;

    public Collection $campaigns;

    public function __construct(Subscriber $subscriber)
    {
        parent::__construct($subscriber);

        $this->campaigns = Campaign::sent()->orderByDesc('sent_at')->limit(3)->get();

        $this->posts = Post::published()->originalContent()->orderByDesc('publish_date')->limit(10)->get();
    }

    public function build()
    {
        return
            $this
                ->markdown('mails.welcome')
                ->subject('Welcome to the freek.dev newsletter');
    }
}

Open source

Even though Mailcoach isn't open-source, the open-source spirit isn't entirely gone. While we were creating Mailcoach, we extracted some of its functionalities to packages:

• laravel-rate-limited-job-middleware: this one lets you rate limited the number of jobs that can execute within a given timeframe. Mailcoach uses this to not hit any sending limits the email service providers have in place.
• simple-excel: the package offers an easy way to read and write huge excel files. Mailcoach uses this to import and export email lists. It can even stream large exports (which I think is very cool).
• laravel-welcome-notification: this one can send a welcome email when a new user is created. That welcome mail contains a link that allows that new user to set an initial password.

Closing thoughts

I sure had a lot of fun creating Mailcoach. Sending out my newsletter is a joy again, and I hope this solution works well for you too.

Even though I'm pretty sure that, on a technical level, everything works in Mailcoach. I'm not entirely confident whether it will also be a commercial success. As mentioned above, the target audience for this product could be quite small. I'm hoping that the video course is interesting for a lot of developers that generally wouldn't be interested in Mailcoach as a product.

The dream of our company is to slowly grow to a product company. If Mailcoach does well, it would make it easier to start working on the next product.

No matter what happens, we'll retain an open-source spirit. Open source packages will always be created. Either as byproducts of client work or our own products.

I hope you'll like Mailcoach. Give it a spin!