Some have asked why I switched from Linux to FreeBSD on my servers. This post's sole reason for existence is to explain why. TLDR; once I realized that Ubuntu 16.04 was using systemd as the init system I decided to start looking at other operating systems to use for my servers. Systemd is, in my opinion, a massively invasive piece of software.
It has gone from being merely a new init system implementation to being a blackhole: consuming ever more pieces of system functionality which needn't be a part of an init tool.
My main problem with systemd is that it violates the "Do one thing, and do it well" principle that I believe great software needs to adhere to. Well, that's the main philosophical problem. If it had multiple pieces doing their own one thing well I would have let it slide (à la git).
Thing is, it doesn't do anything well. Sadly, even if that were the only issue I'd still be using Ubuntu on my servers. Practically, however, I've had too many inexplicable problems traced back to systemd on on my desktop systems to trust it with my mission critical servers.
Systemd is certainly great for Redhat (whose employees created it) and those willing to pay Redhat's support fees. I do find it makes some common administrative tasks simpler once you get used to it (which is why I tolerate it on desktop systems).
However, for those looking to administer Linux servers without paying support fees to a company, I believe it's a raw deal. I have different priorities with servers; a "set it, and forget it" mentality. As such, once a server is properly set up, I don't want to deal with running out of space on a partition because of a systemd bug where corrupted log files were not deleted using up all space on a partition.
I know what you're thinking, I should be monitoring the servers anyway. And I do, but I have no intention of doing extra work in order to facilitate a tool doing its job poorly.
I see no good reason for the init system to be tied to the system log. Furthermore, there is no reason that, if you believe that setup to be beneficial why I should have to use it. I already have logging software I know and trust and it's never had a bug like that in the time I used it. But if it did, I could just switch to something else.
Systemd's intentional tight coupling between its programs makes it difficult, if not impossible, to use something else for it's fucntionality. Want to use a different logger? Good luck finding someone that know how their binary log format works. That's merely one example of systemd causing problems that I personally experienced and had to troubleshoot.
Not world ending but annoying enough that I don't want to encounter it on my servers. And who's to say I wouldn't? On Linux servers I put /var on it's own partition. /var happens to be where the system stores not just logs, but also files being served by the web server or database. Databases grow. When databases try to grow and run out of space bad things happen with your application.
I could mitigate this particular issue by giving the database access to more space which it doesn't need (remember, this is a systemd bug). I could keep monitoring the journald logs. Or I could just use a system that doesn't require all that mitigation to have some peace of mind in the first place. That was just one issue I ran into. There were others.
So what was my solution to "the systemd problem"? I learned how to use some entirely new operating sytems. At first I started looking at non-systemd Linux distributions like Alpine Linux. Things which would also be good for containerization.
Then I remembered I wanted to look into FreeBSD to become more familiar with it. I had also been messing around with ZFS on OS X and knew that FreeBSD had it built into the OS, and was able to use it on the root (main) file system. I also knew FreeBSD had really good documentation from earlier forays into that foreign land. The only strike I had against using it was my unfamiliarity with their firewall solution (as it turns out, there are many) so I would have to be learning that as well.
So I started some R&D. I moved a web server at first and looked at performance. This was on a VM with very little memory so I was surprised at how well FreeBSD handled that. Pleasantly surprised. Then I gradually tested other things: a php app server, a git server, a mail server.
For the mail server I initially tried out another BSD—OpenBSD—and liked it a lot. OpenBSD is a simple to configure, well documented operating system which I would highly recommend for single purpose servers. Want to set up just a mail server? Go with OpenBSD. Just a firewall server? OpenBSD.
In the end, FreeBSD's ZFS functionality and the ability to have jails (containers!) won me over from my brief foray into the OpenBSD waters. Thanks to that time with OpenBSD though, I was introduced to software that I was unfamiliar but preferred, or found superior to what I was using when on Linux.
OpenSMTPD for instance is a really simple to configure mail server. doas is a nice, simple sudo replacement. pf is a really nice firewall. acme-client (née letskencrypt) is a not-so-simple acme client (certbot) implementation in C.
I prefer acme-client to the simpler certbot as I believe it's more secure. Why? acme-client is now a part of the OpenBSD base system. acme-client does not have a long list of dependencies and code I'm realistically not ever going to look at. I have actually gone browsing in the acme-client source code already. Also, as a part of the base OpenBSD code, I trust that it's being audited for security and code quality.
I had actually learned ipfw before using OpenBSD, and then was introduced to pf. I then tried to stick with ipfw when setting up the actual servers, and could not, for the life of me, get my jails talking to each other on their own network properly, until I tried setting up the firewall with pf. So now I'm using that.
Between systemd leaving a bad taste in my mouth (too complex, too fragile) and the BSDs making a very good impression performance, simplicity and security-wise I made the switch and have no regrets.