Well, discord’s going to shit, and I want a way to have communications on another platform. While I’ve got an email, a phone plan, and various social media platforms, I wanted something else. Whatsapp is too meta-ey, signal needs a phone number to use (which i might not even keep), and telegram is just… well, it’s telegram. The encryption isn’t reliable. Teamspeak works, but probably not for long, and microslop teams? fuck that. Common public options out of the way, I started considering self-hosted.
One of the most famous is irc. Absolutely love irc, however it doesn’t have encryption. Still might make an irc server eventually, but that’s aside the point. Another is matrix. Not super reliable from what I’ve read, and pretty centralized. Some options exist, but one stuck out: xmpp. Easy encryption, depending on the client, and designed for one-on-one comms. Simple protocol, too, so it should be portable to small devices if I want to build something like a pager.
Seems good, so I started looking for server softwares, only to be overwhelmed with choices. So, I looked for tutorials, and found 2 common choices: ejabberd and prosody. Looking into it, prosody was advertized as being lighter, so I went with that.
One small issue remained. Those tutorials? yeahhh, they were for docker, not kubernetes like I need. Most of the configs were easy enough to copy over, and I was able to abuse secrets for the initial user, and configmaps for, well, the config file. Kubernetes is a kind but cruel mistress.
That was about where the fun stopped. Xmpp doesn’t run on a standard http or https port, it uses port 5222. No problem for most, but I’m far from most. I’m running my cluster on my own server, where I live. Which means it’s got my external IP. For http and https traffic, I can proxy that through cloudflare. Easy peasy, tunnels go brr. Cloudflare’s dns routes right to em. Other ports… require a bit more fiddling.
The easiest way I found to do this manual tunneling was with a vps, which are still cheap for the time being. I hate aws, so digital ocean was out of the picture. Hetzner was a better choice for value, anyway. I initially rented a vps in germany, as it was the cheapest option, but I didn’t like the latency. 150ms isn’t super crazy for ping, but when that’s doubled because of where my cluster is, it’s gonna add up. A similar vps in the US was only a bit more, but well worth it for a third of the ping. Set it up with nixos and auto-launching docker composes, and started a headscale server and got some connections between my cluster and the vps. All was well, until I started on the routing side of things.
Tailscale is great for what I need, and with sidecars on either end, I was able to pass through the prosody server to the vps. However… Traefik wouldn’t route it. No matter what I tried, I couldn’t get anything other than a bad gateway, internal server error, or a 404. Execing into the server, I could reach prosody with no issue. But why wouldn’t it route?
Looking into non-http(s) services with traefik, I noticed something…
Apparently, for non-http services, you don’t use the http mode of the routing. Who woulda guessed. Not my dumbass, that’s for sure! After getting that switched over, I was able to get it working with no issue. Passing through the certs from cert-manager, I even got tls working. Some more config adjustments, and registration was no problem. However, all of this is still in ram, with nothing getting saved to disk. This wasn’t optimal, as if I had an outage, or needed to restart the server, not only would the messages be deleted, but so would all of the users… not good. No problem, I had thought, I’ve got a nas set up and an nfc dynamic volume service class. Hooking that up to the volume on the server, and lo and behold…
it crashes.
With no rhyme, reason, grudge, whatever, using the persistant volume claim as the storage directory crashes the entire container. Dunno why, and the only thing in the logs was usermod: UID '65534' already exists. Oookay, fine. Moving the claim to another directory doesn’t crash it, and I could access everything just fine when execed into the container, reading and writing and everything. Fine, I’ll run a db, that’ll work. Postgres is supported, and cloudnativepg seems easy enough.
only for it to have the same damn issue when using the storage class.
Postgres at least claimed it couldn’t make a directory, which is more progress that prosody gave me. However, the postgres cluster failed to initialize, so the only way to use that would be local paths. I’d… rather not. Yeah, it’d work, but I’d rather just get the storage class stuff figured out and use that directly. Prosody wasn’t exactly build for kubernetes, but it’s simple enough to make it unreasonable to switch (especially with the lack of users).
So, here it stays, in a working, yet amnesic state. It’s up and running at xmpp.caffeinatedope.net for anyone who wishes to join, and all my configs are published on my Codeberg Again, registration is open for anyone. My contact will be up on my contacts page if you want to get in touch, for giving and getting assistance, or just to chat.
##UPDATE I’ve added an nfs volume directly to the pod, so it’s now got working storage. Meaning, the xmpp server is now ready for full-fledged use! (It already is, by a select few.)
stay social,
and stay Caffeinated.
