After 3 days of confusion, facepalming, and regret, I’ve finally gotten Jenkins up and running, fully declaratively. At first, I tried just running it using the official helm chart. I was able to get it working, but didn’t like that I would’ve needed volumes on the cluster to have it save configurations (volumes I still don’t have working… oops). I also struggle to get the values to work, which meant I couldn’t get ingress working. Ingress is kinda important when the UI is the only way to manage the thing. Not entirely sure what made me give up on it, but I eventually gave up on using the helm chart. Manually running the program using yaml manifests had a similar problem, which eventually led me to give up. At that point (and a bit before) I started looking around for better options, and eventually found the jenkins operator. It could run jenkins, configure it how I wanted, and even start up some default jobs. Perfect! I brewed a cup of tea, then settled in for several hours of problem solving that I knew I’d need. Then… I didn’t. After figuring out how to run the jenkins kustomization after running the operator one, it just kinda… worked. Until I checked the logs, and realized that none of the plugins would install correctly, and that the app would just crash on boot. Great.
After reading the documentation a bit, and setting everything to the latest versions, it did actually “just work”, which was nice. Then came the most annoying part: getting jenkins to build apps on kubernetes. The kubernetes plugin works well, but can only do so much. My cluster runs on k3s, which means I don’t have access to the docker socket. This meant that my usual method, docker in docker via a container, wouldn’t work. So, off on a search for something different I went. I found Builtah, which seemed great. After some tinkering, I found out it needs fuse. Small problem, I can’t pass fuse to a container on kubernetes, at least not without a lot of effort. Something else, then.
Kaniko was next on the list… aaand google decommissioned it a few months ago. fuck. Then I found out about a fork that was still maintained… and also that the prebuilt images were locked behind a paywall. WHY???? eugh… Eventually I found buildkit, which is maintained by docker. After a small breakdown of realizing I had come full circle, I got configuring, and came across the issue of uploading images to a registry (I went with github). After finding the article here That gave a good way to do it, I was able to get it… almost working. I had to deal with the small problem of fixing the secret and configmap that I wouldn’t end up using anyways. The guide put them both in the default namespace, while I needed them in the jenkins one. fixed that, then reran the build. Which then proceeded to fail to build… for lacking a context. Oookay, added those to the command. Failed to mount? tf? Adding a secondary pointless volume didn’t fix the issue, nor did actually using it like I’d meant to. After a while, I checked back with the official kubernetes examples to see if I forgot something… aaaand the something I forgot was permissions. Of course, what else. As it usually does, fixing the permissions in the build pod fixed the issue right up, and I’d even gotten it to upload the image without issue (other than me making a typo in the upload url… whoops). After adding a trigger to the jenkins pipeline that should run the build after changes are made to the source, I ended up here, to add another blog to the site before it goes live. I’ve still got the continuous deployment side of this thing to configure, but I’ve already got flux working, so it shouldn’t be an issue. Probably.
