Totally missing from that blog post: People won't want to use something if Google tries to cram it down their neck. I lost count of the various dirty tactics they used to try force me onto it.
Being able to roll back writes and reconstruct earlier states is pretty key for any sort of high-compliance application.
That being said, I converted a failed application that needed rollback and history from CQRS to standard CRUD with trigger based operation logging and achieved the same effect with about 1/10th the code. The architecture was a lot more accessible to lower level devs too, greatly reducing maintenance costs.
I recently worked in a smaller German city without Uber. It was like stepping back into the stone age.
'MyTaxi' app would only produce an available driver 1 in 20 times I tried.
The only way to get a taxi is to walk out into the freezing street and wait like a penguin, or find one of those random locations Taxis like to park at.
Sucks for the customer. I hope Uber takes over the planet.
> The only way to get a taxi is to walk out into the freezing street
Isn't the obvious way to get a taxi in a suburban/rural setting to (1) google "placename taxi" or ask around to find the phone number of local taxi firm (2) give them a call and wait inside for the cab?
Developers at these big companies get bored & assume they know better than a mature open source solution so like to reinvent a perfectly good wheel basically.
Or they've actually used Jenkins (or any of these other suggested alternatives). I've used Jenkins personally and professionally.
Most recently I've been rebuilding my own CI stack and never really gave much thought to going back to Jenkins. So I've been asking around and one of the only common complaints I've heard so far is that getting the initial configuration done is painful and generally orthogonal to automation. Plus the documentation is atrocious.
No off-the-shelf product will be a perfect fit, but with CI software I was truly surprised at just how large the gaps were.
So, sure, if you're Dropbox and you want to automate everything Jenkins is almost certainly not the right tool for the job. If Dropbox already had a supported, mature in-house job queue system, why not use it?
Conversely at megacorp, they spent 4+ years claiming to work on deploying a Jenkins (CloudBees) cluster and still came up with bupkis. Our own internal job and message queuing systems were astoundingly bad (and support for internal tools was almost entirely forbidden).
At megacorp I absolutely decried any sort of home grown solution. But if Dropbox were to actually tackle the problems of internal testing and support and come up with mature solutions, why shouldn't they use them?
Hi, I'm Kohsuke, the creator of Jenkins. I'm sorry to hear that you had a bad experience.
Would you be willing to letting me interview you so that I can learn where it failed your expectation? I'm honestly trying to learn where we can do things better, and often what's obvious to one person is completely incomprehensible to another. So I think this is a great opportunity for me to learn a fresh perspective.
Jenkins isn't a terrible experience, and I've used it personally and professionally (and would do so again where it's a good fit), but for my current project it missed a few of the requirements. In trying to rationalize the whole NIH thing, I talked to some friends and peers about their CI experiences. I got pretty consistent responses on Jenkins.
My relevant requirements:
1.) The software needs to be self-hostable and run on the BSDs. For the most part this narrows down the options to buildbot and the Java based CI options (Jenkins and GoCd). Travis could probably be run on FreeBSD, but the open source bits are essentially abandoned (e.g. some repos are missing) with no documentation. Nearly everything else these days is strongly tied to Linux via docker. Some free hosted services offer a FreeBSD target, but I'm looking to test on DF/Free/Net/OpenBSD.
2.) The software needs to scale down. The GoCd folks suggested that the agent would need around 500 MB of RAM. I haven't profiled Jenkins, but I can't imagine the agent being that much lighter weight. Certainly the Jenkins server process is glacially slow. By contrast my prototype in Rust is showing memory usage of under 5 MB for each process (agent + server). I expect that to grow a little but, but not by an order of magnitude.
3.) The software needs to handle multi-arch builds. Travis does this extremely well. Buildbot and GoCd, kinda. Jenkins does not handle this use case (e.g. pipelines + matrix builds are not supported). I really like the way Travis basically handles these as sub jobs.
My experience:
A.) The Jenkins documentation is terrible, if it exists at all. I've heard that this has been improved in the year or so since I've looked at Jenkins (but that hasn't been my experience). I mentioned this to one of the CloudBees guys at the DevOps Days conf I went to last year and got an ack that this is a known issue (although CloudBees has driven a ton of Jenkins documentation and improvement). At MegaCorp we paid a fortune to CloudBees, which helped a ton but didn't really help end users. I cannot understate just how much of a detriment the documentation is.
At the opposite end of the spectrum rust (except for the async stuff) and postgres are just a dream come true. If it's any consolation the GoCd documentation is pretty atrocious as well. Almost none of it is up to date with the current UI.
B.) The Jenkins community tends to cargo cult Jenkins-Groovy snippets like crazy, potentially as a result of #A. Having a good community helps documentation and helps when there are gaps in the documentation.
C.) Bootstrapping Jenkins is not something easily done in an automated way. The CLI is not stable and I had tons of trouble trying to get plugins and dependencies sorted without having to drop into the GUI. For homelab stuff I've automated bootstrapping of nearly everything except for Jenkins with Ansible.
I don't think these are new or unknown issues as in talking to friends and peers I've found that the typical responses regarding Jenkins are along the lines of: Jenkins works well enough so that we're not motivated to switch, but A & C are our main pain points.
Thanks for taking time to put this together. Yes, much of it isn't new, but it's always good to hear how these dots are connected in other people's view to form a theme.
I have not. The continuous integration infrastructure at megacorp was merely a symptom of some major cultural problems that over the course of a few years just weren't going to be addressed to my satisfaction, so I left. Paying for Cloudbees was a painful exercise, and I think the guy that was pushing for it ended up getting shitcanned. After all, the whole point of deploying Jenkins was to avoid paying for TeamCity.
For my personal stuff, I haven't looked at buildkite primarily because I'm looking for free, open source solutions. The secondary (but no less important) issue is that I'm also looking for coverage for *BSD. Buildkite lists FreeBSD support so, assuming it's not a Linux binary expected to run under the translation layer, they could presumably offer binaries for the other BSD variants. Right now I've got some buildbot workers setup on some bhyve VMs with df/net/openbsd and some jails with FreeBSD.
The big thing I've noticed is that lots of tools will list FreeBSD support at least, but few of those claims are actually tested. Even fewer mention any of the other BSDs. For example: Elixir deployments on FreeBSD have been broken for months; Hashicorp put out a FreeBSD build of their vault ssh helper that didn't work by design -- but they had no idea because they never tried to validate the build; rust's libc crate lists FreeBSD, NetBSD, and OpenBSD as tier 1 platforms, but they're only testing the crate on an old version of FreeBSD (where they've disabled some key tests due to a misunderstanding rendering the tests not super useful)... plus they're not in a good position to deal with the C ABI changing between releases.
The irony of people saying essentially "Dropbox could have done this just by using open source tools" on hackernews, where people famously announced the Dropbox would fail because it was easy to reproduce using open source tools entertains me.
Maybe, just maybe, the engineers in question considered mature open source tools and decided they didn't serve their needs.
That is the number 1 reason why it failed.