This plugin uses Gearman to support multiple Jenkins masters.
** This project is hosted on git.openstack.org **
** The issue tracker is on storyboard **
We on Openstack infrastructure team use Jenkins extensively. Our jenkins servers, at peak load, runs 20,000+ jobs per day. At that load we require many jenkins slaves (900+) to process all of those build jobs. We have found that our requirement was pushing Jenkins beyond it's limits therefore we've decided to create the Gearman Plugin to support multiple Jenkins masters. The gearman plugin was designed to support extra slaves, allow load balancing of build jobs, and provide redundancy.
Jenkins core does not support multiple masters. You can setup multiple Jenkins masters but there is no coordination between them. One problem with scheduling builds on Jenkins master (“MasterA”) server is that MasterA only knows about its connected slaves. If all slaves on MasterA are busy then MasterA will just put the next scheduled build on the jenkin server queue. Now MasterA needs to wait for an available slave to run the build. This will be very in-efficient if your builds take a long time to run. So.. what if there is another Jenkins master (“MasterB”) that has free slaves to service the next scheduled build on the server's queue? Your probably saying.. “Then slaves on MasterB should run the build instead of waiting for slaves on MasterA to run them”, then I would say "good thought!". However MasterB will never service the builds on MasterA's queue. The client that schedules the builds must know about MasterB and then schedule builds on MasterB. This is what we mean by lack of coordination between masters. This gearman-plugin attempts to fill the gap.
This plugin integrates Gearman with Jenkins and will make it so that any Jenkins slave on any Jenkins master can service a job in the queue. This plugin will essentially replace the Jenkins (master) build queue with the Gearman job queue. The job will stay in the Gearman queue until there is a Jenkins node (master or slave) that can run that job. The gearman job queue is shared by multiple jenkins masters therefore gearman can hand out jobs to the next available slave on any jenkins master.
This assumes some familiarity with Jenkins and Gearman
Here's the corresponding Jenkins master UI:
A gearman client can be written in any language. Here are a few sample clients that work with this plugin
To execute a Jenkins job the gearman client just needs to provide the Gearman hostname, port, function, and UUID to start a jenkins build.
A Gearman request can stop/abort a jenkins build.
The job is stopped differently depending on the current state of the job. The table below explains the state, transitions and when cancellations happen.
You can send a gearman request to update a build's description. To do this you pass in the following parameters: name of project, build number, description.
Our infrastructure employees many 'single use slaves' so what we like to do is run a job and then immediately set the slave offline. You can do this by passing in the parameter 'OFFLINE_NODE_WHEN_COMPLETE'
To configure the gearman plugin to work with multiple Jenkins masters you will need to do following:
Now multiple Jenkins masters will be able to service the same jobs.
The typical workflow for this configuration is something like this:
Instructions to make the gearman plugin send log messages to the Jenkins logger:
Now you should see logs from gearman plugin.
The above images just show how the plugin might work in a simple case. To see the plugin used in production check out openstack jenkins servers, yes that's servers with an s:
All of the above masters use this plugin which means all of them can run any jobs that are sent to gearman server. We have lots of documentation on how we run the system in production.
Skip to end of metadata Go to start of metadata