Jenkins supports the "master/slave" mode, where the workload of building projects are delegated to multiple "slave" nodes, allowing a single Jenkins installation to host a large number of projects, or to provide different environments needed for builds/tests. This document describes this mode and how to use it.
A "master" is an installation of Jenkins. When you weren't using the master/slave support, a master was all you had. Even in the master/slave mode, the role of a master remains the same. It will serve all HTTP requests, and it can still build projects on its own.
Slaves are computers that are set up to build projects for a master. Jenkins runs a separate program called "slave agent" on slaves. In other words, there is no need to install the full Jenkins (package or compiled binaries) on a slave node. There are various ways to start slave agents, but in the end a slave agent and Jenkins master needs to establish a bi-directional byte stream (for example a TCP/IP socket.)
When slaves are registered to a master, a master starts distributing loads to slaves. The exact delegation behavior depends on configuration of each project. Some projects may choose to "stick" to a particular machine for a build, while others may choose to roam freely between slaves. For people accessing Jenkins website, things works mostly transparently. You can still browse javadoc, see test results, download build results from a master, without ever noticing that builds were done by slaves. In other words, the master becomes a sort of "portal" to the entire build farm.
Follow the Step by step guide to set up master and slave machines to quickly start using distributed builds.
Pick the right method depending on your environment and OS that master/slaves run.
Jenkins has a built-in SSH client implementation that it can use to talk to remote sshd and start a slave agent. This is the most convenient and preferred method for Unix slaves, which normally has sshd out-of-the-box. Click Manage Jenkins, then Manage Nodes, then click "New Node." In this set up, you'll supply the connection information (the slave host name, user name, and ssh credential). Note that the slave will need the master's public ssh key copied to ~/.ssh/authorized_keys. (This is a decent howto if you need ssh help). Jenkins will do the rest of the work by itself, including copying the binary needed for a slave agent, and starting/stopping slaves. If your project has external dependencies (like a special ~/.m2/settings.xml, or a special version of java), you'll need to set that up yourself, though.
This is the most convenient set up on Unix. However, if your are on Windows and you don't have ssh commands with cygwin for example, you can use a tool like PuTTY and PuTTYgen to generate your private and public pair of keys.
For connecting to Windows slaves through cygwin sshd, see SSH slaves and Cygwin for more details.
For Windows slaves, Jenkins can use the remote management facility built into Windows 2000 or later (WMI+DCOM, to be more specific.) In this set up, you'll supply the username and the password of the user who has the administrative access to the system, and Jenkins will use that remotely create a Windows service and remotely start/stop them.
This is the most convenient set up on Windows, but does not allow you to run programs that require display interaction (such as GUI tests).
Note : Unlike other Node's configuration type, the Node's name is very important as it is taken as the node's address where to create the service !
If the above turn-key solutions do not provide flexibility necessary, you can write your own script to start a slave. You place this script on the master, and tell Jenkins to run this script whenever it needs to connect to a slave.
Typically, your script uses a remote program execution mechanism like SSH, or other similar means (on Windows, this could be done by the same protocols through cygwin or tools like psexec), but Jenkins doesn't really assume any specific method of connectivity.
What Jenkins expects from your script is that, in the end, it has to execute the slave agent program like java -jar slave.jar, on the right computer, and have its stdin/stdout connect to your script's stdin/stdout. For example, a script that does "ssh myslave java -jar ~/bin/slave.jar" would satisfy this.
A copy of slave.jar can be downloaded from http://yourserver:port/jnlpJars/slave.jar . Many people write scripts in such a way that this 160K jar is downloaded during the running of said script, to ensure that a consistent version of slave.jar is always used. Such an approach eliminates the slave.jar updating issue discussed below. Note that the SSH Slaves plugin does this automatically, so slaves configured using this plugin always use the correct slave.jar.
Launching slaves this way often requires an additional initial set up on slaves (especially on Windows, where remote login mechanism is not available out of box), but the benefits of this approach is that when the connection goes bad, you can use Jenkins's web interface to re-establish the connection.
Another way of doing this is to start a slave agent through Java Web Start (JNLP). In this approach, you'll interactively logon to the slave node, open a browser, and open the slave page. You'll be then presented with the JNLP launch icon. Upon clicking it, Java Web Start will kick in, and it launches a slave agent on the computer where the browser was running.
This mode is convenient when the master cannot initiate a connection to slaves, such as when it runs outside a firewall while the rest of the slaves are in the firewall. OTOH, if the machine with a slave agent goes down, the master has no way of re-launching it on its own.
On Windows, you can do this manually once, then from the launched JNLP slave agent, you can install it as a Windows service so that you don't need to interactively start the slave from then on.
If you need display interaction (e.g. for GUI tests) on Windows and you have a dedicated (virtual) test machine, this is a suitable option. Create a jenkins user account, enable auto-login, and put a shortcut to the JNLP file in the Startup items (after having trusted the slave agent's certificate). This allows one to run tests as a restricted user as well.
This launch mode uses a mechanism very similar to Java Web Start, except that it runs without using GUI, making it convenient for an execution as a daemon on Unix. To do this, configure this slave to be a JNLP slave, take slave.jar as discussed above, and then from the slave, run a command like this:
Make sure to replace "slave-name" with the name of your slave.
Also note that the slaves are a kind of a cluster, and operating a cluster (especially a large one or heterogeneous one) is always a non-trivial task. For example, you need to make sure that all slaves have JDKs, Ant, CVS, and/or any other tools you need for builds. You need to make sure that slaves are up and running, etc. Jenkins is not a clustering middleware, and therefore it doesn't make this any easier. Nevertheless, one can use a server provisioning tool and a configuration management software to facilitate both aspects.
This section describes Kohsuke Kawaguchi's set up of Jenkins slaves that he used to use inside Sun for his day job. His master Jenkins node ran on a SPARC Solaris box, and he had many SPARC Solaris slaves, Opteron Linux slaves, and a few Windows slaves.
Note that in the more recent Jenkins packages, the default JENKINS_HOME (aka home directory for the 'jenkins' user on Linux machines, e.g. Red Hat, CentOS, Ubuntu) is set to /var/lib/jenkins.
Some slaves are faster, while others are slow. Some slaves are closer (network wise) to a master, others are far away. So doing a good build distribution is a challenge. Currently, Jenkins employs the following strategy:
If you have interesting ideas (or better yet, implementations), please let me know.
Jenkins has a notion of a “node monitor” which can check the status of a slave for various conditions, displaying the results and optionally marking the slave offline accordingly. Jenkins bundles several, checking disk space in the workspace; disk space in the temporary partition; swap space; clock skew (compared to the master); and response time.
Plugins can add other monitors.
Administrators can manually mark slaves offline (with an optional published reason) or reconnect them.
Groovy scripts such as Monitor and Restart Offline Slaves can perform batch operations like this. There is also a CLI command to reconnect.
Then there is a background task which automatically reconnects slaves that are thought to be back up. The behavior is configurable per slave (or per cloud, if using cloudy provisioning for slaves) via a “retention strategy”, of which Jenkins bundles several (plugins can contribute others): always keep online if possible; drop offline when not in use; use a schedule; behave according to cloud’s notion of load.
Typically, you start with a master-only installation and then much later you add slaves as your projects grow. When you enable the master/slave mode, Jenkins automatically configures all your existing projects to stick to the master node. This is a precaution to avoid disturbing existing projects, since most likely you won't be able to configure slaves correctly without trial and error. After you configure slaves successfully, you need to individually configure projects to let them roam freely. This is tedious, but it allows you to work on one project at a time.
Projects that are newly created on master/slave-enabled Jenkins will be by default configured to roam freely.
One might consider make the Jenkins master accessible on the public network (so that people can see it), while leaving the build slaves within the firewall (typical reasons: cost and security) There are several ways to make it work:
Note that in both cases, once the master is compromised, all your slaves can be easily compromised (IOW, malicious master can execute arbitrary program on slaves), so both set-up leaves much to be desired in terms of isolating security breach. Build Publisher Plugin provides another way of doing this, in more secure fashion.
Using a well established virtualization infrastructure such as Kernel-based Virtual Machine (KVM), it is quite easy to run multiple slave instances on a single physical node. Such instances can be running various Linux, *BSD UNIX, Solaris, Windows. For Windows, one can have them installed as separate Windows services so they can start up on system startup. While the correct use of executors largely obviates the need for multiple slave instances on the same machine, there are some unique use cases to consider:
Follow these steps to get multiple slaves working on the same Windows box:
When you go to create the second node, it is nice to be able to copy an existing node, and copy the first node you setup. Then you just tweak the Remote FS Root and a couple other settings to make it distinct. When you are done you should have two (or more) Jenkins slave services in the list of Windows services.
Some interesting pages on issues (and resolutions) occurring when using Windows slaves:
Some more general troubleshooting tips:
Skip to end of metadata Go to start of metadata