Jenkins : Swarm Plugin

Plugin Information

View Swarm on the plugin site for more information.

This plugin enables slaves to auto-discover nearby Jenkins master and join it automatically, thereby forming an ad-hoc cluster.
This plugin consists of two pieces:

  1. A self-contained CLI client that discovers a nearby Jenkins (via a UDP broadcast) and joins it
  2. A plugin that needs to be installed on Jenkins master to accept swarm clients

The primary use case of this came from the idea of a "Jenkins slave appliance" — a virtual machine pre-packaged with the swarm client CLI — that someone can just grab and run.

With the swarm client, a person who's willing to contribute some of his computing power to the cluster just needs to run this virtual machine, and the cluster automatically gets additional resources. Because it's packaged as a VM, there is no need to worry about the builds/tests interfering with the host system or altering its settings unexpectedly.

Usage

  1. Install the swarm plugin from the Update Center
  2. Download the CLI agent from here.
  3. Run the CLI agent with java -jar path/to/swarm-client.jar. There are no required command-line options; run with the -help option to see the available options.

Documentation

Available Options

$ java -jar swarm-client.jar --help
 -autoDiscoveryAddress VAL              : Use this address for UDP-based
                                          auto-discovery (default
                                          255.255.255.255)
 -candidateTag VAL                      : Show swarm candidate with tag only
 -deleteExistingClients                 : Deletes any existing slave with the
                                          same name.
 -description VAL                       : Description to be put on the slave
 -disableClientsUniqueId                : Disables client's unique ID.
 -disableSslVerification                : Disables SSL verification in the
                                          HttpClient.
 -executors N                           : Number of executors
 -fsroot FILE                           : Directory where Jenkins places files
 -help (--help)                         : Show the help screen
 -labels VAL                            : Whitespace-separated list of labels
                                          to be assigned for this slave.
                                          Multiple options are allowed.
 -labelsFile VAL                        : File location with space delimited
                                          list of labels.  If the file changes,
                                          the client is restarted.
 -master VAL                            : The complete target Jenkins URL like
                                          'http://server:8080/jenkins/'. If
                                          this option is specified,
                                          auto-discovery will be skipped
 -maxRetryInterval N                    : Max time to wait before retry in
                                          seconds. Default is 60 seconds.
 -mode MODE                             : The mode controlling how Jenkins
                                          allocates jobs to slaves. Can be
                                          either 'normal' (utilize this slave
                                          as much as possible) or 'exclusive'
                                          (leave this machine for tied jobs
                                          only). Default is 'normal'.
 -name VAL                              : Name of the slave
 -noRetryAfterConnected                 : Do not retry if a successful
                                          connection gets closed.
 -password VAL                          : The Jenkins user password
 -passwordEnvVariable VAL               : Environment variable that the
                                          password is stored in
 -passwordFile VAL                      : File containing the Jenkins user
                                          password
 -pidFile VAL                           : File to write PID to
 -retry N                               : Number of retries before giving up.
                                          Unlimited if not specified.
 -retryBackOffStrategy RETRY_BACK_OFF_S : The mode controlling retry wait time.
 TRATEGY                                  Can be either 'none' (use same
                                          interval between retries) or 'linear'
                                          (increase wait time before each retry
                                          up to maxRetryInterval) or
                                          'exponential' (double wait interval
                                          on each retry up to maxRetryInterval).
                                          Default is 'none'.
 -retryInterval N                       : Time to wait before retry in seconds.
                                          Default is 10 seconds.
 -showHostName (--showHostName)         : Show hostname instead of IP address
 -sslFingerprints VAL                   : Whitespace-separated list of accepted
                                          certificate fingerprints
                                          (SHA-256/Hex), otherwise system
                                          truststore will be used. No
                                          revocation, expiration or not yet
                                          valid check will be performed for
                                          custom fingerprints! Multiple options
                                          are allowed.
 -t (--toolLocation)                    : A tool location to be defined on this
                                          slave. It is specified as
                                          'toolName=location'
 -tunnel VAL                            : Connect to the specified host and
                                          port, instead of connecting directly
                                          to Jenkins. Useful when connection to
                                          Jenkins needs to be tunneled. Can be
                                          also HOST: or :PORT, in which case
                                          the missing portion will be
                                          auto-configured like the default
                                          behavior
 -username VAL                          : The Jenkins username for
                                          authentication

FAQ

Error configuration : "xxx.xxx.xxx.xxx doesn't have the URL configuration yet"

This error indicates that the Jenkins master running at the given URL has never configured, and therefore the swarm client cannot determine its HTTP URL. To fix this problem, please go to the system configuration screen of this Jenkins, and click "submit".

Changelog

Version 3.17 (June 2, 2019)
Version 3.16 (May 21, 2019)
  • Swarm Plugin
    • Update minimum Jenkins core requirement to 2.60.3 (#87)
    • Remove unnecessary field (#85)
  • Swarm Plugin Client
    • Disable DTDs completely in all XML parsers to prevent XML External Entity (XXE) attacks (#84)
    • Update Jenkins Remoting version from 3.28 to 3.30 (#78, full changelog)
Version 3.15 (2018 Dec 12)
  • Swarm Plugin
    • PR #75 - Fix the label removal in the "removeLabels" endpoint
  • Swarm Plugin Client
    • Update Jenkins Remoting version from 3.26 to 3.28 to pick up new stability fixes (full changelog)
Version 3.14 (2018 Sep 04)
  • Swarm Plugin Client
    • Update Remoting from 3.21 to 3.26 in order to pick new stability fixes (full changelog)
  • Swarm Plugin
    • Update Swarm Plugin Client to 3.26
Version 3.13 (2018 Jun 08)
  • Swarm Plugin
    • Update minimal Jenkins Core requirement to 2.60.1
  • Swarm Plugin Client
    • Update Remoting to 3.21 to pick logging and "no_proxy" handling fixes
Version 3.12 (2018 Mar 22)
  • Swarm Plugin
    • No changes
  • Swarm Plugin Client
Version 3.11 (2018 Mar 19)
  • Swarm Plugin
    • No changes
  • Swarm Plugin Client
    • JENKINS-50252 - Update Remoting from 3.16 to 3.18 to pick up bug fixes and serialization diagnosability improvements
    • PR #68 - The plugin now trims input strings for password files specified in "-passwordFile"
Version 3.10 (2018 Feb 21)
  • Swarm Plugin
    • PR #62 - Add ability to download the client directory from the plugin installed in Jenkins
      • URL: ${JENKINS_URL}/swarm/swarm-client.jar
  • Swarm Plugin Client
    • No changes
Version 3.9 (2018 Feb 07)
  • Swarm Plugin
    • JENKINS-49292 - Reduce log level from ALL to INFO in sample logging.properties to reduce log spam
  • Swarm Plugin Client
    • PR #66 - Add support of the "–passwordFile" option
Version 3.8 (2018 Jan 10)
Version 3.7 (2017 Dec 22)
  • Swarm Plugin:
    • No changes
  • Swarm Plugin Client: 
    • PR #63 - Require Java 8 (client-side only)
    • PR #63 - Update Remoting from 3.10.2 to 3.15
    • PR #61 - Prevent the infinite reconnect cycle in Remoting Launcher, use the client's failover logic instead
Version 3.6 (2017 Oct 18)
  • Update Remoting in Swarm Client from 3.4.1 to 3.10.2
  • PR #55 - Introduce the "-pidFile" option, which creates a file with the process PID
    • Errata: The current implementation may cause File descriptor leaks in edge cases
  • JENKINS-43674 - Prevent NullPointerException in Swarm client in HTTPS mode without -disableSslVerification or -sslFingerprints
  • JENKINS-42098 - Prevent LinkageError when building a Maven project on a swarm node with new Maven versions
Version 3.5 (2017 Oct 11)
  • SECURITY-597 - Swarm Plugin Client bundled version of the commons-httpclient library, which was vulnerable to MiTM
Version 3.4 (2017 Apr 10)
  • Add option -sslFingerprints providing a possibility to add custom SSL trust anchors without adding them to the system store.
Version 3.3 (2017 Feb 10)
  • Finally a release!
  • Added -logFile and -labelsFile options. Now supports dynamic labels
  • Added support for very large numbers of dynamic labels when using -labelsFile
  • Remove consecutive slashes in plugin urls
  • docker-compose configuration updates
  • Add retry backoff strategy
  • Bumped remoting library to same as Jenkins LTS at the moment
  • Updates to make build and testing pass with new Jenkins plugin parent pom work
Version 3.2 (2017 Feb 8)
  • Failed to release due to INFRA-588
Version 3.1 (2017 Feb 8)
  • Failed to release due to INFRA-588
Version 3.0 (2016 Dec 27)
  • Failed to release due to INFRA-588
Version 2.3 (2016 Nov 28)
  • Failed to release due to INFRA-588
Version 2.2 (2016 July 26)
  • Failed to release due to INFRA-588
Version 2.1 (2016 May 20)
  • Implement #28917 -  Update remoting to one supported by latest LTS
  • MESOS_TASK_ID used as Jenkins slave ID if available as environment variable (for Mesos/Marathon integration).
  • Updating jenkins remoting dependency. Swarm client now matches the jenkins-remoting version in Jenkins-1.625.3 LTS
  • Implement #34593 -  add an option to delete existing clients
  • Add integration test environment based upon Docker compose
Version 2.0 (2015 Aug 3)
Version 1.26 (2015 Jul 21)
  • Re-release of 1.25, some artefacts was not properly deployed. 
Version 1.25 (2015 Jul 21)
  • Correct #29232 - Set the HTTP Connection:close header to ensure the underlying socket is closed (see pull request 29)
  • Add a markdown formatted README to better describe the project for Github viewers
  • Improve end user reporting of hostname lookup errors (see pull request 30)
  • Make javadoc compile with JDK 8
Version 1.24 (2015 Apr 28)
Version 1.23 (2015 Apr 27)
Version 1.22 (2014 Nov 28)
Version 1.21 (2014 Nov 6)
  • Instead of constructing the tool location key, just use the existing descriptor (#25064, see pull request 20)
  • Use latest Jenkins LTS remoting library (1.580.1 Jenkins LTS version)
Version 1.20 (2014 Oct 8)
Version 1.19 (2014 Oct 6)
  • Correct bug introduced by 1.18 where the client did not work _unless_ you set tool locations (#24995, see pull request 18)
Version 1.18 (2014 Oct 2)
Version 1.17 (2014 Sep 30)
  • Add -noRetryAfterConnected and -retry options. These provide optional exit strategies for the default unlimited retry loop.
  • Require a well-formed master URL, ensuring trailing slash
  • [FIXED JENKINS-21892] Update swarm client to send CSRF token
  • Use latest releases of commons-codec, args4j, and remoting
Version 1.16 (2014 Jul 1)
Version 1.15
  • undocumented, or maybe a typo of 1.12?
Version 1.12, 1.11 (2014 Jan 15)
Version 1.10 (2013 Oct 21)
  • swarm 1.9 can't connect to current LTS as slave.jar too old (JENKINS-20138).
Version 1.9 (2013 May 18))
  • Add option for specifying Node.Mode(pull #3).
Version 1.8 (2012 Nov 21)
  • Changing broadcast to send a UDP packet payload of 128 bytes instead of 0
  • Allow slave connection without requiring UDP
  • Adds disableSslVerification option
Version 1.6 (2012 Mar 18)
  • fixed references from Hudson to Jenkins.
  • Swarm Client fails to connect to Jenkins when Authentication is enabled but Authorization is diabled (JENKINS-11663).
  • support remoting 2.12.
Version 1.5 (2011 Aug 11)
  • check whether user has SlaveComputer.CREATE permission.
  • allow authentication in swarm plugin (JENKINS-5504).
Version 1.4 (2010 Aug 14)
  • Fixed broken help links.
  • Node properties save correctly.
  • Added Japanese localization.
Version 1.3 (2010 Jan 14)
Version 1.2 (2009 Dec 30)
  • Minor text correction
Version 1.1 (2009 Jul 15)
  • Added the -master option.
Version 1.0 (2009 May 23)
  • Initial release