Jenkins : JClouds Plugin

This plugin uses JClouds to provide slave launching on most of the currently usable Cloud infrastructures.

Plugin Information

View JClouds on the plugin site for more information.

Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:

JClouds Jenkins plugin provides option to launch jenkins slaves on any Cloud provider supported by JClouds, or on a cloud provider implementing one of the APIs JClouds supports.

Important information for upgrading from versions < 2.9 of this plugin

Due to the fact that all credentals ar now handled by the credentials plugin, the config data must be converted during upgrade and therefore, a rollback can not be performed easily.
If you want to presevere config data, just in case you want to switch back to 2.8.1 you must backup config.xml and credentials.xml from your jenkins' home before performing the update.
Then in case of rollback do the following:

  1. uninstall the plugin
  2. shutdown jenkins
  3. restore config.xml and credentials.xml
  4. startup jenkins
  5. install the old plugin version

Adding a new Cloud Provider

  • Goto Jenkins Configuration page by clicking on Manage menu or browsing to the URL http://localhost:8080/configure
  • Scroll down to Cloud Section
  • Click on the `Add a new cloud` pop-up menu button which should have an option - `Cloud (JClouds)`
  • Click on `Cloud (JClouds)`
  • Fill in the configuration options
    • Profile : the name of the profile e.g, aws-slave-profile
    • Provider Name: Select from the supported providers
    • End Point URL: if your provider API needs an endpoint configuration, add it here, otherwise leave this empty.
    • Max Number of Instances: The maximum number of instances to run from this cloud at one time.
    • Retention Time: How long, in minutes, to wait for a slave to remain idle before disconnecting and terminating it. Defaults to 30.
    • Credentials: Depending on your provider, you either need a username/password pair or a username/privatekey pair. The first one
      is covered by the "Username with password" credentials of jenkins and for the second one, you simply can use the
      "SSH Username with private key" credentials which - despite of what their name suggests - can store any kind of PEM-encoded private key. For
      GCE, select the "JClouds Username with key" credential type and then click the Browse button in order to upload your JSON key from google.
      (In case of openstack-nova keystoneV2, the username is to be specified in the form tenantName:userName).
      New since 2.15 is support for OpenStack keystone-V3: Select "OpenStack keystone V3 credentials" and fill all the fields (The Domain usually is named Default)
    • RSA Private Key/Public Key: Like with the credentials, simply use a "SSH Username with private key" credential. The public key is derived automatically from that.
    • Click on `Test Connection` to validate the cloud settings.
  • Add Cloud Instance Template by clicking on the Add button
  • Fill in configuration options:
    • Name : the name of the instance template e.g. aws-jenkins-slave
    • Number of Executors: How many executors each slave created from this template should have.
    • Description: notes/comments for your reference.
    • Image ID: Use one of the following options:
      • Image ID to use for this slave template, such as EC2 AMIs. Note that EC2 AMIs must include the region as well, e.g., us-east-1/ami-00000. If unshure, you can enter a partial string and the hit the "Check Image Id" button. This searches for matching images and if there are ambiguties, a
        message is shown like: "Did you mean "....exact-image-id..."?
        or
      • OSFamily: Specify the OSFamily - leave empty for default for a cloud provider
      • OS Version : Specify the OSVersion - leave empty for default for a cloud provider
    • Hardware ID: Use one of the following options:
      • Hardware ID on provider for this slave template, such as t1.micro on AWS EC2.
        or
      • RAM : in MB
      • No. of Cores: number of virtual processor cores.
    • Location ID: Location ID where this slave will be deployed. If none is selected jclouds will automatically choose an available one.
    • Labels: (space-separated) labels/tags that you can use to attach a build to this slave template.
    • Init Script: A shell script to be run when the slave is created. A rather simple method for provisioning. If supported by your provider, you should
      prefer the variant using User-Data and cloud-init.
    • Stop on Terminate: If true, suspend slaves rather than terminating them.
  • Click Save to save the configuration changes.
  • Goto Jenkins' home page, click on `Build Executor Status` link on the sidebar.
  • Verify that you have a button with `Provision via JClouds - (YOUR PROFILE NAME) drop down with the slave template name you configured.
  • Click on the slave and see if your slave launched successfully (please wait until the operation completes).

Executing build on the slave

  • To run your build on the newly configured slave computer, just enable the `Restrict where this project can be run` option in the build configuration page.
  • Enter the label which you choose for the instance template in the `Label Expression` text field. This should auto-complete labels for you.
  • Click save to save the configuration options.
  • Schedule the build to check whether the build is executed on the selected slave template.
  • If your cloud provider is charging by the minute (GCE for example), you can enable the checkbox "JClouds Single-use-slave" which destroys the provisioned
    VM after the build job has finished (with a small delay of ~ 1min).

Adding a Blobstore Profile for storing build artifacts

The plugin also provides a way to store your build artifacts on JClouds supported cloud storage providers. You can configure multiple
blobstore profiles and configure the build to copy different files/artifacts to the specified container. Here's how you configure the same.

  • Goto Jenkins Configuration Page
  • Click Ad under the section `JClouds Cloud Storage Settings`
  • Provide the configuration Options:
    • Profile Name: name of the profile e.g. aws-storage
    • Provider Name: Select one of the supported providers.
    • Credentials: Just like the cloud credentials.
  • You can add multiple providers by clicking on Add.
  • Click Save on the bottom of the page, to save the settings.

Publishing artifacts after a build

After you configure a cloudstorage provider, you can enable the publishing file by enabling it under `Post-build Actions` in the build job configuration page.

  • Click on the checkbox `Publish artifacts to JClouds Clouds Storage`
  • You should now see a dropdown with configured storage profiles. Select the one you want to use for this build.
  • Click on Add button next to `Files to upload`.
  • Add the sourced file path (relative to workspace)
  • Add the destination container name.
  • Repeat to add more files if you want to copy.
  • Click save.

When the build is complete and successful, the configured files will be published to the configured blobstore.

Merged cloud-init YAML definitions

When using cloud-init you can provide multiple config data snippets. If you select YAML snippets, thos get merged. E.g:

If snippet 1 contains:

packages:
  - screen
  - openjdk-8-jdk-headless

and snippet 2 contains:

packages:
  - gcc

Then the resulting YAML on the cloud-init (server) side becomes:

packages:
  - gcc
  - screen
  - openjdk-8-jdk-headless

Using the new phone-home feature

When provisioning slaves, there might be too much work on a slave for it to get ready (listening on port 22) in time for jenkins.
Therefore, the plugin provides a new webhook, which is designed to be invoked by a http POST request using cloud-init within the slave when everything is ready to use. When enabled, the usual slave connection setup is delayed until the http POST is received.

  • The corresponding cloud-init configuration looks like this:

    #cloud-config
    phone_home:
      url: http://your.jenkins.url/jclouds-phonehome/
      tries: 3
    
  • You can put this into the User data field in the slave template for example.

Changelog

Version 2.15 (released August 6, 2019)

  • Fixed SECURITY-1482 / CVE-2019-10368 (CSRF) and CVE-2019-10369 (permission check). [Advisory]
  • Suppress binary userData in logs (JENKINS-41989)
  • Improve help for userData
  • Implement workaround for broken cloudbase-init mime handling
  • Added support for cloud statistics plugin (now a mandatory dependency)
  • Added packet provider (Thanks to Ignasi Barrera)
  • Added profitbricks provider (thanks to Ali Bazlamit)
  • Fixed GUI problem (JENKINS-48986) (thanks to Tomasz Wojtun)
  • Upgrade to jclouds-2.1.0, (Fixes JENKINS-42005)
  • Added support for OpenStack keystone-V3 authentication

Version 2.14 (released February 14, 2017)

Version 2.13 (released February 13, 2017)

  • Fixed class loading problem with bouncycastle resulting in InvalidKeySpecException: key spec not recognised. (JENKINS-41186)

Version 2.12 (released December 16, 2016)

  • New template config option "Preferred Address" solves problem with multi-homed slaves. (JENKINS-40342)
  • Use gzip compression of userData on supporting cloud providers; (currently: aws-ec2, openstack-nova and openstack-nova-ec2)
    In the logs, the userData is shown as binary "garbage" if compression is in effect. Since that particular log entry is generated
    from within jclouds, that cannot be changed and works as designed.

Version 2.11 (released December 5, 2016)

  • Upgrade to jclouds-2.0 (Among many improvents, this adds official support for GCE (Google) and DigitalOcean2)
  • Do not send null userData (JENKINS-21108)
  • Generate artificial suffix for Instances on EC2 (JENKINS-19935)
  • Added missing license (JENKINS-40028)
  • Now shows IP address on node details page (JENKINS-14398)
  • Now uses the config-file-provider-plugin for initScripts and userData
  • Allows for multiple cloud-config snippets in userData
  • Merged OpenStack's assignFloatingIP setting into assignPublicIp (same semantic)
  • Improved automatic migration when upgrading the plugin. Now, upgrades from v2.8.1-1
    should go seamlessly.

Version 2.10 (released November 18, 2016)

  • Set Content-Length in blobstore requests (JENKINS-23049)
  • Generally improved blobstore error handling.
  • Blobstore publisher now can be further configured (JENKINS-22178)
  • Added missing dependency on ssh-credential plugin
  • Added config drive flag (JENKINS-23454)
  • Improved deletion of expired slave nodes (JENKINS-27471 and JENKINS-28403)
  • Supplmental instances now properly use phone home feature
  • Properly destroy stale instances after a forced reset/shutdown of jenkins

Version 2.9 (released November 9, 2016)

  • All credential related settings now use the credential-plugin
  • New webhook for synchronizing slave availability (aka. phone-home for slaves)
  • Added CLI commands (jclouds-templates, jclouds-provision)
  • Fixed several potential XSS issues
  • Upgrade to jclouds-1.9.2
  • Safer cleanup of expired slaves
  • Added many missing help texts
  • Fixed classloader isolation problem with jenkins 2.29 (JENKINS-39505)
  • Upgrade to ne credentials API (JENKINS-35560)
  • Fixed reconnection to running slaves after jenkins restart (JENKINS-30556)
  • Added workaround for image cache (JENKINS-28815)
  • Fixed NPE when setting retention time (JENKINS-29136)
  • Fixed floating ip bug on openstack (JENKINS-26083)
  • Fixed scheduling of matrix jobs and added ability to configure node usage mode (JENKINS-25865)
  • Fixed handling of OpenStack security groups (JENKINS-22587)
  • Added validation of various number fields

Version 2.8.1-1 (released December 2, 2015)

  • Add flag for copying and executing slave.jar on Windows images

Version 2.5.1 (released March 5, 2014)

  • Switch back to Java 1.6
  • Jclouds version updated to 1.7.1
  • Allow shading guava and jclouds
  • Added the DigitalOcean provider
  • Added an option to let users install private key in the slave
  • Added option to specify networks requirements for any provider

Version 2.5 (released February 5, 2014)

  • Added OpenStack zones parameter to restrict spawning cloud slaves to them
  • Added capability to run slaves by specifying image name regexp instead of image ID
  • Bumped Java version to 1.7
  • Credential field is now password type
  • JCloudSlave now inherited from AbstractCloudSlave

Version 2.4 (released August 27, 2013)

  • Bump up jclouds to 1.6.2-incubating.
  • Allow the user to not allocate a public IP address on CloudStack. (JENKINS-18461)
  • Various OpenStack fixes
  • Add support for user data and persisting credentials.
  • Various blobstore improvements
  • Don't print auth stack trace (JENKINS-16632)
  • Add forcible synchronized serial delay option (JENKINS-15970)

Version 2.3 (released August 31, 2012)

  • Bump up jclouds to 1.5.0-beta.11.
  • Add the ability to override retention time for an individual template.
  • Now able to specify "Launch Instance" template as a string with parameter replacement enabled.

Version 2.2.2 (released July 18, 2012)

  • Bump up jclouds to 1.5.0-beta.7.
  • Use Remote FS Root for Jenkins user home. (JENKINS-14396)

Version 2.2.1 (released July 11, 2012)

  • Bump up jclouds to 1.5.0-beta.6.

Version 2.2 (released June 22, 2012)

  • Based on JClouds 1.5.0-beta.4, adding support for a bunch of new cloud providers, including CloudStack 3.x, Greenqloud, and more.
  • Added Single Use Slave option - if enabled, the slave will only be used for one build and then deleted.
  • Dramatically improved performance of slave deletion, which should also help avoid lockups of Jenkins itself.

Version 2.1.1 (released May 24, 2012)

  • Fixed a showstopper bug that actually broke everything UI-related. whoops!

Version 2.1.0 (released May 23, 2012)

  • Bumped to JClouds 1.5.0-beta.1.
  • Initial support for spinning up non-slave instances as part of a build.
  • Don't create a Jenkins user on the slave if one already exists.
  • Add a configurable timeout for slave init.
  • Improved logging.

Version 2.0.4 (released May 1, 2012)

  • Hardware ID wasn't being used.
  • Got rid of excess persisted NodeMetadata from slaves.
  • Cleaned up slave template UI.
  • Added option for infinite retention with retention time of -1.

Version 2.0.3 (released Apr 26, 2012)

  • Modified package manifest to mask older guava jars from plugin's classloader, so that the plugin will work consistently on pre-1.463 Jenkins releases.

Version 2.0.2 (released Apr 25, 2012)

  • First real release.
  • Support for multiple clouds on a single Jenkins instance.
  • Auto-provisioning functional.

Attachments: