Plugin Information |
---|
View OpenStack Cloud on the plugin site for more information. |
Older versions of this plugin may not be safe to use. Please review the following warnings before using an older version:
Essentials
In order to provision new nodes when Jenkins load goes high, administrator needs to configure Cloud and one or more Templates on global configuration page. Cloud represents connection to particular OpenStack tenant (project). In order to use several openstack instances or tenants, it is necessary to configure several Clouds in Jenkins. Template can be seen as a definition of particular kind of node. Template is the primary holder of all the attributes for OpenStack machine to be provisioned (image, flavor, etc.) as well as for Jenkins node (number of executors, etc.). Note that the same set of attributes can be configured on Cloud level as well. Such configuration will be then used as default for its templates. For example, if all nodes are supposed to use the same key-pair there is no need to state it in every template, it can be configured on cloud level and leave the filed blank in the templates.
Aside from machine/node attributes, every template require name and labels to be configured. Name will serve both as an identifier of the template as well as a name prefix for Jenkins node and OpenStack machine (that is why some limitations apply here). Labels field expects a set of Jenkins labels that will be assigned to all nodes given template provision. It will also be used to determine which cloud and template to use to process Jenkins load. When there is a build with no label requirements (Restrict where this project can be run on job page), any template can be used to provision the node. Build with label restriction can trigger provisioning only on templates with matching label set. The attributes at template level will inherit all global values (the value in effect is printed under the field). In case required field do not have a default nor current value, it will be reported.
User data
Every template can declare user-data script to be passed to CloudInit to customize the machine that is provisioned. Note that the image needs to support cloud init explicitly.
Before the script is sent to OpenStack, plugin injects several values using ${VARIABLE_NAME
} syntax:
JENKINS_URL
: The URL of the Jenkins instanceSLAVE_JAR_URL
: URL of the slave.jar - the agent executableSLAVE_JNLP_URL
: The endpoint URL for JNLP connectionSLAVE_JNLP_SECRET
: The JNLP 'secret' key. This key authorizes to connect Jenkins agent process to a Jenkins computer. Note that referencing this in server user-data (and then using it to launch agent process) makes it exposed to any user or automation permitted to access the Jenkins agent (including running builds). Also, when OpenStack deployment uses metadata service, which it often does, it must be deployed in a way this is guaranteed not to leak. See OSSN-0074 for further details.SLAVE_LABELS
: Labels of the corresponding Jenkins computer
Reporting
Openstack plugin utilizes Cloud Statistics Plugin that captures failures and time trends of past provisioning attempts.
Configuration As Code
Since version 2.46, JCasC is supported. Example:
Tips and tricks
- User can manually provision slave from particular cloud/template on Manage Jenkins > Manage Nodes.
- Deleting slave provisioned by OpenStack plugin will merely put it temporarily offline, the actual disposal of the slave and machine is performed with delay. That means that user have the possibility to abort that operation putting the slave back online. But most importantly, user can suggest plugin to discard the slave while still occupied by running build this way.
- Plugin identifies OpenStack image/snapshot to provision by its name. The image can be updated/replaced in openstack and the plugin will not require update provided its name have not changed.
- In case maximal instance number is specified on template level, the stricter of the two (global and template value) will be applied. Global value exists to ensure that the number of machines provisioned by plugin will not exceed certain limit. The template value can further restrict that at most N machines of given kind can be utilized at the same time.
- Plugin can report maximal number of instances was reached while there is not adequate number of Jenkins nodes. It is because plugin inspects running OpenStack machines in order to cover machines being provisioned/deleted (that do not have Jenkins nodes) and instances that plugin failed to delete. In case the instance get leaked, please report that as a bug with all relevant FINE level logs attached.
- On every slave provisioned by the plugin, there is an environment variable
OPENSTACK_PUBLIC_IP
declared with the IP address allocated for the machine.
Troubleshooting
Accessing logs
Plugin uses INFO
and above levels to report things user should worry about. For debugging, set it to FINEST
- note the ALL
level is not sufficient to print these. To configure OpenStack plugin logging in Jenkins UI go to Manage Jenkins > System Log > New Log Recorder and use jenkins.plugins.openstack.compute
as the logger name.
Access openstack client from groovy console
Use Jenkins.instance.clouds[0].openstack.@clientProvider.get()
to access the openstack4j client. Users are discouraged to use this anything else but querying the openstack (otherwise there is no way to ensure plugin will work correctly). For older versions fo the plugin use Jenkins.instance.clouds[0].openstack.@client
.
User Data / Cloud init is not evaluated
The image might not support it at all or can fail executing it. Check machine log in OpenStack for further details. Note that for now, there is no guarantee the script will complete before machine is connected to Jenkins and builds are started.
Changelog
Version 2.47
- Issue #254: Improve error reporting during manual provisioning
- Do not count connection time into idle time
Version 2.46.1
Jenkins Configuration as Code (JCasC) is not supported for older releases.
- Correctly handle username in Configuration as Code (JCasC)
Version 2.46
- Configuration as Code (JCasC) support introduced
Version 2.45
- Show correct time in "Pending deletion" messages
- Rework handling of
retentionTime == 0
scenarios
Version 2.44
- Prevent nodes to be deleted while provisioning where
retentionTime == 0
- Warn users JNLP secret in cloud-stats is inherently insecure
- Consult Jenkins computer count for maintaining maximal instance capacity
- Stop tracking servers to delete when cloud is deleted or renamed
Version 2.43
- Use upstream/unpatched openstack4j again
- Use 20 seconds socket timeout when taking to OpenStack
- Invalidate OpenStack client instance when password changes
- Add support for SSH launching over Ipv6
Version 2.42
- Make sure the user id is preserved for manually provisioned nodes
- Fix manual provisioning with multiple clouds configured
Version 2.41
- Prevent provisioning failure when VolumeSnapshot has no description.
- Mark a node used only after a task has complete
- Further fixes ware delivered for use-case where
retentionTime == 0
in later versions
- Further fixes ware delivered for use-case where
Version 2.40
Version 2.39
- Revert glance v2 switch to avoid the regression from 2.38
Version 2.38
- Switched to OpenStack Glance v2 breaking deployments with more than 25 images - use 2.40 or newer
- Use human readable description for server flavor sizes
- Do not wait for provisioning to complete in http handling thread
- Make sure slave with 0 retention time will not be reused
- Interrupt matrix parents when killing computer, too
- Relax template/cloud name restrictions
Version 2.37
- Connection leak regression originally fixed in 2.27 - use 2.38 or newer
- Fix problems introduced by security patch in 2.36
Version 2.36
Version 2.35
- Add support for enforcing minimal number of nodes running per template
Version 2.34
- Add support for provisioning servers in multiple networks
- Avoid chatty logging for periodic work
Version 2.33
- This plugin requires Java 8 from this release on
- Improve diagnostics in case of failed agent launch
- Improved Jenkins orphanned computer detection
Version 2.32
- Support booting to volume
- Rework computer termination to prevent repeated Channel#close
- Note this does not contain the fix for ContainX/openstack4j#1151
Version 2.31.1
- Add hotfix for ContainX/openstack4j#1151
Version 2.31
Version 2.30
- Add support for project domains
- Add ability to skip ssl check
- Report computer "fatal" offline cause when present when destroying computer
Version 2.29 (2017-10-20)
- Fix regression in region handling
- Do not report failed FIP deletion in cloud statistics if failed with 404
- Add support for volumeSnapshots
- Do not use expired login sessions
Version 2.27 (2017-10-03)
Improve reporting of boot timeout
Abort provisioning/launching when server gets deleted
Add support for explicit java path when SSHLauncher is used
Bump openstack4j okhttp connector to avoid occasional connection leaks
Avoid phony cloud-stats warnings logged
Version 2.26
Botched release - changes went to 2.27
Version 2.25 (2017-09-25)
Fix #168: Prevent tracking disposal of the same server several times
Use ok-http to prevent connection blockage (prefer 2.27 with followup fix)
Refactor slave type into describable
Version 2.24 (2017-08-17)
Note this version is affected by httpclient connection leak - use 2.27 instead.
Pipeline step for agentless node provisioning
- Prefer IPv4 address for SSH launcher
Prevent occasional IllegalArgumentException: Failed to instantiate class jenkins.plugins.openstack.compute.SlaveOptions while saving global configuration page
- First attempt to implement Openstack client caching between requests
- This version is affected by occasional httpclient connection blockage
Version 2.23
Issue #128: Do not fail when FIP service is disallowed (403) on paths that does not require floating IP (introduced in 2.21)
- This version is affected by occasional httpclient connection blockage
Version 2.22
- Investigate
SSH channel is closed
/No route to host
(Issue #149)
Version 2.21
Fix #148: Skip unknown variables in user data
Record manual provisioning attempt failure for unexpected exceptions
Issue #128: Request: Allow VMs without floating IPs (followup fix from 2.23 needed)
Version 2.20
No user visible changes included
Version 2.19
Version 2.18
- Discard old nodes asynchronously
- Restore compatibility with config-file-provider 2.14+
- Collect leaked OpenStack servers and Jenkins slaves once they do not have the counterpart
Issue #140: Report meaningful issue in case instance boot times out
Version 2.17
- JENKINS-41024 - Cannot provision new nodes with OpenStack plugin Resolved properly
Version 2.16
Version 2.15 (2017-01-02)
- Do not wait for successful launch while provisioning.
- There should be less failed launch attempts right after the node is provisioned.
- The time statistics are not comparable to the older ones (provisioning time is longer, launching is shorter).
- There should be less failed launch attempts right after the node is provisioned.
Version 2.14 (2016-11-21)
- Bugfix; do not fail when region is empty.
Version 2.13 (2016-10-22)
- Bugfix: avoid classloading issue caused by pom refactoring.
Version 2.8 (2016-06-06)
- Fix floating IP deallocation when machine is deleted (Issue #81)
Version 2.7 (2016-05-16)
- Do not leak servers when floating ip assignment fails.
- Avoid deadlock caused by adding and deleting OpenStack nodes.
- Avoid phony failures in
destroyServer
cause by server disappearing when retrying deletion.
Version 2.6 (2016-05-10)
- Integrate Cloud Statistics Plugin.
Version 2.5 (2016-05-05)
- Make sure plugin can reach all important endpoints when testing connection (JENKINS-34578)
Version 2.4 (2016-05-04)
- Plugin fails to resolve image ID on some OpenStack deployments (JENKINS-34495)
Version 2.3 (2016-04-21)
- Never remove slave put temporarily offline by user
- Plugin can now handle images with blank name
- Fix server deletion retry logic
- Make key-pair field selectable on global config page
Version 2.2 (2016-04-11)
- OpenStack slaves can be put into "pending delete" state pressing "Delete" button while build is in progress.
- Instances out of disk space in /tmp or workspace will be put into "pending delete" state and removed eventually.
- Maximal number of instances limitation implemented for templates.
- Maximal number of instances can be set to more than 10 (regression from 2.1).
Version 2.1 (2016-03-31)
- Machine/slave options can be specified on both cloud level as well as template level (Maximal number of instances limitation is implemented on 2.2).
- Images/snapshots are identified by name, not image id.
- Add support for floating pool name selection.
Version 2.0 (2016-02-22)
- Jobs without label are never scheduled, so does most of matrix combinations(JENKINS-29998)
- Drop support for blobstore. (This is not a rejection of the feature. None of the maintainers have an environment to reproduce this. Please reach us if you care for this feature and have an option to run the tests)
- Drop support for injecting private key from plugin. Should be done by configuration management.
- Replace JClouds backend with openstack4j.
- Move to singlemodule maven project avoiding dependency shading.
Version 1.5 (released February 2015)
- UserData scripts now managed by Config-File-Provider plugin
Version 1.4 (released February 2015)
- InitScript is moved out. use cloud-init plus userData instead
- Fix bug with multiple zones, now plugin restricts user to only one single zone
- get rid of SpoolingBeforeInstanceCreation as it is paid-cloud parameter only
Version 1.3 (released January, 2015)
- Initial release
Provision nodes from OpenStack on demand.
{jenkins-plugin-info:pluginId=openstack-cloud}
Essentials
[Jenkins > Openstack Cloud Plugin > openstack-cloud-config.png]
In order to provision new nodes when Jenkins load goes high, administrator needs to configure Cloud and one or more Templates on global configuration page. Cloud represents connection to particular OpenStack tenant (project). In order to use several openstack instances or tenants, it is necessary to configure several Clouds in Jenkins. Template can be seen as a definition of particular kind of node. Template is the primary holder of all the attributes for OpenStack machine to be provisioned (image, flavor, etc.) as well as for Jenkins node (number of executors, etc.). Note that the same set of attributes can be configured on Cloud level as well. Such configuration will be then used as default for its templates. For example, if all nodes are supposed to use the same key-pair there is no need to state it in every template, it can be configured on cloud level and leave the filed blank in the templates.
Aside from machine/node attributes, every template require name and labels to be configured. Name will serve both as an identifier of the template as well as a name prefix for Jenkins node and OpenStack machine (that is why some limitations apply here). Labels field expects a set of Jenkins labels that will be assigned to all nodes given template provision. It will also be used to determine which cloud and template to use to process Jenkins load. When there is a build with no label requirements (Restrict where this project can be run on job page), any template can be used to provision the node. Build with label restriction can trigger provisioning only on templates with matching label set. The attributes at template level will inherit all global values (the value in effect is printed under the field). In case required field do not have a default nor current value, it will be reported.
[Jenkins > Openstack Cloud Plugin > openstack-template-config.png]
User data
Every template can declare user-data script to be passed to CloudInit to customize the machine that is provisioned. Note that the image needs to support cloud init explicitly.
Before the script is sent to OpenStack, plugin injects several values using ${VARIABLE_NAME} syntax:
JENKINS_URL: The URL of the Jenkins instance
SLAVE_JAR_URL: URL of the slave.jar - the agent executable
SLAVE_JNLP_URL: The endpoint URL for JNLP connection
SLAVE_LABELS: Labels of the corresponding Jenkins computer
Reporting
Openstack plugin utilizes Cloud Statistics Plugin that captures failures and time trends of past provisioning attempts.
Tips and tricks
User can manually provision slave from particular cloud/template on Manage Jenkins > Manage Nodes.
Deleting slave provisioned by OpenStack plugin will merely put it temporarily offline, the actual disposal of the slave and machine is performed with delay. That means that user have the possibility to abort that operation putting the slave back online. But most importantly, user can suggest plugin to discard the slave while still occupied by running build this way.
Plugin identifies OpenStack image/snapshot to provision by its name. The image can be updated/replaced in openstack and the plugin will not require update provided its name have not changed.
In case maximal instance number is specified on template level, the stricter of the two (global and template value) will be applied. Global value exists to ensure that the number of machines provisioned by plugin will not exceed certain limit. The template value can further restrict that at most N machines of given kind can be utilized at the same time.
Plugin can report maximal number of instances was reached while there is not adequate number of Jenkins nodes. It is because plugin inspects running OpenStack machines in order to cover machines being provisioned/deleted (that do not have Jenkins nodes) and instances that plugin failed to delete. In case the instance get leaked, please report that as a bug with all relevant FINE level logs attached.
On every slave provisioned by the plugin, there is an environment variable OPENSTACK_PUBLIC_IP declared with the IP address allocated for the machine.
Troubleshooting
Accessing logs
Plugin uses INFO and above levels to report things user should worry about. For debugging, set it to FINEST - note the ALL level is not sufficient to print these. To configure OpenStack plugin logging in Jenkins UI go to Manage Jenkins > System Log > New Log Recorder and use jenkins.plugins.openstack.compute as the logger name.
Access openstack client from groovy console
Use Jenkins.instance.clouds[0].openstack.@clientProvider.get() to access the openstack4j client. Users are discouraged to use this anything else but querying the openstack (otherwise there is no way to ensure plugin will work correctly). For older versions fo the plugin use Jenkins.instance.clouds[0].openstack.@client.
User Data / Cloud init is not evaluated
The image might not support it at all or can fail executing it. Check machine log in OpenStack for further details. Note that for now, there is no guarantee the script will complete before machine is connected to Jenkins and builds are started.
Changelog
Version 2.32
Support booting to volume
Rework computer termination to prevent repeated Channel#close
Version 2.31
Ensure node names never collide
OpenStackj4 Neutron Networks object incompatible
Version 2.30
Add support for project domains
Add ability to skip ssl check
Report computer "fatal" offline cause when present when destroying computer
Version 2.29 (2017-10-20)
Fix regression in region handling
Do not report failed FIP deletion in cloud statistics if failed with 404
Add support for volumeSnapshots
Do not use expired login sessions
Version 2.27 (2017-10-03)
Improve reporting of boot timeout
Abort provisioning/launching when server gets deleted
Add support for explicit java path when SSHLauncher is used
Bump openstack4j okhttp connector to avoid accasional connection leaks
Avoid phony cloud-stats warnings logged
Version 2.26
Botched release - changes went to 2.27
Version 2.25 (2017-09-25)
Fix #168: Prevent tracking disposal of the same server several times
Use ok-http to prevent connection blockage (prefer 2.27 with followup fix)
Fix #167: Specify node readiness timeout cause
Refactor slave type into describable
Version 2.24 (2017-08-17)
Note this version is affected by httpclient connection leak - use 2.27 instead.
Pipeline step for agentless node provisioning
Prefer IPv4 address for SSH launcher
Prevent occasional IllegalArgumentException: Failed to instantiate class jenkins.plugins.openstack.compute.SlaveOptions while saving global configuration page
First attempt to implement Openstack client caching between requests
This version is affected by occasional httpclient connection blockage
Version 2.23
Issue #128: Do not fail when FIP service is disallowed (403) on paths that does not require floating IP (introduced in 2.21)
This version is affected by occasional httpclient connection blockage
Version 2.22
Investigate SSH channel is closed/No route to host (Issue #149)
Version 2.21
Fix #148: Skip unknown variables in user data
Record manual provisioning attempt failure for unexpected exceptions
Issue #128: Request: Allow VMs without floating IPs (followup fix from 2.23 needed)
Version 2.20
No user visible changes included
Version 2.19
Fixed #84: Destroy leaked floating IPs
Fix #137: Retry when ssh launcher fail to bring the node online silently
Fix #144: Bring the node sidebar links that ware removed accidentally
Fix #109: Generate documentation for variables replaced in user data
Fix waiting for JNLP agents (JENKINS-42207)
Do not discard nodes that are being provisioned
Version 2.18
Discard old nodes asynchronously
Restore compatibility with config-file-provider 2.14+
Collect leaked OpenStack servers and Jenkins slaves once they do not have the counterpart
Issue #140: Report meaningful issue in case instance boot times out
Version 2.17
Restore config-file-provider <2.13 support properly
Version 2.16
Restore config-file-provider <2.13 support - Do not use this version!
Version 2.15 (2017-01-02)
Do not wait for successful launch while provisioning.
There should be less failed launch attempts right after the node is provisioned.
The time statistics are not comparable to the older ones (provisioning time is longer, launching is shorter).
Version 2.14 (2016-11-21)
Bugfix; do not fail when region is empty.
Version 2.13 (2016-10-22)
Bugfix: avoid classloading issue caused by pom refactoring.
Version 2.8 (2016-06-06)
Fix floating IP deallocation when machine is deleted (Issue #81)
Version 2.7 (2016-05-16)
Do not leak servers when floating ip assignment fails.
Avoid deadlock caused by adding and deleting OpenStack nodes.
Avoid phony failures in destroyServer cause by server disappearing when retrying deletion.
Version 2.6 (2016-05-10)
Integrate Cloud Statistics Plugin.
Version 2.5 (2016-05-05)
Make sure plugin can reach all important endpoints when testing connection (JENKINS-34578)
Version 2.4 (2016-05-04)
Plugin fails to resolve image ID on some OpenStack deployments (JENKINS-34495)
Version 2.3 (2016-04-21)
Never remove slave put temporarily offline by user
Plugin can now handle images with blank name
Fix server deletion retry logic
Make key-pair field selectable on global config page
Version 2.2 (2016-04-11)
OpenStack slaves can be put into "pending delete" state pressing "Delete" button while build is in progress.
Instances out of disk space in /tmp or workspace will be put into "pending delete" state and removed eventually.
Maximal number of instances limitation implemented for templates.
Maximal number of instances can be set to more than 10 (regression from 2.1).
Version 2.1 (2016-03-31)
Machine/slave options can be specified on both cloud level as well as template level (Maximal number of instances limitation is implemented on 2.2).
Images/snapshots are identified by name, not image id.
Add support for floating pool name selection.
Version 2.0 (2016-02-22)
Jobs without label are never scheduled, so does most of matrix combinations(JENKINS-29998)
Drop support for blobstore. (This is not a rejection of the feature. None of the maintainers have an environment to reproduce this. Please reach us if you care for this feature and have an option to run the tests)
Drop support for injecting private key from plugin. Should be done by configuration management.
Replace JClouds backend with openstack4j.
Move to singlemodule maven project avoiding dependency shading.
Version 1.5 (released February 2015)
UserData scripts now managed by Config-File-Provider plugin
Version 1.4 (released February 2015)
InitScript is moved out. use cloud-init plus userData instead
Fix bug with multiple zones, now plugin restricts user to only one single zone
get rid of SpoolingBeforeInstanceCreation as it is paid-cloud parameter only
Version 1.3 (released January, 2015)
Initial release
Attachments:
Screen Shot 2015-02-24 at 17.26.34.png (image/png)
openstack-cloud-config.png (image/png)
openstack-cloud-config.png (image/png)
openstack-template-config.png (image/png)