Jenkins : runATH and runPCT step design

Created by Unknown User (rarabaolaza), last modified on Apr 05, 2018

Context

As the first step for  INFRA-1489 - Create runATH step Resolved  a mini design of the new step is needed so the API and implementation details can be validated before making it. Validation and feedback are very important as this step is intended to be used for the community and hence is important not only to follow best practices but also to make it as easy and useful for plugin developers as possible.

API details

Following the example of buildPlugin this step accepts as parameters a map, but every parameter has a sensible default value to make its use as simple as possible for default scenarios, the proposed list of params and the related default values are:

  • athUrl: The URL to get the ATH sources. It can point to a local path or a github destination. The reason for this param is that some changes may need adjustments in the ATH itself or new tests not yet merged into the master branch of the official repo. Defaults to https://github.com/jenkinsci/acceptance-test-harness.git (Or SSH version) Can be overridden from the metadata
  • athRevision: The ATH revision to use can be a branch or tag name or a commit id. Defaults to branch master Can be overridden from the metadata file
  • metadataFile: A String indicating the file path (relative to the repo under test) to use as metadata file for the build, more details about the metadata file will be provided in the implementation details section of this page. Defaults to essentials.yml at the location where this step is invoked 
  • jenkins: URI to the jenkins.war, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". Defaults to "latest". For local war files use the file:// protocol to specify the URI Can be overriden from the metadata file

Implementation details

Metadata file

Even with a call to the step no tests are run unless the metadataFile can be found, recommended standard location for this file is the plugin's repo root level. The file purpose is to serve as flag to indicate that a certain component is under the essentials program and also to provide some metadata for the essentials flow,  every essentials related step must act as a no-op in case this file does not exist

The exact format of this file is to be discussed, but must be a YAML file easily readable for both humans and CI (by using the Pipeline Utility Steps Plugin for example)

Regarding ATH this metadata file is going to include at least:

  • The list of tests to run for the component that calls the step. If no particular set of tests or categories is defined the SmokeTest Category of the ATH will be run
    • The list of Categories to run. Defaults to nothing
  • the run has to fail fast or not. Defaults to false if not specified
  • The number of runs per failing test  (a la maven) Defaults to zero
  • If the ATH should use local versions of the plugins. Defaults to true 
  • The list of browsers to use when running ATH Defaults to firefox
  • athUrl: The URL to get the ATH sources. It can point to a local path or a github destination. If specified it will override the parameter in the runATH step
  • athRevision: The ATH revision to use can be a branch or tag name or a commit id. If specified it will override the parameter in the runATH step
  • jenkins: URI to the jenkins.war file, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". If specified it will override the parameter in the runATH step

The proposed initial format of this file regarding ath runs is:

essentials.yml
--- 
ath:
  athUrl: https://github.com/jenkinsci/acceptance-test-harness.git
  athRevision: acceptance-test-harness-1.59
  jenkins: 2.89.4  
  failFast: false
  rerunFailingTestsCount: 0
  useLocalSnapshots: true
  browsers:
    - firefox
    - chrome
    - ¿explorer|edge? 
  tests: 
    - Test1
    - Test2
    - Test3
  categories:
    - Category1
    - Category2

A very simplified version when you want to use all default values is like:

minimum essentials.yml
ath: 'default'

 

Please note that the list of tests is going to be specified via maven -Dtests parameter whilst categories use the maven -Dgroups parameter, given that when both of them are specified surefire actually executes the intersection of both (that is all tests in the list of tests that also belong to a category in the list of categories) to ensure that all specified tests and categories are executed the step is going to run two different executions in parallel one for tests and another for categories

It may seem that some configuration property of the metadata file can be set in the runATH command. The idea behind this separation is to also use this metadata file (via some mechanism to be developed) to provide a very easy and "standard" way to run the ath locally with a configurable set of parameters. For example by modifying the current run.sh script to read this  file. 

Testing development versions

To run the ATH using development versions of plugins (for example in a PR) this step uses the PLUGINS_DIR property of the ATH if an only if useLocalSnapshots is set to true. In that case, the step expects all local versions of plugins to be previously stashed under the name defined by the env variable 

RUN_ATH_LOCAL_PLUGINS_STASH_NAME or if not defined under "localPlugins"

Step code

Initial implementation of the code will be heavily based on the existing ATH Jenkinsfile

Not supported in the first implementation (but open to discussion or planned for the future)

  • Running against different browsers (the initial implementation will warn user if a browser different than firefox is used and use firefox)
  • Running on different platforms (like windows )

runPCT step

API details

Following the example of buildPlugin this step accepts as parameters a map, but every parameter has a sensible default value to make its use as simple as possible for default scenarios, the proposed list of params and the related default values are:

  • pctUrl: The URL to get the PCT Dockerfile or the pct docker image to use. It can point to a local path of PCT sources (by using the file:// protocol) or a github destination. You can also use this to directly specify a prebuilt PCT docker image by using the docker:// protocol, for example "docker://jenkins/pct" Can be overridden from the metadata file
  • pctRevision: The PCT revision to use in case pctUrl is a github destination can be a branch or tag name or a commit id. Can be overridden from the metadata file
  • metadataFile: A String indicating the file path (relative to the repo under test) to use as metadata file for the build, more details about the metadata file will be provided in the implementation details section of this page. Defaults to essentials.yml at the location where this step is invoked 
  • jenkins: URI to the jenkins.war, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". Defaults to "latest". For local war files use the file:// protocol to specify the URI Can be overriden from the metadata file

Implementation details

Metadata file

Regarding PCT this metadata file is going to include at least:

  • The list of plugins to be exercised by the PCT
  • If the PCT should use local versions of the plugins. Defaults to true 
  • pctUrl: The URL to get the PCT Dockerfile or the pct docker image to use. It can point to a local path of PCT sources (by using the file:// protocol) or a github destination. You can also use this to directly specify a prebuilt PCT docker image by using the docker:// protocol, for example "docker://jenkins/pct". If specified it will override the parameter in the runPCT step
  • pctRevision: he PCT revision to use in case pctUrl is a github destination can be a branch or tag name or a commit id. If specified it will override the parameter in the runPCT step
  • jenkins: URI to the jenkins.war file, Jenkins version or one of "latest", "latest-rc", "lts" and "lts-rc". If specified it will override the parameter in the runPCT step

The proposed initial format of this file regarding ath runs is:

essentials.yml
--- 
pct:
  pctUrl: "https://github.com/jenkinsci/plugin-compat-tester.git"
  pctRevision: "master"
  jenkins: 2.89.4
  useLocalSnapshots: true
  plugins:
    - Credentials
    - ssh-slaves

A very simplified version when you want to use all default values is like:

minimum essentials.yml
pct: 'default'

 

Please note that the PCT is always going to be executed via the pct docker image, if several plugins are specified then several runs of the PCT (one per plugin) are executed in parallel

Testing development versions

To run the PCT using development versions of plugins (for example in a PR) make sure  useLocalSnapshots is set to true. In that case, the step expects all local versions of plugins to be previously stashed under the name defined by the env variable 

RUN_PCT_LOCAL_PLUGIN_SOURCES_STASH_NAME or if not defined under "localPlugins", what has to be stashed is a folder where sources for every plugin to test are stored in a subfolder whose name is the plugin id

Not supported in the first implementation (but open to discussion or planned for the future)

  • Running on different platforms (like windows )