YAML Project support for Jenkins plugin

This plugin introduces a YAML project for Jenkins, which loads a declarative specification of the project from a checked in YAML file.

For example, the following YAML file, will get turned into a simple Freestyle Project with a single shell build step that echos “Hello World”:

kind: !freestyle
builder:
  kind: !shell
  command: echo Hello World

This new project type relies on the simple property that YAML 1.2 is a superset of JSON, so any structured form submission can be parsed as YAML; and that we can serialize the objects parsed from YAML back into JSON with relative ease.

Advantages of YAML

By using YAML instead of raw JSON, we get nice some properties, like the ability to comment your configuration:

kind: !freestyle
builder:
  kind: !shell
  # Print a greeting.
  command: echo Hello World

As well as the ability to replace what would be raw class names (e.g. hudson.tasks.Shell) with friendlier names (e.g. !shell).  While these special short tags require plugins to annotate themselves with, by default we allow users to reference plugins based on their display name.

kind: !freestyle
builder:
  kind: !by-name Execute shell
  # print a greeting
  command: echo Hello World

Getting Started

To facilitate getting started with the YAML project, or to learn to use it with a new plugin, we have added an extension that annotates new projects with the YAML that would be used to generate them.  If we created the previous “Hello World” example through the UI normally, we would see:

NOTE: we try to scrub out a lot of superfluous form data, but it is imperfect/incomplete, which is why this example contains additional fields.

Check this YAML into your repository, e.g. as .jenkins.yaml, and create a new “YAML Project” that syncs the repository and reads your chosen filename.  It’s that easy.

Getting Around

Once you have run your YAML project, you will see a new index page:

On the right side of the page, there is an embedded view of the projects that have been instantiated from the YAML file.  A new version is created whenever you change your YAML, so you can view prior versions of your project by simply browsing the job history.
Embedded in the center of the page, we include the index page for the latest version of the project.  This should match what you would see if you were to click on the latest version in the “Job History” pane.

Version History

Version TBD (TBD)

• TBD

Notes for advanced users and plugin developers

Curating your plugin’s sub-grammar

The grammar of your plugin is entirely guided by your contribution to the structure form submission.  In order to change your plugin’s contribution, you simply need to rename your Jelly form fields, take for instance an older Google Cloud Storage Plugin's jelly:

  <a:credentials title="${%Google Credentials}" field="credentialsId" />
  <f:block>
    <f:hetero-list name="uploads" hasHeader="true"
      descriptors="${descriptor.getUploads()}"
      items="${instance != null ? instance.uploads : descriptor.getDefaultUploads()}"
      addCaption="Add Operation"
      deleteCaption="Delete Operation" />
  </f:block>

This is coupled with its @DataBoundConstructor:

@DataBoundConstructor
public GoogleCloudStorageUploader(
  String credentialsId, @Nullable List<AbstractUpload> uploads) {

To produce the top-level grammar:

publisher:
- kind: !by-name Google Cloud Storage Uploader
  credentialsId: {project-id}
  uploads:
    ...

For the elements of the list of uploads, since AbstractUpload is a abstract Describable, the grammar depends on the implementation, e.g.

- stapler-class: !by-name Build Log Upload
  # Variant-specific field
  logName: build-log.txt
  # Common fields
  bucketNameWithVars: gs://ma-bucket/$GIT_COMMIT
  forFailedJobs: true
- stapler-class: !by-name *Classic Upload*
  # Variant-specific field
  sourceGlobWithVars: '**/target/*.hpi'
  # Common fields
  bucketNameWithVars: gs://ma-bucket/$GIT_COMMIT

We can improve these field names with a simple rename in the Jelly and on the pertinent @DataBoundConstructor.

Backwards compatibility

One of the risks, as you change the grammar of your plugin, is that if people have a dependency on the old grammar that you will break them.  
There a few tricks that can avoid this:

1. Leave your previous @DataBoundConstructor arguments in place, and handle marshalling from a particular set of arguments to your object in the constructor.  This is useful if you are simply renaming arguments without naming collisions.
2. Implement custom newInstance processing on your Descriptor that dispatches to different implementations.  This is useful if, depending on version, the same name is shared by distinct types.

You can either deduce which version of your grammar is being used by null checking multiple flavors of argument, or with an explicit version argument.  It is also possible to explicitly version your grammar through a hidden ‘version: ’ field in your form data, which guides either of the above to the appropriate argument handling.
For example, consider the new ‘ClassicUpload’ @DataBoundConstructor:

@DataBoundConstructor
public ClassicUpload(String bucket, boolean sharedPublicly,
   boolean forFailedJobs, @Nullable UploadModule module,
   String pattern,
   // Legacy arguments for backwards compatibility
   @Deprecated @Nullable String bucketNameWithVars,
   @Deprecated @Nullable String sourceGlobWithVars) {
  super(Objects.firstNonNull(bucket, bucketNameWithVars), sharedPublicly,
    forFailedJobs, module);

  this.sourceGlobWithVars =
    Objects.firstNonNull(pattern, sourceGlobWithVars);
}

NOTE: the new Jelly only specifies ‘bucket’ and ‘pattern’, however, we accept the old names to avoid breaking users of the previous syntax.

This style allows us to simultaneously support both grammars, while only directing users to the latest grammar via the “YAML Project” action that we annotate on jobs created via the UI.

!your-name-here

If you aren’t satisfied with your plugin being accessed via:

!by-name Your Display Name

You have the option to annotate your plugin with a custom yaml tag.  On your Describable, simply annotate either:

@YamlTag(tag = “!your-name-here”)

If you want to reserve multiple pseudonyms, you can use:

@YamlTags({@YamlTag(tag = “!name1”), @YamlTag(tag = “!name2”)})

If there are multiple plugins that occupy a space, you might also want a qualified name.  Some example of this would be:

@YamlTag(tag = “!storage”, arg=“gcs”)

and

@YamlTag(tag = “!storage”, arg=“s3”)

Or in the DSL space:

@YamlTag(tag = “!dsl”, arg=“yaml”)

and

@YamlTag(tag = “!dsl”, arg=“groovy”)

Verbose mode

For diagnostic output, consider enabling verbose output in Jenkins’ global configuration.