Jenkins : Ui Testing White Mountains Release

Configuration (Arrange, Given)

The static code analysis plug-in are configured using the following ways (depends on Jenkins job type).

Freestyle Job, Maven Job, Matrix Job

Use the Add Postbuild Action command "Record static analysis results" (See integration tests for parameter variations). 

Pipeline (Scripted), Pipeline (Scripted Parallel), Multi-Branch-Pipeline (Scripted)

The whole configuration is done using a pipeline script. Use the steps "scanForIssues" (see ScanForIssuesStep) to scan for Issues and the step "publishIssues" (see PublishIssuesStep) to aggregate and publish the results.

Example:

node {
    stage ('Checkout') {
        git branch:'master', url: 'git@github.com:jenkinsci/analysis-model.git'
    }

    stage ('Build') {
        def mvnHome = tool 'mvn-default'

        sh "${mvnHome}/bin/mvn --batch-mode -V -U -e clean verify -Dsurefire.useFile=false -Dmaven.test.failure.ignore"

        junit testResults: '**/target/surefire-reports/TEST-*.xml'

        def java = scanForIssues tool: [$class: 'Java']
        def javadoc = scanForIssues tool: [$class: 'JavaDoc']

        publishIssues issues:[java,javadoc], unstableTotalAll:1
    }

    stage ('Analysis') {
        def mvnHome = tool 'mvn-default'

        sh "${mvnHome}/bin/mvn -batch-mode -V -U -e checkstyle:checkstyle pmd:pmd pmd:cpd findbugs:findbugs spotbugs:spotbugs"

        def checkstyle = scanForIssues tool: [$class: 'CheckStyle'], pattern: '**/target/checkstyle-result.xml'
        publishIssues issues:[checkstyle], unstableTotalAll:1

        def pmd = scanForIssues tool: [$class: 'Pmd'], pattern: '**/target/pmd.xml'
        publishIssues issues:[pmd], unstableTotalAll:1

        def cpd = scanForIssues tool: [$class: 'Cpd'], pattern: '**/target/cpd.xml'
        publishIssues issues:[cpd]

        def findbugs = scanForIssues tool: [$class: 'FindBugs'], pattern: '**/target/findbugsXml.xml'
        publishIssues issues:[findbugs], unstableTotalAll:1

        def spotbugs = scanForIssues tool: [$class: 'SpotBugs'], pattern: '**/target/spotbugsXml.xml'
        publishIssues issues:[spotbugs], unstableTotalAll:1

        def maven = scanForIssues tool: [$class: 'MavenConsole']
        publishIssues issues:[maven]
    }
}

Pipeline (Declarative), Multi-Branch-Pipeline (Declarative)

The whole configuration is done using a declarative pipeline script. Use the step "recordIssues" (see IssuesRecorderStep) to report Issues.

Example:

pipeline {
    agent 'any'
    tools {
        maven 'mvn-3.5.0'
        jdk 'jdk-default'
    }
    stages {
        stage ('Build') {
            steps {
                sh '${M2_HOME}/bin/mvn --batch-mode -V -U -e clean verify -Dsurefire.useFile=false -Dmaven.test.failure.ignore'
            }
        }

        stage ('Analysis') {
            steps {
                sh '${M2_HOME}/bin/mvn --batch-mode -V -U -e checkstyle:checkstyle pmd:pmd pmd:cpd findbugs:findbugs spotbugs:spotbugs'
            }
        }
    }
    post {
        always {
            junit testResults: '**/target/surefire-reports/TEST-*.xml'

            recordIssues enabledForFailure: true, tools: [[pattern: '', tool: [$class: 'MavenConsole']], [pattern: '', tool: [$class: 'Java']], [pattern: '', tool: [$class: 'JavaDoc']]]
            recordIssues enabledForFailure: true, tools: [[pattern: '', tool: [$class: 'CheckStyle']]]
            recordIssues enabledForFailure: true, tools: [[pattern: '', tool: [$class: 'FindBugs']]]
            recordIssues enabledForFailure: true, tools: [[pattern: '', tool: [$class: 'SpotBugs']]]
            recordIssues enabledForFailure: true, tools: [[pattern: '**/target/cpd.xml', tool: [$class: 'Cpd']]]
            recordIssues enabledForFailure: true, tools: [[pattern: '**/target/pmd.xml', tool: [$class: 'Pmd']]]
        }
    }
}

Validation (Assert, Then)

Validation of the results does not depend on the job type chosen above. There following views are available and should be tested using a different page object:

  • Job Page: Sidebar links and Trend Graph
  • Build Page: Sidebar links and Summary Box
  • Result Page: 
    • Doughnut graph showing the trend of new, fixed and outstanding issues
    • Doughnut graph showing the distribution of errors, low, normal or high priority issues
    • Details box showing
      • Distribution by property (Tools, Modules, Packages, Files, Categories, Types)
      • Details Table of all issues
      • Details List of all issues
    • Source Code view of affected file
    • Console Log Code view of affected part in console log
    • Information and Error view
    • Fixed issues details

In order to provide clean code for all assertions a custom assertion class should be created.

Variations

The following set of variations should be tested.

Job Type

  • Freestyle Job 
  • Maven Job
  • Matrix Job
  • Pipeline Job (Scripted)
  • Pipeline Job (Scripted - Parallel Execution)
  • Pipeline Job (Declarative)
  • Pipeline Job (Declarative - Parallel Execution)
  • Multibranch-Pipeline
  • Job in a Folder

Miscellaneous

  • Navigation to Source Files possible (XL)
  • Not Logged in User can navigate to Workspace Source Files (S, extension of previous test case)
  • Name of Packages Tab is variable (Package vs. Namespace) (XL)
  • Reference Build navigable in Build Overview (S)
  • Reference Build navigable from Age Column in Details View (S)
  • Navigation into a sub-set of issues (properties: File, Package, Module, Type, Category, Priority, Origin) (M)
  • Default pattern of tools like Checkstyle is used (S)
  • Composite Parsers (more than one different parser) aggregate warnings (M)
  • Report encodings: check the encoding of a report with different locales (M)
  • Source Code encodings: check the encoding of a source file with different locales (M)
  • Jenkins restart: all information is loaded from disk (S)
  • XML Parser reports an user friendly error if no pattern is provided (i.e. console log scanning active): canScanConsoleLog (S)
  • Zero Warnings should be handled correctly (S)
  • Console Log parsing vs. File parsing (same input should provide same output) (M)
  • Multiple files in the workspace are correctly parsed (M)
  • Module names are correctly resolved (M)
  • Package names are correctly resolved (M)
  • Tests with newest LTS and 2.107.3
  • Master build (default)
  • Agent build on same machine (S)
  • Agent build in docker container (XL)
  • Groovy parser (M)
  • Input variations (see Integration Tests) (M)
  • Verify behavior if builds are deleted (M)
  • Sorting of Tables (M)
  • Filtering of Tables (M)
  • Pagination of Tables (M)
  • Trendgraphs are visible (M)
  • (Not implemented yet: Git Blames are visible)
  • (Not implemented yet: Tokens and EMail)
  •