TAP Plugin

Skip to end of metadata
Go to start of metadata

Plugin Information

Plugin ID tap Changes In Latest Release
Since Latest Release
Latest Release
Latest Release Date
Required Core
Dependencies
1.18
Mar 12, 2014
1.488
Source Code
Issue Tracking
Maintainer(s)
GitHub
Open Issues
Bruno P. Kinoshita (id: kinow)
Usage Installations 2013-Apr 557
2013-May 574
2013-Jun 640
2013-Jul 689
2013-Aug 744
2013-Sep 759
2013-Oct 827
2013-Nov 833
2013-Dec 861
2014-Jan 905
2014-Feb 950
2014-Mar 1015

This plug-in adds support to TAP test result files to Jenkins. It lets you specify an ant-like pattern for a directory that contains your TAP files.

TAP Plug-in has only one dependency, tap4j - a TAP implementation for Java. 

Overview

The plug-in looks for TAP files like the following one.

1..2
ok 1 - Yahoo!
not ok 2 - org.tap4j.Error...

When a TAP stream like the above is found, the plug-in delegates the parsing the tap4j. The results of tap4j parsing are then analysed, organized and displayed to the user as graphs and custom pages.

You can test your TAP streams with tap4j, using InstantTAP.

Understanding TAP streams in 2 minutes

One of the easiest ways to learn something new is by examples. So here we will show some examples of TAP streams. For being human-friendly, it shouldn't confuse you. We will use comments to explain each line.

1..2 # a plan stating that we have two tests cases, from 1 to 2.
ok 1 - Yahoo! # the first test result was executed successfully and has a description of ' - Yahoo!'.
not ok 2 - org.tap4j.Error... # unfortunately, the second test result failed. The description here was used to display some nasty Exception.

Running Perl tests with prove

The plug-in cannot handle prove's default output (since it includes more information than simply TAP, and causes tap4j parser to fail). The best way to handle the prove output is by using Perl module TAP::Harness::Archive. Supposing you have your tests under t/ directory, you can create another directory (say, output) and archive your TAP tests with prove by using a command line similar to the below:

prove t/ --archive output

The result files will be stored under output/t/. You can use a pattern in the plug-in configuration like t/*/.t.

Using attachments

The following is a TAP with attachments, using YAMLish. If you are familiar with YAML, this example should be very easy to read.

1..2
ok 1
not ok 2 - br.eti.kinoshita.selenium.TestListVeterinarians#testGoogle
  ---
  extensions:
      Files:
          my_message.txt:
            File-Title: my_message.txt
            File-Description: Sample message
            File-Size: 31
            File-Name: message.txt
            File-Content: TuNvIGNvbnRhdmFtIGNvbSBtaW5oYSBhc3T6Y2lhIQ==
            File-Type: image/png
  ...

The Files entry has an array of files. You have to Base64 encode your content data.

Subtests (grouping or test suites).

Subtests let you group several TAP streams unto a single one. This way, you can organize your tests in similar fashion to JUnit or TestNG test suites. Indentation is important for TAP subtests.

Subtests and YAMLish are not officially in TAP 13 specification
  1..3
  ok 1 - First test
      1..2
      ok 1 - This is a subtest
      ok 2 - So is this
        1..2
        ok 1 - This is a subtest
        ok 2 - So is this
  ok 2 - An example subtest
  ok 3 - Third test

Configuration

  1. Install the Jenkins TAP Plug-in using the Plug-in Manager or manually by copying the .hpi (or .jpi) file
  2. Check the option to publish TAP, configure a pattern (and other settings)
  3. Execute your build and analyze the results

Screenshots

Jenkins JUnit compatible reports and graphs

Custom actions for TAP too

YAMLish support

Languages Supported

  1. English (American)
  2. Portuguese (Brazil)) Work in Progress
  3. Spanish (Thanks to César Fernandes de Almeida) Work in Progress

Known Limitations

  1. If the file type of a TAP report is considered as binary by the Jenkins webserver then the TAP plugin doesn't consider this file for inclusion in TAP reports (see #15813 for further details). To make sure your TAP report is considered for inclusion use e.g. the file name suffix .tap (so instead of a file named "report" use "report.tap").

Compatibility Matrix

This plug-in is not developed for Hudson, and we don't fix bugs happening in any version of Hudson.
Plugin 1.0 1.1 1.2.x 1.3~1.9 1.10 1.11 1.12 1.13 1.14 1.15 1.16 1.17 1.18
TAP 13 13 13 13 13 13 13 13 13 13 13 13 13
Jenkins 1.399+ 1.399+ 1.399+ 1.399+ 1.488+ 1.488+ 1.488+ 1.488+ 1.488+ 1.488+ 1.488+ 1.488+ 1.488+
Hudson ?
?
?
Not supported Not supported Not supported Not supported Not supported Not supported Not supported Not supported Not supported Not supported

Open Issues

Sponsors

For commercial support, please get contact us via @tupilabs

Resources

  1. This plug-in is going to be part of "Make your tests speak TAP" presentation in JCertif by Bruno P. Kinoshita, in September 2011. In this presentation will be presented too the tap4j project, how to enable TAP in JUnit and TestNG, integrate Perl and Java tests and an Eclipse TAP editor.
  2. Gábor Szabó (2009), Test Reporting system: Smolder wish-list <http://szabgab.com/blog/2009/07/test-reporting-system-smolder-wish-list.html>.
  3. Gábor Szabó (2009), Reporting Test Results <http://szabgab.com/blog/2009/04/reporting-test-results.html>
  4. tap4j - The TAP implementation for Java.
  5. Test Anything Protocol (official webpage).
  6. Performance tests with phantomjs and yslow (uses the plug-in for plotting the TAP results)
  7. TAP Plugin for Matlab
JCertif 2011
JCertif - Make your tests speak TAP - Speaker: Bruno P. Kinoshita
September, 2011 - Brazzaville, Congo

Release Notes

Version 1.18
  1. Updated to tap4j-4.0.8
  2. JENKINS-22047: Add option to reduce noise in logs
  3. JENKINS-22036: NullPointer when there is no Test Plan
  4. JENKINS-17960: Indicate if tests don't go to plan
  5. JENKINS-21917: TAP results graph causes null pointer exception
  6. https://github.com/jenkinsci/tap-plugin/pull/4
Version 1.17
  1. JENKINS-22047: Add option to reduce noise in logs
  2. JENKINS-22036: NullPointer when there is no Test Plan
  3. JENKINS-17960: Indicate if tests don't go to plan
  4. JENKINS-21917: TAP results graph causes null pointer exception
  5. Pull request #4
Version 1.17
  1. JENKINS-20924: Make plans optional in TAP via a configuration
Version 1.16
  1. Updated to tap4j-4.0.5 (better subtests handling)
Version 1.15
  1. JENKINS-16325: TAP Parser can't handle the output from prove
Version 1.14
  1. Security bug reported by Kees J. via e-mail. This issue is related to exposing files that the user running Jenkins has access via the plug-in.
Version 1.13
  1. JENKINS-17960: Indicate if tests don't go to plan
Version 1.12
  1. JENKINS-18885: Parse errors with Git's TAP test suite, part 2
  2. JENKINS-17878: HTML test output in tapResults not escaped
  3. JENKINS-17855: TAP Stream results summary page contains links that fail
  4. JENKINS-17504: TAP Plugin generates bad detail links on "tapTestReport" page
Version 1.11
  1. JENKINS-17859: TAP report table show failed test message on ALL tests after the failed one.
  2. JENKINS-17855: TAP Stream results summary page contains links that fail
  3. JENKINS-17941: Parse errors with Git's TAP test suite
  4. JENKINS-17947: Nested TAP not parsed correctly
  5. JENKINS-17504: TAP Plugin generates bad detail links on "tapTestReport" page
Version 1.10
  1. JENKINS-17245: Tap plug-in can't find TAP attachments
Version 1.9
  1. JENKINS-16262: Tap plug-in can't find TAP attachments
Version 1.8
  1. JENKINS-15914: TAP results table misses first comment line
  2. JENKINS-15322: NOTESTS in TAP response gives parse error and stack trace from plugin
  3. JENKINS-15401: support TODO directive to not fail such tests
  4. JENKINS-15907: When multiple TAP files with same basename match pattern, only one is processed
Version 1.7
  1. JENKINS-15586: TAP plug-in is ignoring given file extension and looking for .tap files
Version 1.6
  1. JENKINS-15419: TAP published results hide JUnit published results
  2. JENKINS-15497: Display link to download TAP attachment
Version 1.2.7
  1. Removed requirement to have the TAP Plan at start or at the end of the TAP Stream. This way, TAP Streams generated using Perl done_testing() now works well with the plug-in
Version 1.2.6
  1. Support JSON within YAMLish data
Version 1.1
  1. Support to Bail out!'s
  2. JENKINS-10562 TAP Plugin fails on slave
Version 1.0
  1. Initial design of the plug-in
  2. Custom UI for TAP test results
  3. JUnit-like graph that displays the test results per build (actually the graph was modeled using TestNG Plug-in as basis)

Roadmap (wish list)

Version 1.x
  1. Add configurations like validate number of tests with test plan, a TODO causes a test to fail
  2. Diagnostics image gallery Done! Fixed as a new plug-in
  3. Diagnostics exception code formatting
  4. Add a link to open the file in the Build workspace (think about remote and local issues) Done!

History

The idea of the plug-in surged after tap4j was created. After learning about Smolder, it became evident that Jenkins could be used as a replacement for it. All that was needed was just adding TAP support to Jenkins and implementing a nice UI to display the test results. After some messages in jenkins-dev-list, Max and Nick commented about their need to show test results in a different manner than how Jenkins was doing at that moment. Soon after that Max, Nick, Bruno (tap4j) and Cesar (tap4j) started to work together, exchanging mail messages and discussing about a initial design for this plug-in.

In July 2011 the first version of the plug-in was ready to be released. The graph code used here was adapted from TestNG Plugin (big thanks to the development team, great work). The diagnostic (YAMLish) was implemented in Jelly + Java + CSS. And the road map was incremented based on what Gábor Szabó posted about Smolder and testing reports in his blog (see resources for links).

Simulating TAP streams with shell build steps

#!/bin/bash


for x in {1..100}; do


cat > $x.tap <<EOF
ok 1
not ok 2
ok 3 # SKIP
not ok 4
ok 5
ok 6
ok 7
EOF


done ;

Labels

Edit
plugin-report plugin-report Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.