Jenkins : Strategies for finding Test Results

After your automated tests are executed the plug-in has to look for Test Results. This way it will know whether an automated test was executed correctly or not and then update the test status in TestLink.

TestNG result seeking strategies

TestNG class name: The plug-in matches the TestNG class name (e.g.: org.tap4j.TestParser) and the key custom field value.

TestNG method name: The plug-in matches the TestNG method name (e.g.: org.tap4j.TestParser#testTokenizer) and the key custom field value.

TestNG suite name: The plug-in matches the TestNG suite name (e.g.: functional tests) and the key custom field value.

JUnit result seeking strategies

JUnit case class name: The plug-in matches the JUnit case class name (e.g.: org.tap4j.TestParser) and the key custom field value.

JUnit case name: The plug-in matches the JUnit case class name (e.g.: testTokenizer) and the key custom field value.

JUnit method name: The plug-in matches the JUnit case class name (e.g.: org.tap4j.TestParser#testTokenizer) and the key custom field value.

JUnit suite name: The plug-in matches the JUnit suite name (e.g.:functional tests) and the key custom field value.

TAP result seeking strategies

TAP file name: The plug-in matches the TAP file name (e.g.: testFtpProtocol.tap) and the key custom field value. 

New proposal

Oliver Merkel and Yury Kryazhev proposed changes to JUnit and TestNG result seeking process. I will try to organize them here for open discussion. Everybody is invited to comment, add suggestions or ideas, even if you are not using the plug-in :)-

1. Combine suite name and test case while looking for test results

If a test case is used in different test suites. It may cause the plug-in to report a test result twice in some situations. We could use the following format: "testSuiteName"."testCaseNameOrClass".

2. Add the possibility to use test methods

This proposal would add an extra option in the plug-in to match custom field values against test method names. I hadn't implemented it in version 1.0 for thinking that the method name could repeat inside other classes and that a test case in TestLink usually would be related to a test case or test suite. However, by adding this option to the plug-in we extend the possibilities of how to use it in different build pipelines. Perhaps this option could be implemented as in #1.

3. Add an option to define whether the plug-in should look for test case or test class name

In JUnit, some tools use test case name, and others the test class name attribute. At present, the plug-in tries to find a class name, and if not present then it uses the test case name. We could let the user decide which option the plug-in must utilize. This way we prevent the plug-in from running in situations where it uses test class name while it should be using the test case name. Below is an example which could cause wrong assertions in the plug-in.

<?xml version="1.0" encoding="UTF-8"?>

<testsuites tests="…" failures="…" disabled="0" errors="…" time="…" name="AllTests">

  <testsuite name="MyClass" tests="…" failures="…" disabled="0" errors="0" time="0">

    <testcase name="MethodOne" status="run" time="0" classname="MyClass" />

    <testcase name="MethodTwo" status="run" time="0" classname="MyClass" />

…

  </testsuite>

</testsuites>