Robot Framework Plugin

Skip to end of metadata
Go to start of metadata

Plugin Information

Plugin ID robot Changes In Latest Release
Since Latest Release
Latest Release
Latest Release Date
Required Core
Dependencies
1.4.2
Mar 14, 2014
1.424
token-macro (version:1.5.1, optional)
Source Code
Issue Tracking
Maintainer(s)
GitHub
Open Issues
Janne Piironen (id: jpiironen)
Usage Installations 2013-Apr 727
2013-May 763
2013-Jun 781
2013-Jul 832
2013-Aug 858
2013-Sep 911
2013-Oct 980
2013-Nov 998
2013-Dec 1017
2014-Jan 1096
2014-Feb 1149
2014-Mar 1265

Description

This plugin collects and publishes Robot Framework test results in Jenkins.

Current features

  • Test suite and case details with trend graphs
  • Environment variable expansion for build paths in configuration
  • Wildcard support for output files for parsing and archiving of multiple Robot result files
  • Collection and archiving of original Robot report HTML files
  • Summary of Robot run on build page
  • Summary of latest Robot run on project page
  • Trend graph of passed tests over builds on project page
  • Marking build failed/unstable/successful according to pass thresholds given by user
  • Option of evaluating only critical tests against the pass thresholds
  • Listviewcolumn to show overall passed/failed tests in project listing
  • Configurable archiving of arbitrary Robot related artifacts per build 
  • Token macros for usage with e.g. email-ext plugin
  • Test visibility in radiator views

Configuration

Basic configuration of testing task

  1. Configure your project
  2. Select: Build -> Add build step -> execute shell/Execute Windows batch command
  3. Add the command to run the tests. You might have direct pybot/jybot command here like "pybot --variable name:value --outputdir my_robot_results my_test_suite" or in case you need to do something more, you can call any script that you use for running the tests. Relevant part from Robot Jenkins Plugin point of view is that this script generates the Robot outputs.
  4. Force your Robot script to return successfully from shell with "exit 0" to empower the plugin in deciding if the build is success/failure (by default Robot exits with error code when there's any failed tests)
  5. Select: Post-build Actions -> Publish Robot Framework test results
  6. Set path where your results are located (in above example command "my_robot_results")
  7. If your output files are named differently than default (i.e. output.xml, report.html and log.html) specify the filenames by pressing the "Advanced..." button and writing the names in relevant fields. The fields support wildcards * and ?.
  8. If you have other artifacts such as screenshots that you want to persist for viewing later in the logs you must configure them under "Advanced... -> Other files to copy". The field accepts comma separated list of files and supports wildcards * and ?.
  9. Set thresholds and optionally disable thresholds for critical tests only to count every test in the pass percentage.

Configuring direct links to log files

Version 1.3.0

Links are automatically generated to whatever files are configured to "Report html" and "Log html" in the plugin configuration. Links to log and report htmls are now only in the summary sections in the middle of the page and the sidebar link from previous versions have been removed.

Version 1.2.3 and earlier

From version 1.2 it's possible to configure a direct link from both sidepanel and project/build page summaries to chosen file persisted with the build. If no file is configured the link won't show up. Steps to do this are as follows:

  1. Go to project configuration
  2. Configure the name of the file to be linked to Log/Report link field
  3. Make sure that you are saving the actual file you want to link with the build by checking the filenames/masks configured in the fields behind "Advanced..." button. By default these are output.xml, report.html and log.html which are the Robot Framework default names
  4. Save the configuration and run a build. The links are persisted per build in order to withstand changes in file naming. Thus the links won't appear in builds that exist already.
  5. Links should appear in the project/build summaries and in build sidepanel.

NOTE: After configuration change etc. the link will appear also in project sidepanel, but for now it doesn't appear there right after the build. Conversely, the project view sidepanel link won't go away until after configuration change/resave in project. We're working on a solution to this.


Sidepanel link


Summary link

Configuring Robot overall pass/fail to show in the project list

  1. Go to Jenkins front page
  2. Select the + sign in the top tabs of the project listing to create a new view or go to http://YOURJENKINSHOST/newView
  3. Name your view and proceed creating a list view
  4. Choose project you want to include to the view. By default the "Robot pass/fail" column will appear to the column listing.
  5. Save view and select it from the top tabs in Jenkins project listing
  6. (Optional) If you want to have the view on by default go to "Manage Jenkins"->"Configure System" (or http://YOURJENKINSHOST/configure )and select your newly created view to be the default one.


List view column in action

Using token macros in e-mail report

Prerequisites: token-macro plugin and email-ext plugin installed.

  1. Go to project configuration
  2. Enable "Editable Email Notification" in post-build actions (configuration reference)
  3. Now you can use the following Robot variables in your custom result email:
  • ${ROBOT_FAILEDCASES} - Expands to list of failed Robot cases. Each case on its own line.
  • ${ROBOT_PASSPERCENTAGE, onlyCritical} - Expands to pass percentage of tests. (passed / total * 100 %),
    onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
  • ${ROBOT_PASSRATIO, onlyCritical} Expands to build result in 'passed / total' format.
    onlyCritical - True if only critical tests should be calculated in percentage. Defaults to false.
  • ${ROBOT_REPORTLINK} - If logfile link is configured in the Robot plugin this link will point to that file for the build. Else show link to Robot reports directory for the build.

Displaying test numbers in build radiator views etc.

If the Robot plugin marks the build as failure the tests will not show up. This is because only unstable and successful builds are considered to have test results per Jenkins.

This means that in order to see the test results in other views you must set your unstable threshold so that the build never goes to failure.

Overall Screenshots

Config

Project view

Build view

Detailed build view

Changelog

Release 1.4.2

  • JENKINS-22060 Plugin does not count suite setup times into duration
  • JENKINS-21736 Make the archiving of output.xml optional

Release 1.4.1

  • JENKINS-21489 URL decoding for test objects fails with special characters
  • JENKINS-21490 Split token-macro tokens further to enable more custom reporting

Release 1.4.0

  • JENKINS-8381 Robot summary and trend graph for multi-configuration projects

Release 1.3.2

  • JENKINS-19479 Robot Framework Plugin Threshold uses rounded percentage for evaluating pass/unstable build status
  • JENKINS-19484 Robot Framework plugin shows result as "Failed!" on individual test case result page

Release 1.3.1

  • JENKINS-19328 Whole directory is copied when "Other files to copy" -field is empty

Release 1.3.0

  • JENKINS-15809 Failed to scout hudson.plugins.robot.tokens.RobotPassPercentageTokenMacro
  • JENKINS-15188 Human readable duration values for test summaries and trend graphs
  • JENKINS-15191 Robot icon is scaled ugly in many places
  • JENKINS-15193 Graph scaling to see the significant changes in test count / duration
  • JENKINS-12795 Log/Report link in Job page is broken when Jenkins is started with a specific prefix
  • JENKINS-19064 Project/build page summary improvements
  • JENKINS-15768 Plug-in doesn't retain directories when copy files using "Other files to copy" setting
  • JENKINS-16615 Link to log.html is broken on project page in matrix project
  • JENKINS-18675 Failure in suite teardown does not fail test cases
  • JENKINS-17328 output.xml kept open after build if parsing error occurs causing subsequent runs to fail
  • JENKINS-15327 Publish Robot Framework test results not support chinese character

Release 1.2.3

  • JENKINS-15194 - Basic token macros created for use in e.g. email-ext plugin
  • JENKINS-15187 - Made the plugin report tests as unit tests in order to see test count in different views e.g. radiator plugins (see config notice above)

Release 1.2.2

  • Changed XML parsing to much more memory efficient StAX. Should solve many outOfMemoryErrors.
  • Changed the way suites with same names are handled
  • Fixed a bug where empty "Other files to copy" field leads to copying the whole directory into saved builds.

Release 1.2.1

  • Compliance with RF 2.6 log file changes (html and javascript files separated)
  • Option to save arbitrary test artifacts with test results

Release 1.2

Release 1.1

  • JENKINS-9078 - Jenkins environment variables available for configurations
  • JENKINS-9079 - Support for GLOB style wildcards in file configurations
  • Detailed views of test cases and suites with trend graphs
  • Fix to JDK 1.5 incompliant exception throw

Release 1.0

  • Initial release of the plugin

Labels

Edit
plugin-report plugin-report Delete
token-producer token-producer Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.

Add Comment