Monitoring external jobs

Skip to end of metadata
Go to start of metadata

Plugin Information

Plugin ID external-monitor-job Changes In Latest Release
Since Latest Release
Latest Release
Latest Release Date
Required Core
Dependencies
1.2
Aug 29, 2013
1.468
Source Code
Issue Tracking
Maintainer(s)
GitHub
Open Issues
n/a (id: jglick)
Usage Installations 2013-Apr 51449
2013-May 53411
2013-Jun 54632
2013-Jul 58476
2013-Aug 59904
2013-Sep 61785
2013-Oct 66269
2013-Nov 66452
2013-Dec 65277
2014-Jan 70247
2014-Feb 72076
2014-Mar 76323

Adds the ability to monitor the result of externally executed jobs.

Jenkins is useful for monitoring the non-interactive execution of processes, such as cron jobs, procmail, inetd-launched processes. Often those tasks are completely unmonitored (which makes it hard for you to notice when things go wrong), or they send e-mails constantly regardless of the success or failure (which results into the same situation as you'll quickly start ignoring them anyway.) Using Jenkins enables you to monitor a large number of such tasks with little overhead.

Setting up a project

Create a new job and choose "Monitor an external job" as the job type.

Monitoring an execution

For Debian / Ubuntu:

sudo apt-get install jenkins-external-tool-monitor
# obviously you should replace localhost with a FQDN if you want to run jobs from other machines.
export JENKINS_HOME=http://@sorins.uk.xensource.com:8080/export JENKINS_HOME=http://@localhost:8080/
java -jar /usr/share/jenkins/external-job-monitor/java/jenkins-core-*.jar "external-build-job-name" command-to-run
java -jar /usr/share/jenkins/external-job-monitor/java/jenkins-core-*.jar "external-build-test" ls -l

Once you set up a project, you can monitor an execution by running a command like this:

$ export JENKINS_HOME=http://user:pw@myserver.acme.org/path/to/jenkins/
$ java -jar /path/to/WEB-INF/lib/jenkins-core-*.jar "job name" <program arg1 arg2...>

For Windows:

> set JENKINS_HOME=http://user:pw@myserver.acme.org/path/to/jenkins/
> java -jar \path\to\WEB-INF\lib\jenkins-core-*.jar "job name" cmd.exe /c <program arg1 arg2...>

If your webserver extracts the jenkins.war file when it deploys Jenkins then you may use the path directly to the WEB-INF/lib directory and all other required jars will be found there. Otherwise you may extract these from the war file:

jenkins-core-*.jar
remoting-*.jar
ant-1.7.0.jar
commons-io-1.4jar
commons-lang-2.4.jar
jna-posix-*.jar
xstream-*.jar

All are found in the WEB-INF/lib path inside the war file. As long as they are all in the same directory, the java -jar /path/to/jenkins-core-*.jar command will find the other required jars.

  • Note: Older versions of Jenkins (before 1.324) also require winstone.jar in order to run this command. This jar is found at the top level directory inside the war file, and must be manually added to the classpath (with -classpath or -cp) in the java command.

The JENKINS_HOME variable is used to locate the server Jenkins is running, so this must be set. Unless your Jenkins job has build permission for guest users, include the username:password@ portion of the URL, as seen in the examples above.

  • Note: Authentication via username:password in JENKINS_HOME is added in Jenkins 1.324; with previous versions either grant anonymous build permission on the job, or use curl to post XML to Jenkins (see below).

You can copy jenkins-core-*.jar and the other required jars to other machines if you want to monitor jobs that are run on a different machine.

stdout and stderr of the program will be recorded, and a non-zero exit code will be considered as a failure.

Monitoring cron jobs

To monitor a cron job, simply run the above set up from your cron script. To avoid receiving e-mails from cron daemon, you might want to write something like:

JENKINS_HOME=http://myserver.acme.org/path/to/jenkins/
0 * * * *     export JENKINS_HOME=$JENKINS_HOME; java -jar jenkins-core-*.jar "backup" backup.sh 2>&1 > /dev/null

Note that you can also move the cron job itself to Jenkins by using free-style software project, which would also allow you to manually execute the job outside the scheduled executions.

Submit a run programatically

The above command submits the execution and its result by sending XML to HTTP. This means you can submit an execution record from any program, as long as you follow the same XML format.

The format is explained below:

<run>
  <log encoding="hexBinary">...hex binary encoded console output...</log>
  <result>... integer indicating the error code. 0 is success and everything else is failure</result>
  <duration>... milliseconds it took to execute this run ...</duration>
  <displayName>... The name to be displayed rather than the build number ...</displayName>
  <description>... Description of the build ...</description>
</run>

The duration element is optional, since 1.429 displayName and description can also optionally be appended, these three elements can be in any order, but must appear after log and result in that order. Console output is hexBinary encoded so that you can pass in any control characters that are otherwise disallowed in XML.

The above XML needs to be sent to http://myhost/jenkins/job/_jobName_/postBuildResult.

A simple example using curl would be (using no real data; the sequence 4142430A encodes a console output of ABC plus a trailing line feed symbol):

$ curl -X POST -d '<run><log encoding="hexBinary">4142430A</log><result>0</result><duration>2000</duration></run>' \
http://user:pass@myhost/jenkins/job/_jobName_/postBuildResult

Submit a run per CLI

The easiest option is for the execution can be submitted per CLI/ssh command. The gzipped log file can be transported through pipe:

$ cat result.log.gz | ssh jenkins set-external-build-result --display 7d552c4ba_Linux_tb21 --job buildbot --result 1 -b --duration 42 --log -

Sometimes build number is needed, as the calling program might need to put an URL to the submitted build. The CLI command above returns the new build number. The command above can be called programatically from an Java application. You can find a sample for it here.

Changelog

Version 1.2 (Aug 29, 2013)

  • new CLI command (pull #5)
  • issue #19377 workaround for log rotation failure
  • NPE fix
  • translations

Labels

plugin-misc plugin-misc Delete
plugin-external plugin-external Delete
Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Sep 06, 2007

    Anonymous says:

    If you get: ----Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup...

    If you get:

    ----Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup.sh: stdout copier" java.lang.NoClassDefFoundError: org/apache/tools/ant/BuildException
            at hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)
    Exception in thread "/opt/kuttig/Dev1-Backup-Scripts/svn_backup.sh: stderr copier" java.lang.NoClassDefFoundError: org/apache/tools/ant/BuildException
            at hudson.util.StreamCopyThread.run(StreamCopyThread.java:27)

    ----call "-Xbootclasspath/a:/usr/local/apache-ant/lib/ant.jar" to your java call (modify the ant path!). So this may result in:
    export HUDSON_HOME=http://localhost:8080/hudson; java -Xbootclasspath/a:/usr/local/apache-ant/lib/ant.jar -jar hudson-core-1.136.jar "SVN_Backup" /opt/svn/svn_backup.sh

    - zeisss 

    1. Sep 16, 2007

      Kohsuke Kawaguchi says:

      This issue is filed as issue #802.

      This issue is filed as issue #802.

  2. Mar 29, 2008

    kevinket - says:

    How about running on a Windows machine. I've created a bat file like this: set ...

    How about running on a Windows machine. I've created a bat file like this:

    set HUDSON_HOME=http://localhost:8080
    java -jar C:\hudson_master\lib\hudson-core-1.200.jar "build_release_12.0" call_build.bat release_12.0

    I get this error (which indicates it cannot file hudsonhome): 

    Exception in thread "main" java.lang.NoClassDefFoundError: hudson/remoting/Callable
            at hudson.Main.getHudsonHome(Main.java:47)
            at hudson.Main.run(Main.java:37)
            at hudson.Main.main(Main.java:29)
    Caused by: java.lang.ClassNotFoundException: hudson.remoting.Callable
            at java.net.URLClassLoader$1.run(Unknown Source)
            at java.security.AccessController.doPrivileged(Native Method)
            at java.net.URLClassLoader.findClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClass(Unknown Source)
            at java.lang.ClassLoader.loadClassInternal(Unknown Source)
            ... 3 more

     I'd think that this should work on a Windows machine as well.

    1. Aug 22, 2008

      Drew Cox says:

      I had this too (on Linux though), fixed it by copying the "remoting" jar out of ...

      I had this too (on Linux though), fixed it by copying the "remoting" jar out of the distro and into the same directory as the hudson-core jar.  The husdon-core jar has a long classpath defined in its manifest.mf, I haven't quite figured out what dependencies are required for remote monitoring, as I now have this error:

      Exception in thread "catalog.full.zip: stdout copier" java.lang.StringIndexOutOfBoundsException: String index out of range: -3
          at java.lang.String.charAt(String.java:687)
          at hudson.util.EncodingStream.write(EncodingStream.java:21)
          at java.io.FilterOutputStream.write(FilterOutputStream.java:108)
          at hudson.util.DualOutputStream.write(DualOutputStream.java:29)
          at hudson.util.StreamCopyThread.run(StreamCopyThread.java:28)

  3. Mar 20, 2009

    pcohen - says:

    Hi I've been using Hudson for 2 months now and I must say that it is a really ni...

    Hi I've been using Hudson for 2 months now and I must say that it is a really nice tools and there are so many available plugins.

    It is really nice.

    Today I started to play with the monitoring of external jobs and had an authentication error even when I set in the HUDSON_HOME env variable the userinfo user:password in the URL.

    Looking in the code Main.java , I added

     

                  String encoding = new sun.misc.BASE64Encoder().encode(new URL(location).getUserInfo().getBytes());
                  con.setRequestProperty ("Authorization", "Basic " + encoding);
    

    That works now but I am not sure this is perfect as it assumes that authentication is expected.

    Anyway that could perhaps be interesting to add such feature one way or another.

     Thanks

    Pascal

  4. Mar 30, 2009

    Bryan Oakley says:

    Is it possible to use this technique to send back data formatted as HTML? I'd li...

    Is it possible to use this technique to send back data formatted as HTML? I'd like to be able to include some hyperlinks and bold text into the information that is generated on the console output page.

  5. Jun 01, 2009

    Josh Koenig says:

    I am also wondering about log formats. Haven't had any luck (yet) getting intell...

    I am also wondering about log formats. Haven't had any luck (yet) getting intelligible results via HTTP POST and the XML format.

  6. Jul 02, 2009

    Prem Polavarapu says:

     Got this working using the following command. java -cp remoting-1.31...

     Got this working using the following command.

    java -cp remoting-1.313.jar:hudson-core-1.313.jar:servlet.jar hudson.Main "Test" Test.groovy

  7. Jul 16, 2009

    moon people says:

    Example of working command: export HUDSON_HOME=http://localhost:8080/hudson java...

    Example of working command:
    export HUDSON_HOME=http://localhost:8080/hudson
    java -classpath remoting.jar:./lib/hudson-core-1.315.jar:/opt/hudson/apache-tomcat-6.0.20/lib/servlet-api.jar hudson.Main "test-mon" ksh ./test_server_health.ksh

    Details:
    I was having a lot of issues trying to get external jobs running successfully, but I figured it out. Here's what I did

    Hudson version: 1.315

    (running all this on OS X 10.5.6)

    1) created dir /opt/hudson
    2) installed apache-tomcat-6.0.20 into that directory (/opt/hudson/apache-tomcat-6.0.20)
    3) dropped hudson.war (version 1.315) into /opt/hudson/apache-tomcat-6.0.20/webapps dir
    4) started up tomcat (you may have a few config tweaks to make in server.xml withing your apache-tomcat-6.0.20 dir)
    5) went to http://localhost:8080/hudson saw that hudson was running ok
    6) Hudson -> New Job -> Monitor an external job (gave it a title of test-mon) -> OK
    7) cd into /opt/hudson/apache-tomcat-6.0.20/webapps/hudson/WEB-INF
    8) set HUDSON_HOME=http://localhost:8080 (this can also be set to an entirely different hudson server. it does not have to be on your local machine to work.)
    9)from the /opt/hudson/apache-tomcat-6.0.20/webapps/hudson/WEB-INF directory I ran the following:
    10) java -classpath remoting.jar:./lib/hudson-core-1.315.jar:/opt/hudson/apache-tomcat-6.0.20/lib/servlet-api.jar hudson.Main "test-mon" ksh ./test_server_health.ksh
    11) Important: make sure to pass the language executor before the script (note: I ran 'ksh ./test_server_health.ksh' I tried many times without the 'ksh' and it doesn't work)
    12) went to http://localhost:8080/hudson and saw my test passed.

    the hardest thing I had to deal with was getting the classpath correct. you need remoting.jar, hudson-core-1.315.jar(of course the versioning will be different for you, 1.315 is the latest version as of today) & a servlet.jar (I chose servlet-api.jar from the apache lib directory out of frustration, so there may be another that exists that is the correct one.)

    in the command I provided I'm running a korn shell script (which is why you see the ksh before the actual script). you could run anything and typically groovy is sited as an example, but it can be bash, perl, ruby.

    also realized that where ever you set HUDSON_HOME to is where the test results get posted, so setting HUDSON_HOME=http://localhost:8080/hudson and looking for results on http://cool.company.com/hudson will make you go crazy

    good luck!!

  8. Oct 05, 2009

    Michael Alsen says:

    Alternative solution for Linux: http://n2.nabble.com/hudson-monitoring-externa...
  9. May 21, 2010

    Jeff Peck says:

    It seems I also need jna-posix-1.0.3.jar (hudson 1.358)

    It seems I also need jna-posix-1.0.3.jar (hudson 1.358)

  10. Jun 24, 2010

    Derek Choi says:

    I am attempting to connect a set of procmail executions and cronjobs with hudson...

    I am attempting to connect a set of procmail executions and cronjobs with hudson, which works great.  However, one thing that I noticed during my testing is if the hudson server itself goes down, the executions don't run at all, which puts a dependency on Hudson CI being up.  Is it possible to pass  a command line argument to ignore errors communicating with hudson and continue with the execution?

    1. Sep 23, 2011

      Andrew Nutter-Upham says:

      This may be too late for Derek, but perhaps it will help others. Here is a perl...

      This may be too late for Derek, but perhaps it will help others.

      Here is a perl script that does the jenkins external job submission. The job will always run; Jenkins notification is secondary.

      https://gist.github.com/1237898

  11. Nov 11, 2011

    Ivan Mounitsyn says:

    1. If you're getting java exeptions when setting up monitoring of external jobs ...

    1. If you're getting java exeptions when setting up monitoring of external jobs that could be caused by the fact that you need commons-io-*.jar added to your client Jenkins setup. The list of 5 jars above seems incomplete.

    2. If your external job is memory-hungry (tries to grab more than 2Gb of ram) make sure you're running 64bit Java. Sounds obvious? Well, for me it wasn't ...

  12. Feb 14, 2012

    Kumar Dev says:

    Hi, I have setup up a jenkins system and trying to set up a job to monitor a cr...

    Hi,

    I have setup up a jenkins system and trying to set up a job to monitor a cron job. My jenkins URL is http://dev1:8080 .This is what I'm doing on server dev02:

    JENKINS_HOME=http://dev01:8080

    export JENKINS_HOME=$JENKINS_HOME; java -jar jenkins-core-*.jar "Dev01 Report" kdev.sh 2>&1 > /dev/null

    I am getting below message

    Unable to access jarfile jenkins-core-*.jar

    I have created a monitoring job Dev01 Report on jenkins dashboard.

    I also tried

    export JENKINS_HOME=$JENKINS_HOME; java -jar jenkins-core-1.451.jar "SUN11 Report" kdev.sh 2>&1 > /dev/null

    but getting same error

    Unable to access jarfile jenkins-core-1.451.jar

    Can someone please help?

    Thanks.

    1. Mar 01, 2012

      Kumar Dev says:

      I managed to get past this one. I had to copy below files on to the remote serve...

      I managed to get past this one. I had to copy below files on to the remote server from $JENKINS_HOME/.jenkins/war/WEB-INF/lib/jenkins-core-*.jar
      remoting-*.jar
      ant-1.7.0.jar
      commons-lang-2.4.jar
      xstream-*.jar

      But now getting this error

      $ java -jar jenkins-core-1.451.jar "Dev02 Report" kdev.sh 2>&1 > /dev/null
      Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/io/input/NullInputStream
              at hudson.Main.remotePost(Main.java:151)
              at hudson.Main.run(Main.java:74)
              at hudson.Main.main(Main.java:56)
      Caused by: java.lang.ClassNotFoundException: org.apache.commons.io.input.NullInputStream
              at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
              ... 3 more
      $

      Do I need to copy more files from $JENKINS_HOME/.jenkins/war/WEB-INF/lib to remote server?Any pointers anyone?

      1. Feb 15, 2012

        Robert Kahlert says:

        You are missing the commons-io*.jar  In my release of Jenkins, its commons-...

        You are missing the commons-io*.jar  In my release of Jenkins, its commons-io-1.4.jar.

        1. Feb 16, 2012

          Kumar Dev says:

          Tried running after copying commons-io*.jar file as well and got below error $ ...

          Tried running after copying commons-io*.jar file as well and got below error

          $ java -jar jenkins-core-1.451.jar "Dev02 Report" kdev.sh 2>&1 > /dev/null

          Exception in thread "main" java.lang.NoClassDefFoundError: org/jruby/ext/posix/POSIX
                  at hudson.Main.remotePost(Main.java:174)
                  at hudson.Main.run(Main.java:74)
                  at hudson.Main.main(Main.java:56)
          Caused by: java.lang.ClassNotFoundException: org.jruby.ext.posix.POSIX
                  at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
                  at java.security.AccessController.doPrivileged(Native Method)
                  at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:307)
                  at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
                  at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
                  ... 3 more

          Then copied jna-posix-1.0.3.jar and it went fine. So to summarize, I think we need below files on the remote server for remote monitoring to work properly. My server/os version is SunOS dev02 5.10 Generic_142909-17 sun4u sparc SUNW,Sun-Fire-V445

          Files copied

          jenkins-core-1.451.jar

          xstream-1.3.1-jenkins-9.jar

          ant-1.8.0.jar

          commons-lang-2.4.jar

          jenkins-core-1.451.jar

          remoting-2.12.jar

          xstream-1.3.1-jenkins-9.jar

          commons-io-1.4.jar

          jna-posix-1.0.3.jar

          Thanks for help.

          1. Mar 04, 2012

            Kumar Dev says:

            I have put together the steps I followed to "Monitor External Jobs" using Jenkin...

            I have put together the steps I followed to "Monitor External Jobs" using Jenkins at link below:

            http://kvdev.wordpress.com/2012/03/01/jenkins-monitoring-external-jobs/

            Hope everyone finds this useful. Feedback welcome.

  13. Apr 24, 2012

    zhewei shou says:

    when i export JENKINS_HOME(which is not localhost path), i need pass username &a...

    when i export JENKINS_HOME(which is not localhost path), i need pass username & password. but what can i do if the password exists character like '@'?

  14. Apr 24, 2012

    zhewei shou says:

    when i export JENKINS_HOME(which is not localhost path), i need pass username &a...

    when i export JENKINS_HOME(which is not localhost path), i need pass username & password. but what can i do if the password exists character like '@'?

  15. Jun 30, 2012

    Dave Kelsey says:

    I'm using the latest rev of jenkins 1.472. I've set it up locally and tried bot...

    I'm using the latest rev of jenkins 1.472.

    I've set it up locally and tried both a remote and local test with ls -l

    This is what I see in the console ouput:

    Started
    FATAL: tried to access field hudson.model.Run.charset from class hudson.model.ExternalRun$2
    java.lang.IllegalAccessError: tried to access field hudson.model.Run.charset from class hudson.model.ExternalRun$2
    	at hudson.model.ExternalRun$2.run(ExternalRun.java:121)
    	at hudson.model.Run.execute(Run.java:1488)
    
    ...
    

    There is a ticket in the issue tracking system:

    https://issues.jenkins-ci.org/browse/JENKINS-14107

    If other people are experiencing this please create an account and vote for the issue.

    1. Jun 30, 2012

      Kumar Dev says:

      Hi, I have set this up to run both remotely and locally. I did face some issues...

      Hi,

      I have set this up to run both remotely and locally. I did face some issues initially but finally managed to fix it. I have put down the steps to setup jenkins from scratch at below link, can you please take a look and confirm that you haven't missed anything:

      http://kvdev.wordpress.com/2012/03/01/jenkins-monitoring-external-jobs/

      I have set this up on solaris, but the steps should work on any *nix system. Please let me know if this helps.

      Thanks.

      1. Jul 13, 2012

        Dave Kelsey says:

        Kumar, Thanks for our help.  Nope.  it didn't work.  I notice yo...

        Kumar,

        Thanks for our help.  Nope.  it didn't work.  I notice you are using version 1.451.    I tried it on solars and on my mac.  I get the same error.

        ...

        Update: Kumar, I rolled back to ver 1.464 and the ls -l example worked.  The latest release doesn't work

  16. Aug 28, 2012

    Aldo Reyes says:

    Hi, Im enjoying every bit of jenkins so far, I need to send a notification emai...

    Hi,

    Im enjoying every bit of jenkins so far, I need to send a notification email after a cron job finish, I see that there is no way to add post build tasks on this kind of project, I also tried the strawboss plugin but that doesnt work on newer jenkins versions (it seems outdated) is there a way to do this?

    I know a workaround is to use a ssh plugin, but I would like to know if there is something native in jenkins or with a plugin that only lets me add a post build step to these kind of jobs, thanks for the great work guys!

  17. Sep 07, 2012

    Frank Merrow says:

    So we have a number of Jenkins . . . and for the most part they are completely s...

    So we have a number of Jenkins . . . and for the most part they are completely separate . . .

    However, we ran into the situation where one Jenkins didn't have enough slaves to run a particular assessment and so it go put onto a separate Jenkins.

    What I would really like is for the status of "that job on that Jenkins" to show up normally on "this Jenkins".

    Is that possible . . . this plugin seems to require a change to the way the extrnal process is involked . . . it really don't want to change the "remote" Jenkins job . . . only have it show up here too.

    Is that possible?

    Frank

  18. Nov 01, 2012

    ozmen adibelli says:

    Hi, I can not find this external-monitor-job plugin in "available plugins"...

    Hi,

    I can not find this external-monitor-job plugin in "available plugins" page. How can I install ?

    1. Nov 01, 2012

      Kumar Dev says:

      Hi, Why do you need such a plugin? I don't think one exists. What is it that yo...

      Hi,

      Why do you need such a plugin? I don't think one exists. What is it that you are exactly trying to do? If you are trying to monitor an external job then you need to configure this in jenkins. Follow the instructions listed at below link and let me know:

      http://kvdev.wordpress.com/2012/03/01/jenkins-monitoring-external-jobs/

      Thanks,

      KVD

  19. Aug 20, 2013

    Vishal Rekala says:

    How do i install this plugin, i don't see it in plugin manager -> Availab...

    How do i install this plugin, i don't see it in plugin manager -> Available plugins???