Spawning processes from build

Skip to end of metadata
Go to start of metadata

Sometimes you'd like to spawn a process from a build that lives longer than the build itself. For example, maybe a part of the build is to launch a new application server with the result of the build. When you do this, you often experience a problem where the build doesn't terminate; you'll see that shell script/ant/maven terminates as expected, but Jenkins just insists on waiting, as if it didn't notice that the build is over.

Starting Jenkins 1.136, Jenkins detects this situation and instead of causing infinite block, it will just print out a warning and let you get going. But you should still understand what's causing this.


Why?

The reason this problem happens is because of file descriptor leak and how they are inherited from one process to another. Jenkins and the child process are connected by three pipes (stdin/stdout/stderr.) This allows Jenkins to capture the output from the child process. Since the child process may write a lot of data to the pipe and quit immediately after that, Jenkins needs to make sure that it drained the pipes before it considers the build to be over. Jenkins does this by waiting for EOF.

When a process terminates for whatever reasons, the operating system closes all the file descriptors it owned. So even if the process didn't close stdout/stderr, Jenkins will nevertheless get EOF.

The complication happens when those file descriptors are inherited to other processes. Let's say the child process forks another process to the background. The background process (AKA daemon) inherits all the file descriptors of the parent, including the writing side of the stdout/stderr pipes that connect the child process and Jenkins. If the daemon forgets to close them, Jenkins won't get EOF for pipes even when the child process exits, because daemon still have those descriptors open. That's how this problem happens.

A good daemon program closes all file descriptors to avoid problems like this, but often there are bad ones that don't follow the rule.

Workarounds

On Unix, you can use a wrapper like this to make the daemon behave. You can call your command like this:

daemonize -E BUILD_ID=dontKillMe /path/to/your/command

Note that this will set the BUILD_ID environment variable for the process being spawned to something other than the current BUILD_ID. Or you can start jenkins with -Dhudson.util.ProcessTree.disable=true - see ProcessTreeKiller for details.

On Windows, the 'at' command can be used to launch a process in the background. See the example below:

<scriptdef name="get-next-minute" language="beanshell">
  <attribute name="property" />

  date = new java.text.SimpleDateFormat("HH:mm")
    .format(new Date(System.currentTimeMillis() + 60000));
  project.setProperty(attributes.get("property"), date);
</scriptdef>

<get-next-minute property="next-minute" />
<exec executable="at">
  <arg value="${next-minute}" />
  <arg value="/interactive" />
  <arg value="${jboss.home}\bin\run.bat" />
</exec>

Another similar workaround on Windows is to use a wrapper script and launch your program through it.

<exec executable="cscript.exe">
   <env key="ANTRUN_TITLE" value="Title for Window" />  <!-- optional -->
   <env key="ANTRUN_OUTPUT" value="output.log" />  <!-- optional -->
   <arg value="//NoLogo" />
   <arg value="antRunAsync.js" />  <!-- this script -->
   <arg value="real executable" />
</exec>

Another workaround for Windows XP and later is to shedule permanent task and force running it from the ant script.
Once run the command:

C:\>SCHTASKS /Create /RU SYSTEM /SC ONSTART /TN Tomcat /TR "C:\Program Files\Apache Software Foundation\Tomcat 6.0\bin\startup.bat"

Note, that ONSTART can be replaced with ONCE if you do not want to keep Tomcat running.
Add the following code to your ant script:

<exec executable="SCHTASKS">
    <arg value="/Run"/>
    <arg value="/TN"/>
    <arg value="Tomcat"/>
</exec>

Another possibility that we can consider is to do something in Jenkins.

Enter labels to add to this page:
Please wait 
Looking for a label? Just start typing.
  1. Jul 04, 2007

    Kohsuke Kawaguchi says:

    For my book keeping purpose, this bug discusses the possibility of using NIO wit...

    For my book keeping purpose, this bug discusses the possibility of using NIO with processes.

  2. May 23, 2009

    Keith Clarke says:

    At least for Tomcat 6, installing Tomcat as a service and using "net start tomca...

    At least for Tomcat 6, installing Tomcat as a service and using "net start tomcat6" and "net stop tomcat6" avoids this problem and is simple.

  3. Jul 29, 2009

    porterhouse91 McGarr says:

    I am currently experiencing this issue on Solaris 10.  I have Hudson runnin...

    I am currently experiencing this issue on Solaris 10.  I have Hudson running and I am trying to setup a job that stops and then starts a local instance of JBoss.  I have been unsuccessful in convincing the SA's to install daemonize as is recommended above.  I am trying to run a shell script to stop/start the JBoss instance, but when the Hudson job completes, the JBoss instance shutdowns.  Does anybody have any recommendations on how we can spawn an instance of JBoss from a Hudson job without using the daemonize script?

    1. Aug 03, 2009

      Duana Stanley says:

      I'm having the same problem trying to start websphere from my ant script via a w...

      I'm having the same problem trying to start websphere from my ant script via a windows batch command (the batch command spawns a child process).  I'd like to know why it works from ant on the command line but not from Hudson.  When run under Hudson the child process that my windows batch file creates gets killed somehow.  I understand there may be some issues around open file descriptors etc, but if it's not a problem for ant, why is it a problem for Hudson? The Hudson build doesn't hang or give any warnings.  I'm working in a limited environment where I can't install services or schedule background tasks. I'm very interested to understand what's behind this problem.

      1. Aug 09, 2009

        Duana Stanley says:

        According to issue 2729 https://hudson.dev.java.net/issues/show_bug.cgi?id=2729 ...

        According to issue 2729 https://hudson.dev.java.net/issues/show_bug.cgi?id=2729 , it sounds like Hudson is willfully killing my build job's beloved spawn.

        They mention a workaround of setting the environment variabe BUILD_ID in the job eg. set BUILD_ID=dontKillMe  which worked for me.

        1. Oct 02, 2010

          Noah Sussman says:

          Thanks for the tip. It took me a while to realize that when a build is done, Hu...

          Thanks for the tip. It took me a while to realize that when a build is done, Hudson kills any child processes that are started by build steps.

          Overriding that default behavior by setting BUILD_ID as you suggest,

          BUILD_ID=allow_to_run_as_daemon start_my_service
          

          worked for me too.

          1. Sep 15, 2011

            Vishal Rekala says:

            I  was having the same issue starting tomcat from shell script.  Setti...

            I  was having the same issue starting tomcat from shell script.  Setting a BUILD_ID solved this problem,  but anybody know how BUILD_ID is related to this issue?

  4. Aug 03, 2009

    Duana Stanley says:

    A simpler way, not involving bean-shell, to get the "next minute" in the ant exa...

    A simpler way, not involving bean-shell, to get the "next minute" in the ant example above using the at command is

    <tstamp>
             <format property="next-minute" pattern="HH:mm" offset="1" unit="minute"/>
    </tstamp>

  5. Aug 04, 2009

    Chetan Sarva says:

    I wasn't able to get the daemonize method to work (not sure why) for a free-styl...

    I wasn't able to get the daemonize method to work (not sure why) for a free-style project on linux but atd works great. I simply schedule my job like so -

     $ echo <command to run> | at `date  -d '+1 minute' +"%H:%M"`

  6. Dec 17, 2009

    pangzi says:

    I use the ' csript.exe'  method run 'c:\apache-tomcat-6.0.18\bin\startup.ba...

    I use the ' csript.exe'  method run 'c:\apache-tomcat-6.0.18\bin\startup.bat'

    on win 2000 and xp ,then it can run on local ant command,but don't run in hudson. --- on win7 ok

    I also try 'at' ,  win 2000 can't execute , xp ok

    why??

    Started by user anonymous
    [workspace] $ cmd /c call C:\DOCUME~1\ADMINI~1\LOCALS~1\Temp\4\hudson2567484063519403955.bat
    C:\Documents and Settings\Administrator\.hudson\jobs\idpro-wap\workspace>cscript //nologo d:\builds\projects\idpro-wap\antRunAsync.js c:\apache-tomcat-6.0.18\bin\startup.bat
    C:\Documents and Settings\Administrator\.hudson\jobs\idpro-wap\workspace>exit 0
    Finished: SUCCESS

    1. Jul 29, 2010

      Shimsha Rao says:

      Hi Pangzi, How did you fix this problem? Even I am facing the similar problem. ...

      Hi Pangzi,

      How did you fix this problem? Even I am facing the similar problem.

      thanks,
      Shimsha

  7. May 06, 2010

    greinhart - says:

    I launch a background command with the plugin  Post build task ... and Huds...

    I launch a background command with the plugin  Post build task ... and Hudson do not kill it.

  8. Sep 08, 2011

    Hila Peleg says:

    After having spent a few hours on this today, it should probably be noted here v...

    After having spent a few hours on this today, it should probably be noted here very specifically that even daemons will be killed once the job finishes since this fix and that its workaround should be used so that the child process can outlive the Jenkins job.

  9. Sep 14, 2011

    Michał Politowski says:

    Apparently in recent Jenkins one has to modify/unset HUDSON_COOKIE, HUDSON_SERVE...

    Apparently in recent Jenkins one has to modify/unset HUDSON_COOKIE, HUDSON_SERVER_COOKIE, JENKINS_COOKIE or JENKINS_SERVER_COOKIE (depending on exact version?) to keep a child process running.

    1. Sep 15, 2011

      Hila Peleg says:

      According to the fix (link above) any environment variable will do. I took the o...

      According to the fix (link above) any environment variable will do. I took the one straight out of the workaround: BUILD_ID=pleaseDontKillMe and it works like a charm.

      1. Sep 19, 2011

        Michał Politowski says:

        Strange, I started with BUILD_ID and it definitely did not work.

        Strange, I started with BUILD_ID and it definitely did not work.

        1. Sep 19, 2011

          Hila Peleg says:

          It seems to be platform dependent. In lin64 and win64 BUILD_ID worked fine for ...

          It seems to be platform dependent.

          In lin64 and win64 BUILD_ID worked fine for me; in win32 it didn't. I changed it to JENKINS_COOKIE for win32 and win64, and it works for win32 but does not for win64. (Lin64 is still doing fine with BUILD_ID...)

          Edit: I take that back; I just re-ran it with JENKINS_COOKIE on win32 and this time it failed. I don't like attributing things to voodoo, but this is starting to make me a little crazy.

        2. Sep 20, 2011

          Michał Politowski says:

          Now BUILD_ID works in free-style jobs, I must have made some other mistake befor...

          Now BUILD_ID works in free-style jobs, I must have made some other mistake before.

          But the problem first happened with a maven project starting a daemon via exec-maven-plugin, and there I still see it.

  10. Dec 07, 2011

    benson margulies says:

    What's the comparison to the ant java task with spawn=true? Does it have this pr...

    What's the comparison to the ant java task with spawn=true? Does it have this problem?

  11. Aug 06, 2012

    malarmannan JB says:

    I am executing Shell script which internally calls expect script then python scr...

    I am executing Shell script which internally calls expect script then python script called from expect script.

    While triggering the script,it has come out after some duration with error

    Error:Process leaked file descriptors. See http://wiki.jenkins-ci.org/display/JENKINS/Spawning+processes+from+build for more information

    iam having thought that May be shell /expect idle mode for more than 10 secs

    it will terminate the job.

    I need concurrent solution for this.

    I have seen some wrapper need to be added for solving the issue.<scriptdef name="get-next-minute" language="beanshell">
    <attribute name="property" />

    date = new java.text.SimpleDateFormat("HH:mm")
    .format(new Date(System.currentTimeMillis() + 60000));
    project.setProperty(attributes.get("property"), date);
    </scriptdef>

    <get-next-minute property="next-minute" />
    <exec executable="at">
    <arg value="$

    Unknown macro: {next-minute}

    " />
    <arg value="/interactive" />
    <arg value="$

    Unknown macro: {jboss.home}

    \bin\run.bat" />
    </exec>As am new to this i dont know where to add /insert the wrapper for solving the issue.

    can any one tell me step by step of adding wrapper or solving the issue

    1. Aug 07, 2012

      sholai rasu says:

      Hi Malarmannan, I've also tried this, i've main shell script -- which ...

      Hi Malarmannan,

      I've also tried this,

      i've main shell script -- which will call an expect script -- and from expect I called a python script...

      now that I tried to start the shell script from jenkin.

      it is starting the expect script successfully.

      but the problem is while triggering the python script, either it is not triggering and not waiting for the process to complete.. it just came out...

      Anyone can help?

  12. Feb 23, 2013

    Joris van der Pol says:

    Hi, is it possible to solve this issue in Jenkins itself? With kind regards, ...

    Hi,

    is it possible to solve this issue in Jenkins itself?

    With kind regards,

    Joris

    1. Feb 24, 2013

      sholai rasu says:

      Hi Joris, we have a framework developed already. using python and shell script....

      Hi Joris,

      we have a framework developed already. using python and shell script.

      It is command line. we wanted to call those scripts from jenkins to add the UI effect.

      there we are stuck.

      Regards,

      Sholai

  13. Aug 07, 2013

    Chris Finn says:

    I ran into this problem in Windows. An easy solution I found was to replace the...

    I ran into this problem in Windows.

    An easy solution I found was to replace the call to start .bat with cscript startBatchFile.wsf.

    In the wsf file I have this code for launching the batch file,

    <job>
        <script language="VBScript">
            Option Explicit
            On Error Resume Next
            Dim WshShell
            set WshShell=CreateObject("WScript.Shell")
            WshShell.run "myBatchFile.bat"
            WScript.Quit
        </script>
    </job>

    This works because WshShell.run doesn't copy the file handlers.

    1. Nov 25

      sameera samarasinghe says:

      Hi Chris, I am also having this problem in Windows( I am using windows 7). In o...

      Hi Chris,

      I am also having this problem in Windows( I am using windows 7). In order to use your method do I have to install VB? How can I call this startBatchFile.wsf file from the Jenkins job?

      Regards!

      Sameera