Sun's Hudson deployment : Archived artifact is missing

The latest incarnation is in http://kohsuke.sfbay/hudson/view/GFv3/job/glassfish-v3/4524/

The error is:

Deploying module: glassfish-v3 ? Phobos connector for Glassfish V3 #4524 timestamp=java.util.GregorianCalendar[time=1213668692269,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="US/Pacific",offset=-28800000,dstSavings=3600000,useDaylight=true,transitions=185,lastRule=java.util.SimpleTimeZone[id=US/Pacific,offset=-28800000,dstSavings=3600000,useDaylight=true,startYear=0,startMode=3,startMonth=2,startDay=8,startDayOfWeek=1,startTime=7200000,startTimeMode=0,endMode=3,endMonth=10,endDay=1,endDayOfWeek=1,endTime=7200000,endTimeMode=0]],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2008,MONTH=5,WEEK_OF_YEAR=25,WEEK_OF_MONTH=3,DAY_OF_MONTH=16,DAY_OF_YEAR=168,DAY_OF_WEEK=2,DAY_OF_WEEK_IN_MONTH=3,AM_PM=1,HOUR=7,HOUR_OF_DAY=19,MINUTE=11,SECOND=32,MILLISECOND=269,ZONE_OFFSET=-28800000,DST_OFFSET=3600000]

FATAL: Archived artifact is missing: /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-16_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/pom.xml
java.io.IOException: Archived artifact is missing: /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-16_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/pom.xml
	at hudson.maven.reporters.MavenArtifact.getFile(MavenArtifact.java:147)
	at hudson.maven.reporters.MavenArtifactRecord.deploy(MavenArtifactRecord.java:93)
	at hudson.maven.reporters.MavenAggregatedArtifactRecord.deploy(MavenAggregatedArtifactRecord.java:56)
	at hudson.maven.RedeployPublisher.perform(RedeployPublisher.java:67)
	at hudson.model.AbstractBuild$AbstractRunner.performAllBuildStep(AbstractBuild.java:309)
	at hudson.maven.MavenModuleSetBuild$RunnerImpl.post2(MavenModuleSetBuild.java:434)
	at hudson.model.AbstractBuild$AbstractRunner.post(AbstractBuild.java:282)
	at hudson.model.Run.run(Run.java:789)
	at hudson.maven.MavenModuleSetBuild.run(MavenModuleSetBuild.java:202)
	at hudson.model.ResourceController.execute(ResourceController.java:70)
	at hudson.model.Executor.run(Executor.java:82)

First, what is the build number of 2008/06/16 19:11:32?

hudson@kohsuke2 ~/server/jobs/glassfish-v3/builds $ less 2008-06-16_19-11-20/build.xml 
<?xml version='1.0' encoding='UTF-8'?>
<hudson.maven.MavenModuleSetBuild>
  <builtOn>sebastian</builtOn>
  <hudsonVersion>1.226-SNAPSHOT (private-06/17/2008 01:22-kohsuke)</hudsonVersion>
  <scm class="hudson.scm.SubversionChangeLogParser"/>
  <culprits/>
  <number>4524</number>
  <result>FAILURE</result>

The error is that POM is not archived. And indeed it is not. We just have jars:

hudson@kohsuke2 ~/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-16_19-11-32/arc
hive $ ls -lR
.:
total 2
drwxr-sr-x   3 hudson   hudson       512 Jun 16 19:18 org.glassfish.extras

./org.glassfish.extras:
total 2
drwxr-sr-x   3 hudson   hudson       512 Jun 16 19:18 gf-phobos-connector

./org.glassfish.extras/gf-phobos-connector:
total 2
drwxr-sr-x   2 hudson   hudson       512 Jun 16 19:18 10.0-SNAPSHOT

./org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT:
total 30
-rw-r--r--   1 hudson   hudson      5296 Jun 16 19:18 gf-phobos-connector-10.0-SNAPSHOT-sources.jar
-rw-r--r--   1 hudson   hudson      8223 Jun 16 19:18 gf-phobos-connector-10.0-SNAPSHOT.jar

But earlier we seem to be attempting to archive this:

[HUDSON] Archiving /files/hudson/workspace/glassfish-v3/v3/extras/phobos/pom.xml to /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-12_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/pom.xml
[HUDSON] Archiving /files/hudson/workspace/glassfish-v3/v3/extras/phobos/target/gf-phobos-connector-10.0-SNAPSHOT.jar to /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-16_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/gf-phobos-connector-10.0-SNAPSHOT.jar
[HUDSON] Archiving /files/hudson/workspace/glassfish-v3/v3/extras/phobos/target/gf-phobos-connector-10.0-SNAPSHOT-sources.jar to /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-16_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/gf-phobos-connector-10.0-SNAPSHOT-sources.jar

.... oh, wait!!! If I format the above 3 lines more nicely, we see that POM went to the wrong time stamp: 2008-06-12_19-11-32, not 2008-06-16_19-11-32

[HUDSON] Archiving
  /files/hudson/workspace/glassfish-v3/v3/extras/phobos/pom.xml
  to
  /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-12_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/pom.xml
[HUDSON] Archiving
  /files/hudson/workspace/glassfish-v3/v3/extras/phobos/target/gf-phobos-connector-10.0-SNAPSHOT.jar
  to
  /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-16_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/gf-phobos-connector-10.0-SNAPSHOT.jar
[HUDSON] Archiving
  /files/hudson/workspace/glassfish-v3/v3/extras/phobos/target/gf-phobos-connector-10.0-SNAPSHOT-sources.jar
  to 
  /files/hudson/server/jobs/glassfish-v3/modules/org.glassfish.extras$gf-phobos-connector/builds/2008-06-16_19-11-32/archive/org.glassfish.extras/gf-phobos-connector/10.0-SNAPSHOT/gf-phobos-connector-10.0-SNAPSHOT-sources.jar

... and this directory only contains pom.xml, so no such build exists. Also confirmed below:

hudson@kohsuke2 ~/server/jobs/glassfish-v3/builds $ ls | less
...
2008-06-10_23-15-11
2008-06-11_06-15-13
2008-06-11_13-06-39
2008-06-12_13-05-11
2008-06-13_01-23-24
2008-06-13_20-30-24
2008-06-14_01-55-07
...

this points to a calendar corruption. Calendar doesn't say if it's thread safe or not, and various documents suggests it's not. The Calendar.updateTime() method (which is used from the getTimeInMillis()) is certainly thread unsafe, although it's hard to say if that's causing this error or not.