Jenkins Infrastructure : issues.jenkins-ci.org

Where are things?

Stock JIRA is installed at /srv/jira under its own user account 'jira'.

Changes to stock JIRA

conf/server.xml needs to be modified to accomodate for the fact that SSL is terminated at the front SSL (more details): In particular, the HTTP connector needs to have the following added:

scheme="https" proxyName="issues.jenkins-ci.org" proxyPort="443"

To bounce the server

sudo /etc/init.d/jira stop && sleep 10 && sudo /etc/init.d/jira start

(restart sometimes doesn't restart the server. Better do start/stop separately to be safe)

Customization

When upgrading to the newer version, be sure to make these changes.

  • Stop access logs, as we use Apache to take them. This is in /conf/server.xml. Comment out a valve definition.

Incident records

Out of disk - Aug 30th, 2011

Stupid machine ran out of disk space, rtyler synced some older data from /srv/jira/home/export locally for temporary archival. (Will eventually re-upload to lettuce once a proper backup partition is in place.

Rebounce Aug 11th, 2011

Server running since July 14th. "jmap -heap" indicates complete memory fillup

JVM version is 19.1-b02

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio = 40
   MaxHeapFreeRatio = 70
   MaxHeapSize      = 536870912 (512.0MB)
   NewSize          = 1310720 (1.25MB)
   MaxNewSize       = 17592186044415 MB
   OldSize          = 5439488 (5.1875MB)
   NewRatio         = 2
   SurvivorRatio    = 8
   PermSize         = 21757952 (20.75MB)
   MaxPermSize      = 201326592 (192.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 127008768 (121.125MB)
   used     = 127008768 (121.125MB)
   free     = 0 (0.0MB)
   100.0% used
From Space:
   capacity = 24313856 (23.1875MB)
   used     = 0 (0.0MB)
   free     = 24313856 (23.1875MB)
   0.0% used
To Space:
   capacity = 25362432 (24.1875MB)
   used     = 0 (0.0MB)
   free     = 25362432 (24.1875MB)
   0.0% used
PS Old Generation
   capacity = 357957632 (341.375MB)
   used     = 357957632 (341.375MB)
   free     = 0 (0.0MB)
   100.0% used
PS Perm Generation
   capacity = 128974848 (123.0MB)
   used     = 128654088 (122.69409942626953MB)
   free     = 320760 (0.30590057373046875MB)
   99.7513003465606% used

Heap analysis indicates webwork.util.Query.queriesMap ate up all the memory — retention size 175MB.

Rebounce Nov 1st, 2011

Heap fill up, with the same webwork.util.Query.queriesMap eating all the memory. https://support.atlassian.com/browse/JSP-95456 is filed for this.