Where are things?
Stock Confluence is installed at /srv/wiki under its own user account 'wiki'.
To bounce the server
- Apache runs auto export plugin to divert much of the load to statically generated HTML without bothering Confluence. Something we can look into. Example of static page and its corresponding Confluence page.
- Performance tuning guide from Atlassian
There's a ticket filed in Atlassian support for the recent outages. So let's keep records of when/how Confluence failed. (newer ones first)
See How to do a post-mortem analysis for what data to collect before relaunching a new instance
March 16th 18:49 PT
Upgraded JDK to 6u24 since investigation in CSP-58700 seems to indicate that there have been 7 JDK crashes while JIT-ing the exact same method. This KB article appears spot on.
March 16th afternoon PT
With the help of OSUSL, the VM now has 2.5GB heap. I've modified the VM parameters to "-Xmx768m -XX:MaxPermSize=256m". Previously it was 512m and 192m respectively.
March 16th (1st time)
JVM crash on out of memory error (full report):
It appears that the JVM crashed when it was trying to reallocate the oldgen from 300MB-ish to 500MB-ish because the kernel didn't have enough swap space to underwrite the new allocation.
March 16th (2nd time)
Unresponsive JVM. "jmap -heap" reported that all the heap spaces have fully filled up. Presumably the JVM went into the excessive GC mode although I couldn't confirm it.
March 16th (3rd time)
Andrew restarted it. No details.