This is my notes on how I setup Jenkins and Gerrit for continuous delivery on my servers.
There seems to be a lot of bits and pieces to master to get this up and running. I have noted down the steps I have done so that someone else can repeat it for their own setup by seeing what I have done.
My objective is to setup gerrit for commit review and for each commit the jenkins master will schedule 2 jobs one each for linux and windows. After it is all done it will collect the job results and if all are successful it should mark the gerrit commit as verified.
Machines used: I have one linux server for gerrit. One server for jenkins. One server for linux master. One server for windows master. Some linux slaves. Some windows slaves. The purpose of the linux master and windows master is to compile the software which just takes only a few minutes. All other slaves are used to split the testing time. If tests are executed one after the other they take several hours. So I get the lin/win master to compile the software and the slaves will download the compiled software and run in parallel different sets of tests that do not depend on each other. So the testing time goes down to the time taken for the longest test set.
Install Postgresql.
This is used as a backend for Gerrit. I used the following script.
wget https://ftp.postgresql.org/pub/source/v9.4.4/postgresql-9.4.4.tar.gz
tar zxf postgresql-9.4.4.tar.gz
cd postgresql-9.4.4
yum install -y readline-devel
./configure
gmake world
gmake install-world
useradd pguser
mkdir -p /home/pguser/data
chown -R pguser:pguser /home/pguser
echo "to init run these following"
echo "su pguser"
echo "/usr/local/pgsql/bin/initdb -D /home/pguser/data"
echo "to start run these following commands"
echo "su pguser"
echo "cd /home/pguser"
echo "/usr/local/pgsql/bin/postgres -D /home/pguser/data >logfile 2>&1 &"
echo "exit"
Install Git
Git must be installed before gerrit.
yum install -y git
Install Gerrit.
yum install -y java
/usr/local/pgsql/bin/createuser --username=pguser -RDIElPS gerrit
/usr/local/pgsql/bin/createdb --username=pguser -E UTF-8 -O gerrit reviewdb
Give a password for the gerrit user and enter it below during gerrit setup
useradd gerrit
su gerrit
cd /home/gerrit/
mkdir reviewsite
wget https://www.gerritcodereview.com/download/gerrit-2.11.3.war
java -jar gerrit-2.11.3.war init -d /home/gerrit/reviewsite
Press enter to all questions except the following:
Database server type [h2]:postgresql
Authentication method []:http
Get username from custom HTTP header []? y
Install verified label []? y
Behind reverse proxy []? y
Proxy uses SSL (https://) []? y
Subdirectory on proxy server []: /r/
Canonical URL : https://review.sveena.com/r/
Install plugin reviewnotes ? y
Install plugin replication ? y
Install commit-message-length-validator plugin? y
To start gerrit if it is not already started:
su gerrit
cd /home/gerrit/reviewsite
bin/gerrit.sh start
If that doesn't start try this:
Start postgres sql first.
su gerrit
cd /home/gerrit/reviewsite
java -jar bin/gerrit.war daemon -d /home/gerrit/reviewsite/ &
Install Jenkins.
I used the following script. I am using a centos machine for the master, some linux slaves and some windows slaves.
I move the jenkins folder from /var/lib/jenkins to /home/jenkins because I like to backup only the home folder.
wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat-stable/jenkins.repo
rpm --import https://jenkins-ci.org/redhat/jenkins-ci.org.key
yum -y install java jenkins git
service jenkins stop
cd /usr/lib/jenkins
mv jenkins.war jenkins.bak
wget http://mirrors.jenkins-ci.org/war/latest/jenkins.war
mv /var/lib/jenkins /home/
sed -i -e 's/\/var\/lib\/jenkins/\/home\/jenkins/g' /etc/sysconfig/jenkins
service jenkins start
chkconfig jenkins on
Setup Nginx:
Nginx is used as a reverse proxy with client certificates on both the Gerrit and Jenkins machines.
Client certificates are simple, easy and more secure than mere passwords.
I haven't figured out how to setup Jenkins (with open anyone can sign up) client certificates yet, I still have to log in.
But Gerrit works fine with open client certficates.
Direct your users to get a free client certificate at https://userid.sveena.com
In the nginx.conf
server {
server_name car.sveena.com; #car stands for continuous automatic releases
rewrite ^ https://car.sveena.com$request_uri? permanent;
}
server {
listen 443 ssl;
server_name car.sveena.com;
root /home/car;
ssl_certificate /home/sveena.com.crt2;
ssl_certificate_key /home/sveena.com.key2;
ssl_verify_client on;
ssl_verify_depth 2;
location = /robots.txt {
}
location = /496.htm {
root /home;
}
client_body_buffer_size 3m;# these are required to install jenkins plugins
client_max_body_size 3m;
location / {
proxy_pass http://127.0.0.1:8080/;
proxy_redirect $scheme://$host:$server_port/ /; # this is required for reverse proxy to work correctly
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-User $ssl_client_s_dn_id;
proxy_set_header X-Forwarded-Groups "Authenticated";
}
}
server {
server_name review.sveena.com;
rewrite ^ https://review.sveena.com$request_uri? permanent;
}
server {
listen 443 ssl;
server_name review.sveena.com commit2.sveena.com; #commit2.sveena.com is replication backup of gerrit server
root /home/commit;
ssl_certificate /home/sveena.com.crt2;
ssl_certificate_key /home/sveena.com.key2;
ssl_verify_client on;
ssl_verify_depth 2;
location = /496.htm { root /home; }
location /r/ {
proxy_pass http://127.0.0.1:8081;
proxy_set_header X-Forwarded-For $remote_addr;
proxy_set_header Host $host;
proxy_set_header SM_USER $ssl_client_s_dn_id;
}
}
Secure Gerrit:
The first user to open your Gerrit site in a browser becomes the administrator.
Secure Jenkins:
Open your Jenkins site in a browser.
Click Manage Jenkins.
Click Configure Global Security.
Check Enable Security.
Check Jenkins own user database under security realm.
Check Matrix based security.
Type your name in User/group to add and click add.
Select everything for your name.
Check Read under overall for anonymous, so others can see your configuration.
Restart the browser and access Jenkins site again.
Click create an account and sign up.
Click Manage Jenkins.
Click Configure Global Security.
Uncheck allow users to sign up.
Click Save.
Setup Nodes:
You must create the nodes BEFORE you create a project otherwise there will be no option to add slaves configuration matrix in the project setup. This took me many days to figure out.
On the linux slaves run the following:
yum install -y ntp
service ntpd start
chkconfig ntpd on
yum install -y java
useradd jenkins
passwd jenkins
su jenkins
mkdir /home/jenkins/work
Create a node:
Click Jenkins home/Manage Jenkins/Manage Nodes/New node.
Give the node name, select dumb node and click ok.
Set # of executors to the number of processing cores.
Enter /home/jenkins/work for the Remote root directory.
For both windows and linux slaves:
choose Launch method as 'Launch slave agents on Unix machines via SSH'
Enter host address and click credentials add.
In the Add credentials dialog:
Give Kind as Username with password.
Scope as global, username as jenkins and the password for that user and click Add.
For labels enter the following for different machines:
Machine |
Used For |
Labels |
|---|---|---|
Linux Master |
Compiling |
LinuxMaster |
Linux Slave |
Testing |
Linux |
Windows Master |
Compiling |
WindowsMaster |
Windows Slave |
Testing |
Windows |
Labels will be used to run specific configurations of a project on specific groups of machines.
On the windows machines:
Install openssh:
Open Internet Explorer.
Goto Java.com and download and install java.
Go to cygwin.com
Download setup-x86-64.exe from the home page and save as cygwin-setup-x86-64.exe (so that you don't have to download it again. If everyone named their babies as cute baby we wouldn't be able to tell each other apart. If every setup file is called setup how will you know which one is which without running them?)
Run the setup file.
Search cygrunsrv and openssh and select them and click next to install them.
Click yes to create a start menu item for cygwin terminal.
Run cygwin64 terminal as adminstrator.
Type ssh-host-config in the terminal and press enter.
Enter yes for all questions except for the question do you want to use a different name.
Enter ntsec for the question 'Enter the value of CYGWIN for the daemon'.
In the cygwin terminal type:
ln -s /cygdrive/c/ProgramData/Oracle/Java/javapath/java /bin/java
mkdir -p /home/jenkins/work
chown -R jenkins /home/jenkins
Open windows firewall from the control panel. (for windows 2008 server).
Click advanced settings.
Click Inbound rules.
Click New Rule.
Select Port and click next.
For specific local ports Enter 22 or another number of your choice.
Click Next.
Select allow the connection and click next.
Check Domain, Private and Public and click next.
For Name Give SSH and click finish.
Click Start/Administrative Tools/Services.
Right click CYGWIN sshd and click start.
Click start/control panel/User Accounts/Add or remove user accounts
Click create a new account.
Give Jenkins as name of account, select standard user and click create account.
Double click Jenkins user and click create a password.
Enter a password twice and click create password.
Click on the time on the task bar.
Click change date and time settings.
Click Internet time.
Click change settings.
Check Synchronize wih an Internet time server
Click update now.
Click OK.
Install Mingw-w64:
After many weeks of googling and hair pulling and banging my head on the keyboard I have at last found a satisfactory set of c++ compilers and tool chains for automated install and builds on the windows machines.
I needed to compile to both 32 bit and 64 bit native targets on the same machine but I wanted windows native threads instead of posix threads. I found mingw-w64, which uses gcc.
Go to http://sourceforge.net/projects/mingw-w64/files/
Select Toolchains targetting Win64 and Toolchains targetting Win32, choose Personal builds. Choose mingw-builds. Choose latest version (currently 5.2.0). Choose threads-win32. Choose one of the authors.
Choose <target><version>-release-win32<author>-*.7z and download it. Unzip them to c:\mingw64 and c:\mingw32 respectively. Write a simple program and compile them to exe files to make sure they both work. We will later use these paths in our scripts.
Import previous git project into Gerrit:
Users will pull from sveena-master but will commit to sveena-commit. This setup is to recheck that all tests pass again in the master after code is approved and merged into the mainline.
su gerrit
cd /home/gerrit/reviewsite/git
git clone git://open.source.sveena.com/git/sveena sveena-master.git --bare
git clone sveena-master.git/ sveena-commit.git --bare
cd ..
bin/gerrit.sh restart
Setup Gerrit:
Create groups.
Sign in to your gerrit site as administrator.
Click People/Create New Group.
Give group name as Committer.
Click Create Group.
Click General.
Check Make group visible to all users.
Click Save Group Options.
Similarly create another group called Reviewer.
Create a key for jenkins user on Gerrit
At a linux prompt type:
ssh-keygen -t rsa -b 4096
For 'Enter file in which to save the key (/root/.ssh/id_rsa):' type /root/.ssh/jg_rsa
cat /root/.ssh/jg_rsa.pub | ssh -p 29418 shankar@review.sveena.com gerrit create-account --group "'Non-Interactive Users'" --ssh-key - jgwatcher
Upload the jg_rsa file to the Jenkins server as /home/jenkins/secrets/.jg_rsa
In the Gerrit Web UI
Click Projects/List/All Projects.
Click Access.
Click Edit.
Under Reference:Refs/*
Under Read
Click Add Group
Type Non Interactive Users and click add.
Make sure ALLOW is selected.
Under Reference:Refs/heads/*
Under Label Code-Review
Click Add Group
Type Non Interactive Users and click add.
Select -1 to +1
Under Reference:Refs/heads/*
Click Add Permission and select Label Verified.
Type Non interactive users and click add.
Select -1 to +1
Make sure under Global Capabilities
Stream Events is allowed for non interactive users.
Goto the bottom of the screen and click save changes.
Setup Gerrit Trigger:
Sign in to your Jenkins site.
Click Manage Jenkins.
Click Manage Plugins.
In the updates tab.
Check all the updates and click Download and install after restart.
Click Available Tab.
Type git in the filter field.
Select GIT Plugin
Click Install without restart.
Type gerrit in Filter field.
Select Gerrit Trigger and click Install without restart.
From Home page click Manage Jenkins.
Click Gerrit Trigger.
Click Add New Server.
Give name as sveena and click OK.
Give hostname as review.sveena.com.
For Frontend URL type https://review.sveena.com/r/
For username give jgwatcher
For SSH key file type /home/jenkins/secrets/.jg_rsa
Give the password for the file.
Click Test Connection. If the result is Bad SSH file or password. Re-encode the file using des3 if the encryption is aes256 it doesn't work.
Click Save.
If the Server status is Red, Remove the password on the jg_rsa file.
Edit /home/jenkins/gerrit-trigger.xml
Remove the password between gerritAuthFileKeyPassword Tags.
service jenkins restart.
That fixed the server contact.
Setup Project:
Sign in to your Jenkins site.
Click New Item.
Check Multi Configuration Project.
Give Item name as sveena-commit.
Click OK.
In the project configuration screen.
Check Gerrit Event.
For Choose a server field, choose a server duh! If no server is listed goto manage jenkins/gerrit trigger and add a server.
Click Add against Trigger On and select Patchset created.
Click Add and select Draft published.
Under Gerrit Projects click on the box next to plain and select sveena-commit. If nothing is listed then the server connection is not working, you have to fix that first.
Under Branches pattern click on plain and select path. In the text box next to it type **.
Click Add project and select sveena-master.
Under Branches pattern click on plain and select path. In the text box next to it type **.
Under Configuration Matrix click Add axis.
Click Slaves.(If you do not find this option then it means you have not setup any nodes. Goto manage jenkins/manage nodes and create the slaves first)
Open Labels branch and check Linux, LinuxMaster, Windows and WindowsMaster labels.
Click Add Axis and select User-defined Axis.
Type name as TestSet
Give Values as Compile TestSet0 TestSet1 TestSet2 TestSet3
Check Combination Filter
For Filter Type ((label=="WindowsMaster" || label=="LinuxMaster") && TestSet=="Compile") || !(label=="WindowsMaster" || label=="LinuxMaster" || TestSet=="Compile")
Important: There seems to be bug in jenkins where if you screw up the combination filter and a combination gets switched on you cannot later on switch it off even if you change the filter formula. You will have to delete the project and create a new one with the correct formula. I spent many days to find this.
Check Execute touchstone builds first.
For filter type label=="LinuxMaster" || label=="WindowsMaster"
Click Add build step
Select Execute shell
For command type printenv
Click Save.
Click BuildNow. This should run all the configurations and show success. It merely prints out the environment variables of all the builds. We will write the build scripts later.
Setup Gerrit Events log:
Setting up Gerrit Events log will make you love the Jenkins Plugins ecosystem, where you go to Manage Plugins check plugin click install. Contrast this with the number of hoops you need to jump before you can get gerrit events log.
First I searched how to install gerrit event log on the gerrit site.
I was directed to https://gerrit.googlesource.com/plugins/events-log
where I was instructed to
git clone https://gerrit.googlesource.com/plugins/events-log
OK now what? This doesn't give me a cookie jar. I have to compile it. Oh well. More googling.
To compile this I need to do:
git clone https://gerrit.googlesource.com/gerrit
cd gerrit
git checkout v2.11.2
git submodule init
git submodule update
git clone https://gerrit.googlesource.com/plugins/events-log plugins/events-log
buck build plugins/events-log
Oh, I need buck. What buck? What is buck? The buck doesn't stop here. More googling.
Turns out buck is available here: https://buckbuild.com/setup/install.html
Oh that is not a binary either.
I have to compile it. I am not on OS X so I need the manual build. Let me see what I need for that.
I need:
Oracle JDK 7
Apache Ant 1.8 (or newer)
Python 2.6 or 2.7
Git
This is getting out of hand. More googling for how to get those. Hope I touch bottom at least now.
Git: I have. Python: Check.
yum -y install java-1.8.0-openjdk java-1.8.0-openjdk-devel
Apache Ant? I don't know any ants. More googling.
OK to get ant I did:
wget http://www.eu.apache.org/dist//ant/binaries/apache-ant-1.9.6-bin.tar.gz
tar zxf apache-ant-1.9.6-bin.tar.gz
cd apache-ant-1.9.6
Whew at least this is a binary. Why the deuce doesn't everyone distribute binaries in the first place? Especially as these builds can be automated by using continuous delivery.
After lots of build failures, I decided to live without events log for now.
If the build server fails I will just have to schedule builds manually for missed patches.
If you found my notes useful, please leave a comment. That will prevent random bozos from deleting something that is useful and replacing it with crap.