Continuous integration/Jenkins job builder

shortcut: CI/JJB

Jenkins Job builder (abbreviated JJB) is a python script to maintain and simplify configuration of Jenkins jobs. Jenkins internally saves the configuration of jobs in an XML format. JJB outputs the configuration in that format and feeds them to Jenkins through its HTTP API.

Upstream documentation is available at: http://ci.openstack.org/jenkins-job-builder/.

Note: Which jobs are triggered for which events is handled by Zuul.

How it works

All Jenkins jobs are managed by JJB. To modify jobs, we edit the YAML files in the integration/config repository (submit a change, have it reviewed/merged by someone else).

To deploy the change, unlike we do for mediawiki deployment, it is not deployed from a local checkout on the server, but from your own workstation.

So make sure you have JJB installed. To update it, pull the latest changes from the repository, and then generate the XML files and push them to Jenkins with your credentials.

Install JJB

Wikimedia maintains a fork of JJB in integration/jenkins-job-builder.git. This has been made to allow us to version patches (just in case) in production before having them reviewed upstream. To install JJB clone from the WMF repository.

git clone https://gerrit.wikimedia.org/r/p/integration/jenkins-job-builder.git

Enable the software.

If your python install requires root to install into the Library, run as root (e.g. default Mac OS X)

$ sudo pip install -e .

Or, if your python install only has a writable library for your current user:

$ pip install --user -e .

The above will install the required dependencies and create a dummy site package pointing to your current working directory. Once completed, your should be able to run it using jenkins-jobs. (If you try to run ./jenkins_jobs/cmd.py directly it will not generate the XML files.)

What could go wrong

pyconfig.h

If you get an error message that contains stuff like:

 ext/_yaml.c:8:22: fatal error: pyconfig.h: No such file or directory
  #include "pyconfig.h"
                      ^
 compilation terminated.

then you need to install python development dependencies. In Ubuntu that is:

sudo apt-get install python-dev

six.moves

$ jenkins-jobs test config/ -o output/
Traceback (most recent call last):
  File "/usr/local/bin/jenkins-jobs", line 6, in <module>
    from jenkins_jobs.cmd import main
  File "/Users/krinkle/Development/wikimedia/integration/jenkins-job-builder/jenkins_jobs/cmd.py", line 17, in <module>
    from six.moves import configparser, StringIO
ImportError: No module named six.moves

This happens with the Python 2.7 that ships with Mac OS X 10.9. Installing Pythong 2.7.8 from Homebrew, and using its pip to build jenkins-job-buider fixes this.

Configure JJB

Clone the repository and symlink config/jjb to config in the working copy of your JJB clone:

$ cd /path/to/integration
$ ls
  ..
  jenkins-job-builder
  ..
$ git clone ssh://gerrit.wikimedia.org:29418/integration/config.git
$ cd jenkins-job-builder
$ ln -s ../config/jjb config

JJB Authentication

JJB requires a basic authentication file using the INI format. This file holds the location and credentials to a Jenkins installation.

JJB only needs these credentials to be able to update or delete jobs. To simply generate the XML files, you do not need any credentials (just create a dummy file).

Our main installation at https://integration.wikimedia.org/ci/ only allows users in the wmf LDAP group to modify jobs (granted to Wikimedia Foundation employees and contractors).

To get your API token, log in and open the "Configure" tab of your account (JENKINS_URL/user/USERNAME/configure; currently https://integration.wikimedia.org/ci/user/USERNAME/configure). Click the "Show API Token" button to reveal your API token.

Create /path/to/jenkins-job-builder/etc/jenkins_jobs.ini:

[job_builder]
allow_duplicates=True

[jenkins]
user=USERNAME
password=API_TOKEN
url=https://integration.wikimedia.org/ci/

If you want to test a local Jenkins installation, create an .ini file like this:

[job_builder]
allow_duplicates=True

[jenkins]
user=
password=
url=http://localhost:8080/

Verify installation

$ cd /path/to/jenkins-job-builder
$ mkdir output/  # place where XML files will be written to
$ jenkins-jobs test config/ -o output/

output/ should now contains some files (exact list depending on the current configuration):

$ ls -1 output/
mediawiki-core-install-sqlite
mediawiki-core-lint
mediawiki-core-merge
mediawiki-core-phpunit-api
mediawiki-core-phpunit-databaseless
mediawiki-core-phpunit-misc
mediawiki-core-phpunit-parser
operations-puppet-validate
...

Each file contains an XML representation of a Jenkins job.

Congratulations!

Modifying jobs

See also Install JJB (you need to have it installed locally in order to verify the files are valid)

• Modify jobs by editing the YAML files in the integration/config.git repository.
• Read the documentation to learn what features are available (JJB internally maps the YAML format to the XML format that the Jenkins API takes, properties might have slightly different names or might work differently or might not be supported).
• Use an editor with YAML syntax highlighting and ideally YAML linting as well. The file format rely on indentation for its structure which often leads to mistakes by people not used to that (similar to how Python works).
• Once done editing, run the following in /path/to/jenkins-job-builder:
  

  $ jenkins-jobs test config/ -o output/
  

  This verified everything works as expected (aside from the YAML syntax, incorrect indentation can break things, or naming collisions with Python interfaces).
• Commit your modifications to a local topic branch (keep master clean), and send to Gerrit for review.

Deploy changes

See also Install JJB (you need to have it installed and authenticated)

Check out the relevant change from Gerrit into the jenkins-job-builder-config repo. Ensure the change is rebased onto latest master to avoid accidentally undoing change to generic macros that are included in all jobs.

1. Before letting JJB parse the YAML files into XML. Ensure sure you are running the latest version of JJB by pulling master and re-running the quick installer.
2. Running the jenkins-jobs test command will validate the YAML files and generate the XML files in output/ directory.
   If that commands results in error, there is either a YAML syntax error or incorrect JJB commands inside the YAML file. Resolve this before continuing!
3. Push the updated job configuration to Jenkins.

Example

# Checkout config change

krinkle in integration/config (master)
$ git review -d 139035
 Downloading refs/changes/35/139035/3 from gerrit

# Ensure jenkins-job-builder is installed and up to date

krinkle in integration/config (review/ori_livneh/139035)
$ cd ../jenkins-job-builder

krinkle at integration/jenkins-job-builder (master)
$ ls -l config
 config@ -> ../config/jjb

krinkle in integration/jenkins-job-builder (master)
$ git pull origin master
 From ssh://gerrit.wikimedia.org:29418/integration/jenkins-job-builder
 Already up-to-date.

krinkle in integration/jenkins-job-builder (master)
$ pip install -e .
 running develop...
 Installing pip script to /usr/local/bin
 Finished processing dependencies for jenkins-job-builder

# Compile job XML files

krinkle in integration/jenkins-job-builder (master)
$ rm output/* && jenkins-jobs test config/ -o output/

krinkle in integration/jenkins-job-builder (master)
$ cat output/operations-apache-config-lint 
  ...

# Push one or more jobs configurations to Jenkins (updates or creates the given job)

krinkle in integration/jenkins-job-builder (master)
$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/ 'operations-apache-config-lint'
 INFO:root:Updating jobs in config/ (['operations-apache-config-lint'])
 INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-apache-config-lint

Synchronizing all jobs

You should not be doing this

This will easily take over an hour and might overwrite things you don't intend to. It's like doing scap instead of sync-file in Wikimedia's deployment. In addition it doesn't allow you to control the order of things, so you might update jobs with dependencies before the dependencies, which is problematic as we're pushing to a production instance that is actively used.

$ jenkins-jobs --conf etc/jenkins_jobs.ini update config/
INFO:root:Updating jobs in config/ (None)
INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet
INFO:jenkins_jobs.builder:Reconfiguring jenkins job operations-puppet-validate
.....
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-merge
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-install-sqlite
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-lint
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jshint
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-jsduck-publish
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-qunit
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-api
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-databaseless
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-misc
INFO:jenkins_jobs.builder:Reconfiguring jenkins job mediawiki-core-phpunit-parser
....

Notes

• You can enable debugging output by passing -l debug
• Jenkins job builder maintains a cache of jobs and will not resubmit a job if it considers it already up to date. You will want to delete the file ~/.jenkins_jobs_cache.yml to force the update.