Backup and Restore Jenkins Data — 1

How to backup and restore your Jenkins data — part 1

What is Jenkins?

According to the Jenkins documentation, Jenkins is a self-contained, open-source automation server that can be used to automate all sorts of tasks related to building, testing, and delivering or deploying software.

Jenkins can be installed through native system packages, Docker, or even run standalone by any machine with Java Runtime Environment (JRE) installed.

The task in hand:

We are assuming that your Jenkins container is running on an instance with the Ubuntu operating system. And the Jenkins is running on a docker container.

How can you make sure to backup all data accumulated by the Jenkins container and restore Jenkins without losing any data on another instance in case the Jenkins server fails?

Your solution must resolve the issue in a cloud-agnostic manner therefore you will only use free and open-source technologies and tools.

Solution 1:

Backup to hard disc. You can backup data securely on the server from container volume to the server’s hard disk. This will give you the chance of recovering all data as of the time of backup however if the server fails better solution would be to use another server to backup data and restore Jenkins there.

We assumed that the Jenkins is running on a Docker container therefore Jenkins data is resting in a volume attached to the container. Using Duplicati, we can regularly copy the content of the container volume to a folder in another instance. And we can restore data from the folder to a new Jenkins container at the destination. We will go through the solution details below.

Solution 2:

Backup to hard disc with Duplicati using Jenkins job. This part will be explored in the next articles.

Solution 3:

Backup to s3 bucket. This part will be explored in detail in one of the following articles.

Pre-requisites:

Docker, Docker Compose, Duplicati, Jenkins.

Solution summary

This task requires regular backups, and the backup must be secure therefore the tools we use must give us backup automation with encryption. We have decided to use Duplicati.

Duplicati is a backup client that securely stores encrypted, incremental, compressed remote backups of local files on cloud storage services and remote file servers. (https://en.wikipedia.org/wiki/Duplicati)

For easier understanding we have split the solution into 4 stages as follows:

Stage 1

Setting up a new environment so that we can backup Jenkins data and restore Jenkins from the backup folder. We will call this new Jenkins server.

Spin up an instance on a public cloud provider with Ubuntu 20.04 operating system.

Install docker and docker-compose:

Uninstall old versions:

$ sudo apt-get remove docker docker-engine docker.io containerd runc

Set up the repository:

$ sudo apt-get update -y
$ sudo apt-get upgrade -y
$ sudo apt-get install -y \
apt-transport-https \
ca-certificates \
curl \
wget \
gnupg \
lsb-release \
software-properties-common

Add Docker’s official GPG key:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg — dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
$ echo \
“deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable” | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

Install docker-ce:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
$ sudo add-apt-repository “deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable”

Install Docker Engine:

$ sudo apt-get update -y
$ sudo apt-get install -y docker-ce docker-ce-cli containerd.io
$ sudo systemctl start docker
$ sudo systemctl enable docker
$ sudo usermod -a -G docker ubuntu

Install docker-compose:

$ sudo curl -L “https://github.com/docker/compose/releases/download/1.27.4/docker-compose-$(uname -s)-$(uname -m)” -o /usr/local/bin/docker-compose
$ sudo chmod +x /usr/local/bin/docker-compose

Install Duplicati on a container:

$ sudo docker pull linuxserver/duplicati
$ sudo docker create — name=duplicati -p 8200:8200 linuxserver/duplicati
$ sudo docker start duplicati

This duplicati application runs inside of a container so we need to configure it. Let’s go ahead and create the new container, and mount our directory to be backed up on the /source directory inside the container:

$ docker create — name=duplicati -v ~/duplicati_home:/source -p 8200:8200

If you are using docker or docker-compose file to run duplicati please make sure that the folder where Jenkins backup data rest (stored by duplicati from old Jenkins server) mount to the docker container’s volume. Your volumes section of docker-compose YAML file should look like:

volumes:
- /usr/lib/duplicati/config:/config # Contains all relevant configuration files.
- /usr/lib/duplicati/backups:/backups # Path to store local backups.
- /usr/lib/duplicati/source:/source # Path to store local backups.
- /jenkins_backup_duplicati:/jenkins_backup_duplicati # using the same name for connection between backup folder and volume inside the duplicati container
- /var/lib/docker/volumes/jenkins_jenkins-data/_data:/restore_jenkins_here # left side for Jenkins data which stored in volume for Jenkins container. the right side is inside duplicati container to be able to easily restore data. IMPORTANT NOTE: remember that docker-compose uses folder name as the project name. eg: previously file was in the infrastructure folder therefore the path was …/infrastructure_jenkins-data/… however as the folder name in this location is Jenkins, therefore, the path must be …/jenkins_jenkins-data/…

Install Jenkins:

Please install Jenkins using the official image from docker hub (https://hub.docker.com/r/jenkins/jenkins)

$ sudo docker pull jenkins/jenkins
$ docker run -d — name=jenkins jenkins/jenkins

Once the first backup has been sent from the existing Jenkins server you can open duplicati user interface:

http://<new jenkins server ip address>:8200

Stage 2

Setting up Duplicati on the existing Jenkins machine.

Install docker, docker-compose, and Duplicati container as per the instructions on stage 1 above. The only difference here is to make sure that bind mounts are correctly identified on the docker-compose file. Your volumes section of docker-compose YAML file should look like:

- /usr/lib/duplicati/config:/config # Contains all relevant configuration files.
- /usr/lib/duplicati/backups:/backups # Path to store local backups.
- /usr/lib/duplicati/source:/source # Path to local source.
- /var/lib/docker/volumes/infrastructure_jenkins-data/_data/:/jenkins_backup # source (left side) must be your jenkins volume folder. Right hand side after column (:) is the point where you can see the jenkins’s existing data from duplicati interface.

Stage 3

Make Duplicati configuration from duplicati user interface.

Open duplicati user interface on web browser:

http://<jenkins server ip address>:8200

Once the user interface open, please create your first Jenkins data backup following below instructions;

add backup
Configure a new backup
fill General backup settings
name
choose encryption
enter passphrase
Config Backup destination
Storage Type : SFTP (SSH)
Server and port : <new jenkis server ip address> : <22>
Path on server : /jenkins_backup/
enter username and password for user
Test connection
The folder /jenkins_backup/ does not exist. Create it now? (click yes when prompted)
Expected result: Connection worked!
config source data
choose “jenkins_backup” folder (please refer to notes if you want to backup some specific sub-folders or files)
config Schedule cycle
choose Backup retention
Save

Once you clicked the save button the Duplicati task was completed. You can either wait for the next day to see the backup or you can run the task now so that you can see the backed-up data on the new Jenkins server.

Stage 4

Jenkins on the new Jenkins server now should be able to run using data from the Jenkins backup file.

Stop your jenkins container:

$ docker ps
$ sudo docker stop container <container_id>

Restore your backup on Duplicati user interface

select menu > restore
Where do you want to restore from?
Direct restore from backup files …
Backup location
select “jenkins_backup_duplicati”
Test connection
if successful click next
Encryption
enter your passphrase
Select files to restore
Restore options
Where do you want to restore the files to? Select pick location
Select “/restore_jenkins_here”
select overwrite
tick Restore read/write permissions

Start Jenkins container

$ sudo docker start jenkins

Reload your configuration on Jenkins

Go to http://<new Jenkins server public IP address>:8080/ > Manage Jenkins > Reload Configuration from Disk

and click OK to reload.

Now you should be able to see an identical Jenkins screen on the new Jenkins environment. However please bear in mind that the data used for the new Jenkins is not fully up to date or synchronized and reflects Jenkins data accumulated on the existing Jenkins machine as of the time duplicati start taking the latest backup from the existing Jenkins server. Therefore, it is important that you must make sure to set duplicati to take smart backup ideally once the day’s Jenkins activity is completed.

We will further explore Jenkins backup and restore using a combination of duplicati and Jenkins job and share the details with another article soon. See you at the second article.

Best regards

Authors:

M. Altun

F. Sari

S. Erdem

09Nov2021, London

DevOps Engineer @ Finspire Technology