Magento2 Continuous Integration and Continuous Deployment using Jenkins

suranjan horrow

20 June 2017

In Continuous Integration and Continuous Deployment environment, there are many steps involved while deploying codes to the production server. Some of these steps are as follows-:

Developers push their latest codes to the repository
Code is fetched from the repository and built is made
The built code is deployed to the staging server
In staging server, some test cases are run
After passing the test cases, code is deployed to the production server

Jenkins is a great tool which can be used to setup continuous integration and continuous deployment environment for Magento2.

In this tutorial, I will show you how to setup a continuous integration and deployment environment for magento2 using magedeploy2. If you want to know more about the magedeployer2 visit this link https://github.com/mwr/magedeploy2-base.git

I am using 4 servers in this setup

Git server
Jenkins server
Staging server
Production server

Below is the architecture of my continuous integration and deployment environment

When code is pushed to the git repository job trigger is generated at Jenkins through web hooks.
At first step Jenkins will setup build environment for magento2.
In second step it will fetch latest code from the git repository and build the code.
In third step the built code is deployed to the staging server.
In last step the built code is deployed to the production server.

Setting up your production server

Install apache and php dependencies

sudo add-apt-repository ppa:ondrej/php

sudo apt-get update

sudo apt-get install apache2 php7.0 php7.0-{curl,gd,mbstring,mcrypt,json,dom,intl,xsl,readline,bcmath,zip,soap}

sudo apt-get install git zip realpath

In production server, user magento is running the apache service. So I need to change the user www-data to magento in the /etc/apache2/envvars file

export APACHE_RUN_USER=magento

export APACHE_RUN_GROUP=magento

Virtual Host configuration

Create a new file /etc/apache2/sites-enabled/magento.conf and add the following codes

ServerName mysite.com

DocumentRoot /home/magento/www/current

CustomLog /var/log/apache2/mysite_access.log combined

ErrorLog /var/log/apache2/mysite_error.log

Options FollowSymLinks MultiViews

Require all granted

AllowOverride all

</Directory>

</VirtualHost>

Download n98-magerun2

wget https://files.magerun.net/n98-magerun2.phar

chmod +x n98-magerun2.phar

sudo cp n98-magerun2.phar /usr/local/bin/

Install composer

php -r "copy('https://getcomposer.org/installer', 'composer-setup.php');"

php -r "if (hash_file('SHA384', 'composer-setup.php') === '669656bab3166a7aff8a7506b8cb2d1c292f042046c5a994c43155c0be6190fa0355160742ab2e1c88d40d5be660b410') { echo 'Installer verified'; } else { echo 'Installer corrupt'; unlink('composer-setup.php'); } echo PHP_EOL;"

php composer-setup.php

chmod +x composer.phar

mv composer.phar /usr/local/bin/composer

Restart apache service

phpenmod mcrypt

phpenmod mbstring

a2enmod rewrite

a2enmod php7.0

service apache2 restart

Install mysql server

1	sudo apt-get install mysql-server-5.6

create database and user for magento

mysql -u root -p

create database production;

create user 'dbuserprod'@'%' ideintified by 'qh3gf4ce';

grant all privileges on production.* to 'dbuserprod'@'%';

Setting up your staging server

Follow the same procedure as in the production server. Just make some changes in the virtual host configuration and database setup. In staging server user test is running apache service. Open /etc/apache2/envvars file and change user www-data to test

export APACHE_RUN_USER=test

export APACHE_RUN_GROUP=test

Virtual Host configuration

create a file /etc/apache2/sites-enabled/magento.conf and add the following content

ServerName staging.mysite.com

DocumentRoot /home/test/www/current

CustomLog /var/log/apache2/mysite_access.log combined

ErrorLog /var/log/apache2/mysite_error.log

Options FollowSymLinks MultiViews

Require all granted

AllowOverride all

</Directory>

</VirtualHost>

create database and user for magento

mysql -u root -p

create database staging;

create user 'dbuserstage'@'%' ideintified by 'msdjo335s';

grant all privileges on staging.* to 'dbuserstage'@'%';

Setting up your Jenkins server

create database and user for magento

mysql -u root -p

create database build;

create user 'dbuserbuild'@'%' ideintified by 'p0jhser847';

grant all privileges on build.* to 'dbuserbuild'@'%';

Download and setup composer as given in the production server setup.

create and login as non-root user

1 2	adduser jenkins su - jenkins

Download magedeploy2-base

1	composer create-project mwltr/magedeploy2-base magento2

create .config/composer/auth.json file and add your magento2 credential for repo.magento.com

{

"http-basic": {

"repo.magento.com": {

"username": "<Enter your key here>",

"password": "<Enter your key here>"

}

create your default magedeploy2.php, staging.php and production.php

mkdir site

touch site/magedeploy2.php

touch site/staging.php

touch site/production.php

touch site/config

cp -rv magento2/vendor/n98/n98-deployer/src/Deployer/Task site/

change the content of the files according to your need. Every time you run the Jenkins job the defaults php files inside site folder will be used for the build and deployment tasks.

magedeploy2.php

<?php

return [

'env' => [

'git_bin' => getenv('GIT_BIN') ?: '/usr/bin/git',

'php_bin' => getenv('PHP_BIN') ?: '/usr/bin/php',

'tar_bin' => getenv('TAR_BIN') ?: '/bin/tar',

'mysql_bin' => getenv('MYSQL_BIN') ?: '/usr/bin/mysql',

'composer_bin' => getenv('COMPOSER_BIN') ?: '/usr/local/bin/composer',

'deployer_bin' => getenv('DEPLOYER_BIN') ?: '/home/jenkins/magento2/vendor/bin/dep',

'deploy' => [

'git_url' => 'https://username:password@mygitrepo.com/repo1/magedeploy.git',

'git_dir' => 'shop',

'app_dir' => 'shop',

'artifacts_dir' => 'artifacts',

'themes' => [

['code' => 'Magento/backend','languages' => ['en_US','de_DE']],

['code' => 'Magento/luma','languages' => ['de_DE']],

['code' => 'Magento/luma','languages' => ['en_US']],

'artifacts' => [

'shop.tar.gz' => ['dir' => '.','options' => ['--exclude-vcs','--checkpoint=5000']],

'clean_dirs' => [

'var/cache',

'var/di',

'var/generation',

'build' => [

'db' => [

'db-host' => getenv('DB_HOST') ?: '127.0.0.1',

'db-name' => getenv('DB_NAME') ?: 'build',

'db-user' => getenv('DB_USER') ?: 'dbuserbuild',

'db-password' => getenv('DB_PASSWORD') ?: 'p0jser847',

'admin-email' => 'suranjan.horrow869@webkul.com',

'admin-firstname' => 'Suranjan',

'admin-lastname' => 'Horrow',

'admin-password' => 'dswe234532r',

'admin-user' => 'admin',

'backend-frontname' => 'admin',

'base-url' => 'http://mysite.com',

'base-url-secure' => 'https://mysite.com',

'currency' => 'EUR',

'language' => 'en_US',

'session-save' => 'files',

'timezone' => 'Asia/Kolkata',

'use-rewrites' => '1',

'use-secure' => '0',

'use-secure-admin' => '0',

];

I am using https://username:password@mygitrepo.com/repo1/magedeploy.git in git url because I do not want prompt for username and password while cloning from git. We are providing the username and password in the url. So it will be easy for the automation process.

Here, https://mygitrepo.com/repo1/magedeploy.git is your actual git url for your repo.

staging.php

<?php

/**

* @see LICENSE

namespace Deployer;

use N98\Deployer\RoleManager;

$deployPath = '/home/test/www';

$staging = host('192.168.1.67'); // IP of the staging server

$staging->set('bin/n98_magerun2', 'n98-magerun2.phar');

$staging->user('test'); // user of the staging server

$staging->configFile('.ssh/config');

$staging->set('deploy_path', $deployPath);

$staging->stage('staging');

$staging->set('config_store_env', 'staging');

$staging->roles('web','db','staging');

production.php

<?php

/**

* @see LICENSE

namespace Deployer;

use N98\Deployer\RoleManager;

$deployPath = '/home/magento/www';

$sshConfigFile = '.ssh/config';

$production = host('192.168.1.113'); // IP of the production server

$production->set('bin/n98_magerun2', 'n98-magerun2.phar');

$production->user('magento'); // user of the production server

$production->configFile($sshConfigFile);

$production->set('deploy_path', $deployPath);

$production->stage('production');

$production->set('config_store_env', 'production');

$production->roles('web','db','production');

config

StrictHostKeyChecking no

ControlMaster auto

ControlPath ~/tmp/ssh-mux-%h-%p-%r

The files inside /home/testuser/site/Task are

BuildTasks.php
CleanupTasks.php
DeployTasks.php
MagentoTasks.php
SystemTasks.php
TaskAbstract.php

Some of the tasks required sudo access. So it will be difficult for us to automate the process because it will ask for password when sudo access is needed. We will remove sudo related tasks in these files.

In BuildTasks.php comment these lines

#\Deployer\run("sudo chown -RH $owner $dir");

#\Deployer\run("sudo chmod -R $mode $dir");

In CleanupTasks.php remove sudo

1 2	\Deployer\run("rm -rf {{deploy_path}}/releases/$release");

In SystemTasks.php comment these lines

#\Deployer\run("sudo service $service restart");

If your user in staging and production server has sudo access you do not need to edit these files.

Create password less authentication from Jenkins server to staging and production server

Go to home directory of user jenkins

1 2	ssh-copy-id test@192.168.1.67 # staging server ssh-copy-id magento@192.168.1.113 # production server

Now, you can execute command on remote server without password

Create scripts for running deploy tasks

I have created 4 scripts for deploying Magento

setup.sh
build.sh
stage.sh
prod.sh

setup.sh

#!/bin/bash

set -ex

if [ -d /home/jenkins/magento2 ]

then

rm -rf magento2

composer create-project mwltr/magedeploy2-base magento2

build.sh

#!/bin/bash

set -ex

cp site/magedeploy2.php magento2/

cd /home/jenkins/magento2

./vendor/bin/robo validate

./vendor/bin/robo deploy:magento-setup master

./vendor/bin/robo deploy:artifacts-generate

stage.sh

#!/bin/bash

set -ex

cd /home/jenkins/magento2

cp ../site/staging.php config/

cp ../site/config .ssh/

cp -rv ../site/Task vendor/n98/n98-deployer/src/Deployer/

./vendor/bin/dep deploy:prepare staging

./vendor/bin/dep server:setup staging

ssh test@192.168.1.67 php /home/test/www/current/bin/magento setup:install --db-host="127.0.0.1" --db-name="staging" --db-user="dbuserstage" --admin-email="suranjan.horrow869@webkul.com" --db-password="msdjo335s" --admin-firstname="Suranjan" --admin-lastname="Horrow" --admin-password="webkul123" --admin-user="admin" --backend-frontname="admin" --base-url="http://staging.mysite.com/" --base-url-secure="https://staging.mysite.com/" --language="en_US" --use-rewrites=1

cd /home/jenkins/magento2

./vendor/bin/dep deploy staging

prod.sh

#!/bin/bash

set -ex

cd /home/jenkins/magento2

cp ../site/production.php config/

cp ../site/config .ssh/

cp -rv ../site/Task vendor/n98/n98-deployer/src/Deployer/

./vendor/bin/dep deploy:prepare production

./vendor/bin/dep server:setup production

ssh magento@192.168.1.113 php /home/magento/www/current/bin/magento setup:install --db-host="192.168.1.113" --db-name="production" --db-user="dbuserprod" --admin-email="suranjan.horrow869@webkul.com" --db-password="qh3gf4ce" --admin-firstname="Suranjan" --admin-lastname="Horrow" --admin-password="webkul123" --admin-user="admin" --backend-frontname="admin" --base-url="http://mysite.com/" --base-url-secure="https://mysite.com/" --language="en_US" --use-rewrites=1

cd /home/jenkins/magento2

./vendor/bin/dep deploy production

Now we need to execute these shell scripts through Jenkins pipeline.

Create pipeline jobs in Jenkins

Go to New Item -> Pipeline

Create a name for your pipeline and your job will be created.

Then got to Build Triggers and select Trigger builds remotely. Give your Authentication Token. The Authentication Token will be used in git webhook.

In Pipeline script copy the following codes.

node('master') {

stage('Set up')

sh 'su - jenkins -c "bash setup.sh"'

stage('Build')

sh 'su - jenkins -c "bash build.sh"'

stage('Deploy staging')

sh 'su - jenkins -c "bash stage.sh"'

stage('Deploy production')

sh 'su - jenkins -c "bash prod.sh"'

}

In this pipeline script we are defining 4 stages. If one of the stage is failed, the Jenkins job will be stopped.

Now, save your Jenkins pipeline job.

Setting up your git repository

Create a repository for magento2. We will upload our files in this repository.

Now go to Settings of this repo and select Web hooks. Write the url you want to trigger when someone pushes the code into git.

When someone pushes the code http://example.com/hook.php will be called. In hook.php I am executing a curl command which will trigger the Jenkins job we have previously defined.

hook.php

<?php

shell_exec('curl http://user:pass@192.168.1.94:8080/jenkins/job/deploy_magento/build?token=sgfed35e3d2');

In this code user and pass is the username and password of Jenkins admin user. The job name is deploy_magento and sgfed35e3d2 is the Authentication Token we defined at the creation of the Jenkins Job.

Now, we have completed setting up our server. For installing magento2 upload magento2 setup files in the git repository. This will trigger the job defined in Jenkins. Job will start automatically and start executing the scripts. When the job is completed you can visit your staging server and production server in the browser. If you want to install modules in magento2, just push your module codes to the git repository and it will be automatically tested and deployed to the production server.

If your job failed for some reason, you can check the build logs. This is all about setting up Continuous Integration and Continuous Deployment environment for magento2. If you have any query regarding the setup procedure, you can ask me in the comment.

IN CASE OF ANY QUERY, CONTACT US