Integrate Magento 2 with Varnish-Cache and Redis-Server with SSL Termination using Docker-Compose

Updated 17 July 2018

If you have been following our Magento 2 Docker architectures series, you are almost there to configure high level architecture for your e-commerce store. Although in all my previous blogs in this series, I have been using Magento 2 only but these docker architecture can be used for other frameworks as well. So before exploring today’s architecture, let us review our journey so far.

In our previous blogs, we have covered,

 

 

All these above mentioned architecture can be used as per the server set up requirements. But there few things missing from these architectures that might not be needed in beginning but they can be essential need as our e-commerce store grows.

We all have seen that our previous architectures lack SSL and do not have any mechanism to take database backup on the scheduled basis. Although we had secured our application code keeping it on our host but database is as important as server code.

So in this blog, in addition to previous setups we will also configure SSL for our Magento 2 store and we will create a bash script that will make regular database backups on our docker hosts. Also as we had discussed in our last blog that we will achieve an extra layer of optimisation, so in this blog we will also integrate Redis-server as well.

 

Introduction to Redis

 

As Varnish-Cache works as a HTTP accelerator, Redis alongside its various features,  can be used as database cache. Quoting from docs itself,

 

Redis is an open source, BSD licensed, advanced key-value store that can optionally be used in Magento for back end and session storage. It can be used to cache the database resulting in exploitation of less database resources, and provides a tunable persistent cache. It is a good alternative to memcacheD.

 

When first time page is loaded, a database is queried on the server. Redis caches the query. Next time other user loads the page the results are provided from the redis without quering the actual database. It implements a persistent object cache (no expiration). An object cache works by caching the SQL queries in memory which are needed to load the web page. When the data of main database server is updatedc,then corresponding key in the redis is invalidated. So it provides the updated data instead of caching the data. If a query is not available in redis, the database provides the result and it adds the result to its cache.

Magento supports many backend caches like MemcacheD and APC that are commonly used. However, Redis has become a popular and powerful cache system for Magento and other web applications.

 

Need For Nginx as SSL Termination

 

You all, who are not much familiar with nginx-varnish relationship, might have thought that why we are going to use nginx for SSL instead of configuring SSL with apache2 server itself. Take a note that, Varnish being a reverse proxy caching server sits in front of apache2 server. Also as Varnish is a HTTP accelerator it cannot deal with HTTPS traffic. So we must deploy a way to direct both HTTP and HTTPS traffic to Varnish cache server which in turn, if needed, forward it apache2 server.

Nginx comes in action here. Nginx serves as a reverse proxy server that receives traffic on port 80 and 443 and then proxy pass it to listening port of Varnish Cache server.

So continuing our legacy of multi-container Docker architecture, we will be using separate containers for apache2 server, mysql-server, varnish-cache server, redis-server and nginx-server (for ssl termination) for its integration with Magento 2 on Ubuntu 16.04.

The main directory holding all the files/directories will be considered as the project name. Custom project name can also be in set docker-compose.yml file. Here we are creating separate directories for apache2, mysql-server, redis-server, varnish cache server and nginx server setup that hold their Dockerfile(s) and their associated volumes.

Take a note that: following the same approach, Magento 2 files will be placed on our host. As it is a good practice to keep application files on host so that it will not be lost if containers or images get accidentally removed. These magento files will be mapped from host to running docker container.

We are also using supervisor to control all five of our servers in their respective containers. Apart from controlling these servers, supervisor is also running various commands and scripts that will be mentioned later in this blog.

To begin with create a directory on your Ubuntu 16.04 server for this project. Our directory architecture will be something like:

 

  • docker-compose.yml
  • web_server

Dockerfile

supervisord.conf

  •  database_server

Dockerfile

mysql.sh

supervisord.conf

  • cache_server

Dockerfile

default.vcl

supervisord.conf

varnish

  • redis_server

Dockerfile

supervisord.conf

  • ssl_server

Dockerfile

default

nginx.conf

supervisord.conf

  • magento2

Unarchived Magento 2 files and directories.

 

The docker-compose.yml file is shown below:

 


 

Here in YAML file above, we are defining five services: ssl_server, redis_server, cache_server, web_server and database_server.

 

  • The ssl_server is associated with Nginx server configuration. Container name is nginx, linked to all the other services and port 80 and 443 is allocated to it. There are three files that are being mapped from host to docker container and “context” points to it Dockerfile installing nginx version 1.10.
  • The redis_server is associated with Redis-server configuration. Container name is redis, linked to web_server and port 6379 is allocated to it. There is one file being mapped from host to docker and “context” points to its Dockerfile installing redis-server.
  • The cache_server is associated with our Varnish cache server configuration. Container name defined is varnish, linked to web_server and port 6081 & 6082 is allocated to it. There are 3 files that are being mapped from host to docker container and “context” points to its Dockerfile installing Varnish version 4.1.
  • The web_server is associated with our apache server configuration. Container name defined for this service is apache2, linked to database_server and port 8080 is allocated to it. There are four volumes or files are being mapped from host to docker container and “context” under “build” points to location of its Dockerfile.
  • Also, the service database_server is associated with mysql-server. Container name is defined as mysql and port 3306 is allocated to it. Mysql root password will be passed as build argument and their are two volumes/files mapped from host to docker container. Same as web_server, “context” points to location of its Dockerfile installing mysql-server-5.7.

 

Lets take a look in our web_server directory. It contains 2 files. Dockerfile is shown below:

 


 

And at last we have supervisord.conf file that supervisor use to run apache2 server and ownership commands. Its contents are shown below:

 


 

Moving on to our database_server directory, it contains 3 files. Dockerfile is shown below:

 


 

As we have discussed many times we cannot perform any operation from Dockerfile that requires a particular service to be running. As with case of database, we cannot create database from Dockerfile as mysql service is not running. For database and its user creation, we will create a bash script that will run whenever a container will launch, hence creating mentioned database and its user

We are using “mysql.sh” as the bash script as mentioned in Dockerfile. Bash script “msyql.sh” resides on our host parallel to Dockerfile.

Contents of mysql.sh is shown below:

 


 

Apart from mysql.sh and Dockerfile, we are also mapping supervisord.conf file. Its contents are shown below:

 


 

Moving on and taking a look on our cache_server directory, It contains 4 files. Dockerfile contents are shown below:

 


 

We also have server configuration file named varnish whose contents are shown below.

 


 

And most importantly, our Varnish configuration language file. This VCL file is provided by Magento 2 itself and it works perfectly. Its contents are shown below.

 


 

Take a note that we have mentioned apache2 (apache container name) for backend host in our default.vcl file as our magento code will be mapped to apache container. Also, we also have supervisor for controlling varnish server. Its contents are shown below:

 


 

Proceeding to redis_server directory, we have Dockerfile as,

 


 

and the supervisord.conf file,

 


 

And at last, lets take a look on our ssl_server directory. Its Dockerfile is shown below:

 


 

And its default configuration file,

 


 

As you can see that, nginx is listening to port 80 and 443 and forwarding the traffic to varnish container. In our configuration, we have used private SSL certificates. You can use your own certificates and mention their path in default configuration file.

Contents of nginx.conf file is shown below:

 


 

And its supervisord.conf file,

 


 

And our last directory is magento2 directory. In our case, we have download Magento 2.2.5 from https://magento.com/tech-resources/download and unarchived it in magento2 directory. This directory will be mapped with /var/www/html directory in apache2 docker container.

After extracting Magento 2 files in the desired directory, our project directory setup will be completed. Now in order to build the images with docker-compose.yml file, go to project parent directory and run command,

 


 

This command will build up all three images for apache2, mysql and varnish server. To check the built images, run command,

 


 

Now to run the containers as a part of single project being as mentioned in docker-compose.yml file, run the command:

 


 

Your containers will get running. To list running containers under docker-compose, run command:

 


 

 

Now, your server setup is all ready, now hit your domain name or IP to install Magento 2 store and configure it with Varnish Cache server as we did in https://cloudkul.com/blog/magento-2-and-varnish-cache-integration-with-docker-compose/. Also test the Varnish cache using varnishhist tool.

NOTE:- Use name or id of the mysql container as database host.

Now, to configure redis-server, go to magento2 directory in your main project directory and open file ~/app/etc/env.php. It will be something just like this,

 


 

Add following piece of code in the last second line of this file:

 


 

Please take a note that we have mentioned name of our redis-server container in server value in above code.

Now go to redis-server container and restart the server as,

 


 

You can also check if Redis-server is able to set keys or not,

 


 

You can also use ‘info’ command to get information and statistics about the server as,

 


 

Backing Up Databases from Mysql Docker Container

 

Our databases inside Docker Containers are as critical as our application code. So in order to keep their backup we schedule a shell script that will take backups of all the databases present in mysql-server container and keep them in archived from on our host.

The database backup bash script is shown below:

 


 

Now setup the cron to take regular backup.

 


 

Add the script as,

 


 

It will take database backups twice a day at interval of 12 hours.

So far we have discussed an optimised architecture of Magento 2 using docker-compose tool. Having very complex and code structure, Magento 2 needs optimisation which we have achieved with Varnish-cache, redis-server cache and also our store is configured with SSL. Please refer to repository https://github.com/webkul/magento2-varnish-redis-ssl-docker-compose setup same architecture.

In our later blogs, we explore see more applications of docker. Stay tuned.

author
. . .

Leave a Comment

Your email address will not be published. Required fields are marked*


Be the first to comment.

Start a Project




    Message Sent!

    If you have more details or questions, you can reply to the received confirmation email.

    Back to Home