25K Views

Magento 2 and Varnish Cache Integration With Docker-Compose

In our last blog, after being introduced to docker-compose, we had discussed the deployment of multi-container architecture in Docker. Using this tool we had setup Magento 2 running with apache2 server and mysql-server on separate containers interlinked with each other. We had also discussed the volume mapping, ports allocation, containers linking etc techniques for docker-compose.yml file.

As we are now familiar with docker and docker-compose, their working and their integration with applications (Magento 2 in our case), we will now take a further step ahead. In our previous blogs, we have installed Magento 2 with default configurations. We were focused on infrastructure performance rather than application optmisation. Moving further, in this blog we will discuss integration of Varnish cache with Magento 2 on the architecture of Docker and of course we will be using docker-compose to setup this architecture as we did earlier.

 

Introduction To Varnish Cache

 

If you are familiar with Magento 2 and own your store then you might have heard of Varnish cache once in a while. If not all of you, then those who are struggling with high traffic on their Magento 2 store, they must have considered using some cache server at some point. Those who are not familiar with Varnish cache, following lines are for you,

 

Quoting from Varnish docs, Varnish cache is a web application accelerator also known as a caching HTTP reverse proxy. It is a powerful open source HTTP engine/reverse HTTP proxy that can speed up a website by up to 1000 percent by doing exactly what its name implies: caching (or storing) a copy of a webpage the first time a user visits. Varnish cache visits your server once to cache the page, then all future requests for the same page will be served by Varnish cache. 

 

Unlike other web-accelerators, Varnish cache deals only with HTTP traffic. The key element in Varnish cache setup is its Varnish Configuration Language (VCL) file. VCL files are needed to be customised for specific frameworks or applications. Default VCL file does not work directly with all the applications. If you consider Varnish Cache as a radio, then you have it tune it according your server needs. Luckily for us, Magento 2 works out of box with Varnish Cache and provides its own VCL file for its setup. Also from Magento 2 docs,

 

“We strongly recommend you use Varnish in production. The built-in full-page caching (to either the file system or database) is much slower than Varnish, and Varnish is designed to accelerate HTTP traffic.”

 

Magento 2 And Varnish Cache Setup With Docker-Compose

 

As we are still on our multi-container docker architecture, we will be using separate containers for apache2 server, mysql-server and varnish cache server for its integration with Magento 2 on Ubuntu 16.04. As separate containers will require inter-linking with each other, we will be using docker-compose tool to achieve this architecture.

The parent directory holding all the files/directories will be considered as the project name. Custom project name can also be in set docker-compose.yml file. Here we are creating separate directories for apache2, mysql server and varnish cache server setup that hold their Dockerfile(s) and their associated volumes.

Take a note that: following the same approach, Magento 2 files will be placed on our host. As it is a good practice to keep application files on host so that it will not be lost if containers or images get accidentally removed. These magento files will be mapped from host to running docker container.

We are also using supervisor to control all three of our servers in their respective containers. Apart from controlling these servers, supervisor is also running various commands and scripts that will be mentioned later in this blog.

To begin with create a directory on your Ubuntu 16.04 server for this project. Our directory architecture will be something like:

 

  • docker-compose.yml
  • web_server 

Dockerfile

supervisord.conf

  •  database_server

Dockerfile

mysql.sh

supervisord.conf

  • cache_server

Dockerfile

default.vcl

supervisord.conf

varnish

  • magento2

Unarchived Magento 2 files and directories.

 

The docker-compose.yml file is shown below:

 

 

Here in YAML file above, we are defining three services: cache_server, web_server and database_server.

 

  • The cache_server is associated with our Varnish cache server configuration. Container name defined is varnish, linked to web_server and port 80 & 6082 is allocated to it. There are three files that are being mapped from host to docker container and “context” points to its Dockerfile installing Varnish version 4.1.
  • The web_server is associated with our apache server configuration. Container name defined for this service is apache2, linked to database_server and port 8080 is allocated to it. There are four volumes or files are being mapped from host to docker container and “context” under “build” points to location of its Dockerfile.
  • Also, the service database_server is associated with mysql-server. Container name is defined as mysql and port 3306 is allocated to it. Mysql root password will be passed as build argument and their are two volumes/files mapped from host to docker container. Same as web_server, “context” points to location of its Dockerfile installing mysql-server-5.7.

 

Lets take a look in our web_server directory. It contains 2 files. Dockerfile is shown below:

 

 

And at last we have supervisord.conf file that supervisor use to run apache2 server and ownership commands. Its contents are shown below:

 

 

Moving on to our database_server directory, it contains 3 files. Dockerfile is shown below:

 

 

So as we have seen in our last blog as well that Dockerfile just install packages as it is instructed to do by the commands.

As we have mentioned earlier consider image as rest package where no services or processes are running. That is why we cannot perform any operation from Dockerfile that requires a particular service to be running. As with case of database, we cannot create database from Dockerfile as mysql service is not running. For database and its user creation, we will create a bash script that will run whenever a container will launch, hence creating mentioned database and its user.

We are using “mysql.sh” as the bash script as mentioned in Dockerfile. Bash script “msyql.sh” resides on our host parallel to Dockerfile.

Contents of mysql.sh is shown below:

 

 

Apart from mysql.sh and Dockerfile, we are also mapping supervisord.conf file. Its contents are shown below:

 

 

Moving on and taking a look on our cache_server directory, It contains 4 files. Dockerfile contents are shown below:

 

 

We also have server configuration file named varnish whose contents are shown below.

 

 

And most importantly, our Varnish configuration language file. This VCL file is provided by Magento 2 itself and it works perfectly. Its contents are shown below.

 

 

Take a note that we have mentioned apache2 (apache container name) for backend host in our default.vcl file as our magento code will be mapped to apache container. Also, we also have supervisor for controlling varnish server. Its contents are shown below:

 

 

And our last directory is magento2 directory. In our case, we have download Magento 2.1.6 from https://magento.com/tech-resources/download and unarchived it in magento2 directory. This directory will be mapped with /var/www/html directory in apache2 docker container.

After extracting Magento 2 files in the desired directory, our project directory setup will be completed. Now in order to build the images with docker-compose.yml file, go to project parent directory and run command,

This command will build up all three images for apache2, mysql and varnish server. To check the built images, run command,

Now to run the containers as a part of single project being as mentioned in docker-compose.yml file, run the command:

Your containers will get running. To list running containers under docker-compose, run command:

 

Then proceed further and install the Magento 2.

After successful installation, we need to configure Magento 2 from admin panel to use Varnish Cache instead of its built-in cache. So, login to your admin panel. Go to STORES –> Configuration –> ADVANCED –> System –> Full Page Cache as,

Select Varnish Cache in Caching Application and scroll down,

 Mention the backend host, backend port and Access list. As we already have our VCL file “default.vcl”, we don’t need to export it. Now, clear all the cache from Magento admin. Now, Varnish Server has been integrated with our Magento 2 store.

As  you might have noticed that we are running apache2 server (our backend server) on port 8080 and Varnish cache server  on port 80. So here’s how Varnish is working: Varnish acts a reverse proxy server typically sits at port 80 and directs client requests to the back-end apache2 server at port 8080. Whenever a client makes a request, Varnish server checks the content within the cache and incase data not found it send the request to backend server and fetch the content to client and keep a copy of the data as cache. When the same request is made, Varnish does not bother apache2 server, it just fetch the data from the cache. It provides an additional level of abstraction and control to ensure the smooth flow of network traffic between clients and servers.

To check if varnish is working or not, go to your varnish container,

Now, run command:

And hit your domain URL, when first time you hit it you will see ‘#’ on the terminal representing cache miss as first request is forwarded to main server. Next time as you hit the browser, you will see ‘|’ increasing and ‘#’ decreasing representing cache hits.

So far we have discussed how to setup Magento 2 and integrate it with Varnish Cache server using docker-compose on multiple docker container architecture. Please refer to repository https://github.com/webkul/magento2-varnish-docker-compose to deploy same architecture.

In our next blog we will add an extra layer in our current architecture by integrating redis-server with Magento 2.

Category(s) Uncategorized
. . .

Comment

Add Your Comment

Be the first to comment.

css.php