Integrate Magento 2 with EFS

When it comes to high end server architectures, couple of things that come in mind are giant servers with huge RAM and multi-core CPUs, Load Balancer, an Auto scaling manager, a shared file system,  Cache mechanism etc. For deploying this architecture we can either build all of its components from scratch or we can lookup to someone who can provide these features with flexibility and ease so that we can focus on our application.

Amazon Web Services, along with various services, provides Elastic Load Balancing, Auto scaling, Elastic File System etc to implement desired architecture. AWS indeed is very flexible and scalable but its services might not work perfectly out of box for our application without proper tuning.

 

Magento 2 with EFS:

 

Here today we will discuss the implementation of Amazon Elastic File System (EFS) with well known E-commerce platform Magento 2. Previously we have discussed what is Amazon Elastic File System and How to create an Elastic File System. Quoting from AWS docs,

 

Elastic File System is simple, scalable and elastic file storage system for our EC2 instances. It is analogous to NAS (network attached storage), which is a file level data storage connected to a network and providing data to a heterogeneous group of clients across the network.

 

EFS can be mounted to multiple EC2 instances irrespective of its size, location and availability. It is highly available and accessible across various availability zones. It provides data consistency to all attached EC2 instances. In our architecture, we will mount Magento 2 media directory on EFS and this directory will shared across various EC2 instances. All EC2 instances sharing Magento 2 media directory will be deployed under a Elastic Load Balancer. We can check introduction and implementation of Elastic Load Balancer in our older blogs.

ELB can automatically distribute traffic among various instances and manage our load as per their availability, justifying the term elastic associated with it. It monitors the health of registered targets and routes traffic only to the healthy targets. Also, it serves as an end point for clients, and it can add or remove desired number of instances, which ensures high availability.

So all the EC2 servers within Load Balancer will fetch same contents from media directory. Any changes in media directory (mounted on EFS) will be instantly reflected on all the EC2 instances sharing it.

Now the another obvious question arises that what about other files and directories in Magento 2 ? should they be mounted on EFS as well?  If not, how the changes in other files are going to reflect across multiple servers.

Well, we can surely mount all the files and directories of Magento 2 in single or multiple EFS and there modifications can be instantly reflected to all the attached instances. But problem is that Amazon EFS is slow !!

From the AWS support forum, it can be quotated as:

 

While the distributed architecture of Amazon EFS enables high levels of availability, durability, and scalability, it results in a small latency overhead for each file operation. If the workload involves reading/writing lots of small files, the per-operation latencies can result in EFS performance being different from a local file system performance. This is why performance is degraded when dealing with small R/W IOPS – it is a limitation of the EFS architecture.

 

So as we are familiar that changes in Magento 2 application code (files, modules etc) requires upgradation, code compilation, static content generations followed by re-indexing and cache clearance. Even on local file system (EBS, in case AWS), these process may take considerable amount of time depending upon files count. As EFS is already slow, these time taking upgrading process may lead to operation timeouts and much slower process rate if whole Magento 2 is mounted upon EFS. That is why we are restricting our mount directory only to media folder as files generated while Magento 2 code upgrade will be on local EBS and will not consume much time.

 

Version Control System:

 

So the approach we consider better to update rest of the code is via GIT management. Our application can be placed on GIT or any other version control system suitable to our architecture. File modifications can be done on GIT and then updated code can be pulled to our primary EC2 instance. Then, we can upgrade our code, compile it, deploy static content and clear all the cache. After all the updates and code being properly tested, we can create AMI of updated EC2 instance and launch instances, managing it under elastic load balancer.

So putting it together, our architecture will look something like this:-

 

 

We can also deploy Jenkins or use a customised script to automate data pull from GIT, its upgrade (along with needful Magento 2 commands to update code), AMI creation, EC2 instance launch and Load balancer modification.

 

Mounting Media Directory On EFS:

 

For Elastic File System creation, you can check the blog on EFS setup. However here we will discuss setting up EFS client on Magento 2 server.

After creating EFS in General Purpose mode and setting up mount targets, we will install EFS client on our server (Ubuntu in our case).

Now we will download latest Magento 2 version and unarchive it on our server. Go to your Magento 2 directory and rename pub/media folder (for taking backup). Now create  a directory as media in pub directory and mount it on EFS as,

Now copy the contents of that renamed ~/pub/media folder to media directory mounted on EFS. Our Magento 2 is ready to be installed.

After successful mount, we can create entry in /etc/fstab or /etc/rc.local to mount filesystem on boot.

There are still other ways to be explored to reduce overhead while performing operations on EFS. In our later blogs, we will check out other ways that can be used to deploy high end Magento 2 architecture with EFS and other services.