Introduction
Running out of disk space on your AWS EC2 instance can cause downtime or data loss. In this blog, you’ll learn how to automatically auto-resize EBS volumes when disk usage exceeds a set threshold.
Also If you’re concerned about protecting your AWS infrastructure, check out our article on AWS Shield protection against DDoS attacks.
Overview
This solution automatically monitors disk usage on an EC2 instance. If it exceeds a defined threshold (e.g., 85%), the instance:
- Publishes a message to an SNS topic.
- The SNS topic triggers a Lambda function.
- The Lambda function increases the EBS volume size.
- The EC2 script detects the change and auto-resize the partition and filesystem accordingly.
Prerequisites
- An EC2 instance with:
- Root volume using EBS.
- Filesystem: ext4 or xfs.
- IAM Instance Role (e.g., InstanceRole) with:
|
1 2 3 4 5 |
sns:Publish ec2:DescribeInstances ec2:ModifyVolume ec2:DescribeVolumes ec2:DescribeVolumesModifications |
An SNS topic (e.g., disk-alerts) with a subscription to your Lambda function.
A Lambda function (disk-auto-manage) with execution role that has EC2 modify permissions.
Step 1: EC2 Script to Monitor Disk
We use a Bash script that:
- Checks disk usage.
- If usage exceeds threshold, sends an SNS alert.
- Waits for the volume to be resized.
- Then auto-resize the Linux partition and filesystem.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 |
#!/bin/bash # Ensure AWS CLI is installed if ! command -v aws &> /dev/null; then echo "AWS CLI not found. Installing..." curl -s "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip" apt install unzip -y unzip -qq awscliv2.zip sudo ./aws/install > /dev/null 2>&1 export PATH=$PATH:/usr/local/bin echo "AWS CLI installed." fi # Disk usage threshold THRESHOLD=85 USAGE=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//') # Use IMDSv2 to fetch instance ID TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" \ -H "X-aws-ec2-metadata-token-ttl-seconds: 21600") INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \ http://169.254.169.254/latest/meta-data/instance-id) REGION=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" \ http://169.254.169.254/latest/dynamic/instance-identity/document | grep region | awk -F\" '{print $4}') # If usage exceeds threshold, send alert and wait if [ "$USAGE" -gt "$THRESHOLD" ]; then echo "Disk usage is ${USAGE}%, exceeding threshold." # Publish SNS alert aws sns publish \ --region us-east-2 \ --topic-arn arn:aws:sns:[Region][Your_sns_arn]:disk-alerts \ --message "DISK_ALERT:$INSTANCE_ID:$REGION:$USAGE" # Wait for Lambda to increase volume echo "Waiting 45 seconds for volume expansion..." sleep 45 # Get the root partition (e.g., /dev/nvme0n1p1) ROOT_PART=$(df / | tail -1 | awk '{print $1}') DISK=$(lsblk -no pkname "$ROOT_PART" | head -1) DEVICE="/dev/$DISK" echo "Resolved ROOT_PART=$ROOT_PART, DISK=$DISK, DEVICE=$DEVICE" DISK_SIZE=$(lsblk -b -no SIZE "$DEVICE" | head -1) PART_SIZE=$(lsblk -b -no SIZE "$ROOT_PART" | head -1) if [[ "$DISK_SIZE" -gt "$PART_SIZE" ]]; then echo "Detected volume size increase. Proceeding with resize..." # Extend the partition sudo growpart "$DEVICE" 1 FS_TYPE=$(df -T / | awk 'NR==2 {print $2}') if [ "$FS_TYPE" == "ext4" ]; then echo "Resizing ext4 filesystem..." sudo resize2fs "$ROOT_PART" elif [ "$FS_TYPE" == "xfs" ]; then echo "Resizing xfs filesystem..." sudo xfs_growfs / else echo "Unsupported filesystem: $FS_TYPE" fi echo "Resize complete." # Publish SNS notification for successful resize aws sns publish \ --region us-east-2 \ --topic-arn arn:aws:sns:[Region][Your_sns_arn]:disk-alerts \ --message "DISK_RESIZED:$INSTANCE_ID:$REGION:OldSize=${PART_SIZE},NewSize=${DISK_SIZE},Usage=${USAGE}%" else echo "No volume size change detected after wait period." echo "DISK_SIZE=$DISK_SIZE, PART_SIZE=$PART_SIZE" fi else echo "Disk usage is ${USAGE}%, below threshold." fi |
Note: Replace [Region] with the region where your SNS is set up, and update the ARN value with your SNS topic’s ARN.
Step 2: IAM Role & SNS Setup
Attach an IAM role (e.g., InstanceRole) to the instance with permissions for SNS publishing.
Create an SNS topic (e.g., disk-alerts) in us-east-2 and link it to a Lambda.
Step 3: Lambda to Resize EBS Volume
The Lambda will:
- Parse the SNS message.
- Get the EBS volume ID from EC2 metadata.
- Check current volume size.
- If the volume size is below 100 GB, increase it by 20 GB, if it is 100 GB or more, increase it by 50 GB
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 |
import boto3 def lambda_handler(event, context): print("Incoming Event:", event) try: message = event['Records'][0]['Sns']['Message'] print("SNS Message:", message) if message.startswith("DISK_ALERT"): print("Processing disk usage alert...") # Parse message parts _, instance_id, region, usage = message.split(":") print(f"Instance: {instance_id}, Region: {region}, Disk Usage: {usage}%") ec2 = boto3.client('ec2', region_name=region) # Get instance details instance = ec2.describe_instances(InstanceIds=[instance_id])['Reservations'][0]['Instances'][0] root_device = instance['RootDeviceName'] print(f"Root device: {root_device}") # Get root volume ID volume_id = next( vol['Ebs']['VolumeId'] for vol in instance['BlockDeviceMappings'] if vol['DeviceName'] == root_device ) print(f"🔗 Volume ID: {volume_id}") # Get current volume size volume = ec2.describe_volumes(VolumeIds=[volume_id])['Volumes'][0] current_size = int(volume['Size']) # Decide new size if current_size < 100: new_size = current_size + 20 else: new_size = current_size + 50 print(f"Current size: {current_size} GB → Increasing to: {new_size} GB") # Check for ongoing modification try: mods = ec2.describe_volumes_modifications(VolumeIds=[volume_id])['VolumesModifications'] mod_state = mods[0]['ModificationState'] if mods else 'none' except ec2.exceptions.ClientError as e: if "InvalidVolumeModification.NotFound" in str(e): mod_state = 'none' else: raise print(f"Volume modification state: {mod_state}") # Trigger resize if not already modifying if mod_state not in ['modifying', 'optimizing']: ec2.modify_volume(VolumeId=volume_id, Size=new_size) print("Volume resize initiated.") else: print(f"Skipping resize, volume is already in '{mod_state}' state.") except Exception as e: print(f"Lambda failed: {str(e)}") raise return {"status": "complete"} |
Make sure the Lambda IAM role allows:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
{ "Effect": "Allow", "Action": [ "ec2:ModifyVolume", "ec2:DescribeVolumes", "ec2:DescribeInstances", "ec2:DescribeVolumesModifications" ], "Resource": "*" } |
Step 4: Set Cron Job
crontab -e
|
1 |
*/30 * * * * /usr/local/bin/disk_check.sh >> /var/log/disk_resize.log 2>&1 |
Final Output
Now, your EC2 will:
- Monitor disk usage every 30 min.
- Alert and expand volume automatically.
- And auto-resize filesystem without downtime.