When Just IOPS Aren’t Enough – Optimal EBS Bandwidth Calculation

adminComputers

The current EBS features in Amazon EC2 environment offers really good performance and the marketing material and documentation boosts how different EBS types give IOPS, but a lesser talked thing is the bandwidth which EBS can offer. Traditional SATA attached magnetic hard disks can usually provide speeds from 40-150 MiB/s but really low amount of IOPS – something between 50-150 depending on rotation speed. With the Provisioned and GP2 EBS volume types it’s easy to focus on just getting enough IOPS but it’s important not to forget how much bandwidth the instance can get from the disks.

The EBS bandwidth in EC2 is limited by a few different factors:

  • The maximum bandwidth of the EBS block device
  • EC2 instance type
  • Is EBS-Optimised bit turned on for the instance

In most instance types the EBS traffic and the instance network traffic flows on the same physical NIC attached. In these cases the EBS-Optimised bit simply adds a QOS marker to the EBS packets. According to AWS documentation some instance types have a dedicated network interface for the EBS traffic and thus they don’t need to offer separated EBS-Optimised mode. This can be seen as the instances are natively always EBS-Optimised in the AWS documentation. It seems that the c4, d2 and m4 instance types seems to have this dedicated EBS NIC in the physical host.

The EBS type itself has bandwidth limitations. According to the documentation GP2 has maximum throughput of 160 MiB/s, Provisioned volume 320 MiB/s and magnetic 40-90 MiB/s. The GP2 and Provisioned bandwidth is determined on the volume size, so a small volume will not achieve the maximum bandwidth.

So to get maximum bandwidth in a single EC2 instance with EBS you should choose an instance type which is always EBS optimised, calculate maximum bandwidth of your EBS instance types and usually combine several EBS volumes together with stripped LVM to obtain best performance. For example a c4.4xlarge has max EBS bandwidth of 250 MiB/s and would require two GP2 EBS volumes (2 * 160 MiB/s) to max it out. According to my tests a single striped LVM logical volume which is backed up with two EBS volumes can achieve a constant read or write speed of 250 MiB/s while also transferring at 1.8 Gbps speed over the ethernet network to another machine.