Troubleshooting Network Using iPerf

👋 Hi! I’m Bibin Wilson. In each edition, I share practical tips, guides, and the latest trends in DevOps and MLOps to make your day-to-day DevOps tasks more efficient. If someone forwarded this email to you, you can subscribe here to never miss out!

In this edition, we'll explore how to identify bandwidth issues between two servers.

Troubleshooting network issues is one of the key tasks for a DevOps engineer.

Even if an organization has a dedicated network team, a DevOps engineer should still perform an initial investigation into network problems to either identify or rule out network issues.

Use Case

Let’s say you are working in a cloud environment with an application server in one region or zone and a database or API in another region or zone.

Users are complaining about slow response times, and you suspect there might be a network issue between the two servers.

This is where iPerf can help.

It allows you to measure the bandwidth between the servers, which can highlight whether the network is causing the slowdown.

For example, your network connection can handle up to 1 Gbps of data (this is your bandwidth), you are only getting 200 Mbps (this is your throughput).

Examples

When deploying applications across multiple cloud regions or a combination of on-premises and cloud environments, iperf is used to test network performance between these locations.

For example, it helps validate VPN throughput in a site-to-site VPN.

You can also determine how long it takes for data to travel between two points. For instance, you can use iperf to measure the latency between an API backend and a database.

In Kubernetes environments, iperf can test network performance between pods, nodes, or services. This helps identify issues with overlay networks, CNI plugins, or network policies. (Refer example here)

iPerf Setup

To test network bandwidth between two servers using iperf, for example, one server in the cloud and another in an on-premises environment, follow these steps:

It could also be between a Kubernetes pod in the cloud and a server on-premises.

Either way, you need to install iperf on both servers.

To install iperf.

sudo apt -y install iperf3

Run iperf in server mode on one machine and in client mode on the other to perform the test.

For example, to run iperf as a server, you just need to run:

$ iperf3 -s

------------------------------------
Server listening on 5201

From the iperf client, you just need to run the test using the IP address of server 01.

For example, let’s say server 01 has the IP address 172.31.28.28:

iperf3 -c 172.31.28.28

You will get an output like the one below.

The highlighted Bitrate (Gbits/sec) shows the speed of data transfer, along with other information that can help you understand the bandwidth.

To get the average throughput, you just need to add up all 10 values from the output (just an example)

Divide by total intervals (10 seconds): 10.31/10 = 1.03Gbit/sec

So, in my test, the average throughput for the entire duration is approximately 1.03 Gbits/sec.

Now, if you're testing a 10 Gbps link (as an assumption), the throughput is using only about 10% (1.03 Gbps) of the capacity.

This could indicate an an issue like throttling, congestion, or hardware limitations.