Photo: Brett Sayles / Pexels
When a team plans to collect or process a large volume of publicly available data, bandwidth is often chosen using an overly simple approach: take the total volume, divide it by the number of days, and calculate the average transfer rate.
Mathematically, this calculation is correct. But in real-world infrastructure, it shows only the theoretical minimum. Latency, retries, remote server limits, downtime, and protocol overhead can significantly increase the bandwidth required.
The Theoretical Minimum
For decimal terabytes, the average required transfer rate can be calculated as follows:
`required Mbps = volume in TB × 8 000 000 ÷ seconds`, where `seconds = days × 86 400`.
Suppose a project needs to process 100 TB of data in 30 days. There are 2 592 000 seconds in 30 days, so:
`100 × 8 000 000 ÷ 2 592 000 = 308,6 Mbps`
At first glance, a 400 Mbps connection appears sufficient. But the figure of 308.6 Mbps assumes that data is transferred continuously, around the clock, without a single error. In a real system, this almost never happens.
Why the Average Transfer Rate Is Not Enough
Latency. The longer a remote server takes to respond to a request, the more concurrent connections are needed to use the available bandwidth. It is important to look at both the average response time and the P50 median, as well as P95—the value below which 95% of requests fall. If P50 is 300 ms and P95 reaches two seconds, a calculation based on the average will be overly optimistic.
Retries. Some requests fail because of temporary server unavailability, network failures, or timeouts. If 10% of traffic needs to be retried, the project effectively transfers around 110 TB instead of 100 TB.
Downtime. A system rarely operates at maximum speed 100% of the time: restarts, updates, reduced overnight workloads, remote resource limits, and maintenance all affect performance. It is therefore useful to apply a utilization factor: a value of 0.70 means that, on average, 70% of theoretical bandwidth can be used.
A Practical Model
For a more realistic estimate:
`planned Mbps = theoretical Mbps × (1 + retry fraction) ÷ utilization factor`
For our example, the theoretical transfer rate is 308.6 Mbps, retries account for 10%, and the utilization factor is 70%.
`308,6 × 1,10 ÷ 0,70 ≈ 485 Mbps`
In other words, the project theoretically needs only 308.6 Mbps, but for practical operation, it is more sensible to budget for around 485 Mbps. This is not a universal figure: it varies with the workload, latency, response sizes, and connection quality.
A Python Calculation Example
“`python from dataclasses import dataclass
SECONDS_PER_DAY = 86_400 BITS_PER_TERABYTE = 8_000_000
@dataclass(frozen=True) class BandwidthEstimate: theoretical_mbps: float planned_mbps: float
def estimate_bandwidth(traffic_tb: float, days: int, utilization: float = 0.70, retry_fraction: float = 0.10) -> BandwidthEstimate: if traffic_tb <= 0: raise ValueError(“traffic_tb must be positive”) if days <= 0: raise ValueError(“days must be positive”) if not 0 < utilization <= 1: raise ValueError(“utilization must be between 0 and 1”) if retry_fraction < 0: raise ValueError(“retry_fraction cannot be negative”)
seconds = days * SECONDS_PER_DAY theoretical = traffic_tb * BITS_PER_TERABYTE / seconds planned = theoretical * (1 + retry_fraction) / utilization return BandwidthEstimate(round(theoretical, 1), round(planned, 1))
print(estimate_bandwidth(traffic_tb=100, days=30)) # BandwidthEstimate(theoretical_mbps=308.6, planned_mbps=484.9) “`
What “1 Gbps = 324 TB” Actually Means
These figures represent the connection’s physical capacity, rather than a guaranteed usable data volume. A 1 Gbps connection can indeed theoretically transfer 324 TB in 30 days. But if actual utilization is 70% and 10% of requests are retried, the usable volume will be significantly lower.
Why Testing Must Be Done from the Actual Region
For projects operating from Kazakhstan or accessing infrastructure in other countries, it is important to run tests from the same region where the production system will run. Results are affected by:
- international routing;
- the target server’s location;
- the time of day and network congestion;
- DNS and connection setup time;
- response size;
- the number of concurrent connections.
A test from a European data center does not always reflect the behavior of a system deployed in Kazakhstan.
How to Handle 429 and 403 Responses
HTTP 429 and 403 responses should not be treated as signals to keep increasing the number of connections indefinitely. A 429 status code usually means that the remote service is limiting the request rate: concurrency should be reduced and bounded exponential backoff applied. A 403 status code may indicate a lack of permission, an authentication requirement, or restrictions imposed by the resource owner.
The goal of a well-designed system is to maximize the number of permitted, successful responses, rather than the total number of requests sent.
Calculation Steps Before Scaling
- Determine the total volume of data you are permitted to access.
- Establish the available time frame.
- Calculate the theoretical average transfer rate.
- Estimate the proportion of retries.
- Choose a realistic utilization factor.
- Measure P50 and P95 latency.
- Run a small, controlled test.
- Increase the workload only while the success rate remains stable.
Conclusion
Bandwidth is a data transfer rate, rather than a data volume. A connection with no data cap still has a specific speed. Conversely, a high advertised speed does not guarantee that a project can continuously use 100% of it.
Infrastructure should therefore be selected using more than a theoretical formula. A reliable estimate combines data volume, available time, latency, retries, downtime, and actual measurements from the region where the system will operate.
Michael Guy writes practical articles on Python, network infrastructure, and the responsible collection of publicly available data. The author’s views may not reflect those of the editorial team.