On Bandwidth and Backhaul Provisioning
I’ve been asked this question (and many like it) plenty of times before, and so what follows is something of an essay that I have written out from scratch many times before. So this time, I've written it all down as an online article, so that next time someone asks, I can just provide this url! ;-)
Basic Constraints:
There are a many ways to address backhaul requirements, and some are better than others. Essentially it all comes down to two basic concepts: Rate and Contention.
Rate is the base throughput capacity of the link: e.g. 8megabits ADSL, 10Mbit ethernet, 4Mbit SHDSL etc.
Contention is about the total number of simultaneous requests – there may well be 50 users, but they are not always downloading at the same time.
Oversubscription:
Most ISP services available provide an ‘oversubscribed’ service – which essentially means that there are other users sharing the same backhaul link. For example, you might buy an 1meg adsl service from bigpond in Redfern. Sure, that ADSL service is capable of carrying 1meg down, but there are probably 1000 other homes in Redfern that also have an 1Meg service connected to the same exchange as you are, and there is probably only a 100meg backhaul to that exchange. That means that there is potentially 1meg * 1000 = 1000 megabits demand on a 100megabit backhaul. That’s what we would call a 10:1 contention ratio. Most ISPs will deliver a service way in excess of 10:1 – a typical residential ADSL service is at least 20:1 and some satellite services can be as high as 50:1 or 100:1.
So when you are shopping around for a backhaul service for your project, it is equally important to know what the contention rate on that service is as knowing what speed it is.
If you buy an 8meg service with a 10:1 contention rate and put 50 users on it, then you blow out to a 500:1 contention rate which is way too high. If you buy a 1:1 contention service, then you might be able to put 50 users online and not hit too much congestion too often.
Download Limits:
The next variable you need to consider is download capacity. An 8 meg DSL service (assuming it is uncontended) can deliver about 1,000 gigs per month if it is running flat chat 10 hours a day. Assuming that most users will be using their services during 10 hours of peak time during the day throughput the month, then it is useful to work on a concept of selling gigabytes: you have 1000 gigs to share among 50 users, so you can allocate 20 gigs each, and be reasonably confident that they have a decent chance of getting their 20 gigs in a month without causing too much congestion too often. You can ‘encourage’ users to limit their downloads to the allocated amount by hitting them hard with big penalties if they go over, or by reducing their access speed to a trickle.
If you do the latter, then you need to be careful about start-of-month-binge-download-syndrome: where all those download junkies get their quota back on the first of each month and all start downloading flat out – result network congestion to the max. One way to overcome this effect is to stagger billing dates throughput the month so that there is no single day when a large number of users suddenly increase their download to the max. The DuxAdmin application is specifically designed to assist with this technique.
Other ways involve careful set up of bandwidth access policies to rate limit individual connections over extended period of time, so that a download starts off at full speed, but shapes down over time so that a web page is received very fast, but a 10Gig movie file downloads at full speed for the first minute, then half speed for the second minute, then 10th speed for the remainder. The Mikrotik RouterOS features some very powerful traffic management systems to facilitate many different bandwidth control policies, including the kind of scheme described here.
Multiple Links:
If the backhaul technologies available simply can’t give you the capacity (rate AND contention) required, then you may need to consider multiple backhauls, which introduces some more interesting complexities.
First of all, if you bring in two 8meg services, that does not usually mean that you can offer customers a 16 meg download service. Sure, if they start off multiple downloads, then the total sum of all downloads might reach up to 16 meg, but a single stream will not usually achieve that rate. If you have access to the other end of the ADSL links – like if you have some rack space in the ISP colo centre for example, then you *can* put a router at the site and one at the colo and set up “bonded” links over multiple services. This can have the effect of providing capacity for a single 16 meg download stream over two 8meg adsl services. It can be somewhat expensive though, and may not be necessary for every application.
A more common technique is to set up each link as an independent service and send connection requests out one service or the other according to some scheme. One scheme might be a ‘round robin’ approach, where the first connection from any client is sent through the first service, then the next connection request goes out the second link regardless if it is the same client or a different one. This can have a bad effect on some applications that are implemented over multiple connections but require a constant client IP address, because each outbound connection request will carry the source IP address of the ISP interface that it was transmitted on.
An alternative load balancing scheme scheme might be to choose the service link to send a connection request from a client, and then always send requests from that client using that link. The next client to make a request will be sent using the second link, and so on. The selected links might be set to time-out after a certain time to avoid an uneven build up of too many clients using one link and just a few using some other link.
Another scheme involves selection of links based on traffic type or destination – like send all web traffic through one link, all mail traffic via a second, you tube traffic through a third link, and everything else through a fourth.
The best methods usually form a combination of all three. Once again, Mikrotik RouterOS comes to the rescue by providing a sophisticated platform capable of delivering pretty much any routing design you can think of!
A third method to use multiple ISP links is to use BGP routing services to provide both load sharing and automated fail-over for both inbound and outbound traffic. This is particularly useful if you are hosting selected services inside your local network (web sites and mail servers, etc) but also requires that you hold your own public IP address space.
ISP Service Constraints:
Last of all, it is always important to consider the terms of use that come with the service as defined by the ISP. After getting this far through this discussion, you will probably be understanding how important it is to have obedient customers that behave according to expected parameters. All it takes is for one user to regularly hammer the service out of normal expectations to throw performance out of whack for every other user sharing the backhaul. It is this reason that most ISPs will insist on stringent ‘acceptable use’ policies, and will often quite explicitly prohibit any kind of network sharing – especially in public access wireless hotspot or communal internet service arrangements. Therefore, you should always carefully read through the intended ISP service agreement before proceeding.
I hope that this all helps to highlight the relevant points that you need to consider when designing backhaul services for your projects. If you need any further information or clarification, feel free to ask.