Limiting Outgoing and Incoming Bandwidth in Linux

From Bubba.org

Jump to: navigation, search

Limiting Outgoing and Incoming Bandwidth in Linux (for 2.2+ Kernels) - 12/3/2000

Overview

The point of this document is to walk you through setting up bandwidth limiting for outgoing and incoming network packets under Linux 2.2.x+ kernel series. The limiting of the outgoing bandwidth is accomplished using the built-in QoS Class Based Queueing (CBQ) functions in the 2.2.x kernel. QoS is considered part of the 2.2.x series, but it was not introduced until the 2.2.9 kernel, so you will 2.2.9 or better to use these features. The limiting of the incoming bandwidth is accomplished using a stand-alone kernel module called rshaper. NOTE: rshaper also requires slight modifications to your network driver source code and is not officially part of the 2.2 kernel (so use it at your own risk). Note: Some people have noted that you can use Ingres Filters, rather than rshaper, to limit incoming bandwidth, but I haven't seen a working implementation of this thus far.

===Before You Begin You first need to download a few utilities:

And if you wish to limit incoming traffic as well, you'll need rshaper:


Installation

You'll first need to configure your kernel for QoS and CBQ. This is accomplished under the "Networking Options" menu. Here you need to enable Routing Messages and the Kernel/User netlink socket. Next, under "QoS and/or Fair Queueing" menu, it is probably best to enable all of the options. Below is the Networking Options portion of my kernel config. You should probably mimic this setup (unless you want to build them as modules). If you do choose to build them as modules, then you'll have to load all the specifc QoS modules (sch_cbq sch_tbf sch_sfq sch_prio cls_u32) after your machine boots (or allow them to be autoloaded). Since I don't like dealing with a slew of modules, I usually compile everything in statically.

            <*> Packet socket                                                         
            [*] Kernel/User netlink socket                                            
            [*] Routing messages                                                      
            <*> Netlink device emulation                                              
            [*] Network firewalls                                                     
            [*] Socket Filtering                                                      
            <*> Unix domain sockets                                                   
            [*] TCP/IP networking                                                     
            [*] IP: multicasting                                                      
            [*] IP: advanced router                                                   
            [*] IP: policy routing                                                    
            [*] IP: equal cost multipath                                              
            [*] IP: use TOS value as routing key                                      
            [ ] IP: verbose route monitoring                                          
            [ ] IP: large routing tables                                              
            [*] IP: fast network address translation                                  
            [ ] IP: kernel level autoconfiguration                                    
            [*] IP: firewalling                                                       
            [ ] IP: firewall packet netlink device                                    
            [*] IP: use FWMARK value as routing key                              
            [*] IP: masquerading                                                      
            --- Protocol-specific masquerading support will be built as modules.      
            [*] IP: ICMP masquerading                                                 
            --- Protocol-specific masquerading support will be built as modules.      
            [*] IP: masquerading special modules support                              
            <*> IP: ipautofw masq support (EXPERIMENTAL)                              
            <*> IP: ipportfw masq support (EXPERIMENTAL)                              
            <*> IP: ip fwmark masq-forwarding support (EXPERIMENTAL)                  
            [ ] IP: optimize as router not host                                       
            < > IP: tunneling                                                         
            [*] IP: aliasing support                                                  
            [ ] IP: ARP daemon support (EXPERIMENTAL)                                 
            [*] IP: TCP syncookie support (not enabled per default)                   
            --- (it is safe to leave these untouched)                                 
            <*> IP: Reverse ARP                                                       
            [*] IP: Allow large windows (not recommended if <16Mb of memory)          
            < > The IPv6 protocol (EXPERIMENTAL)                                      
            ---                                                                       
            < > The IPX protocol                                                      
            < > Appletalk DDP                                                         
            < > CCITT X.25 Packet Layer (EXPERIMENTAL)                                
            < > LAPB Data Link Driver (EXPERIMENTAL)                                  
            [ ] Bridging (EXPERIMENTAL)                                               
            [ ] 802.2 LLC (EXPERIMENTAL)                                              
            < > Acorn Econet/AUN protocols (EXPERIMENTAL)                             
            < > WAN router
            [*] CPU is too slow to handle full bandwidth                              
            QoS and/or fair queueing  --->     
                           [*] QoS and/or fair queueing                               
                           <*> CBQ packet scheduler                                   
                           <*> CSZ packet scheduler                                   
                           <*> The simplest PRIO pseudoscheduler                     
                           <*> RED queue                                              
                           <*> SFQ queue                                              
                           <*> TEQL queue                                             
                           <*> TBF queue                                              
                           [*] QoS support                                            
                           [*] Rate estimator
                           [*] Packet classifier API                                  
                           <*> Routing table based classifier                         
                           <*> Firewall based classifier                              
                           <*> U32 classifier                                         
                           <*> Special RSVP classifier                                
                           < > Special RSVP classifier for IPv6                       
                           [*] Ingres traffic policing                                

While you are in the kernel config, if you have decided to use rshaper and limit incoming traffic, then go ahead and modularize your network device driver (if its not already a module).

                   < M > Sun LANCE support                                        

Doing so will be helpful in modifying the driver and setting it up for use with rshaper. If you aren't going to be using rshaper, then you may skip the next step.

Installing rshaper

Unpackage the source, "make" and then read the README! (from the README)

  • In global space add the following line:
          int (*net_shaper_rx_hook)(struct sk_buff *skb) = NULL;
  • Only for Linux-2.2 (not for Linux-2.0), add this too:
          EXPORT_SYMBOL(net_shaper_rx_hook)

this line must appear after the definition of the hook.

  • In the receive function of the driver change
           netif_rx(skb)

to

           if (net_shaper_rx_hook)
                   (*net_shaper_rx_hook)(skb);
           else
                   netif_rx(skb);

This should be it unless you have more than one network device or if you are not compiling your network driver as a module. If this is the case, please see the README included with rshaper.

Enabling Changes

Compile your kernel and modules, install them and reboot if necessary.

Now, we'll setup our outgoing bandwidth limitations (note, replace eth0 with your network device).

Setup the queue, with your max bandwidth, mine is 10Mbit

 # tc qdisc add dev eth0 root handle 10: cbq bandwidth 10Mbit avpkt 1000

Add your class. Again, bandwidth is 10Mbit. weight should be bandwidth / 10.

 # tc class add dev eth0 parent 10:0 classid 10:1 cbq bandwidth 10Mbit rate 10Mbit allot 1514 weight 1Mbit prio 8 maxburst 20 avpkt 1000

Add your subclass where you'll limit the bandwidth. I'm limiting at a rate of 1024Kbit's, weight should be less than the limit, preferrably by a factor of 10 as above.

 # tc class add dev eth0 parent 10:1 classid 10:100 cbq bandwidth 10Mbit  rate 1024kbit allot 1514 weight 128Kbit prio 5 maxburst 20 avpkt 1000  bounded

We've added the queues, now you must tell the kernel they exist and how to manage them. We'll be using stochastic fairness queuing (sfq)

 # tc qdisc add dev eth0 parent 10:100 sfq quantum 1514b perturb 15

Now we have to tell the kernel to send packets to the queue. Notice the ip/mask. Change this depending on your target network to filter.

 # tc filter add dev eth0 parent 10:0 protocol ip prio 100 u32 match ip dst 192.168.1.0/24 flowid 10:100

If you get RTNETLINK answers: Invalid argument errors, chances are you didn't compile in all the necessary items into the kernel. Check my config above and make sure necessary things are there. For instance, if the last "tc" command above delivers this error, it means that QoS u32 queueing is not compiled into the kernel or the module isn't loaded.

If you do not see any errors, then chances are you have correctly limited the outgoing bandwidth of the above device. Issue the "tc -d qdisc" command. It should look similar to this:

# tc -d qdisc
 qdisc sfq 8001: dev eth0 quantum 1514b limit 128p flows 128/1024 perturb 15sec 
 qdisc cbq 10: dev eth0 rate 10Mbit cell 8b (bounded,isolated) prio \ 
 no-transmit/8 weight 10Mbit allot 1514b level 2 ewma 5 avpkt 1000b maxidle 23us 

You can verify this by connecting to your machine from a remote machine and comparing your download speeds. This is best done on an internal LAN where you're speeds are pretty much guaranteed to be max possible throughput.

If you have chosen to also limit incoming bandwidth using the rshaper utility, then you'll need to load the rshaper.o module at this time.

 # insmod /usr/lib/rshaper.o
  

Then you need to setup your incoming bandwidth limitation using the rshaperctl utility. For more information on this, see the rshaper README. In this case my local network is 192.168.1.0/24.

 # rshaperctl 192.168.1.0/24 128000

This shapes all incoming traffic on the local network to 128KB/s. Running rshapectl with no options will show you the status of shaped hosts.

 # rshaperctl 
 host                     mask         bytes-per-sec      max-queue-len (s)
 192.168.1.0          255.255.255.0         128000                     4

To test this setup, connect from your local machine to a remote machine and compare your download speeds from this machine. Again, this is best done on an internal LAN where you're speeds are pretty much guaranteed.

Script

Here is the script I use to start cbq on my machine:

#!/bin/sh
 
FILTERMASK="192.168.1.0/24"
 
# bandwidth in Kbits
OUT_BANDWIDTH="1024"
 
# bandwidth in bytes
IN_BANDWIDTH="128000"
 
DEV="eth0"
 
###################################################################
PATH=/usr/sbin:/sbin
 
echo -n "Shaping outgoing traffic to ${OUT_BANDWIDTH}Kbit/s "
tc qdisc add dev $DEV root handle 10: cbq bandwidth 10Mbit avpkt 1000
 
echo -n .
 
tc class add dev $DEV parent 10:0 classid 10:1 cbq bandwidth 10Mbit \
rate 10Mbit allot 1514 weight 1Mbit prio 8 maxburst 20 avpkt 1000
echo -n .
 
tc class add dev $DEV parent 10:1 classid 10:100 cbq bandwidth 10Mbit \
rate ${OUT_BANDWIDTH}kbit allot 1514 weight 85kbit prio 5 maxburst 20 \
avpkt 1000 bounded
 
echo -n .
 
tc qdisc add dev $DEV parent 10:100 sfq quantum 1514b perturb 15
 
echo -n .
 
tc filter add dev $DEV parent 10:0 protocol ip prio 100 u32 match ip \
dst $FILTERMASK flowid 10:100
 
echo -n .
echo
 
echo -n "Shaping incoming traffic to ${IN_BANDWIDTH}bytes/s "
modprobe rshaper
rshaperctl $FILTERMASK $IN_BANDWIDTH
echo -n .
echo


Questions/Comments/Information

The information contained herein was gathered from a weekends worth of searches on google. Hopefully you'll find it useful and usable. I claim to be no expert on this topic by any means, so I don't know if I'll be much help if you run into problems, but I'll do my best to help. Contact Bubba

Credits