EdgeRouter – WAN load-balance and Tunnels

In this Knowledge Base article we’ll cover some of the special considerations that are necessary with combining WAN load-balancing and tunnels.  For simplicity this example will use a GRE tunnnel (easier debugging with packet capture on non-encyrpted data), but if theory should work for other tunnels.

Our topology:

 

Router R1                                                                         Router R2

 

eth0      WAN1   20.0.0.2/30                                             eth0  WAN  30.0.0.2/30

eth1      LAN      192.168.1.1/24                                        eth1  LAN   172.16.1.1/24

pppoe0 WAN2

tun0      GRE     40.0.01/30                                               tun0   GRE  40.0.0.2/30

We’ll start with the GRE tunnel, then add load-balancing after.

We want our GRE tunnel to go from R1 eth0 to R2 eth0.  So for R1 we add:

 tunnel tun0 {       address 40.0.0.1/30       encapsulation gre       local-ip 20.0.0.2       remote-ip 30.0.0.2   } 

For R2:

 tunnel tun0 {       address 40.0.0.2/30       encapsulation gre       local-ip 30.0.0.2       remote-ip 20.0.0.2   }  

 

Then we need to open the firewall for protocol GRE for WAN_LOCAL:

 

ubnt@R1# show firewall name WAN_LOCAL rule 40   action accept   description "Allow GRE"   protocol gre   source {       address 30.0.0.2   }  

 

Then we’ll likely need TCP mss-clamp:

 

ubnt@R1# show firewall options    mss-clamp {       interface-type tun       interface-type pppoe       mss 1412   }  

 

Last we need to add a static route to tell the system that R2’s LAN network should be sent out tun0 interface.

     interface-route 172.16.1.0/24 {           next-hop-interface tun0 {           }       }

At this point our GRE tunnel should work, but

ubnt@R1:~$ ping 172.16.1.1  PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.  64 bytes from 172.16.1.1: icmp_req=1 ttl=64 time=0.832 ms  64 bytes from 172.16.1.1: icmp_req=2 ttl=64 time=0.741 ms  64 bytes from 172.16.1.1: icmp_req=4 ttl=64 time=0.635 ms  64 bytes from 172.16.1.1: icmp_req=5 ttl=64 time=0.654 ms  64 bytes from 172.16.1.1: icmp_req=6 ttl=64 time=0.644 ms  64 bytes from 172.16.1.1: icmp_req=7 ttl=64 time=0.614 ms  ^C  --- 172.16.1.1 ping statistics ---  8 packets transmitted, 6 received, 25% packet loss, time 7006ms  rtt min/avg/max/mdev = 0.614/0.686/0.832/0.082 ms 

So it kind of works, but 25% packet loss is not good.  One theory is that since we have 2 Internet connection, that some packets are going out the wrong interface.  To test that we’ll take down the 2nd WAN interface.

ubnt@R1:~$ disconnect interface pppoe0   Bringing interface pppoe0 down...  ubnt@R1:~$   ubnt@R1:~$   ubnt@R1:~$ ping 172.16.1.1  PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data.  64 bytes from 172.16.1.1: icmp_req=1 ttl=64 time=0.789 ms  64 bytes from 172.16.1.1: icmp_req=2 ttl=64 time=0.606 ms  64 bytes from 172.16.1.1: icmp_req=3 ttl=64 time=0.585 ms  64 bytes from 172.16.1.1: icmp_req=4 ttl=64 time=0.667 ms  64 bytes from 172.16.1.1: icmp_req=5 ttl=64 time=0.650 ms  64 bytes from 172.16.1.1: icmp_req=6 ttl=64 time=0.707 ms  64 bytes from 172.16.1.1: icmp_req=7 ttl=64 time=0.752 ms  64 bytes from 172.16.1.1: icmp_req=8 ttl=64 time=0.613 ms  ^C  --- 172.16.1.1 ping statistics ---  8 packets transmitted, 8 received, 0% packet loss, time 7007ms  rtt min/avg/max/mdev = 0.585/0.671/0.789/0.069 ms

Problem solved! But how do we use the 2nd WAN. One way is to add a static route for the far GRE end-point and force it to go out eth0.

     route 30.0.0.2/32 {           next-hop 20.0.0.1 {           }       }  

Now we can bring back up the pppoe0 interface and still the GRE tunnel works fine.

The full configurations for router R1 and R2 for this part of the KB are:

R1 /config/config.boot GRE

R2 /config/config.boot GRE

 

Load-Balancing

 

Now we’ll add WAN load-balance to R1.  Start with the load-balance section:

 

load-balance {      group WLB {          interface eth0 {          }          interface pppoe0 {          }      }  }  

This is all we really need, but we’re going to change the ping target to an address so that the health check doesn’t need DNS.

load-balance {      group WLB {          interface eth0 {              route-test {                  type {                      ping {                          target 8.8.8.8                      }                  }              }          }          interface pppoe0 {              route-test {                  type {                      ping {                          target 8.8.8.8                      }                  }              }          }      }  }  

 Then we’ll add a firewall modify rule

         modify BALANCE {          rule 10 {              action modify              description "Do not load-balance LAN to LAN traffic"              destination {                  address 192.168.1.0/24              }              modify {                  table main              }          }          rule 20 {              action modify              description "load-balance the rest of LAN to WAN traffic"              modify {                  lb-group WLB              }          }      }    

Then apply the modify rule to the LAN “in”.

    ethernet eth1 {          address 192.168.1.1/24          description LAN          firewall {              in {                  modify BALANCE              }          }      }  

 At this point our LAN traffic is being balanced over both WAN’s fairly evenly:

 

ubnt@R1:~$ show load-balance status   Group WLB    interface   : eth0    carrier     : up    status      : active    gateway     : 20.0.0.1    weight      : 50    flows        WAN Out : 682        WAN In  : 0      Local Out : 20      interface   : pppoe0    carrier     : up    status      : active    gateway     : pppoe0    weight      : 50    flows        WAN Out : 672        WAN In  : 0      Local Out : 22  

 

But our GRE tunnel now doesn’t work.  The first problem is that we really don’t want to load-balance the LAN traffic that is supposed to go over the GRE tunnel.  So we’ll add a rule before the lb-group rule like:

 

ubnt@R1# show firewall modify    modify BALANCE {       rule 10 {           action modify           description "Do not load-balance LAN to LAN traffic"           destination {               address 192.168.1.0/24           }           modify {               table main           }       }       rule 20 {           action modify           description "Do not load-balance traffic for gre tunnel"           destination {               address 172.16.1.0/24           }           modify {               table main           }       }       rule 30 {           action modify           description "load-balance the rest of LAN to WAN traffic"           modify {               lb-group WLB           }       }   }    

 This is mostly working now, but sometimes I need to take down the pppoe interface for the GRE tunnel to come up correctly.  The next issue is related to how the load-balance feature is currently implemented with a separate routing table per WAN interface that only has a default route for that WAN.  This works fine for LAN to WAN sessions, but breaks when we need a connected route from the “main” routing table.  So the current work around is to instead of using the system generated separate routing table, instead create your own that has the default route for that WAN and the connected routes that are needed.

 

ubnt@R1# show protocols static table    table 1 {       interface-route 20.0.0.0/30 {           next-hop-interface eth0 {           }       }       interface-route 192.168.1.0/24 {           next-hop-interface eth1 {           }       }       route 0.0.0.0/0 {           next-hop 20.0.0.1 {           }       }   }   table 2 {       interface-route 0.0.0.0/0 {           next-hop-interface pppoe0 {           }       }       interface-route 20.0.0.0/30 {           next-hop-interface eth0 {           }       }       interface-route 192.168.1.0/24 {           next-hop-interface eth1 {           }       }   }  

 

And then change the load-balance config to use the over-ride routing table.

ubnt@R1# show load-balance              group WLB {       interface eth0 {           route {               table 1           }           route-test {               type {                   ping {                       target 8.8.8.8                   }               }           }       }       interface pppoe0 {           route {               table 2           }           route-test {               type {                   ping {                       target 8.8.8.8                   }               }           }       }   }  

The full config file for R1 with GRE and WAN load-balance is:

 

R1 /config/config.boot GRE with WAN load-balance

 

Below is the example config files for an IPSec tunnel between R1 and R2 while R1 has WAN load-balancing configured.

 

R1 /config/config.boot IPSec to R2 with WAN load-balance

R2 /config/config.boot IPSec to R1

 

Leave a Reply

Your email address will not be published. Required fields are marked *