Our topology:
Router R1
eth0 WAN1 20.0.0.2/30
eth1 LAN 192.168.1.1/24
pppoe0 WAN2
tun0 GRE 40.0.01/30
We’ll start with the GRE tunnel, then add load-balancing after.
We want our GRE tunnel to go from R1 eth0 to R2 eth0. So for R1 we add:
tunnel tun0 { address 40.0.0.1/30 encapsulation gre local-ip 20.0.0.2 remote-ip 30.0.0.2 }
For R2:
tunnel tun0 { address 40.0.0.2/30 encapsulation gre local-ip 30.0.0.2 remote-ip 20.0.0.2 }
Then we need to open the firewall for protocol GRE for WAN_LOCAL:
ubnt@R1# show firewall name WAN_LOCAL rule 40 action accept description "Allow GRE" protocol gre source { address 30.0.0.2 }
Then we’ll likely need TCP mss-clamp:
ubnt@R1# show firewall options mss-clamp { interface-type tun interface-type pppoe mss 1412 }
Last we need to add a static route to tell the system that R2’s LAN network should be sent out tun0 interface.
interface-route 172.16.1.0/24 { next-hop-interface tun0 { } }
At this point our GRE tunnel should work, but
ubnt@R1:~$ ping 172.16.1.1 PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data. 64 bytes from 172.16.1.1: icmp_req=1 ttl=64 time=0.832 ms 64 bytes from 172.16.1.1: icmp_req=2 ttl=64 time=0.741 ms 64 bytes from 172.16.1.1: icmp_req=4 ttl=64 time=0.635 ms 64 bytes from 172.16.1.1: icmp_req=5 ttl=64 time=0.654 ms 64 bytes from 172.16.1.1: icmp_req=6 ttl=64 time=0.644 ms 64 bytes from 172.16.1.1: icmp_req=7 ttl=64 time=0.614 ms ^C --- 172.16.1.1 ping statistics --- 8 packets transmitted, 6 received, 25% packet loss, time 7006ms rtt min/avg/max/mdev = 0.614/0.686/0.832/0.082 ms
So it kind of works, but 25% packet loss is not good. One theory is that since we have 2 Internet connection, that some packets are going out the wrong interface. To test that we’ll take down the 2nd WAN interface.
ubnt@R1:~$ disconnect interface pppoe0 Bringing interface pppoe0 down... ubnt@R1:~$ ubnt@R1:~$ ubnt@R1:~$ ping 172.16.1.1 PING 172.16.1.1 (172.16.1.1) 56(84) bytes of data. 64 bytes from 172.16.1.1: icmp_req=1 ttl=64 time=0.789 ms 64 bytes from 172.16.1.1: icmp_req=2 ttl=64 time=0.606 ms 64 bytes from 172.16.1.1: icmp_req=3 ttl=64 time=0.585 ms 64 bytes from 172.16.1.1: icmp_req=4 ttl=64 time=0.667 ms 64 bytes from 172.16.1.1: icmp_req=5 ttl=64 time=0.650 ms 64 bytes from 172.16.1.1: icmp_req=6 ttl=64 time=0.707 ms 64 bytes from 172.16.1.1: icmp_req=7 ttl=64 time=0.752 ms 64 bytes from 172.16.1.1: icmp_req=8 ttl=64 time=0.613 ms ^C --- 172.16.1.1 ping statistics --- 8 packets transmitted, 8 received, 0% packet loss, time 7007ms rtt min/avg/max/mdev = 0.585/0.671/0.789/0.069 ms
Problem solved! But how do we use the 2nd WAN. One way is to add a static route for the far GRE end-point and force it to go out eth0.
route 30.0.0.2/32 { next-hop 20.0.0.1 { } }
Now we can bring back up the pppoe0 interface and still the GRE tunnel works fine.
The full configurations for router R1 and R2 for this part of the KB are:
Load-Balancing
Now we’ll add WAN load-balance to R1. Start with the load-balance section:
load-balance { group WLB { interface eth0 { } interface pppoe0 { } } }
This is all we really need, but we’re going to change the ping target to an address so that the health check doesn’t need DNS.
load-balance { group WLB { interface eth0 { route-test { type { ping { target 8.8.8.8 } } } } interface pppoe0 { route-test { type { ping { target 8.8.8.8 } } } } } }
Then we’ll add a firewall modify rule
modify BALANCE { rule 10 { action modify description "Do not load-balance LAN to LAN traffic" destination { address 192.168.1.0/24 } modify { table main } } rule 20 { action modify description "load-balance the rest of LAN to WAN traffic" modify { lb-group WLB } } }
Then apply the modify rule to the LAN “in”.
ethernet eth1 { address 192.168.1.1/24 description LAN firewall { in { modify BALANCE } } }
At this point our LAN traffic is being balanced over both WAN’s fairly evenly:
ubnt@R1:~$ show load-balance status Group WLB interface : eth0 carrier : up status : active gateway : 20.0.0.1 weight : 50 flows WAN Out : 682 WAN In : 0 Local Out : 20 interface : pppoe0 carrier : up status : active gateway : pppoe0 weight : 50 flows WAN Out : 672 WAN In : 0 Local Out : 22
But our GRE tunnel now doesn’t work. The first problem is that we really don’t want to load-balance the LAN traffic that is supposed to go over the GRE tunnel. So we’ll add a rule before the lb-group rule like:
ubnt@R1# show firewall modify modify BALANCE { rule 10 { action modify description "Do not load-balance LAN to LAN traffic" destination { address 192.168.1.0/24 } modify { table main } } rule 20 { action modify description "Do not load-balance traffic for gre tunnel" destination { address 172.16.1.0/24 } modify { table main } } rule 30 { action modify description "load-balance the rest of LAN to WAN traffic" modify { lb-group WLB } } }
This is mostly working now, but sometimes I need to take down the pppoe interface for the GRE tunnel to come up correctly. The next issue is related to how the load-balance feature is currently implemented with a separate routing table per WAN interface that only has a default route for that WAN. This works fine for LAN to WAN sessions, but breaks when we need a connected route from the “main” routing table. So the current work around is to instead of using the system generated separate routing table, instead create your own that has the default route for that WAN and the connected routes that are needed.
ubnt@R1# show protocols static table table 1 { interface-route 20.0.0.0/30 { next-hop-interface eth0 { } } interface-route 192.168.1.0/24 { next-hop-interface eth1 { } } route 0.0.0.0/0 { next-hop 20.0.0.1 { } } } table 2 { interface-route 0.0.0.0/0 { next-hop-interface pppoe0 { } } interface-route 20.0.0.0/30 { next-hop-interface eth0 { } } interface-route 192.168.1.0/24 { next-hop-interface eth1 { } } }
And then change the load-balance config to use the over-ride routing table.
ubnt@R1# show load-balance group WLB { interface eth0 { route { table 1 } route-test { type { ping { target 8.8.8.8 } } } } interface pppoe0 { route { table 2 } route-test { type { ping { target 8.8.8.8 } } } } }
The full config file for R1 with GRE and WAN load-balance is:
R1 /config/config.boot GRE with WAN load-balance
Below is the example config files for an IPSec tunnel between R1 and R2 while R1 has WAN load-balancing configured.
R1 /config/config.boot IPSec to R2 with WAN load-balance
R2 /config/config.boot IPSec to R1