Ubiquiti EdgeRouter 12 (ER-12) firewall configuration fun

Fun is very relative term. This setup task hits a pain point on the learning curve. I generally know what I want, but I don't know exactly how to execute the task on this new platform. There will be a bit frustration involved this process.

First thought. Phew, there's a lot's of stuff to configure. But funny thing was that while everything was still broken, and NAT and routing configuration was missing I decided for fun to check if dnskv.com works, and it did. This is a good example how it can in some cases help, even if when everything else is blocked or broken. Routing, forwarding and firewall(s) aren't yet configured at all, but DNS still works, because it uses different system routing.

Next LAN and WAN configured, then NAT and firewall. Ok. Nice, it hanged. Probably due to firewall blocking access to the management console, it happens. Factory reset. This is exactly why it's a good idea to play with the device(s), you're installing a few weeks before production use, so all the problems and snags you'll be finding won't affect production environments. As well as you can try configurations which do break stuff, just to see what will happen. - Tomorrow, let's retry with different config and approach.

VPN configuration is also super sketchy. UI is lacking tons of features and options. Where the VPN tunnel terminates is also very unclear. All this requires setup and testing. Or just accepting lot of downtime and randomization ins production to see, if it happens to work or not.

Now LAN networks are configured, and firewall rules between those are working. I'll probably won't even bother with IPv6 because it would take so much extra time to configure it.

IPsec and 6in4 tunnel configuration is on going, let's see how that works out.

IPsec tunnels, lot's of tuning. It was pretty easy to setup the tunnels. But the firewall is especially confusing about this. With Zyxel IPsec VPNs got it's own interfaces. In this case, by default there's no interface for VPNs. Let's see how we can firewall the tunnels. It seems that by default there's "full trust" to all tunnels, which is naturally very bad approach.

Found reason for that, but it seems that fixing the issue takes lots of reconfiguration. The way the IPsec tunnels work by default is kind funny from the firewall perspective. Because the packets never come in, but still go out through the firewall. I can block the outbound "return traffic" but I can't block the inbound traffic. Yet if packets can't go out, is it same as the packets couldn't get in? Practically almost, but not exactly.

Changed from tunnel mode to virtual interface mode. Now everything seems to be working fine, until the tunnel is opened. Then traffic mysteriously disappears from the virtual interface. Also the IPsec tunnel doesn't automatically start when there's traffic, even if

First thoughts about this. Wow, this must be slowest thing to configure ever. Documentation is non-existing. Configuration requires 10000 tests. First to test if everything is working and then testing everything that's not supposed to work, etc. Haven't seen this annoying devices for a while, yet this seems to be standard quality of many products. It's up to the customer to make it work. You'll need to read about 100 different discussions how things work, then read some source code and then conclude how to make things work like you want to.

Why referring to programming languages? Well, because I ended up reading source code of "vyatta-vti-config.pl" to get things to work. It seems that there's some more into the VTI than I would like to. I would love to use policy based VPN, but still route traffic through virtual interface. Wondering if such configuration is possible.

Enough time spent so far. I'll re-read all pages I've found about the IPsec VPN-setup, and if I don't find suitable answers then I'll ask in Ubiquiti forums. I thought about that, but after all I didn't need to do it. I figured out the way my-self.

It seems that after long investigation, there's especial option which needs to be set with firewall rules, so that the rules will be applied to the IPsec traffic. Need to try that later... I hope it'll resolve this. This was extremely long and from technical perspective painfully simple thing to deal with.

Configuring potential VTI interface was bit confusing, but after reading some documentation I figured out that the VTI isn't the solution to the matter I'm looking for. Also using the VTI requires that the remote end is also using VTI configuration. So it's not just something which can be locally switched on or off. Another slightly confusing thing about VTIs were that does the VTI interface local address matter at all? And how it's exactly linked to routing? Should I actually have a rule which will route traffic into the VTI and then the next ends VTI interface address is configured as the next-hop? Or what. I ignored this approach but those questions remained unanswered. (vti bind)

After abandoning the VTI approach, I found the very likely root cause of the traffic magically getting through: show vpn ipsec auto-firewall-nat-exclude.

After disabling that and analyzing everything in detail, the issue was finally found. After doing all the required steps and adding separate WAN IN and LAN_OUT IPSEC rules, things started to work out as planned. That's good. IPsec and firewalling is now sorted out. Next step is full IPv6 configuration. IPv6 tunnel was really easy to setup, and it worked almost immediately like a charm. But Next LAN addresses and routing + firewalls need to be configured.

There's no ipv6 firewall management in the GUI so everything needs to be done via CLI. Not a problem, just bit time consuming. Didn't bother yet after the initial testing that it does work.

When I talked a few guys who've configured bunch of Ubiquiti EdgeRouters and MicroTik routers, their pro tip was

Some of the configuration was so time consuming to enter, that I wrote Python 3 program, which configures the firewalls using SSH. Done. Now everything is pretty clear, rest is just minor fine tuning and finalizing the configuration. All basic and core concepts are now clear and most of the hard things solved. - This is the way I like working. First solve the hard things, because the easy things aren't worth of thinking, before hard ones are solved. Because, if the hard problems stop the project, then all the resources spent on easy problems are probably wasted.

Now everything's pretty good. Final fine tuning of firewall rules and such is still required. Mostly allowing specific production traffic in and out, after I've tested everything.

IPv6 not working / being hard, well... It turned out that after enjoying relaxed weekend, the problem was solved pretty quickly when staring at the configuration dump... Wait, what, what is that... Oh why why why. There's a typo in the IPv6 address assigned to local interface. Oh well... - Fixed, working. Still firewall rules need to be created manually, but now all the "magic" of the related problems is suddenly gone, and it's just not strangely broken and not working.

Finally even more firewall confusion. I asked around from several persons what does this mean, and nobody could explain why these are named like this. The usual situation, everything's so confusing, that to figure out how things work out, you'll end up doing extensive testing. Very bad way, but that's how it is with a lot of software. There's documentation, but it's bad and misleading. To truly know things, you'll need either read the source code or in case of proprietary software, do extensive testing and even then you can't be 100% sure what you're missing.

What's the difference between these options:
1) Don't match on IPsec packets
2) Match inbound IPsec packets
3) Match inbound non-IPsec packets

Ok, there's word inbound there, sure. But otherwise difference between those options. What if I just want to do generic firewall rule which doesn't give a bleep if it's IPsec or not? That should be the default option afaik. Also that's really confusing, because that actually means "from IPsec tunnel", not that the packet would be "IPsec packet". Does the option two "Match inbound IPsec packets" also match non-IPsec packets? - Oh why there isn't IPsec neutral option? - After some extra testing it turned out that the 1 was exactly what I'm looking for. But the wording is bit confusing. It basically means that it doesn't care if it's IPsec or not. So the match statement isn't being used at all. I figured out this by checking the firewall configuration data which is updated by the GUI and reading it's documentation.

I also asked above the questions where the VPN tunnels terminate, well, it seems that the VPN tunnels actually terminate / connect to the WAN interface.

When mentioning other annoying things. Version 2.0.9 firewall management doesn't support IPv6 at all in GUI. That's annoying, but not a show stopper. Another thing is that when you're rearranging firewall rules, the UI is really buggy. You'll need to move the rule. Then the UI freezes up, it does NOT get updated! Until you save the new order and then you'll see the changes you've made. So when you move rule 3 between rule 1 and 2, freeze, after save you'll see what you did. This also prevents you from moving more than one rule at a time. - Sometimes software and documentation is so bad, you'll start to think they're just teasing and trolling you. - Nothing new about that.

Don't do too much daily, so you won't get too annoyed. Next day I continued with finalizing the IPv6 configuration, and now it's also working and prefect.

Configuring and verifying the logging also took surprising amount of work. To test and see that everything which is required to be logged gets logged, and unnecessary stuff doesn't After initial configuration the logs grew at totally insane pace, until I started to reduce the amount of logging.

After all this tuning, the switch over went very very nicely. Just a few firewall rules needed fine tuning because the IPsec related options mentioned above were incorrect and traffic did hit interfaces default rule which is always drop. - Job done.

I'll probably still need to bit work with the logging settings, but that's the usual way of doing things. First too much logs, and then just go through the logs and see what's not necessary.

2021-02-21