IPFW NG (WIP)

This is a page for work-in-progress ipfw modernization project, IPFW NetGraph (or Next Generation if you insist, but that words are worn out). This page is more developer-oriented, but please keep in mind that CLI interface must be user-friendly (being more user-friendly at that level is oen of the goals).

Our ipfw(4) is good, but outdated a lot. It's a crazy mix of shiny blocks, brilliant ideas and ancient squeaky farts. It's opcode-based structure, amongst some other cool things, has very great potential - it theory - which often stays still unimplemented in practice for many years. And everyone who had to manage rulesets of thousands rules knows it is manageable, but requires effort. In fact, in some cases even half-a-hundred-rules setup for doing something non-trivial and complex could be very hard to grok - imagine e.g. a NAT for several ext channels with “hairpinning” port redirects (from LAN to gw's ext addr so then again to LAN).

So this drives the motivation to do ipfw's… rewrite? No. Rewriting anything from scratch is almost always a very bad idea and loss of userbase accustomed to previous syntax etc. and angry due to new bugs. So this will be rework, preserving as many backwards compatibility for users (POLA) as possible. Given that ipfw already has many good ideas in it's internals, this is doable.

The first thing that simplifies ruleset management and was demanded by users, was ability to support multiple rulesets. That is, a thing like Cisco ACLs - one can have a separate ruleset for every interface/direction (in or out). Something slightly more good is a thing like pf anchors or iptables “chains” - when one can call a chain like a subroutine and then return from it. I've already implemented ipfw call/ipfw return actions (available in 8.3R and 9.0R), but that was just a hack very quick to implement, a stub for those who need at least something to ease maintainability immediately. A more complete consistent solution must be created.

The second big thing in ipfw which is inadequate to current needs is dynamic rules. But they still have to be rewritten just to support multiple rulesets, as it will be shown below.

To be generic and Right Thing™, this is a HUGE work, because we need to implement the following:

• each ipfw chain becomes independent netgraph(4) node
• generic ng_pfil node usable not only for firewalls
• chains may be called from each other (see iptables)
• chains (actually netgraph nodes) may be bind to ifaces or any other place
• main unnamed chain is called from pfil(9) as before
• rewrite ipfw & dummynet management from setsockopt() to netgraph messages
• completely rewrite ipfw dynamic rules to not conflict with multiple rulesets, as now they just jump to parent static rule (need to be more like pf or iptables states). This item is hard but essential (you'll get a mess jumping to another ruleset), and ipfw contexts don't handle it
• while here, do other needed things, e.g. adding support for modules in both kernel and userland, loadable opcodes, keywords, etc.
• what yet? may be generic instructions and “registers” like in bpf(4) to support arythmetics etc. in rules? like e.g. denying packets every other second; or begin to shape a connection after a first 100Rb of traffic

ipfw chain mychain create
ipfw chain common create default-policy allow
ipfw chain lanusers create
ipfw chain mychain add 200 ...
ipfw chain mychain delete 300 ...
ipfw chain mychain destroy
ipfw add 100 call common ip from any to any   # call from main ruleset
ipfw ng_bpf mypktbody create
ipfw ng_bpf mypktbody config proto udp and udp[42:4]=0x01020304
ipfw add 200 deny ngmatch mypktbody
ipfw bind mychain to interface em0 direction out
ipfw bind lanusers to interface vlan100
ipfw pfil-order move ipfilter bottom

Hope that's enough user-intuitive. And netgraph-based design here is very straightforward: for examples above, you just create netgraph nodes named “ipfwmychain”, “ipfwcommon”, etc., then do send NGM_IPFW_ADD message to the node with name of appropriate “chain” keyword, or main ng_ipfw (“ipfw:”) if none.

Netgraph here is not a stack, but a clean alternative to hack raw_ip.c for new setsockopt()'s. Currently, ipfw is not very modular - to implement a new command, you need to introduce new socket option and put into a file totally unrelated to ipfw. And support part's of ipfw code there - just to check if module is loaded, for example. Or invent a home-grown protocol to multiplex all new commands into one optname. And try to support previous versions (it was hard to keep compatible interfaces when ABI changed in 8.2)… all of that sounds not hard and not important, but still ipfw's control is nowadays a total mess (remember at least ipfw nat broken buffer? it was broken for several major releases…).

And in Netgraph all that problems are already solved in a clean way (netgraph messages do provide clean versioning, etc.). As a bonus, advanced user is able to manually assign node (ruleset lives in node) in complex configuration to some non-typical point (usually the /sbin/ipfw will do). In fact, Netgraph main use here is to provide something like Linux' Netlink alternative for us.

NB: For typical uses all Netgraph complexity must be hidden from user behind the habitual /sbin/ipfw. This must follow the mpd's model - you may know something about Netgraph, but you don't have to (as a consequence, mpd is widely used, but Netgraph itself, the Netgraph-scary-to-user, is not). The system, however, should be transparent to advanced user knowing what he is doing.

The main reason is that it is just a pointer to static rule:

• suppose you have router with interfaces X, Y.
• you have TCP/UDP session from A to B, incoming on X and leaving through Y
• you have per-interface rulesets, one for X on input and one for Y

Then, if a packet entered via iface X, and there is a state (dynamic rule) there, then, whatever you write in rules for Y - the first keep-state will accept packet, no matter what rule, it will be accepted.

You can try to expoit the fact that in ipfw, dynamic rule just jumps to parent rule's action part, and, if that was “skipto”, continues on a static ruleset (where it could be checked, which interface).

What do I mean here by “jump to parent's rule's action part” ? Suppose your router have channels to two ISPs and you want to emulate pf's reply-to behaviour. It is not obvious from man page, but if you read sources and grok how dynrules in ipfw2 work, you can do it like this:

ipfw add 100 skipto 300 tag 1 in recv $ext_if1 keep-state
ipfw add 200 skipto 300 tag 2 in recv $ext_if2 keep-state
ipfw add 300 allow { recv $ext_if1 or recv $ext_if2 }      # incoming from world
ipfw add 400 allow in recv $int_if                         # allow replies on internal pass
ipfw add 500 fwd $gw1 tagged 1                       # the same replies are now on out-
ipfw add 600 fwd $gw2 tagged 2                       # going pass, fwd them where appropriate

Here first packet from outside via ext_if2 (note if2!) will create a state (dynrule) at rule 200, then there will come reply from our LAN “in via $int_if”. On rule 100 there will be implicit check-state (remember if2? now it looks like triggered on if1, but that's just implicit check-state) which directs packet to action part of parent rule, which is in rule 200, and it is “skipto 300 tag 2”. Then processing continues as if there were no dynamic rules, and goes via rules 300-600. Thus every dynamic rule here is like a subroutine with rules 300-600. And a channel to ISP - where to direct a packet - differentiated only by the tag.

Looks like magic? Hard to grok? Hard to write? Hard to debug? Crying “Gods, how pf's reply-to is dozens time simplier”? That's how today's ipfw looks like. Things are doable, but…

But this becomes unmaintainable mess even on 3-4 interfaces (it's like assembly language programming), and if you have hundreds of ifaces, you may encounter there is not enough numbers in 65536 to handle all X*Y combinations (not all ifaces need be stateful) even if config is somehow generated automatically. There is at least one company in real life which really comes up to such situation.

And now… Recall you have several rulesets, one per interface (X and Y), and have just a usual ipfw2's dynrules with a skipto. A skipto somewhere in ruleset for X, and packet in opposite direction (from B to A) meets a check-state in another ruleset for Y. And then suddenly jumps to… first ruleset for X. Oooops.

That's why current dynamic rules is absolutely not suited for multiple rulesets and must be rewritten. The other reason is to simplify other things, six rules in the example above just for one reply-to are too complex. Want a cleaner solution.

Really, ipfw and many other components of network subsytem suffer from the fact each component is forced to do it's own connection state tracking, e.g. NAT, dynamic rules, Netflow, all are separate. This can be good sometimes, but:

• this consumes memory
• it's a PITA when you try to combine them, e.g. ipfw nat + keep-state (dynamic rules) can cause kernel panics, infinite recursion, etc.

We need unified state tracking in our firewall which will do all needed operations, it will be like Linux' conntrack or pf's states, in that it must:

• support NAT and redirects natively, including NAT64 (RFC 6146)
• be designed the way to support “ipfwsync” (a-la pfsync)
• be able to be manipulated well and to support related connections: e.g. dynamic rule for ftp-data connection will be related to main dynamic rule, or another subsystem is able to insert a dynrule in EXPECTED state for yet not started connection (useful e.g. for P2P tracking).

We need loadable modules able to implement loadable opcodes. For example, leave first 128 opcodes for built-in functions (present nowadays) and other 128 for dynamically registered modules. Currently there is only one module, ipfw_nat, and it is implemented as a hack. DIFFUSE is another example, it tried better but still not in the tree. In fact, having a framework for modules will encourage people to write them, DIFFUSE may have been done and committed much earlier if we had a framework, who knows…

Another application of modules is utilizing hooks in dynamic rules. E.g. a helper for tracking FTP to allow data connections.

Totally a mess now, but we can't rewrite it to lex/yacc because all historical quirks of syntax do not fit into clean grammar. Given that modules are wanted, the idea should be borrowed from GEOM - a userland module for each kernel module, consistent keywords in the beginning of command line with passing rest of it to module, etc.

More human-friendly and morer readable? Let's introduce a directory /etc/ipfw where files, if present, will allow to control behaviour. E.g. we can have /etc/ipfw/tables listing pairs number and name, just like /etc/protocols do (idea stolen from route tables naming in file in iproute2). Benefits:

• One can have any comfortable mapping of strings to table (e.g. unique client contract ID to table) without wasting space for string names in kernel memory.
• Machine-friendly: pf have names, but compiles them to internal numbers on every reload, and this is not visible to other subsystem. Keeping numbers in file still allows everything in the system use the same numbers (let's be open and extensible).

Example:

# cat /etc/ipfw/tables
1       badguys
2       client-2000421-3000454
# ipfw list
00001 deny ip from table(badguys) to any
00002 nat ip from table(client-2000421-3000454) to any out via em0
00003 pipe 1 ip from table(3) to any in

Here two tables are named and will be recognized in commands like ipfw add deny ip from table(badguys) to any, and table 3 is not named and shown as is, like in previous versions (POLA, don't use if you don't want, etc.).

Another thing to be present in the parser is a macro/aliases file to be expanded before actual command processing. The obvious use will be for things like regexps:

^all$ -> ip4 from any to any

to be able to write e.g.

ipfw add block all

instead of traditional string. Of course, that's trivial, the more useful application is to support older syntax. For example, suppose new dynrules NAT engine will be called by keywords nat44 and nat64 (for traditional IPv4-to-IPv4 NAT and for IPv6-toIPv4 NAT, respectively). And older libalias-based ipfw nat will be renamed to libalias (“ipfw add libalias ip from $lan to any”). But /etc/ipfw/preparse for two major releases will contain lines:

nat -> libalias
sho -> show
sh -> show

and all older scripts for ipfw nat will still work with libalias as before - POLA preserved. And you'll no longer get an annoying message:

# ipfw sh
ipfw: DEPRECATED: 'sh' matched 'show' as a sub-string

but could also define your own handy aliases.

Simple substitution is of little use, of course, although the “'sh' matched 'show'” was marked deprecated because it's one of the things making /sbin/ipfw parser code the big mess and yet another obstackle on the way to convert parser to clean lex/yacc parsing. The goal is to be able to rewrite /sbin/ipfw to speak only new syntax, and support as much of older syntax quirks as possible via this preparsing mechanism. Consider the following old syntax:

add 60001 deny { tcp or udp } from { table(3) or me } 80,443 to { 127.0.0.1 or 127.0.0.2 or 127.0.0.3 } established

It is old, it is messy to process in parser, it regularly show bugs in GNATS with various corner cases. It should really die, all these “proto from X to Y”, in favor of more clean regular grammar of just an options list (section “RULE OPTIONS (MATCH PATTERNS)” in man page). But we can't let it just die because of backward compatibility and users accustomed to that syntax. A more complex preparser, may be with several sections in lex style, may be provided to deal with some subset of that. It is still questionable, which part and how exactly, but something like this should be done.

TO BE CONTINUED