Shortcut Forwarding Engine -------------------------- Welcome to "Shortcut" :-) Here's a quick FAQ: Q) What is Shortcut? A) Shortcut is an in-Linux-kernel IP packet forwarding engine. It's designed to offer very high speed IP packet forwarding based on IP connection tracking. It's dramatically faster than the standard netfilter-based NAT forwarding path but is designed to synchronise state back to netfilter/conntrack so that it doesn't need to deal with all of the complexities of special cases. Q) What versions of IP does it support? A) The current version only supports IPv4 but will be extended to support IPv6 in the future. Q) What transport protocols does it support? A) TCP and UDP. It also knows enough about ICMP to spot ICMP error messages related to TCP and UDP and handle things accordingly. Q) Is there a design spec for this software? A) Not at the moment. I'll write one when I get more time. The code is intended to be a good tutorial though - it's very heavily commented. If you find yourself reading something and not understanding it then I take that to mean I've probably not done a sufficently good job of explaining what it's doing in the comments. Let me know - I will try to fix it :-) Q) Why was it written? A) It was written as a demonstration of what can be done to provide high performance forwarding inside the kernel. There were two initial motivations: 1) To provide a platform to enable research into how QoS analysis systems can offload work and avoid huge Linux overheads. 2) To provide a tool to investigate the behaviour of various processors, SoCs and software sets so that we can characterize and design new network processor SoCs. Q) How much faster is it than the Linux kernel forwarding path? A) At the time of pushing this to github it's been tested on a QCA AP135. This has a Scorpion (QCA Scopion, not the QMC one :-)) SoC, QCA9550. The SoC's processor is a MIPS74K running at 720 MHz and with a DDR2 memory subsystem that offers a peak of 600 MT/s (16-bit transfers). Running IPv4 NAT forwarding of UDP between the board's 2 GMAC ports and using a SmartBits 200 as a traffic generator Linux is able to forward 70k PPS. Once the SFE code is invoked this will increase to 350k PPS! There's also a slightly hacky mode which causes SFE to bypass the Linux bridge layer, but this isn't really ready for use because it doesn't have sufficient MAC address checks or integration of statistics back to the Ethernet bridge, but that runs at 436k PPS. Q) Are there any diagnostics? A) Yes, this is a research tool after all! There's a complex way to do this that's more general purpose and a simple one - here's the simple one: mknod /dev/sfe c 253 0 The file /dev/sfe is an XML-ish output and provides details of all the network connections currently being offloaded. It also reports the numbers of packets that took various "exception" paths within the code. In addition it provides a summary of the number of connections, attempts to accelerate connections, cancel accelerations, etc. It also reports the numbers of packets that were forwarded and not forwarded by the engine and has some stats on the effectiveness of the hashing algorithm it uses. Q) How does the code interact with Linux? A) There are four minor patches required to make this software run with Linux. These are currently against a 3.3.8 or 3.4.0 kernel: * (net/core/dev.c) adds a hook to allow packets to be extracted out. * (net/netfilter/nf_conntrack_proto_tcp.c) exposes a state variable inside netfilter that's necessary to enable TCP sequence and ACK checking within the offload path. Note that this specific patch is against the QCA QSDK patched version of 3.3.8 - there's a slightly braindead "performance" patch in that kernel, courtesy of the OpenWrt community that makes the Linux forwarding path slightly faster at the expense of losing functionality :-( * (net/Kconfig) adds the shortcut-fe option. * (net/Makefile) adds the shortcut-fe build support. Once these are applied and the module is loaded then everything else is automatic :-) The patches are in this git repo. Q) Are any of the pieces reused from other projects? A) Yes! Some of the forwarding concepts are reused from the Ubicom Network Accelerator that morphed into part of the Akronite NSS. This code has all been substantially changed though to accomodate Linux's needs. There are also some pieces that I borrowed from the QCA "FastNAT" software written by Xiaoping Fan . Xiaoping's code was the first actual demonstration within QCA that this in-kernel concept could yield signficant performance gains. Enjoy! Dave Hudson