From owner-freebsd-net@FreeBSD.ORG Sat Aug 25 04:12:39 2012 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id D360E106564A for ; Sat, 25 Aug 2012 04:12:39 +0000 (UTC) (envelope-from anshukla@juniper.net) Received: from exprod7og123.obsmtp.com (exprod7og123.obsmtp.com [64.18.2.24]) by mx1.freebsd.org (Postfix) with ESMTP id 7F8308FC12 for ; Sat, 25 Aug 2012 04:12:39 +0000 (UTC) Received: from P-EMHUB02-HQ.jnpr.net ([66.129.224.36]) (using TLSv1) by exprod7ob123.postini.com ([64.18.6.12]) with SMTP ID DSNKUDhQsdHZod7srxXKiARYD5x5L5xlpoKh@postini.com; Fri, 24 Aug 2012 21:12:39 PDT Received: from EMBX02-HQ.jnpr.net ([fe80::18fe:d666:b43e:f97e]) by P-EMHUB02-HQ.jnpr.net ([fe80::88f9:77fd:dfc:4d51%11]) with mapi; Fri, 24 Aug 2012 21:11:48 -0700 From: Anuranjan Shukla To: "freebsd-net@freebsd.org" Date: Fri, 24 Aug 2012 21:11:45 -0700 Thread-Topic: Proposal for changes to network device drivers and network stack (RFC) Thread-Index: Ac2Cd76xUaKMoSnfRb6WxYbFKGyXpA== Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: user-agent: Microsoft-MacOutlook/14.2.3.120616 acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Proposal for changes to network device drivers and network stack (RFC) X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Aug 2012 04:12:39 -0000 At Juniper Networks, we've been using FreeBSD to build JUNOS (Juniper's network operating system). So far the additions and changes to the functionality were made inline, making the task of upgrading to new versions of FreeBSD progressively difficult. We've been looking at JUNOS to see if we can build it off of a clean FreeBSD base rather than making changes to the OS inline. As part of that work, we've come up with a few expansive change proposals to FreeBSD kernel that will make this task possible for us, and hopefully also contribute something of interest to the community. If the community is in agreement with these, we'd like to contribute/commit them to FreeBSD. This is a proposal and an RFC. The actual nomenclature is open to ideas (naming etc). From Juniper, Marcel (marcel@freebsd.org) will be attending the upcoming DevSummit at Cambridge. He's indicated that interested folks are welcome to chat with him about this stuff during the summit. The changes we propose are (the code/diffs etc are indicated at the end of this email): - Network Device Drivers - Building FreeBSD kernel without network stack, network stack as a module - Changes to mbuf and socket structures (minor member additions) Network Device Drivers: ----------------------- As we indicated during DevSummit 2012, JUNOS extended the interface functionality in a big way to support logical interfaces, interface hierarchies and scaling in general. Not surprisingly this resulted in changing the drivers to use our custom interface structure(s). A simple way to resolve this without impacting the rest of the large codebase is to avoid directly accessing (get/set) the ifnet structure from the drivers. Using get/set functions to update the functionality would make the driver more 'flexible' for the network stack to work with in situations where the stack wants to extend the interface functionality. For eg, em_start_locked(struct ifnet *ifp, struct tx_ring *txr) { - struct adapter *adapter =3D ifp->if_softc; + struct adapter *adapter =3D if_getsoftc(ifp); struct mbuf *m_head; =20 EM_TX_LOCK_ASSERT(txr); =20 - if ((ifp->if_drv_flags & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) !=3D + if ((if_getdrvflags(ifp) & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) !=3D IFF_DRV_RUNNING) return; =20 if (!adapter->link_active) return; =20 - while (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) { + while (!if_sendq_empty(ifp)) { /* Call cleanup if number of TX descriptors low */ if (txr->tx_avail <=3D EM_TX_CLEANUP_THRESHOLD) em_txeof(txr); if (txr->tx_avail < EM_MAX_SCATTER) { - ifp->if_drv_flags |=3D IFF_DRV_OACTIVE; + if_setdrvflagbits(ifp,IFF_DRV_OACTIVE, 0); break; } - IFQ_DRV_DEQUEUE(&ifp->if_snd, m_head); + m_head =3D if_dequeue(ifp); if (m_head =3D=3D NULL) break; /* @@ -1010,7 +1009,7 @@ em_start_locked(struct ifnet *ifp, struct tx_ring if (em_xmit(txr, &m_head)) { if (m_head =3D=3D NULL) break; - IFQ_DRV_PREPEND(&ifp->if_snd, m_head); + if_sendq_prepend(ifp, m_head); break; This allows Juniper to have its own interface structure(s) instead of ifnet, and still be able to use the driver without modification. Since the notion of ifnet is abstracted away, other users can also find this useful in plugging in functionality without having muck around in the driver code. The ifnet split/restructuring was discussed in DevSummit at BSDCan in May 2012. This change can also aid in that work. This change can be applied to drivers in a phased way. Clearly, it won't have any impact on drivers that haven't been changed. At Juniper we're planning on converting em,fxp and tsec. Are there any strong feelings on whether the phase-wise change is ok or not? Building FreeBSD without the network stack (network stack as a module) ---------------------------------------------------------------------- Today, not compiling networking stack related files in the kernel breaks the kernel build due to dependencies the OS has on the network stack (calling into functions in the network stack). Network stack module isn't there. We've added these in JUNOS. The benefits for us are obvious (we can load our own version of network stack if we desire!), but most likely this functionality will benefit others too. The detailed implementation is indicated later in this email. In short the changes are: - Load network stack as a module. For now via loader, not dynamically loaded. (Is there interest in dynamic loading?). - To facilitate calling network stack functionality from the generic kernel, a new interface has been defined with the kobj framework. - Some files and tunables needed to move to generic kernel areas (accept filters) - Generic socket code calls into network stack for interface and route related ioctls. Network stack now registers these ioctl groups - Other changes: uuid generation (register/query uuid sources), fib/sctp system calls (moved to network stack code, with system calls register dynamically). Changes to mbuf and socket structures -------------------------------------- A couple additions to these structures help JUNOS incorporate cool things like interface/route indices and logical routing. For us the diff today looks=20 something like: struct pkthdr { + uint32_t rcvidx; /* rcv interface index */ + uint32_t rnhidx; /* route or nexthop index */ struct ifnet *rcvif; /* rcv interface */ struct socket { int so_fibnum; /* routing domain for this socket */ uint32_t so_user_cookie; + u_int so_oqueue; /* manage send prioritizing based on application needs */ + u_short so_lrid; /* logical routing */ }; A question for the community is if it's there're strong objections to adding fields to these structures. If so, do we have another way or a suggestion to consider? For a detailed look, diffs can be found at: http://people.freebsd.org/~marcel/Juniper/ddi-mbuf-socket.diff http://people.freebsd.org/~marcel/Juniper/netstack.diff These will provide a good idea of the changes. They're not in final shape yet.