Date: Wed, 5 Sep 2012 16:15:39 -0400 From: George Neville-Neil <gnn@neville-neil.com> To: Anuranjan Shukla <anshukla@juniper.net> Cc: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Re: Proposal for changes to network device drivers and network stack (RFC) Message-ID: <5F3C03B6-01D0-42DE-BE9E-323DBDC90C8E@neville-neil.com> In-Reply-To: <CC5C760E.18207%anshukla@juniper.net> References: <CC5C760E.18207%anshukla@juniper.net>
next in thread | previous in thread | raw e-mail | index | archive | help
On Aug 25, 2012, at 00:11 , Anuranjan Shukla <anshukla@juniper.net> wrote:
> At Juniper Networks, we've been using FreeBSD to build JUNOS (Juniper's
> network operating system). So far the additions and changes to the
> functionality were made inline, making the task of upgrading to new
> versions of FreeBSD progressively difficult. We've been looking at JUNOS
> to see if we can build it off of a clean FreeBSD base rather than making
> changes to the OS inline. As part of that work, we've come up with a few
> expansive change proposals to FreeBSD kernel that will make this task
> possible for us, and hopefully also contribute something of interest to
> the community. If the community is in agreement with these, we'd like to
> contribute/commit them to FreeBSD.
>
> This is a proposal and an RFC. The actual nomenclature is open to ideas
> (naming etc). From Juniper, Marcel (marcel@freebsd.org) will be attending
> the upcoming DevSummit at Cambridge. He's indicated that interested folks
> are welcome to chat with him about this stuff during the summit.
>
Hello Anu,
Yes, a bunch of this was discussed at the DevSummit, and I think there is
already some agreement about these proposals, at least there was
in the room, now to get some on net@.
> The changes we propose are (the code/diffs etc are indicated
> at the end of this email):
>
> - Network Device Drivers
> - Building FreeBSD kernel without network stack, network stack as a module
> - Changes to mbuf and socket structures (minor member additions)
>
> Network Device Drivers:
> -----------------------
> As we indicated during DevSummit 2012, JUNOS extended the interface
> functionality in a big way to support logical interfaces, interface
> hierarchies and scaling in general. Not surprisingly this resulted in
> changing the drivers to use our custom interface structure(s). A simple
> way to resolve this without impacting the rest of the large codebase is to
> avoid directly accessing (get/set) the ifnet structure from the drivers.
> Using get/set functions to update the functionality would make the driver
> more 'flexible' for the network stack to work with in situations where the
> stack wants to extend the interface functionality.
>
> For eg,
>
> em_start_locked(struct ifnet *ifp, struct tx_ring *txr)
> {
> - struct adapter *adapter = ifp->if_softc;
> + struct adapter *adapter = if_getsoftc(ifp);
> struct mbuf *m_head;
>
> EM_TX_LOCK_ASSERT(txr);
>
> - if ((ifp->if_drv_flags & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) !=
> + if ((if_getdrvflags(ifp) & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) !=
> IFF_DRV_RUNNING)
> return;
>
> if (!adapter->link_active)
> return;
>
> - while (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) {
> + while (!if_sendq_empty(ifp)) {
> /* Call cleanup if number of TX descriptors low */
> if (txr->tx_avail <= EM_TX_CLEANUP_THRESHOLD)
> em_txeof(txr);
> if (txr->tx_avail < EM_MAX_SCATTER) {
> - ifp->if_drv_flags |= IFF_DRV_OACTIVE;
> + if_setdrvflagbits(ifp,IFF_DRV_OACTIVE, 0);
> break;
> }
> - IFQ_DRV_DEQUEUE(&ifp->if_snd, m_head);
> + m_head = if_dequeue(ifp);
> if (m_head == NULL)
> break;
> /*
> @@ -1010,7 +1009,7 @@ em_start_locked(struct ifnet *ifp, struct tx_ring
> if (em_xmit(txr, &m_head)) {
> if (m_head == NULL)
> break;
> - IFQ_DRV_PREPEND(&ifp->if_snd, m_head);
> + if_sendq_prepend(ifp, m_head);
> break;
>
> This allows Juniper to have its own interface structure(s) instead of
> ifnet, and still be able to use the driver without modification. Since the
> notion of ifnet is abstracted away, other users can also find this useful
> in plugging in functionality without having muck around in the driver code.
>
> The ifnet split/restructuring was discussed in DevSummit at BSDCan in May
> 2012. This change can also aid in that work.
>
> This change can be applied to drivers in a phased way. Clearly, it won't
> have any impact on drivers that haven't been changed. At Juniper we're
> planning on converting em,fxp and tsec. Are there any strong feelings on
> whether the phase-wise change is ok or not?
>
>
I think these changes might be aided by something that bz@ and I talked
about after the network session, to wit, we could stand to abstract a good
deal of code away from the drivers, and up into the ethernet and other
modules. I think trying to reduce the amount of code in the drivers,
much of which is common code, is the way for us to go. As part
of that we can start to use the functions that you mention.
> Building FreeBSD without the network stack (network stack as a module)
> ----------------------------------------------------------------------
> Today, not compiling networking stack related files in the kernel breaks
> the kernel build due to dependencies the OS has on the network stack
> (calling into functions in the network stack). Network stack module isn't
> there. We've added these in JUNOS. The benefits for us are obvious (we can
> load our own version of network stack if we desire!), but most likely this
> functionality will benefit others too.
>
> The detailed implementation is indicated later in this email. In short the
> changes are:
>
> - Load network stack as a module. For now via loader, not dynamically
> loaded. (Is there interest in dynamic loading?).
> - To facilitate calling network stack functionality from the generic
> kernel, a new interface has been defined with the kobj framework.
> - Some files and tunables needed to move to generic kernel areas (accept
> filters)
> - Generic socket code calls into network stack for interface and route
> related ioctls. Network stack now registers these ioctl groups
> - Other changes: uuid generation (register/query uuid sources), fib/sctp
> system calls (moved to network stack code, with system calls register
> dynamically).
>
This would be interesting for many reasons, and I think it would be a good
contribution. Does the work you've done in this area handle the VNET
stuff that is in the stack as well? That is, how well does the network stack
as a module play with the vnet architecture?
> Changes to mbuf and socket structures
> --------------------------------------
> A couple additions to these structures help JUNOS incorporate cool things
> like interface/route indices and logical routing. For us the diff today
> looks
> something like:
>
> struct pkthdr {
> + uint32_t rcvidx; /* rcv interface index */
> + uint32_t rnhidx; /* route or nexthop index */
> struct ifnet *rcvif; /* rcv interface */
>
These seem straightforward, but, of course must go into a .0 release.
> struct socket {
>
> int so_fibnum; /* routing domain for this socket */
> uint32_t so_user_cookie;
> + u_int so_oqueue; /* manage send prioritizing based on application
> needs */
> + u_short so_lrid; /* logical routing */
> };
>
I'd be interested to know how this is used.
> A question for the community is if it's there're strong objections to
> adding fields to these structures. If so, do we have another way or a
> suggestion to consider?
>
So long as you're happy living with them in a .0 that's, I believe,
fine.
> For a detailed look, diffs can be found at:
> http://people.freebsd.org/~marcel/Juniper/ddi-mbuf-socket.diff
> http://people.freebsd.org/~marcel/Juniper/netstack.diff
>
> These will provide a good idea of the changes. They're not in final shape
> yet.
>
OK, I think we should keep this moving along.
Best,
George
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?5F3C03B6-01D0-42DE-BE9E-323DBDC90C8E>