Date: Fri, 24 Aug 2012 21:11:45 -0700 From: Anuranjan Shukla <anshukla@juniper.net> To: "freebsd-net@freebsd.org" <freebsd-net@freebsd.org> Subject: Proposal for changes to network device drivers and network stack (RFC) Message-ID: <CC5C760E.18207%anshukla@juniper.net>
next in thread | raw e-mail | index | archive | help
At Juniper Networks, we've been using FreeBSD to build JUNOS (Juniper's
network operating system). So far the additions and changes to the
functionality were made inline, making the task of upgrading to new
versions of FreeBSD progressively difficult. We've been looking at JUNOS
to see if we can build it off of a clean FreeBSD base rather than making
changes to the OS inline. As part of that work, we've come up with a few
expansive change proposals to FreeBSD kernel that will make this task
possible for us, and hopefully also contribute something of interest to
the community. If the community is in agreement with these, we'd like to
contribute/commit them to FreeBSD.
This is a proposal and an RFC. The actual nomenclature is open to ideas
(naming etc). From Juniper, Marcel (marcel@freebsd.org) will be attending
the upcoming DevSummit at Cambridge. He's indicated that interested folks
are welcome to chat with him about this stuff during the summit.
The changes we propose are (the code/diffs etc are indicated
at the end of this email):
- Network Device Drivers
- Building FreeBSD kernel without network stack, network stack as a module
- Changes to mbuf and socket structures (minor member additions)
Network Device Drivers:
-----------------------
As we indicated during DevSummit 2012, JUNOS extended the interface
functionality in a big way to support logical interfaces, interface
hierarchies and scaling in general. Not surprisingly this resulted in
changing the drivers to use our custom interface structure(s). A simple
way to resolve this without impacting the rest of the large codebase is to
avoid directly accessing (get/set) the ifnet structure from the drivers.
Using get/set functions to update the functionality would make the driver
more 'flexible' for the network stack to work with in situations where the
stack wants to extend the interface functionality.
For eg,
em_start_locked(struct ifnet *ifp, struct tx_ring *txr)
{
- struct adapter *adapter =3D ifp->if_softc;
+ struct adapter *adapter =3D if_getsoftc(ifp);
struct mbuf *m_head;
=20
EM_TX_LOCK_ASSERT(txr);
=20
- if ((ifp->if_drv_flags & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) !=3D
+ if ((if_getdrvflags(ifp) & (IFF_DRV_RUNNING|IFF_DRV_OACTIVE)) !=3D
IFF_DRV_RUNNING)
return;
=20
if (!adapter->link_active)
return;
=20
- while (!IFQ_DRV_IS_EMPTY(&ifp->if_snd)) {
+ while (!if_sendq_empty(ifp)) {
/* Call cleanup if number of TX descriptors low */
if (txr->tx_avail <=3D EM_TX_CLEANUP_THRESHOLD)
em_txeof(txr);
if (txr->tx_avail < EM_MAX_SCATTER) {
- ifp->if_drv_flags |=3D IFF_DRV_OACTIVE;
+ if_setdrvflagbits(ifp,IFF_DRV_OACTIVE, 0);
break;
}
- IFQ_DRV_DEQUEUE(&ifp->if_snd, m_head);
+ m_head =3D if_dequeue(ifp);
if (m_head =3D=3D NULL)
break;
/*
@@ -1010,7 +1009,7 @@ em_start_locked(struct ifnet *ifp, struct tx_ring
if (em_xmit(txr, &m_head)) {
if (m_head =3D=3D NULL)
break;
- IFQ_DRV_PREPEND(&ifp->if_snd, m_head);
+ if_sendq_prepend(ifp, m_head);
break;
This allows Juniper to have its own interface structure(s) instead of
ifnet, and still be able to use the driver without modification. Since the
notion of ifnet is abstracted away, other users can also find this useful
in plugging in functionality without having muck around in the driver code.
The ifnet split/restructuring was discussed in DevSummit at BSDCan in May
2012. This change can also aid in that work.
This change can be applied to drivers in a phased way. Clearly, it won't
have any impact on drivers that haven't been changed. At Juniper we're
planning on converting em,fxp and tsec. Are there any strong feelings on
whether the phase-wise change is ok or not?
Building FreeBSD without the network stack (network stack as a module)
----------------------------------------------------------------------
Today, not compiling networking stack related files in the kernel breaks
the kernel build due to dependencies the OS has on the network stack
(calling into functions in the network stack). Network stack module isn't
there. We've added these in JUNOS. The benefits for us are obvious (we can
load our own version of network stack if we desire!), but most likely this
functionality will benefit others too.
The detailed implementation is indicated later in this email. In short the
changes are:
- Load network stack as a module. For now via loader, not dynamically
loaded. (Is there interest in dynamic loading?).
- To facilitate calling network stack functionality from the generic
kernel, a new interface has been defined with the kobj framework.
- Some files and tunables needed to move to generic kernel areas (accept
filters)
- Generic socket code calls into network stack for interface and route
related ioctls. Network stack now registers these ioctl groups
- Other changes: uuid generation (register/query uuid sources), fib/sctp
system calls (moved to network stack code, with system calls register
dynamically).
Changes to mbuf and socket structures
--------------------------------------
A couple additions to these structures help JUNOS incorporate cool things
like interface/route indices and logical routing. For us the diff today
looks=20
something like:
struct pkthdr {
+ uint32_t rcvidx; /* rcv interface index */
+ uint32_t rnhidx; /* route or nexthop index */
struct ifnet *rcvif; /* rcv interface */
struct socket {
int so_fibnum; /* routing domain for this socket */
uint32_t so_user_cookie;
+ u_int so_oqueue; /* manage send prioritizing based on application
needs */
+ u_short so_lrid; /* logical routing */
};
A question for the community is if it's there're strong objections to
adding fields to these structures. If so, do we have another way or a
suggestion to consider?
For a detailed look, diffs can be found at:
http://people.freebsd.org/~marcel/Juniper/ddi-mbuf-socket.diff
http://people.freebsd.org/~marcel/Juniper/netstack.diff
These will provide a good idea of the changes. They're not in final shape
yet.
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?CC5C760E.18207%anshukla>