Date: Mon, 5 Aug 2013 11:32:02 -0600 From: Scott Long <scott4long@yahoo.com> To: Adrian Chadd <adrian@freebsd.org> Cc: current@freebsd.org, Bryan Venteicher <bryanv@daemoninthecloset.org>, Navdeep Parhar <np@freebsd.org>, net@freebsd.org, Giuseppe Lettieri <g.lettieri@iet.unipi.it>, Luigi Rizzo <rizzo@iet.unipi.it> Subject: Re: [net] protecting interfaces from races between control and data ? Message-ID: <195AEBBF-4297-4570-9D38-954FEC8D08DB@yahoo.com> In-Reply-To: <CAJ-Vmomd9bD9cXJtOWnuL_yuM%2BF%2Bt6zaaEgq2PxiT_wxQi%2Bejg@mail.gmail.com> References: <20130805082307.GA35162@onelab2.iet.unipi.it> <2034715395.855.1375714772487.JavaMail.root@daemoninthecloset.org> <CAJ-VmokT6YKPR7CXsoCavEmWv3W8urZu4eBVgKWaj9iMaVJFZg@mail.gmail.com> <CA%2BhQ2%2BhuoCCweq7fjoYmH3nyhmhb5DzukEdPSMtaJEWa8Ft0JQ@mail.gmail.com> <51FFDD1E.1000206@FreeBSD.org> <CAJ-Vmo=Q9AqdBJ0%2B4AiX4%2BWreYuZx6VGGYw=MZ4XhMB1P2yMww@mail.gmail.com> <CAJ-Vmomd9bD9cXJtOWnuL_yuM%2BF%2Bt6zaaEgq2PxiT_wxQi%2Bejg@mail.gmail.com>
next in thread | previous in thread | raw e-mail | index | archive | help
On Aug 5, 2013, at 11:20 AM, Adrian Chadd <adrian@freebsd.org> wrote: > .. and I bet it's not a design pattern, and this is total conjecture = on my part: >=20 > * the original drivers weren't SMP safe; > * noone really sat down and figured out how to correctly synchronise > all of this stuff; > * people did the minimum amount of work to keep the driver from > immediately crashing, but didn't really think things through at a > larger scale. >=20 > Almost every driver is this way Luigi. :-) >=20 >=20 Yes, but let me expand. The original work to make NIC drivers SMP = focused around just putting everything under 1 lock. The lock was acquired in if_start and held through the transmission of the packet, it was held in if_ioctl all the way through whatever action was taken, and it was held = in the interrupt handler over all of the RX and TX-complete processing. = This worked fine up until the RX path called if_input() with the netisr path = set for direct dispatch. In this mode, the code could flow up the stack and then immediately back down into the if_start handler for the driver, where it would try to acquire the same lock again. Also, it meant that forwarding packets between to interfaces would have the lock from the RX context of one interface held into the TX context of the other = interface. Obviously, this was a big mess, so the "solution" was to just drop the lock around the call to if_input(). No consideration was made for = driver state that might change during the lock release, nor was any = consideration made for the performance impact of dropping the lock on every packet and then having to contend with top-half TX threads to pick it back up. But this became the quick-fix pattern. As for the original question of why the RX path can operate unlocked, = it's fairly easy. The RX machinery of most NICs is fairly separate from the = TX machinery, so little synchronization is needed. When there's a state = change in the driver, in terms of an interface going down, a queue size = changing, or the driver being unloaded, it's up to the driver to pause and drain = the RX processing prior to this state change. It worked well when I did it = in if_em, and it appears to work well with cxgbe. Scott
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?195AEBBF-4297-4570-9D38-954FEC8D08DB>