From owner-freebsd-current@FreeBSD.ORG Mon Aug 5 17:32:12 2013 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) (using TLSv1 with cipher ADH-AES256-SHA (256/256 bits)) (No client certificate requested) by hub.freebsd.org (Postfix) with ESMTP id 35332251 for ; Mon, 5 Aug 2013 17:32:12 +0000 (UTC) (envelope-from scott4long@yahoo.com) Received: from nm13.bullet.mail.ne1.yahoo.com (nm13.bullet.mail.ne1.yahoo.com [98.138.90.76]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.freebsd.org (Postfix) with ESMTPS id D9AA62A84 for ; Mon, 5 Aug 2013 17:32:11 +0000 (UTC) Received: from [98.138.90.56] by nm13.bullet.mail.ne1.yahoo.com with NNFMP; 05 Aug 2013 17:32:04 -0000 Received: from [98.138.226.127] by tm9.bullet.mail.ne1.yahoo.com with NNFMP; 05 Aug 2013 17:32:04 -0000 Received: from [127.0.0.1] by smtp206.mail.ne1.yahoo.com with NNFMP; 05 Aug 2013 17:32:04 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s1024; t=1375723924; bh=YiS4NvFfaKQ8oewMV8XxgvkG12NOLehq+k67gYJZaok=; h=X-Yahoo-Newman-Id:X-Yahoo-Newman-Property:X-YMail-OSG:X-Yahoo-SMTP:X-Rocket-Received:Content-Type:Mime-Version:Subject:From:In-Reply-To:Date:Cc:Content-Transfer-Encoding:Message-Id:References:To:X-Mailer; b=bszIC1LJKRZMcjmLLa4+S54WpbnnkBNUQgSZuR5vRjBm+09N4HA8e2LDABIsUvDGTlqSIjWOMdT+9TDKzfe3kV1oS1LzoM8Piy3iMmmeU1I0GjHGd+XGp6D15yyneKn1AzCP9W79zwFj6N4GJw15gXUxKO9rjgKLW9qMq1NBvkM= X-Yahoo-Newman-Id: 697595.10846.bm@smtp206.mail.ne1.yahoo.com X-Yahoo-Newman-Property: ymail-3 X-YMail-OSG: qJrqZAkVM1n1eOIWn9ZHv7Bd1ajC776U11nuLqzRMtf.vl0 trVo7Gmmdr_AtuZsuYUYtGO69JamoOQkDHdXKUzccGSeotnZzCQrIxarpj3z jVm8NU0bHmCgTeKSEgv3uUFFH5.JYuuncnwWjTpDuBblyroyTnBwapUa8v_b q5GYrd6eYcMNKt1ZdWkTkOlYslg7olzBsUl1AOJJ7_igPT.wf33mILokhhDh S2nc.jmTYF0ug3rR9Tzmd0MEVH5PM8Yi9hqGzQkTp9gpTjxwvqqKxCzYt0LZ UPWlGCLFyVC33JJsA5_xIJ0rPgUCbaEx2owVV7uS6hNbDbYcINMeMllIhA4m y4yP9joYI5yjoxNE5omceTcUHDlmC4CYQJoBf.ZQEbkvtOc3wH7KRnY.K0Wp ko.15VteDiPGq.kiSmrdrswrVeJzLhJHUU2NpS1RlPfePFtxRlkK7T1KFEZO 4I.gcYBqa5BOlMSC0J30xF4Rah93NgiBAxza_ZxFkjz0byrvNfC8TRHUnzze dqpHcDL3bk0Y5O28ihS7m4z05jZTE6Qoswzzas3K0BIxE5..DyWsdnr9MfA- - X-Yahoo-SMTP: clhABp.swBB7fs.LwIJpv3jkWgo2NU8- X-Rocket-Received: from lgmc-adee.corp.netflix.com (scott4long@69.53.237.126 with ) by smtp206.mail.ne1.yahoo.com with SMTP; 05 Aug 2013 10:32:04 -0700 PDT Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\)) Subject: Re: [net] protecting interfaces from races between control and data ? From: Scott Long In-Reply-To: Date: Mon, 5 Aug 2013 11:32:02 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <195AEBBF-4297-4570-9D38-954FEC8D08DB@yahoo.com> References: <20130805082307.GA35162@onelab2.iet.unipi.it> <2034715395.855.1375714772487.JavaMail.root@daemoninthecloset.org> <51FFDD1E.1000206@FreeBSD.org> To: Adrian Chadd X-Mailer: Apple Mail (2.1508) X-Mailman-Approved-At: Mon, 05 Aug 2013 17:46:39 +0000 Cc: current@freebsd.org, Bryan Venteicher , Navdeep Parhar , net@freebsd.org, Giuseppe Lettieri , Luigi Rizzo X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 05 Aug 2013 17:32:12 -0000 On Aug 5, 2013, at 11:20 AM, Adrian Chadd wrote: > .. and I bet it's not a design pattern, and this is total conjecture = on my part: >=20 > * the original drivers weren't SMP safe; > * noone really sat down and figured out how to correctly synchronise > all of this stuff; > * people did the minimum amount of work to keep the driver from > immediately crashing, but didn't really think things through at a > larger scale. >=20 > Almost every driver is this way Luigi. :-) >=20 >=20 Yes, but let me expand. The original work to make NIC drivers SMP = focused around just putting everything under 1 lock. The lock was acquired in if_start and held through the transmission of the packet, it was held in if_ioctl all the way through whatever action was taken, and it was held = in the interrupt handler over all of the RX and TX-complete processing. = This worked fine up until the RX path called if_input() with the netisr path = set for direct dispatch. In this mode, the code could flow up the stack and then immediately back down into the if_start handler for the driver, where it would try to acquire the same lock again. Also, it meant that forwarding packets between to interfaces would have the lock from the RX context of one interface held into the TX context of the other = interface. Obviously, this was a big mess, so the "solution" was to just drop the lock around the call to if_input(). No consideration was made for = driver state that might change during the lock release, nor was any = consideration made for the performance impact of dropping the lock on every packet and then having to contend with top-half TX threads to pick it back up. But this became the quick-fix pattern. As for the original question of why the RX path can operate unlocked, = it's fairly easy. The RX machinery of most NICs is fairly separate from the = TX machinery, so little synchronization is needed. When there's a state = change in the driver, in terms of an interface going down, a queue size = changing, or the driver being unloaded, it's up to the driver to pause and drain = the RX processing prior to this state change. It worked well when I did it = in if_em, and it appears to work well with cxgbe. Scott