From owner-freebsd-net@FreeBSD.ORG Thu Oct 19 07:07:09 2006 Return-Path: X-Original-To: freebsd-net@freebsd.org Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6856916A492 for ; Thu, 19 Oct 2006 07:07:09 +0000 (UTC) (envelope-from bde@zeta.org.au) Received: from mailout1.pacific.net.au (mailout1-3.pacific.net.au [61.8.2.210]) by mx1.FreeBSD.org (Postfix) with ESMTP id B534E43D5F for ; Thu, 19 Oct 2006 07:07:08 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy2.pacific.net.au (mailproxy2.pacific.net.au [61.8.2.163]) by mailout1.pacific.net.au (Postfix) with ESMTP id 74FAE5DFCFF; Thu, 19 Oct 2006 17:07:07 +1000 (EST) Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) by mailproxy2.pacific.net.au (Postfix) with ESMTP id 21E042741F; Thu, 19 Oct 2006 17:07:06 +1000 (EST) Date: Thu, 19 Oct 2006 17:07:02 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: Scott Long In-Reply-To: <45371B91.5090507@samsco.org> Message-ID: <20061019164814.L76712@delplex.bde.org> References: <2a41acea0610181046k822afd1qcec4187dc8514187@mail.gmail.com> <2a41acea0610181531y732cd5sa7bf733cc445491c@mail.gmail.com> <20061018224233.GA1632@xor.obsecurity.org> <20061019110950.X75878@delplex.bde.org> <4536EF19.2060201@samsco.org> <20061019141748.Y76352@delplex.bde.org> <45371B91.5090507@samsco.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: Kip Macy , freebsd-net , Jack Vogel , Kris Kennaway Subject: Re: em network issues X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Oct 2006 07:07:09 -0000 On Thu, 19 Oct 2006, Scott Long wrote: > Bruce Evans wrote: >>>> On Wed, 18 Oct 2006, Kris Kennaway wrote: >>>>> I have been working with someone's system that has em shared with fxp, >>>>> and a simple fetch over the em (e.g. of a 10 GB file of zeroes) is >>>>> enough to produce watchdog timeouts after a few seconds. >>>> >>>> em_intr_fast() has no locking whatsoever. I would be very surprised >>>> if it even seemed to work for SMP. For UP, masking of CPU interrupts >>>> (as is automatic in fast interrupt handlers) might provide sufficient >>>> locking, ... >> >> I barely noticed the point about it being shared. With sharing, and >> probably especially with fast and normal interrupt handlers sharing an >> IRQ, locking is more needed. There are many possibilities for races. >> One likely one is: >> - em interrupt task running. Device interrupts are disabled, so the >> task thinks it won't be interfered with by the em interrupt handler. > > What interference are you talking about? em_intr_fast changes no state > in the driver softc (aside from the silly bookkeeping). It only reads > from one register, and writes to no registers or shared memory. It disables interrupts. To do that, it calls em_disable_intr(). The hardware is simple enough for em_disable_intr() not to have to make many state changes, but it certainly has to make at least 1 to work. It uses several layers of macros which I think ends up doing a write to 1 register in bus space. >> - shared fxp interrupt. The em interrupt handler is called. Without >> any explicit synchonization, bad things may happen and apparently do. >> In the UP case, there is some implicit synchronization which may help >> but is hard to understand. > > Can you be more specific as to the 'bad things'? Not very. Maybe interrupts don't get reenabled as intended. Then the symptoms get mutated by watchdog timeouts. Bruce