From owner-freebsd-stable@FreeBSD.ORG Mon Jun 20 08:35:43 2005 Return-Path: X-Original-To: stable@freebsd.org Delivered-To: freebsd-stable@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2879A16A41C; Mon, 20 Jun 2005 08:35:43 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [204.156.12.53]) by mx1.FreeBSD.org (Postfix) with ESMTP id EA29443D4C; Mon, 20 Jun 2005 08:35:42 +0000 (GMT) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [204.156.12.50]) by cyrus.watson.org (Postfix) with ESMTP id 3C17246BA3; Mon, 20 Jun 2005 04:35:42 -0400 (EDT) Date: Mon, 20 Jun 2005 09:38:27 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: =?ISO-8859-1?Q?Eirik_=D8verby?= In-Reply-To: Message-ID: <20050620092829.E19830@fledge.watson.org> References: <8149D7F8-3FA2-48F5-BF03-9AF813448BF0@anduin.net> <20050619185338.J6413@fledge.watson.org> MIME-Version: 1.0 Content-Type: MULTIPART/MIXED; BOUNDARY="0-980453451-1119256707=:19830" Cc: stable@freebsd.org, mlaier@FreeBSD.org Subject: Re: NFS-related hang in 5.4? X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Jun 2005 08:35:43 -0000 This message is in MIME format. The first part should be readable text, while the remaining parts are likely unreadable without MIME-aware tools. --0-980453451-1119256707=:19830 Content-Type: TEXT/PLAIN; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE On Mon, 20 Jun 2005, Eirik =D8verby wrote: >> Hmm. Looks like a bug in dummynet. ipfw should not be directly re-=20 >> injecting UDP traffic back into the input path from an outbound path,=20 >> or it risks re-entering, generating lock order problems, etc. It should= =20 >> be getting dropped into the netisr queue to be processed from the=20 >> netisr context. > > This problem would exist across all 5.4 installations, both i386 and=20 > amd64? Would it depend on heavy load, or could it theoretically happen=20 > at any time when there's traffic? All three of my fbsd5 servers (dual=20 > opteron, dual p3-1ghz, dual p3-700mhz) are experiencing random hangs=20 > with ~a few weeks between, impression is that if running single-cpu mode= =20 > they are all stable. All using dummynet in a comparable manner. Ideas? Yes. Basically, the network stack avoids recursion in processing for=20 "complicated" packets by deferring processing an offending packet to a=20 thread called the 'netisr'. Whenever the stack reaches a possible=20 recursion point on a packet, it's supposed to queue the packet for=20 processing 'later' in a per-protocol queue, unwind, and then when the=20 netisr runs, pick up and continue processing. In the stack trace you=20 provide, dummynet appears to immediately immediately invoke the in-bound=20 network path from the out-bound network path, walking back into the=20 network stack from the outbound path. This is generally forbidden, for a= =20 variety of reasons: - We do allow the in-bound path to call the out-bound path, so that protocols like TCP, and services like NFS can turn around packets without a context switch. If further recursion is permitted, the stack may overflow. - Both paths may hold network stack locks over calls in either direction -- specifically, we allow protocol locks to be held over calls into the socket layer, as the protocol layer drives operation; if a recursive call is made, deadlocks can occur due to violating the lock order. This is what is happening in your case. Pretty much all network code is entirely architecture-independent, so bugs= =20 typically span architectures, although race conditions can sometimes be=20 hard to reproduce if they require precise timing and multiple processors. >> Is it possible to configure dummynet out of your configuration, and see= =20 >> if the problem goes away? > > I'm running a test right now, will let you know in the morning. Thanks. Robert N M Watson --0-980453451-1119256707=:19830--