From owner-freebsd-current@FreeBSD.ORG Tue Jul 3 13:45:24 2007 Return-Path: X-Original-To: freebsd-current@freebsd.org Delivered-To: freebsd-current@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 31ADA16A421 for ; Tue, 3 Jul 2007 13:45:24 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from cyrus.watson.org (cyrus.watson.org [209.31.154.42]) by mx1.freebsd.org (Postfix) with ESMTP id 03D4413C45A for ; Tue, 3 Jul 2007 13:45:23 +0000 (UTC) (envelope-from rwatson@FreeBSD.org) Received: from fledge.watson.org (fledge.watson.org [209.31.154.41]) by cyrus.watson.org (Postfix) with ESMTP id 6D487480A4; Tue, 3 Jul 2007 09:45:23 -0400 (EDT) Date: Tue, 3 Jul 2007 14:45:23 +0100 (BST) From: Robert Watson X-X-Sender: robert@fledge.watson.org To: Ian FREISLICH In-Reply-To: Message-ID: <20070703143617.D29272@fledge.watson.org> References: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: bu7cher@yandex.ru, freebsd-current@freebsd.org Subject: Re: Panic in ipfw X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Jul 2007 13:45:24 -0000 On Tue, 3 Jul 2007, Ian FREISLICH wrote: > "Andrey V. Elsukov" wrote: >> >>> I got this panic yesterday on a fairly busy firewall. I have some private >>> patches to ip_fw2.c and to the em driver (see the earlier "em0 hijacking >>> traffic to port 623" thread). I don't think this panic is a result of >>> those changes. >> >>> It occurred round about the time an address was added to an interface. >> >> I have a patch that can help you (i guess..). Can you test this patch? >> >> http://butcher.heavennet.ru/patches/kernel/inaddr_locking/ > > Thanks. Wow, that looks like it touches a lot more than just ipfw. It took > about 1.5 years of production at 2.3 billion backets a day to trigger this > condition twice. It's going to be difficult to tell if this patch fixes the > problem. This, FYI, is actually the reason why the locking isn't there now -- when prioritizing things to make MPSAFE, performance work, etc, address/ifnet lists, with the exception of multicast address lists, it was clear that they were basically static data structures. Andrey's patch addresses problems I've wanted to work on for several years, and I'm very pleased he's working on it. It's only part of the solution to the long-term problem there -- we have a number of other synchronization issues for data structures that are nearly almost always static, and some life cycle issues with ifnet registration. To date, I think this is really the first bug report I've seen that I could authoritatively point the finger at missing synchronization at the ifnet/ifaddr layer as the source. My hope is that these will be addressed in FreeBSD 8.x, especially with the upcoming read-mostly locks, which will have almost zero cost to acquire for read protection, exactly what we'd like to see for these code paths. Some of these changes may be mergeable to the 7.x branch, but will need lots of time to "burn in", as the risks of such changes are non-trivial. I did some initial work to properly lock down the ifaddr address lists for netinet, and found they required moderate rearrangement of the address management ioctl paths, which are pretty complex without being rerranged :-). Robert N M Watson Computer Laboratory University of Cambridge