From owner-freebsd-current@FreeBSD.ORG Wed Jun 5 10:13:55 2013 Return-Path: Delivered-To: current@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 56EA5268; Wed, 5 Jun 2013 10:13:55 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id DE2F61F94; Wed, 5 Jun 2013 10:13:54 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r55ADjT6001771; Wed, 5 Jun 2013 13:13:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r55ADjT6001771 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r55ADjf1001770; Wed, 5 Jun 2013 13:13:45 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Wed, 5 Jun 2013 13:13:45 +0300 From: Konstantin Belousov To: Gleb Smirnoff Subject: Re: Recurring panic Message-ID: <20130605101345.GY3047@kib.kiev.ua> References: <20130605095043.GB67170@glebius.int.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="YkfI5/73A7RnyCZx" Content-Disposition: inline In-Reply-To: <20130605095043.GB67170@glebius.int.ru> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Ian FREISLICH , current@freebsd.org X-BeenThere: freebsd-current@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussions about the use of FreeBSD-current List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Jun 2013 10:13:55 -0000 --YkfI5/73A7RnyCZx Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Jun 05, 2013 at 01:50:43PM +0400, Gleb Smirnoff wrote: > On Wed, Jun 05, 2013 at 10:18:21AM +0200, Ian FREISLICH wrote: > I> I have the following recurring panic on all my heavily network > I> loaded -CURRENT routers. The current process is always different. > I>=20 > I> Gleb, can you please chime in with what you've managed to uncover. >=20 > The panics appear on selfd mutex. The mtx_lock value is a free mutex, but > it has 1 extra bit set: >=20 > (kgdb) p/x sfp->sf_mtx->mtx_lock > $3 =3D 0x1000004 >=20 > Rarely (only one panic observed) more than one bit is set: >=20 > $3 =3D 0x21000004 >=20 > It is important that selfd mutexes are taken from mtxpool(9), which > is allocated at a early boot stage. Thus, across reboots all possible > sfp->sf_mtx mutexes usually fall into the same virtual memory region. > I'm not sure, but I suppose, they fall into same physical region. >=20 > This can lead one to idea that RAM in the box has problems. But it > is running ECC memory, and it doesn't experience other random panics. >=20 > The only special about the box is that it is running pf(4) with huge > ruleset and a lot of traffic. So the pf(4) is the number one suspected, > albeit it isn't closely related to selfds. >=20 So is the virtual address of the corrupted word same for each panic ? If yes, set up the hw watchpoint in ddb. --YkfI5/73A7RnyCZx Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRrw9YAAoJEJDCuSvBvK1BC3AP/1nNmlp8xX6IEItncs8bpTEH +WYrQvncmtyy3rfPkveiRBYvFfNWqPEti0OHBIXXue10R+zb2er6ExybRckIy69t K3DZ3Mm0usejNNZXkSRGL4mNwMSPXj9D0SDroOOLYNbFWj394A1zz7aXsqUTq5sU S4QyANHOVf/nmRDg/HOJaJIY+IZKSecHlI7k8VpySB7UXvzdcnQ388yOChw12Zze nqkdFFvF3ocax+6Oeqz404UffH8dEyCqQtyMeVZXKi5DjNZJWkezKSwq2Vg7GLfp IsRlS4F84ejRjmQ2P5i0wYWybIS75tsESksscF12hVJ6x5pfAVXQwHNss6FAf6dX VTxzsmHxqiHxJHydMUZBZAoaZTG97FNGv7VxQi/35bS/e5ANGU+yOZvhXEq/JNEz T+eb9/9kyB5SC4FkeFzBXhYMjX40zUTdDx7QRqfFZA2t+gcQB1ZWcPJLl59SS/4O iuhLo2VFy4YWvUVG5B6A7tGERfv73QmxILZQ+WOkbc2ezc1Av/Y/FLbWTPXjFMoE 8I4F2R9QmqBxtHOX2IaqXsavGqgODOBob3o2KIJWx3E1xbYigyOsI6juqvFfYv3z WvietFJNcsyOdtzwx1GJyWD/xL7325RZmUR2+JEO5rxxk9LDdESVC0K8rD6cnLDX aLxtEt9Q9JswsTodqYAx =+GKt -----END PGP SIGNATURE----- --YkfI5/73A7RnyCZx--