From owner-freebsd-pf@FreeBSD.ORG Wed Jun 8 23:23:44 2005 Return-Path: X-Original-To: pf@freebsd.org Delivered-To: freebsd-pf@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3941716A41F for ; Wed, 8 Jun 2005 23:23:44 +0000 (GMT) (envelope-from mgrooms@seton.org) Received: from zixvpm01.seton.org (zixvpm01.seton.org [207.193.126.161]) by mx1.FreeBSD.org (Postfix) with ESMTP id 62ACC43D48 for ; Wed, 8 Jun 2005 23:23:42 +0000 (GMT) (envelope-from mgrooms@seton.org) Received: from zixvpm01.seton.org (ZixVPM [127.0.0.1]) by Outbound.seton.org (Proprietary) with ESMTP id 0D5EB3600C1 for ; Wed, 8 Jun 2005 18:23:42 -0500 (CDT) Received: from mx1-out.seton.org (unknown [10.21.254.249]) by zixvpm01.seton.org (Proprietary) with ESMTP id A89AC33005A; Wed, 8 Jun 2005 18:23:41 -0500 (CDT) Received: from localhost (unknown [127.0.0.1]) by mx1-out.seton.org (Postfix) with ESMTP id 9B4F58014E25; Wed, 8 Jun 2005 18:23:41 -0500 (CDT) Received: from mx1-out.seton.org ([10.21.254.249]) by localhost (mx1 [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id 21531-33; Wed, 8 Jun 2005 18:23:41 -0500 (CDT) Received: from ausexfe01.seton.org (ausexfe01.seton.org [10.20.10.211]) by mx1-out.seton.org (Postfix) with ESMTP id 513A78014E24; Wed, 8 Jun 2005 18:23:41 -0500 (CDT) Received: from AUSEX2VS1.seton.org ([10.20.10.74]) by ausexfe01.seton.org with Microsoft SMTPSVC(6.0.3790.211); Wed, 8 Jun 2005 18:23:40 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.5.7226.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Date: Wed, 8 Jun 2005 18:23:15 -0500 Message-ID: <28FCC7CB4CF6EA43AF83BCA2096E97D013E563@AUSEX2VS1.seton.org> X-MS-Has-Attach: X-MS-TNEF-Correlator: Thread-Topic: 5.4-RELEASE lockups on amd64 SMP Thread-Index: AcVsNMiyFNlXC+NBTTGZHcmjZddM7wASRnT9AADKZIg= From: "Grooms, Matthew" To: "Grooms, Matthew" , "Max Laier" X-OriginalArrivalTime: 08 Jun 2005 23:23:40.0014 (UTC) FILETIME=[1A7148E0:01C56C81] X-Virus-Scanned: by amavisd-new at seton.org Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: pf@freebsd.org, glebius@freebsd.org, freebsd-stable@freebsd.org, Palle Girgensohn , Kris Kennaway Subject: RE: 5.4-RELEASE lockups on amd64 SMP X-BeenThere: freebsd-pf@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: "Technical discussion and general questions about packet filter \(pf\)" List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Jun 2005 23:23:44 -0000 Max, With your patch applied, I get a panic very quickly during the boot = cycle with output that looks like this ... net.inet.carp.preempt: 0 -> 1 Setting hostname: ---. em: Link is up 100 Mbps Full Duplex panic: mtx_lock() of spin mutex (null) @ ../../../net/if.c:1983 cpuid =3D 1 KDB: enter: panic [thread pid 282 tid 100157 ] Stopped at kdb_enter+0x2f: nop db> trace Tracing pid 282 tid 100157 td 0xffffff000af78280 kdb_enter() at kdb_enter+0x2f panic() at panic+0x249 _mtx_lock_flags() at _mtx_lock_flags+0xd6 if_handoff() at if_handoff+0x49 pfsync_sendout() at pfsync_sendout+0x268 pfsyncioctl() at pfsyncioctl+0x497 in_control() at in_control+0x8cb ifioctl() at ifioctl+0x178 sooo_ioctl() at soo_ioctl+0x2d6 ioctl() at ioctl+0xfc syscall() at syscall+0x4ab Xfast_syscall() at Xfast_syscall+0xa8 --- syscall (54, FreeBSD ELF64, ioctl), rip =3D 0x800793340, rsp =3D = 0x7fffffffeca8, rbp =3D 0x7fffffffef8b --- db> show locks eclusive sleep mutex pf task mtx r =3D 0 (0xffffffff80752f60) locked @ = contrib/pf/net/if_pfsync.c:973 Rebooting the machine with the same kernel produces an identical panic. = Let me know what else I can do to help. Right now I have just been = rebooting back to a UP kernel which has never shown any sign of = problems. Matthew Grooms -----Original Message----- From: Grooms, Matthew Sent: Wed 6/8/2005 6:22 PM To: Max Laier Cc: Palle Girgensohn; Kris Kennaway; freebsd-stable@freebsd.org; = glebius@freebsd.org; pf@freebsd.org Subject: RE: 5.4-RELEASE lockups on amd64 SMP =20 Matthew, can you try the attached diff. Available for 5 and CURRENT. I recall = that=20 this problem was seen before, strange that I didn't see the problem. = Sounds=20 familiar to you? Please try the patch and let me know if that helps. = Thanks=20 a lot. On Wednesday 08 June 2005 01:35, Matthew Grooms wrote: > Once again, here are the backtraces for the panic and lor ... > > Tracing id 110 tid 100089 td 0xffffff012f3f0c80 > kdb_enter() at kdb_enter+0x2f > panic() at panic+0x249 > uma_dbg_free() at uma_dbg_free+0x188 > uma_zfree_arg() at uma_zfree_arg+0x1b0 > pf_purge_expired_states() at pf_purge_expired_states+0x41 > pfsync_input at pfsync_input+xb35 > pf_input() at ip_input+0x10f > netisr_processqueue() at netisr_processqueue+0x17 > swi_net() at swi_net+0xa8 > ithread_loop() at ithread_loop+0xd9 > fork_exit() at fork_exit+0xc3 > fork_trampoline() at fork_trampoline+0xe > --- trap 0, rip =3D 0, rsp =3D 0xffffffffb44f9d00, rbp =3D 0 --- > db> continue > boot() called on cpu#0 > Uptime: 13h42m43s > Dumping 4864 MB > 16 32 ... > > lock order reversal ... > alltraps_with_regs_pushed() at alltraps_with_regs_pushed+0x5 > pf_state_tree_lan_ext_RB_REMOVE() at = pf_state_tree_lan_ext_RB_REMOVE+0x10c This LOR is a consequence of the fault, so it can be disregarded. --=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News