From owner-freebsd-ipfw@freebsd.org Tue Nov 3 14:04:40 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A0209A24DAC for ; Tue, 3 Nov 2015 14:04:40 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 257921D9A for ; Tue, 3 Nov 2015 14:04:40 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id 07408A24DA8; Tue, 3 Nov 2015 14:04:40 +0000 (UTC) Delivered-To: ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E27DFA24DA7; Tue, 3 Nov 2015 14:04:39 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4ED001D59; Tue, 3 Nov 2015 14:04:38 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id tA3E4aYE027892; Tue, 3 Nov 2015 06:04:36 -0800 (PST) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id tA3E4aAh027891; Tue, 3 Nov 2015 06:04:36 -0800 (PST) (envelope-from david) Date: Tue, 3 Nov 2015 06:04:36 -0800 From: David Wolfskill To: ipfw@freebsd.org Cc: current@freebsd.org, net@freebsd.org Subject: panic: refcount inconsistency: found: 0 total: 1 Message-ID: <20151103140436.GJ21127@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , ipfw@freebsd.org, current@freebsd.org, net@freebsd.org MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="C7PTD44AewjTsiSV" Content-Disposition: inline User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 14:04:40 -0000 --C7PTD44AewjTsiSV Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable This was on my laptop; yesterday, it built & booted: FreeBSD g1-252.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #230 r2902= 70M/290270:1100085: Mon Nov 2 05:03:07 PST 2015 root@g1-252.catwhisker= =2Eorg:/common/S4/obj/usr/src/sys/CANARY amd64 OK; today, after building: FreeBSD localhost 11.0-CURRENT FreeBSD 11.0-CURRENT #231 r290334M/290334:1= 100086: Tue Nov 3 04:51:24 PST 2015 root@g1-252.catwhisker.org:/common= /S4/obj/usr/src/sys/CANARY amd64 I tried booting it, and during the transition to multi-user mode, once ipfw was being invoked, I got the above-cited panic. Circumvention was to leave it disconnected from a network (turn off the WiFi switch, in my case), so we don't get a chance to use the network. I was able to get a dump by explicitly typing "call doadump" -- an earlier attempt at "panic" didn't capture one. Stack trace: #0 doadump (textdump=3D0) at pcpu.h:221 221 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump (textdump=3D0) at pcpu.h:221 #1 0xffffffff8037b6b6 in db_fncall (dummy1=3D,=20 dummy2=3D, dummy3=3D,=20 dummy4=3D) at /usr/src/sys/ddb/db_command.c:568 #2 0xffffffff8037b14e in db_command (cmd_table=3D0x0) at /usr/src/sys/ddb/db_command.c:440 #3 0xffffffff8037aee4 in db_command_loop () at /usr/src/sys/ddb/db_command.c:493 #4 0xffffffff8037d97b in db_trap (type=3D, code=3D0) at /usr/src/sys/ddb/db_main.c:251 #5 0xffffffff80a270f3 in kdb_trap (type=3D3, code=3D0, tf=3D) at /usr/src/sys/kern/subr_kdb.c:654 #6 0xffffffff80db6668 in trap (frame=3D0xfffffe060bdde1d0) at /usr/src/sys/amd64/amd64/trap.c:549 #7 0xffffffff80d961f7 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:234 #8 0xffffffff80a267db in kdb_enter (why=3D0xffffffff812a5566 "panic",=20 msg=3D0x80
) at cpufunc.h:63 #9 0xffffffff809ea01f in vpanic (fmt=3D,=20 ap=3D) at /usr/src/sys/kern/kern_shutdown.c:750 #10 0xffffffff809e9e76 in kassert_panic (fmt=3D) at /usr/src/sys/kern/kern_shutdown.c:647 #11 0xffffffff80c2a788 in ipfw_rewrite_rule_uidx (chain=3D0xffffffff81be531= 0,=20 ci=3D0xfffffe060bdde4b8) at /usr/src/sys/netpfil/ipfw/ip_fw_table.c:3395 #12 0xffffffff80c267c3 in commit_rules (chain=3D0xffffffff81be5310,=20 rci=3D0xfffffe060bdde4b8, count=3D1) at /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:678 #13 0xffffffff80c25d80 in add_rules (chain=3D0xffffffff81be5310,=20 op3=3D, sd=3D) at /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:2594 #14 0xffffffff80c232f4 in ipfw_ctl3 (sopt=3D0xfffffe060bdde920) at /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:3242 #15 0xffffffff80b3d8b1 in rip_ctloutput (so=3D,=20 sopt=3D0xfffffe060bdde920) at /usr/src/sys/netinet/raw_ip.c:588 #16 0xffffffff80a72bc6 in sogetopt (so=3D0xfffff80009e658b8,=20 sopt=3D0xfffffe060bdde920) at /usr/src/sys/kern/uipc_socket.c:2731 #17 0xffffffff80a7729e in kern_getsockopt (td=3D0xfffff800098119a0,=20 s=3D, level=3D,=20 name=3D, val=3D, valseg=3D464= ,=20 valsize=3D0xfffffe060bdde98c) at /usr/src/sys/kern/uipc_syscalls.c:1540 #18 0xffffffff80a771a0 in sys_getsockopt (td=3D0xfffff800098119a0,=20 uap=3D0xfffffe060bddea40) at /usr/src/sys/kern/uipc_syscalls.c:1486 #19 0xffffffff80db7519 in amd64_syscall (td=3D0xfffff800098119a0, traced=3D= 0) at subr_syscall.c:140 #20 0xffffffff80d964db in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:394 #21 0x0000000800b2cbea in ?? () Previous frame inner to this frame (corrupt stack?) Current language: auto; currently minimal (kgdb)=20 I've copied the vmcore.z & core.txt.7 to ; gzipped copies are also available: Index of /~david/FreeBSD/head/ipfw Icon Name Last modified Size Description _____________________________________________________________________ [PARENTDIR] Parent Directory - [TXT] core.txt.7 2015-11-03 05:22 155K [ ] core.txt.7.gz 2015-11-03 05:22 35K [ ] vmcore.7 2015-11-03 05:22 528M [ ] vmcore.7.gz 2015-11-03 05:22 45M _____________________________________________________________________ I'll start taking a closer look at recent changes (e.g., in src/sys/netpfil/ipfw), but I'm not really all that familiar with the code. Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --C7PTD44AewjTsiSV Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJWOL70XxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7YqwP/3ctxamCUq5Xt5CxCle4HVSG LL3gi+6WEC6FbYMO7SjKdHw/OREkja/mBkNDcI76ps7W+o3xMNW0YwMX5/Wr/aNR T7KTGNkTDK6MP04DimtMscQXqTYVLerd3hOKNoGw/vrGUQZwYbhO9jBlVAoBP2Ax aobqivKD5ajHngRnOKfdGQKucfx75mKNbwypej0w6GnDo0vhkETfwvsYSFIdOvUD JFPSXq5raGJKpjazE6tXvxFi+zN/XHQ2CnYni/1Q8pbuNc17TMBIA73zC0sDbTp4 9hiA22/G7EdAfC+PpRNfx2afwd39yNx5zlTAAKXzpA+12HLRVhX2q3y5xaMDlrCK r82csuRGX+lq0w//I6S2zMKolx9fxj6GOQrg38uAdjgwNaywRdm5V9es9KfmYlBO BTLLY1szyYZOzql8GeVI7D2zXC2jxhZZ8YCWOlJtrvLhcrYYI5tjUdQd88HQSIan Irn9Vrf6cMJwrFw7qiZIzPrCi1QTknkFaq5ybVAA84TG0YA4XGWhC3zeVLpUFMKp v4lRI/tAPhtAShu/zA8u58nqVUJj9Nvg/U2K4JS+nPI2NTrKEOVIeh1LpGM3lZS2 hUTjqnHZNBEBCXgvxcPjNkwL5noT+DUWX8V5fNpnCT8mWCXPYIHge6cLpW0jSO0r kCVxbpPFuJXMOz2KhldA =Xz+w -----END PGP SIGNATURE----- --C7PTD44AewjTsiSV-- From owner-freebsd-ipfw@freebsd.org Tue Nov 3 15:15:41 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 911ADA25DA4 for ; Tue, 3 Nov 2015 15:15:41 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 713A01311 for ; Tue, 3 Nov 2015 15:15:41 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 6D40EA25DA1; Tue, 3 Nov 2015 15:15:41 +0000 (UTC) Delivered-To: ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6B4B9A25D9F; Tue, 3 Nov 2015 15:15:41 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from forward15h.cmail.yandex.net (forward15h.cmail.yandex.net [IPv6:2a02:6b8:0:f35::a0]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Certum Level IV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id EEC0D130F; Tue, 3 Nov 2015 15:15:40 +0000 (UTC) (envelope-from melifaro@ipfw.ru) Received: from web30h.yandex.ru (web30h.yandex.ru [IPv6:2a02:6b8:0:f05::40]) by forward15h.cmail.yandex.net (Yandex) with ESMTP id 6EEB921380; Tue, 3 Nov 2015 18:15:37 +0300 (MSK) Received: from 127.0.0.1 (localhost [127.0.0.1]) by web30h.yandex.ru (Yandex) with ESMTP id 5787E26C27EE; Tue, 3 Nov 2015 18:15:36 +0300 (MSK) Received: by web30h.yandex.ru with HTTP; Tue, 03 Nov 2015 18:15:35 +0300 From: Alexander V. Chernikov Envelope-From: melifaro@ipfw.ru To: David Wolfskill , "ipfw@freebsd.org" Cc: "current@freebsd.org" , "net@freebsd.org" In-Reply-To: <20151103140436.GJ21127@albert.catwhisker.org> References: null <20151103140436.GJ21127@albert.catwhisker.org> Subject: Re: panic: refcount inconsistency: found: 0 total: 1 MIME-Version: 1.0 Message-Id: <693401446563735@web30h.yandex.ru> X-Mailer: Yamail [ http://yandex.ru ] 5.0 Date: Tue, 03 Nov 2015 18:15:35 +0300 Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset=koi8-r X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 15:15:41 -0000 03.11.2015, 17:05, "David Wolfskill" : > This was on my laptop; yesterday, it built & booted: > > FreeBSD g1-252.catwhisker.org 11.0-CURRENT FreeBSD 11.0-CURRENT #230 r290270M/290270:1100085: Mon Nov 2 05:03:07 PST 2015 root@g1-252.catwhisker.org:/common/S4/obj/usr/src/sys/CANARY amd64 > > OK; today, after building: > > FreeBSD localhost 11.0-CURRENT FreeBSD 11.0-CURRENT #231 r290334M/290334:1100086: Tue Nov 3 04:51:24 PST 2015 root@g1-252.catwhisker.org:/common/S4/obj/usr/src/sys/CANARY amd64 > > I tried booting it, and during the transition to multi-user mode, > once ipfw was being invoked, I got the above-cited panic. Circumvention > was to leave it disconnected from a network (turn off the WiFi > switch, in my case), so we don't get a chance to use the network. It is most probably related with r290334. Would you mind reverting it and checking if ipfw works correctly ? > > I was able to get a dump by explicitly typing "call doadump" -- an > earlier attempt at "panic" didn't capture one. Stack trace: > > #0 doadump (textdump=0) at pcpu.h:221 > 221 pcpu.h: No such file or directory. > ššššššššin pcpu.h > (kgdb) #0 doadump (textdump=0) at pcpu.h:221 > #1 0xffffffff8037b6b6 in db_fncall (dummy1=, > ššššdummy2=, dummy3=, > ššššdummy4=) at /usr/src/sys/ddb/db_command.c:568 > #2 0xffffffff8037b14e in db_command (cmd_table=0x0) > ššššat /usr/src/sys/ddb/db_command.c:440 > #3 0xffffffff8037aee4 in db_command_loop () > ššššat /usr/src/sys/ddb/db_command.c:493 > #4 0xffffffff8037d97b in db_trap (type=, code=0) > ššššat /usr/src/sys/ddb/db_main.c:251 > #5 0xffffffff80a270f3 in kdb_trap (type=3, code=0, tf=) > ššššat /usr/src/sys/kern/subr_kdb.c:654 > #6 0xffffffff80db6668 in trap (frame=0xfffffe060bdde1d0) > ššššat /usr/src/sys/amd64/amd64/trap.c:549 > #7 0xffffffff80d961f7 in calltrap () > ššššat /usr/src/sys/amd64/amd64/exception.S:234 > #8 0xffffffff80a267db in kdb_enter (why=0xffffffff812a5566 "panic", > ššššmsg=0x80
) at cpufunc.h:63 > #9 0xffffffff809ea01f in vpanic (fmt=, > ššššap=) at /usr/src/sys/kern/kern_shutdown.c:750 > #10 0xffffffff809e9e76 in kassert_panic (fmt=) > ššššat /usr/src/sys/kern/kern_shutdown.c:647 > #11 0xffffffff80c2a788 in ipfw_rewrite_rule_uidx (chain=0xffffffff81be5310, > ššššci=0xfffffe060bdde4b8) at /usr/src/sys/netpfil/ipfw/ip_fw_table.c:3395 > #12 0xffffffff80c267c3 in commit_rules (chain=0xffffffff81be5310, > ššššrci=0xfffffe060bdde4b8, count=1) > ššššat /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:678 > #13 0xffffffff80c25d80 in add_rules (chain=0xffffffff81be5310, > ššššop3=, sd=) > ššššat /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:2594 > #14 0xffffffff80c232f4 in ipfw_ctl3 (sopt=0xfffffe060bdde920) > ššššat /usr/src/sys/netpfil/ipfw/ip_fw_sockopt.c:3242 > #15 0xffffffff80b3d8b1 in rip_ctloutput (so=, > ššššsopt=0xfffffe060bdde920) at /usr/src/sys/netinet/raw_ip.c:588 > #16 0xffffffff80a72bc6 in sogetopt (so=0xfffff80009e658b8, > ššššsopt=0xfffffe060bdde920) at /usr/src/sys/kern/uipc_socket.c:2731 > #17 0xffffffff80a7729e in kern_getsockopt (td=0xfffff800098119a0, > ššššs=, level=, > ššššname=, val=, valseg=464, > ššššvalsize=0xfffffe060bdde98c) at /usr/src/sys/kern/uipc_syscalls.c:1540 > #18 0xffffffff80a771a0 in sys_getsockopt (td=0xfffff800098119a0, > ššššuap=0xfffffe060bddea40) at /usr/src/sys/kern/uipc_syscalls.c:1486 > #19 0xffffffff80db7519 in amd64_syscall (td=0xfffff800098119a0, traced=0) > ššššat subr_syscall.c:140 > #20 0xffffffff80d964db in Xfast_syscall () > ššššat /usr/src/sys/amd64/amd64/exception.S:394 > #21 0x0000000800b2cbea in ?? () > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) > > I've copied the vmcore.z & core.txt.7 to > ; gzipped > copies are also available: > > ššššššššššššššššššššIndex of /~david/FreeBSD/head/ipfw > > šIcon Name Last modified Size Description > šš_____________________________________________________________________ > š[PARENTDIR] Parent Directory - > š[TXT] core.txt.7 2015-11-03 05:22 155K > š[ ] core.txt.7.gz 2015-11-03 05:22 35K > š[ ] vmcore.7 2015-11-03 05:22 528M > š[ ] vmcore.7.gz 2015-11-03 05:22 45M > šš_____________________________________________________________________ > > I'll start taking a closer look at recent changes (e.g., in > src/sys/netpfil/ipfw), but I'm not really all that familiar with > the code. > > Peace, > david > -- > David H. Wolfskill david@catwhisker.org > Those who would murder in the name of God or prophet are blasphemous cowards. > > See http://www.catwhisker.org/~david/publickey.gpg for my public key. From owner-freebsd-ipfw@freebsd.org Tue Nov 3 16:39:54 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05FF4A24472 for ; Tue, 3 Nov 2015 16:39:54 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id DB04E17E9 for ; Tue, 3 Nov 2015 16:39:53 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id D8AB8A2446F; Tue, 3 Nov 2015 16:39:53 +0000 (UTC) Delivered-To: ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE154A2446B; Tue, 3 Nov 2015 16:39:53 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8A25417E8; Tue, 3 Nov 2015 16:39:53 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id tA3Gdqki029110; Tue, 3 Nov 2015 08:39:52 -0800 (PST) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id tA3GdqFK029109; Tue, 3 Nov 2015 08:39:52 -0800 (PST) (envelope-from david) Date: Tue, 3 Nov 2015 08:39:52 -0800 From: David Wolfskill To: "Alexander V. Chernikov" Cc: "ipfw@freebsd.org" , "current@freebsd.org" , "net@freebsd.org" Subject: Re: panic: refcount inconsistency: found: 0 total: 1 Message-ID: <20151103163952.GP21127@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , "Alexander V. Chernikov" , "ipfw@freebsd.org" , "current@freebsd.org" , "net@freebsd.org" References: <20151103140436.GJ21127@albert.catwhisker.org> <693401446563735@web30h.yandex.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="r/w8vo2lxBmCPGjQ" Content-Disposition: inline In-Reply-To: <693401446563735@web30h.yandex.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 16:39:54 -0000 --r/w8vo2lxBmCPGjQ Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 03, 2015 at 06:15:35PM +0300, Alexander V. Chernikov wrote: > ... > > I tried booting it, and during the transition to multi-user mode, > > once ipfw was being invoked, I got the above-cited panic. Circumvention > > was to leave it disconnected from a network (turn off the WiFi > > switch, in my case), so we don't get a chance to use the network. > It is most probably related with r290334. Would you mind reverting it and= checking if ipfw works correctly ? > .... OK; after: Script started on Tue Nov 3 08:22:37 2015 g1-252(10.2-S)[1] (cd /S4/usr/src/ && svn diff -c 290334 >/tmp/r290334)^M g1-252(10.2-S)[2] (cd /S4/usr/src/ && svn patch --reverse-diff /tmp/r290334= )^M U sys/netpfil/ipfw/ip_fw_sockopt.c U sys/netpfil/ipfw/ip_fw_table.c g1-252(10.2-S)[3] exit followed by a "make buildkernel", I now have: FreeBSD localhost 11.0-CURRENT FreeBSD 11.0-CURRENT #232 r290334M/290334:1= 100086: Tue Nov 3 08:27:20 PST 2015 root@localhost:/common/S4/obj/usr/= src/sys/CANARY amd64 and localhost(11.0-C)[3] ifconfig wlan0=20 wlan0: flags=3D8843 metric 0 mtu 15= 00 ether 00:24:d6:7a:03:ce inet 198.135.49.33 netmask 0xfffffc00 broadcast 198.135.51.255=20 =2E.. as I type, and IPFW is isn use at a *guess*, we'll need a bit more effort to keep "found" and "ci->object_opcodes" in sync, at least by the time the KASSERT on src/sys/netpfil/ipfw/ip_fw_table.c:3395 looks at the values. Thanks! Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --r/w8vo2lxBmCPGjQ Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJWOONYXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7VRIP/3i6WSUOZHXNnuYmo3Rj5PFK j9X1xP4PRIEYk5nKrrevEEvm7P4BfQ3snhK5dBPXbHwweIjYnzk0wdet38MKKX1u FSK+gJ0/asN0F2kCmudkLyZ3Es63ZdMMEKNRTrVyJQuVejSLLXCPfUJkEs9MSj/K H/TrqhRFMX3yw6PDuN7mumqVeiawylTKz3Cb5qFNcrJSRiY+1WxQRNQ5QJQCGSXx zo3ecXqmzjMm0eijrYV7TyIJ0HJKKa4vIsWu0wpejoBhBP8rJLyXwY26QZxS9WBE jvCApt2zkutYPN2uoq7w6V+Fsk6lN/t5TbQvtneUI2qMHjprRhfJ531i95RCYd98 wWaLe35WUXAA4OdE+zocscd8JF4+2Gn3Ug/Io5c+1IjKY1Tpgm2euQyWy+VD4DJ2 CzlCpwm+HN9REhK1iHfvXeEqT6LEQWd9KS14/AbUzXmXtyq9e/1OAWz/kPPERDQU 6mwSFGeoC+DDGrHyD+OQEbqLOeq+wWPy0dql8n53Y1CDuG/wthTjsxHvnF/4uANw Mv3P8JRVUVHLKscqCIAjqgW9W2wY+VaTG8LUS2cx0drEAlmoamXs+oD+P54HP4tk VDXMIYSJ8UQ3N3KXRTlkMFcGKiQQxososZaTpQiuhG9mEIXQN+WvnzmggoaPo2PU LqUrt3IGlvNmx80daG7F =Tmh8 -----END PGP SIGNATURE----- --r/w8vo2lxBmCPGjQ-- From owner-freebsd-ipfw@freebsd.org Tue Nov 3 21:10:06 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 51ADAA25C95 for ; Tue, 3 Nov 2015 21:10:06 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 3654E1030 for ; Tue, 3 Nov 2015 21:10:06 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id 30A0FA25C8F; Tue, 3 Nov 2015 21:10:06 +0000 (UTC) Delivered-To: ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2FF63A25C8B; Tue, 3 Nov 2015 21:10:06 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 75BDE102E; Tue, 3 Nov 2015 21:10:04 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id tA3LA1L6031313; Tue, 3 Nov 2015 13:10:01 -0800 (PST) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id tA3LA16e031312; Tue, 3 Nov 2015 13:10:01 -0800 (PST) (envelope-from david) Date: Tue, 3 Nov 2015 13:10:01 -0800 From: David Wolfskill To: "Alexander V. Chernikov" , "ipfw@freebsd.org" , "current@freebsd.org" , "net@freebsd.org" Subject: Re: panic: refcount inconsistency: found: 0 total: 1 Message-ID: <20151103211001.GT21127@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , "Alexander V. Chernikov" , "ipfw@freebsd.org" , "current@freebsd.org" , "net@freebsd.org" References: <20151103140436.GJ21127@albert.catwhisker.org> <693401446563735@web30h.yandex.ru> <20151103163952.GP21127@albert.catwhisker.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="lpUp1egui7PDlNtH" Content-Disposition: inline In-Reply-To: <20151103163952.GP21127@albert.catwhisker.org> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 21:10:06 -0000 --lpUp1egui7PDlNtH Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Nov 03, 2015 at 08:39:52AM -0800, David Wolfskill wrote: > On Tue, Nov 03, 2015 at 06:15:35PM +0300, Alexander V. Chernikov wrote: > > ... > > > I tried booting it, and during the transition to multi-user mode, > > > once ipfw was being invoked, I got the above-cited panic. Circumventi= on > > > was to leave it disconnected from a network (turn off the WiFi > > > switch, in my case), so we don't get a chance to use the network. > > It is most probably related with r290334. Would you mind reverting it a= nd checking if ipfw works correctly ? > > .... >=20 > ... [Reverting r290334 gets things working again -- dhw] >=20 > at a *guess*, we'll need a bit more effort to keep "found" and > "ci->object_opcodes" in sync, at least by the time the KASSERT on > src/sys/netpfil/ipfw/ip_fw_table.c:3395 looks at the values. > ... So... looking at the code a bit (and bearing in mind that I still am unfamiliar with said code, and I hack more than I "write" code)... here's the original bit of src/sys/netpfil/ipfw/ip_fw_table.c in question, with the modification from r290334 shown: =2E.. /* * Finds and bumps refcount for objects referenced by given @rule. * Auto-creates non-existing tables. * Fills in @oib array with userland/kernel indexes. * * Returns 0 on success. */ static int ref_rule_objects(struct ip_fw_chain *ch, struct ip_fw *rule, struct rule_check_info *ci, struct obj_idx *oib, struct tid_info *ti) { =2E.. if (error !=3D 0) { /* Unref everything we have already done */ unref_oib_objects(ch, rule->cmd, oib, pidx); IPFW_UH_WUNLOCK(ch); return (error); } IPFW_UH_WUNLOCK(ch); found =3D pidx - oib; KASSERT(found =3D=3D ci->object_opcodes, ("refcount inconsistency: found: %d total: %d", found, ci->object_opcodes)); =20 /* Perform auto-creation for non-existing objects */ if (numnew !=3D 0) error =3D create_objects_compat(ch, rule->cmd, oib, pidx, t= i); =20 + /* Calculate real number of dynamic objects */ + ci->object_opcodes =3D (uint16_t)(pidx - oib); + return (error); } =2E.. The added code to "Calculate real number of dynamic objects" is apparently setting ci->object_opcodes to the same value that "found" was just set to (pidx - oib -- cast to uint16_t in the case of the added code). But that appears to come a bit late for the KASSERT. Moving the KASSERT relative to the added code (so the KASSERT comes after) would be an option, but I'm not sure it makes sense to actually do the KASSERT unless we have some reason to believe that the difference between pidx and oib might change in the interval represented by a couple lines of code AND we have not coded to handle that situation any more "gracefully" than to panic. So... with the change from r290334, what's the point of the KASSERT? Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --lpUp1egui7PDlNtH Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJWOSKpXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7M3QP/RwN2RNOPwqO6jVxTs4fJa5e Z+53TPcDIL1EAtGS9/s5ENOduCEFFbDH/4rBg7HhP7mtRtpucWINRcel8/XF/ydU hRy+NAr/IncpTU/wWuq+af0FpwPiBWERSK92mneWoihrgpGyeDAYRhYftbkg9OkE OKlVYvXnZt6KPciC1ib6JZXvKKrQMKROHrjNmjnMUQe+q1ZuqvuW5oikTB8yesJa 1zQqUMDFdenIrzVv/9nINbCjAZtVNhdbsSfg2OfEnv6WpZE12DJkIKKlNvOH8w5n cfrfLeaAvHHif/CUUb8NO15E8D0GdXmDkAUUKi82BEldB337o6lbxWOffBlAym2L iY2Z0FUiUgCo04FMMlARKBu1ipuWMUSB6H4D15r28bdCXQWiprpT7fSZiBTph5jv fJnu/mkxTEZfsste7JfJJkGPG8EOEY2QhnYeDkdyvrDvXcDUHeuSySPHcea3SW1B t7w2gvjONqmInQ/pD3pF1N5BD8pNKPu6qIfFujNrl/TDBdsihjYNcZIn4VwfM5KU gocdPXhusTjbzO/BkWXqs1SddcZtrQHnY6WZKmKK5tusQuhujOlmy+NJlx6ZjU8A fY07aYdRouIbu2MUPbrCeeyNY/axh3NTPZXnA8U3goUbHtGMqijTUxBhiWYzNOPG VEg8C/HMAia4t4HrsEGV =B1g6 -----END PGP SIGNATURE----- --lpUp1egui7PDlNtH-- From owner-freebsd-ipfw@freebsd.org Tue Nov 3 22:25:45 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59C3BA2508B for ; Tue, 3 Nov 2015 22:25:45 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 1DE961036 for ; Tue, 3 Nov 2015 22:25:45 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: by mailman.ysv.freebsd.org (Postfix) id 162E5A25088; Tue, 3 Nov 2015 22:25:45 +0000 (UTC) Delivered-To: ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 152F7A25084; Tue, 3 Nov 2015 22:25:45 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward13j.cmail.yandex.net (forward13j.cmail.yandex.net [5.255.227.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "forwards.mail.yandex.net", Issuer "Certum Level IV CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B8B9F1032; Tue, 3 Nov 2015 22:25:44 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from smtp14.mail.yandex.net (smtp14.mail.yandex.net [IPv6:2a02:6b8:0:801:1::13]) by forward13j.cmail.yandex.net (Yandex) with ESMTP id 02F08217BC; Wed, 4 Nov 2015 01:25:39 +0300 (MSK) Received: from smtp14.mail.yandex.net (localhost [127.0.0.1]) by smtp14.mail.yandex.net (Yandex) with ESMTP id 196781B603EF; Wed, 4 Nov 2015 01:25:38 +0300 (MSK) Received: by smtp14.mail.yandex.net (nwsmtp/Yandex) with ESMTPSA id EQzKOFqcSY-Pc24eGDs; Wed, 4 Nov 2015 01:25:38 +0300 (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client certificate not present) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1446589538; bh=Tx0Fm1r5SO7kVas2ANGmmPSDsQ+/YIUunixGAth8LbQ=; h=Subject:To:References:From:Message-ID:Date:User-Agent: MIME-Version:In-Reply-To:Content-Type; b=ULJ+36GVO2hu6EFCmND5ix/iEiGvhiRFjQp2GK6y5OP2K98LZVvMb7PFWxSqQ+P16 Bdtk9jCNru6sNTnFY48Do1MxR1IeJW5VWeijFtq7WH3wGU4Asz1IIXi3671mRub9aV Xl6qGHtIhlcEF8W5rESVjoc3tSmQOmXWiDeGIzeM= Authentication-Results: smtp14.mail.yandex.net; dkim=pass header.i=@yandex.ru X-Yandex-ForeignMX: US Subject: Re: panic: refcount inconsistency: found: 0 total: 1 To: David Wolfskill , "Alexander V. Chernikov" , "ipfw@freebsd.org" , "current@freebsd.org" , "net@freebsd.org" References: <20151103140436.GJ21127@albert.catwhisker.org> <693401446563735@web30h.yandex.ru> <20151103163952.GP21127@albert.catwhisker.org> <20151103211001.GT21127@albert.catwhisker.org> From: "Andrey V. Elsukov" Message-ID: <563933EE.1050909@yandex.ru> Date: Wed, 4 Nov 2015 01:23:42 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:38.0) Gecko/20100101 Thunderbird/38.2.0 MIME-Version: 1.0 In-Reply-To: <20151103211001.GT21127@albert.catwhisker.org> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="haPd523UHhOx2ICdN2cukHB5n3APTq374" X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 03 Nov 2015 22:25:45 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --haPd523UHhOx2ICdN2cukHB5n3APTq374 Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: quoted-printable On 04.11.2015 00:10, David Wolfskill wrote: > So... with the change from r290334, what's the point of the KASSERT? Yes, you are right. We changed this code and use it some time, but without INVARIANTS. I just removed this KASSERT in r290345. Can you try this revision? Sorry for the breakage. --=20 WBR, Andrey V. Elsukov --haPd523UHhOx2ICdN2cukHB5n3APTq374 Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBCAAGBQJWOTPuAAoJEAHF6gQQyKF6kP4H/iymjkU5mWjJ9OsL5FLGaR94 Ai7qyV0gAHvVgl6Wj0Te5uYDoO7XyHSzGSQQwE3EFlfQpouyzveXgAqOQl82KZe3 AZh8DZToiV19S4HT1bgNgb/QKD1wrdEywCCiJ9hZtAOMa0nT4nZAihr7FpPSFkBz V7ML2aKlgUJ1P5GhweHWTblu9jFhthm/E6OzPgvkL8AySEXx+hH2YcLvRS3k1WsC 2NW24HRspWPjkKrd4gCnXCs5j4bTvfQzZ3QW2Kf15ckrnIamvuYngd3X+SrjVZwf DVkOrPcENvAhDAUlF8oTUiH70f3V5p8loDUcBciJbmnzfbMG94DHcmTOusiRtJs= =1JOj -----END PGP SIGNATURE----- --haPd523UHhOx2ICdN2cukHB5n3APTq374-- From owner-freebsd-ipfw@freebsd.org Wed Nov 4 02:33:28 2015 Return-Path: Delivered-To: freebsd-ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 75ACCA25EAC for ; Wed, 4 Nov 2015 02:33:28 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 5005F18C2 for ; Wed, 4 Nov 2015 02:33:28 +0000 (UTC) (envelope-from david@catwhisker.org) Received: by mailman.ysv.freebsd.org (Postfix) id 4F084A25EA8; Wed, 4 Nov 2015 02:33:28 +0000 (UTC) Delivered-To: ipfw@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3336BA25EA5; Wed, 4 Nov 2015 02:33:28 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 866DF18BE; Wed, 4 Nov 2015 02:33:26 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id tA42XJmB033986; Tue, 3 Nov 2015 18:33:19 -0800 (PST) (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id tA42XJce033985; Tue, 3 Nov 2015 18:33:19 -0800 (PST) (envelope-from david) Date: Tue, 3 Nov 2015 18:33:19 -0800 From: David Wolfskill To: "Andrey V. Elsukov" Cc: "Alexander V. Chernikov" , "ipfw@freebsd.org" , "current@freebsd.org" , "net@freebsd.org" Subject: Re: panic: refcount inconsistency: found: 0 total: 1 Message-ID: <20151104023319.GZ21127@albert.catwhisker.org> Mail-Followup-To: David Wolfskill , "Andrey V. Elsukov" , "Alexander V. Chernikov" , "ipfw@freebsd.org" , "current@freebsd.org" , "net@freebsd.org" References: <20151103140436.GJ21127@albert.catwhisker.org> <693401446563735@web30h.yandex.ru> <20151103163952.GP21127@albert.catwhisker.org> <20151103211001.GT21127@albert.catwhisker.org> <563933EE.1050909@yandex.ru> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="6ysXqiu0yoUmUNJB" Content-Disposition: inline In-Reply-To: <563933EE.1050909@yandex.ru> User-Agent: Mutt/1.5.24 (2015-08-30) X-BeenThere: freebsd-ipfw@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: IPFW Technical Discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 04 Nov 2015 02:33:28 -0000 --6ysXqiu0yoUmUNJB Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Nov 04, 2015 at 01:23:42AM +0300, Andrey V. Elsukov wrote: > On 04.11.2015 00:10, David Wolfskill wrote: > > So... with the change from r290334, what's the point of the KASSERT? >=20 > Yes, you are right. We changed this code and use it some time, but > without INVARIANTS. I just removed this KASSERT in r290345. Can you try > this revision? Sorry for the breakage. > .... OK; I first did "svn revert /usr/src/sys/netpfil/ipfw/", then (having saved a copy of r290345's commit message as /tmp/r290345), did "svn patch --strip 1 /tmp/r290345 /usr/src", then built & installed a new kernel, which booted without a panic: FreeBSD 11.0-CURRENT FreeBSD 11.0-CURRENT #233 r290334M/290334:1100086: T= ue Nov 3 17:18:26 PST 2015 root@localhost:/common/S4/obj/usr/src/sys/C= ANARY amd64 I was un the shuttle bus at the time; apparently the DHCP server in the access point wasn't behaving itself, as I got a DHCPNAK from it -- running either head or stable/10. I booted up head here at home, and DHCP works fine: FreeBSD 11.0-CURRENT FreeBSD 11.0-CURRENT #233 r290334M/290334:1100086: T= ue Nov 3 17:18:26 PST 2015 root@localhost:/common/S4/obj/usr/src/sys/C= ANARY amd64 So: looks like awin to me! :-) Peace, david --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --=20 David H. Wolfskill david@catwhisker.org Those who would murder in the name of God or prophet are blasphemous coward= s. See http://www.catwhisker.org/~david/publickey.gpg for my public key. --6ysXqiu0yoUmUNJB Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- Version: GnuPG v2 iQJ8BAEBCgBmBQJWOW5vXxSAAAAAAC4AKGlzc3Vlci1mcHJAbm90YXRpb25zLm9w ZW5wZ3AuZmlmdGhob3JzZW1hbi5uZXQ4RThEMDY4QTIxMjc1MDZFRDIzODYzRTc4 QTY3RjlDOERFRjQxOTNCAAoJEIpn+cje9Bk7FtgP/1VtwOEkCY2jQQm02mALkh6M vjUaVhdV8EUw3u3Y7ziuuCDEsnSb2raDl0TQDa3pPcRWjRkPGfHpA1Ep1XbIOMLl QvckSq5i9QcKXHBkpxWrm1xXfIvE2tQOM3eJfJWGNH2+eGA5+6qaH1wANqYOEAV3 /oqZixR5KXm9PiowXs/QHWjfxHw2hi3dm51fzwNWurTdtJKvilNeHsdaxiDiL00/ Obnmrv/tIc7OLU/dOWNUtj7TdBjGKeCgUGtsNXI7UoFi9JIE83X8/3VTjvnH1RD5 CyCjXZrPL2Gfu1y7ZAZ0QWiSRY+foug5FKmZcNw68BMljk828eMyBSVLTkbPDsV5 k6QD5ZDZIIFTe3udyPzPUyTjoxEbH/mXONLHlcRt0ZMW4C38hXO6aRhSu+wd/FIH LVOqFB8OlJvWiMqJu5HIQrw53c5tACCbyrxdd3vmr5h3BW009+4bbuzp78juLrKd JKC6YkI9DSHp0Gui9ucQ7fXZZun6VAaUG7NwZnicwPbj2e3FSEHd4/GDIPIh4A7y goTIZFqXuWV2ENGa7tY7k6IdFzA7YEwEA96kiGsbb0IOF4ZF699P4L3l3cuwn9GX welGKUE4+L0zr5eElJ7qfdBr5rSDBAJWGYqWcDHwJYe6v3RWnpnID4aYaa0zCOb/ VY5VDVt0I++PdAcyMRZO =Wrhk -----END PGP SIGNATURE----- --6ysXqiu0yoUmUNJB--