From owner-freebsd-net@FreeBSD.ORG Thu Dec 9 17:28:25 2004 Return-Path: Delivered-To: freebsd-net@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9F91916A4CE for ; Thu, 9 Dec 2004 17:28:25 +0000 (GMT) Received: from parrot.aev.net (host29-15.pool8174.interbusiness.it [81.74.15.29]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8DD0643D48 for ; Thu, 9 Dec 2004 17:28:22 +0000 (GMT) (envelope-from andrea.venturoli@netfence.it) Received: from soth.ventu (adsl-ull-78-5.41-151.net24.it [151.41.5.78]) (authenticated bits=128) by parrot.aev.net (8.13.1/8.13.1) with ESMTP id iB9HZBBd073707 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Thu, 9 Dec 2004 18:35:18 +0100 (CET) (envelope-from andrea.venturoli@netfence.it) Received: from netfence.it (xanatar.ventu [10.1.2.6]) by soth.ventu (8.13.1/8.12.10) with ESMTP id iB9HQaIf076903; Thu, 9 Dec 2004 18:26:36 +0100 (CET) (envelope-from andrea.venturoli@netfence.it) Message-ID: <41B88B45.4040407@netfence.it> Date: Thu, 09 Dec 2004 18:28:37 +0100 From: Andrea Venturoli Organization: NetFence User-Agent: Mozilla/5.0 (OS/2; U; Warp 4.5; en-US; rv:1.6) Gecko/20040117 X-Accept-Language: it,en,fr,de MIME-Version: 1.0 To: freebsd-net@freebsd.org Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg=sha1; boundary="------------ms000105030507000008060907" X-Scanned-By: MIMEDefang 2.45 Subject: panic with 4.10p4 and ipfw2 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 09 Dec 2004 17:28:25 -0000 This is a cryptographically signed message in MIME format. --------------ms000105030507000008060907 Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Hello. A box of mine, which acts as firewall/bridge, is experiencing frequent panics. As said in the subject line, it's a 4.10-RELEASE-p4 with ipfw2 enabled in the kernel. I've run through post mortem kernel analisys and found out that the crashes are always related to ipfw2; specifically I get: > panic: free: multiple frees Here is the complete backtrack: > #0 dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487 > #1 0xc0150993 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:316 > #2 0xc0150db8 in poweroff_wait (junk=0xc02354ac, howto=-1071427665) > at /usr/src/sys/kern/kern_shutdown.c:595 > #3 0xc0208a3e in trap_fatal (frame=0xc023a3e4, eva=48) > at /usr/src/sys/i386/i386/trap.c:974 > #4 0xc0208711 in trap_pfault (frame=0xc023a3e4, usermode=0, eva=48) > at /usr/src/sys/i386/i386/trap.c:867 > #5 0xc02082fb in trap (frame={tf_fs = 16, tf_es = 16, tf_ds = 16, tf_edi = 0, > tf_esi = 0, tf_ebp = -1071406036, tf_isp = -1071406064, > tf_ebx = -1071330820, tf_edx = 6864896, tf_ecx = -1054588914, > tf_eax = 0, tf_trapno = 12, tf_err = 0, tf_eip = -1071892584, tf_cs = 8, > tf_eflags = 66182, tf_esp = -967647568, tf_ss = 0}) > at /usr/src/sys/i386/i386/trap.c:466 > #6 0xc01c3798 in acquire_lock (lk=0xc024c9fc) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:266 > #7 0xc01c8e7c in softdep_count_dependencies (bp=0xc652deb0, wantcount=0) > at /usr/src/sys/ufs/ffs/ffs_softdep.c:4792 > #8 0xc01cc0d8 in ffs_fsync (ap=0xc023a4a0) > at /usr/src/sys/ufs/ffs/ffs_vnops.c:168 > #9 0xc01cabab in ffs_sync (mp=0xc123fc00, waitfor=2, cred=0xc0a3e800, > p=0xc026dbe0) at vnode_if.h:558 > #10 0xc0181737 in sync (p=0xc026dbe0, uap=0x0) > at /usr/src/sys/kern/vfs_syscalls.c:583 > #11 0xc015072e in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:235 > #12 0xc0150db8 in poweroff_wait (junk=0xc0218cff, howto=-1051816704) > at /usr/src/sys/kern/kern_shutdown.c:595 > #13 0xc014c41f in free (addr=0xc18fc100, type=0xc0249420) > at /usr/src/sys/kern/kern_malloc.c:385 > #14 0xc01a56ce in lookup_dyn_rule (pkt=0xc023a650, match_direction=0xc023a5c8, > tcp=0xc0b26b50) at /usr/src/sys/netinet/ip_fw2.c:784 > #15 0xc01a6ae7 in ipfw_chk (args=0xc023a630) > at /usr/src/sys/netinet/ip_fw2.c:1900 > #16 0xc01aa5f5 in ip_output (m0=0xc0b26b00, opt=0x0, ro=0xd0bfb0fc, flags=0, > imo=0x0, inp=0xd0bfb0c0) at /usr/src/sys/netinet/ip_output.c:733 > #17 0xc01afc51 in tcp_output (tp=0xd0bfb180) > at /usr/src/sys/netinet/tcp_output.c:953 > #18 0xc01ae977 in tcp_input (m=0xc0b26b00, off0=20, proto=6) > at /usr/src/sys/netinet/tcp_input.c:2229 > #19 0xc01a8f1c in ip_input (m=0xc0b26b00) > at /usr/src/sys/netinet/ip_input.c:934 > #20 0xc01a8f7b in ipintr () at /usr/src/sys/netinet/ip_input.c:955 > #21 0xc01fbd89 in swi_net_next () > #22 0xc0156a69 in softclock () at /usr/src/sys/kern/kern_timeout.c:131 > #23 0xc01fbd43 in doreti_swi () So, free is called from the following fragment: > /** > * lookup a dynamic rule. > */ > static ipfw_dyn_rule * > lookup_dyn_rule(struct ipfw_flow_id *pkt, int *match_direction, > struct tcphdr *tcp) > { > /* > * stateful ipfw extensions. > * Lookup into dynamic session queue > */ > #define MATCH_REVERSE 0 > #define MATCH_FORWARD 1 > #define MATCH_NONE 2 > #define MATCH_UNKNOWN 3 > int i, dir = MATCH_NONE; > ipfw_dyn_rule *prev, *q=NULL; > > if (ipfw_dyn_v == NULL) > goto done; /* not found */ > i = hash_packet( pkt ); > for (prev=NULL, q = ipfw_dyn_v[i] ; q != NULL ; ) { > if (q->dyn_type == O_LIMIT_PARENT && q->count) > goto next; > if (TIME_LEQ( q->expire, time_second)) { /* expire entry */ > => UNLINK_DYN_RULE(prev, ipfw_dyn_v[i], q); > continue; > } > if (pkt->proto == q->id.proto && > q->dyn_type != O_LIMIT_PARENT) { I'm no kernel expert, so take my observation for what they might be worth, but: > (kgdb) p *q > $24 = {next = 0xc18a2d00, rule = 0xc6523b3c, parent = 0xd0001, > pcnt = 13916504069872025600, bcnt = 11709303859986432, id = {dst_ip = 0, > src_ip = 0, dst_port = 15744, src_port = 49469, proto = 164 '\244', > flags = 129 '\201'}, expire = 0, bucket = 4294967295, state = 4294967295, > ack_fwd = 0, ack_rev = 0, dyn_type = 0, count = 0} > (kgdb) These values do not make much sense to me... maybe the mess has already happened? Any hint? Is ipfw2 known to be broken in 4_10? Should I upgrade to 4_STABLE? Or is it just a matter of finding a better configuration for all the relevant sysctl (which are all set to their default values)? Really any help is appreciated!!! bye & Thanks av. --------------ms000105030507000008060907 Content-Type: application/x-pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExCzAJBgUrDgMCGgUAMIAGCSqGSIb3DQEHAQAAoIIHdDCC A7YwggMfoAMCAQICAQQwDQYJKoZIhvcNAQEEBQAwgZExCzAJBgNVBAYTAklUMRAwDgYDVQQI EwdCb2xvZ25hMRAwDgYDVQQHEwdCb2xvZ25hMREwDwYDVQQKEwhOZXRGZW5jZTEkMCIGA1UE AxMbTmV0RmVuY2UgQ2VydGlmaWNhdGUgTWFzdGVyMSUwIwYJKoZIhvcNAQkBFhZwb3N0bWFz dGVyQG5ldGZlbmNlLml0MB4XDTA0MDcwNjE0NTczM1oXDTA3MDcwNjE0NTczM1owgY4xCzAJ BgNVBAYTAklUMRAwDgYDVQQIEwdCb2xvZ25hMRIwEAYDVQQHEwlWaWxsYW5vdmExETAPBgNV BAoTCE5ldEZlbmNlMRkwFwYDVQQDExBBbmRyZWEgVmVudHVyb2xpMSswKQYJKoZIhvcNAQkB FhxhbmRyZWEudmVudHVyb2xpQG5ldGZlbmNlLml0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCB iQKBgQDNut5pvhl9kcTaS4DwvTZA7I6G2cuNTi0zVYf3ObXn+eV5KPrUe+iIj5tsJHhZSFie Mmr/sU8+tmtL9w4F+IYqoJZMrj3pyBLwUuyG/UCA97j7iOunDXdQaezt3RjPTHDgZ1Estw42 O/tdOksw1RGz/Pcl1nYDAx3Qxb1s+CQDwwIDAQABo4IBHTCCARkwCQYDVR0TBAIwADAsBglg hkgBhvhCAQ0EHxYdT3BlblNTTCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFL+G 85rXpNrBqicRBxrsKKXAbTqxMIG+BgNVHSMEgbYwgbOAFGuQcrsEAmOxnk0uGCXT6UCTGFCD oYGXpIGUMIGRMQswCQYDVQQGEwJJVDEQMA4GA1UECBMHQm9sb2duYTEQMA4GA1UEBxMHQm9s b2duYTERMA8GA1UEChMITmV0RmVuY2UxJDAiBgNVBAMTG05ldEZlbmNlIENlcnRpZmljYXRl IE1hc3RlcjElMCMGCSqGSIb3DQEJARYWcG9zdG1hc3RlckBuZXRmZW5jZS5pdIIBADANBgkq hkiG9w0BAQQFAAOBgQCmYkfL9jqgRsIobKrZstcYZZXA0hMVKfMwUYddCKWJYjpoynV7k5pC CKo8ZLdfj9mhCnOkGLbhVpQL7ySMzAH5IbD2qgowAf06zV3rsmzmspw5iyL6iZQsJpoT+owg FtSpw2UgUuhWmq004PqgasdzlaJg3LRS1ACHmyPZKRUYaTCCA7YwggMfoAMCAQICAQQwDQYJ KoZIhvcNAQEEBQAwgZExCzAJBgNVBAYTAklUMRAwDgYDVQQIEwdCb2xvZ25hMRAwDgYDVQQH EwdCb2xvZ25hMREwDwYDVQQKEwhOZXRGZW5jZTEkMCIGA1UEAxMbTmV0RmVuY2UgQ2VydGlm aWNhdGUgTWFzdGVyMSUwIwYJKoZIhvcNAQkBFhZwb3N0bWFzdGVyQG5ldGZlbmNlLml0MB4X DTA0MDcwNjE0NTczM1oXDTA3MDcwNjE0NTczM1owgY4xCzAJBgNVBAYTAklUMRAwDgYDVQQI EwdCb2xvZ25hMRIwEAYDVQQHEwlWaWxsYW5vdmExETAPBgNVBAoTCE5ldEZlbmNlMRkwFwYD VQQDExBBbmRyZWEgVmVudHVyb2xpMSswKQYJKoZIhvcNAQkBFhxhbmRyZWEudmVudHVyb2xp QG5ldGZlbmNlLml0MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDNut5pvhl9kcTaS4Dw vTZA7I6G2cuNTi0zVYf3ObXn+eV5KPrUe+iIj5tsJHhZSFieMmr/sU8+tmtL9w4F+IYqoJZM rj3pyBLwUuyG/UCA97j7iOunDXdQaezt3RjPTHDgZ1Estw42O/tdOksw1RGz/Pcl1nYDAx3Q xb1s+CQDwwIDAQABo4IBHTCCARkwCQYDVR0TBAIwADAsBglghkgBhvhCAQ0EHxYdT3BlblNT TCBHZW5lcmF0ZWQgQ2VydGlmaWNhdGUwHQYDVR0OBBYEFL+G85rXpNrBqicRBxrsKKXAbTqx MIG+BgNVHSMEgbYwgbOAFGuQcrsEAmOxnk0uGCXT6UCTGFCDoYGXpIGUMIGRMQswCQYDVQQG EwJJVDEQMA4GA1UECBMHQm9sb2duYTEQMA4GA1UEBxMHQm9sb2duYTERMA8GA1UEChMITmV0 RmVuY2UxJDAiBgNVBAMTG05ldEZlbmNlIENlcnRpZmljYXRlIE1hc3RlcjElMCMGCSqGSIb3 DQEJARYWcG9zdG1hc3RlckBuZXRmZW5jZS5pdIIBADANBgkqhkiG9w0BAQQFAAOBgQCmYkfL 9jqgRsIobKrZstcYZZXA0hMVKfMwUYddCKWJYjpoynV7k5pCCKo8ZLdfj9mhCnOkGLbhVpQL 7ySMzAH5IbD2qgowAf06zV3rsmzmspw5iyL6iZQsJpoT+owgFtSpw2UgUuhWmq004Pqgasdz laJg3LRS1ACHmyPZKRUYaTGCA0swggNHAgEBMIGXMIGRMQswCQYDVQQGEwJJVDEQMA4GA1UE CBMHQm9sb2duYTEQMA4GA1UEBxMHQm9sb2duYTERMA8GA1UEChMITmV0RmVuY2UxJDAiBgNV BAMTG05ldEZlbmNlIENlcnRpZmljYXRlIE1hc3RlcjElMCMGCSqGSIb3DQEJARYWcG9zdG1h c3RlckBuZXRmZW5jZS5pdAIBBDAJBgUrDgMCGgUAoIICCTAYBgkqhkiG9w0BCQMxCwYJKoZI hvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0wNDEyMDkxNzI4MzdaMCMGCSqGSIb3DQEJBDEWBBRp WQVh/yO00Tcqkmx8QypKk6IP9DBSBgkqhkiG9w0BCQ8xRTBDMAoGCCqGSIb3DQMHMA4GCCqG SIb3DQMCAgIAgDANBggqhkiG9w0DAgIBQDAHBgUrDgMCBzANBggqhkiG9w0DAgIBKDCBqAYJ KwYBBAGCNxAEMYGaMIGXMIGRMQswCQYDVQQGEwJJVDEQMA4GA1UECBMHQm9sb2duYTEQMA4G A1UEBxMHQm9sb2duYTERMA8GA1UEChMITmV0RmVuY2UxJDAiBgNVBAMTG05ldEZlbmNlIENl cnRpZmljYXRlIE1hc3RlcjElMCMGCSqGSIb3DQEJARYWcG9zdG1hc3RlckBuZXRmZW5jZS5p dAIBBDCBqgYLKoZIhvcNAQkQAgsxgZqggZcwgZExCzAJBgNVBAYTAklUMRAwDgYDVQQIEwdC b2xvZ25hMRAwDgYDVQQHEwdCb2xvZ25hMREwDwYDVQQKEwhOZXRGZW5jZTEkMCIGA1UEAxMb TmV0RmVuY2UgQ2VydGlmaWNhdGUgTWFzdGVyMSUwIwYJKoZIhvcNAQkBFhZwb3N0bWFzdGVy QG5ldGZlbmNlLml0AgEEMA0GCSqGSIb3DQEBAQUABIGAcxFATotdGXQj6j/sPjSigS4GnkMB 97BRnUtslBCQLXRX2k6MACIfmqFInFIKy5xyFojwqJpS5YYHsAwhTm+Q/zZ8tZNmOlUYmPRL ANyMi6Xs81jwe62a3IFNUBaJ06RKf5JJV2Ru1yoCw0mXeuleGnxeUro/3kwEXNo2cxQjCnMA AAAAAAA= --------------ms000105030507000008060907--