From owner-freebsd-threads@FreeBSD.ORG Tue Sep 12 19:56:02 2006 Return-Path: X-Original-To: freebsd-threads@FreeBSD.org Delivered-To: freebsd-threads@FreeBSD.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4998216A407; Tue, 12 Sep 2006 19:56:02 +0000 (UTC) (envelope-from anders@FreeBSD.org) Received: from fupp.net (totem.fix.no [80.91.36.20]) by mx1.FreeBSD.org (Postfix) with ESMTP id F005C43D62; Tue, 12 Sep 2006 19:55:49 +0000 (GMT) (envelope-from anders@FreeBSD.org) Received: from localhost (totem.fix.no [80.91.36.20]) by fupp.net (Postfix) with ESMTP id 4BAB48D9896; Tue, 12 Sep 2006 21:55:48 +0200 (CEST) Received: from fupp.net ([80.91.36.20]) by localhost (totem.fix.no [80.91.36.20]) (amavisd-new, port 10024) with LMTP id 73858-01-4; Tue, 12 Sep 2006 21:55:47 +0200 (CEST) Received: by fupp.net (Postfix, from userid 1000) id AE26E8D9888; Tue, 12 Sep 2006 21:55:47 +0200 (CEST) Date: Tue, 12 Sep 2006 21:55:47 +0200 From: Anders Nordby To: FreeBSD-gnats-submit@FreeBSD.org, freebsd-threads@FreeBSD.org Message-ID: <20060912195547.GA71462@totem.fix.no> References: <20060911075431.D12758D9874@fupp.net> <200609110800.k8B80ie3041853@freefall.freebsd.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <200609110800.k8B80ie3041853@freefall.freebsd.org> X-PGP-Key: http://anders.fix.no/pgp/ X-PGP-Key-FingerPrint: 1E0F C53C D8DF 6A8F EAAD 19C5 D12A BC9F 0083 5956 User-Agent: Mutt/1.5.11 Cc: Suleiman Souhlal Subject: Re: threads/103127: Kernel panic while using thread features in Squid 2.6 X-BeenThere: freebsd-threads@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Threading on FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 12 Sep 2006 19:56:02 -0000 Using libthr instead of pthread didn't help at all. It might be a kqueue issue after all. On recommendation by Pawel Worach, I set these sysctls: debug.trace_on_panic=1 debug.debugger_on_panic=0 Then I could get more debugging info: # dmesg -M /var/crash/vmcore.0 -N /usr/obj/usr/src/sys/CACHE/kernel.debug (..) Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 06 fault virtual address = 0x0 fault code = supervisor read, page not present instruction pointer = 0x20:0x0 stack pointer = 0x28:0xeb886b98 frame pointer = 0x28:0xeb886bcc code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 545 (squid) trap number = 12 panic: page fault cpuid = 0 KDB: stack backtrace: kdb_backtrace(100,c9599480,28,eb886b58,c) at kdb_backtrace+0x29 panic(c087f113,c08ce651,0,fffff,c959d49b) at panic+0x114 trap_fatal(eb886b58,0,c9599480,c8f8a708,c) at trap_fatal+0x2ce trap_pfault(eb886b58,0,0) at trap_pfault+0x1d7 trap(8,c9590028,ce020028,c9feb0b4,eb886c30) at trap+0x2fd calltrap() at calltrap+0x5 --- trap 0xc, eip = 0, esp = 0xeb886b98, ebp = 0xeb886bcc --- (null)(c9560800,eb886c1c,c9599480,1,0) at 0 kern_kevent(c9599480,3,37,80,eb886cc8) at kern_kevent+0xc9 kevent(c9599480,eb886d04,6,4bacc,202) at kevent+0x55 syscall(3b,3b,bfbf003b,82180b1,0) at syscall+0x2bf Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (363, FreeBSD ELF32, kevent), eip = 0xb71e24af, esp = 0xbfbfe9cc, ebp = 0xbfbfea18 --- Uptime: 21h18m39s Dumping 3839 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 3839MB (982778 pages) 3824 3808 3792 3776 3760 3744 3728 3712 3696 3680 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520 3504 3488 3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296 3280 3264 3248 3232 3216 3200 3184 3168 3152 3136 3120 3104 3088 3072 3056 3040 3024 3008 2992 2976 2960 2944 2928 2912 2896 2880 2864 2848 2832 2816 2800 2784 2768 2752 2736 2720 2704 2688 2672 2656 2640 2624 2608 2592 2576 2560 2544 2528 2512 2496 2480 2464 2448 2432 2416 2400 2384 2368 2352 2336 2320 2304 2288 2272 2256 2240 2224 2208 2192 2176 2160 2144 2128 2112 2096 2080 2064 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720 704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16 Checking those kevent's, I get: (kgdb) l *kevent+0x55 0xc06542fd is in kevent (/usr/src/sys/kern/kern_event.c:571). 566 return (error); 567 tsp = &ts; 568 } else 569 tsp = NULL; 570 571 return (kern_kevent(td, uap->fd, uap->nchanges, uap->nevents, 572 &k_ops, tsp)); 573 } 574 575 /* (kgdb) l *kern_kevent+0xc9 0xc0654439 is in kern_kevent (/usr/src/sys/kern/kern_event.c:637). 632 goto done; 633 changes = keva; 634 for (i = 0; i < n; i++) { 635 kevp = &changes[i]; 636 kevp->flags &= ~EV_SYSFLAGS; 637 error = kqueue_register(kq, kevp, td, 1); 638 if (error) { 639 if (nevents != 0) { 640 kevp->flags = EV_ERROR; 641 kevp->data = error; (kgdb) This is in FreeBSD 6.1-RELEASE-p5. One physical processor, but two logical (HyperThreading enabled). Regards, Anders.