Date: Mon, 11 Sep 2006 09:54:31 +0200 (CEST) From: Anders Nordby <anders@FreeBSD.org> To: FreeBSD-gnats-submit@FreeBSD.org Cc: tmseck@netcologne.de, adrian@creative.net.au Subject: threads/103127: Kernel panic while using thread features in Squid 2.6 Message-ID: <20060911075431.D12758D9874@fupp.net> Resent-Message-ID: <200609110800.k8B80ic9041871@freefall.freebsd.org>
next in thread | raw e-mail | index | archive | help
>Number: 103127
>Category: threads
>Synopsis: Kernel panic while using thread features in Squid 2.6
>Confidential: no
>Severity: serious
>Priority: medium
>Responsible: freebsd-threads
>State: open
>Quarter:
>Keywords:
>Date-Required:
>Class: sw-bug
>Submitter-Id: current-users
>Arrival-Date: Mon Sep 11 08:00:44 GMT 2006
>Closed-Date:
>Last-Modified:
>Originator: Anders Nordby <anders@FreeBSD.org>
>Release: FreeBSD 6.1-RELEASE-p5 i386
>Organization:
-
>Environment:
System: FreeBSD cache3.foobar.no 6.1-RELEASE-p5 FreeBSD 6.1-RELEASE-p5 #0: Thu Sep 7 15:53:24 CEST 2006 root@cache3.foobar.no:/usr/obj/usr/src/sys/CACHE i386
Squid Cache: Version 2.6.STABLE3
configure options: '--bindir=/usr/local/sbin' '--sbindir=/usr/local/sbin' '--datadir=/usr/local/etc/squid' '--libexecdir=/usr/local/libexec/squid' '--localstatedir=/usr/local/squid' '--sysconfdir=/usr/local/etc/squid' '--enable-removal-policies=lru heap' '--disable-linux-netfilter' '--disable-linux-tproxy' '--disable-epoll' '--enable-auth=basic ntlm digest' '--enable-basic-auth-helpers=NCSA PAM MSNT SMB YP' '--enable-digest-auth-helpers=password' '--enable-external-acl-helpers=ip_user session unix_group wbinfo_group' '--enable-ntlm-auth-helpers=SMB' '--enable-async-io' '--with-pthreads' '--enable-storeio=ufs diskd null aufs' '--enable-kqueue' '--enable-err-languages=Azerbaijani Bulgarian Catalan Czech Danish Dutch English Estonian Finnish French German Greek Hebrew Hungarian Italian Japanese Korean Lithuanian Polish Portuguese Romanian Russian-1251 Russian-koi8-r Serbian Simplify_Chinese Slovak Spanish Swedish Traditional_Chinese Turkish' '--enable-default-err-language=E
nglish' '--enable-dlmalloc' '--enable-snmp' '--enable-async-io=50' '--prefix=/usr/local' 'i386-portbld-freebsd6.1' 'LDFLAGS=' 'CFLAGS=-O2 -fno-strict-aliasing -pipe ' 'CPPFLAGS=' 'host_alias=i386-portbld-freebsd6.1' 'build_alias=i386-portbld-freebsd6.1' 'target_alias=i386-portbld-freebsd6.1' 'CC=cc'
>Description:
Recently I've been trying to switch to threads-based scheduling of
disk I/O requests, that is to use the aufs cache store type. It seems fast,
by using it Squid doesn't block on disk I/O so easily when under load.
Now and then I get a kernel panic, however:
root@cache3:/usr/src# kgdb /usr/obj/usr/src/sys/CACHE/kernel.debug /var/crash/vmcore.0
[GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 06
fault virtual address = 0x0
fault code = supervisor read, page not present
instruction pointer = 0x20:0x0
stack pointer = 0x28:0xeb9a7b98
frame pointer = 0x28:0xeb9a7bcc
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 542 (squid)
trap number = 12
panic: page fault
cpuid = 0
Uptime: 2d4h20m9s
Dumping 3839 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 3839MB (982778 pages) 3824 3808 3792 3776 3760 3744 3728 3712 3696 3680 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520 3504 3488 3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296 3280 3264 3248 3232 3216 3200 3184 3168 3152 3136 3120 3104 3088 3072 3056 3040 3024 3008 2992 2976 2960 2944 2928 2912 2896 2880 2864 2848 2832 2816 2800 2784 2768 2752 2736 2720 2704 2688 2672 2656 2640 2624 2608 2592 2576 2560 2544 2528 2512 2496 2480 2464 2448 2432 2416 2400 2384 2368 2352 2336 2320 2304 2288 2272 2256 2240 2224 2208 2192 2176 2160 2144 2128 2112 2096 2080 2064 2048 2032 2016 2000 1984 1968 1952 1936 1920 1904 1888 1872 1856 1840 1824 1808 1792 1776 1760 1744 1728 1712 1696 1680 1664 1648 1632 1616 1600 1584 1568 1552 1536 1520 1504 1488 1472 1456 1440 1424 1408 1392 1376 1360 1344 1328 1312 1296 1280 1264 1248 1232 1216 1200 1184 1168 1152 1136 1120 1104 1088 1072 1056 1040 1024 1008 992 976 960 944 928 912 896 880 864 848 832 816 800 784 768 752 736 720
704 688 672 656 640 624 608 592 576 560 544 528 512 496 480 464 448 432 416 400 384 368 352 336 320 304 288 272 256 240 224 208 192 176 160 144 128 112 96 80 64 48 32 16
#0 doadump () at pcpu.h:165
165 pcpu.h: No such file or directory.
in pcpu.h
(kgdb) bt
#0 doadump () at pcpu.h:165
#1 0xc0672901 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:402
#2 0xc0672c59 in panic (fmt=0xc087f113 "%s")
at /usr/src/sys/kern/kern_shutdown.c:558
#3 0xc08493e6 in trap_fatal (frame=0xeb9a7b58, eva=0)
at /usr/src/sys/i386/i386/trap.c:836
#4 0xc08490ef in trap_pfault (frame=0xeb9a7b58, usermode=0, eva=0)
at /usr/src/sys/i386/i386/trap.c:744
#5 0xc0848d05 in trap (frame=
{tf_fs = -916914168, tf_es = -916914136, tf_ds = -896335832, tf_edi = -917610112, tf_esi = -342197140, tf_ebp = -342197300, tf_isp = -342197372, tf_ebx = 4, tf_edx = -919717272, tf_ecx = -917610156, tf_eax = -1064162880, tf_trapno = 12, tf_err = 0, tf_eip = 0, tf_cs = 32, tf_eflags = 66118, tf_esp = -1067102273, tf_ss = -917610156}) at /usr/src/sys/i386/i386/trap.c:434
#6 0xc08361fa in calltrap () at /usr/src/sys/i386/i386/exception.s:139
#7 0x00000000 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb) quit
root@cache3:/usr/src# exit
The panics seems to happen when load is low as well as high.
>How-To-Repeat:
Install FreeBSD 6.1, and then ports/www/squid26:
cd /usr/ports/www/squid26
make SQUID_CONFIGURE_ARGS="--enable-dlmalloc --enable-snmp --enable-async-io=50"
WITH_SQUID_AUFS=yes OPTIONS="" install
Set up some aufs cache dirs on different disks (I am using SCSI disks in HP
Proliant DL 380):
cache_dir aufs /data/cache 15000 64 512
cache_dir aufs /data01/cache 15000 64 512
cache_dir aufs /data02/cache 15000 64 512
cache_dir aufs /data03/cache 15000 64 512
cache_dir aufs /data04/cache 15000 64 512
>Fix:
N/A
I am going to be trying libthr(3), meanwhile, to see if I can avoid the
problem then.
Let me know if there is anything more I can do to help debug the problem.
>Release-Note:
>Audit-Trail:
>Unformatted:
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20060911075431.D12758D9874>