Date: Thu, 6 Nov 2008 11:29:31 +0100 From: Herve Boulouis <amon@aelita.org> To: freebsd-stable@freebsd.org Subject: Multiple panics with 7.1-PRERELEASE amd64 and varnish Message-ID: <20081106102931.GD596@ra.aabs>
next in thread | raw e-mail | index | archive | help
Hi,
Just put 3 boxes with varnish into production last night and I've got a few differents panics in 8hours. Traffic is very low
(each box generates 10 Mbit/s).
All boxes have the same 7.1-PRERELEASE (from around end of september) and 4GB of ram and varnish is launched with the following command line :
/usr/local/sbin/varnishd -f /usr/local/etc/varnish/default.vcl \
-u varnish \
-n /opt/varnish \
-s file,/opt/varnish/storage.bin,4G \
-P /var/run/varnishd.pid \
-t 7464960000s \
-w 32,4096,120 \
-p listen_depth=4096 \
-p thread_pool_min=32 \
-p thread_pool_max=4096 \
-p lru_interval=3600 \
-T 0.0.0.0:3000 \
-h classic,500009
Sysctl tunings :
www2:~# cat /etc/sysctl.conf
net.inet.tcp.tso=0
debug.minidump=1
net.inet.tcp.rfc1323=1
kern.ipc.maxsockbuf=1024000
net.inet.tcp.sendspace=512000
net.inet.tcp.recvspace=512000
www2:~# cat /boot/loader.conf
kern.ipc.nmbclusters=65536
kern.maxfiles=2097152
kern.maxfilesperproc=104856
kern.ipc.somaxconn=16384
net.inet.ip.portrange.last=65535
kern.threads.max_threads_per_proc=4096
kern.ipc.maxpipekva=104857600
What is strange is that those 3 boxes where hammered with siege and we had no problems
with them generating 900Mbit/s of traffic.
Are these panics known ? Can they be related to the sysctl ?
Do i need to CC freebsd-amd64 ?
Panic #1 (Box 2):
#2 0xffffffff803c6b8d in boot (howto=260) at ../../../kern/kern_shutdown.c:418
#3 0xffffffff803c6e48 in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:572
#4 0xffffffff806260b6 in trap_fatal (frame=0xffffffffcd816890, eva=138) at ../../../amd64/amd64/trap.c:764
#5 0xffffffff80626311 in trap_pfault (frame=0xffffffffcd816890, usermode=0) at ../../../amd64/amd64/trap.c:680
#6 0xffffffff80626c53 in trap (frame=0xffffffffcd816890) at ../../../amd64/amd64/trap.c:449
#7 0xffffffff8060d5be in calltrap () at ../../../amd64/amd64/exception.S:209
#8 0xffffffff80599dda in vm_object_clear_flag (object=0x0, bits=Variable "bits" is not available.
) at ../../../vm/vm_object.c:269
#9 0xffffffff804302a6 in cluster_wbuild (vp=0xffffff000eabfbd0, size=16384, start_lbn=58905, len=2) at ../../../kern/vfs_cluster.c:925
#10 0xffffffff804275de in vfs_bio_awrite (bp=0xffffffffa4d0b660) at ../../../kern/vfs_bio.c:1668
#11 0xffffffff8057800e in ffs_syncvnode (vp=0xffffff000eabfbd0, waitfor=Variable "waitfor" is not available.
) at ../../../ufs/ffs/ffs_vnops.c:283
#12 0xffffffff80573c91 in ffs_sync (mp=0xffffff000e5cc6f0, waitfor=3, td=0xffffff000e568a50) at ../../../ufs/ffs/ffs_vfsops.c:1234
#13 0xffffffff8043f821 in sync_fsync (ap=Variable "ap" is not available.
) at ../../../kern/vfs_subr.c:3217
#14 0xffffffff8065f552 in VOP_FSYNC_APV (vop=Variable "vop" is not available.
) at vnode_if.c:1007
#15 0xffffffff8043ff71 in sched_sync () at vnode_if.h:538
#16 0xffffffff803a6a01 in fork_exit (callout=0xffffffff8043f8f0 <sched_sync>, arg=0x0, frame=0xffffffffcd816c80) at ../../../kern/kern_fork.c:804
#17 0xffffffff8060d98e in fork_trampoline () at ../../../amd64/amd64/exception.S:455
Panic #2 (Box 2):
#2 0xffffffff803c6b8d in boot (howto=260) at ../../../kern/kern_shutdown.c:418
#3 0xffffffff803c6e48 in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:572
#4 0xffffffff806260b6 in trap_fatal (frame=0xffffffffcd816890, eva=138) at ../../../amd64/amd64/trap.c:764
#5 0xffffffff80626311 in trap_pfault (frame=0xffffffffcd816890, usermode=0) at ../../../amd64/amd64/trap.c:680
#6 0xffffffff80626c53 in trap (frame=0xffffffffcd816890) at ../../../amd64/amd64/trap.c:449
#7 0xffffffff8060d5be in calltrap () at ../../../amd64/amd64/exception.S:209
#8 0xffffffff80599dda in vm_object_clear_flag (object=0x0, bits=Variable "bits" is not available.
) at ../../../vm/vm_object.c:269
#9 0xffffffff804302a6 in cluster_wbuild (vp=0xffffff000ea4adc8, size=16384, start_lbn=20817, len=5) at ../../../kern/vfs_cluster.c:925
#10 0xffffffff804275de in vfs_bio_awrite (bp=0xffffffffa4d83660) at ../../../kern/vfs_bio.c:1668
#11 0xffffffff8057800e in ffs_syncvnode (vp=0xffffff000ea4adc8, waitfor=Variable "waitfor" is not available.
) at ../../../ufs/ffs/ffs_vnops.c:283
#12 0xffffffff80573c91 in ffs_sync (mp=0xffffff000e6c96f0, waitfor=3, td=0xffffff000e568a50) at ../../../ufs/ffs/ffs_vfsops.c:1234
#13 0xffffffff8043f821 in sync_fsync (ap=Variable "ap" is not available.
) at ../../../kern/vfs_subr.c:3217
#14 0xffffffff8065f552 in VOP_FSYNC_APV (vop=Variable "vop" is not available.
) at vnode_if.c:1007
#15 0xffffffff8043ff71 in sched_sync () at vnode_if.h:538
#16 0xffffffff803a6a01 in fork_exit (callout=0xffffffff8043f8f0 <sched_sync>, arg=0x0, frame=0xffffffffcd816c80) at ../../../kern/kern_fork.c:804
#17 0xffffffff8060d98e in fork_trampoline () at ../../../amd64/amd64/exception.S:455
Panic #3 (Box 1):
#3 0xffffffff803c6e48 in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:572
#4 0xffffffff8059d76b in vm_page_unwire (m=0xffffff00d9e3ff70, activate=0) at ../../../vm/vm_page.c:1410
#5 0xffffffff80429cc8 in vfs_vmio_release (bp=0xffffffffa4ca5860) at ../../../kern/vfs_bio.c:1539
#6 0xffffffff8042b72b in getnewbuf (slpflag=0, slptimeo=0, size=Variable "size" is not available.
) at ../../../kern/vfs_bio.c:1847
#7 0xffffffff8042ccbe in getblk (vp=0xffffff001c6493f0, blkno=0, size=2048, slpflag=Variable "slpflag" is not available.
) at ../../../kern/vfs_bio.c:2602
#8 0xffffffff8042d65c in breadn (vp=0xffffff001c6493f0, blkno=Variable "blkno" is not available.
) at ../../../kern/vfs_bio.c:786
#9 0xffffffff8042d759 in bread (vp=Variable "vp" is not available.
) at ../../../kern/vfs_bio.c:734
#10 0xffffffff80578c92 in ffs_read (ap=Variable "ap" is not available.
) at ../../../ufs/ffs/ffs_vnops.c:502
#11 0xffffffff8065f222 in VOP_READ_APV (vop=Variable "vop" is not available.
) at vnode_if.c:637
#12 0xffffffff805838b0 in ufs_readdir (ap=0xffffffffcfca4a80) at vnode_if.h:344
#13 0xffffffff8065f5d6 in VOP_READDIR_APV (vop=Variable "vop" is not available.
) at vnode_if.c:1407
#14 0xffffffff80448d6f in getdirentries (td=0xffffff000ef42000, uap=dwarf2_read_address: Corrupted DWARF expression.
) at vnode_if.h:747
#15 0xffffffff8062668d in syscall (frame=0xffffffffcfca4c80) at ../../../amd64/amd64/trap.c:907
#16 0xffffffff8060d7cb in Xfast_syscall () at ../../../amd64/amd64/exception.S:330
Panic #4 (Box 0):
#1 0x0000000000000000 in ?? ()
#2 0xffffffff803c6b8d in boot (howto=260) at ../../../kern/kern_shutdown.c:418
#3 0xffffffff803c6e48 in panic (fmt=Variable "fmt" is not available.
) at ../../../kern/kern_shutdown.c:572
#4 0xffffffff806260b6 in trap_fatal (frame=0xffffffffcd8162a0, eva=138) at ../../../amd64/amd64/trap.c:764
#5 0xffffffff80626311 in trap_pfault (frame=0xffffffffcd8162a0, usermode=0) at ../../../amd64/amd64/trap.c:680
#6 0xffffffff80626c53 in trap (frame=0xffffffffcd8162a0) at ../../../amd64/amd64/trap.c:449
#7 0xffffffff8060d5be in calltrap () at ../../../amd64/amd64/exception.S:209
#8 0xffffffff80599dda in vm_object_clear_flag (object=0x0, bits=Variable "bits" is not available.
) at ../../../vm/vm_object.c:269
#9 0xffffffff804302a6 in cluster_wbuild (vp=0xffffff000ea991f8, size=16384, start_lbn=57442, len=2) at ../../../kern/vfs_cluster.c:925
#10 0xffffffff804305ff in cluster_write (vp=0xffffff000ea991f8, bp=0xffffffffa5394360, filesize=4294967296, seqcount=127) at ../../../kern/vfs_cluster.c:570
#11 0xffffffff8057897a in ffs_write (ap=0xffffffffcd8166a0) at ../../../ufs/ffs/ffs_vnops.c:771
#12 0xffffffff80660834 in VOP_WRITE_APV (vop=0xffffffff80869280, a=0xffffffffcd8166a0) at vnode_if.c:691
#13 0xffffffff805a4ee5 in vnode_pager_generic_putpages (vp=0xffffff000ea991f8, m=0xffffffffcd816860, bytecount=Variable "bytecount" is not available.
) at vnode_if.h:373
#14 0xffffffff80430c31 in vop_stdputpages (ap=Variable "ap" is not available.
) at ../../../kern/vfs_default.c:550
#15 0xffffffff8065fc36 in VOP_PUTPAGES_APV (vop=Variable "vop" is not available.
) at vnode_if.c:2189
#16 0xffffffff805a5085 in vnode_pager_putpages (object=0xffffff000e6b6d00, m=0xffffffffcd816860, count=2, sync=8, rtvals=0xffffffffcd8167e0) at vnode_if.h:1164
#17 0xffffffff8059f79e in vm_pageout_flush (mc=0xffffffffcd816860, count=2, flags=8) at vm_pager.h:147
#18 0xffffffff8059baab in vm_object_page_collect_flush (object=0xffffff000e6b6d00, p=0xffffff00db42f178, curgeneration=Variable "curgeneration" is not available.
) at ../../../vm/vm_object.c:973
#19 0xffffffff8059bed6 in vm_object_page_clean (object=0xffffff000e6b6d00, start=0, end=Variable "end" is not available.
) at ../../../vm/vm_object.c:865
#20 0xffffffff8043f4ee in vfs_msync (mp=0xffffff000e69e378, flags=2) at ../../../kern/vfs_subr.c:2995
#21 0xffffffff8043f80a in sync_fsync (ap=Variable "ap" is not available.
) at ../../../kern/vfs_subr.c:3216
#22 0xffffffff8065f552 in VOP_FSYNC_APV (vop=Variable "vop" is not available.
) at vnode_if.c:1007
#23 0xffffffff8043ff71 in sched_sync () at vnode_if.h:538
#24 0xffffffff803a6a01 in fork_exit (callout=0xffffffff8043f8f0 <sched_sync>, arg=0x0, frame=0xffffffffcd816c80) at ../../../kern/kern_fork.c:804
#25 0xffffffff8060d98e in fork_trampoline () at ../../../amd64/amd64/exception.S:455
--
Herve Boulouis
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20081106102931.GD596>