From owner-freebsd-fs@FreeBSD.ORG Fri Jun 1 12:50:11 2012 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id A9F991065678; Fri, 1 Jun 2012 12:50:11 +0000 (UTC) (envelope-from Kashyap.Desai@lsi.com) Received: from na3sys009aog123.obsmtp.com (na3sys009aog123.obsmtp.com [74.125.149.149]) by mx1.freebsd.org (Postfix) with ESMTP id 494CC8FC25; Fri, 1 Jun 2012 12:50:09 +0000 (UTC) Received: from paledge01.lsi.com ([192.19.193.42]) (using TLSv1) by na3sys009aob123.postini.com ([74.125.148.12]) with SMTP ID DSNKT8i6ezGNwyCrE9UVStQIApRr3Ia6BriA@postini.com; Fri, 01 Jun 2012 05:50:09 PDT Received: from PALCAS01.lsi.com (128.94.213.117) by PALEDGE01.lsi.com (192.19.193.42) with Microsoft SMTP Server (TLS) id 8.3.213.0; Fri, 1 Jun 2012 08:56:13 -0400 Received: from inbexch01.lsi.com (135.36.98.37) by PALCAS01.lsi.com (128.94.213.117) with Microsoft SMTP Server (TLS) id 8.3.213.0; Fri, 1 Jun 2012 08:50:02 -0400 Received: from inbmail01.lsi.com ([135.36.98.64]) by inbexch01.lsi.com ([135.36.98.37]) with mapi; Fri, 1 Jun 2012 18:19:59 +0530 From: "Desai, Kashyap" To: Konstantin Belousov Date: Fri, 1 Jun 2012 18:19:56 +0530 Thread-Topic: Kernel panic in FreeBSD-8.3 from UFS Thread-Index: Ac0/9DQxCjvBzL4sTc2DlPByoowCKAAAKAOw Message-ID: References: <20120601124338.GU2358@deviant.kiev.zoral.com.ua> In-Reply-To: <20120601124338.GU2358@deviant.kiev.zoral.com.ua> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: acceptlanguage: en-US Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "freebsd-fs@freebsd.org" , "freebsd-scsi@freebsd.org" , "McConnell, Stephen" Subject: RE: Kernel panic in FreeBSD-8.3 from UFS X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 01 Jun 2012 12:50:11 -0000 Thanks for the information. *YES* to me also looks like memory leaks only..= but it is CAM XPT who is using "366195K " memory.. See below output of "vmstat -m" vmstat -m Type InUse MemUse HighUse Requests Size(s) feeder 7 1K - 7 16 acpiintr 1 1K - 1 32 isadev 9 1K - 9 64 acpica 3179 172K - 73127 16,32,64,128,256,512,1024,2048 cdev 7 1K - 7 128 acpitask 1 1K - 1 1024 sigio 1 1K - 1 32 filedesc 50 14K - 2926 16,256,512,1024 kenv 121 9K - 130 16,32,64,128,4096 kqueue 0 0K - 266 128,1024 CAM dev queue 8 1K - 8 128 proc-args 26 2K - 4746 16,32,64,128 hhook 2 1K - 2 128 ithread 128 11K - 128 16,64,128 CAM queue 43 9K - 595257 16,32,64,128,256,512,1024,2048= ,4096 KTRACE 100 13K - 100 128 acpisem 21 3K - 21 64,128 linker 157 6K - 166 16,32,256 lockf 1751 61K - 2311 32,64,128,256,512,1024 loginclass 2 1K - 96 64 ip6ndp 12 1K - 13 64,128 ip6opt 0 0K - 3 32 temp 56 233K - 11165 16,32,64,128,256,512,1024,2048= ,4096 devbuf 5248 4507K - 5360 16,32,64,128,256,512,1024,2048= ,4096 module 493 31K - 493 64,128 mtx_pool 2 8K - 2 4096 CAM SIM 8 1K - 8 128 subproc 216 219K - 3091 256,4096 proc 2 8K - 2 4096 session 18 2K - 109 64 pgrp 25 2K - 129 64 cred 62 6K - 13960 64,128 uidinfo 3 2K - 88 64,1024 plimit 18 5K - 1389 256 scsi_cd 0 0K - 4 16 CAM periph 22 3K - 84532 16,32,64,128 CAM XPT 183208 366195K - 722021 16,32,64,256,1024,2048 sysctltmp 0 0K - 453 16,32,64,128,4096 sysctloid 5010 158K - 5286 16,32,64,128 sysctl 0 0K - 763 16,32,64 tidhash 1 8K - 1 =20 callout 7 1792K - 7 =20 umtx 750 71K - 750 64,128 p1003.1b 1 1K - 1 16 SWAP 2 4373K - 2 64 bus-sc 97 417K - 6298 16,32,64,128,256,512,1024,2048= ,4096 bus 1382 64K - 9711 16,32,64,128,256,1024 devstat 16 33K - 16 16,4096 eventhandler 73 4K - 73 32,64,128 UART 6 3K - 6 16,256,1024 kobj 358 716K - 634 2048 Per-cpu 1 1K - 1 16 ata_pci 2 1K - 2 32 rman 226 13K - 424 16,32,64 sbuf 0 0K - 1636 16,32,64,128,256,512,1024,2048= ,4096 scsi_da 0 0K - 186 16 stack 0 0K - 2 128 taskqueue 15 1K - 15 16,64 Unitno 14 1K - 5912 16,64 iov 0 0K - 1847877 16,64,128,256 select 9 1K - 9 64 ioctlops 0 0K - 5848 16,32,64,128,256,512,1024,2048 msg 4 25K - 4 1024,4096 sem 4 101K - 4 1024,4096 shm 1 12K - 1 =20 tty 21 11K - 23 512,2048 mbuf_tag 0 0K - 549 32,64 shmfd 1 4K - 1 4096 pcb 18 79K - 567 16,32,64,512,1024,2048,4096 soname 3 1K - 16885 16,32,128 vfscache 1 512K - 1 =20 cl_savebuf 0 0K - 48 32 vfs_hash 1 256K - 1 =20 acpidev 50 2K - 50 32 vnodes 2 1K - 2 128 vnodemarker 0 0K - 6497 512 mount 94 4K - 197 16,32,64,128,256 BPF 10 1K - 10 64 ether_multi 21 1K - 24 16,32,64 ifaddr 90 17K - 90 16,32,64,128,256,512,2048 ifnet 11 11K - 11 64,1024 USBdev 35 9K - 35 32,128,1024 clone 6 24K - 6 4096 arpcom 2 1K - 2 16 lltable 23 6K - 23 256 USB 66 40K - 69 16,32,64,128,256,1024,4096 routetbl 29 4K - 245 16,64,128,256 igmp 10 2K - 10 128 in_multi 1 1K - 1 128 sctp_iter 0 0K - 3 256 sctp_ifn 2 1K - 2 128 sctp_ifa 4 1K - 4 128 sctp_vrf 1 1K - 1 64 sctp_a_it 0 0K - 3 16 hostcache 1 16K - 1 =20 syncache 1 72K - 1 =20 entropy 1024 64K - 1024 64 in6_multi 15 2K - 15 16,256 pci_link 16 2K - 16 64,128 mld 10 2K - 10 128 rpc 2 1K - 2 128 audit_evclass 179 3K - 218 16 jblocks 2 1K - 2 128 savedino 0 0K - 121 256 sbdep 0 0K - 464 32 jsegdep 1 1K - 6778 32 jseg 1 1K - 4558 128 jfreefrag 0 0K - 179 64 jnewblk 0 0K - 5965 64 jremref 0 0K - 317 64 jaddref 0 0K - 317 64 freedep 0 0K - 9 32 freework 1 1K - 268 32,128 newdirblk 0 0K - 6 32 dirrem 0 0K - 305 64 mkdir 0 0K - 12 64 diradd 0 0K - 305 64 freefile 0 0K - 72 32 freeblks 0 0K - 157 128 freefrag 0 0K - 179 64 indirdep 1 1K - 4235 64 newblk 2 65K - 5966 128 bmsafemap 2 5K - 4389 128,4096 inodedep 2 257K - 4997 256 pagedep 1 64K - 51 128 ufs_dirhash 8 4K - 24 16,32,64,512 ufs_mount 21 390K - 21 256,4096 vm_pgdata 2 65K - 2 64 UMAHash 1 1K - 1 256 acpi_perf 2 1K - 2 256 DEVFS1 127 32K - 187 256 atkbddev 2 1K - 2 32 DEVFS3 141 18K - 223 128,256 DEVFS 24 1K - 25 16,64 memdesc 1 4K - 1 4096 apmdev 1 1K - 1 64 io_apic 2 2K - 2 1024 pfs_nodes 21 3K - 21 128 msi 3 1K - 3 64 nexusdev 5 1K - 5 16 GEOM 117 19K - 2291 16,32,64,128,256,512,1024,2048 SCSI SES 2 4K - 2 2048 kbdmux 7 18K - 7 16,256,1024,2048 mps 22 280K - 24141 16,32,64,128,256,512,2048,4096 mps_user 0 0K - 662 32,64 ` Kashyap > -----Original Message----- > From: Konstantin Belousov [mailto:kostikbel@gmail.com] > Sent: Friday, June 01, 2012 6:14 PM > To: Desai, Kashyap > Cc: freebsd-scsi@freebsd.org; freebsd-fs@freebsd.org; McConnell, Stephen > Subject: Re: Kernel panic in FreeBSD-8.3 from UFS >=20 > On Fri, Jun 01, 2012 at 05:30:39PM +0530, Desai, Kashyap wrote: > > Hi, > > > > We have seen kernel panic while doing IO along with HBA reset. > > This looks to be very rare but not sure if someone can help me to > > understand what is a issue here. To me it does not look any issue with > > underline Device Driver > > > > See below back trace. > You did not specified the panic message. Was it 'kmem_map too small' ? >=20 > Unless HBA driver causes memory leak, this is probably indeed unrelated. > > > > > > #0 doadump (textdump=3D1) at pcpu.h:244 > > 244 pcpu.h: No such file or directory. > > in pcpu.h > > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:244 > > #1 0xc0a1845a in kern_reboot (howto=3D260) > > at /usr/src/sys/kern/kern_shutdown.c:442 > > #2 0xc0a186f1 in panic (fmt=3DVariable "fmt" is not available. > > ) at /usr/src/sys/kern/kern_shutdown.c:607 > > #3 0xc0c7ceda in kmem_malloc (map=3D0xc15c808c, size=3D32768, flags=3D= 2) > > at /usr/src/sys/vm/vm_kern.c:334 > > #4 0xc0c708e7 in page_alloc (zone=3D0x0, bytes=3D32768, pflag=3D0xf198= 39bf > "\002", > > wait=3D2) at /usr/src/sys/vm/uma_core.c:994 > > #5 0xc0c72fe0 in uma_large_malloc (size=3D32768, wait=3D2) > > at /usr/src/sys/vm/uma_core.c:3067 > > #6 0xc0a04fac in malloc (size=3D32768, mtp=3D0xc102b808, flags=3D2) > > at /usr/src/sys/kern/kern_malloc.c:492 > > #7 0xc0c42e89 in softdep_disk_io_initiation (bp=3D0xdef881fc) > > at /usr/src/sys/ufs/ffs/ffs_softdep.c:10126 > > #8 0xc0c5208f in ffs_geom_strategy (bo=3D0xc5fc30ac, bp=3D0xdef881fc) > > at buf.h:411 > > #9 0xc0c65a43 in ufs_strategy (ap=3D0xf1983b00) > > at /usr/src/sys/ufs/ufs/ufs_vnops.c:2317 > > #10 0xc0d6a6dd in VOP_STRATEGY_APV (vop=3D0xc102e4a0, a=3D0xf1983b00) > > at vnode_if.c:2171 > > #11 0xc0a8d19e in bufstrategy (bo=3D0xc6b901bc, bp=3D0xdef881fc) at > > vnode_if.h:940 > > #12 0xc0a9352e in bufwrite (bp=3D0xdef881fc) at buf.h:404 > > #13 0xc0a8db5c in vfs_bio_awrite (bp=3D0xdef881fc) at buf.h:392 > > #14 0xc0c584c5 in ffs_syncvnode (vp=3D0xc6b90110, waitfor=3D1) > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:288 > > #15 0xc0c58739 in ffs_fsync (ap=3D0xf1983c4c) > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:187 > > #16 0xc0d69712 in VOP_FSYNC_APV (vop=3D0xc102dfc0, a=3D0xf1983c4c) > > at vnode_if.c:1267 > > #17 0xc0ab5d49 in sys_fsync (td=3D0xc64ea8a0, uap=3D0xf1983cec) at > > vnode_if.h:549 > > #18 0xc0d49315 in syscall (frame=3D0xf1983d28) at subr_syscall.c:131 > > #19 0xc0d32af1 in Xint0x80_syscall () > > at /usr/src/sys/i386/i386/exception.s:266 > > #20 0x00000033 in ?? ( > > > > > > To me it looks like UFS is doing something to crash the kernel. >=20 > You might try to use vmstat -z and vmstat -m on core to see what has > used KVA.