Date: Fri, 1 Jun 2012 15:58:24 +0300 From: Konstantin Belousov <kostikbel@gmail.com> To: "Desai, Kashyap" <Kashyap.Desai@lsi.com> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>, "McConnell, Stephen" <Stephen.McConnell@lsi.com> Subject: Re: Kernel panic in FreeBSD-8.3 from UFS Message-ID: <20120601125824.GV2358@deviant.kiev.zoral.com.ua> In-Reply-To: <B2FD678A64EAAD45B089B123FDFC3ED72B9F6C1F7F@inbmail01.lsi.com> References: <B2FD678A64EAAD45B089B123FDFC3ED72B9F6C1F79@inbmail01.lsi.com> <20120601124338.GU2358@deviant.kiev.zoral.com.ua> <B2FD678A64EAAD45B089B123FDFC3ED72B9F6C1F7F@inbmail01.lsi.com>
next in thread | previous in thread | raw e-mail | index | archive | help
--Azlbkg4pdDb7tRhV Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Fri, Jun 01, 2012 at 06:19:56PM +0530, Desai, Kashyap wrote: >=20 > Thanks for the information. *YES* to me also looks like memory leaks only= .. but it is CAM XPT who is using "366195K " memory.. >=20 Yes, it seems that cam should be investigated next. It is indeed most likely not related to UFS at all, and just happens to trace through the UFS code since you might have run fs-intensive test and filesystem just called allocator most often. > See below output of "vmstat -m" >=20 > vmstat -m >=20 > Type InUse MemUse HighUse Requests Size(s) > feeder 7 1K - 7 16 > acpiintr 1 1K - 1 32 > isadev 9 1K - 9 64 > acpica 3179 172K - 73127 16,32,64,128,256,512,1024,20= 48 > cdev 7 1K - 7 128 > acpitask 1 1K - 1 1024 > sigio 1 1K - 1 32 > filedesc 50 14K - 2926 16,256,512,1024 > kenv 121 9K - 130 16,32,64,128,4096 > kqueue 0 0K - 266 128,1024 > CAM dev queue 8 1K - 8 128 > proc-args 26 2K - 4746 16,32,64,128 > hhook 2 1K - 2 128 > ithread 128 11K - 128 16,64,128 > CAM queue 43 9K - 595257 16,32,64,128,256,512,1024,20= 48,4096 > KTRACE 100 13K - 100 128 > acpisem 21 3K - 21 64,128 > linker 157 6K - 166 16,32,256 > lockf 1751 61K - 2311 32,64,128,256,512,1024 > loginclass 2 1K - 96 64 > ip6ndp 12 1K - 13 64,128 > ip6opt 0 0K - 3 32 > temp 56 233K - 11165 16,32,64,128,256,512,1024,20= 48,4096 > devbuf 5248 4507K - 5360 16,32,64,128,256,512,1024,20= 48,4096 > module 493 31K - 493 64,128 > mtx_pool 2 8K - 2 4096 > CAM SIM 8 1K - 8 128 > subproc 216 219K - 3091 256,4096 > proc 2 8K - 2 4096 > session 18 2K - 109 64 > pgrp 25 2K - 129 64 > cred 62 6K - 13960 64,128 > uidinfo 3 2K - 88 64,1024 > plimit 18 5K - 1389 256 > scsi_cd 0 0K - 4 16 > CAM periph 22 3K - 84532 16,32,64,128 > CAM XPT 183208 366195K - 722021 16,32,64,256,1024,2048 > sysctltmp 0 0K - 453 16,32,64,128,4096 > sysctloid 5010 158K - 5286 16,32,64,128 > sysctl 0 0K - 763 16,32,64 > tidhash 1 8K - 1 =20 > callout 7 1792K - 7 =20 > umtx 750 71K - 750 64,128 > p1003.1b 1 1K - 1 16 > SWAP 2 4373K - 2 64 > bus-sc 97 417K - 6298 16,32,64,128,256,512,1024,20= 48,4096 > bus 1382 64K - 9711 16,32,64,128,256,1024 > devstat 16 33K - 16 16,4096 > eventhandler 73 4K - 73 32,64,128 > UART 6 3K - 6 16,256,1024 > kobj 358 716K - 634 2048 > Per-cpu 1 1K - 1 16 > ata_pci 2 1K - 2 32 > rman 226 13K - 424 16,32,64 > sbuf 0 0K - 1636 16,32,64,128,256,512,1024,20= 48,4096 > scsi_da 0 0K - 186 16 > stack 0 0K - 2 128 > taskqueue 15 1K - 15 16,64 > Unitno 14 1K - 5912 16,64 > iov 0 0K - 1847877 16,64,128,256 > select 9 1K - 9 64 > ioctlops 0 0K - 5848 16,32,64,128,256,512,1024,20= 48 > msg 4 25K - 4 1024,4096 > sem 4 101K - 4 1024,4096 > shm 1 12K - 1 =20 > tty 21 11K - 23 512,2048 > mbuf_tag 0 0K - 549 32,64 > shmfd 1 4K - 1 4096 > pcb 18 79K - 567 16,32,64,512,1024,2048,4096 > soname 3 1K - 16885 16,32,128 > vfscache 1 512K - 1 =20 > cl_savebuf 0 0K - 48 32 > vfs_hash 1 256K - 1 =20 > acpidev 50 2K - 50 32 > vnodes 2 1K - 2 128 > vnodemarker 0 0K - 6497 512 > mount 94 4K - 197 16,32,64,128,256 > BPF 10 1K - 10 64 > ether_multi 21 1K - 24 16,32,64 > ifaddr 90 17K - 90 16,32,64,128,256,512,2048 > ifnet 11 11K - 11 64,1024 > USBdev 35 9K - 35 32,128,1024 > clone 6 24K - 6 4096 > arpcom 2 1K - 2 16 > lltable 23 6K - 23 256 > USB 66 40K - 69 16,32,64,128,256,1024,4096 > routetbl 29 4K - 245 16,64,128,256 > igmp 10 2K - 10 128 > in_multi 1 1K - 1 128 > sctp_iter 0 0K - 3 256 > sctp_ifn 2 1K - 2 128 > sctp_ifa 4 1K - 4 128 > sctp_vrf 1 1K - 1 64 > sctp_a_it 0 0K - 3 16 > hostcache 1 16K - 1 =20 > syncache 1 72K - 1 =20 > entropy 1024 64K - 1024 64 > in6_multi 15 2K - 15 16,256 > pci_link 16 2K - 16 64,128 > mld 10 2K - 10 128 > rpc 2 1K - 2 128 > audit_evclass 179 3K - 218 16 > jblocks 2 1K - 2 128 > savedino 0 0K - 121 256 > sbdep 0 0K - 464 32 > jsegdep 1 1K - 6778 32 > jseg 1 1K - 4558 128 > jfreefrag 0 0K - 179 64 > jnewblk 0 0K - 5965 64 > jremref 0 0K - 317 64 > jaddref 0 0K - 317 64 > freedep 0 0K - 9 32 > freework 1 1K - 268 32,128 > newdirblk 0 0K - 6 32 > dirrem 0 0K - 305 64 > mkdir 0 0K - 12 64 > diradd 0 0K - 305 64 > freefile 0 0K - 72 32 > freeblks 0 0K - 157 128 > freefrag 0 0K - 179 64 > indirdep 1 1K - 4235 64 > newblk 2 65K - 5966 128 > bmsafemap 2 5K - 4389 128,4096 > inodedep 2 257K - 4997 256 > pagedep 1 64K - 51 128 > ufs_dirhash 8 4K - 24 16,32,64,512 > ufs_mount 21 390K - 21 256,4096 > vm_pgdata 2 65K - 2 64 > UMAHash 1 1K - 1 256 > acpi_perf 2 1K - 2 256 > DEVFS1 127 32K - 187 256 > atkbddev 2 1K - 2 32 > DEVFS3 141 18K - 223 128,256 > DEVFS 24 1K - 25 16,64 > memdesc 1 4K - 1 4096 > apmdev 1 1K - 1 64 > io_apic 2 2K - 2 1024 > pfs_nodes 21 3K - 21 128 > msi 3 1K - 3 64 > nexusdev 5 1K - 5 16 > GEOM 117 19K - 2291 16,32,64,128,256,512,1024,20= 48 > SCSI SES 2 4K - 2 2048 > kbdmux 7 18K - 7 16,256,1024,2048 > mps 22 280K - 24141 16,32,64,128,256,512,2048,40= 96 > mps_user 0 0K - 662 32,64 >=20 >=20 > ` Kashyap >=20 > > -----Original Message----- > > From: Konstantin Belousov [mailto:kostikbel@gmail.com] > > Sent: Friday, June 01, 2012 6:14 PM > > To: Desai, Kashyap > > Cc: freebsd-scsi@freebsd.org; freebsd-fs@freebsd.org; McConnell, Stephen > > Subject: Re: Kernel panic in FreeBSD-8.3 from UFS > >=20 > > On Fri, Jun 01, 2012 at 05:30:39PM +0530, Desai, Kashyap wrote: > > > Hi, > > > > > > We have seen kernel panic while doing IO along with HBA reset. > > > This looks to be very rare but not sure if someone can help me to > > > understand what is a issue here. To me it does not look any issue with > > > underline Device Driver <mps> > > > > > > See below back trace. > > You did not specified the panic message. Was it 'kmem_map too small' ? > >=20 > > Unless HBA driver causes memory leak, this is probably indeed unrelated. > > > > > > > > > #0 doadump (textdump=3D1) at pcpu.h:244 > > > 244 pcpu.h: No such file or directory. > > > in pcpu.h > > > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:244 > > > #1 0xc0a1845a in kern_reboot (howto=3D260) > > > at /usr/src/sys/kern/kern_shutdown.c:442 > > > #2 0xc0a186f1 in panic (fmt=3DVariable "fmt" is not available. > > > ) at /usr/src/sys/kern/kern_shutdown.c:607 > > > #3 0xc0c7ceda in kmem_malloc (map=3D0xc15c808c, size=3D32768, flags= =3D2) > > > at /usr/src/sys/vm/vm_kern.c:334 > > > #4 0xc0c708e7 in page_alloc (zone=3D0x0, bytes=3D32768, pflag=3D0xf1= 9839bf > > "\002", > > > wait=3D2) at /usr/src/sys/vm/uma_core.c:994 > > > #5 0xc0c72fe0 in uma_large_malloc (size=3D32768, wait=3D2) > > > at /usr/src/sys/vm/uma_core.c:3067 > > > #6 0xc0a04fac in malloc (size=3D32768, mtp=3D0xc102b808, flags=3D2) > > > at /usr/src/sys/kern/kern_malloc.c:492 > > > #7 0xc0c42e89 in softdep_disk_io_initiation (bp=3D0xdef881fc) > > > at /usr/src/sys/ufs/ffs/ffs_softdep.c:10126 > > > #8 0xc0c5208f in ffs_geom_strategy (bo=3D0xc5fc30ac, bp=3D0xdef881fc) > > > at buf.h:411 > > > #9 0xc0c65a43 in ufs_strategy (ap=3D0xf1983b00) > > > at /usr/src/sys/ufs/ufs/ufs_vnops.c:2317 > > > #10 0xc0d6a6dd in VOP_STRATEGY_APV (vop=3D0xc102e4a0, a=3D0xf1983b00) > > > at vnode_if.c:2171 > > > #11 0xc0a8d19e in bufstrategy (bo=3D0xc6b901bc, bp=3D0xdef881fc) at > > > vnode_if.h:940 > > > #12 0xc0a9352e in bufwrite (bp=3D0xdef881fc) at buf.h:404 > > > #13 0xc0a8db5c in vfs_bio_awrite (bp=3D0xdef881fc) at buf.h:392 > > > #14 0xc0c584c5 in ffs_syncvnode (vp=3D0xc6b90110, waitfor=3D1) > > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:288 > > > #15 0xc0c58739 in ffs_fsync (ap=3D0xf1983c4c) > > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:187 > > > #16 0xc0d69712 in VOP_FSYNC_APV (vop=3D0xc102dfc0, a=3D0xf1983c4c) > > > at vnode_if.c:1267 > > > #17 0xc0ab5d49 in sys_fsync (td=3D0xc64ea8a0, uap=3D0xf1983cec) at > > > vnode_if.h:549 > > > #18 0xc0d49315 in syscall (frame=3D0xf1983d28) at subr_syscall.c:131 > > > #19 0xc0d32af1 in Xint0x80_syscall () > > > at /usr/src/sys/i386/i386/exception.s:266 > > > #20 0x00000033 in ?? ( > > > > > > > > > To me it looks like UFS is doing something to crash the kernel. > >=20 > > You might try to use vmstat -z and vmstat -m on core to see what has > > used KVA. --Azlbkg4pdDb7tRhV Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.12 (FreeBSD) iEYEARECAAYFAk/IvG8ACgkQC3+MBN1Mb4je2gCgqc5sAMpSoKxbz+NPCLz1fAa7 4FMAnj3hA9UgE3oSu6KLkverbyhQ1Goz =0QcW -----END PGP SIGNATURE----- --Azlbkg4pdDb7tRhV--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?20120601125824.GV2358>