Date: Tue, 5 Jun 2012 17:49:05 +0530 From: "Desai, Kashyap" <Kashyap.Desai@lsi.com> To: Konstantin Belousov <kostikbel@gmail.com>, "Kenneth D. Merry" <ken@FreeBSD.org> Cc: "freebsd-fs@freebsd.org" <freebsd-fs@freebsd.org>, "freebsd-scsi@freebsd.org" <freebsd-scsi@freebsd.org>, "McConnell, Stephen" <Stephen.McConnell@lsi.com>, "Reddy, Sreekanth" <Sreekanth.Reddy@lsi.com> Subject: RE: Kernel panic in FreeBSD-8.3 from UFS Message-ID: <B2FD678A64EAAD45B089B123FDFC3ED72B9F6C21A7@inbmail01.lsi.com> In-Reply-To: <20120601125824.GV2358@deviant.kiev.zoral.com.ua> References: <B2FD678A64EAAD45B089B123FDFC3ED72B9F6C1F79@inbmail01.lsi.com> <20120601124338.GU2358@deviant.kiev.zoral.com.ua> <B2FD678A64EAAD45B089B123FDFC3ED72B9F6C1F7F@inbmail01.lsi.com> <20120601125824.GV2358@deviant.kiev.zoral.com.ua>
next in thread | previous in thread | raw e-mail | index | archive | help
--_002_B2FD678A64EAAD45B089B123FDFC3ED72B9F6C21A7inbmail01lsic_
Content-Type: text/plain; charset="us-ascii"
Content-Transfer-Encoding: quoted-printable
Hi All,
We found some potential area of memory leak in CAM layer.=20
CAM XPT Memory leak is due to following function in scsi/scsi_all.c
int
scsi_command_string(struct ccb_scsiio *csio, struct sbuf *sb) =
=20
In above function, CAM layer allocate memory for ccb device as below
if ((cgd =3D (struct ccb_getdev*)xpt_alloc_ccb_nowait()) =3D=3D NULL)
_But_, unfortunately we never free the allocated memory and we see memory l=
eak of 2K every time when someone is calling=20
Scsi_command_string from kernel mode.
Attached is a proposed patch for this issue.
` Kashyap
> -----Original Message-----
> From: Konstantin Belousov [mailto:kostikbel@gmail.com]
> Sent: Friday, June 01, 2012 6:28 PM
> To: Desai, Kashyap
> Cc: freebsd-scsi@freebsd.org; freebsd-fs@freebsd.org; McConnell, Stephen
> Subject: Re: Kernel panic in FreeBSD-8.3 from UFS
>=20
> On Fri, Jun 01, 2012 at 06:19:56PM +0530, Desai, Kashyap wrote:
> >
> > Thanks for the information. *YES* to me also looks like memory leaks
> only.. but it is CAM XPT who is using "366195K " memory..
> >
> Yes, it seems that cam should be investigated next.
>=20
> It is indeed most likely not related to UFS at all, and just happens to
> trace through the UFS code since you might have run fs-intensive test
> and filesystem just called allocator most often.
>=20
> > See below output of "vmstat -m"
> >
> > vmstat -m
> >
> > Type InUse MemUse HighUse Requests Size(s)
> > feeder 7 1K - 7 16
> > acpiintr 1 1K - 1 32
> > isadev 9 1K - 9 64
> > acpica 3179 172K - 73127
> 16,32,64,128,256,512,1024,2048
> > cdev 7 1K - 7 128
> > acpitask 1 1K - 1 1024
> > sigio 1 1K - 1 32
> > filedesc 50 14K - 2926 16,256,512,1024
> > kenv 121 9K - 130 16,32,64,128,4096
> > kqueue 0 0K - 266 128,1024
> > CAM dev queue 8 1K - 8 128
> > proc-args 26 2K - 4746 16,32,64,128
> > hhook 2 1K - 2 128
> > ithread 128 11K - 128 16,64,128
> > CAM queue 43 9K - 595257
> 16,32,64,128,256,512,1024,2048,4096
> > KTRACE 100 13K - 100 128
> > acpisem 21 3K - 21 64,128
> > linker 157 6K - 166 16,32,256
> > lockf 1751 61K - 2311 32,64,128,256,512,1024
> > loginclass 2 1K - 96 64
> > ip6ndp 12 1K - 13 64,128
> > ip6opt 0 0K - 3 32
> > temp 56 233K - 11165
> 16,32,64,128,256,512,1024,2048,4096
> > devbuf 5248 4507K - 5360
> 16,32,64,128,256,512,1024,2048,4096
> > module 493 31K - 493 64,128
> > mtx_pool 2 8K - 2 4096
> > CAM SIM 8 1K - 8 128
> > subproc 216 219K - 3091 256,4096
> > proc 2 8K - 2 4096
> > session 18 2K - 109 64
> > pgrp 25 2K - 129 64
> > cred 62 6K - 13960 64,128
> > uidinfo 3 2K - 88 64,1024
> > plimit 18 5K - 1389 256
> > scsi_cd 0 0K - 4 16
> > CAM periph 22 3K - 84532 16,32,64,128
> > CAM XPT 183208 366195K - 722021 16,32,64,256,1024,2048
> > sysctltmp 0 0K - 453 16,32,64,128,4096
> > sysctloid 5010 158K - 5286 16,32,64,128
> > sysctl 0 0K - 763 16,32,64
> > tidhash 1 8K - 1
> > callout 7 1792K - 7
> > umtx 750 71K - 750 64,128
> > p1003.1b 1 1K - 1 16
> > SWAP 2 4373K - 2 64
> > bus-sc 97 417K - 6298
> 16,32,64,128,256,512,1024,2048,4096
> > bus 1382 64K - 9711 16,32,64,128,256,1024
> > devstat 16 33K - 16 16,4096
> > eventhandler 73 4K - 73 32,64,128
> > UART 6 3K - 6 16,256,1024
> > kobj 358 716K - 634 2048
> > Per-cpu 1 1K - 1 16
> > ata_pci 2 1K - 2 32
> > rman 226 13K - 424 16,32,64
> > sbuf 0 0K - 1636
> 16,32,64,128,256,512,1024,2048,4096
> > scsi_da 0 0K - 186 16
> > stack 0 0K - 2 128
> > taskqueue 15 1K - 15 16,64
> > Unitno 14 1K - 5912 16,64
> > iov 0 0K - 1847877 16,64,128,256
> > select 9 1K - 9 64
> > ioctlops 0 0K - 5848
> 16,32,64,128,256,512,1024,2048
> > msg 4 25K - 4 1024,4096
> > sem 4 101K - 4 1024,4096
> > shm 1 12K - 1
> > tty 21 11K - 23 512,2048
> > mbuf_tag 0 0K - 549 32,64
> > shmfd 1 4K - 1 4096
> > pcb 18 79K - 567
> 16,32,64,512,1024,2048,4096
> > soname 3 1K - 16885 16,32,128
> > vfscache 1 512K - 1
> > cl_savebuf 0 0K - 48 32
> > vfs_hash 1 256K - 1
> > acpidev 50 2K - 50 32
> > vnodes 2 1K - 2 128
> > vnodemarker 0 0K - 6497 512
> > mount 94 4K - 197 16,32,64,128,256
> > BPF 10 1K - 10 64
> > ether_multi 21 1K - 24 16,32,64
> > ifaddr 90 17K - 90 16,32,64,128,256,512,2048
> > ifnet 11 11K - 11 64,1024
> > USBdev 35 9K - 35 32,128,1024
> > clone 6 24K - 6 4096
> > arpcom 2 1K - 2 16
> > lltable 23 6K - 23 256
> > USB 66 40K - 69
> 16,32,64,128,256,1024,4096
> > routetbl 29 4K - 245 16,64,128,256
> > igmp 10 2K - 10 128
> > in_multi 1 1K - 1 128
> > sctp_iter 0 0K - 3 256
> > sctp_ifn 2 1K - 2 128
> > sctp_ifa 4 1K - 4 128
> > sctp_vrf 1 1K - 1 64
> > sctp_a_it 0 0K - 3 16
> > hostcache 1 16K - 1
> > syncache 1 72K - 1
> > entropy 1024 64K - 1024 64
> > in6_multi 15 2K - 15 16,256
> > pci_link 16 2K - 16 64,128
> > mld 10 2K - 10 128
> > rpc 2 1K - 2 128
> > audit_evclass 179 3K - 218 16
> > jblocks 2 1K - 2 128
> > savedino 0 0K - 121 256
> > sbdep 0 0K - 464 32
> > jsegdep 1 1K - 6778 32
> > jseg 1 1K - 4558 128
> > jfreefrag 0 0K - 179 64
> > jnewblk 0 0K - 5965 64
> > jremref 0 0K - 317 64
> > jaddref 0 0K - 317 64
> > freedep 0 0K - 9 32
> > freework 1 1K - 268 32,128
> > newdirblk 0 0K - 6 32
> > dirrem 0 0K - 305 64
> > mkdir 0 0K - 12 64
> > diradd 0 0K - 305 64
> > freefile 0 0K - 72 32
> > freeblks 0 0K - 157 128
> > freefrag 0 0K - 179 64
> > indirdep 1 1K - 4235 64
> > newblk 2 65K - 5966 128
> > bmsafemap 2 5K - 4389 128,4096
> > inodedep 2 257K - 4997 256
> > pagedep 1 64K - 51 128
> > ufs_dirhash 8 4K - 24 16,32,64,512
> > ufs_mount 21 390K - 21 256,4096
> > vm_pgdata 2 65K - 2 64
> > UMAHash 1 1K - 1 256
> > acpi_perf 2 1K - 2 256
> > DEVFS1 127 32K - 187 256
> > atkbddev 2 1K - 2 32
> > DEVFS3 141 18K - 223 128,256
> > DEVFS 24 1K - 25 16,64
> > memdesc 1 4K - 1 4096
> > apmdev 1 1K - 1 64
> > io_apic 2 2K - 2 1024
> > pfs_nodes 21 3K - 21 128
> > msi 3 1K - 3 64
> > nexusdev 5 1K - 5 16
> > GEOM 117 19K - 2291
> 16,32,64,128,256,512,1024,2048
> > SCSI SES 2 4K - 2 2048
> > kbdmux 7 18K - 7 16,256,1024,2048
> > mps 22 280K - 24141
> 16,32,64,128,256,512,2048,4096
> > mps_user 0 0K - 662 32,64
> >
> >
> > ` Kashyap
> >
> > > -----Original Message-----
> > > From: Konstantin Belousov [mailto:kostikbel@gmail.com]
> > > Sent: Friday, June 01, 2012 6:14 PM
> > > To: Desai, Kashyap
> > > Cc: freebsd-scsi@freebsd.org; freebsd-fs@freebsd.org; McConnell,
> > > Stephen
> > > Subject: Re: Kernel panic in FreeBSD-8.3 from UFS
> > >
> > > On Fri, Jun 01, 2012 at 05:30:39PM +0530, Desai, Kashyap wrote:
> > > > Hi,
> > > >
> > > > We have seen kernel panic while doing IO along with HBA reset.
> > > > This looks to be very rare but not sure if someone can help me to
> > > > understand what is a issue here. To me it does not look any issue
> > > > with underline Device Driver <mps>
> > > >
> > > > See below back trace.
> > > You did not specified the panic message. Was it 'kmem_map too small'
> ?
> > >
> > > Unless HBA driver causes memory leak, this is probably indeed
> unrelated.
> > > >
> > > >
> > > > #0 doadump (textdump=3D1) at pcpu.h:244
> > > > 244 pcpu.h: No such file or directory.
> > > > in pcpu.h
> > > > (kgdb) #0 doadump (textdump=3D1) at pcpu.h:244
> > > > #1 0xc0a1845a in kern_reboot (howto=3D260)
> > > > at /usr/src/sys/kern/kern_shutdown.c:442
> > > > #2 0xc0a186f1 in panic (fmt=3DVariable "fmt" is not available.
> > > > ) at /usr/src/sys/kern/kern_shutdown.c:607
> > > > #3 0xc0c7ceda in kmem_malloc (map=3D0xc15c808c, size=3D32768,
> flags=3D2)
> > > > at /usr/src/sys/vm/vm_kern.c:334
> > > > #4 0xc0c708e7 in page_alloc (zone=3D0x0, bytes=3D32768,
> > > > pflag=3D0xf19839bf
> > > "\002",
> > > > wait=3D2) at /usr/src/sys/vm/uma_core.c:994
> > > > #5 0xc0c72fe0 in uma_large_malloc (size=3D32768, wait=3D2)
> > > > at /usr/src/sys/vm/uma_core.c:3067
> > > > #6 0xc0a04fac in malloc (size=3D32768, mtp=3D0xc102b808, flags=3D2=
)
> > > > at /usr/src/sys/kern/kern_malloc.c:492
> > > > #7 0xc0c42e89 in softdep_disk_io_initiation (bp=3D0xdef881fc)
> > > > at /usr/src/sys/ufs/ffs/ffs_softdep.c:10126
> > > > #8 0xc0c5208f in ffs_geom_strategy (bo=3D0xc5fc30ac, bp=3D0xdef881=
fc)
> > > > at buf.h:411
> > > > #9 0xc0c65a43 in ufs_strategy (ap=3D0xf1983b00)
> > > > at /usr/src/sys/ufs/ufs/ufs_vnops.c:2317
> > > > #10 0xc0d6a6dd in VOP_STRATEGY_APV (vop=3D0xc102e4a0, a=3D0xf1983b0=
0)
> > > > at vnode_if.c:2171
> > > > #11 0xc0a8d19e in bufstrategy (bo=3D0xc6b901bc, bp=3D0xdef881fc) at
> > > > vnode_if.h:940
> > > > #12 0xc0a9352e in bufwrite (bp=3D0xdef881fc) at buf.h:404
> > > > #13 0xc0a8db5c in vfs_bio_awrite (bp=3D0xdef881fc) at buf.h:392
> > > > #14 0xc0c584c5 in ffs_syncvnode (vp=3D0xc6b90110, waitfor=3D1)
> > > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:288
> > > > #15 0xc0c58739 in ffs_fsync (ap=3D0xf1983c4c)
> > > > at /usr/src/sys/ufs/ffs/ffs_vnops.c:187
> > > > #16 0xc0d69712 in VOP_FSYNC_APV (vop=3D0xc102dfc0, a=3D0xf1983c4c)
> > > > at vnode_if.c:1267
> > > > #17 0xc0ab5d49 in sys_fsync (td=3D0xc64ea8a0, uap=3D0xf1983cec) at
> > > > vnode_if.h:549
> > > > #18 0xc0d49315 in syscall (frame=3D0xf1983d28) at subr_syscall.c:13=
1
> > > > #19 0xc0d32af1 in Xint0x80_syscall ()
> > > > at /usr/src/sys/i386/i386/exception.s:266
> > > > #20 0x00000033 in ?? (
> > > >
> > > >
> > > > To me it looks like UFS is doing something to crash the kernel.
> > >
> > > You might try to use vmstat -z and vmstat -m on core to see what has
> > > used KVA.
--_002_B2FD678A64EAAD45B089B123FDFC3ED72B9F6C21A7inbmail01lsic_
Content-Type: application/octet-stream; name="xpt_free_ccb.patch"
Content-Description: xpt_free_ccb.patch
Content-Disposition: attachment; filename="xpt_free_ccb.patch"; size=372;
creation-date="Tue, 05 Jun 2012 12:17:56 GMT";
modification-date="Tue, 05 Jun 2012 17:12:59 GMT"
Content-Transfer-Encoding: base64
LS0tIHNjc2kvc2NzaV9hbGwuYwkyMDEyLTAxLTI1IDA2OjQ2OjE3LjAwMDAwMDAwMCArMDUzMAor
Kysgc2NzaS9zY3NpX2FsbF9uZXcuYwkyMDEyLTA2LTA2IDAxOjM2OjE2LjAwMDAwMDAwMCArMDUz
MApAQCAtMzA1Niw3ICszMDU2LDkgQEAgc2NzaV9jb21tYW5kX3N0cmluZyhzdHJ1Y3QgY2FtX2Rl
dmljZSAqZAogCQkJICAgIHNjc2lfY2RiX3N0cmluZyhjc2lvLT5jZGJfaW8uY2RiX2J5dGVzLCBj
ZGJfc3RyLAogCQkJCQkgICAgc2l6ZW9mKGNkYl9zdHIpKSk7CiAJfQotCisjaWZkZWYgX0tFUk5F
TAorICAgIHhwdF9mcmVlX2NjYigodW5pb24gY2NiKiljZ2QpOworI2VuZGlmIC8qIF9LRVJORUwv
IV9LRVJORUwgKi8KIAlyZXR1cm4oMCk7CiB9CiAK
--_002_B2FD678A64EAAD45B089B123FDFC3ED72B9F6C21A7inbmail01lsic_--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?B2FD678A64EAAD45B089B123FDFC3ED72B9F6C21A7>