From owner-freebsd-questions@FreeBSD.ORG Thu Jan 14 16:16:14 2010 Return-Path: Delivered-To: freebsd-questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4B320106566C for ; Thu, 14 Jan 2010 16:16:14 +0000 (UTC) (envelope-from ivoras@gmail.com) Received: from mail-fx0-f227.google.com (mail-fx0-f227.google.com [209.85.220.227]) by mx1.freebsd.org (Postfix) with ESMTP id CB32F8FC1B for ; Thu, 14 Jan 2010 16:16:13 +0000 (UTC) Received: by fxm27 with SMTP id 27so991796fxm.3 for ; Thu, 14 Jan 2010 08:16:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:sender:received:in-reply-to :references:from:date:x-google-sender-auth:message-id:subject:to:cc :content-type:content-transfer-encoding; bh=iX7MFuMc0oXYwU/bXmK5KOEGd9hKYHRBSekgPq9HFyw=; b=AlZSZZyIyepL3GhLYTLHICfXnbaWOduNsZPKho+XQ+O0pIXBEmbDM29dIogCfkTNnQ ERYV+yRrHoIIbjEfe75ULR89VoJbStqudQ1NFz+/bM9Ey+9zYOdp4kpXSqI0gJlq3X3G B0f1Hz6NnWMrXozWnuCHJ0NrlaQIonWhKdjkI= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :content-transfer-encoding; b=KOa6qXfVJ3RSUbfb0h7PawTNPvStIvocXjXhABUsa07mcWsyH7o9+PJE9YvjlSTbQt jsxccURkhO8HRmuhtle/yiRyiNu6uQRx6OszH0ljVbAJ+IF7Ej6AJV1TZQ5RBeHAjFXl uUo9J+XA2nhuBNNhlLfal3Cpv7tgC6Tn74+Uo= MIME-Version: 1.0 Sender: ivoras@gmail.com Received: by 10.216.85.5 with SMTP id t5mr353621wee.142.1263485767659; Thu, 14 Jan 2010 08:16:07 -0800 (PST) In-Reply-To: References: <8418112cdfada93d83ca0cb5307c1d21.squirrel@email.polands.org> <9bbcef731001131035x604cdea1t81b14589cb10ad25@mail.gmail.com> <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> <3aa09fd8723749d1fa65f1b9a6faac60.squirrel@email.polands.org> <27117211dd662bcf93055f4351243396.squirrel@email.polands.org> <9bbcef731001140650h5d887843ubc6d555da993e8b6@mail.gmail.com> From: Ivan Voras Date: Thu, 14 Jan 2010 17:15:47 +0100 X-Google-Sender-Auth: 1a7f25e6100da177 Message-ID: <9bbcef731001140815h5ee1d672je58c8ec91382e8d4@mail.gmail.com> To: Doug Poland Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-questions@freebsd.org Subject: Re: 8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Jan 2010 16:16:14 -0000 2010/1/14 Doug Poland : > > On Thu, January 14, 2010 08:50, Ivan Voras wrote: >> 2010/1/14 Doug Poland : >>>>> >>>>> kstat.zfs.misc.arcstats.size >>>>> >>>>> seemed to fluctuate between about 164,000,00 and 180,000,000 bytes >>>>> during this last run >>>> >>>> Is that with or without panicking? >>>> >>> with a panic >>> >>> >>>> If the system did panic then it looks like the problem is a memory >>>> leak somewhere else in the kernel, which you could confirm by >>>> monitoring vmstat -z. >>>> >>> I'll give that a try. =C2=A0Am I looking for specific items in vmstat >>> -z? =C2=A0 arc*, zil*, zfs*, zio*? =C2=A0Please advise. >> >> You should look for whatever is allocating all your memory between 180 >> MB (which is your ARC size) and 1.2 GB (which is your kmem size). >> > > OK, another run, this time back to vfs.zfs.arc_max=3D512M in > /boot/loader.conf, and a panic: > > panic: kmem malloc(131072): kmem map too small: 1294258176 total > allocated > > I admit I do not fully understand what metrics are important to proper > analysis of this issue. =C2=A0In this case, I was watching the following > within 1 second of the panic: > > sysctl kstat.zfs.misc.arcstats.size: 41739944 > sysctl vfs.numvnodes: 678 > sysctl vfs.zfs.arc_max: 536870912 > sysctl vfs.zfs.arc_meta_limit: 134217728 > sysctl vfs.zfs.arc_meta_used: 7228584 > sysctl vfs.zfs.arc_min: 67108864 > sysctl vfs.zfs.cache_flush_disable: 0 > sysctl vfs.zfs.debug: 0 > sysctl vfs.zfs.mdcomp_disable: 0 > sysctl vfs.zfs.prefetch_disable: 1 > sysctl vfs.zfs.recover: 0 > sysctl vfs.zfs.scrub_limit: 10 > sysctl vfs.zfs.super_owner: 0 > sysctl vfs.zfs.txg.synctime: 5 > sysctl vfs.zfs.txg.timeout: 30 > sysctl vfs.zfs.vdev.aggregation_limit: 131072 > sysctl vfs.zfs.vdev.cache.bshift: 16 > sysctl vfs.zfs.vdev.cache.max: 16384 > sysctl vfs.zfs.vdev.cache.size: 10485760 > sysctl vfs.zfs.vdev.max_pending: 35 > sysctl vfs.zfs.vdev.min_pending: 4 > sysctl vfs.zfs.vdev.ramp_rate: 2 > sysctl vfs.zfs.vdev.time_shift: 6 > sysctl vfs.zfs.version.acl: 1 > sysctl vfs.zfs.version.dmu_backup_header: 2 > sysctl vfs.zfs.version.dmu_backup_stream: 1 > sysctl vfs.zfs.version.spa: 13 > sysctl vfs.zfs.version.vdev_boot: 1 > sysctl vfs.zfs.version.zpl: 3 > sysctl vfs.zfs.zfetch.array_rd_sz: 1048576 > sysctl vfs.zfs.zfetch.block_cap: 256 > sysctl vfs.zfs.zfetch.max_streams: 8 > sysctl vfs.zfs.zfetch.min_sec_reap: 2 > sysctl vfs.zfs.zil_disable: 0 > sysctl vm.kmem_size: 1327202304 > sysctl vm.kmem_size_max: 329853485875 > sysctl vm.kmem_size_min: 0 > sysctl vm.kmem_size_scale: 3 > > > vmstat -z | egrep -i 'zfs|zil|arc|zio|files' > ITEM =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 SIZE =C2=A0 =C2=A0 LIMIT =C2=A0 =C2=A0 =C2=A0USED =C2=A0 =C2=A0 =C2=A0F= REE =C2=A0REQUESTS > Files: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 80, =C2=A0 =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 =C2=A0116, =C2=A0 =C2= =A0 =C2=A0199, =C2=A0 850713 > zio_cache: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0720, = =C2=A0 =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A053562, =C2=A0 =C2=A0 =C2=A0 98, = 86386955 > arc_buf_hdr_t: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0208, =C2=A0 =C2= =A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 1193, =C2=A0 =C2=A0 =C2=A0 31, =C2=A0 =C2= =A011990 > arc_buf_t: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 72, = =C2=A0 =C2=A0 =C2=A0 =C2=A00, =C2=A0 =C2=A0 1180, =C2=A0 =C2=A0 =C2=A0120, = =C2=A0 =C2=A011990 > zil_lwb_cache: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0200, =C2=A0 =C2= =A0 =C2=A0 =C2=A00, =C2=A0 =C2=A011580, =C2=A0 =C2=A0 2594, =C2=A0 =C2=A062= 407 > zfs_znode_cache: =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0376, =C2=A0 =C2=A0 =C2= =A0 =C2=A00, =C2=A0 =C2=A0 =C2=A0605, =C2=A0 =C2=A0 =C2=A0 55, =C2=A0 =C2= =A0 =C2=A0654 > > vmstat -m |grep solaris|sed 's/K//'|awk '{print "vm.solaris:", $3*1024}' > > > =C2=A0solaris: 1285068800 > > > The value I see as the culprit is vmstat -m | grep solaris. =C2=A0This > value fluctuates wildly during the run and is always near kmem_size at > the time of the panic. > > Again, I'm not sure what to look for here, and you are patiently > helping me along in this process. =C2=A0If you have any tips or can point > me to docs on how to easily monitor these values, I will endeavor to > do so. The only really important ones should be kstat.zfs.misc.arcstats.size (which you very rarely print) and vm.kmem_size. The "solaris" entry above should be near kstat.zfs.misc.arcstats.size in all cases. But I don't have any more ideas here. Try taking this post (also include kstst.zfs.misc.arcstats.size) to the freebsd-fs@ mailing list.