From owner-freebsd-questions@FreeBSD.ORG Wed Jan 13 20:26:43 2010 Return-Path: Delivered-To: questions@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 418B3106566C; Wed, 13 Jan 2010 20:26:43 +0000 (UTC) (envelope-from doug@polands.org) Received: from hrndva-omtalb.mail.rr.com (hrndva-omtalb.mail.rr.com [71.74.56.122]) by mx1.freebsd.org (Postfix) with ESMTP id D51C28FC1B; Wed, 13 Jan 2010 20:26:42 +0000 (UTC) X-Authority-Analysis: v=1.0 c=1 a=sVhNVL3m-NYA:10 a=tH3TCQxNhl/vWwTfLVWeHw==:17 a=bqq2Vc5EAAAA:8 a=1h4evn_gICQKwsDCVZEA:9 a=u8VXOzhpAQL4BUjkZOCpdqEahSgA:4 a=5ERLOmoKdHQA:10 X-Cloudmark-Score: 0 X-Originating-IP: 75.87.219.217 Received: from [75.87.219.217] ([75.87.219.217:63113] helo=haran.polands.org) by hrndva-oedge04.mail.rr.com (envelope-from ) (ecelerity 2.2.2.39 r()) with ESMTP id 29/6E-18377-18C2E4B4; Wed, 13 Jan 2010 20:26:42 +0000 Received: from email.polands.org (ammon.polands.org [172.16.1.7]) by haran.polands.org (8.14.3/8.14.3) with ESMTP id o0DKQfgj088172; Wed, 13 Jan 2010 14:26:41 -0600 (CST) (envelope-from doug@polands.org) Received: from 209.103.214.35 (SquirrelMail authenticated user djp) by email.polands.org with HTTP; Wed, 13 Jan 2010 14:26:41 -0600 Message-ID: <158692d18f40556dc57008f1661afd48.squirrel@email.polands.org> In-Reply-To: <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> References: <8418112cdfada93d83ca0cb5307c1d21.squirrel@email.polands.org> <9bbcef731001131035x604cdea1t81b14589cb10ad25@mail.gmail.com> <9bbcef731001131157h256c4d14mbb241bc4326405f8@mail.gmail.com> Date: Wed, 13 Jan 2010 14:26:41 -0600 From: "Doug Poland" To: "Ivan Voras" User-Agent: SquirrelMail/1.4.20-RC2 MIME-Version: 1.0 Content-Type: text/plain;charset=iso-8859-1 Content-Transfer-Encoding: 8bit X-Priority: 3 (Normal) Importance: Normal Cc: questions@freebsd.org Subject: Re: 8.0-R-p2 ZFS: unixbench causing kmem exhaustion panic X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 13 Jan 2010 20:26:43 -0000 On Wed, January 13, 2010 13:57, Ivan Voras wrote: > 2010/1/13 Doug Poland : >> This is the state of the machine when it panicked this time: >> >> panic: kmem_malloc(131072): kmem_map too small: 1296957440 total >> allocated >> cpuid = 1 >> >> /boot/loader.conf: vfs.zfs.arc_max=512M >> vfs.numvnodes: 660 >> vfs.zfs.arc_max: 536870912 >> vfs.zfs.arc_meta_limit: 134217728 >> vfs.zfs.arc_meta_used: 7006136 >> vfs.zfs.arc_min: 67108864 >> vfs.zfs.zil_disable: 0 >> vm.kmem_size: 1327202304 >> vm.kmem_size_max: 329853485875 > > (from the size of arc_max I assume you did remember to reboot after > changing loader.conf and before testing again but just checking - did > you?) > Yes, I did reboot > Can you monitor and record kstat.zfs.misc.arcstats.size sysctl while > the test is running (and crashing)? > Certainly > This looks curious - your kmem_max is ~~ 1.2 GB, arc_max is 0.5 GB and > you are still having panics. Is there anything unusual about your > system? Like unusually slow CPU, unusually fast or slow drives? > Don't think there is anything unusual. This is 5 year old HP DL385. It has two 2.6GHz Opteron 252 CPUs. The disks are 6x36GB P-SCSI. There are behind an HP Smart Array 6i controller. I had to configure each drive as "RAID0" in order make it visible to the OS. Kinda hokey if you ask me. dmesg | grep -i CPU CPU: AMD Opteron(tm) Processor 252 (2605.92-MHz K8-class CPU) FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs smartctl -a /dev/da0 Device: COMPAQ RAID 0 VOLUME Version: OK Device type: disk Local Time is: Wed Jan 13 14:21:44 2010 CST Device does not support SMART dmesg | grep -i smart ciss0: port 0x5000-0x50ff mem 0xf7ef0000-0xf7ef1fff,0xf7e80000-0xf7ebffff irq 24 at device 4.0 on pci2 > I don't have any ideas smarter than reducing arc_max by half then try > again and continue reducing it until it works. It would be very > helpful if you could monitor the kstat.zfs.misc.arcstats.size sysctl > while you are doing the tests to document what is happening to the > system. If it by any chance stays the same you should probably monitor > "vmstat -m". > OK, will do monitor on the next run. Thanks for your help so far. -- Regards, Doug