From owner-freebsd-fs@FreeBSD.ORG Tue Nov 6 13:18:22 2007 Return-Path: Delivered-To: freebsd-fs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 180BA16A417; Tue, 6 Nov 2007 13:18:22 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from proxypop2.sarenet.es (proxypop2.sarenet.es [194.30.0.95]) by mx1.freebsd.org (Postfix) with ESMTP id C122013C48E; Tue, 6 Nov 2007 13:18:21 +0000 (UTC) (envelope-from BORJAMAR@SARENET.ES) Received: from localhost (unknown [212.81.200.214]) by proxypop2.sarenet.es (Postfix) with ESMTP id EBBAE731E2; Tue, 6 Nov 2007 14:18:12 +0100 (CET) Message-Id: From: Borja Marcos To: Pawel Jakub Dawidek In-Reply-To: <20071106100015.GB5268@garage.freebsd.pl> Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Content-Transfer-Encoding: 7bit Mime-Version: 1.0 (Apple Message framework v912) Date: Tue, 6 Nov 2007 14:18:04 +0100 References: <20071005000046.GC92272@garage.freebsd.pl> <20071008121523.GM2327@garage.freebsd.pl> <20071105215035.GC26730@heff.fud.org.nz> <2e77fc10711051531k41e7224dq6aaedb35cad8d9f2@mail.gmail.com> <6214AB9C-9F9B-4B9D-8B05-0B3DF5F6C16D@SARENET.ES> <20071106100015.GB5268@garage.freebsd.pl> X-Mailer: Apple Mail (2.912) Cc: freebsd-fs@freebsd.org, freebsd-current@freebsd.org Subject: Re: ZFS kmem_map too small. X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 06 Nov 2007 13:18:22 -0000 On Nov 6, 2007, at 11:00 AM, Pawel Jakub Dawidek wrote: > If you use vm_kern.c.2.patch, can you show loader.conf and exact > command > that can provke the panic? I can elaborate a bit. I repeated the test while I kept some connections open, watching zpool iostat, the Bonnie++ command output, the make buildworld, and pinging the machine continuously. Wired memory reached 1696 MB according to top, and the programs seemed to be stopped. I didn't see more activity from top, zpool iostat, or make buildworld. But the kernel was still answering to pings, although with a very long delay, an average of 300 - 500 ms. I'm connected to the same local network, and the typical delay I get with the machine is around 0.3 ms. It was until things started to go wrong. The machine kept answering pings (very late, but with no packet loss), until I pressed ctrl-C at the terminal session where I was running Bonnie++. That seemed to trigger the panic, kmem_map to small. And I've found something. I was using SCHED_ULE instead of SCHED_4BSD. I've switched back to SCHED_4BSD (I apologise I didn't mention it, I thought it shouldn't make a difference) and the kmem usage climbs much slower, sustaning a 30 MB/s write throughput during the Bonnie's "writing intelligently" test. Using ULE, wired memory climbs and tops 1.6 GB in less than a minute, while using 4BSD it's climbing much slower. I've seen it reach 1.2 GB, go back to 800 MB... and suddenly go down to 256 MB, while it keeps sustaining a write throughput of about 25 - 30 MB/s (according to zpool iostat at 10 second intervals). I've compiled the kernel with debugger support and this is what the backtrace reads: panic kmem_malloc(131072): kmem_map too small: 1608417280 total allocated cpuid=1 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x17a kmem_malloc at kmem_malloc+0x65e uma_large_malloc() at uma_large_malloc+0x4a malloc() at malloc+0x7d vdev_queue_io_to_issue() at vdev_queue_io_to_issue+0x147 vdev_queue_io_done() at vdev_queue_io_done+0x9c vdev_geom_io_done() at vdev_geom_io_done+0x11 taskq_thread() at taskq_thread+0x17b fork_exit() at fork_exit+0x11f fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xfffffffffee34d30, rbp = 0 --- KDB: enter: panic Hope it makes sense. Anything else I can provide? Borja ---------------- "The thing he realised about the windows was this: because they had been converted into openable windows after they had first been designed to be impregnable, they were, in fact, much less secure than if they had been designed as openable windows in the first place." Douglas Adams, "Mostly Harmless"