From owner-freebsd-bugs@FreeBSD.ORG Fri Sep 24 11:45:15 2010 Return-Path: Delivered-To: freebsd-bugs@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 232BD1065674 for ; Fri, 24 Sep 2010 11:45:15 +0000 (UTC) (envelope-from leroy.vanlogchem@gmail.com) Received: from mail-qy0-f182.google.com (mail-qy0-f182.google.com [209.85.216.182]) by mx1.freebsd.org (Postfix) with ESMTP id CD5EF8FC14 for ; Fri, 24 Sep 2010 11:45:14 +0000 (UTC) Received: by qyk7 with SMTP id 7so842505qyk.13 for ; Fri, 24 Sep 2010 04:45:14 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:received:received:date:message-id :subject:from:to:content-type; bh=YOHAij+U/cMWCD/7RitVhC3K164K1U1se2WqXDX4NDA=; b=OyxVtonGrsmME3TlaEHePOXd8KrBxbN+6IgygCsIMeKj3PRK8toaNZSHf5A1tWeOZa s6Qp5prEBKl2CMrlbBtr8Tv4+qUDHNiAbDaCgKnQXiXlEbqfl8AaHM0W9C2bdjOBB0ZN JjNs7QmrQuK3xyy18PnVnF9OjvvgYY2jPYq9s= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:date:message-id:subject:from:to:content-type; b=wps+/1yDY2jlHT1OASwh2sDpUt0YKo8330dwty6Gcd1uwq4rBdE5mR8XIY7swT+d36 dErZuytxRujmEX9GdDXP0U6B26ejRr7jH+nUu8+dX8dlQRVkwQwZx4IROwuMwHwRBgv6 aUhN/0dspRLGeyibZy6XawE9gMBf8G7+QDdC4= MIME-Version: 1.0 Received: by 10.224.65.169 with SMTP id j41mr2255359qai.154.1285327332801; Fri, 24 Sep 2010 04:22:12 -0700 (PDT) Received: by 10.229.2.25 with HTTP; Fri, 24 Sep 2010 04:22:12 -0700 (PDT) Date: Fri, 24 Sep 2010 13:22:12 +0200 Message-ID: From: Leroy van Logchem To: freebsd-bugs@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 Subject: 8.1-RELEASE: [zfs] [kmem] zfs destroy snapshot results in panic: kmem_map too small X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 24 Sep 2010 11:45:15 -0000 ----------------------------------------------------------------------------- Problem : Kernel panic "kmem_malloc(114688): kmem_map too small Trigger : Destroy ZFS snapshots (each bigger >80GB) Version : FreeBSD 8.1-RELEASE (GENERIC AMD64 but with DDB) Workaround : kmem / zfs arc tuning in /boot/loader.conf System Info : Manufacturer: Supermicro Mainboard : X8DTN Cpu(s) Two : Intel(R) Xeon(R) E5520 @ 2.27GHz Memory : 10240 MB (speed: 1066 MHz) Storage : mpt0 Adapter: Board Name: SAS3081E Board Assembly: L3-00159-02D Chip Name: C1068E 24 drives are connected to mpt0: /dev/da[0 up to 23] Each drive : 512 # sectorsize 2000398934016 # mediasize in bytes (1.8T) 3907029168 # mediasize in sectors 243201 # Cylinders according to firmware. 255 # Heads according to firmware. 63 # Sectors according to firmware. JK1130YAHUSJMT # Disk ident. # zpool list NAME SIZE USED AVAIL CAP HEALTH ALTROOT tank 43.5T 31.0T 12.5T 71% ONLINE - ----------------------------------------------------------------------------- This server has been running since dec 2009 without downtime. We keep 14 daily zfs snapshots. The problem of our nightly kernel panic started when the size of each snapshot exceeded about 80 GB. A cronjob first creates new snapshots just fine and destroys expired (older >14 days) snapshots. During the command "zfs destroy " the memory free seen in top(1) drops very quickly and soon after that this panic is generated: panic: kmem_malloc(114688): kmem_map too small: 3307884544 total allocated cpuid = 2 KDB: stack backtrace: db_trace_self_wrapper() at db_trace_self_wrapper+0x2a panic() at panic+0x182 kmem_malloc() at kmem_malloc+0x5b5 uma_large_malloc() at uma_large_malloc+0x4a malloc() at malloc+0x14b zio_compress_data() at zio_compress_data+0xa2 zio_write_bp_init() at zio_write_bp_init+0xc2 zio_exectute() at zio_execute+0x77 taskq_run_safe() at taskq_run_safe+0x13 taskqueue_run() at taskqueue_run+0x91 taskqueue_thread_loop() at taskqueue_thread_loop+0x3f fork_exit() at fork_exit+0x12a fork_trampoline() at fork_trampoline+0xe --- trap 0, rip = 0, rsp = 0xffffff81261edd30, rbp = 0 --- panic: kmem_malloc(118784): kmem_map too small: 3307884544 total allocated cpuid = 2 During these panics no tuning or /boot/loader.conf values where present. Only after hitting this behaviour yesterday i created boot/loader.conf: vm.kmem_size=18G vfs.zfs.arc_max=6G After which the snapshots could be destroyed without panic. Questions: Are users required to tune? If so, are there reliable guidelines regarding memory and ZFS? Are the -quickly- chosen values any good? (service window was closing) --- Leroy