From owner-freebsd-fs@freebsd.org Wed Oct 3 08:49:53 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6EA8E10BCED8 for ; Wed, 3 Oct 2018 08:49:53 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Received: from smtp-relay-int.realworks.nl (smtp-relay-int.realworks.nl [194.109.157.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 0281E72650 for ; Wed, 3 Oct 2018 08:49:52 +0000 (UTC) (envelope-from ronald-lists@klop.ws) Date: Wed, 3 Oct 2018 10:49:44 +0200 (CEST) From: Ronald Klop To: freebsd-fs@freebsd.org Message-ID: <492380418.1.1538556584486@localhost> In-Reply-To: <72e4915f-3d2f-3a3c-ada1-44f797b7244f@multiplay.co.uk> References: <029ac041-39a9-3a42-4dda-7ce94188d83c@unice.fr> <72e4915f-3d2f-3a3c-ada1-44f797b7244f@multiplay.co.uk> Subject: Re: ZFS mix disks diffrent size and speed in a pool MIME-Version: 1.0 X-Mailer: Realworks (427.25715-265900) Importance: Normal X-Priority: 3 (Normal) Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 08:49:53 -0000 After all the disks are replaced the extra space can be used. See man zpool: autoexpand=on | off Controls automatic pool expansion when the underlying LUN is grown. If set to "on", the pool will be resized according to the size of the expanded device. If the device is part of a mirror or raidz then all devices within that mirror/raidz group must be expanded before the new space is made available to the pool. The default behavior is "off". This property can also be referred to by its shortened column name, expand. So it can be nice to start using bigger disks (2+ TB) if you know you need more space in the future and the price is ok. Regards, Ronald. Van: Steven Hartland Datum: woensdag, 3 oktober 2018 10:12 Aan: freebsd-fs@freebsd.org Onderwerp: Re: ZFS mix disks diffrent size and speed in a pool > > That will be fine, you wont benefit from the extra space or speed but it will just work. > > On 03/10/2018 09:07, Jean-Marc LACROIX wrote: > > Hello, > > > > we actually have a storage solution based on a DELL 630 + 3 JBODs > DELL MD 1420 (HBA mode), > > > > The ZFS pool is made of 9 raidz2 vdevs + 1 mirror log + 2 spares > > > > All disks are of the same type in capacity and speed. > > > > The problem is that DELL is unable to provide the SAME kind of disk > (capacity AND speed) > > > > when a disk has to be replaced. So the question is: > > > > Is it possible to replace a 1TB 7200rpm by a 1.2TB 10000rpm disk > without any consequences for the existing pool ? > > > > THanks in advance > > > > Best Regards > > > > JM > > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > > > From owner-freebsd-fs@freebsd.org Wed Oct 3 14:32:29 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CB1B310C4ED7 for ; Wed, 3 Oct 2018 14:32:28 +0000 (UTC) (envelope-from jgitlin@goboomtown.com) Received: from mail-yw1-xc31.google.com (mail-yw1-xc31.google.com [IPv6:2607:f8b0:4864:20::c31]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 664AA7D307 for ; Wed, 3 Oct 2018 14:32:28 +0000 (UTC) (envelope-from jgitlin@goboomtown.com) Received: by mail-yw1-xc31.google.com with SMTP id d126-v6so2355983ywa.5 for ; Wed, 03 Oct 2018 07:32:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=goboomtown-com.20150623.gappssmtp.com; s=20150623; h=from:message-id:mime-version:subject:date:in-reply-to:cc:to :references; bh=fjnwbuzPql1MMeMXV4ao89ZK0ceN0WkrayQhvO4fmMI=; b=bL0zA3zxDJotUssjRYP0XH7LeVMlyj+pEjSZ8cIeBLof2iQIBM3doZc19g7nkmY6+Y oGYyGDY9ypjPZpZNkgIfzQUG/ckKDdhW8yDWulb7R36EJ/IZ+k9V8vLqMBczHUS9VOU/ gl7Ci/7nP5ODB56336rA3t/VWi9gj9RWGe2o9H7qRlUn4athUp4P8iGkd+u9bkQjrb77 ceSiuF5LzugNJCuKScPh7HB8bYAynUlDCEaovjrUw59li8sVerV1QXeWnawvG0vg/qoc UZ/jbeVMF+deH9OLtMabqrYFINOdjPTax1plyQzols+7bwZ/aJl+yR03AUZNer6bBEJR HkvA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:message-id:mime-version:subject:date :in-reply-to:cc:to:references; bh=fjnwbuzPql1MMeMXV4ao89ZK0ceN0WkrayQhvO4fmMI=; b=JFzYJjFe3ptwmQ03lip4uEheFv+vxN2J6k3j1XHSIAX2Dlu3ReHio3FuIGctLiKjtA e5BZQIWt02G3Qhr/21Eghlt9Itmx2P1nhzLT9EhW3GkqJEwOAYaUYRO/K8o3P0uW+LFz Edv3JkS3NfGgh9Jo6pMhkHHa1ZH/NaTbKacS9eBQovV31EcV1b6CTkVW3EvwOHxuSBvc lE0uXaZu/KHi0EvzgqFj3ooomU0aEBxm0hJfU8tliQ0PdE9sRDfh0D0ywANkxPUKceKa fE+TAveg+bTqsxR8BdPPJSl+2RcHJUWgHtM99RLlanNwPiPTQtyPxG2YpHp8mWnUxiw7 PZDw== X-Gm-Message-State: ABuFfohXJQR9tQ4lrFwi0DQpdrfq+ICy6DcEs0xn9zhXk3MA1XC04XkY +QHPMQ7c8PY/jlRmv25S3VZWauV7JRBPtw== X-Google-Smtp-Source: ACcGV63OLTDX+N1EJdASAcWLavY3Y2BRXSpzKpkANdJBot6kjT/xyAuZ8fcSo2mKOqE/MXiLUaoFdA== X-Received: by 2002:a81:81c6:: with SMTP id r189-v6mr1057852ywf.494.1538577146950; Wed, 03 Oct 2018 07:32:26 -0700 (PDT) Received: from yyz.farcry.sitepalette.com (047-134-241-203.res.spectrum.com. [47.134.241.203]) by smtp.gmail.com with ESMTPSA id e82-v6sm689716ywa.60.2018.10.03.07.32.26 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 03 Oct 2018 07:32:26 -0700 (PDT) From: Josh Gitlin Message-Id: Mime-Version: 1.0 (Mac OS X Mail 11.5 \(3445.9.1\)) Subject: Re: Troubleshooting kernel panic with zfs Date: Wed, 3 Oct 2018 10:32:25 -0400 In-Reply-To: To: freebsd-fs@freebsd.org References: X-Mailer: Apple Mail (2.3445.9.1) Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.27 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 03 Oct 2018 14:32:29 -0000 Following up on this, a bug was just posted to the stable@freebsd.org = list where the stack trace exactly matches = what I was seeing. See: = https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D231296 On our end we reduced ARC and other memory tunables and have not seen a = panic *yet* but they were unpredictable before so I am not 100% sure = that we've resolved the issue. CC'ing rainer@ultra-secure.de who posted the similar bug to stable@ -- =09 Josh Gitlin Senior Full Stack Developer (415) 690-1610 x155 Stay up to date and join the conversation in Relay = . > On Sep 20, 2018, at 8:27 PM, Josh Gitlin = wrote: >=20 > I am working to debug/troubleshoot a kernel panic with a FreeBSD ZFS = iSCSI server, specifically trying to determine if it's a bug or (more = likely) a misconfiguration in our settings. Server is running = 11.2-RELEASE-p2 with 15.6 GiB of RAM and has a single zpool with 4x2 = mirrored vdevs, 2x mirrored zil and 2x l2arc. Server runs pretty much = nothing other than SSH and iSCSI (via ctld) and serves VM virtual disks = to hypervisor servers over 10gbe LAN. >=20 > The server experienced a kernel panic and we unfortunately did not = have dumpdev set in /etc/rc.conf (we have since corrected this) so the = only info I have is what was on the screen before I rebooted it. = (Because it's a production system I couldn't mess around and had to = reboot ASAP) >=20 > trap number =3D 12 > panic: page fault > cpuid =3D 6 > KDB: stack backtrace: > #0 0xffffffff80b3d567 at kdb_backtrace+0x67 > #1 0xffffffff80af6b07 at vpanic+0x177 > #2 0xffffffff80af6983 at panic+0x43 > #3 0xffffffff80f77fcf at trap_fatal+0x35f > #4 0xffffffff80f78029 at trap_pfault+0x49 > #5 0xffffffff80f777f7 at trap+0x2c7 > #6 0xffffffff80f57dac at calltrap+0x8 > #7 0xffffffff80dee7e2 at kmem_back+0xf2 > #8 0xffffffff80dee6c0 at kmem_malloc+0x60 > #9 0xffffffff80de6172 at keg_alloc_slab+0xe2 > #10 0xffffffff80de8b7e at keg_fetch_slab+0x14e > #11 0xffffffff80de8364 at zone_fetch_slab+0x64 > #12 0xffffffff80de848f at zone_import+0x3f > #13 0xffffffff80de4b99 at uma_zalloc_arg+0x3d9 > #14 0xffffffff826e6ab2 at zio_write_compress+0x1e2 > #15 0xffffffff826e574c at zio_execute+0xac > #16 0xffffffff80bled74 at taskqueue_run_locked+0x154 > #17 0xffffffff80b4fed8 at taskqueue_thread_loop+0x98 > Uptime: 18d18h31m6s > mpr0: Sending StopUnit: path (xpt0:mpr0:0:10:ffffffff): handle 10=20 > mpr0: Incrementing SSU count > mpr0: Sending StopUnit: path (xpt0:mpr0:0:13:ffffffff): handle 13=20 > mpr0: Incrementing SSU count > mpr0: Sending StopUnit: path Ixpt0:mpr0:0:16:ffffffff): handle 16=20 > mpr0: Incrementing SSU count >=20 > My hunch is that, given this was inside kmem_malloc, we were unable to = allocate memory for a zio_write_compress call (the pool does have ZFS = compression on) and hence this is a tuning issue and not a bug... but I = am looking for confirmation and/or suggested changes/troubleshooting = steps. The ZFS tuning configuration has been stable for years, to it may = be a change in behavior or traffic... If this looks like it might be a = bug, I will be able to get more information from a minidump if it = reoccurs and can follow up on this thread. >=20 > Any advice or suggestions are welcome! >=20 > [jgitlin@zfs3 ~]$ zpool status > pool: srv > state: ONLINE > scan: scrub repaired 0 in 2h32m with 0 errors on Tue Sep 11 20:32:18 = 2018 > config: >=20 > NAME STATE READ WRITE CKSUM > srv ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > gpt/s5 ONLINE 0 0 0 > gpt/s9 ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > gpt/s6 ONLINE 0 0 0 > gpt/s10 ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > gpt/s7 ONLINE 0 0 0 > gpt/s11 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > gpt/s8 ONLINE 0 0 0 > gpt/s12 ONLINE 0 0 0 > logs > mirror-4 ONLINE 0 0 0 > gpt/s2-zil ONLINE 0 0 0 > gpt/s3-zil ONLINE 0 0 0 > cache > gpt/s2-cache ONLINE 0 0 0 > gpt/s3-cache ONLINE 0 0 0 >=20 > errors: No known data errors >=20 > ZFS tuning: >=20 > vfs.zfs.delay_min_dirty_percent=3D90 > vfs.zfs.dirty_data_max=3D4294967296 > vfs.zfs.dirty_data_sync=3D3221225472 > vfs.zfs.prefetch_disable=3D1 > vfs.zfs.top_maxinflight=3D128 > vfs.zfs.trim.txg_delay=3D8 > vfs.zfs.txg.timeout=3D20 > vfs.zfs.vdev.aggregation_limit=3D524288 > vfs.zfs.vdev.scrub_max_active=3D3 > vfs.zfs.l2arc_write_boost=3D134217728 > vfs.zfs.l2arc_write_max=3D134217728 > vfs.zfs.l2arc_feed_min_ms=3D200 > vfs.zfs.min_auto_ashift=3D12 >=20 >=20 > -- > =09 > Josh Gitlin > Senior DevOps Engineer > (415) 690-1610 x155 >=20 > Stay up to date and join the conversation in Relay = . >=20