From owner-freebsd-fs@freebsd.org Tue Jan 22 15:21:18 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CC67814A50BC for ; Tue, 22 Jan 2019 15:21:18 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x532.google.com (mail-ed1-x532.google.com [IPv6:2a00:1450:4864:20::532]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id B1B1A9069A for ; Tue, 22 Jan 2019 15:21:17 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x532.google.com with SMTP id p6so19663975eds.0 for ; Tue, 22 Jan 2019 07:21:17 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-transfer-encoding:content-language; bh=0j2pkuXIH3Y3ovsBp09AjVexgb3QAIvPl/A1Qs9ScB0=; b=d5DUtEeOHVzSZr2OmvadW73o1XQkmObKrxQz8jmIrlK7roYQh8jOG7T8fGjTzXguau u4+tCwPC6lXmYP9Vj4HljZSukj/BGy3W1X1lZLAMVB/dxv4eoDrxcpHbRyBMR0OwRwvl hJHTlqmFaTDoZ09kTemR7AsZas10zQi1okAdOz44qaBNAXQ1YiqBXdckiDxWve3bqwdL 0mROxNU2+KF2+jwJt/CgojyMPi6/4WS6yIKPbohcuwIT7whJgwoFa6ptwetkbdx/KgnA 5dc/JEvaIZnBtdpHTH4B3kxlCL09UPfUJZji1XONBSWY9mSVoevcPGf5qZmqgEHMZEt4 EtpQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-transfer-encoding :content-language; bh=0j2pkuXIH3Y3ovsBp09AjVexgb3QAIvPl/A1Qs9ScB0=; b=DMkQ7o8eP1ooJRaPtGCjiUqG6L5J2zBpN2oTeotirLH3aw/7CS0j3wLEhupppCEUz/ 0WShnbypKiExlIHJjZND3XXsrlFZC+4F3XQ+IzF+/EHBdZrk+P68VeopxFrGc0tfT/13 bcE2+XaG3gQtpK8bILAorEFGRMwcdiYKUFK02GOEIxbhpmAxKGm7yuj4iY1kpp2IgV9d Jt8RTradA8dWjeUjS9fFeFcj2acovo0CtP4AdcYFA5OkF+y3gDfhfDCHx4VL1PKxR9es jXa+17pQNOhcIFk5MmJw4eVXthYzC0dolUi025MLUAU7KU0PIPPJ7nyp8s7xNpJORaPA 1uRQ== X-Gm-Message-State: AJcUukeBSBXpkuPpWcwwMnsOXiLAva7g3lwc7ZT6rXzjGGairTHQDOXj BeSb1ICOvl+vauMrMbA0JvmaoNbJznQ= X-Google-Smtp-Source: ALg8bN5Sr8aJQwbfPAFvzdcM3zhQP9daFam/YYR7uj/v/bzsfSLDctflSqEc44U4JqQ5TJFQ2n9hZw== X-Received: by 2002:a05:6402:643:: with SMTP id u3mr31027054edx.276.1548170475488; Tue, 22 Jan 2019 07:21:15 -0800 (PST) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id a38sm9548765edd.44.2019.01.22.07.21.14 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 22 Jan 2019 07:21:14 -0800 (PST) Subject: Re: ZFS on Hardware RAID To: Borja Marcos Cc: jdelisle , freebsd-fs References: <1180280695.63420.1547910313494.JavaMail.zimbra@gausus.net> <92646202.63422.1547910433715.JavaMail.zimbra@gausus.net> <1691666278.63816.1547976245836.JavaMail.zimbra@gausus.net> <92746659-4B3F-415C-BB6A-6C99837AFAF2@sarenet.es> <335e44ec-7c76-8dbd-f587-46e6a9266efc@multiplay.co.uk> From: Steven Hartland Message-ID: Date: Tue, 22 Jan 2019 15:21:15 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Rspamd-Queue-Id: B1B1A9069A X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=d5DUtEeO; spf=pass (mx1.freebsd.org: domain of killing@multiplay.co.uk designates 2a00:1450:4864:20::532 as permitted sender) smtp.mailfrom=killing@multiplay.co.uk X-Spamd-Result: default: False [-6.11 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: ASPMX.L.GOOGLE.COM]; RCVD_IN_DNSWL_NONE(0.00)[2.3.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.99)[-0.985,0]; IP_SCORE(-2.61)[ip: (-8.93), ipnet: 2a00:1450::/32(-2.18), asn: 15169(-1.89), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; FREEMAIL_CC(0.00)[gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 22 Jan 2019 15:21:19 -0000 On 22/01/2019 11:52, Borja Marcos wrote: > >> On 21 Jan 2019, at 16:40, Steven Hartland wrote: >> >> Typically yes it does. >>> The same could be said of other functionalities available when you talk directly >>> to the disks. >>> >>> >>> >>>> That's why I did as I describe above. I didn't have the money to buy an >>>> HBA until a year or two later, and I had good backups, so I used the PERC >>>> 6/e. >>>> >>> Depending on the LSI model used in that card you can make it behave like an HBA. >>> >>> >> Indeed you can for most, but its not an easy or error free option, requires force flashing from IR to IT firmware, and if it goes wrong you can end up with a bricked controller. >> >> We've used LSI's in IR mode (RAID) but as individual disks JBOD with ZFS raid2z or mirroring on top for a long time and on lots of machines, not had any issues to report. So while I would never use hardware RAID from those controllers with ZFS, JBOD on them is an easy and reliable option in our experience. > Sorry to insist, but maybe we are talking of different stuff? My apologies in advance if I am confused and/or my > information is outdated. > > JBOD mode: a mode offered by some LSI Logic IR controllers to create virtual devices mapped so that each one is effectively. > For example, with the mfi driver the “disks” appear as “mfisyspd” devices while the “conventional” logical volumes appear as “mfid”. My information could be outdated though? Cards that use mfi are quite a different beasts, they are more a pure RAID controller, than cards which have both IT & IR modes, where the RAID capabilities are more limited. The later present as daX devices where the former presents as mfid unless you force otherwise using mfi/mfip as you detail below. > HBA mode: when the card is in IT mode *or* it does expose the actual targets to the CAM layer. In the past I did it > by manually patching the drivers and I’ve kept systems running smoothly for many years despite using IR cards. > Currently you can use hw.mfi.allow_cam_disk_passthrough tunable to achieve the same effect without ugly tinkering. > > Unless I am terribly wrong, when I tried the JBOD mode as defined above (with LSI SAS3 cards and IR firmware) the mfisyspd > devices were not actual CAM devices. > > Actually I find “JBOD” an artificial and confusing term because it can mean so many things. I would have rather preferred > LSI to avoid it and provide either real, transparent access to the actual targets insteaf of somewhat “virtualizing” them. > > Again, I may be confused? I haven’t tried to configure a LSI IR card as “JBOD” for a long time. Maybe now they do > expose the targets instead of defining logical volumes? I can definitely confirm LSI controllers which present individual daX devices can support TRIM / UMAP based off what the device reports. As you say its quite confusing, so hope that clarifies?     Regards     Steve