From owner-freebsd-fs@freebsd.org Mon Jan 21 15:40:23 2019 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D934514ACF57 for ; Mon, 21 Jan 2019 15:40:22 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: from mail-ed1-x536.google.com (mail-ed1-x536.google.com [IPv6:2a00:1450:4864:20::536]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 90EC88E7A9 for ; Mon, 21 Jan 2019 15:40:21 +0000 (UTC) (envelope-from killing@multiplay.co.uk) Received: by mail-ed1-x536.google.com with SMTP id a20so16901862edc.8 for ; Mon, 21 Jan 2019 07:40:21 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=multiplay-co-uk.20150623.gappssmtp.com; s=20150623; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language; bh=TAtvgJqfD9+stMlaNiZjDHcxJoINacw11hxS3UREFY4=; b=Q3qeZB4dPhsEfS6zDiGGsnz+SVsS3K+adwQtJkwAT3pg3wmVGTAGi4IXZCLXBS+gi1 TuiyVh2EogaZmcBUPCvNk38SOIG4l+mnSX5oz06XV5MvXvxtXUvYXoKjMIvB3/BG7BS9 M7ElwCuvivPWElFqtt1boXXOAtYuWM8Qd/9zIYS+KE5DJy71obK/ocYUhKK1/WLtOlca lLMAXD9Gt23Gvoi+FArzU5Esdk5SZ5IvaANH1yqfXtNbXNlDvRqqh0U6w1C1coFPf6JO fkUkSHy0ptlGxKo65fx/GjyDH24kWQgpCLbh4RVNkrGO/f+hrnLHQYzB+pCrHoGsIXPx xRqg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language; bh=TAtvgJqfD9+stMlaNiZjDHcxJoINacw11hxS3UREFY4=; b=OAm00lA3nXdEI7Q/XOIj6O8cUNfYvGyLYKdp7OB9t44iJ1kg/0f/kqDgy1fbqBajgn 4fmJ1a5nf8LrvMfGdwgs1F1EnsVujSav5j1PrPQK1uWo66C+J5qDjLySO3VPb7s4Psq6 QU7jlj1asXlX9MgCi95ade8l0r7WnRQ9KAHPNKkPsTvkdNEr1aF2RrjAe4gqVr3liC2u xZhelOW1JQ96sJL1JAKp+fuZgBr5CLQbOUc3iSfaw8nxNbiH/Z9IUQufouuk+Ql8MzqW wgR9+vrCgW8bfwAkDm7zSTfNvEG/gM8PXPnVV4WBAjnyVJXD0CZCG7j0/fKB9XJsP/ID kURw== X-Gm-Message-State: AJcUukdph3MQepdgfZPayIYaDHVIme1CG4ZGSyyfXzTYke9j6kG7fCST ym2tP/7Tf8SLoeNVspv+Y0yDxfoC+gA= X-Google-Smtp-Source: ALg8bN6OeNGqwtuLl1xfYrtacrWtufS+PDiQ0b1/OsmL8UOgBG1hFzLF4mDuYzGAkFWtBVavnqgU4g== X-Received: by 2002:a50:fc07:: with SMTP id i7mr27361516edr.153.1548085219574; Mon, 21 Jan 2019 07:40:19 -0800 (PST) Received: from [10.44.128.75] ([161.12.40.153]) by smtp.gmail.com with ESMTPSA id b19-v6sm5111196ejp.77.2019.01.21.07.40.18 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 21 Jan 2019 07:40:18 -0800 (PST) Subject: Re: ZFS on Hardware RAID To: Borja Marcos , jdelisle Cc: freebsd-fs References: <1180280695.63420.1547910313494.JavaMail.zimbra@gausus.net> <92646202.63422.1547910433715.JavaMail.zimbra@gausus.net> <1691666278.63816.1547976245836.JavaMail.zimbra@gausus.net> <92746659-4B3F-415C-BB6A-6C99837AFAF2@sarenet.es> From: Steven Hartland Message-ID: <335e44ec-7c76-8dbd-f587-46e6a9266efc@multiplay.co.uk> Date: Mon, 21 Jan 2019 15:40:18 +0000 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <92746659-4B3F-415C-BB6A-6C99837AFAF2@sarenet.es> Content-Language: en-US X-Rspamd-Queue-Id: 90EC88E7A9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=multiplay-co-uk.20150623.gappssmtp.com header.s=20150623 header.b=Q3qeZB4d; spf=pass (mx1.freebsd.org: domain of killing@multiplay.co.uk designates 2a00:1450:4864:20::536 as permitted sender) smtp.mailfrom=killing@multiplay.co.uk X-Spamd-Result: default: False [-6.12 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[multiplay-co-uk.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[multiplay.co.uk]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[multiplay-co-uk.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[ASPMX.L.GOOGLE.COM,ALT2.ASPMX.L.GOOGLE.COM,ALT1.ASPMX.L.GOOGLE.COM,ASPMX3.GOOGLEMAIL.COM,ASPMX2.GOOGLEMAIL.COM]; RCVD_IN_DNSWL_NONE(0.00)[6.3.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.97)[-0.973,0]; IP_SCORE(-2.64)[ip: (-9.10), ipnet: 2a00:1450::/32(-2.15), asn: 15169(-1.86), country: US(-0.08)]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[] Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 21 Jan 2019 15:40:23 -0000 On 21/01/2019 14:17, Borja Marcos wrote: > >> On 20 Jan 2019, at 16:33, jdelisle wrote: >> I can say for certain that with Dell PERC 6/e RAID (SAS 8087ELP based) >> cards, when the virtual devices are entire disks each in a RAID0 >> configuration, the way the adapter writes data to them is effectively the >> same as JBOD. You can take a RAID0 drive from a PERC 6/e controller and >> use in with an HBA/ "IT Mode" JBOD controller without issue. I personally >> migrated 14x 1TB drives off PERC 6/e (all RAID0's) to a SAS 6gbps HBA, and >> my ZFS VDEV's and pools were totally fine. You can move back and forth no >> problem. I can't say this is the case for ALL RAID controllers, I've only >> done this with an old PERC 6/e. > However a single disk RAID0 volume (what manufacturers call a JBOD) is not the same > as a disk. With the “virtual JBOD” you lose direct CAM access, which means that some > advanced options won’t be supported. > > For example, if you are using SSDs ZFS knows when to issue a TRIM (or, in general, > BIO_DELETE) command. Does the “virtual JBOD” offer that option? Typically yes it does. > The same could be said of other functionalities available when you talk directly > to the disks. > > >> That's why I did as I describe above. I didn't have the money to buy an >> HBA until a year or two later, and I had good backups, so I used the PERC >> 6/e. > Depending on the LSI model used in that card you can make it behave like an HBA. > Indeed you can for most, but its not an easy or error free option, requires *force *flashing from IR to IT firmware, and if it goes wrong you can end up with a bricked controller. We've used LSI's in IR mode (RAID) but as individual disks JBOD with ZFS raid2z or mirroring on top for a long time and on lots of machines, not had any issues to report. So while I would never use hardware RAID from those controllers with ZFS, JBOD on them is an easy and reliable option in our experience.     Regards     Steve