From owner-freebsd-stable@freebsd.org Thu Apr 11 18:52:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8DF4B15662F0 for ; Thu, 11 Apr 2019 18:52:41 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-it1-x12b.google.com (mail-it1-x12b.google.com [IPv6:2607:f8b0:4864:20::12b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A5FEF721F7 for ; Thu, 11 Apr 2019 18:52:40 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-it1-x12b.google.com with SMTP id f22so11309950ita.3 for ; Thu, 11 Apr 2019 11:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=p3A0JfbjDgC860HhNTG3zjTJ40nC+sjNhDOYNmPrkt4=; b=N5HRsEriDa6AOACNNJAUiZEomHsc2yLEzruxuRkm+txvfZlQLXHoY+1fgjUHRN0kzK yCk2sy6+9pqX0YFbT+RtQAbmAlLYaDTZgEp51inrs7tL/B36j5cfy5mcvenmjXl2lNlB rAylmhljjdTscLnKV2lfzr3IARgHpon5xnFgZO74gfTAycHquSuPI9MzAAlMSw9zW2uJ Tat2VBLQ3E5DKPAAp0Qpwg06J7GDHVSpvwK05tLE1ADyoxno0GjtTkN3Ren8d97982yR p1ulCiM2bg1adH62oow/XnK0crc6hHfA+NBp3c1Stv1ZkyYkMCx7yEGB3VRvC44SWLc+ PqGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=p3A0JfbjDgC860HhNTG3zjTJ40nC+sjNhDOYNmPrkt4=; b=KMQXWOdIQJlqr6kLyxJQI71zi9jqiN8hXPJH6TIjSLK3aek/wbdYwrjIpGI9mm56ot 7nrekAaSRugl5LH8WxLwatT40amAcMb+s+2q+GHSZLzV5+kV1biunnxw3lBLxjI7WOCP OmC4WfwFB6pjRQRQh3SR0FQ5M4p4oFYWpxDZTt+LYh2aiko65+ntJokMRCHmiFB1Npsf 2161iH0irXKrmvIAbEM8N4+dvR5nd1BLoSZpYHa5KMgMtANqMcGWt2WhiVEiCS/Wt7BJ +0V6La+fSxatBomslwEbZ0hSrjwkl6NG7SScyGvY95meCXa75s92LyaShTduDgw11dUH Mm7g== X-Gm-Message-State: APjAAAXlRD4H/w7fYFvZ9twDoe/A1CUTxfG3EOYFXqgG+C4kP3X5sHiP ACAHwwlwppHuOOnYbZHQ55sWXurY5pBRCJWzEA== X-Google-Smtp-Source: APXvYqyZHxOXY+ug5J/7/oAGHdzbWSnPd5P/Q56jgch2N371Weuo/sveX0jZxHcntCQlB2kKNmZCr2bQKK9XTsTRaOc= X-Received: by 2002:a24:13c7:: with SMTP id 190mr10199447itz.9.1555008759746; Thu, 11 Apr 2019 11:52:39 -0700 (PDT) MIME-Version: 1.0 References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> In-Reply-To: <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> From: Zaphod Beeblebrox Date: Thu, 11 Apr 2019 14:52:28 -0400 Message-ID: Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Karl Denninger Cc: FreeBSD Stable X-Rspamd-Queue-Id: A5FEF721F7 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=N5HRsEri; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2607:f8b0:4864:20::12b as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-4.40 / 15.00]; R_SPF_ALLOW(0.00)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RBL_SEM_IPV6(1.00)[b.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.bl.ipv6.spameatingmonkey.net]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; MIME_BASE64_TEXT(0.10)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; NEURAL_HAM_SHORT(-0.94)[-0.938,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[b.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.55)[ip: (-7.55), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 18:52:41 -0000 On Wed, Apr 10, 2019 at 10:41 AM Karl Denninger wrote: > In this specific case the adapter in question is... > > mps0: port 0xc000-0xc0ff mem > 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 > mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd > mps0: IOCCapabilities: > 1285c > > Which is indeed a "dumb" HBA (in IT mode), and Zeephod says he connects > his drives via dumb on-MoBo direct SATA connections. > Maybe I'm in good company. My current setup has 8 of the disks connected to: mps0: port 0xb000-0xb0ff mem 0xfe240000-0xfe24ffff,0xfe200000-0xfe23ffff irq 32 at device 0.0 on pci6 mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 5a85c ... just with a cable that breaks out each of the 2 connectors into 4 SATA-style connectors, and the other 8 disks (plus boot disks and SSD cache/log) connected to ports on... - ahci0: port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xfe900000-0xfe9001ff irq 44 at device 0.0 on pci2 - ahci2: port 0xa050-0xa057,0xa040-0xa043,0xa030-0xa037,0xa020-0xa023,0xa000-0xa01f mem 0xfe610000-0xfe6107ff irq 40 at device 0.0 on pci7 - ahci3: port 0xf040-0xf047,0xf030-0xf033,0xf020-0xf027,0xf010-0xf013,0xf000-0xf00f mem 0xfea07000-0xfea073ff irq 19 at device 17.0 on pci0 ... each drive connected to a single port. I can actually reproduce this at will. Because I have 16 drives, when one fails, I need to find it. I pull the sata cable for a drive, determine if it's the drive in question, if not, reconnect, "ONLINE" it and wait for resilver to stop... usually only a minute or two. ... if I do this 4 to 6 odd times to find a drive (I can tell, in general, that a drive is part of the SAS controller or the SATA controllers... so I'm only looking among 8, ever) ... then I "REPLACE" the problem drive. More often than not, the a scrub will find a few problems. In fact, it appears that the most recent scrub is an example: [1:7:306]dgilbert@vr:~> zpool status pool: vr1 state: ONLINE scan: scrub repaired 32K in 47h16m with 0 errors on Mon Apr 1 23:12:03 2019 config: NAME STATE READ WRITE CKSUM vr1 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/v1-d0 ONLINE 0 0 0 gpt/v1-d1 ONLINE 0 0 0 gpt/v1-d2 ONLINE 0 0 0 gpt/v1-d3 ONLINE 0 0 0 gpt/v1-d4 ONLINE 0 0 0 gpt/v1-d5 ONLINE 0 0 0 gpt/v1-d6 ONLINE 0 0 0 gpt/v1-d7 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 gpt/v1-e0c ONLINE 0 0 0 gpt/v1-e1b ONLINE 0 0 0 gpt/v1-e2b ONLINE 0 0 0 gpt/v1-e3b ONLINE 0 0 0 gpt/v1-e4b ONLINE 0 0 0 gpt/v1-e5a ONLINE 0 0 0 gpt/v1-e6a ONLINE 0 0 0 gpt/v1-e7c ONLINE 0 0 0 logs gpt/vr1log ONLINE 0 0 0 cache gpt/vr1cache ONLINE 0 0 0 errors: No known data errors ... it doesn't say it now, but there were 5 CKSUM errors on one of the drives that I had trial-removed (and not on the one replaced).