From owner-freebsd-questions@freebsd.org Wed Aug 28 20:45:53 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C6BCDE60A5 for ; Wed, 28 Aug 2019 20:45:53 +0000 (UTC) (envelope-from Albert.Shih@obspm.fr) Received: from mx-p1.obspm.fr (mx-p1.obspm.fr [145.238.193.20]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "*.obspm.fr", Issuer "TERENA SSL CA 3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46Jd74639Rz3Cdx for ; Wed, 28 Aug 2019 20:45:52 +0000 (UTC) (envelope-from Albert.Shih@obspm.fr) Received: from io.chezmoi.fr (vpn.obspm.fr [145.238.186.39]) (authenticated bits=0) by mx-p1.obspm.fr (8.14.4/8.14.4/DIO Observatoire de Paris - 15/04/10) with ESMTP id x7SKjmlJ299376 (version=TLSv1/SSLv3 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 28 Aug 2019 22:45:50 +0200 Date: Thu, 29 Aug 2019 00:45:47 +0200 From: Albert Shih To: freebsd-questions@freebsd.org Subject: Verry serious problem with ZFS & 12.0 Message-ID: <20190828224547.GA1557@io.chezmoi.fr> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit User-Agent: Mutt/1.12.1 (2019-06-15) X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.5.11 (mx-p1.obspm.fr [145.238.193.20]); Wed, 28 Aug 2019 22:45:50 +0200 (CEST) X-Virus-Scanned: clamav-milter 0.100.3 at mx-p1.obspm.fr X-Virus-Status: Clean X-Rspamd-Queue-Id: 46Jd74639Rz3Cdx X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of Albert.Shih@obspm.fr designates 145.238.193.20 as permitted sender) smtp.mailfrom=Albert.Shih@obspm.fr X-Spamd-Result: default: False [-0.03 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.15)[0.151,0]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.66)[-0.658,0]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_SHORT(0.35)[0.349,0]; RCVD_IN_DNSWL_MED(-0.20)[20.193.238.145.list.dnswl.org : 127.0.11.2]; DMARC_NA(0.00)[obspm.fr]; IP_SCORE(0.63)[asn: 2200(3.14), country: FR(-0.00)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:2200, ipnet:145.238.0.0/16, country:FR]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Aug 2019 20:45:53 -0000 Hi After update 4 servers from 11.2 to 12.0 without any problem, wait few weeks to see if everything work well, and it did. I just upgrade my mail server. During the upgrade I also upgrade all firmware for the hardware. And now I got a very serious issue with my server. Configuration : Dell PowerEdge R740Xd with H730P, 192 Go Ram, 2 SAS mechanical disk for the system, 2 SSD (in a zfs pool) for the mail index (cyrus), and 28 mechanical disk (in a second zfs pool) for the mailbox. The problem: After running few days the zfs pool with the 2 SSD are not responding. The system are perfectly working. The second zpool (mechanical disk) are perfectly working. I got zero log, zero message in the console or in dmesg. The arc_size are correct, it's around 70-75 %. The moment the zfs pool become not responding are random, not related to any activity (human or cron). The only option I pass for the kernel related to ZFS are vfs.zfs.min_auto_ashift=12 and vfs.zfs.prefetch_disable=1. Without the second one the system no responding (under 11.2) when the server send (through zfs send) the data to another server. After the first problem I make a zfs upgrade, thinking maybe that's the problem so I'm not sure I can downgrade to 11.2 (and 11.2 are EOL) In your opinion : 1/ What should I do to try to find the problem ? 2/ Do you think that's a hardware/firmware problem or FreeBSD problem, the point is the second zpool are working perfectly so I'm thinking at some firmware/hardware/compatibility problem. Regards. -- Albert SHIH DIO bātiment 15 Observatoire de Paris Heure local/Local time: Thu 29 Aug 2019 12:26:55 AM CEST