From owner-freebsd-questions@freebsd.org Wed Aug 28 23:25:59 2019 Return-Path: Delivered-To: freebsd-questions@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 74711E9242 for ; Wed, 28 Aug 2019 23:25:59 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from holgerdanske.com (holgerdanske.com [184.105.128.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "holgerdanske.com", Issuer "holgerdanske.com" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 46Jhgp2bVCz3Mby for ; Wed, 28 Aug 2019 23:25:58 +0000 (UTC) (envelope-from dpchrist@holgerdanske.com) Received: from 99.100.19.101 ([99.100.19.101]) by holgerdanske.com with ESMTPSA (ECDHE-RSA-AES128-GCM-SHA256:TLSv1.2:Kx=ECDH:Au=RSA:Enc=AESGCM(128):Mac=AEAD) (SMTP-AUTH username dpchrist@holgerdanske.com, mechanism PLAIN) for ; Wed, 28 Aug 2019 16:25:53 -0700 Subject: Re: Verry serious problem with ZFS & 12.0 To: freebsd-questions@freebsd.org References: <20190828224547.GA1557@io.chezmoi.fr> From: David Christensen Message-ID: Date: Wed, 28 Aug 2019 16:25:38 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <20190828224547.GA1557@io.chezmoi.fr> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 46Jhgp2bVCz3Mby X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of dpchrist@holgerdanske.com has no SPF policy when checking 184.105.128.27) smtp.mailfrom=dpchrist@holgerdanske.com X-Spamd-Result: default: False [-2.24 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-0.97)[-0.970,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-0.47)[ipnet: 184.104.0.0/15(0.81), asn: 6939(-3.09), country: US(-0.05)]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-questions@freebsd.org]; TO_DN_NONE(0.00)[]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.97)[-0.965,0]; DMARC_NA(0.00)[holgerdanske.com]; NEURAL_HAM_SHORT(-0.74)[-0.738,0]; RCVD_IN_DNSWL_NONE(0.00)[27.128.105.184.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:184.104.0.0/15, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 28 Aug 2019 23:25:59 -0000 On 8/28/19 3:45 PM, Albert Shih wrote: > Hi > > After update 4 servers from 11.2 to 12.0 without any problem, wait few > weeks to see if everything work well, and it did. I just upgrade my mail > server. > > During the upgrade I also upgrade all firmware for the hardware. > > And now I got a very serious issue with my server. > > Configuration : > > Dell PowerEdge R740Xd with H730P, 192 Go Ram, 2 SAS mechanical disk for the system, > 2 SSD (in a zfs pool) for the mail index (cyrus), and 28 mechanical disk > (in a second zfs pool) for the mailbox. > > The problem: > > After running few days the zfs pool with the 2 SSD are not responding. > > The system are perfectly working. > > The second zpool (mechanical disk) are perfectly working. > > I got zero log, zero message in the console or in dmesg. > > The arc_size are correct, it's around 70-75 %. > > The moment the zfs pool become not responding are random, not related to > any activity (human or cron). > > The only option I pass for the kernel related to ZFS are vfs.zfs.min_auto_ashift=12 and > vfs.zfs.prefetch_disable=1. Without the second one the system no > responding (under 11.2) when the server send (through zfs send) the data to another > server. > > After the first problem I make a zfs upgrade, thinking maybe that's the > problem so I'm not sure I can downgrade to 11.2 (and 11.2 are EOL) > > In your opinion : > > 1/ What should I do to try to find the problem ? > > 2/ Do you think that's a hardware/firmware problem or FreeBSD problem, > the point is the second zpool are working perfectly so I'm thinking at > some firmware/hardware/compatibility problem. Please run the following commands in a terminal and post the actual prompt, the actual command entered, and the actual output produced by the command (redact as required). Substitute "SSDPOOL", below, with the actual name of your SSD pool. Substitute MAILD with the actual name of your mail service: freebsd-version uname -a pkg version | grep MAILD zpool list zpool status SSDPOOL zfs list -d 1 SSDPOOL top -S -d3 top -mio -d3 Are mail services still operational without the mail index? Can you configure your mail service to not use a mail index? Can you put two fast drives into the mail server, create a mirror, and put the mail index there? David