From owner-freebsd-questions@freebsd.org Wed May 30 02:30:07 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id D9471EF4C3D for ; Wed, 30 May 2018 02:30:06 +0000 (UTC) (envelope-from drizzt321@gmail.com) Received: from mail-ua0-x22c.google.com (mail-ua0-x22c.google.com [IPv6:2607:f8b0:400c:c08::22c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 614B969F79 for ; Wed, 30 May 2018 02:30:06 +0000 (UTC) (envelope-from drizzt321@gmail.com) Received: by mail-ua0-x22c.google.com with SMTP id i3-v6so11442057uad.4 for ; Tue, 29 May 2018 19:30:06 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:from:date:message-id:subject:to; bh=K5+Jfcj4bgjVX+OFzgmcxj5qoAFlgEVi+2JAl1dJFyo=; b=PNmMhogMxwUn/Xx1ZRBrQgBUU46hvRqCwvNIcnE4feD2clsqH7juqbmDGgwxq87IJ+ DRkjpcTWLJuV0yy4OYYGLNbs5YnP0fdI9Oj5z8BBtQzdByhF1AwwcKwmhaJ5GVG0p2Xt nvoxYpfMySvmlgdO2KVXUh41kI1ThH8VClrVeDL70kKUJOoJ4ct4hUqSjMCNkEHQeNtk NHOsQ6RX6DcrLIOtbBleBLM9q+vTCuV6jy7YCH8m6wg5pMWRei7t74qsgpYA2QlVW/rN vyBeIpwhmhagSdSW5xNbpOGJHyitPdSAfD3UI95+zskAAhxYpjFWElrcByL411LVrMcB SDFQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=K5+Jfcj4bgjVX+OFzgmcxj5qoAFlgEVi+2JAl1dJFyo=; b=IjTr5uISM4uXei+PIyspJigc2SQJqPNYmW0Gka4vznGZEGpUmTDXaa59zFnpSxhKrk PiTmMT2L4YXZ/NUIOc2UoJrcrRNzswDTtmxHKiZns5ropFk29L9YsqQXXyoAxKrLWpey TE5nwKqJCeSforiv8dvetid2Zgs/hBCqHYAfpqlBUfrsbHcR5cttsCxourH+ZnPkB/Ym hYZHf4EIsgCuIReWBCTPJynAUhlj0lqki1EmYFxFLnYFy6HkSaJHINlulV2Y4zC2b3Gv mzQXV6j84zoquoYI8EXkn6ADu3XmWMeXzOgMks3fX2PIWVLI+VdRhLC6U1feRMXJTARQ 67kg== X-Gm-Message-State: ALKqPwdgwvzNFwYDnHahkiTBo/QuMM9kN6gAkLbLQpN/rNPctSKQrENe XsJTvAplh2aTdW0x/jBXSUuImd+2v9x8th4bhAje0NJI X-Google-Smtp-Source: ADUXVKKcuf40JJNUPC05gqK+p09gr0lwEccdePnR5/a5gJ0jGRFj4cUA627N2TuFPF5KtCWXpUMWFpKc2OmXm32Y+GE= X-Received: by 2002:a9f:2635:: with SMTP id 50-v6mr533332uag.41.1527647404811; Tue, 29 May 2018 19:30:04 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a67:6206:0:0:0:0:0 with HTTP; Tue, 29 May 2018 19:29:34 -0700 (PDT) From: Aaron Date: Tue, 29 May 2018 19:29:34 -0700 Message-ID: Subject: ZFS mirror keeps going offline/removed To: FreeBSD Questions Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 30 May 2018 02:30:07 -0000 So recently my ZFS mirror keeps getting both drives marked as removed which makes my mirror go offline. I don't see anything obvious in my /var/log/messages (other than as below), although the errors always seem to be in metadata. The latest error seems to be in a non-metadata location, although not a file but a zfs directory. Whenever I run zpool clear -F -n bulk it ends up back online and after a scrub no errors, but it keeps happening. Based on the below paste, what could I look at that would tell me which drive, or both, or what the issue is? I'm running an old dual Xeon x5650, ECC memory (ran memtest86+ on it before installing, no errors). --Aaron /var/log/messages May 29 19:20:33 darkserver kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 May 29 19:20:33 darkserver kernel: ada2: s/n ZDH1CTDZ detached May 29 19:20:33 darkserver kernel: (ada2:ahcich2:0:0:0): Periph destroyed May 29 19:20:33 darkserver ZFS: vdev state changed, pool_guid=9608097630476313818 vdev_guid=4999141564141954755 May 29 19:20:33 darkserver ZFS: vdev is removed, pool_guid=9608097630476313818 vdev_guid=4999141564141954755 May 29 19:20:38 darkserver kernel: ada8 at ahcich9 bus 0 scbus9 target 0 lun 0 May 29 19:20:38 darkserver kernel: ada8: s/n ZDH1CS9M detached May 29 19:20:38 darkserver kernel: (ada8:ahcich9:0:0:0): Periph destroyed May 29 19:21:19 darkserver kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 May 29 19:21:19 darkserver kernel: ada2: ACS-3 ATA SATA 3.x device May 29 19:21:19 darkserver kernel: ada2: Serial Number ZDH1CTDZ May 29 19:21:19 darkserver kernel: ada2: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes) May 29 19:21:19 darkserver kernel: ada2: Command Queueing enabled May 29 19:21:19 darkserver kernel: ada2: 3815447MB (7814037168 512 byte sectors) May 29 19:21:19 darkserver kernel: ada2: quirks=0x1<4K> May 29 19:21:21 darkserver kernel: ada8 at ahcich9 bus 0 scbus9 target 0 lun 0 May 29 19:21:21 darkserver kernel: ada8: ACS-3 ATA SATA 3.x device May 29 19:21:21 darkserver kernel: ada8: Serial Number ZDH1CS9M May 29 19:21:21 darkserver kernel: ada8: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) May 29 19:21:21 darkserver kernel: ada8: Command Queueing enabled May 29 19:21:21 darkserver kernel: ada8: 3815447MB (7814037168 512 byte sectors) May 29 19:21:21 darkserver kernel: ada8: quirks=0x1<4K> # zpool status -x -v pool: bulk state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://illumos.org/msg/ZFS-8000-HC scan: scrub repaired 0 in 7h22m with 0 errors on Tue May 29 04:27:28 2018 config: NAME STATE READ WRITE CKSUM bulk UNAVAIL 0 0 0 mirror-0 UNAVAIL 1 0 0 4999141564141954755 REMOVED 0 0 0 was /dev/ada2 16676652866205005686 REMOVED 0 0 0 was /dev/ada8 errors: Permanent errors have been detected in the following files: :<0x0> :<0x1> :<0x1b> :<0x33> bulk/video:<0xc0ad>