From owner-freebsd-fs@freebsd.org Thu Sep 22 20:15:32 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 46123BE5399 for ; Thu, 22 Sep 2016 20:15:32 +0000 (UTC) (envelope-from aaronkmiller@gmail.com) Received: from mail-io0-x22e.google.com (mail-io0-x22e.google.com [IPv6:2607:f8b0:4001:c06::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 11B5C8E9 for ; Thu, 22 Sep 2016 20:15:32 +0000 (UTC) (envelope-from aaronkmiller@gmail.com) Received: by mail-io0-x22e.google.com with SMTP id e66so8057377iod.1 for ; Thu, 22 Sep 2016 13:15:32 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:from:date:message-id:subject:to; bh=m8B4aVz8nQ8h1wgCA07gJbFdro3SfbMppHjhZ+LsuCM=; b=T8R4lk4WvQiH1a2o5udoRJESMi6CJ/Wv45yqnYMxTwYPKw1d1I5grkfAXOuC9PtUj4 9+PCX2Gtvwk7DTRMVz0ihlAWpbc+4qlJDWkXvh3La1y7OPSPXRA9tpHkedX+02e+fkDC klx3NBrrlyuAEg1CYaD56rxohWC57+TDXZH3M5g6uIXtQh0M8L/LQQX550eL6uVIJSqy s0+9Y1MlWZX8S2ORRtd/KRURW7flSSFFA5ZAlCZQ5Jp1Qbyg8kqaF22JoRNtB/Rl4asC Pao3EiRnfnimSjUaYOBl/q1eMvsg43S8+9v8oxhYYsMVr2aEnbJxWQ14QDreoXj4EbIf z/8g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:from:date:message-id:subject:to; bh=m8B4aVz8nQ8h1wgCA07gJbFdro3SfbMppHjhZ+LsuCM=; b=IOJwu1URaFBwSbmOdWcZCus+zEyOnyEX+dA8xlHv3pRcgQ2Lpn80gJU0uxbqBsNy9F r/ktlpSMPIo6x6ZOnvf3pl5coHJrz8vpOPyTAtUhOjcFmJGAz9t7rXn9V8TdNXG9nIbK +nCWI2cRlafyYyIah0/0exAg3H2XW7qMu5QJ/dOrPS5SHfmzUnUUSLxd/h8tbpHRccop CTSOgFnLK/90GjtkG4VFOb34PgHAay8ihM5oH24vRUmhOXslBdWwch0/z1cGs1o/I06f u0ngtJPr8iriUpkPJUFUZBtbmKFluwH1+r3p3/VFURmiFSrYeeSSNoBjozmEoeuof7s3 GjUQ== X-Gm-Message-State: AE9vXwOU1ri086ywmQ+0jUn/HoEn2a6e2fgwv+tt7mGSy7gAqAClGrFado2JIQq2gxP+f0d6eMAvh+77ItK2Ww== X-Received: by 10.107.17.216 with SMTP id 85mr5508604ior.96.1474575331254; Thu, 22 Sep 2016 13:15:31 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.71.209 with HTTP; Thu, 22 Sep 2016 13:14:50 -0700 (PDT) From: Aaron Miller Date: Thu, 22 Sep 2016 13:14:50 -0700 Message-ID: Subject: Logical unit not ready, manual intervention required To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 22 Sep 2016 20:15:32 -0000 Hello all, I hope I'm posting to the right list and someone can help me. After rebooting my FreeBSD 10.3 machine the iSCSI LUN it is hosting is no longer accessible on my SAN. The underlying zpool and zvol seem fine: root@freebsd:~ # zpool status pool: spool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM spool ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 da0 ONLINE 0 0 0 da1 ONLINE 0 0 0 mirror-1 ONLINE 0 0 0 da2 ONLINE 0 0 0 da3 ONLINE 0 0 0 mirror-2 ONLINE 0 0 0 da4 ONLINE 0 0 0 da5 ONLINE 0 0 0 mirror-3 ONLINE 0 0 0 da6 ONLINE 0 0 0 da7 ONLINE 0 0 0 mirror-4 ONLINE 0 0 0 da8 ONLINE 0 0 0 da9 ONLINE 0 0 0 mirror-5 ONLINE 0 0 0 da10 ONLINE 0 0 0 da11 ONLINE 0 0 0 root@freebsd:~ # zfs list NAME USED AVAIL REFER MOUNTPOINT spool 3.16T 0 19K /spool spool/volume1 3.16T 3.15T 5.18G - root@freebsd:~ # zfs get all spool/volume1 NAME PROPERTY VALUE SOURCE spool/volume1 type volume - spool/volume1 creation Tue Sep 6 8:23 2016 - spool/volume1 used 3.16T - spool/volume1 available 3.15T - spool/volume1 referenced 5.18G - spool/volume1 compressratio 1.00x - spool/volume1 reservation none default spool/volume1 volsize 3.06T local spool/volume1 volblocksize 8K - spool/volume1 checksum on default spool/volume1 compression off default spool/volume1 readonly off default spool/volume1 copies 1 default spool/volume1 refreservation 3.16T local spool/volume1 primarycache all default spool/volume1 secondarycache all default spool/volume1 usedbysnapshots 0 - spool/volume1 usedbydataset 5.18G - spool/volume1 usedbychildren 0 - spool/volume1 usedbyrefreservation 3.15T - spool/volume1 logbias latency default spool/volume1 dedup off default spool/volume1 mlslabel - spool/volume1 sync standard default spool/volume1 refcompressratio 1.00x - spool/volume1 written 5.18G - spool/volume1 logicalused 5.15G - spool/volume1 logicalreferenced 5.15G - spool/volume1 volmode default default spool/volume1 snapshot_limit none default spool/volume1 snapshot_count none default spool/volume1 redundant_metadata all default The LUN seems to start okay? root@freebsd:/var/log # ctladm start 0 (7:0:0/0): LUN started successfully I do have some errors in the log but I'm not sure if they are related: Sep 22 19:12:03 freebsd kernel: GEOM: da0: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da0: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da1: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da1: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da2: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da2: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da3: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da3: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da4: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da4: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da5: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da5: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da6: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da6: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da7: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da7: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da8: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da8: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da9: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da9: using the secondary instead -- recovery strongly advised. Sep 22 19:12:03 freebsd kernel: GEOM: da10: the primary GPT table is corrupt or invalid. Sep 22 19:12:03 freebsd kernel: GEOM: da10: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: da11: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: da11: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZY8NW1L%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZY8NW1L%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WV8UEL%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WV8UEL%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRVTEL%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRVTEL%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN71J%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN71J%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXYHAKJ%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXYHAKJ%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZB4PJ%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZB4PJ%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN7HJ%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN7HJ%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKHP0J%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKHP0J%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKJB9J%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKJB9J%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WT1S8L%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WT1S8L%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRD6PL%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRD6PL%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZYD0VWM%20%20%20%20%20%20%20%20: the primary GPT table is corrupt or invalid. Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZYD0VWM%20%20%20%20%20%20%20%20: using the secondary instead -- recovery strongly advised. There's a lot of this noise when an initiator is online and trying to connect: Sep 22 19:40:49 freebsd ctld[6983]: 10.33.80.62: read: connection lost Sep 22 19:40:49 freebsd ctld[659]: child process 6983 terminated with exit status 1 Sep 22 19:40:49 freebsd ctld[6984]: 10.33.80.62: read: connection lost Sep 22 19:40:49 freebsd ctld[659]: child process 6984 terminated with exit status 1 Sep 22 19:40:49 freebsd ctld[6985]: 10.33.80.62 (iqn.1993-08.org.debian:01:46952f23d3e): read: connection lost Sep 22 19:40:49 freebsd ctld[659]: child process 6985 terminated with exit status 1 Sep 22 19:40:57 freebsd ctld[6987]: 10.33.80.62 (iqn.1993-08.org.debian:01:46952f23d3e): read: connection lost On the initiator which is running debian-based proxmox there is a lot of this in the log: Sep 22 12:49:53 proxmox kernel: [ 136.579835] sd 1:0:0:0: [sdb] Add. Sense: Logical unit not ready, manual intervention required Sep 22 12:49:53 proxmox kernel: [ 136.580246] sd 1:0:0:0: [sdb] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE Sep 22 12:49:53 proxmox kernel: [ 136.580250] sd 1:0:0:0: [sdb] Sense Key : Not Ready [current] Which after some googling sounds like I need to run 'ctladm start 0 -o', right? Well for some reason that doesn't work: root@freebsd:~ # ctladm start 0 -o ctladm: illegal option -- o ctladm: illegal option -- o (7:0:0/0): LUN started successfully No idea why it's saying illegal option here? It's listed in the command help: root@freebsd:~ # ctladm Usage: Primary commands: ctladm start [dev_id][general options] [-i] [-o] But that option is missing from 'man ctladm'? Folks on #freebsd have suggested that maybe it was added after freebsd 10.3 but I've found man pages online from 9.x that have it. Any assistance greatly appreciated! Thanks!