From owner-freebsd-fs@freebsd.org Fri Sep 23 22:48:44 2016 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5A5DBBE8B0D for ; Fri, 23 Sep 2016 22:48:44 +0000 (UTC) (envelope-from aaronkmiller@gmail.com) Received: from mail-io0-x22a.google.com (mail-io0-x22a.google.com [IPv6:2607:f8b0:4001:c06::22a]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2433BBFE for ; Fri, 23 Sep 2016 22:48:44 +0000 (UTC) (envelope-from aaronkmiller@gmail.com) Received: by mail-io0-x22a.google.com with SMTP id r145so132682921ior.0 for ; Fri, 23 Sep 2016 15:48:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to; bh=mGAAKZsgV6GZ2Tqp4PnKgMmtm9nxXHcKwOKg/e/T/rw=; b=UW5raAQ4rZUeez1ppYWQxBH8eKNeGawkkTy3KKRltCGLW3HiQo5EV1TjpTApJEjYn8 FBsNJuZ/5++2vtgDYo09ge76Weq/4780SyG6uPKvMLlTmJd55H8jK6olObsywpHY7izS rtivZ/E2Yr/PL2Tq+FlHoEClB7z5N5fboV3C0H76XjoDxfFczqzx7K3k64votCtu+0q/ UhRSURTosBDGl6mURkQwYw/UIcJ6NBsWCwlBgR3Qg5hbSXENTmjeoyDdz3+U2fScJS/2 y2ZJapfazbZZ2gES5f6qaUM8Cd+VvdNuE510kK6N/I6KiUzqGUOg2h2JL18H6KdZTTZ/ Jttw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to; bh=mGAAKZsgV6GZ2Tqp4PnKgMmtm9nxXHcKwOKg/e/T/rw=; b=UjvQ5TK86UbXT5mv1WrFLi4UqarSjFzXsnikgawVx345GXWKVKVr2zTw9xyhXWgTRJ YLdTEpYoQZIHxg+vuNn+S5oAi9aivdeEbncmlEns9kX8t/Oaz9ev+dzqKehfhD3cFw1K qeCuzY6tJ5vvXI4a+b/RLug4LY2ktBlM3P/JaNPQd7BdQIZTmJGjClWB9LvV/evACKAZ k67NKZQfxmoDBogP9C6Wa81HYXRaIQJYxYD2iUJnGEpyQnJL2n082FBxNIDr6FBYadel LZCkd8DvLeQ3KaoHCBEKr4ryGfWEog9YXJHCRh8b79NJ5h+dJXXqlifttQcDtCgrpOXc 6/gA== X-Gm-Message-State: AE9vXwO0Hia1RRmtLjmhrdHJOqveQzLIQuqIF/5jsCMjl7/ma7LxmBy2SgAgwf7K5D0wYWSw7aTzb1mrNJL1hQ== X-Received: by 10.107.17.216 with SMTP id 85mr13265425ior.96.1474670923309; Fri, 23 Sep 2016 15:48:43 -0700 (PDT) MIME-Version: 1.0 Received: by 10.36.71.209 with HTTP; Fri, 23 Sep 2016 15:48:02 -0700 (PDT) In-Reply-To: References: From: Aaron Miller Date: Fri, 23 Sep 2016 15:48:02 -0700 Message-ID: Subject: Re: Logical unit not ready, manual intervention required To: freebsd-fs@freebsd.org Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.23 X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 23 Sep 2016 22:48:44 -0000 I figured it out. I did not have a size defined for my LUN in /etc/ctl.conf. Added that, restarted ctld and now it's working. On Thu, Sep 22, 2016 at 1:14 PM, Aaron Miller wrote: > Hello all, I hope I'm posting to the right list and someone can help me. > After rebooting my FreeBSD 10.3 machine the iSCSI LUN it is hosting is no > longer accessible on my SAN. > > The underlying zpool and zvol seem fine: > > root@freebsd:~ # zpool status > pool: spool > state: ONLINE > scan: none requested > config: > > NAME STATE READ WRITE CKSUM > spool ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > da0 ONLINE 0 0 0 > da1 ONLINE 0 0 0 > mirror-1 ONLINE 0 0 0 > da2 ONLINE 0 0 0 > da3 ONLINE 0 0 0 > mirror-2 ONLINE 0 0 0 > da4 ONLINE 0 0 0 > da5 ONLINE 0 0 0 > mirror-3 ONLINE 0 0 0 > da6 ONLINE 0 0 0 > da7 ONLINE 0 0 0 > mirror-4 ONLINE 0 0 0 > da8 ONLINE 0 0 0 > da9 ONLINE 0 0 0 > mirror-5 ONLINE 0 0 0 > da10 ONLINE 0 0 0 > da11 ONLINE 0 0 0 > > root@freebsd:~ # zfs list > NAME USED AVAIL REFER MOUNTPOINT > spool 3.16T 0 19K /spool > spool/volume1 3.16T 3.15T 5.18G - > > root@freebsd:~ # zfs get all spool/volume1 > NAME PROPERTY VALUE SOURCE > spool/volume1 type volume - > spool/volume1 creation Tue Sep 6 8:23 2016 - > spool/volume1 used 3.16T - > spool/volume1 available 3.15T - > spool/volume1 referenced 5.18G - > spool/volume1 compressratio 1.00x - > spool/volume1 reservation none default > spool/volume1 volsize 3.06T local > spool/volume1 volblocksize 8K - > spool/volume1 checksum on default > spool/volume1 compression off default > spool/volume1 readonly off default > spool/volume1 copies 1 default > spool/volume1 refreservation 3.16T local > spool/volume1 primarycache all default > spool/volume1 secondarycache all default > spool/volume1 usedbysnapshots 0 - > spool/volume1 usedbydataset 5.18G - > spool/volume1 usedbychildren 0 - > spool/volume1 usedbyrefreservation 3.15T - > spool/volume1 logbias latency default > spool/volume1 dedup off default > spool/volume1 mlslabel - > spool/volume1 sync standard default > spool/volume1 refcompressratio 1.00x - > spool/volume1 written 5.18G - > spool/volume1 logicalused 5.15G - > spool/volume1 logicalreferenced 5.15G - > spool/volume1 volmode default default > spool/volume1 snapshot_limit none default > spool/volume1 snapshot_count none default > spool/volume1 redundant_metadata all default > > > The LUN seems to start okay? > > root@freebsd:/var/log # ctladm start 0 > (7:0:0/0): LUN started successfully > > > I do have some errors in the log but I'm not sure if they are related: > > Sep 22 19:12:03 freebsd kernel: GEOM: da0: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da0: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da1: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da1: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da2: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da2: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da3: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da3: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da4: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da4: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da5: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da5: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da6: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da6: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da7: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da7: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da8: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da8: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da9: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da9: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:03 freebsd kernel: GEOM: da10: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:03 freebsd kernel: GEOM: da10: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: da11: the primary GPT table is > corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: da11: using the secondary instead -- > recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZY8NW1L%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZY8NW1L%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WV8UEL%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WV8UEL%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRVTEL%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRVTEL%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN71J%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN71J%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXYHAKJ%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXYHAKJ%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZB4PJ%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZB4PJ%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN7HJ%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZXZN7HJ%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKHP0J%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKHP0J%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKJB9J%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZWKJB9J%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WT1S8L%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WT1S8L%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRD6PL%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-J9WRD6PL%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZYD0VWM%20%20%20%20%20%20%20%20: > the primary GPT table is corrupt or invalid. > Sep 22 19:12:04 freebsd kernel: GEOM: diskid/DISK-JZYD0VWM%20%20%20%20%20%20%20%20: > using the secondary instead -- recovery strongly advised. > > > There's a lot of this noise when an initiator is online and trying to > connect: > > Sep 22 19:40:49 freebsd ctld[6983]: 10.33.80.62: read: connection lost > Sep 22 19:40:49 freebsd ctld[659]: child process 6983 terminated with exit > status 1 > Sep 22 19:40:49 freebsd ctld[6984]: 10.33.80.62: read: connection lost > Sep 22 19:40:49 freebsd ctld[659]: child process 6984 terminated with exit > status 1 > Sep 22 19:40:49 freebsd ctld[6985]: 10.33.80.62 (iqn.1993-08.org.debian:01:46952f23d3e): > read: connection lost > Sep 22 19:40:49 freebsd ctld[659]: child process 6985 terminated with exit > status 1 > Sep 22 19:40:57 freebsd ctld[6987]: 10.33.80.62 (iqn.1993-08.org.debian:01:46952f23d3e): > read: connection lost > > > On the initiator which is running debian-based proxmox there is a lot of > this in the log: > > Sep 22 12:49:53 proxmox kernel: [ 136.579835] sd 1:0:0:0: [sdb] Add. > Sense: Logical unit not ready, manual intervention required > Sep 22 12:49:53 proxmox kernel: [ 136.580246] sd 1:0:0:0: [sdb] Read > Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE > Sep 22 12:49:53 proxmox kernel: [ 136.580250] sd 1:0:0:0: [sdb] Sense Key > : Not Ready [current] > > > Which after some googling sounds like I need to run 'ctladm start 0 -o', > right? Well for some reason that doesn't work: > > root@freebsd:~ # ctladm start 0 -o > ctladm: illegal option -- o > ctladm: illegal option -- o > (7:0:0/0): LUN started successfully > > > No idea why it's saying illegal option here? It's listed in the command > help: > > root@freebsd:~ # ctladm > Usage: > Primary commands: > > ctladm start [dev_id][general options] [-i] [-o] > > > But that option is missing from 'man ctladm'? Folks on #freebsd have > suggested that maybe it was added after freebsd 10.3 but I've found man > pages online from 9.x that have it. > > > Any assistance greatly appreciated! Thanks! >