From owner-freebsd-fs@freebsd.org Sun Dec 16 07:28:02 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59E02133FDFA for ; Sun, 16 Dec 2018 07:28:02 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: from mail-wm1-x342.google.com (mail-wm1-x342.google.com [IPv6:2a00:1450:4864:20::342]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5BE967449C for ; Sun, 16 Dec 2018 07:28:01 +0000 (UTC) (envelope-from kayasaman@gmail.com) Received: by mail-wm1-x342.google.com with SMTP id g67so9537338wmd.2 for ; Sat, 15 Dec 2018 23:28:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:subject:message-id:date:user-agent:mime-version :content-transfer-encoding:content-language; bh=zqwdm2drma+jOzB0s5zTDzO8vvW1lXG7LwplmwKZ+iQ=; b=Xyg879AmvIrQrImLpc/2qE4a5KSJtcwPR6/ug94XbUCcNgThMc4ISPDr1Vo0POfTzh 2P1Xo0ODUJiG44WHEDb46/9mabbI8LpYJ2qoKzzVSx7jsC4Tpvyn7FOELIQzel7oufXD KY9zE0iuimoiO8yvVCmLROBpBt2RI3L0vYeIfOhwCWKKoB5cAmPvA/43jIGsUhlccQBP qmP3im3oLa8nBxwZyRG9D4jPaIvSv64RGh8wtEnyVJG+jdNDQXKRPQQ0sUXP8BjnsDuf FufV8vgfmlu84k9yX4ax2iRhfmI9MI+F3N/AmYFb2Ax+rDZ+oTlmy1I7NOFtYVUBNHTA iF2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:message-id:date:user-agent :mime-version:content-transfer-encoding:content-language; bh=zqwdm2drma+jOzB0s5zTDzO8vvW1lXG7LwplmwKZ+iQ=; b=pnrON7W3sjP+OFWeDU4PQ74IHUSnjaBEY87rzUZveULujGOVByNx6QlZO29SAA7teZ OrryfBQJFcjfJZCyqQLMi2cjXT22BjZyV3bzwP9HAutGNSiIdp6D3cqNPprkdrBfchFi x5sk8VZnJVnVQ4MnHAkG++H+EP1N2okmZTtckMzwPB2bBrk8QYOiv9u7A+fhWmkubBEy kkBZMtrYSHDAYlmvi16yGbffNg2OTQgKKaYN9iTFDQFqFoFXlTd91Hfy2XN1qf5VSdoq FkPNIg+d+IV/22K7S7qMG3t+hhM57o3tfD446XSQf/yxKaszDjr0E4u54oArwdeRD49J qFyA== X-Gm-Message-State: AA+aEWbzHBEtaD0WG5yojfQ1m2m4vzfskCPafquzipWHabWQ+skIfNk0 3o//GwlryFp9p1VWh9VO7AK3fbNm X-Google-Smtp-Source: AFSGD/XUt4yabjVZH7ee0r97dyVY+dtYYNPZ5XZg4irMTdDVvfADf+9hPeJUD3x5jwhksb2nAQ0fiA== X-Received: by 2002:a1c:2e0c:: with SMTP id u12mr7980336wmu.81.1544945279585; Sat, 15 Dec 2018 23:27:59 -0800 (PST) Received: from Sting-Ray.optiplex-networks.com (optiplexnetworks.plus.com. [212.159.80.17]) by smtp.gmail.com with ESMTPSA id o82sm9540966wmo.29.2018.12.15.23.27.58 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Sat, 15 Dec 2018 23:27:58 -0800 (PST) To: freebsd-fs@freebsd.org From: Kaya Saman Subject: Upgrade to 12-Release system not booting kernel panic Message-ID: <4a9526c1-2c46-eebe-b92d-d10e522d8941@gmail.com> Date: Sun, 16 Dec 2018 07:27:58 +0000 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.3.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US-large X-Rspamd-Queue-Id: 5BE967449C X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Xyg879Am; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of kayasaman@gmail.com designates 2a00:1450:4864:20::342 as permitted sender) smtp.mailfrom=kayasaman@gmail.com X-Spamd-Result: default: False [-3.99 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.76)[-0.762,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.997,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.98)[-0.983,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-0.24)[ip: (1.73), ipnet: 2a00:1450::/32(-1.52), asn: 15169(-1.32), country: US(-0.08)]; RCVD_IN_DNSWL_NONE(0.00)[2.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 16 Dec 2018 07:28:02 -0000 Hi, I've been digging around a little on this one and it seems that there is some ZFS wierdness going on. My original install was an 11.2-RELEASE fresh install and upon an upgrade to 12-RELEASE the system is having issues booting. I have a ZFS root pool called zroot; by default this mounts at zroot/ROOT/default and a few other non-root / bootable ZFS pools. The system is a SuperMicro SC216 chassis with LSI non-RAID HBA. I have the boot disks ada0 and ada1 plugged into the rear of the chassis and directly into the systemboard which is also a SuperMicro. These drives are both Samsung SSD's. The 22 drive slots at the front of the chassis are occupied by the other various data pools. So here is some strangeness... if I remove all 22 drives from the front, the system boots fine but straight after boot goes into kernel panic mode and reboots before I can even look at the error or get to the login prompt. With the non-root pools installed at the BTX loader after scanning through all the bios drives I get a bunch of: read 264 from ... to 0x...., error 0x10 errors then: ZFS: i/o error - all block copies unavailable ZFS: can't read MOS of pool after this the system simply hangs? I have tried looking around but everything mentioning the MOS error is talking about the root pool, a particularly good reference is here: http://freebsd.1045724.x6.nabble.com/ZFS-i-o-error-in-recent-12-0-td6245865.html In fact I did try to boot with a USB stick and go into Live mode then import all the pools on the system. This works without any issue! The pools are fine the data is there everything looks normal. - I also rebuilt the zpool.cache according to the link just incase there was some kind of corruption there, however upon reboot I still get the same issue?? Looking at a bug report with a kernel panic: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=220923#c17 I have attempted to add: kern.cam.scsi_delay="50000" kern.cam.boot_delay="50000" into the /boot/loader.conf file but unfortunately the issue still continues :-( I wonder if there is a way to tell to tell the system to only look at certain drives for booting?? There is this line in my loader.conf: vfs.root.mountfrom="zfs:zroot" It maybe the wrong hunch I have but it seems like the system is looking for "zroot" on all pools instead of the actual root pool hence the above errors?? Would anyone be able to suggest anything or have any ideas about how to get the system back online and booting?? Thanks. Kaya