From owner-freebsd-stable@freebsd.org Sun Jul 21 21:42:02 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C3A2FBFAD1 for ; Sun, 21 Jul 2019 21:42:02 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-yw1-xc33.google.com (mail-yw1-xc33.google.com [IPv6:2607:f8b0:4864:20::c33]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D54176DBEF for ; Sun, 21 Jul 2019 21:42:01 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by mail-yw1-xc33.google.com with SMTP id f187so15210849ywa.5 for ; Sun, 21 Jul 2019 14:42:01 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:references:from:openpgp:autocrypt:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=eUzuW6P1M/TilvdbZuQsINtqvmZOK04Kre9iGOg4bDI=; b=qIqiu0LKSlnlnjaALCIFNycTjjidzXNvMf+sF25bjX8dGmYjT8mU58INf9tgvfCFx7 YjLcuRf2aS5G39sEGOrDslcfNCc08n6IhS9J/IH8kBSXNSSP+UyakNizjRpaPiQEJYfC +msMdvQOav/FJsPdwwoeVW9nxjmNe06SvUC7rbI+JFKfGNaLRHMgLxfvPYfBAdN4gVsv y0Rvwj4ZDWeyTCObGFJpwfG7y1j8qIxzbgSkxy3/Tmcm2Kceed1rdGO2464ulotMynFz Ag0W2W1jLBGED0nNPtO8ZFfbYeH8ApAUI0DbOZy9GMMqaSpGtT4F1+skCEVVD0iIh+vh loYg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:references:from:openpgp :autocrypt:message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=eUzuW6P1M/TilvdbZuQsINtqvmZOK04Kre9iGOg4bDI=; b=U0KUzMj/QSd7ISY6OyiSagYR8qqPGhtacAkEWy9iBkiNKHsD+WpN5GoxcdZOa4Zkjh 5nLfsIl9N94gkcULEcsmUPQD8N4yvxnhPHCd0Xx3TJQoYVLhXcqW6b83bKZPPeMPO+Ra bsqK4hNQKGRma3aKLzZXJBnQyj3Qr0mjq+SG/t0mJeBgHgitHiKn+gETEZnrS9f49nfT 6oiHr78Uf+rB5xgU0YvURg8hLTbRkL8gKD5RhoE8o+7lp9/dVgmZaxjmL9DNZiO510Pc o536h4Q18epbcdRg71oQ9XRoB/TplPgz34lIlIZUIHwfyTC1z5FYqu4zLbFNe7J1CH3N FNpg== X-Gm-Message-State: APjAAAVnpBMJB6wfsmQT+RUhpa97JDL0M6uCa2tW5Oum3NULdN9lIn/6 KVRjo9teFAd391u7Po12+zNn6NBy X-Google-Smtp-Source: APXvYqxusB90MnJKY52hjuyWjWhpLCoIQjgBSfq9CYNbUJOL4wxw7+qJSaad/PgxdM3RwdrIzWGyYw== X-Received: by 2002:a0d:d795:: with SMTP id z143mr39357589ywd.305.1563745320670; Sun, 21 Jul 2019 14:42:00 -0700 (PDT) Received: from spectre.mavhome.dp.ua ([2600:1700:3580:3560:228:f8ff:fe04:d12]) by smtp.gmail.com with ESMTPSA id u123sm9279413ywu.75.2019.07.21.14.41.59 (version=TLS1_3 cipher=AEAD-AES128-GCM-SHA256 bits=128/128); Sun, 21 Jul 2019 14:41:59 -0700 (PDT) Sender: Alexander Motin Subject: Re: ZFS root mount regression To: Eugene Grosbein , Garrett Wollman , freebsd-stable@freebsd.org References: <23858.2573.932364.128957@khavrinen.csail.mit.edu> <73cddcd9-97f0-e73f-da9d-2a454fd3ea1a@grosbein.net> From: Alexander Motin Openpgp: preference=signencrypt Autocrypt: addr=mav@FreeBSD.org; prefer-encrypt=mutual; keydata= mQENBFOzxAwBCADkPrax0pI2W/ig0CK9nRJJwsHitAGEZ2HZiFEuti+6/4UVxj81yr4ak/4g 9bKUyC7rMEAp/ZHNhd+MFCPAAcHPvtovnfykqE/vuosCS3wlSLloix2iKVLks0CwbLHGAyne 46lTQW74Xl/33c3W1Z6d8jD9gVFT/xaVzZ0U9xdzOmsYAZaAj4ki0tuxO9F7L+ct9grRe7iP g8t9hai7BL4ee3VRwk2JXnKb7UvBiVITKYWKz1jRvZIrjPokgEcCLOSlv7x/1kjuFnj3xWZU 7HSFFT8J93epBbrSSCsYsppIk2fZH41kaaFXsMQfTPH8wkeM6qwrvOh4HiQM08R+9tThABEB AAG0IUFsZXhhbmRlciBNb3RpbiA8bWF2QEZyZWVCU0Qub3JnPokBVwQTAQoAQQIbAwULCQgH AwUVCgkICwUWAwIBAAIeAQIXgAIZARYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMKuBQkN McyiAAoJEIMYw5VbqyJ/tuUIAOG3ONOSNYqjK4eTZ1TVh9jdUBAhWk5nhDFnODN49Wj0AbYm 7aIqy8O1hnCDSZG5LttjSAo3UfXJZDKQM0BLb0gpRMBnAYqO6tdolLNqAbPGJBnGoPjsh24y 6KcbDaNnis+lD4GwPXwQM+92wZGhCUFElPV9NciZGVS65TNIgk7X+yEjjhD1MSWKKijZ1r9Z zIt4OzUTxxNOvzdlABZS88nNRdJkatOQJPmFdd1mpP6UzTNCiLUo1pIqOEtJgvVVDYq5WHY6 tciWWYdmZG/tIBexJmv2mV2OLVjXR6ZeKmntVH14H72/wRHJuYHQC+r5SVRcWWayrThsY6jZ Yr4+raS5AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6Z AXgDtmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8Flv mI/c40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt 3ytU8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZ R1EdEIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm5 9R8AEQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczM AAoJEIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLq A6xe6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHu uC5vgPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15Gc sS9YcQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9 TevwGsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCg lz65AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6ZAXgD tmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8FlvmI/c 40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt3ytU 8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZR1Ed EIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm59R8A EQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczMAAoJ EIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLqA6xe 6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHuuC5v gPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15GcsS9Y cQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9Tevw Gsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCglz4= Message-ID: <841d26dd-7433-2e6d-9011-76ed7ad3d5d2@FreeBSD.org> Date: Sun, 21 Jul 2019 17:41:59 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <73cddcd9-97f0-e73f-da9d-2a454fd3ea1a@grosbein.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D54176DBEF X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=qIqiu0LK; spf=pass (mx1.freebsd.org: domain of mavbsd@gmail.com designates 2607:f8b0:4864:20::c33 as permitted sender) smtp.mailfrom=mavbsd@gmail.com X-Spamd-Result: default: False [-6.05 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.92)[-0.918,0]; FORGED_SENDER(0.30)[mav@FreeBSD.org,mavbsd@gmail.com]; IP_SCORE(-2.92)[ip: (-9.01), ipnet: 2607:f8b0::/32(-3.11), asn: 15169(-2.42), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[mav@FreeBSD.org,mavbsd@gmail.com]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.3.c.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 21 Jul 2019 21:42:02 -0000 Hi, I am not sure how the original description leads to conclusion that problem is related to parallel mounting. From my point of view it sounds like a problem that root pool mounting happens based on name, not pool GUID that needs to be passed from the loader. We have seen problem like that ourselves too when boot pool names collide. So I doubt it is a new problem, just nobody got to fixing it yet. On 20.07.2019 06:41, Eugene Grosbein wrote: > CC'ing Alexander Motin who comitted the change. > > 20.07.2019 1:21, Garrett Wollman wrote: > >> I recently upgraded several file servers from 11.2 to 11.3. All of >> them boot from a ZFS pool called "tank" (the data is in a different >> pool). In a couple of instances (which caused me to have to take a >> late-evening 140-mile drive to the remote data center where they are >> located), the servers crashed at the root mount phase. In one case, >> it bailed out with error 5 (I believe that's [EIO]) to the usual >> mountroot prompt. In the second case, the kernel panicked instead. >> >> The root cause (no pun intended) on both servers was a disk which was >> supplied by the vendor with a label on it that claimed to be part of >> the "tank" pool, and for some reason the 11.3 kernel was trying to >> mount that (faulted) pool rather than the real one. The disks and >> pool configuration were unchanged from 11.2 (and probably 11.1 as >> well) so I am puzzled. >> >> Other than laboriously running "zpool labelclear -f /dev/somedisk" for >> every piece of media that comes into my hands, is there anything else >> I could have done to avoid this? > > Both 11.3-RELEASE announcement and Release Notes mention this: > >> The ZFS filesystem has been updated to implement parallel mounting. > > I strongly suggest reading Release documentation in case of troubles > after upgrade, at least. Or better, read *before* updating. > > I guess this parallelism created some race for your case. > > Unfortunately, a way to fall back to sequential mounting seems undocumented. > libzfs checks for ZFS_SERIAL_MOUNT environment variable to exist having any value. > I'm not sure how you set it for mounting root, maybe it will use kenv, > so try adding to /boot/loader.conf: > > ZFS_SERIAL_MOUNT=1 > > Alexander should have more knowledge on this. > > And of course, attaching unrelated device having label conflicting > with root pool is asking for trouble. Re-label it ASAP. > -- Alexander Motin