From owner-freebsd-fs@freebsd.org Sun Apr 5 12:40:53 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4E4BB2B5F5B for ; Sun, 5 Apr 2020 12:40:53 +0000 (UTC) (envelope-from mail@florian-schulze.net) Received: from wout1-smtp.messagingengine.com (wout1-smtp.messagingengine.com [64.147.123.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48wCvC5BjGz4vSK for ; Sun, 5 Apr 2020 12:40:38 +0000 (UTC) (envelope-from mail@florian-schulze.net) Received: from compute7.internal (compute7.nyi.internal [10.202.2.47]) by mailout.west.internal (Postfix) with ESMTP id EBDCB895; Sun, 5 Apr 2020 08:40:27 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute7.internal (MEProxy); Sun, 05 Apr 2020 08:40:28 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= florian-schulze.net; h=from:to:cc:subject:date:message-id :in-reply-to:references:mime-version:content-type :content-transfer-encoding; s=fm3; bh=uCj9+5soh4pNUBNMJESq8TOWvM oFG/Nwg/TAQy3fv6c=; b=RtxLFtD6waWFCvn0lr2kkdvsadBewCxBrmFAja3+x+ 9umE7s0a5WpUytlehfEPhoavJWc2n/ykP+ARPi3D6WQan7Ik3b47qwmZBw3eh5AK EEBY/pjA2VALgqop4n8rI3a8daAIYatjushzNdXaCmQdMviJRupXXE2YdK1YTPtV GV4TvyDGM7q6Qcx7HhNFwP5AuKnW//z0Y/yXt19JtKW8K4cKhW0wa67sqL+50kyq qBFBb791IyL1AA1Ixyv+eH/bC7Nbjs2xPyfemsTr/u1H/lbBv5LdavepkBkSksg8 ua/BeAZNeuzfhYXQQ/IiOEpxtNLkCCtoKTkYe3Two+6g== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm2; bh=uCj9+5soh4pNUBNMJESq8TOWvMoFG/Nwg/TAQy3fv 6c=; b=fPPsHp+W1SPbnFiqlSVLOQBGZf2JRj2uBW3d/MT4t2VFih4hHS4a+/pMn 6wTfa89EqUB/wt/fHUl/v0ZgOEPPXA9nmkp06nJjrZTAPY5a13kxvyteEeqCgv21 NZcZko0E/TtL3u3A1cbrg48wyZgnzCalS+zGCApuRc5sEf9ybtipAqEoNtaFLw6K eZ0ieJ2GrsfdcjFKbrOyvqcQMfn+I1qlwd/+GNamyrQYYpkWCzQkI4Ev5/GloAx0 t4qHFQPpNwzArRMY/EzHHJUyPQZFy9hP2nkTLFvckRnR4gD4SNn2dHgRkICpwbzA RUaWildlKlvCZ5xUXsPidat/kTxQA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduhedruddugdehhecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenuc fjughrpefhvffufffokfgjfhggtgfgsehtkehmtdertdejnecuhfhrohhmpedfhfhlohhr ihgrnhcuufgthhhulhiivgdfuceomhgrihhlsehflhhorhhirghnqdhstghhuhhliigvrd hnvghtqeenucffohhmrghinhepfhhrvggvsghsugdrohhrghenucfkphepkeegrddukeeg rddujedvrdejjeenucevlhhushhtvghrufhiiigvpedtnecurfgrrhgrmhepmhgrihhlfh hrohhmpehmrghilhesfhhlohhrihgrnhdqshgthhhulhiivgdrnhgvth X-ME-Proxy: Received: from [192.168.1.182] (p54b8ac4d.dip0.t-ipconnect.de [84.184.172.77]) by mail.messagingengine.com (Postfix) with ESMTPA id 542003280059; Sun, 5 Apr 2020 08:40:26 -0400 (EDT) From: "Florian Schulze" To: "Artem Kuchin" Cc: freebsd-fs@freebsd.org Subject: Re: gpart bootcode Operation not permitted Date: Sun, 05 Apr 2020 14:40:23 +0200 X-Mailer: MailMate (1.13.1r5671) Message-ID: In-Reply-To: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> MIME-Version: 1.0 Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Transfer-Encoding: 8bit Autocrypt: addr=mail@florian-schulze.net; keydata=mQENBFliSUoBCADLKPuy1qhTI+A3XRleyZ7WlXhRfYiXBCSMiIqUb+nx88xKLVAclRJqMnqlJHzAJcb4JW9i3riRU+V1rVG9BDaBCCih5Si0BQO5W84SpaadTdcE3rUO8Ihdxc6dJN9DiAtp26WyO1fXSA2NkwpakIerxlF91knkrnSa/VgLrlB+Ahtcisbsn9XYSnv0wr/njQJOfJMsmJfWAKXWWaWDWb0Z4/UITsBe259Ppkf6bTYS3j7oCPqygPzdyc14ILINb0N3ksT3tKLiS4b2gh7RAO4uf/C8TvlZSYvoX/sel5xFFxcgmqG+V0d4jjhdvY5zFHU+/Egr5y3HPgSQjevEHsNxABEBAAG0KkZsb3JpYW4gU2NodWx6ZSA8bWFpbEBmbG9yaWFuLXNjaHVsemUubmV0PokBOQQTAQgAIwUCWWJJSgIbAwcLCQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJEC0N2jSLcB7qWo4H/jEgS6Id1TkRtlh+vZanpswS83sH8pOfc20qHbPEM2a1pk9aEgvvnckaeLiGVYaHKZccxRaE2E6zGKW26sI5WPcMehCZkiYf4zxfoppWsWDIGAsuptByHT+etkQVEANwNdDM5hIn4Hq0e6gsH1zu68SBhH4FtW4+ukCK3WrHZ0QJBoECU1otWxhP5BqOO8h9OCY2vESxHC2nH+y3vsqVL43JQWmBUv7uaaX4OusdWUZCW8jlYnWd//cmt3RKHh4PCKY7JK0itzoUi+DnnHBN7jxz2FFjX3+BinUeZX3bpXQhw3Xsv1ZXW9aeyGLgzYOuLdBSXmXJcetyJ0heCrOwZUS5AQ0EWWJJSgEIANlJdoChOEMMB7vh3YB0HjrBpK2+iyWC1Y1VCWOpKg/PiYq5GmZDMh1SGq+s/3ZVG5kAYcSKqdHSBqC8sRjY/gU/hysR3w9IU0J5/j1jTFZsw09vTfloEVRAJaAEyOAKvgUw+ fwiGhJsJPS9W8dqieVovqft5DpzNIBztPJxfiDrlHvKbxrwyCEW9+EAHMVJf7F6wqp0rAnlxZQujasL7Urx1Kl0RKT9HT8QHyzTJFpbpuO0hApiWCWD5GCy3AIc0KK3jJIz3ODCVFSGhbLiV0j9FB4j3wmM5JSvsyMhLZAkaFy/kqmlaVeroLWMBbmKUfEA968b37O+kILc3iXYQmMAEQEAAYkBHwQYAQgACQUCWWJJSgIbDAAKCRAtDdo0i3Ae6oyuB/4rh42bgTYncPkMv0l9jsKTB61wwe2Imxfxd52Gk3Ev5slaSFYl2r2FwhTFCX5Ay5XXqoueIG2AlZY2do7jnuLW66dye4GAI67PgKkZfzWJqD+q3bOimnmdlyKM1x2AxGnwPBEdjc4sShFm/uPOPPGJu4f8I9PKjGI62RJUCGHN9KxYJclomhNYe6n1xDsF0e3lDEtWn+KL1WesHO/NY2Mv//qGs9dqIu7rdAXrXiBQaSUz8ol0o4Z924qEWX+23tlWn3XLZPbFPJAB8sPHKWapt3nR/zaPJqfqkqwKWnWFdiKaKIB9GhlXmBlq+0alpjNjbicGlOF1YGF4agbFyoCb X-Rspamd-Queue-Id: 48wCvC5BjGz4vSK X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=florian-schulze.net header.s=fm3 header.b=RtxLFtD6; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=fPPsHp+W; dmarc=none; spf=pass (mx1.freebsd.org: domain of mail@florian-schulze.net designates 64.147.123.24 as permitted sender) smtp.mailfrom=mail@florian-schulze.net X-Spamd-Result: default: False [-6.09 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[florian-schulze.net:s=fm3,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:64.147.123.24]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[florian-schulze.net]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[florian-schulze.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; RBL_BLOCKLISTDE_FAIL(0.00)[24.123.147.64.bl.blocklist.de:query timed out]; IP_SCORE(-3.49)[ip: (-9.81), ipnet: 64.147.123.0/24(-4.92), asn: 11403(-2.69), country: US(-0.05)]; ZRD_FAIL(0.00)[query timed out]; RCVD_IN_DNSWL_LOW(-0.10)[24.123.147.64.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11403, ipnet:64.147.123.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[77.172.184.84.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 05 Apr 2020 12:40:53 -0000 Hi! Not sure if this applies to you, but for me the issue was that the boot partition was too small. I removed the swap, removed the swap partition, resized the boot partition, recreated swap partition, re-added swap and then wrote the boot code. Regards, Florian Schulze On 3 Apr 2020, at 15:23, Artem Kuchin wrote: > I am lost with this problem. > > I have gmirror: > > # gmirror status >        Name    Status  Components > mirror/boot  COMPLETE  ada0p1 (ACTIVE) >                        ada1p1 (ACTIVE) > mirror/swap  COMPLETE  ada0p2 (ACTIVE) >                        ada1p2 (ACTIVE) > mirror/root  COMPLETE  ada0p3 (ACTIVE) >                        ada1p3 (ACTIVE) > > > # gpart show > =>       34  976773101  ada0  GPT  (466G) >          34        128     1  freebsd-boot  (64K) >         162    8388608     2  freebsd-swap  (4.0G) >     8388770  968384365     3  freebsd-ufs  (462G) > > =>       34  976773101  ada1  GPT  (466G) >          34        128     1  freebsd-boot  (64K) >         162    8388608     2  freebsd-swap  (4.0G) >     8388770  968384365     3  freebsd-ufs  (462G) > > > And i want to update bootcode on both disks after freebsd updating > from source. > > > i Do > > sysctl -w kern.geom.debugflags=16 > > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 > gpart: /dev/ada0p1: Operation not permitted > > > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 > gpart: /dev/ada1p1: Operation not permitted > > > #gpart list > > Geom name: ada0 > modified: false > state: OK > fwheads: 16 > fwsectors: 63 > last: 976773134 > first: 34 > entries: 128 > scheme: GPT > Providers: > 1. Name: ada0p1 >    Mediasize: 65536 (64K) >    Sectorsize: 512 >    Stripesize: 0 >    Stripeoffset: 17408 >    Mode: r1w1e1 >    efimedia: > HD(1,GPT,346dc1b6-ca13-11e2-a56f-001cc0cc7f7d,0x22,0x80) >    rawuuid: 346dc1b6-ca13-11e2-a56f-001cc0cc7f7d >    rawtype: 83bd6b9d-7f41-11dc-be0b-001560b84f0f >    label: boot0 >    length: 65536 >    offset: 17408 >    type: freebsd-boot >    index: 1 >    end: 161 >    start: 34 > 2. Name: ada0p2 >    Mediasize: 4294967296 (4.0G) >    Sectorsize: 512 >    Stripesize: 0 >    Stripeoffset: 82944 >    Mode: r1w1e1 >    efimedia: > HD(2,GPT,3ebf609f-ca13-11e2-a56f-001cc0cc7f7d,0xa2,0x800000) >    rawuuid: 3ebf609f-ca13-11e2-a56f-001cc0cc7f7d >    rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b >    label: swap0 >    length: 4294967296 >    offset: 82944 >    type: freebsd-swap >    index: 2 >    end: 8388769 >    start: 162 > 3. Name: ada0p3 >    Mediasize: 495812794880 (462G) >    Sectorsize: 512 >    Stripesize: 0 >    Stripeoffset: 82944 >    Mode: r1w1e1 >    efimedia: > HD(3,GPT,4797d671-ca13-11e2-a56f-001cc0cc7f7d,0x8000a2,0x39b85f6d) >    rawuuid: 4797d671-ca13-11e2-a56f-001cc0cc7f7d >    rawtype: 516e7cb6-6ecf-11d6-8ff8-00022d09712b >    label: root0 >    length: 495812794880 >    offset: 4295050240 >    type: freebsd-ufs >    index: 3 >    end: 976773134 >    start: 8388770 > Consumers: > 1. Name: ada0 >    Mediasize: 500107862016 (466G) >    Sectorsize: 512 >    Mode: r3w3e6 > > Same for ada1 > > What's wrong? > > > -- > > Artem > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" From owner-freebsd-fs@freebsd.org Mon Apr 6 08:03:22 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 82A4F2A7BB0 for ; Mon, 6 Apr 2020 08:03:22 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from forward101o.mail.yandex.net (forward101o.mail.yandex.net [37.140.190.181]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48wjhn2rXjz3CCl for ; Mon, 6 Apr 2020 08:03:21 +0000 (UTC) (envelope-from bu7cher@yandex.ru) Received: from mxback10g.mail.yandex.net (mxback10g.mail.yandex.net [IPv6:2a02:6b8:0:1472:2741:0:8b7:171]) by forward101o.mail.yandex.net (Yandex) with ESMTP id 315913C013DE; Mon, 6 Apr 2020 11:03:19 +0300 (MSK) Received: from sas1-e00c2743cdb8.qloud-c.yandex.net (sas1-e00c2743cdb8.qloud-c.yandex.net [2a02:6b8:c14:3a22:0:640:e00c:2743]) by mxback10g.mail.yandex.net (mxback/Yandex) with ESMTP id jxH0Z57vj2-3JCSOw2s; Mon, 06 Apr 2020 11:03:19 +0300 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yandex.ru; s=mail; t=1586160199; bh=HgU3u0pUZ9APztS8sf9mJBdNS+NAOLf94cmnxNCoZ2o=; h=In-Reply-To:From:Date:References:To:Subject:Message-ID; b=S/r8SVxC1YDf3BfuduP+r+ITiOGrDi+CcW/erN64B2R3CuX3Tri8cn019CjrZFiHV /xBGvfqZW9fG/gZ8kb6mzCwP2vGvgzR69estM535IMs0q6BqJqVp2Tj2/JPGKltBoh VU0ZiwV5yzfj34UXj/1U+2X3rpNaBdWXuAYOjp4Y= Received: by sas1-e00c2743cdb8.qloud-c.yandex.net (smtp/Yandex) with ESMTPSA id AEvwiexgfK-3I2e8IqT; Mon, 06 Apr 2020 11:03:18 +0300 (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)) (Client certificate not present) Subject: Re: gpart bootcode Operation not permitted To: Artem Kuchin , freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> From: "Andrey V. Elsukov" Openpgp: id=E6591E1B41DA1516F0C9BC0001C5EA0410C8A17A Autocrypt: addr=bu7cher@yandex.ru; prefer-encrypt=mutual; keydata= mQENBEwBF1kBCADB9sXFhBEUy8qQ4X63Y8eBatYMHGEFWN9ypS5lI3RE6qQW2EYbxNk7qUC5 21YIIS1mMFVBEfvR7J9uc7yaYgFCEb6Sce1RSO4ULN2mRKGHP3/Sl0ijZEjWHV91hY1YTHEF ZW/0GYinDf56sYpDDehaBF5wkWIo1+QK5nmj3vl0DIDCMNd7QEiWpyLVwECgLX2eOAXByT8B bCqVhJGcG6iFP7/B9Ll6uX5gb8thM9LM+ibwErDBVDGiOgvfxqidab7fdkh893IBCXa82H9N CNwnEtcgzh+BSKK5BgvPohFMgRwjti37TSxwLu63QejRGbZWSz3OK3jMOoF63tCgn7FvABEB AAG0JUFuZHJleSBWLiBFbHN1a292IDxidTdjaGVyQHlhbmRleC5ydT6JATgEEwECACIFAkwB F1kCGwMGCwkIBwMCBhUIAgkKCwQWAgMBAh4BAheAAAoJEAHF6gQQyKF6qmYIAI6ekfm1VA4T vqankI1ISE6ku4jV7UlpIQlEbE7/8n3Zd6teJ+pGOQhN5qk8QE7utdPdbktAzi+x7LIJVzUw 4TywZLXGrkP7VKYkfg6oyCGyzITghefQeJtr2TN4hYCkzPWpylkue8MtmqfZv/6royqwTbN+ +E09FQNvTgRUYJYTeQ1qOsxNRycwvw3dr2rOfuxShbzaHBB1pBIjGrMg8fC5pd65ACH5zuFV A0CoTNGMDrEZSfBkTW604UUHFFXeCoC3dwDZRKOWJ3GmMXns65Ai5YkA63BSHEE1Qle3VBhd cG1w0CB5FBV3pB27UVnf0jEbysrDqW4qN7XMRFSWNAy5AQ0ETAEXWQEIAJ2p6l9LBoqdH/0J PEFDY2t2gTvAuzz+8zs3R03dFuHcNbOwjvWCG0aOmVpAzkRa8egn5JB4sZaFUtKPYJEQ1Iu+ LUBwgvtXf4vWpzC67zs2dDuiW4LamH5p6xkTD61aHR7mCB3bg2TUjrDWn2Jt44cvoYxj3dz4 S49U1rc9ZPgD5axCNv45j72tggWlZvpefThP7xT1OlNTUqye2gAwQravXpZkl5JG4eOqJVIU X316iE3qso0iXRUtO7OseBf0PiVmk+wCahdreHOeOxK5jMhYkPKVn7z1sZiB7W2H2TojbmcK HZC22sz7Z/H36Lhg1+/RCnGzdEcjGc8oFHXHCxUAEQEAAYkBHwQYAQIACQUCTAEXWQIbDAAK CRABxeoEEMihegkYCAC3ivGYNe2taNm/4Nx5GPdzuaAJGKWksV+w9mo7dQvU+NmI2az5w8vw 98OmX7G0OV9snxMW+6cyNqBrVFTu33VVNzz9pnqNCHxGvj5dL5ltP160JV2zw2bUwJBYsgYQ WfyJJIM7l3gv5ZS3DGqaGIm9gOK1ANxfrR5PgPzvI9VxDhlr2juEVMZYAqPLEJe+SSxbwLoz BcFCNdDAyXcaAzXsx/E02YWm1hIWNRxanAe7Vlg7OL+gvLpdtrYCMg28PNqKNyrQ87LQ49O9 50IIZDOtNFeR0FGucjcLPdS9PiEqCoH7/waJxWp6ydJ+g4OYRBYNM0EmMgy1N85JJrV1mi5i Message-ID: <9b871532-23a1-4029-ecb4-ead8ba060dd8@yandex.ru> Date: Mon, 6 Apr 2020 10:59:32 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="qO51It5OFaIxhVMv0s3xMr4OPjzytTHGg" X-Rspamd-Queue-Id: 48wjhn2rXjz3CCl X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=yandex.ru header.s=mail header.b=S/r8SVxC; dmarc=pass (policy=none) header.from=yandex.ru; spf=pass (mx1.freebsd.org: domain of bu7cher@yandex.ru designates 37.140.190.181 as permitted sender) smtp.mailfrom=bu7cher@yandex.ru X-Spamd-Result: default: False [-5.10 / 15.00]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[yandex.ru]; R_SPF_ALLOW(-0.20)[+ip4:37.140.128.0/18]; HAS_ATTACHMENT(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[yandex.ru:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[yandex.ru,none]; SIGNED_PGP(-2.00)[]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(0.00)[ip: (-9.72), ipnet: 37.140.128.0/18(-4.88), asn: 13238(-3.85), country: RU(0.01)]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~]; FREEMAIL_ENVFROM(0.00)[yandex.ru]; ASN(0.00)[asn:13238, ipnet:37.140.128.0/18, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[yandex.ru.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[yandex.ru:s=mail]; RCVD_TLS_LAST(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/mixed,text/plain]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[181.190.140.37.list.dnswl.org : 127.0.5.0] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Apr 2020 08:03:22 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --qO51It5OFaIxhVMv0s3xMr4OPjzytTHGg Content-Type: multipart/mixed; boundary="VhS3rVpm4m4qZLfolyE5AVER8L7JSmCPB"; protected-headers="v1" From: "Andrey V. Elsukov" To: Artem Kuchin , freebsd-fs@freebsd.org Message-ID: <9b871532-23a1-4029-ecb4-ead8ba060dd8@yandex.ru> Subject: Re: gpart bootcode Operation not permitted References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> In-Reply-To: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> --VhS3rVpm4m4qZLfolyE5AVER8L7JSmCPB Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 03.04.2020 16:23, Artem Kuchin wrote: > sysctl -w kern.geom.debugflags=3D16 This is not needed. It will be better if you forget this option. > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 > gpart: /dev/ada0p1: Operation not permitted >=20 >=20 > # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 > gpart: /dev/ada1p1: Operation not permitted >=20 > What's wrong? In your case you only can write pmbr using gpart(8): # gpart bootcode -b /boot/pmbr ada0 # gpart bootcode -b /boot/pmbr ada1 Then you can write gptboot to the mirror using dd(1): # dd if=3D/boot/gptboot of=3D/dev/mirror/boot You can't write gptboot to the partition using gpart(8) because it is exclusively used by geom_mirror. --=20 WBR, Andrey V. Elsukov --VhS3rVpm4m4qZLfolyE5AVER8L7JSmCPB-- --qO51It5OFaIxhVMv0s3xMr4OPjzytTHGg Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- Comment: Using GnuPG with Thunderbird - https://www.enigmail.net/ iQEzBAEBCAAdFiEE5lkeG0HaFRbwybwAAcXqBBDIoXoFAl6K4WQACgkQAcXqBBDI oXpGNwf/cCAaMVtYZuJOXmYAx0S7/3j1dOVuSkjq1yz7sxZHwbqDeG4EdPzpy8sE AuksIz4i8EfOkCq136DqOv9AKLIoFS9zWFsIMXFNyM37H5i7zjCxEe/D+aDStzDr jwBXR5hmqnhu8oleGKWFWyXUkkthj8hdjeSJoSeW6WwE3U52gBr//H1UR/fymnW1 AZTrvdW1m9l3nVHlFVj7P/BdWZLymay7u7rZ3AfvhCVDPe/lcv7v9k+tkfMw67tQ M8Tphaq1evlTsMfni9rGf3sBXzv8Z1bBTnJZUH7j33Cfmw8onKP9ab7WyOxJdobV 0rnul7n8/KJLz4c69dnxY7PT6Mxj6w== =fhR6 -----END PGP SIGNATURE----- --qO51It5OFaIxhVMv0s3xMr4OPjzytTHGg-- From owner-freebsd-fs@freebsd.org Mon Apr 6 11:56:20 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 4D43A2AD2F3 for ; Mon, 6 Apr 2020 11:56:20 +0000 (UTC) (envelope-from artem@artem.ru) Received: from smtp5.mail.ru (smtp5.mail.ru [94.100.179.24]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 48wpsZ3mSYz3R66 for ; Mon, 6 Apr 2020 11:56:18 +0000 (UTC) (envelope-from artem@artem.ru) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=mail.ru; s=mail2; h=Content-Transfer-Encoding:Content-Type:In-Reply-To:MIME-Version:Date:Message-ID:From:References:To:Subject; bh=VOrFxbClHN1meTFg2L/4T08VHdMJlbj7RW4x6GZUcy8=; b=oUpWFw0hO197bK1iQEhbHBWp7J0A02Gp/irNY5bmU+bPO0MJbaLZz+H7qb7yqrK2CsrDqk4IarRMgY7GDhEo4kAIO7t3PeEk+J3e1CS8s92lc5PCIPSZZpy4k0jdV3e/8dNeNr6agemvQskxLaY95bxqQP2oL0X3+RxiGL2FaV8=; Received: by smtp5.mail.ru with esmtpa (envelope-from ) id 1jLQMR-00072B-6I; Mon, 06 Apr 2020 14:56:15 +0300 Subject: Re: gpart bootcode Operation not permitted To: Miroslav Lachman <000.fbsd@quip.cz>, freebsd-fs@freebsd.org References: <27955efa-01f2-88f6-6a28-d9d8a62dfa2a@artem.ru> <48ba23fa-13dc-4a74-579c-2028479f302a@quip.cz> <1605a04a-ad2c-5b2c-445f-fb8ccd7211d6@artem.ru> <159cf2f5-1898-019e-9f02-29750ff7fa7e@quip.cz> From: Artem Kuchin Message-ID: <07de3b0c-cf60-f061-f2dd-33597ff576fd@artem.ru> Date: Mon, 6 Apr 2020 14:56:03 +0300 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.6.0 MIME-Version: 1.0 In-Reply-To: <159cf2f5-1898-019e-9f02-29750ff7fa7e@quip.cz> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: ru X-7564579A: 646B95376F6C166E X-77F55803: 0A44E481635329DB0E1AA8A03B392317D32E5E48865217365060145B739F5F5CB3409941E66BE95BF688BCB05C26794D00EEFFA3BE1FE418D4222F2BC55466BB2D5C35E11E7FD390B659AE49F45CE3A7 X-7FA49CB5: 0D63561A33F958A56A8221842B0B777F915DEF13E7C17871AB3F61A9443B25348941B15DA834481FA18204E546F3947C5B63D382EEF4D896F6B57BC7E64490618DEB871D839B7333395957E7521B51C2545D4CF71C94A83E9FA2833FD35BB23D27C277FBC8AE2E8B972D20C7389F068BA471835C12D1D977C4224003CC8364767815B9869FA544D8D32BA5DBAC0009BE9E8FC8737B5C22493BD8BFB0F436B6B276E601842F6C81A12EF20D2F80756B5F012D6517FE479FCD76E601842F6C81A127C277FBC8AE2E8BD5E8704C5CAD72113AA81AA40904B5D99449624AB7ADAF37CCFFBAE954C2DE44725E5C173C3A84C34B08FA16E56A400835872C767BF85DA2F004C906525384306FED454B719173D6462275124DF8B9C9DE2850DD75B2526BE5BFE6E7EFDEDCD789D4C264860C145E X-D57D3AED: Y8kq8+OzVozcFQziTi/Zi1xwo7H2ZNxGP5qz8aO2mjTJzjHGC4ogvVuzB3zfVUBtENeZ6b5av1fnCBE34JUDkWdM6QxE+Ga5d8voMtmXfSoScvtxS3Y8PJuf5RTkiCWD X-Mailru-Sender: 0E9E14D9EC491FBA79C5613A73A5E7B2A57F58173E705661E3B46E400B9082C839E2C803C56CC8A98A4382C47DA47812C77752E0C033A69E376A1339FE8876DF1FC4F5A70058821069EB1F849E6DBC830DA7A0AF5A3A8387 X-Mras: Ok X-Rspamd-Queue-Id: 48wpsZ3mSYz3R66 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=mail.ru header.s=mail2 header.b=oUpWFw0h; dmarc=none; spf=none (mx1.freebsd.org: domain of artem@artem.ru has no SPF policy when checking 94.100.179.24) smtp.mailfrom=artem@artem.ru X-Spamd-Result: default: False [-2.33 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; R_DKIM_ALLOW(-0.20)[mail.ru:s=mail2]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-0.997,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[artem.ru]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[mail.ru:+]; RCPT_COUNT_TWO(0.00)[2]; R_SPF_NA(0.00)[]; RCVD_COUNT_ZERO(0.00)[0]; RCVD_IN_DNSWL_LOW(-0.10)[24.179.100.94.list.dnswl.org : 127.0.5.1]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:47764, ipnet:94.100.176.0/20, country:RU]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(0.06)[ipnet: 94.100.176.0/20(0.06), asn: 47764(0.24), country: RU(0.01)] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 06 Apr 2020 11:56:20 -0000 03.04.2020 22:44, Miroslav Lachman пишет: > Artem Kuchin wrote on 2020/04/03 21:23: >> 03.04.2020 17:42, Miroslav Lachman пишет: >>> >>> I think you can use "gmirror stop" on boot partition, then "gmirror >>> clear" and then update both individual boot partitions by gpart. >>> >>> >> >> I am afraid to do it. Gmirror stops mirroring, okay. What what >> gmirror clear does? What metadata is cleared and what it is used for? >> >> How to restart mirroring after that and how to make sure that mirror >> is 100% complete? >> >> Is it possible to exclude only boot partition from mirroring? > > Of course that's what I am suggesting - just split up mirroring of > boot partition and keep mirroring on swap and root! > > Gmirror clear deletes the very last sector on the given partition > where metadata of gmirror are stored. It does not touch the data on > the partitions. You don't need to worry. > > You have 3 mirrors. They are visible in /dev/mirror or by command > "gmirror status": > boot > swap > root > > As the "man gmirror" shows: >      gmirror stop [-fv] name ... > > You can run: gmirror stop -v boot > > It stops only the "boot" mirror. swap and root are still mirrored. > > Then you can clear metadata on each provider: > > gmirror clear ada0p1 > gmirror clear ada1p1 > > Or you can use "gmirror destroy -v boot" istead of 3 command above. It > should stop the "boot" mirror and then clear metadata on both providers. > > The you will have / your system will use 2 independent boot > partitions: ada0p1 and ada1p1. > The machine should be able to boot from any of those 2 disks because > each of them has valid boot partitions / boot code. > > Gpart bootcode should work for them on each system upgrade. > > gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 > gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 > > Kind regards > Miroslav Lachman Hmm. tried what you wrote but it went a little different, failed to clear  and needed to use destroy anyway # gmirror stop -v boot Done. # gmirror status        Name    Status  Components mirror/swap  COMPLETE  ada0p2 (ACTIVE)                        ada1p2 (ACTIVE) mirror/root  COMPLETE  ada0p3 (ACTIVE)                        ada1p3 (ACTIVE) mirror/boot  COMPLETE  gpt/boot1 (ACTIVE)                        gpt/boot0 (ACTIVE) # gmirror clear ada0p1 Can't clear metadata on ada0p1: Operation not permitted. gmirror: Not fully done. # gmirror clear ada1p1 Can't clear metadata on ada1p1: Operation not permitted. gmirror: Not fully done. # gmirror destroy -v boot Done. # gmirror status        Name    Status  Components mirror/swap  COMPLETE  ada0p2 (ACTIVE)                        ada1p2 (ACTIVE) mirror/root  COMPLETE  ada0p3 (ACTIVE)                        ada1p3 (ACTIVE) # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada0 partcode written to ada0p1 bootcode written to ada0 # gpart bootcode -b /boot/pmbr -p /boot/gptboot -i 1 ada1 partcode written to ada1p1 bootcode written to ada1 Now it can boot from any disk. Thank you. Artem From owner-freebsd-fs@freebsd.org Wed Apr 8 01:20:07 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id E21502A5732 for ; Wed, 8 Apr 2020 01:20:07 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) Received: from CAN01-TO1-obe.outbound.protection.outlook.com (mail-eopbgr670088.outbound.protection.outlook.com [40.107.67.88]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "mail.protection.outlook.com", Issuer "GlobalSign Organization Validation CA - SHA256 - G3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 48xmfZ5J07z4VHh for ; Wed, 8 Apr 2020 01:20:06 +0000 (UTC) (envelope-from rmacklem@uoguelph.ca) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=eRRXv4u/ZYw967jYa6re5yXAg53Gp0LbJskuSAT98xRTtDsOZkB1Xaz2G0Lcos0ALpKbVrvz9slaPFflO8VhUnbAKr33ZSzZRyTHW1YDtG/hpknpJfZTt2RlRrfDbVzGq/2D7axNjJOoPuomcUVB6J6js7pWr9z4WRW57+XSC7vMvjCkmV2RnSXe6b6DOb684CvwIiDhdndBpAwOmXGkYyZ451emI19vOquCLz+SVYEhxVUtrjckk6ZZNrXyvKI7nTf6ufVEpVDPPidYPEfKG2UOrgJtHdPPKo4oY4xFUDY37EnzIMxv0EcmliGjDjpdt8DwrUlQchS95roEGwhQbQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=pt9BzlTpHl7Os/WPn1hZEIGQ4IpEadRIChieoxDgabs=; b=Xdg0/Yw27TgWENOqV3v1SG8Ahk/9SuThYGtfwICECm4eJgM9Am7UgTRwk2st84TwFA7/MjfRJdeXX1BkI6rg3X1uUJ7iHGmLQFfJfLiSaWqmkP9MjVQ6lr7PdQ5sua3zRvQBm6yMhnlG3Me+bcH/NaCDm0L2tzr0aU3uakKEaJC84zWmmpicShY/HIzAjSj/HviaVprUpV3KutneVnLcb55Iz3KwKI0Ptx6gWHMn/aQ48JtEpXKO49YsTPZR6oaPoW++55Me5jQTGMiY/+h312PEawZvkD9xJ3VOkTmqS44+cecBVJ6MNotjxGWYSte+Xk/QPT+keQ/hxuB8oHhKOw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=uoguelph.ca; dmarc=pass action=none header.from=uoguelph.ca; dkim=pass header.d=uoguelph.ca; arc=none Received: from YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM (10.255.13.83) by YTBPR01MB3231.CANPRD01.PROD.OUTLOOK.COM (10.255.12.203) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2878.15; Wed, 8 Apr 2020 01:20:05 +0000 Received: from YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM ([fe80::bc28:7213:fda5:e06c]) by YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM ([fe80::bc28:7213:fda5:e06c%7]) with mapi id 15.20.2878.022; Wed, 8 Apr 2020 01:20:05 +0000 From: Rick Macklem To: Luoqi Chen CC: Peter Eriksson , freebsd-fs Subject: Re: Linux could write to read only files on FreeBSD NFS server Thread-Topic: Linux could write to read only files on FreeBSD NFS server Thread-Index: AQHV7ZU6To48aBpgW0mPQBk2wdIrMagvh4MAgAATIQCAAALPjYAACluAgADNX9KAAFvlAIADoSCIgACDYoCAAPdIjIADg18AgDU6dHE= Date: Wed, 8 Apr 2020 01:20:04 +0000 Message-ID: References: <707243CD-C67E-4DAD-AC5A-68EC11CFFDFD@lysator.liu.se> <6EC06026-DA28-4CAC-8D56-5C7856D4625E@lysator.liu.se> <202002281113.01SBDlsl017697@higson.cam.lispworks.com> <8247CFFC-C324-40BB-B0DD-B469A3B35851@lysator.liu.se> , In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: b7e209d1-7251-444d-70b6-08d7db5af80a x-ms-traffictypediagnostic: YTBPR01MB3231: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:3513; x-forefront-prvs: 0367A50BB1 x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:YTBPR01MB3664.CANPRD01.PROD.OUTLOOK.COM; PTR:; CAT:NONE; SFTY:; SFS:(10009020)(39860400002)(346002)(396003)(376002)(366004)(136003)(6916009)(786003)(7696005)(81166006)(8676002)(2906002)(8936002)(33656002)(81156014)(9686003)(55016002)(478600001)(64756008)(66476007)(86362001)(6506007)(54906003)(66556008)(316002)(5660300002)(4326008)(52536014)(76116006)(71200400001)(66946007)(66446008)(186003); DIR:OUT; SFP:1101; received-spf: None (protection.outlook.com: uoguelph.ca does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: 3E/kLUlAmZq89Cyn1sG3hXsJyvp4LeD0jx48j/sqkN1VXZxok5uyBPDZfW7blXgb8XIFxW7AuQvbM8FxKAUs9erCATJA/dP4TZ0sr2/EiUWWgWaBm3dMDqJeRTn3IZhtbD2EF9XiTSrAUOpU0kbPb9r7yMA4W1hu6opd1dIIq8p4EKIjZvyNoBG7MjNNw6O3iq4+IHxu+qDQnVfMtEVs1atLXbI+cDA9NYP6uR1or9CCydFHQmksjVknkil1DvcDlh8EBwETI+oduSRZMJC9I+giO3gE5vR9EKNfoK1DLIYrK0jr0P478J8iY8nnb2X5dcAWJnwj6Eqvdbw9yhSDKm2kJmoZpDVqO2GcV2ak3tAWwBXN+vZiNcGC01tSeSlkSMLrA26zTp930gInJ94u2DYvwfCnFxboVJWWOt5SrQAv9Dmf5ZWcpcGCAG2qA+rJ x-ms-exchange-antispam-messagedata: ShgBlPH1rquqCIYHAXboPiXYfRxXujsiaIanIE3uQRkS/z/N7gkEGUfGJF4ru5z7QjuM6nswc/MuWkExY2qY90N6gHdEy9cK/KwZNW4DCKDbomzMIkkFaskBFvTbbxuiWOGdipEtDjJ63aOlEfjd1r38j5YMZYZ3c8kFIWew7nCyAelDax8xeIwIL5QXGsiqWJdi1TjvKMTaffXlpGWZMQ== x-ms-exchange-transport-forked: True Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: uoguelph.ca X-MS-Exchange-CrossTenant-Network-Message-Id: b7e209d1-7251-444d-70b6-08d7db5af80a X-MS-Exchange-CrossTenant-originalarrivaltime: 08 Apr 2020 01:20:05.0418 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: be62a12b-2cad-49a1-a5fa-85f4f3156a7d X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: 4ccZp1TJSt5q8dypM6kIzUm58gj1KeXQ0W1jqlhwSH5twoHjzwj5QjPdoz9MMvH5G+m7P56CIKibsaWICi5OYA== X-MS-Exchange-Transport-CrossTenantHeadersStamped: YTBPR01MB3231 X-Rspamd-Queue-Id: 48xmfZ5J07z4VHh X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of rmacklem@uoguelph.ca designates 40.107.67.88 as permitted sender) smtp.mailfrom=rmacklem@uoguelph.ca X-Spamd-Result: default: False [-4.69 / 15.00]; RCVD_TLS_LAST(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:40.107.0.0/16]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[uoguelph.ca]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[88.67.107.40.list.dnswl.org : 127.0.3.0]; IP_SCORE(-1.39)[ipnet: 40.64.0.0/10(-3.76), asn: 8075(-3.15), country: US(-0.05)]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:8075, ipnet:40.64.0.0/10, country:US]; ARC_ALLOW(-1.00)[i=1] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 08 Apr 2020 01:20:07 -0000 Luoqi Chen wrote:=0A= >On Mon, Mar 2, 2020 at 2:48 PM Rick Macklem > wrote:=0A= >>Anyhow, if others test the patch and like it, I can commit it controlled = via a sysctl.=0A= >>I'd just have to decide whether it should be enabled by default or not.= =0A= >>=0A= >>rick=0A= >=0A= >It has been more than a decade since I last developed for freebsd, I don't= have=0A= >anything running -current, but I did try the fix under 12-RELEASE, and can= confirm=0A= >it works. But my test is too ad hoc, it may need more thorough regression.= =0A= I have just committed a patch to head as r359720 that I believe handles thi= s.=0A= The difference between this patch and the one I posted some weeks ago is th= at=0A= it only performs the "does file owner have permission" check for the case= =0A= where the NFSv4 Open is not creating the file.=0A= This appears to be what the Linux server does and allows both the Linux and= =0A= FreeBSD clients to exhibit correct behaviour.=0A= =0A= Since having it enabled does not seem to affect the FreeBSD client's behavi= our,=0A= I have it enabled by default, with a sysctl called vfs.nfsd.v4openaccess th= at can=0A= be set to 0 to revert the NFSv4 server to the old behaviour.=0A= =0A= This should be MFC'd to stable/12 and stable/11 in 2 weeks.=0A= =0A= Thanks for reporting this, rick=0A= ps: I finally was able to do interop. testing with Linux to see what was go= ing on.=0A= =0A= -luoqi=0A= From owner-freebsd-fs@freebsd.org Sat Apr 11 17:36:41 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 0CE0E2BE41F for ; Sat, 11 Apr 2020 17:36:41 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) Received: from xtrwsxqr.outbound-mail.sendgrid.net (xtrwsxqr.outbound-mail.sendgrid.net [167.89.106.69]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 49029z4gyKz3Ph7 for ; Sat, 11 Apr 2020 17:36:39 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=from:content-type:content-transfer-encoding:mime-version:subject:to; s=s1; bh=50yhssOYidXTlUBY1vxW4pLi02F5jOulGkJoUUhZfvw=; b=kEn7g27av/sCjioqzW7wWZf0puHyf+t4z2Tvn7bjvdFlKUV9pe6rVp3t5844xOnRtWcq vQTZH/LJNxHbO6XdD5KB8M60JkrtiMsUGvyxvmrU6Qz7n7BdosFsBh/7QL7ikNsKDDZxaU qdJpE376QcSqlpQp/oSbZ02kST7sDZprs= Received: by filter0072p3las1.sendgrid.net with SMTP id filter0072p3las1-25929-5E920025-7E 2020-04-11 17:36:37.467380761 +0000 UTC m=+849917.549179951 Received: from mail.distal.com (unknown) by ismtpd0088p1mdw1.sendgrid.net (SG) with ESMTP id f2ecHczMQSClfGzLEGgvdQ Sat, 11 Apr 2020 17:36:37.357 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [2001:470:e24c:200:14a4:6888:ae02:b3bb]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 4d32cc50 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 13:36:35 -0400 (EDT) From: Chris Ross Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: ZFS server has gone crazy slow Message-Id: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> Date: Sat, 11 Apr 2020 17:36:37 +0000 (UTC) X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXZBmcg47PFtWNkjuk?= =?us-ascii?Q?OHD3k9B4AgosUkZt9jcDqCvebQRVsgzS=2FGHvMXh?= =?us-ascii?Q?KhjfCD5FlkPfv79=2FlaPX5rxUjImWBhLjCAG8azS?= =?us-ascii?Q?I6nPaQEBGxHl9vsoyKxChT6nafpJeWUHojxXJJb?= =?us-ascii?Q?uG0XKYghJ2U7Lyz8Cz16G+jdmpiEIVnETriqQd?= To: freebsd-fs , freebsd-stable@freebsd.org X-Rspamd-Queue-Id: 49029z4gyKz3Ph7 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=kEn7g27a; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.106.69 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-3.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.67)[ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[69.106.89.167.list.dnswl.org : 127.0.15.0]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[69.106.89.167.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[13739864-7f7e-freebsd-fs=freebsd.org,freebsd] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 17:36:41 -0000 I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirror (= of two GPT disks) as it=E2=80=99s disk. It=E2=80=99s many years old, and h= as only been misbehaving like this for a day or so. I=E2=80=99m trying to = figure out what=E2=80=99s wrong. I confirmed that internet connectivity isn=E2=80=99t the problem, and a reb= oot didn=E2=80=99t fix it. (The reboot took 10-15 minutes to finish going = multi-user, starting daemons, due to the underlying problem described below= .) Truss=E2=80=99ing a very basic command (date), I can see that close() and e= xit() calls are taking 1-2 seconds. All of the files being opened are on Z= FS, but I don=E2=80=99t know if that=E2=80=99s for sure related. Similarly= , using shell builtin =E2=80=9Cecho foo=E2=80=9D always is immediate, but = =E2=80=9C/bin/echo=E2=80=9D sometimes works quickly, but sometimes the clos= e() on /var/run/ld-elf.so.hints takes 3-5 seconds. I _think_ this is a filesystem problem. It=E2=80=99s very hard to diagnose= because logging in, and doing anything, takes many seconds per command. z= pool status shows my mirror as online, so I=E2=80=99m not sure where I shou= ld check. I=E2=80=99d appreciate any help! Thanks much=E2=80=A6 - Chris From owner-freebsd-fs@freebsd.org Sat Apr 11 17:48:41 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1BA2A2BE9EF for ; Sat, 11 Apr 2020 17:48:41 +0000 (UTC) (envelope-from fuz@fuz.su) Received: from fuz.su (fuz.su [IPv6:2001:41d0:8:e508::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "amnesiac", Issuer "amnesiac" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4902Rq4gfvz3Qc8 for ; Sat, 11 Apr 2020 17:48:39 +0000 (UTC) (envelope-from fuz@fuz.su) Received: from fuz.su (localhost [127.0.0.1]) by fuz.su (8.15.2/8.15.2) with ESMTPS id 03BHmVb8054531 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO) for ; Sat, 11 Apr 2020 19:48:31 +0200 (CEST) (envelope-from fuz@fuz.su) Received: (from fuz@localhost) by fuz.su (8.15.2/8.15.2/Submit) id 03BHmVsh054530 for freebsd-fs@freebsd.org; Sat, 11 Apr 2020 19:48:31 +0200 (CEST) (envelope-from fuz) Date: Sat, 11 Apr 2020 19:48:31 +0200 From: Robert Clausecker To: freebsd-fs Subject: Re: ZFS server has gone crazy slow Message-ID: <20200411174831.GA54397@fuz.su> References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> X-Rspamd-Queue-Id: 4902Rq4gfvz3Qc8 X-Spamd-Bar: ++++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of fuz@fuz.su has no SPF policy when checking 2001:41d0:8:e508::1) smtp.mailfrom=fuz@fuz.su X-Spamd-Result: default: False [4.27 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; DMARC_NA(0.00)[fuz.su]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_MEDIUM(0.98)[0.984,0]; IP_SCORE(1.39)[ipnet: 2001:41d0::/32(4.92), asn: 16276(2.03), country: FR(0.00)]; TO_DN_ALL(0.00)[]; NEURAL_SPAM_LONG(1.00)[0.999,0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16276, ipnet:2001:41d0::/32, country:FR]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 17:48:41 -0000 Hi Chris, What does zpool status say about the pool? Also check if the SMART values of these disks are okay. Yours, Robert Clausecker On Sat, Apr 11, 2020 at 05:36:37PM +0000, Chris Ross wrote: > I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS > mirror (of two GPT disks) as it’s disk. It’s many years old, and has > only been misbehaving like this for a day or so. I’m trying to figure > out what’s wrong. > > I confirmed that internet connectivity isn’t the problem, and a reboot > didn’t fix it. (The reboot took 10-15 minutes to finish going > multi-user, starting daemons, due to the underlying problem described > below.) > > Truss’ing a very basic command (date), I can see that close() and > exit() calls are taking 1-2 seconds. All of the files being opened > are on ZFS, but I don’t know if that’s for sure related. Similarly, > using shell builtin “echo foo” always is immediate, but “/bin/echo” > sometimes works quickly, but sometimes the close() on > /var/run/ld-elf.so.hints takes 3-5 seconds. > > I _think_ this is a filesystem problem. It’s very hard to diagnose > because logging in, and doing anything, takes many seconds per > command. zpool status shows my mirror as online, so I’m not sure > where I should check. > > I’d appreciate any help! Thanks much… > > - Chris > > > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org" > -- () ascii ribbon campaign - for an 8-bit clean world /\ - against html email - against proprietary attachments From owner-freebsd-fs@freebsd.org Sat Apr 11 18:02:50 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 659EE2BEF37 for ; Sat, 11 Apr 2020 18:02:50 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) Received: from xtrwsqdf.outbound-mail.sendgrid.net (xtrwsqdf.outbound-mail.sendgrid.net [167.89.100.223]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4902m92rn3z3wrw for ; Sat, 11 Apr 2020 18:02:48 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=content-type:mime-version:subject:from:in-reply-to: content-transfer-encoding:references:to:cc; s=s1; bh=TVpVorQCzYxSX5+Wn/+1BHYUIGwI7/iTZ+X/z9KofGI=; b=ijDLZpQvqNP4TnHDdWYIAQOCZXtaVXHUqG12mHFAw12enP3sJIrpUPkaUDmZD9YYu6LL aqsyOThebouzpzWpmvbruM3lGxHOvmJkTac0kK41dg7GYzJMGfk/66OgDiGVTxEuKOO/N8 kbvYzPJhPWwicryJyT8jU3D3Y6PIq+CdU= Received: by filterdrecv-p3iad2-8ddf98858-cq957 with SMTP id filterdrecv-p3iad2-8ddf98858-cq957-19-5E920647-51 2020-04-11 18:02:47.446323411 +0000 UTC m=+1443317.594130220 Received: from mail.distal.com (unknown) by ismtpd0050p1iad1.sendgrid.net (SG) with ESMTP id QAHo62m9T5iOhLu4LhB53w Sat, 11 Apr 2020 18:02:47.288 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [2001:470:e24c:200:14a4:6888:ae02:b3bb]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id e65237bf (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 14:02:46 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: ZFS server has gone crazy slow From: Chris Ross In-Reply-To: <20200411174831.GA54397@fuz.su> Date: Sat, 11 Apr 2020 18:02:47 +0000 (UTC) Content-Transfer-Encoding: quoted-printable Message-Id: <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXZXch77HK5mEB291x?= =?us-ascii?Q?DDHUc0txwnvDp=2F6ZY=2Ff7B=2FByS2TkncLNpjXOy46?= =?us-ascii?Q?0ZqWzEIaV3vdMHzUQDgWhfwHxFraMA3zwSgPMgl?= =?us-ascii?Q?Uo8=2Fe08xilYLDpo2Ls8DFqTbuBauMLutdAZorpP?= =?us-ascii?Q?qI8cyeFCajWtUoJJRfi=2FY83=2FJX6IRKBZm7XaEl7?= =?us-ascii?Q?sndPxmRnKtbMOl1kZfrPg=3D=3D?= To: Robert Clausecker Cc: freebsd-fs , Chris Ross X-Rspamd-Queue-Id: 4902m92rn3z3wrw X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=ijDLZpQv; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.100.223 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-3.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17]; MV_CASE(0.50)[]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-1.67)[ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[223.100.89.167.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; TAGGED_FROM(0.00)[13739864-7f7e-freebsd-fs=freebsd.org,freebsd]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 18:02:50 -0000 > On Apr 11, 2020, at 13:48, Robert Clausecker wrote: >=20 > Hi Chris, >=20 > What does zpool status say about the pool? Also check if the SMART value= s > of these disks are okay. Looks okay, I think. Full output below. pool: zroot state: ONLINE status: Some supported features are not enabled on the pool. The pool can still be used, but some features are unavailable. action: Enable all features using 'zpool upgrade'. Once this is done, the pool may no longer be accessible by software that does not support the features. See zpool-features(7) for details. scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 gpt/disk0 ONLINE 0 0 0 gpt/disk1 ONLINE 0 0 0 errors: No known data errors And, I=E2=80=99m not sure how to check SMART values. This is a Dell server= , and I=E2=80=99m 95% sure I have it=E2=80=99s RAID controller (Dell PERC 6= , mfi0) just set to provide JBOD for the two disks. Will the smartmontools= package be useful with disks on an mfi controller? - Chris From owner-freebsd-fs@freebsd.org Sat Apr 11 18:11:46 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 13F002BF20B for ; Sat, 11 Apr 2020 18:11:46 +0000 (UTC) (envelope-from ipluta@wp.pl) Received: from mx3.wp.pl (mx3.wp.pl [212.77.101.9]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4902yS490zz3xS6 for ; Sat, 11 Apr 2020 18:11:44 +0000 (UTC) (envelope-from ipluta@wp.pl) Received: (wp-smtpd smtp.wp.pl 19071 invoked from network); 11 Apr 2020 20:11:41 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=wp.pl; s=1024a; t=1586628701; bh=MC8acvZ4EipUHjiKsbk/+0J/NBf7GpcrWyCJXMujdck=; h=Subject:To:Cc:From; b=r36Uio9ZjPBkQkVLzJcIPhQAfYyyRpHwe+SDxPRLri6ePPgMJjdAMeiMv3QIhc2jY cG8oK8mVJVzMGRGvE2GMUa9sOujgRn5nbhNiGkZoBkBl4d/Pp1tjxiAqlvos/YZDkL En9pwtPnPY3DTbeb24QF1+DB2J+l97fx6sJwMprY= Received: from aazf8.neoplus.adsl.tpnet.pl (HELO [10.0.0.81]) (ipluta@wp.pl@[83.6.143.8]) (envelope-sender ) by smtp.wp.pl (WP-SMTPD) with ECDHE-RSA-AES256-GCM-SHA384 encrypted SMTP for ; 11 Apr 2020 20:11:41 +0200 Subject: Re: ZFS server has gone crazy slow To: Chris Ross , Robert Clausecker Cc: freebsd-fs References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> From: "Ireneusz Pluta/wp.pl" Message-ID: <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> Date: Sat, 11 Apr 2020 20:10:43 +0200 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.7.0 MIME-Version: 1.0 In-Reply-To: <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: pl X-WP-MailID: 55b1bb3ba7ac357982460e42be2f9a83 X-WP-AV: skaner antywirusowy Poczty Wirtualnej Polski X-WP-SPAM: NO 0000000 [sSOU] X-Rspamd-Queue-Id: 4902yS490zz3xS6 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=wp.pl header.s=1024a header.b=r36Uio9Z; dmarc=pass (policy=none) header.from=wp.pl; spf=pass (mx1.freebsd.org: domain of ipluta@wp.pl designates 212.77.101.9 as permitted sender) smtp.mailfrom=ipluta@wp.pl X-Spamd-Result: default: False [-3.00 / 15.00]; R_SPF_ALLOW(-0.20)[+ip4:212.77.96.0/19]; FREEMAIL_FROM(0.00)[wp.pl]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[wp.pl:+]; DMARC_POLICY_ALLOW(-0.50)[wp.pl,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_ENVFROM(0.00)[wp.pl]; ASN(0.00)[asn:12827, ipnet:212.77.101.0/24, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[wp.pl.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[wp.pl:s=1024a]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TAGGED_RCPT(0.00)[freebsd]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE_FREEMAIL(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[9.101.77.212.list.dnswl.org : 127.0.5.0]; IP_SCORE(0.00)[ip: (-7.65), ipnet: 212.77.101.0/24(-3.98), asn: 12827(-2.87), country: PL(0.06)]; RWL_MAILSPIKE_POSSIBLE(0.00)[9.101.77.212.rep.mailspike.net : 127.0.0.17]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 18:11:46 -0000 W dniu 2020-04-11 o 20:02, Chris Ross pisze: > And, I’m not sure how to check SMART values. This is a Dell server, and I’m 95% sure I have it’s RAID controller (Dell PERC 6, mfi0) just set to provide JBOD for the two disks. Will the smartmontools package be useful with disks on an mfi controller? Does the controller expose these jbods as /dev/da? devices to the system ? From owner-freebsd-fs@freebsd.org Sat Apr 11 18:41:42 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 01A322BFDBF for ; Sat, 11 Apr 2020 18:41:42 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) Received: from xtrwsqdf.outbound-mail.sendgrid.net (xtrwsqdf.outbound-mail.sendgrid.net [167.89.100.223]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4903d0366mz406Q for ; Sat, 11 Apr 2020 18:41:40 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=content-type:mime-version:subject:from:in-reply-to: content-transfer-encoding:references:to:cc; s=s1; bh=QPl02ssWqCfpd8Fci0bpDopQtSTwy1rMFoAVCZLAw1k=; b=Fmhif1UhQG7mkaiMQ9DYF6m6ldlmqPOEwtbUNODd2IUDte05NDxyhRurfqZPyxK+/Ae0 sQmkUlDtTGyn93xDaOyWmhJ9cYu0jCHbBUMt8IpHGNFNLl6VeX9pW5hF4RzbqFlHQbkqvj 97qbkigtvbaSxqzomJFdmBgjTzkiSDWaA= Received: by filterdrecv-p3iad2-8ddf98858-f4h4l with SMTP id filterdrecv-p3iad2-8ddf98858-f4h4l-19-5E920F61-31 2020-04-11 18:41:37.505406232 +0000 UTC m=+1445650.824638391 Received: from mail.distal.com (unknown) by ismtpd0117p1mdw1.sendgrid.net (SG) with ESMTP id U7BAIbKwQVyx6GagocLS4w Sat, 11 Apr 2020 18:41:37.377 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [2001:470:e24c:200:14a4:6888:ae02:b3bb]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 2e44148d (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 14:41:36 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: ZFS server has gone crazy slow From: Chris Ross In-Reply-To: <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> Date: Sat, 11 Apr 2020 18:41:37 +0000 (UTC) Content-Transfer-Encoding: quoted-printable Message-Id: References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXdF33uVZzQoN2Gb9j?= =?us-ascii?Q?+9vBOXXzj7CDtgHY4gWTcwh0HnVeoqIacxK4gmX?= =?us-ascii?Q?ZVwomrgxP7=2FJkBeY9CzGuVKQBme7KtznJ=2FcaBS7?= =?us-ascii?Q?j7CkPJeVNgUt36YMcIk3XbydZdAHZeDW6Ck3YZd?= =?us-ascii?Q?A0VxRFrl+j5aOCCIprfydXI1EXANsaTzMzuaBU8?= =?us-ascii?Q?63Ceu=2Fh6V7NBTN+usymvw=3D=3D?= To: Robert Clausecker Cc: freebsd-fs , Chris Ross X-Rspamd-Queue-Id: 4903d0366mz406Q X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=Fmhif1Uh; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.100.223 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-4.95 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17]; MV_CASE(0.50)[]; TAGGED_RCPT(0.00)[freebsd]; MIME_GOOD(-0.10)[text/plain]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; IP_SCORE(-2.75)[ip: (-5.41), ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[223.100.89.167.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[13739864-7f7e-freebsd-fs=freebsd.org,freebsd] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 18:41:42 -0000 > On Apr 11, 2020, at 14:02, Chris Ross wrote: > I _think_ this is a filesystem problem. [=E2=80=A6] Okay, this is pretty certain now. I=E2=80=99m trying to disable some daemo= ns (squid specifically), and to move a large log file out of the way, I did= : % mv access.log access.log.old && touch access.log ...as the squid user. For 10 minutes or more, that mv was in disk wait: squid 2365 0.0 0.0 4240 1892 3 D+ 14:22 0:00.02 mv access= .log access.log.old I didn=E2=80=99t truss that mv to know why, but that should never happen. = A rename in the same directory is about the most simple action to perform= =E2=80=A6 I=E2=80=99m going to reboot now that I=E2=80=99ve disabled squid and moved = it=E2=80=99s logs (it did eventually complete, somewhere in the 11-14 minut= e range). I asked the daemon to stop 30+ minutes ago, it still hasn=E2=80= =99t exited=E2=80=A6. I doubt that will fix the issues I=E2=80=99m having, = but it=E2=80=99s a start. - Chris From owner-freebsd-fs@freebsd.org Sat Apr 11 19:25:59 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id A83D1278A03 for ; Sat, 11 Apr 2020 19:25:59 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) Received: from xtrwsxqr.outbound-mail.sendgrid.net (xtrwsxqr.outbound-mail.sendgrid.net [167.89.106.69]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4904c65bdZz429Q for ; Sat, 11 Apr 2020 19:25:58 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=content-type:mime-version:subject:from:in-reply-to: content-transfer-encoding:references:to:cc; s=s1; bh=L3QU1aqrr/dCTxcLfEoi1w/gjK/CJSvAUy84I6UkH9g=; b=GpiM4OdcHB6D16ruVkTJJVwFTAwYh89AYZ2u7ColCSHmycYNHP0/s0Tu1POCN6c2H7cp cXionFHzJ/oFJjoGD3GtXJG/YIklS+L+LnL6LhGJXlTSH38ttbx/nGW33dEfQvP1wbNfVZ Bdf7hJF2d7X9EkbROFbnaiEv82NmC6yYI= Received: by filter0081p3las1.sendgrid.net with SMTP id filter0081p3las1-23682-5E9219C5-20 2020-04-11 19:25:57.242616472 +0000 UTC m=+856498.432381552 Received: from mail.distal.com (unknown) by ismtpd0039p1iad1.sendgrid.net (SG) with ESMTP id vR56VFraQ86V-X3Z0BRvVw Sat, 11 Apr 2020 19:25:57.023 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [206.138.151.12]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id 99ccdff1 (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 14:59:16 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: ZFS server has gone crazy slow From: Chris Ross In-Reply-To: <68328a40-0e3d-f9cf-510b-9cbfd7cb8acd@grosbein.net> Date: Sat, 11 Apr 2020 19:25:57 +0000 (UTC) Content-Transfer-Encoding: quoted-printable Message-Id: <654D00F8-DBEC-49BD-B871-7EB830F49D50@distal.com> References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <68328a40-0e3d-f9cf-510b-9cbfd7cb8acd@grosbein.net> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXcFUA0lhjELtoV3Mz?= =?us-ascii?Q?d0JINwqw+ym612NsdzaAma+ZokkaT5h2jF+7492?= =?us-ascii?Q?xJJOarFaPq3ghR47aSEsoKnFBF8YX89wfFw2x7q?= =?us-ascii?Q?tUXsehr12Bryfdk2m56zA=2FosYAQK=2FwYM4lXqDub?= =?us-ascii?Q?IegwgBBgmkW=2Fy2FgansJ7=2F1FjpOWAcM5b8V3RW?= To: Eugene Grosbein Cc: freebsd-fs , freebsd-stable@freebsd.org X-Rspamd-Queue-Id: 4904c65bdZz429Q X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=GpiM4Odc; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.106.69 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-3.87 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; IP_SCORE(-1.67)[ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; RCVD_IN_DNSWL_NONE(0.00)[69.106.89.167.list.dnswl.org : 127.0.15.0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[69.106.89.167.rep.mailspike.net : 127.0.0.17]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[13739864-7f7e-freebsd-fs=freebsd.org,freebsd] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 19:25:59 -0000 > On Apr 11, 2020, at 14:33, Eugene Grosbein wrote: >=20 > 12.04.2020 0:36, Chris Ross wrote: >=20 >> I have a FreeBSD 11.3-STABLE server that is my router, using a ZFS mirro= r (of two GPT disks) as it=E2=80=99s disk. It=E2=80=99s many years old, an= d has only been misbehaving like this for a day or so. I=E2=80=99m trying = to figure out what=E2=80=99s wrong. >>=20 >> [=E2=80=A6] >>=20 >> I _think_ this is a filesystem problem. It=E2=80=99s very hard to diagn= ose because logging in, and doing anything, takes many seconds per command.= zpool status shows my mirror as online, so I=E2=80=99m not sure where I s= hould check. >>=20 >> I=E2=80=99d appreciate any help! Thanks much=E2=80=A6 >=20 > First of all you should check if any of your ZFS pools is low on space. Wow. I=E2=80=99m so embarrassed that I didn=E2=80=99t notice that myself. = You mentioned it, and now I look back at df output and see that the filesy= stems are all very nearly full! It=E2=80=99s very slowly booting now, but assumedly after it comes online, = I=E2=80=99ll be able to rectify that situation and hopefully that will be t= he issue. Thanks, and sorry that I hadn=E2=80=99t seen that myself! - Chris From owner-freebsd-fs@freebsd.org Sat Apr 11 19:26:46 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 9B147278B63 for ; Sat, 11 Apr 2020 19:26:46 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) Received: from xtrwsxqr.outbound-mail.sendgrid.net (xtrwsxqr.outbound-mail.sendgrid.net [167.89.106.69]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4904d205Ljz42Tc for ; Sat, 11 Apr 2020 19:26:45 +0000 (UTC) (envelope-from bounces+13739864-7f7e-freebsd-fs=freebsd.org@em848.distal.com) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=distal.com; h=content-type:mime-version:subject:from:in-reply-to: content-transfer-encoding:references:to:cc; s=s1; bh=A7VHvf3Ej7zZMr/VXZyh3dKzyxe4ppQCPQQBco1Pq44=; b=OKLmUBnlfSnRKHIhyBTELpu/Lz8Fg1yJtAphj4SMRwUJsSQHWXW4roRnDiAxoOih6kk7 Y9Gw6DeBOvzggOGzKm96FvD9oDnlqVVwEshoYN8Bq6Y/k0uTIgcW62+ypKqzsU1cBK9qMU a5UWW7k8YRy+uBSX8JfJv4/A788sRMSKY= Received: by filterdrecv-p3iad2-8ddf98858-mzxhd with SMTP id filterdrecv-p3iad2-8ddf98858-mzxhd-20-5E9219F0-15 2020-04-11 19:26:40.533023621 +0000 UTC m=+1448354.630915560 Received: from mail.distal.com (unknown) by ismtpd0053p1iad1.sendgrid.net (SG) with ESMTP id W7E0blXZS6-AGDq6F315Xg Sat, 11 Apr 2020 19:26:40.436 +0000 (UTC) Received: from magrathea.distal.com (magrathea.distal.com [2001:470:e24c:200:14a4:6888:ae02:b3bb]) by tristain.distal.com (OpenSMTPD) with ESMTPSA id f713286f (TLSv1.2:ECDHE-RSA-AES256-GCM-SHA384:256:NO); Sat, 11 Apr 2020 15:26:40 -0400 (EDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: ZFS server has gone crazy slow From: Chris Ross In-Reply-To: <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> Date: Sat, 11 Apr 2020 19:26:40 +0000 (UTC) Content-Transfer-Encoding: quoted-printable Message-Id: <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> X-Mailer: Apple Mail (2.3608.80.23.2.2) X-SG-EID: =?us-ascii?Q?gDj=2Futz1vvM0Gg5Dx3C984MHE5rSknXbUiMX+7YYKy8H35YTA3nJpmg1IQ5mw9?= =?us-ascii?Q?RS2bD=2FH4Uluvnmedtzs9ncXatiP41xWyXwwbae2?= =?us-ascii?Q?R+buzy8N6IfrIRaugYCYsrK8v8Jf7zcl6OcCKOV?= =?us-ascii?Q?Sv+bBMMF1RppEZ15za7idmGbAFAMYYkJWQtqNpE?= =?us-ascii?Q?n15+ye1u4NqmDS6qkIkbg1IDzfsPuZzNVEyibs5?= =?us-ascii?Q?O5vPcBmx=2FaynZXOhH77bkjLjO3DAZMP4W6bFP4=2F?= =?us-ascii?Q?D3E0YGRlhdmKkesCLavGQ=3D=3D?= To: "Ireneusz Pluta/wp.pl" Cc: Robert Clausecker , freebsd-fs X-Rspamd-Queue-Id: 4904d205Ljz42Tc X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=distal.com header.s=s1 header.b=OKLmUBnl; dmarc=pass (policy=none) header.from=distal.com; spf=pass (mx1.freebsd.org: domain of bounces@em848.distal.com designates 167.89.106.69 as permitted sender) smtp.mailfrom=bounces@em848.distal.com X-Spamd-Result: default: False [-3.87 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:167.89.0.0/17:c]; MV_CASE(0.50)[]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[distal.com:+]; DMARC_POLICY_ALLOW(-0.50)[distal.com,none]; FORGED_SENDER(0.30)[cross@distal.com,bounces@em848.distal.com]; FREEMAIL_TO(0.00)[wp.pl]; IP_SCORE(-1.67)[ipnet: 167.89.96.0/20(-4.61), asn: 11377(-3.67), country: US(-0.05)]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:11377, ipnet:167.89.96.0/20, country:US]; FROM_NEQ_ENVFROM(0.00)[cross@distal.com,bounces@em848.distal.com]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[distal.com:s=s1]; TAGGED_FROM(0.00)[13739864-7f7e-freebsd-fs=freebsd.org,freebsd]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[69.106.89.167.list.dnswl.org : 127.0.15.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[69.106.89.167.rep.mailspike.net : 127.0.0.17] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 19:26:46 -0000 > On Apr 11, 2020, at 14:10, Ireneusz Pluta/wp.pl wrote: > W dniu 2020-04-11 o 20:02, Chris Ross pisze: >> And, I=E2=80=99m not sure how to check SMART values. This is a Dell ser= ver, and I=E2=80=99m 95% sure I have it=E2=80=99s RAID controller (Dell PER= C 6, mfi0) just set to provide JBOD for the two disks. Will the smartmonto= ols package be useful with disks on an mfi controller? > Does the controller expose these jbods as /dev/da? devices to the system = ? mfid devices. And, googling showed that I might want to use mfip to allow = me to access the drives for smartctl, and I=E2=80=99ll work on that more in= a bit.. But, Eugene was kind enough to point out something that should=E2=80=99ve b= een obvious, that I=E2=80=99d filled my ZFS volumes. I had a snapshot of e= verything from a while ago, and one of the filesystems contains large data = files. Deleting the snapshot on that volume freed things back to functiona= l. Thanks all. - Chris From owner-freebsd-fs@freebsd.org Sat Apr 11 21:24:35 2020 Return-Path: Delivered-To: freebsd-fs@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8565D27BBDE for ; Sat, 11 Apr 2020 21:24:35 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (mail.lysator.liu.se [IPv6:2001:6b0:17:f0a0::3]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4907Dy2ph0z48JP for ; Sat, 11 Apr 2020 21:24:34 +0000 (UTC) (envelope-from pen@lysator.liu.se) Received: from mail.lysator.liu.se (localhost [127.0.0.1]) by mail.lysator.liu.se (Postfix) with ESMTP id 2251D40004 for ; Sat, 11 Apr 2020 23:24:29 +0200 (CEST) Received: by mail.lysator.liu.se (Postfix, from userid 1004) id 107C440003; Sat, 11 Apr 2020 23:24:28 +0200 (CEST) X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on bernadotte.lysator.liu.se X-Spam-Level: X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,AWL autolearn=disabled version=3.4.2 X-Spam-Score: -1.0 Received: from [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842] (unknown [IPv6:2001:9b1:28ff:d901:f4ee:5e2b:488:2842]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.lysator.liu.se (Postfix) with ESMTPSA id 7BA6940003; Sat, 11 Apr 2020 23:24:18 +0200 (CEST) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 13.0 \(3608.60.0.2.5\)) Subject: Re: ZFS server has gone crazy slow From: Peter Eriksson In-Reply-To: <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> Date: Sat, 11 Apr 2020 23:24:18 +0200 Content-Transfer-Encoding: quoted-printable Message-Id: References: <2182C27C-A5D3-41BF-9CE9-7C6883E43074@distal.com> <20200411174831.GA54397@fuz.su> <6190573D-BCA7-44F9-86BD-0DCBB1F69D1D@distal.com> <6fd7a561-462e-242d-5057-51c52d716d68@wp.pl> <7AA1EA07-6041-464A-A39A-158ACD1DC11C@distal.com> To: freebsd-fs X-Mailer: Apple Mail (2.3608.60.0.2.5) X-Virus-Scanned: ClamAV using ClamSMTP X-Rspamd-Queue-Id: 4907Dy2ph0z48JP X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=pass (policy=none) header.from=liu.se; spf=pass (mx1.freebsd.org: domain of pen@lysator.liu.se designates 2001:6b0:17:f0a0::3 as permitted sender) smtp.mailfrom=pen@lysator.liu.se X-Spamd-Result: default: False [-3.29 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:mail.lysator.liu.se]; MV_CASE(0.50)[]; TAGGED_RCPT(0.00)[freebsd]; PREVIOUSLY_DELIVERED(0.00)[freebsd-fs@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; IP_SCORE(-0.99)[ip: (-3.05), ipnet: 2001:6b0::/32(-1.03), asn: 1653(-0.85), country: EU(-0.01)]; DMARC_POLICY_ALLOW(-0.50)[liu.se,none]; RCVD_IN_DNSWL_NONE(0.00)[3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.a.0.f.7.1.0.0.0.b.6.0.1.0.0.2.list.dnswl.org : 127.0.11.0]; RCVD_TLS_LAST(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:1653, ipnet:2001:6b0::/32, country:EU]; FREEMAIL_CC(0.00)[wp.pl]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 11 Apr 2020 21:24:35 -0000 Yes, this is a know =E2=80=9Cfeature=E2=80=9D of ZFS. When filesystems = are nearly full (or near quota limits) it will tune down the write = transaction sizes which causes I/O (in order to make sure it doesn=E2=80=99= t write more than the quota, and try to opimize for storage instead of = latency/speed) to inflate and things basically grinds to a halt=E2=80=A6=20= Another fun thing that might happen is if you reboot your server and = happen to have a lot of queued up writes in the ZIL (for example if you = did a =E2=80=9Czfs destroy -d -r POOL@snapshots=E2=80=9D = (deferred(background) destroys of snapshots) and do a hard reboot while = it=E2=80=99s busy it will =E2=80=9Cwrite out=E2=80=9D those queued = transactions at filesystem mount time during the boot sequence - which = may take a long time (many hours if your have big pools and some of them = have filesystems (datasets) that are nearly full - ours pools are = 100-400TB). Taking snapshots will also grind to a crawl if you are near = quota limits on filesystems. Fun fun :-) (We have modified our rc.d startup scripts so we do the =E2=80=9Czfs = mount -a=E2=80=9D part in the background so the rest of the system = doesn=E2=80=99t have to wait for it (so we can get a ssh login prompt = and login and check the progress of the filesystem mounts. (And then = =E2=80=9Cnfs exports=E2=80=9D, nfsd and samba also waits in the = background for that to finish before starting up. A bit of a hack but it = works (a real parallel server manager with dependencies would have been = better :-) - Peter > On 11 Apr 2020, at 21:26, Chris Ross wrote: >=20 >=20 >=20 >> On Apr 11, 2020, at 14:10, Ireneusz Pluta/wp.pl wrote: >> W dniu 2020-04-11 o 20:02, Chris Ross pisze: >>> And, I=E2=80=99m not sure how to check SMART values. This is a Dell = server, and I=E2=80=99m 95% sure I have it=E2=80=99s RAID controller = (Dell PERC 6, mfi0) just set to provide JBOD for the two disks. Will = the smartmontools package be useful with disks on an mfi controller? >> Does the controller expose these jbods as /dev/da? devices to the = system ? >=20 > mfid devices. And, googling showed that I might want to use mfip to = allow me to access the drives for smartctl, and I=E2=80=99ll work on = that more in a bit.. >=20 > But, Eugene was kind enough to point out something that should=E2=80=99v= e been obvious, that I=E2=80=99d filled my ZFS volumes. I had a = snapshot of everything from a while ago, and one of the filesystems = contains large data files. Deleting the snapshot on that volume freed = things back to functional. >=20 > Thanks all. >=20 > - Chris > _______________________________________________ > freebsd-fs@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-fs > To unsubscribe, send any mail to "freebsd-fs-unsubscribe@freebsd.org"