From owner-freebsd-stable@freebsd.org Sun Apr 7 02:23:07 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id A5CEB1575116 for ; Sun, 7 Apr 2019 02:23:07 +0000 (UTC) (envelope-from graham@menhennitt.com.au) Received: from ostrich.birch.relay.mailchannels.net (ostrich.birch.relay.mailchannels.net [23.83.209.138]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id AF312764AC for ; Sun, 7 Apr 2019 02:23:05 +0000 (UTC) (envelope-from graham@menhennitt.com.au) X-Sender-Id: dreamhost|x-authsender|graham@menhennitt.com.au Received: from relay.mailchannels.net (localhost [127.0.0.1]) by relay.mailchannels.net (Postfix) with ESMTP id 1C8666A140E for ; Sun, 7 Apr 2019 01:45:34 +0000 (UTC) Received: from pdx1-sub0-mail-a65.g.dreamhost.com (100-96-7-60.trex.outbound.svc.cluster.local [100.96.7.60]) (Authenticated sender: dreamhost) by relay.mailchannels.net (Postfix) with ESMTPA id 85AD96A160B for ; Sun, 7 Apr 2019 01:45:33 +0000 (UTC) X-Sender-Id: dreamhost|x-authsender|graham@menhennitt.com.au Received: from pdx1-sub0-mail-a65.g.dreamhost.com ([TEMPUNAVAIL]. [64.90.62.162]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384) by 0.0.0.0:2500 (trex/5.17.2); Sun, 07 Apr 2019 01:45:34 +0000 X-MC-Relay: Neutral X-MailChannels-SenderId: dreamhost|x-authsender|graham@menhennitt.com.au X-MailChannels-Auth-Id: dreamhost X-Stupid-Abiding: 7822394202c77e72_1554601533894_1533067929 X-MC-Loop-Signature: 1554601533894:911321468 X-MC-Ingress-Time: 1554601533893 Received: from pdx1-sub0-mail-a65.g.dreamhost.com (localhost [127.0.0.1]) by pdx1-sub0-mail-a65.g.dreamhost.com (Postfix) with ESMTP id 75FA77FE09 for ; Sat, 6 Apr 2019 18:45:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=menhennitt.com.au; h= subject:to:references:from:message-id:date:mime-version :in-reply-to:content-type:content-transfer-encoding; s= menhennitt.com.au; bh=MOlcd8Qb6f4Knx2H1MQsmNRaFfc=; b=aGrXU1bCCq aH00lt8dvFe/Du+HQO+KnWUgMtNu4YAOk0ykr2AqWd+j93B2L7OwvN1M+gyJIZQN 7O817Ju1etN0AM1rPoWJKb8zicwNcWEs/AjNdGp3A4OoMq0MiwcPWd3+Quv3Cpef aN75y/sK6cTv3LSJI8g059Jt1RUyr5JbI= Received: from [203.3.73.6] (unknown [203.220.148.60]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) (Authenticated sender: graham@menhennitt.com.au) by pdx1-sub0-mail-a65.g.dreamhost.com (Postfix) with ESMTPSA id 438E67FE0C for ; Sat, 6 Apr 2019 18:45:28 -0700 (PDT) Subject: Re: em performs worse than igb (latency wise) in 12? To: freebsd-stable@freebsd.org References: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> X-DH-BACKEND: pdx1-sub0-mail-a65 From: Graham Menhennitt Message-ID: <81e63eaf-352d-4b29-a4c0-9085e699931e@menhennitt.com.au> Date: Sun, 7 Apr 2019 11:45:26 +1000 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.13; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US X-VR-OUT-STATUS: OK X-VR-OUT-SCORE: 0 X-VR-OUT-SPAMCAUSE: gggruggvucftvghtrhhoucdtuddrgeduuddruddtgdehtdcutefuodetggdotefrodftvfcurfhrohhfihhlvgemucggtfgfnhhsuhgsshgtrhhisggvpdfftffgtefojffquffvnecuuegrihhlohhuthemuceftddtnecunecujfgurhepuffvfhfhkffffgggjggtgfesthekredttdefjeenucfhrhhomhepifhrrghhrghmucfovghnhhgvnhhnihhtthcuoehgrhgrhhgrmhesmhgvnhhhvghnnhhithhtrdgtohhmrdgruheqnecuffhomhgrihhnpehfrhgvvggsshgurdhorhhgnecukfhppedvtdefrddvvddtrddugeekrdeitdenucfrrghrrghmpehmohguvgepshhmthhppdhhvghloheplgdvtdefrdefrdejfedriegnpdhinhgvthepvddtfedrvddvtddrudegkedriedtpdhrvghtuhhrnhdqphgrthhhpefirhgrhhgrmhcuofgvnhhhvghnnhhithhtuceoghhrrghhrghmsehmvghnhhgvnhhnihhtthdrtghomhdrrghuqedpmhgrihhlfhhrohhmpehgrhgrhhgrmhesmhgvnhhhvghnnhhithhtrdgtohhmrdgruhdpnhhrtghpthhtohepfhhrvggvsghsugdqshhtrggslhgvsehfrhgvvggsshgurdhorhhgnecuvehluhhsthgvrhfuihiivgeptd Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: AF312764AC X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; dkim=pass header.d=menhennitt.com.au header.s=menhennitt.com.au header.b=aGrXU1bC; spf=pass (mx1.freebsd.org: domain of graham@menhennitt.com.au designates 23.83.209.138 as permitted sender) smtp.mailfrom=graham@menhennitt.com.au X-Spamd-Result: default: False [-0.54 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[menhennitt.com.au:s=menhennitt.com.au]; RCVD_COUNT_FIVE(0.00)[6]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:23.83.208.1/20]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.98)[-0.982,0]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_SHORT(0.34)[0.340,0]; DKIM_TRACE(0.00)[menhennitt.com.au:+]; MX_GOOD(-0.01)[vade-in1.mail.dreamhost.com,vade-in2.mail.dreamhost.com]; RCVD_IN_DNSWL_NONE(0.00)[138.209.83.23.list.dnswl.org : 127.0.3.0]; NEURAL_HAM_MEDIUM(-0.51)[-0.511,0]; IP_SCORE(0.12)[ipnet: 23.83.208.0/21(0.38), asn: 36483(0.31), country: CA(-0.09)]; DMARC_NA(0.00)[menhennitt.com.au]; FROM_EQ_ENVFROM(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:36483, ipnet:23.83.208.0/21, country:CA]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Apr 2019 02:23:07 -0000 Not that it's at all relevant to the question here, but... It does mostly work without em in the 12 kernel - I'm not sure how, but=20 it does. I upgraded to 12-stable via source but didn't add em to my custom=20 kernel. Most things worked - basic network functionality. But I had=20 problems with ipfw and igb. Adding em to the kernel fixed them. Graham On 6/4/19 6:12 am, Kris von Mach wrote: > On 4/6/2019 2:56 AM, Pete French wrote: >> Something odd going on there there - I am using 12-STABLE and I have=20 >> igb just fine, and it attaches to the same hardware that 11 did: > > It does work in 12, throughput is great, just that the latency is=20 > higher than 11. > > igb0: flags=3D8843 metric 0 mtu= =20 > 1500 > options=3De527bb=20 > > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ether 38:ea:a7:8d:c1:6c > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet 208.72.56.19 netmask 0x= fffffc00 broadcast 208.72.59.255 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet6 fe80::3aea:a7ff:fe8d:c= 16c%igb0 prefixlen 64 scopeid 0x1 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 inet6 2602:ffb8::208:72:56:9= prefixlen 64 > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 media: Ethernet autoselect (= 1000baseT ) > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 status: active > =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 nd6 options=3D21 > >> Do you have a custom kernel, and if so did you see this note in=20 >> UPDATING? > > Yes I do, but it includes all of GENERIC which includes em drivers,=20 > otherwise it wouldn't even work with the network card. > > my custom kernel: > > include GENERIC > ident=C2=A0=C2=A0 CUSTOM > makeoptions WITH_EXTRA_TCP_STACKS=3D1 > options TCPHPTS > options SC_KERNEL_CONS_ATTR=3D(FG_GREEN|BG_BLACK) > options IPSTEALTH > options=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 AHC_REG_PRETTY_PRINT=C2=A0 = # Print register bitfields in debug > options=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 AHD_REG_PRETTY_PRINT=C2=A0 = # Print register bitfields in debug > device cryptodev > device aesni > > I did try without RACK just in case that was the culprit. > > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.or= g" From owner-freebsd-stable@freebsd.org Sun Apr 7 15:36:46 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2BAE91558EEF; Sun, 7 Apr 2019 15:36:46 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out3-smtp.messagingengine.com (out3-smtp.messagingengine.com [66.111.4.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id E390B9493B; Sun, 7 Apr 2019 15:36:43 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 0AB95210F2; Sun, 7 Apr 2019 11:36:43 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Sun, 07 Apr 2019 11:36:43 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:cc:subject:message-id:mime-version:content-type; s= fm2; bh=zGCT3CmTqGVetzGIOhw9+7WNGXlGtzmRRN3f2L3hir0=; b=rvVQwxiy /6iuy4UT5mJ4m1DKrN4kYnmAMhyF9Vf1SiwX59TdN7FurJC8mO9uKxBV1SMmupKO CD5GCMck/PvWtXqCf1eNVXjVz6YK3PddO/0na+kp3GZKKib84gU1GvLXqbCUiKAH nBbI3X4vyflmrYvdrSmF0btZjUJRLaqRlxWJAHohh6nxqaxij/1Jd/RWuH3IlE5R cC5UUqNw0dqA9g1J4ud8dKkjwp3tjtynJOY8uVTYl+h/lNAC+JjA9mtLgVSWvaAg oa2BeFBzLRJLAFUAxm4RrdvbOYjtokYvUZ/dleCNTRj/Jh1mzdi6uyQS79UhaEdN iBCVTs8aihGpNQ== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:message-id :mime-version:subject:to:x-me-proxy:x-me-proxy:x-me-sender :x-me-sender:x-sasl-enc; s=fm2; bh=zGCT3CmTqGVetzGIOhw9+7WNGXlGt zmRRN3f2L3hir0=; b=blmSv60FoYttUdqu8MIev5VKQH6cRYgaJgaT2CeE4PR/N iutP4nr2YUA587XGq3G+sdmxbOfyKJfc/3elMsXtzoh3MqMVxKC0T7rbOYlkrYHP Y4XIJkLTXmVaJVoYWXVeiePMpas8XndxtqoSnSG+42pa6v3eZCuJQ03WI/YN3wP9 auIKQln52M16CMu/Du/pAvkee+VQW2/xFB5hg4VsOrR/rM823lZUhtGQoyTc2uCw dPTm7I7Xu6JX8ttVpaVFS/IJeqmAdeaBeHl8ZxlJoxUmJpNx4Tt2qQ9O1iWXWC8A 08vFAmWKHrFNVBOlToFcL+p33WDss7DnyBb+QAm9A== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddruddugdekleculddtuddrgedutddrtddtmd cutefuodetggdotefrodftvfcurfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdp uffrtefokffrpgfnqfghnecuuegrihhlohhuthemuceftddtnecunecujfgurhepfffhvf fukfggtggufgesghdtreertdervdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgv tghhqdhlihhsthhsseiihiigshhtrdhnvghtqeenucfkphepkedvrdejtddrledurddutd dunecurfgrrhgrmhepmhgrihhlfhhrohhmpehtvggthhdqlhhishhtshesiiihgihsthdr nhgvthenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id 2F4FCE45F3; Sun, 7 Apr 2019 11:36:42 -0400 (EDT) Date: Sun, 7 Apr 2019 16:36:40 +0100 From: tech-lists To: freebsd-fs@freebsd.org Cc: freebsd-stable@freebsd.org Subject: about zfs and ashift and changing ashift on existing zpool Message-ID: <20190407153639.GA41753@rpi3.zyxst.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="EVF5PPMfhYS0aIcm" Content-Disposition: inline User-Agent: Mutt/1.11.4 (2019-03-13) X-Rspamd-Queue-Id: E390B9493B X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=rvVQwxiy; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=blmSv60F; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 66.111.4.27 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-9.18 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.27]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[zyxst.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; IP_SCORE(-3.49)[ip: (-9.45), ipnet: 66.111.4.0/24(-4.59), asn: 11403(-3.35), country: US(-0.06)]; RCVD_IN_DNSWL_LOW(-0.10)[27.4.111.66.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 07 Apr 2019 15:36:46 -0000 --EVF5PPMfhYS0aIcm Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hello, I have this in sysctl.conf on a desktop machine (12-stable): vfs.zfs.min_auto_ashift=3D12 this has not always been there. I guess the zpool pre-dates it. I only noticed it because have recently had to replace a disk in its zfs array when I saw this: % zpool status pool: storage state: ONLINE status: One or more devices is currently being resilvered. The pool will continue to function, possibly in a degraded state. action: Wait for the resilver to complete. scan: resilver in progress since Sun Apr 7 03:09:42 2019 3.46T scanned at 79.5M/s, 2.73T issued at 62.8M/s, 3.46T total 931G resilvered, 78.94% done, 0 days 03:22:41 to go config: NAME STATE READ WRITE CKSUM storage ONLINE 0 0 0 raidz1-0 ONLINE 0 0 0 replacing-0 ONLINE 0 0 1.65K ada2 ONLINE 0 0 0 ada1 ONLINE 0 0 0 block size: 512B configured, 4= 096B native ada3 ONLINE 0 0 0 ada4 ONLINE 0 0 0 What I'd like to know is: 1. is the above situation harmful to data 2. given that vfs.zfs.min_auto_ashift=3D12, why does it still say 512B configured for ada1 which is the new disk, or.. 3. does "configured" pertain to the pool, the disk, or both 4. what would be involved in making them all 4096B 5. does a 512B disk wear out faster than 4096B (all other things being equal) Given that the machine and disks were new in 2016, I can't understand why z= fs didn't default to 4096B on installation thanks, --=20 J. --EVF5PPMfhYS0aIcm Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAlyqGPwACgkQs8o7QhFz NAU4zQ//cpkfnOep4ZM6n3HiNrBZ6Jqcfam8VYoow7X3roQm5JoSiIj9lYd0BAOF qupSwygvQRx2L14MnfPfCHcIcJAi+Oj/hsT6t3GxyNnK5wcQr6ng24BQ0x6dnzgh OU2kxq0jn0S/NIs09IcAa4zaeXqxCcnIwHbyPAD8ilU261nibdrnqW+bzLUZarYj SXbap9G5MWA6U1mBZsmGdVRZpyGAM/G1xQ5BT+ovNezSr2ssABoAtSy9DsQnbgmU plpq697kZWRokNcV25ASlyK5+nC71RawxDQ/YnMAVU0ZWItAWUkyI1LLjXahZtDV lmBfNMcP79iAcxJvfca9tp4sESlUG3eV/kHnm6kgD9XLDXDyu7jQfpfbgpB+4a8o 0O/BzXpNQD78E3W4zHaKBVswman+KHbbzcXUGc6Icm2sS6BTExXJYFMtjZI9Zt9g ho13EYQWAAVRX0FZOGXkn4xFAOVU4o3D4WHdiDjS9bINNVY5h8Y0/f/L1Uv9I4CB S64UfdkqLFaI2YSikZjtwnRr7vhdwovRdErILkZCp0UrC0GnqWRknosd+Kc+1wQT 4/djfaI6W9aUWUlDSQfml8MW03Fm7BtISvCTzg/nAUm4PnNmxkT025FrR12LD/9Y 5heGKN/1HNTbEWu5uL2cGM85yA1nHAqBomdDPYzdHkRzUc+Fiu0= =PrOZ -----END PGP SIGNATURE----- --EVF5PPMfhYS0aIcm-- From owner-freebsd-stable@freebsd.org Mon Apr 8 06:45:08 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E6D6B15767C2 for ; Mon, 8 Apr 2019 06:45:07 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mx0.gentlemail.de (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 859F28A560 for ; Mon, 8 Apr 2019 06:45:01 +0000 (UTC) (envelope-from freebsd@omnilan.de) Received: from mh0.gentlemail.de (ezra.dcm1.omnilan.net [IPv6:2a00:e10:2800::a135]) by mx0.gentlemail.de (8.14.5/8.14.5) with ESMTP id x386iw49045652; Mon, 8 Apr 2019 08:44:58 +0200 (CEST) (envelope-from freebsd@omnilan.de) Received: from titan.inop.mo1.omnilan.net (s1.omnilan.de [217.91.127.234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by mh0.gentlemail.de (Postfix) with ESMTPSA id 4BA53A8D; Mon, 8 Apr 2019 08:44:58 +0200 (CEST) Subject: Re: EFI loader doesn't handle md_preload (md_image) correct? From: Harry Schmalzbauer To: Toomas Soome Cc: freebsd-stable@freebsd.org References: <591B12C6.4040301@omnilan.de> <591B1A8B.6070803@omnilan.de> <591B1EA4.600@omnilan.de> <591B2523.6040101@omnilan.de> <7CF3AC8F-A778-40AE-B457-9B96AE5B4719@me.com> <591B284B.6070204@omnilan.de> Organization: OmniLAN Message-ID: <30f7b617-1814-ad21-a445-a62758150dc3@omnilan.de> Date: Mon, 8 Apr 2019 08:44:57 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <591B284B.6070204@omnilan.de> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.2.7 (mx0.gentlemail.de [IPv6:2a00:e10:2800::a130]); Mon, 08 Apr 2019 08:44:58 +0200 (CEST) X-Milter: Spamilter (Reciever: mx0.gentlemail.de; Sender-ip: ; Sender-helo: mh0.gentlemail.de; ) X-Rspamd-Queue-Id: 859F28A560 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of freebsd@omnilan.de designates 2a00:e10:2800::a130 as permitted sender) smtp.mailfrom=freebsd@omnilan.de X-Spamd-Result: default: False [-5.66 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[omnilan.de]; HAS_ORG_HEADER(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mx0.gentlemail.de]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.85)[-0.851,0]; RCVD_TLS_LAST(0.00)[]; FREEMAIL_TO(0.00)[me.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:25074, ipnet:2a00:e10:2800::/64, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; IP_SCORE(-3.50)[ip: (-9.18), ipnet: 2a00:e10:2800::/64(-4.66), asn: 25074(-3.67), country: DE(-0.01)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2019 06:45:08 -0000 Am 16.05.2017 um 18:26 schrieb Harry Schmalzbauer: > Bezüglich Toomas Soome's Nachricht vom 16.05.2017 18:20 (localtime): >>> On 16. mai 2017, at 19:13, Harry Schmalzbauer wrote: >>> >>> Bezüglich Toomas Soome's Nachricht vom 16.05.2017 18:00 (localtime): >>>>> On 16. mai 2017, at 18:45, Harry Schmalzbauer >>>> > wrote: >>>>> >>>>> Bezüglich Harry Schmalzbauer's Nachricht vom 16.05.2017 17:28 (localtime): >>>>>> Bezüglich Toomas Soome's Nachricht vom 16.05.2017 16:57 (localtime): >>>>>>>> On 16. mai 2017, at 17:55, Harry Schmalzbauer >>>>>>> > wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> unfortunately I had some trouble with my preferred MFS-root setups. >>>>>>>> It seems EFI loader doesn't handle type md_image correctly. >>>>>>>> >>>>>>>> If I load any md_image with loader invoked by gptboot or gptzfsboot, >>>>>>>> 'lsmod' >>>>>>>> shows "elf kernel", "elf obj module(s)" and "md_image". >>>>>>>> >>>>>>>> Using the same loader.conf, but EFI loader, the md_image-file is >>>>>>>> prompted and sems to be loaded, but not registered. There's no >>>>>>>> md_image >>>>>>>> with 'lsmod', hence it's not astonsihing that kernel doesn't attach md0 >>>>>>>> so booting fails since there's no rootfs. >>>>>>>> >>>>>>>> Any help highly appreciated, hope Toomas doesn't mind beeing >>>>>>>> initially CC'd. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> -harry >>>>>>> The first question is, how large is the md_image and what other >>>>>>> modules are loaded? >>>>>> Thanks for your quick response. >>>>>> >>>>>> The images are 50-500MB uncompressed (provided by gzip compressed file). >>>>>> Small ammount of elf modules, 5, each ~50kB. >>>>> On the real HW, there's vmm and some more: >>>>> Id Refs Address Size Name >>>>> 1 46 0xffffffff80200000 16M kernel >>>>> 2 1 0xffffffff8121d000 86K unionfs.ko >>>>> 3 1 0xffffffff81233000 3.1M zfs.ko >>>>> 4 2 0xffffffff81545000 51K opensolaris.ko >>>>> 5 7 0xffffffff81552000 279K usb.ko >>>>> 6 1 0xffffffff81598000 67K ukbd.ko >>>>> 7 1 0xffffffff815a9000 51K umass.ko >>>>> 8 1 0xffffffff815b6000 46K aesni.ko >>>>> 9 1 0xffffffff815c3000 54K uhci.ko >>>>> 10 1 0xffffffff815d1000 65K ehci.ko >>>>> 11 1 0xffffffff815e2000 15K cc_htcp.ko >>>>> 12 1 0xffffffff815e6000 3.4M vmm.ko >>>>> 13 1 0xffffffffa3a21000 12K ums.ko >>>>> 14 1 0xffffffffa3a24000 9.1K uhid.ko >>>>> >>>>> Providing md_image uncompressed doesn't change anything. >>>>> >>>>> Will deploy a /usr separated rootfs, which is only ~100MB uncompressed >>>>> and see if that changes anything. >>>>> That's all I can provide, code is far beyond my knowledge... >>>>> >>>>> -harry >>>> >>>> The issue is, that current UEFI implementation is using 64MB staging >>>> memory for loading the kernel and modules and files. When the boot is >>>> called, the relocation code will put the bits from staging area into the >>>> final places. The BIOS version does not need such staging area, and that >>>> will explain the difference. >>>> >>>> I actually have different implementation to address the same problem, >>>> but thats for illumos case, and will need some work to make it usable >>>> for freebsd; the idea is actually simple - allocate staging area per >>>> loaded file and relocate the bits into the place by component, not as >>>> continuous large chunk (this would also allow to avoid the mines like >>>> planted by hyperv;), but right now there is no very quick real solution >>>> other than just build efi loader with larger staging size. >>> Ic, thanks for the explanation. >>> While not aware about the purpose of the staging area nor the >>> consequences of enlarging it, do you think it's feasable increasing it >>> to 768Mib? >>> >>> At least now I have an idea baout the issue and an explanation why >>> reducing md_imgae to 100MB hasn't helped – still more than 64... >>> >>> Any quick hint where to define the staging area size highly appreciated, >>> fi there are no hard objections against a 768MB size. >>> >>> -harry >> The problem is that before UEFI Boot Services are not switched off, the memory is managed (and owned) by the firmware, > Hmm, I've been expecting something like that (owend by firmware) ;-) > > So I'll stay with CSM for now, and will happily be an early adopter if > you need someone to try anything (-stable mergable). Hello Toomas, thanks for your ongoing FreeBSD commits, saw your recent libstand improvements and the efiloader commit. Which remembers me nagging the skilled ones for my unmet needs ;-) I guess nobody had time to look at the MFS-root limitation with EFI vs. BIOS. If you have any news/plans, please share. The ability to boot via EFI gives a much better console experience/usability for admins, but on MFS-root system, I'm still forced to use the old loader path, because of the 64MB size limit. Do you think there's a chance that this will be resolved for FreeBSD? Thanks, -harry From owner-freebsd-stable@freebsd.org Mon Apr 8 07:19:11 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3B8801577412 for ; Mon, 8 Apr 2019 07:19:11 +0000 (UTC) (envelope-from tsoome@me.com) Received: from pv50p00im-ztbu10011701.me.com (pv50p00im-ztbu10011701.me.com [17.58.6.53]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DF4468B9DD for ; Mon, 8 Apr 2019 07:19:07 +0000 (UTC) (envelope-from tsoome@me.com) Received: from [10.131.72.237] (113-100-157-37.dyn.estpak.ee [37.157.100.113]) by pv50p00im-ztbu10011701.me.com (Postfix) with ESMTPSA id B14518A018B; Mon, 8 Apr 2019 07:18:59 +0000 (UTC) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: EFI loader doesn't handle md_preload (md_image) correct? From: Toomas Soome X-Mailer: iPhone Mail (16E227) In-Reply-To: <30f7b617-1814-ad21-a445-a62758150dc3@omnilan.de> Date: Mon, 8 Apr 2019 10:18:51 +0300 Cc: freebsd-stable@freebsd.org Content-Transfer-Encoding: quoted-printable Message-Id: <2417CFDF-9744-4600-853D-4A3759CB3137@me.com> References: <591B12C6.4040301@omnilan.de> <591B1A8B.6070803@omnilan.de> <591B1EA4.600@omnilan.de> <591B2523.6040101@omnilan.de> <7CF3AC8F-A778-40AE-B457-9B96AE5B4719@me.com> <591B284B.6070204@omnilan.de> <30f7b617-1814-ad21-a445-a62758150dc3@omnilan.de> To: Harry Schmalzbauer X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:, , definitions=2019-04-08_04:, , signatures=0 X-Proofpoint-Spam-Details: rule=notspam policy=default score=0 suspectscore=0 malwarescore=0 phishscore=0 bulkscore=0 spamscore=0 clxscore=1011 mlxscore=0 mlxlogscore=999 adultscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.0.1-1812120000 definitions=main-1904080069 X-Rspamd-Queue-Id: DF4468B9DD X-Spamd-Bar: ---- X-Spamd-Result: default: False [-4.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:17.58.0.0/16]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[me.com]; DKIM_TRACE(0.00)[me.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[me.com,quarantine]; MX_GOOD(-0.01)[mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, m x6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.co m, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud.com, mx3.mail.icloud.com, mx1.mail.icloud.com, mx6.mail.icloud.com, mx4.mail.icloud.com, mx5.mail.icloud.com, mx2.mail.icloud .com,mx3.mail.icloud.com,mx1.mail.icloud.com,mx6.mail.icloud.com,mx4.mail.icloud.com,mx5.mail.icloud.com,mx2.mail.icloud.com,mx3.mail.icloud.com,mx1.mail.icloud.com,mx6.mail.icloud.com,mx4.mail.icloud.com,mx5.mail.icloud.com,mx2.mail.icloud.com,mx3.mail.icloud.com,mx1.mail.icloud.com,mx6.mail.icloud.com,mx4.mail.icloud.com,mx5.mail.icloud.com,mx2.mail.icloud.com,mx3.mail.icloud.com,mx1.mail.icloud.com,mx6.mail.icloud.com,mx4.mail.icloud.com,mx5.mail.icloud.com,mx2.mail.icloud.com,mx3.mail.icloud.com,mx1.mail.icloud.com,mx6.mail.icloud.com,mx4.mail.icloud.com,mx5.mail.icloud.com,mx2.mail.icloud.com,mx3.mail.icloud.com]; NEURAL_HAM_SHORT(-0.53)[-0.534,0]; RCVD_IN_DNSWL_LOW(-0.10)[53.6.58.17.list.dnswl.org : 127.0.5.1]; SUBJECT_ENDS_QUESTION(1.00)[]; FREEMAIL_ENVFROM(0.00)[me.com]; ASN(0.00)[asn:714, ipnet:17.58.0.0/20, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[me.com:s=04042017]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DWL_DNSWL_LOW(-1.00)[me.com.dwl.dnswl.org : 127.0.5.1]; TO_MATCH_ENVRCPT_SOME(0.00)[]; IP_SCORE(-0.86)[ipnet: 17.58.0.0/20(-2.12), asn: 714(-2.11), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2019 07:19:11 -0000 Yes, I still do remember it, just need to find time to port the feature.=20 Of course the root cause there is much more complicated (we can not assume t= o have contigous space above 1MB), but we mostly can cope... Sent from my iPhone > On 8 Apr 2019, at 09:44, Harry Schmalzbauer wrote: >=20 >> Am 16.05.2017 um 18:26 schrieb Harry Schmalzbauer: >> Bez=C3=BCglich Toomas Soome's Nachricht vom 16.05.2017 18:20 (localtime):= >>>> On 16. mai 2017, at 19:13, Harry Schmalzbauer wrot= e: >>>>=20 >>>> Bez=C3=BCglich Toomas Soome's Nachricht vom 16.05.2017 18:00 (localtime= ): >>>>>> On 16. mai 2017, at 18:45, Harry Schmalzbauer >>>>> > wrote: >>>>>>=20 >>>>>> Bez=C3=BCglich Harry Schmalzbauer's Nachricht vom 16.05.2017 17:28 (l= ocaltime): >>>>>>> Bez=C3=BCglich Toomas Soome's Nachricht vom 16.05.2017 16:57 (localt= ime): >>>>>>>>> On 16. mai 2017, at 17:55, Harry Schmalzbauer >>>>>>>> > wrote: >>>>>>>>>=20 >>>>>>>>> Hello, >>>>>>>>>=20 >>>>>>>>> unfortunately I had some trouble with my preferred MFS-root setups= . >>>>>>>>> It seems EFI loader doesn't handle type md_image correctly. >>>>>>>>>=20 >>>>>>>>> If I load any md_image with loader invoked by gptboot or gptzfsboo= t, >>>>>>>>> 'lsmod' >>>>>>>>> shows "elf kernel", "elf obj module(s)" and "md_image". >>>>>>>>>=20 >>>>>>>>> Using the same loader.conf, but EFI loader, the md_image-file is >>>>>>>>> prompted and sems to be loaded, but not registered. There's no >>>>>>>>> md_image >>>>>>>>> with 'lsmod', hence it's not astonsihing that kernel doesn't attac= h md0 >>>>>>>>> so booting fails since there's no rootfs. >>>>>>>>>=20 >>>>>>>>> Any help highly appreciated, hope Toomas doesn't mind beeing >>>>>>>>> initially CC'd. >>>>>>>>>=20 >>>>>>>>> Thanks, >>>>>>>>>=20 >>>>>>>>> -harry >>>>>>>> The first question is, how large is the md_image and what other >>>>>>>> modules are loaded? >>>>>>> Thanks for your quick response. >>>>>>>=20 >>>>>>> The images are 50-500MB uncompressed (provided by gzip compressed fi= le). >>>>>>> Small ammount of elf modules, 5, each ~50kB. >>>>>> On the real HW, there's vmm and some more: >>>>>> Id Refs Address Size Name >>>>>> 1 46 0xffffffff80200000 16M kernel >>>>>> 2 1 0xffffffff8121d000 86K unionfs.ko >>>>>> 3 1 0xffffffff81233000 3.1M zfs.ko >>>>>> 4 2 0xffffffff81545000 51K opensolaris.ko >>>>>> 5 7 0xffffffff81552000 279K usb.ko >>>>>> 6 1 0xffffffff81598000 67K ukbd.ko >>>>>> 7 1 0xffffffff815a9000 51K umass.ko >>>>>> 8 1 0xffffffff815b6000 46K aesni.ko >>>>>> 9 1 0xffffffff815c3000 54K uhci.ko >>>>>> 10 1 0xffffffff815d1000 65K ehci.ko >>>>>> 11 1 0xffffffff815e2000 15K cc_htcp.ko >>>>>> 12 1 0xffffffff815e6000 3.4M vmm.ko >>>>>> 13 1 0xffffffffa3a21000 12K ums.ko >>>>>> 14 1 0xffffffffa3a24000 9.1K uhid.ko >>>>>>=20 >>>>>> Providing md_image uncompressed doesn't change anything. >>>>>>=20 >>>>>> Will deploy a /usr separated rootfs, which is only ~100MB uncompresse= d >>>>>> and see if that changes anything. >>>>>> That's all I can provide, code is far beyond my knowledge... >>>>>>=20 >>>>>> -harry >>>>>=20 >>>>> The issue is, that current UEFI implementation is using 64MB staging >>>>> memory for loading the kernel and modules and files. When the boot is >>>>> called, the relocation code will put the bits from staging area into t= he >>>>> final places. The BIOS version does not need such staging area, and th= at >>>>> will explain the difference. >>>>>=20 >>>>> I actually have different implementation to address the same problem, >>>>> but thats for illumos case, and will need some work to make it usable >>>>> for freebsd; the idea is actually simple - allocate staging area per >>>>> loaded file and relocate the bits into the place by component, not as >>>>> continuous large chunk (this would also allow to avoid the mines like >>>>> planted by hyperv;), but right now there is no very quick real solutio= n >>>>> other than just build efi loader with larger staging size. >>>> Ic, thanks for the explanation. >>>> While not aware about the purpose of the staging area nor the >>>> consequences of enlarging it, do you think it's feasable increasing it >>>> to 768Mib? >>>>=20 >>>> At least now I have an idea baout the issue and an explanation why >>>> reducing md_imgae to 100MB hasn't helped =E2=80=93 still more than 64..= . >>>>=20 >>>> Any quick hint where to define the staging area size highly appreciated= , >>>> fi there are no hard objections against a 768MB size. >>>>=20 >>>> -harry >>> The problem is that before UEFI Boot Services are not switched off, the m= emory is managed (and owned) by the firmware, >> Hmm, I've been expecting something like that (owend by firmware) ;-) >>=20 >> So I'll stay with CSM for now, and will happily be an early adopter if >> you need someone to try anything (-stable mergable). >=20 > Hello Toomas, >=20 > thanks for your ongoing FreeBSD commits, saw your recent libstand improvem= ents and the efiloader commit. > Which remembers me nagging the skilled ones for my unmet needs ;-) >=20 > I guess nobody had time to look at the MFS-root limitation with EFI vs. BI= OS. > If you have any news/plans, please share. > The ability to boot via EFI gives a much better console experience/usabili= ty for admins, but on MFS-root system, I'm still forced to use the old loade= r path, because of the 64MB size limit. >=20 > Do you think there's a chance that this will be resolved for FreeBSD? >=20 > Thanks, >=20 > -harry >=20 From owner-freebsd-stable@freebsd.org Mon Apr 8 07:43:03 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2D130157802A for ; Mon, 8 Apr 2019 07:43:03 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from echo.brtsvcs.net (echo.brtsvcs.net [IPv6:2607:f740:c::4ae]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D17238C8C8 for ; Mon, 8 Apr 2019 07:43:01 +0000 (UTC) (envelope-from list_freebsd@bluerosetech.com) Received: from chombo.houseloki.net (catnip [73.240.250.185]) by echo.brtsvcs.net (Postfix) with ESMTPS id 86E0538D32 for ; Mon, 8 Apr 2019 00:42:52 -0700 (PDT) Received: from [10.26.25.100] (unknown [10.26.25.100]) by chombo.houseloki.net (Postfix) with ESMTPSA id DDA3B50E5 for ; Mon, 8 Apr 2019 00:42:51 -0700 (PDT) To: freebsd-stable@freebsd.org From: Mel Pilgrim Subject: Can someone MFC usb/234503 Message-ID: <748ac1e6-53f0-53fa-c9e6-f19110dd90eb@bluerosetech.com> Date: Mon, 8 Apr 2019 00:42:49 -0700 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: D17238C8C8 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of list_freebsd@bluerosetech.com designates 2607:f740:c::4ae as permitted sender) smtp.mailfrom=list_freebsd@bluerosetech.com X-Spamd-Result: default: False [-6.75 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[bluerosetech.com]; MX_GOOD(-0.01)[echo.brtsvcs.net,foxtrot.brtsvcs.net]; NEURAL_HAM_SHORT(-0.98)[-0.981,0]; IP_SCORE(-3.46)[ip: (-8.86), ipnet: 2607:f740:c::/48(-4.47), asn: 36236(-3.89), country: US(-0.06)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:36236, ipnet:2607:f740:c::/48, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[185.250.240.73.zen.spamhaus.org : 127.0.0.10] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2019 07:43:03 -0000 The patch was committed to head on March 11 and flagged for MFC, but it hasn't be merged to stable yet. I've been running this modification on 11.2-R and 12.0-R using the affected devices as system disks for a little over two months without issue. Can someone merge this into stable/11 and stable/12? It's important (to me, at least) that it make it into 11.3-R, and the slush for it isn't far off. From owner-freebsd-stable@freebsd.org Mon Apr 8 21:28:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7A67E1569797; Mon, 8 Apr 2019 21:28:41 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from vtr.rulingia.com (vtr.rulingia.com [IPv6:2001:19f0:5801:ebe:5400:1ff:fe53:30fd]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "vtr.rulingia.com", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 7E20A8CD6A; Mon, 8 Apr 2019 21:28:40 +0000 (UTC) (envelope-from peter@rulingia.com) Received: from server.rulingia.com (ppp59-167-167-3.static.internode.on.net [59.167.167.3]) by vtr.rulingia.com (8.15.2/8.15.2) with ESMTPS id x38LSSr5048704 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK); Tue, 9 Apr 2019 07:28:34 +1000 (AEST) (envelope-from peter@rulingia.com) X-Bogosity: Ham, spamicity=0.000000 Received: from server.rulingia.com (localhost.rulingia.com [127.0.0.1]) by server.rulingia.com (8.15.2/8.15.2) with ESMTPS id x38LSM2E045124 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Tue, 9 Apr 2019 07:28:23 +1000 (AEST) (envelope-from peter@server.rulingia.com) Received: (from peter@localhost) by server.rulingia.com (8.15.2/8.15.2/Submit) id x38LSM7M045123; Tue, 9 Apr 2019 07:28:22 +1000 (AEST) (envelope-from peter) Date: Tue, 9 Apr 2019 07:28:22 +1000 From: Peter Jeremy To: tech-lists Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org Subject: Re: about zfs and ashift and changing ashift on existing zpool Message-ID: <20190408212822.GD13734@server.rulingia.com> References: <20190407153639.GA41753@rpi3.zyxst.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="jRHKVT23PllUwdXP" Content-Disposition: inline In-Reply-To: <20190407153639.GA41753@rpi3.zyxst.net> X-PGP-Key: http://www.rulingia.com/keys/peter.pgp User-Agent: Mutt/1.11.4 (2019-03-13) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 08 Apr 2019 21:28:41 -0000 --jRHKVT23PllUwdXP Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On 2019-Apr-07 16:36:40 +0100, tech-lists wrote: >storage ONLINE 0 0 0 > raidz1-0 ONLINE 0 0 0 > replacing-0 ONLINE 0 0 1.65K > ada2 ONLINE 0 0 0 > ada1 ONLINE 0 0 0 block size: 512B configured, = 4096B native > ada3 ONLINE 0 0 0 > ada4 ONLINE 0 0 0 > >What I'd like to know is: > >1. is the above situation harmful to data In general no. The only danger is that ZFS is updating the uberblock replicas at the start and end of the volume assuming 512B sectors which means you are at a higher risk or losing one of the replica sets if a power failure occurs during an uberblock update. >2. given that vfs.zfs.min_auto_ashift=3D12, why does it still say 512B > configured for ada1 which is the new disk, or.. The pool is configured with ashift=3D9. >3. does "configured" pertain to the pool, the disk, or both "configured" relates to the pool - all vdevs match the pool >4. what would be involved in making them all 4096B Rebuild the pool - backup/destroy/create/restore >5. does a 512B disk wear out faster than 4096B (all other things being > equal) It shouldn't. It does mean that the disk is doing read/modify/write at the physical sector level but that should be masked by the drive cache. >Given that the machine and disks were new in 2016, I can't understand why = zfs >didn't default to 4096B on installation I can't answer that easily. The current version of ZFS looks at the native disk blocksize to determine the pool ashift but I'm not sure how things were in 2016. Possibilities include: * The pool was built explicitly with ashift=3D9 * The initial disks reported 512B native (I think this is most likely) * That version of ZFS was using logical, rather than native blocksize. My guess (given that only ada1 is reporting a blocksize mismatch) is that your disks reported a 512B native blocksize. In the absence of any overrid= e, ZFS will then build an ashift=3D9 pool. --=20 Peter Jeremy --jRHKVT23PllUwdXP Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQKTBAEBCgB9FiEE7rKYbDBnHnTmXCJ+FqWXoOSiCzQFAlyrvPZfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEVF QjI5ODZDMzA2NzFFNzRFNjVDMjI3RTE2QTU5N0EwRTRBMjBCMzQACgkQFqWXoOSi CzQbEg//dTqzj9xBLYWErHc7ot1sH54+Mnwsgu7bYthA6E2uXIjT3lqK1+2X68Yn NStQiv9SIljbrCVlyN6aejgx1dFuSsd90DGOpPtwdFvHx8QkXIcXl0xzyef5R+7q XNfSyltWVa/DxGnH+7ve98PaQQTIfgn3WG4zn4tx69+XwwMOkhPlF6E0TC4XnST9 o+Qpv9BGzkRGYZsYy4gNMRFVhvUhoZvTys+k2euC7x9onZ4L/OnbeY6CAD/1Wj54 lZpAVgBB6Ms+lUWVvDPVtIKqA3RoDvlwefLgdJee6gnlNZ1vzQ/KBVr2DnqgQtbu xFhZB9j1tph+P184HcH8fKziYa+fudXGI9A9y1snga2hVLDSnPX1MlC0tt+4uHYs PJqCVWj9nWrw/x0B5z9nVAbLK74qS6QAe9Eodjp0p0tJ8sh/hHzHEB5llK44l8TL mlzMdgf5/PZlw8N+1TfbU8TfNWej+ImDRUa2L8n/vgU695Z2fMYSOBnqo/S3ZlKj 77Z45V05bQqUGz4XSn9VLG6jo5joibpH/gwyjr3extFWkXTEK36ZWoMKxW1ndlzX +FX5tNDhj6psntcRlDj/S5embKFFElKvyV2/CWNrIwJZvxiQ0HCTyiaW56vdflUX r91OkXSjk17hEJB8Zh95JMr/zUcYybofIak/JviHml0vHof4mJ4= =yHzn -----END PGP SIGNATURE----- --jRHKVT23PllUwdXP-- From owner-freebsd-stable@freebsd.org Tue Apr 9 00:22:31 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1861C156D972; Tue, 9 Apr 2019 00:22:31 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from hz.grosbein.net (hz.grosbein.net [IPv6:2a01:4f8:d12:604::2]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "hz.grosbein.net", Issuer "hz.grosbein.net" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id E25686B3E0; Tue, 9 Apr 2019 00:22:19 +0000 (UTC) (envelope-from eugen@grosbein.net) Received: from eg.sd.rdtc.ru (eg.sd.rdtc.ru [IPv6:2a03:3100:c:13:0:0:0:5]) by hz.grosbein.net (8.15.2/8.15.2) with ESMTPS id x390M58m037896 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT); Tue, 9 Apr 2019 02:22:09 +0200 (CEST) (envelope-from eugen@grosbein.net) X-Envelope-From: eugen@grosbein.net X-Envelope-To: kpn@neutralgood.org Received: from [10.58.0.4] ([10.58.0.4]) by eg.sd.rdtc.ru (8.15.2/8.15.2) with ESMTPS id x390M4YI051921 (version=TLSv1.2 cipher=DHE-RSA-AES128-SHA bits=128 verify=NOT); Tue, 9 Apr 2019 07:22:04 +0700 (+07) (envelope-from eugen@grosbein.net) Subject: Re: about zfs and ashift and changing ashift on existing zpool To: "Kevin P. Neal" , Peter Jeremy References: <20190407153639.GA41753@rpi3.zyxst.net> <20190408212822.GD13734@server.rulingia.com> <20190409000009.GA65388@neutralgood.org> Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, tech-lists , freebsd-geom@freebsd.org, Alexander Motin From: Eugene Grosbein Message-ID: <9590cb82-64be-a2f9-a812-36f0ea324e4d@grosbein.net> Date: Tue, 9 Apr 2019 07:21:57 +0700 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <20190409000009.GA65388@neutralgood.org> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit X-Spam-Status: No, score=0.3 required=5.0 tests=BAYES_00,LOCAL_FROM,SPF_PASS autolearn=no autolearn_force=no version=3.4.2 X-Spam-Report: * -2.3 BAYES_00 BODY: Bayes spam probability is 0 to 1% * [score: 0.0000] * -0.0 SPF_PASS SPF: sender matches SPF record * 2.6 LOCAL_FROM From my domains X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on hz.grosbein.net X-Rspamd-Queue-Id: E25686B3E0 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; spf=permerror (mx1.freebsd.org: domain of eugen@grosbein.net uses mechanism not recognized by this client) smtp.mailfrom=eugen@grosbein.net X-Spamd-Result: default: False [-3.55 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; MX_INVALID(0.50)[greylisted]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[grosbein.net]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; R_SPF_PERMFAIL(0.00)[]; NEURAL_HAM_SHORT(-0.82)[-0.822,0]; RCPT_COUNT_SEVEN(0.00)[7]; IP_SCORE(-1.13)[ip: (-1.48), ipnet: 2a01:4f8::/29(-2.16), asn: 24940(-2.01), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:24940, ipnet:2a01:4f8::/29, country:DE]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 00:22:31 -0000 09.04.2019 7:00, Kevin P. Neal wrote: >> My guess (given that only ada1 is reporting a blocksize mismatch) is that >> your disks reported a 512B native blocksize. In the absence of any override, >> ZFS will then build an ashift=9 pool. [skip] > smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.2-RELEASE-p4 amd64] (local build) > Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org > > === START OF INFORMATION SECTION === > Vendor: SEAGATE > Product: ST2400MM0129 > Revision: C003 > Compliance: SPC-4 > User Capacity: 2,400,476,553,216 bytes [2.40 TB] > Logical block size: 512 bytes > Physical block size: 4096 bytes Maybe it't time to prefer "Physical block size" over "Logical block size" in relevant GEOMs like GEOM_DISK, so upper levels such as ZFS would do the right thing automatically. From owner-freebsd-stable@freebsd.org Tue Apr 9 00:56:03 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 69C63156E5F1; Tue, 9 Apr 2019 00:56:03 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: from mail-oi1-x22e.google.com (mail-oi1-x22e.google.com [IPv6:2607:f8b0:4864:20::22e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 899D16C811; Tue, 9 Apr 2019 00:56:02 +0000 (UTC) (envelope-from mavbsd@gmail.com) Received: by mail-oi1-x22e.google.com with SMTP id e5so12156013oii.0; Mon, 08 Apr 2019 17:56:02 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:subject:to:cc:references:from:openpgp:autocrypt:message-id :date:user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=pS0tp7ULDb5hmpCqEAmUjvYnnznYgybUw5Nmo28ViGY=; b=IR/iukTmsXrsCGfBSgUSld2DfCefYtOxh2kqbtBCn9hbEX16bEuKInQCccodC4pvtI AyutHssSIzGXlExW4C20zMTIHFXdbDyu2Yyh1bCidIFGkreYZ7JwYZ4BzIZ+/OyqDxxq JmFH0a/mkf8CTWJ+Nq939mt1lQ+UOryUuHT9whX1VsG4khH6igb4dLLA9QW2QA/dObwf iTdTJ81fsh5sW5wUOsxDVQ0TdKCxZ/GosuzvdiAP3QGp9vQCD+Sr+9ao8fIYkuDhBEpU CrCFVsIo7eOe3LrWsLUPRtkn1lrcrTqppiLdiBzAAuJZUA+gkEeQbGIZliWbqw4cbDCl oGSQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:subject:to:cc:references:from:openpgp :autocrypt:message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=pS0tp7ULDb5hmpCqEAmUjvYnnznYgybUw5Nmo28ViGY=; b=lSrxEjXhcNLnpvugBrSsGzoyHYB94jBM90fWdAS2kyVwcDM/hs8oIm24vgNDQ5Ii0m 6KUFSlY56AzKZ2f9/QJpXmNT5P0LtflX97iSSouXsC4hOdQ6xFAMIjuBIdCfyyv0qY3k V5TViyGerLDSnZ2mkqBdmPbh0PX4xMGEJgDV1thaPkw7EYL5BLUAevoHFl4/VwMcmibU Ahf600tiMHaEufOWhkpeEjEKOKX1hHSP2DNXWt61A9FqfkFLP52a1I0c5KA1DHEJOG33 B1ckhea+j36vwppHRzcO+uvDL2nwpFFDpRUGz5WJ/Y316E18gpPbUlxZVgmlvGh3iAK8 EAcQ== X-Gm-Message-State: APjAAAX5xxq2q+Gq0D/h5FsY19mn40PS67V37jx9HdOvDUC81RQUAbC0 va9GxGX9xbuSvfDNmDLI7lN7fnwV X-Google-Smtp-Source: APXvYqwJ2mJaH96f5FaMB6bTKpeuo+1/nJh3KlXc41DMxlDY2wrCEmo817F5qLqXD1CzzSxFOeN8Kg== X-Received: by 2002:aca:3504:: with SMTP id c4mr17896736oia.74.1554771361187; Mon, 08 Apr 2019 17:56:01 -0700 (PDT) Received: from spectre.mavhome.dp.ua ([2600:1700:3580:3560:228:f8ff:fe04:d12]) by smtp.gmail.com with ESMTPSA id 96sm13272087otf.17.2019.04.08.17.56.00 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Mon, 08 Apr 2019 17:56:00 -0700 (PDT) Sender: Alexander Motin Subject: Re: about zfs and ashift and changing ashift on existing zpool To: Eugene Grosbein , "Kevin P. Neal" , Peter Jeremy Cc: freebsd-fs@freebsd.org, freebsd-stable@freebsd.org, tech-lists , freebsd-geom@freebsd.org References: <20190407153639.GA41753@rpi3.zyxst.net> <20190408212822.GD13734@server.rulingia.com> <20190409000009.GA65388@neutralgood.org> <9590cb82-64be-a2f9-a812-36f0ea324e4d@grosbein.net> From: Alexander Motin Openpgp: preference=signencrypt Autocrypt: addr=mav@FreeBSD.org; prefer-encrypt=mutual; keydata= mQENBFOzxAwBCADkPrax0pI2W/ig0CK9nRJJwsHitAGEZ2HZiFEuti+6/4UVxj81yr4ak/4g 9bKUyC7rMEAp/ZHNhd+MFCPAAcHPvtovnfykqE/vuosCS3wlSLloix2iKVLks0CwbLHGAyne 46lTQW74Xl/33c3W1Z6d8jD9gVFT/xaVzZ0U9xdzOmsYAZaAj4ki0tuxO9F7L+ct9grRe7iP g8t9hai7BL4ee3VRwk2JXnKb7UvBiVITKYWKz1jRvZIrjPokgEcCLOSlv7x/1kjuFnj3xWZU 7HSFFT8J93epBbrSSCsYsppIk2fZH41kaaFXsMQfTPH8wkeM6qwrvOh4HiQM08R+9tThABEB AAG0IUFsZXhhbmRlciBNb3RpbiA8bWF2QEZyZWVCU0Qub3JnPokBVwQTAQoAQQIbAwULCQgH AwUVCgkICwUWAwIBAAIeAQIXgAIZARYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMKuBQkN McyiAAoJEIMYw5VbqyJ/tuUIAOG3ONOSNYqjK4eTZ1TVh9jdUBAhWk5nhDFnODN49Wj0AbYm 7aIqy8O1hnCDSZG5LttjSAo3UfXJZDKQM0BLb0gpRMBnAYqO6tdolLNqAbPGJBnGoPjsh24y 6KcbDaNnis+lD4GwPXwQM+92wZGhCUFElPV9NciZGVS65TNIgk7X+yEjjhD1MSWKKijZ1r9Z zIt4OzUTxxNOvzdlABZS88nNRdJkatOQJPmFdd1mpP6UzTNCiLUo1pIqOEtJgvVVDYq5WHY6 tciWWYdmZG/tIBexJmv2mV2OLVjXR6ZeKmntVH14H72/wRHJuYHQC+r5SVRcWWayrThsY6jZ Yr4+raS5AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6Z AXgDtmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8Flv mI/c40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt 3ytU8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZ R1EdEIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm5 9R8AEQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczM AAoJEIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLq A6xe6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHu uC5vgPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15Gc sS9YcQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9 TevwGsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCg lz65AQ0EU7PEDAEIAOZgWf2cJIu+58IzP2dkXE/urj3tr4OqrB/yHGWUf71Lz6D0Fi6ZAXgD tmcFLGPfMyWuLAvSM+xmoguk7zC4hRBYvQycmIhuqBq1jO1Wp/Z+lpoPM/1cDYLn8FlvmI/c 40MhUZh345DA4jYWWaZNjQHUWVQ1fPf595vdVVMPT/abE8E5DaF6fSkRmqFTmfYRkfbt3ytU 8NdUapDcJVY7cEP2nJBVNZPnOIObR/ZIgSxjjrG5o34yXoqeup8JvwEv+/NylzzuyXEZR1Ed EIzQ/a1nh/0j4NXtzZEqKW4aTWlmSqb6wN8jh1OSOOqkYsfnE3nfxcZbxi4IRoNQYlm59R8A EQEAAYkBPAQYAQoAJgIbDBYhBOmM88TmnMPNDledVYMYw5VbqyJ/BQJZYMLYBQkNMczMAAoJ EIMYw5VbqyJ/TqgH/RQHClkvecE0262lwKoP/m0Mh4I5TLRgoJJn8S7G1BnqohYJkiLqA6xe 6urGD7OqdNAl12UbrjWbdJV+zvea3vJoM4MZuYiYrGaXWxzFXqWJcPwMU9sAh8MRghHuuC5v gPb45Tnftw9/+n0i8GfVhQhOqepUGdQg4NPcXviSkoAvig6pp9Lcxisn0groUQKt15GcsS9Y cQWg3j9Hnipc6Mu416HX98Fb113NHJqc2geTHLkRyuBFOoyIqB6N9GKjzOAIzxxsVdl9Tevw Gsrp4M4/RFzWbSgsbOnbE7454lmuVZGfReEjnUm8RHp9Q2UWKXlp3exlZjvOp/uVEpCglz4= Message-ID: Date: Mon, 8 Apr 2019 20:55:59 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.4.0 MIME-Version: 1.0 In-Reply-To: <9590cb82-64be-a2f9-a812-36f0ea324e4d@grosbein.net> Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 899D16C811 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=IR/iukTm; spf=pass (mx1.freebsd.org: domain of mavbsd@gmail.com designates 2607:f8b0:4864:20::22e as permitted sender) smtp.mailfrom=mavbsd@gmail.com X-Spamd-Result: default: False [-5.96 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_SHORT(-0.99)[-0.994,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_SEVEN(0.00)[7]; RCVD_IN_DNSWL_NONE(0.00)[e.2.2.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.75)[ip: (-8.57), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)]; FORGED_SENDER(0.30)[mav@FreeBSD.org,mavbsd@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[mav@FreeBSD.org,mavbsd@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 00:56:03 -0000 On 08.04.2019 20:21, Eugene Grosbein wrote: > 09.04.2019 7:00, Kevin P. Neal wrote: > >>> My guess (given that only ada1 is reporting a blocksize mismatch) is that >>> your disks reported a 512B native blocksize. In the absence of any override, >>> ZFS will then build an ashift=9 pool. > > [skip] > >> smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.2-RELEASE-p4 amd64] (local build) >> Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org >> >> === START OF INFORMATION SECTION === >> Vendor: SEAGATE >> Product: ST2400MM0129 >> Revision: C003 >> Compliance: SPC-4 >> User Capacity: 2,400,476,553,216 bytes [2.40 TB] >> Logical block size: 512 bytes >> Physical block size: 4096 bytes > > Maybe it't time to prefer "Physical block size" over "Logical block size" in relevant GEOMs > like GEOM_DISK, so upper levels such as ZFS would do the right thing automatically. No. It is a bad idea. Changing logical block size for existing disks will most likely result in breaking compatibility and inability to read previously written data. ZFS already uses physical block size when possible -- on pool creation or new vdev addition. When not possible (pool already created wrong) it just complains about it, so that user would know that his configuration is imperfect and he should not expect full performance. -- Alexander Motin From owner-freebsd-stable@freebsd.org Tue Apr 9 01:25:56 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9F336156F903; Tue, 9 Apr 2019 01:25:56 +0000 (UTC) (envelope-from imb@protected-networks.net) Received: from mail.protected-networks.net (mail.protected-networks.net [202.12.127.228]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.protected-networks.net", Issuer "Protected Networks CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id A66276DBC7; Tue, 9 Apr 2019 01:25:53 +0000 (UTC) (envelope-from imb@protected-networks.net) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= protected-networks.net; h=content-transfer-encoding :content-language:content-type:content-type:in-reply-to :mime-version:user-agent:date:date:message-id:from:from :references:subject:subject; s=201508; t=1554773144; bh=zK4gi2f4 X8bCWOeWwieqF63chhx0O+OD8z6e5dGhkDk=; b=mZLU7tL0+ZEHm8EOpdKJ6Nti RBC+CVDJzHXkUqthnI2hZECYa/09Myby4RBcQnPGJ/9nx7UWAhUo+HIDRaH/bz8l LVTZUheSMnCWRp2YIO/1/55KBCm4u2IEYOttHrYpAuOg++xCZ51ep1smWTZqzWkV MQxUbdoD/2XBqn274cs= Received: from toshi.auburn.protected-networks.net (toshi.auburn.protected-networks.net [192.168.1.10]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: imb@mail.protected-networks.net) by mail.protected-networks.net (Postfix) with ESMTPSA id 7D15F1285F; Mon, 8 Apr 2019 21:25:44 -0400 (EDT) Subject: Re: about zfs and ashift and changing ashift on existing zpool To: Alexander Motin , Eugene Grosbein , "Kevin P. Neal" , Peter Jeremy Cc: freebsd-fs@freebsd.org, freebsd-geom@freebsd.org, freebsd-stable@freebsd.org, tech-lists References: <20190407153639.GA41753@rpi3.zyxst.net> <20190408212822.GD13734@server.rulingia.com> <20190409000009.GA65388@neutralgood.org> <9590cb82-64be-a2f9-a812-36f0ea324e4d@grosbein.net> From: Michael Butler Openpgp: preference=signencrypt Message-ID: Date: Mon, 8 Apr 2019 21:25:43 -0400 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: A66276DBC7 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=protected-networks.net header.s=201508 header.b=mZLU7tL0; spf=pass (mx1.freebsd.org: domain of imb@protected-networks.net designates 202.12.127.228 as permitted sender) smtp.mailfrom=imb@protected-networks.net X-Spamd-Result: default: False [-1.83 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[protected-networks.net:s=201508]; NEURAL_HAM_MEDIUM(-0.85)[-0.853,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+mx]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[protected-networks.net]; NEURAL_SPAM_SHORT(0.54)[0.542,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[protected-networks.net:+]; MX_GOOD(-0.01)[sarah.protected-networks.net,mail.protected-networks.net]; RCPT_COUNT_SEVEN(0.00)[8]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; IP_SCORE(-0.01)[country: US(-0.06)]; ASN(0.00)[asn:5716, ipnet:202.12.127.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 01:25:56 -0000 On 2019-04-08 20:55, Alexander Motin wrote: > On 08.04.2019 20:21, Eugene Grosbein wrote: >> 09.04.2019 7:00, Kevin P. Neal wrote: >> >>>> My guess (given that only ada1 is reporting a blocksize mismatch) is that >>>> your disks reported a 512B native blocksize. In the absence of any override, >>>> ZFS will then build an ashift=9 pool. >> >> [skip] >> >>> smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.2-RELEASE-p4 amd64] (local build) >>> Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org >>> >>> === START OF INFORMATION SECTION === >>> Vendor: SEAGATE >>> Product: ST2400MM0129 >>> Revision: C003 >>> Compliance: SPC-4 >>> User Capacity: 2,400,476,553,216 bytes [2.40 TB] >>> Logical block size: 512 bytes >>> Physical block size: 4096 bytes >> >> Maybe it't time to prefer "Physical block size" over "Logical block size" in relevant GEOMs >> like GEOM_DISK, so upper levels such as ZFS would do the right thing automatically. > > No. It is a bad idea. Changing logical block size for existing disks > will most likely result in breaking compatibility and inability to read > previously written data. ZFS already uses physical block size when > possible -- on pool creation or new vdev addition. When not possible > (pool already created wrong) it just complains about it, so that user > would know that his configuration is imperfect and he should not expect > full performance. And some drives just present 512 bytes for both .. no idea if this is consistent with the underlying silicon :-( I built a ZFS pool on it using 4k blocks anyway. smartctl 7.0 2018-12-30 r4883 [FreeBSD 13.0-CURRENT amd64] (local build) Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WDS100T2B0A-00SM50 Serial Number: 1837B0803409 LU WWN Device Id: 5 001b44 8b99f7560 Firmware Version: X61190WD User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: Solid State Device Form Factor: 2.5 inches Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Mon Apr 8 21:22:15 2019 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled AAM feature is: Unavailable APM level is: 128 (minimum power consumption without standby) Rd look-ahead is: Enabled Write cache is: Enabled DSN feature is: Unavailable ATA Security is: Disabled, frozen [SEC2] Wt Cache Reorder: Unavailable imb From owner-freebsd-stable@freebsd.org Tue Apr 9 11:33:10 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B4F20157E016 for ; Tue, 9 Apr 2019 11:33:10 +0000 (UTC) (envelope-from mach@swishmail.com) Received: from vorlon.swishmail.com (vorlon.swishmail.com [208.72.56.19]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6E959897DE for ; Tue, 9 Apr 2019 11:33:09 +0000 (UTC) (envelope-from mach@swishmail.com) Received: (qmail 15244 invoked by uid 89); 9 Apr 2019 11:33:02 -0000 Received: from unknown (HELO ?IPv6:2001:b030:14e:100:959b:801a:9ec8:14a1?) (mach@swishmail.com@2001:b030:14e:100:959b:801a:9ec8:14a1) by 2602:ffb8::208:72:56:19 with ESMTPSA (ECDHE-RSA-AES128-GCM-SHA256 encrypted, authenticated); 9 Apr 2019 11:33:02 -0000 Subject: Re: em performs worse than igb (latency wise) in 12? To: freebsd-stable@freebsd.org References: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> <4f9b9259-f5a1-ecc6-366e-4a26de0ca3dc@protected-networks.net> From: Kris von Mach Message-ID: <2af4d1fe-1581-baa7-e38e-bba631639deb@swishmail.com> Date: Tue, 9 Apr 2019 19:33:00 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Content-Language: en-US X-Rspamd-Queue-Id: 6E959897DE X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of mach@swishmail.com designates 208.72.56.19 as permitted sender) smtp.mailfrom=mach@swishmail.com X-Spamd-Result: default: False [1.29 / 15.00]; ARC_NA(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:208.72.56.0/22]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; TO_DN_NONE(0.00)[]; NEURAL_SPAM_MEDIUM(0.44)[0.444,0]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-0.30)[-0.305,0]; RCVD_COUNT_THREE(0.00)[3]; DMARC_NA(0.00)[swishmail.com]; NEURAL_SPAM_SHORT(0.57)[0.571,0]; MX_GOOD(-0.01)[mxfilter1.nyc.swishmail.com,mxfilter2.nyc.swishmail.com]; IP_SCORE(-0.01)[country: US(-0.06)]; RCVD_IN_DNSWL_LOW(-0.10)[19.56.72.208.list.dnswl.org : 127.0.5.1]; R_DKIM_NA(0.00)[]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:14469, ipnet:208.72.56.0/22, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_EQ_ENVFROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 11:33:10 -0000 On 4/7/2019 6:49 AM, Matthew Macy wrote: > On Sat, Apr 6, 2019 at 1:23 PM Michael Butler > wrote: >> I'd be interested to see if substituting the port net/intel-em-kmod >> has any effect on the issue, > I would as well. igb, em, and lem are all the same driver in 12. This > makes maintenance a lot easier. However, the older NICs have a lot of > errata workarounds that aren't explicitly commented as such. My first > guess is this card suffers from one such errata workaround that has > been dropped in the update. I've tried net/intel-em-kmod, it actually became worse went from 100 requests/sec to about 90. That makes sense about maintenance. Though I believe i350 is less than 5 years old, HP's 366FLR version is 4. So it's not that old, and at least in gigabit level nic, is one of the best afaik. Is there some other 1gig nic that is recommended for 12? Or is it time to switch to 10gbit? I've heard good things about Chelsio for 10gbit. I went back to 11-Stable for now. From owner-freebsd-stable@freebsd.org Tue Apr 9 13:49:00 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 91A9615818F1; Tue, 9 Apr 2019 13:49:00 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from out2-smtp.messagingengine.com (out2-smtp.messagingengine.com [66.111.4.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 6465B8E0A1; Tue, 9 Apr 2019 13:48:58 +0000 (UTC) (envelope-from tech-lists@zyxst.net) Received: from compute4.internal (compute4.nyi.internal [10.202.2.44]) by mailout.nyi.internal (Postfix) with ESMTP id 0E18E267F3; Tue, 9 Apr 2019 09:48:52 -0400 (EDT) Received: from mailfrontend1 ([10.202.2.162]) by compute4.internal (MEProxy); Tue, 09 Apr 2019 09:48:52 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=zyxst.net; h= date:from:to:cc:subject:message-id:references:mime-version :content-type:in-reply-to; s=fm2; bh=mtpK4gTSDshRbqxO5kmtZ2cf79p Yp/tAW7woEtuvG0M=; b=qT0Tq7+I4hxZHZo2SRy48vFO+Zcz3yc07UQi5r8KhKI XWzW4vjvJAx4wDG10nEn7Qnfa0jq/MLyoTC9V/yufHYWryJ07Gk0Sr/oapCMAdBa HX6uEKXjWVnFgmic2Pderx9bVjHgrMc8d4aLn0rGvFhbe3VCFAD9Hv4lHB0vX+IY F28OlYHpUXlAihHXGrjABz3CB/7hU67NwfDunGsCBRUuYBBNWgRjhRVhdiwB0KOb bLghsXOPPFzmZJPOO982mnM+cA++i6mvPcf2l4AXgBP0mVSakZpXVHEzL+IoaaWN g7tOmE8TVgHLo6VaIaPds3X1ezQahi9i4FIKoAzdxZg== DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-type:date:from:in-reply-to :message-id:mime-version:references:subject:to:x-me-proxy :x-me-proxy:x-me-sender:x-me-sender:x-sasl-enc; s=fm2; bh=mtpK4g TSDshRbqxO5kmtZ2cf79pYp/tAW7woEtuvG0M=; b=ItuIyy8JjTl5Ap9gVp7ahF /QMKsMv95DOebF1egi92XxtJHWIOuliVft5HPGe9P4K5nRiZpnSAkI0+Vi9rCM91 uwZr3VT++z80DdmIgsoM70hsU5PmoKdbws8FTY02H7sUvk7T7Wln5iDgJ1Bh38Xg 2Nfx4bg1FHZysJSPNkERzTbFsQJZGz8AiV9CUrjVEag8GCHmznqEiOzCjsiO76D8 TnDKlVNGY1IpM3fPcfAGXqKS05liEcd7dnkzvkzH4GRwI3p9GtjGpVMKh0YuyU+A eTT4662yyXHSwiSzRTxYq2o0hfhz9y4X4AKUs0dJmtXuTFtHImTbeW0anPS2WXFQ == X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgeduuddrudehgdeilecutefuodetggdotefrodftvf curfhrohhfihhlvgemucfhrghsthforghilhdpqfgfvfdpuffrtefokffrpgfnqfghnecu uegrihhlohhuthemuceftddtnecunecujfgurhepfffhvffukfhfgggtuggjfgesghdtre ertdervdenucfhrhhomhepthgvtghhqdhlihhsthhsuceothgvtghhqdhlihhsthhsseii hiigshhtrdhnvghtqeenucffohhmrghinhepshhmrghrthhmohhnthhoohhlshdrohhrgh enucfkphepkedvrdejtddrledurddutddunecurfgrrhgrmhepmhgrihhlfhhrohhmpeht vggthhdqlhhishhtshesiiihgihsthdrnhgvthenucevlhhushhtvghrufhiiigvpedt X-ME-Proxy: Received: from rpi3.zyxst.net (rpi3.zyxst.net [82.70.91.101]) by mail.messagingengine.com (Postfix) with ESMTPA id 402E1E4519; Tue, 9 Apr 2019 09:48:50 -0400 (EDT) Date: Tue, 9 Apr 2019 14:48:47 +0100 From: tech-lists To: freebsd-stable@freebsd.org Cc: freebsd-fs@freebsd.org Subject: Re: about zfs and ashift and changing ashift on existing zpool Message-ID: <20190409134847.GA57588@rpi3.zyxst.net> References: <20190407153639.GA41753@rpi3.zyxst.net> <20190408212822.GD13734@server.rulingia.com> <20190409000009.GA65388@neutralgood.org> <9590cb82-64be-a2f9-a812-36f0ea324e4d@grosbein.net> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="uAKRQypu60I7Lcqm" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.11.4 (2019-03-13) X-Rspamd-Queue-Id: 6465B8E0A1 X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=zyxst.net header.s=fm2 header.b=qT0Tq7+I; dkim=pass header.d=messagingengine.com header.s=fm2 header.b=ItuIyy8J; spf=pass (mx1.freebsd.org: domain of tech-lists@zyxst.net designates 66.111.4.26 as permitted sender) smtp.mailfrom=tech-lists@zyxst.net X-Spamd-Result: default: False [-9.10 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_DKIM_ALLOW(-0.20)[zyxst.net:s=fm2,messagingengine.com:s=fm2]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:66.111.4.26]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; TO_DN_NONE(0.00)[]; DMARC_NA(0.00)[zyxst.net]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[4]; DKIM_TRACE(0.00)[zyxst.net:+,messagingengine.com:+]; RCPT_COUNT_TWO(0.00)[2]; MX_GOOD(-0.01)[in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com,in2-smtp.messagingengine.com,in1-smtp.messagingengine.com]; SIGNED_PGP(-2.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.997,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:11403, ipnet:66.111.4.0/24, country:US]; IP_SCORE(-3.39)[ip: (-8.97), ipnet: 66.111.4.0/24(-4.58), asn: 11403(-3.35), country: US(-0.06)]; RCVD_IN_DNSWL_LOW(-0.10)[26.4.111.66.list.dnswl.org : 127.0.5.1] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 13:49:00 -0000 --uAKRQypu60I7Lcqm Content-Type: text/plain; charset=us-ascii; format=flowed Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Mon, Apr 08, 2019 at 09:25:43PM -0400, Michael Butler wrote: >On 2019-04-08 20:55, Alexander Motin wrote: >> On 08.04.2019 20:21, Eugene Grosbein wrote: >>> 09.04.2019 7:00, Kevin P. Neal wrote: >>> >>>>> My guess (given that only ada1 is reporting a blocksize mismatch) is = that >>>>> your disks reported a 512B native blocksize. In the absence of any o= verride, >>>>> ZFS will then build an ashift=3D9 pool. >>> >>> [skip] >>> >>>> smartctl 7.0 2018-12-30 r4883 [FreeBSD 11.2-RELEASE-p4 amd64] (local b= uild) >>>> Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontool= s.org >>>> >>>> =3D=3D=3D START OF INFORMATION SECTION =3D=3D=3D >>>> Vendor: SEAGATE >>>> Product: ST2400MM0129 >>>> Revision: C003 >>>> Compliance: SPC-4 >>>> User Capacity: 2,400,476,553,216 bytes [2.40 TB] >>>> Logical block size: 512 bytes >>>> Physical block size: 4096 bytes >>> >>> Maybe it't time to prefer "Physical block size" over "Logical block siz= e" in relevant GEOMs >>> like GEOM_DISK, so upper levels such as ZFS would do the right thing au= tomatically. >> >> No. It is a bad idea. Changing logical block size for existing disks >> will most likely result in breaking compatibility and inability to read >> previously written data. ZFS already uses physical block size when >> possible -- on pool creation or new vdev addition. When not possible >> (pool already created wrong) it just complains about it, so that user >> would know that his configuration is imperfect and he should not expect >> full performance. > >And some drives just present 512 bytes for both .. no idea if this is >consistent with the underlying silicon :-( I built a ZFS pool on it >using 4k blocks anyway. > >smartctl 7.0 2018-12-30 r4883 [FreeBSD 13.0-CURRENT amd64] (local build) >Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org > >=3D=3D=3D START OF INFORMATION SECTION =3D=3D=3D >Device Model: WDC WDS100T2B0A-00SM50 >Serial Number: 1837B0803409 >LU WWN Device Id: 5 001b44 8b99f7560 >Firmware Version: X61190WD >User Capacity: 1,000,204,886,016 bytes [1.00 TB] >Sector Size: 512 bytes logical/physical >Rotation Rate: Solid State Device >Form Factor: 2.5 inches >Device is: Not in smartctl database [for details use: -P showall] >ATA Version is: ACS-4 T13/BSR INCITS 529 revision 5 >SATA Version is: SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s) >Local Time is: Mon Apr 8 21:22:15 2019 EDT >SMART support is: Available - device has SMART capability. >SMART support is: Enabled >AAM feature is: Unavailable >APM level is: 128 (minimum power consumption without standby) >Rd look-ahead is: Enabled >Write cache is: Enabled >DSN feature is: Unavailable >ATA Security is: Disabled, frozen [SEC2] >Wt Cache Reorder: Unavailable Yeah it's weird isn't it. So it seems it's not an issue with zfs at all as far as I can see. This is one of the drives that was replaced, and it's identical to the other two making up the array. So not unreasonably ashift was 9, as all three drives making up the array were 512 logical/physical. =3D=3D=3D START OF INFORMATION SECTION =3D=3D=3D Model Family: Western Digital Black Device Model: WDC WD4001FAEX-00MJRA0 Firmware Version: 01.01L01 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Apr 9 12:47:01 2019 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled I replaced one of them with an 8tb drive: TART OF INFORMATION SECTION =3D=3D=3D Model Family: Seagate Archive HDD Device Model: ST8000AS0002-1NA17Z Firmware Version: AR13 User Capacity: 8,001,563,222,016 bytes [8.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5980 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s) Local Time is: Tue Apr 9 12:55:55 2019 BST SMART support is: Available - device has SMART capability. SMART support is: Enabled so the 2nd drive is emulating 512. But ZFS seems to see through that and correctly determines it's a 4k drive. In any case, the fix was to make a new pool (which automatically set ashift to 12 when the 8Tb disk was added) then zfs send from the old pool to the new one, then destroy the old pool. Fortunately this was easy because the system had zfs installed as an afterthought. So no root-on-zfs. The OS is on a SSD. All I can say is that zpool performance of a 4k drive in an a9 zpool is non-ideal. The new pool feels quicker (even though the disks aren't built for speed), and I've learned something new :D --=20 J. --uAKRQypu60I7Lcqm Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQIzBAABCAAdFiEE8n3tWhxW11Ccvv9/s8o7QhFzNAUFAlysorQACgkQs8o7QhFz NAV/Gg/+I5X1JnOvyhBH/oWzvCGEHDfl0eoFHAAr19RAthGTpzXj/1n30c4Eu+HN spg/jkGa562PyNEg7N5/65Il+vSXNISDKvBXAbkY8oO0V5OjRyPEkOu2HxfLjZKl t9J4CMgYDgeVJgHJY2hfXUD68tLIgvmRIkCu97IR91W6HAesQToymk2ZAXIziI8m Zd2Q1JYnQjY8elsqRGp3S4NiCtxqDpiQfEz5xNCEX/sqN2E68vsQ3q0OadInnb2V ILreKKz4bqZeocU+VFxyLi1lHe7AGLe2rzXH3Fjd2Xd6xItXAGRBfPGpW5tevD2U cm2YRWIzCTfC4UN1PvepWgrh4YOS5luQGdRU4oM9p1HgmvdB6VZ4OKdKKu9Oc9Rt roC4oYZOjPGKkryQvE2SmhK4XvpExrHDEpz6VcxSoXWMVOfA5B049ZFVMd5xsHvQ wv+kMsxQsREkev7e58g+UqKI3LHwzStMJkOUFq44EOarlPiUfeciW7ODRoolRYCA splmuJBB7ugoFKBtTo6JVdDlpczhTk43zU8kX3CvvbC4We01KYWXrjynXGWUq3S8 1+NLb26Y0oYLzReTm5ewolCDK9UO17jAO6KnhIvSGkRyQNNcSpWJ7fvtLAyJP2YQ ai7QRSWpkl6R/JvK7U3Vm9t5YvYyKuOQZMLbECEu+nTrxugcvgw= =2AUJ -----END PGP SIGNATURE----- --uAKRQypu60I7Lcqm-- From owner-freebsd-stable@freebsd.org Tue Apr 9 18:19:31 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 6A02E1568263 for ; Tue, 9 Apr 2019 18:19:31 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: from mail-yw1-xc33.google.com (mail-yw1-xc33.google.com [IPv6:2607:f8b0:4864:20::c33]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2061171C20 for ; Tue, 9 Apr 2019 18:19:30 +0000 (UTC) (envelope-from ncrogers@gmail.com) Received: by mail-yw1-xc33.google.com with SMTP id l15so6587651ywe.8 for ; Tue, 09 Apr 2019 11:19:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=IZk2Ay+1fj8d+KC2u9CqBQloU1smT50kr0NQwCjppUQ=; b=DSoZFBoogw9eKTqEDRhXkOrUF1LLr4yympdGo3nKgQSeM5bAbg8zpQkg4xrrk+3Jgj bjOu9bzaVZrdXxdHiv9Qa+U2URFd8FcRNvfFTRscfESafDPPYjiGglI7ugH8KSZbjNcy ptjUoGpEy3JWKpHf57SwjH2FnbFVOXNSmgkVSvvB4hfEsQxcJH0ueKplP1zIwgyC/u3c +AgxdjpMYODCoR4b1W7TouS4vwo0Oni3Wv5xKoUTLWXEOvFhimR33uKcUw/dGOF9b2vd MP7zvRTJZcKIYzBOk5jCJRe7Y5q76YEfAHd2Kx7cO7pF6ObVmeEttkFmzRab+ljSdQrJ iifQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=IZk2Ay+1fj8d+KC2u9CqBQloU1smT50kr0NQwCjppUQ=; b=TgI341HR5VJxoaV0MWx6TQv4V8x3pTMDSmMc7n9CGuGLT2Usc9vxUA2q0swnBDZrg4 7qK3mwh/aU8x9HlbLIRxGp0kK5WnVeB190VgkkgeSYEyi5G9Mfj+alyJcnK4DFx1MW1c lByKaaCmzJtK1bPnC6lSln78R14uxldgZRZN6HQ59LDHx6KPkNZIPPS9MI3BhoclKAZQ P4MgdffldxoRqxl6InRRb9LWe0dS2GmNVr1wbZM8KP5Ygzt1M4jI1HrH5XP2GIBe9kLG WBUeo82qFfWeN6VMN+X+U+Psl4gWXFY1PjCyilqfc5e1T0HVlzD1d+h3Y8F+yB4W5MBi aHew== X-Gm-Message-State: APjAAAUUUOqkznNWirs5rtk3ppO+/UOy1VJhsuVbcGGKl+9zwt1zfHZ3 d4CcmupwwQWic4OLFRvIHc8PzcMjeNXiUMfj2y8= X-Google-Smtp-Source: APXvYqyPcO1CSSc/HU1JYiD/nslZOjxHQBqGnDU7OMmrBaXc3i8+4/z/xYCB5l6u1w2JoJivnFlRv2B//idrBWYeMpY= X-Received: by 2002:a81:9914:: with SMTP id q20mr30652720ywg.35.1554833969331; Tue, 09 Apr 2019 11:19:29 -0700 (PDT) MIME-Version: 1.0 References: <7673edad-1e50-7e9b-961e-f28ab7a0f41e@ingresso.co.uk> <81e63eaf-352d-4b29-a4c0-9085e699931e@menhennitt.com.au> In-Reply-To: <81e63eaf-352d-4b29-a4c0-9085e699931e@menhennitt.com.au> From: Nick Rogers Date: Tue, 9 Apr 2019 14:19:18 -0400 Message-ID: Subject: Re: em performs worse than igb (latency wise) in 12? To: Graham Menhennitt Cc: FreeBSD STABLE X-Rspamd-Queue-Id: 2061171C20 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=DSoZFBoo; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ncrogers@gmail.com designates 2607:f8b0:4864:20::c33 as permitted sender) smtp.mailfrom=ncrogers@gmail.com X-Spamd-Result: default: False [-5.73 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.99)[-0.994,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.73)[ip: (-8.45), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.3.c.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; SUBJECT_ENDS_QUESTION(1.00)[] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 18:19:31 -0000 On Sat, Apr 6, 2019 at 10:24 PM Graham Menhennitt wrote: > Not that it's at all relevant to the question here, but... > > It does mostly work without em in the 12 kernel - I'm not sure how, but > it does. > > I upgraded to 12-stable via source but didn't add em to my custom > kernel. Most things worked - basic network functionality. But I had > problems with ipfw and igb. Adding em to the kernel fixed them. > FWIW the latest GENERIC kernel includes the iflib, em, etc devices as far as I can tell. I found the new UPDATING entry about iflib "no longer unconditionally compiled into the kernel" a bit confusing... So long as you are including GENERIC it should be the same as 12-RELEASE. > Graham > > On 6/4/19 6:12 am, Kris von Mach wrote: > > On 4/6/2019 2:56 AM, Pete French wrote: > >> Something odd going on there there - I am using 12-STABLE and I have > >> igb just fine, and it attaches to the same hardware that 11 did: > > > > It does work in 12, throughput is great, just that the latency is > > higher than 11. > > > > igb0: flags=8843 metric 0 mtu > > 1500 > > > options=e527bb > > > > > ether 38:ea:a7:8d:c1:6c > > inet 208.72.56.19 netmask 0xfffffc00 broadcast 208.72.59.255 > > inet6 fe80::3aea:a7ff:fe8d:c16c%igb0 prefixlen 64 scopeid 0x1 > > inet6 2602:ffb8::208:72:56:9 prefixlen 64 > > media: Ethernet autoselect (1000baseT ) > > status: active > > nd6 options=21 > > > >> Do you have a custom kernel, and if so did you see this note in > >> UPDATING? > > > > Yes I do, but it includes all of GENERIC which includes em drivers, > > otherwise it wouldn't even work with the network card. > > > > my custom kernel: > > > > include GENERIC > > ident CUSTOM > > makeoptions WITH_EXTRA_TCP_STACKS=1 > > options TCPHPTS > > options SC_KERNEL_CONS_ATTR=(FG_GREEN|BG_BLACK) > > options IPSTEALTH > > options AHC_REG_PRETTY_PRINT # Print register bitfields in debug > > options AHD_REG_PRETTY_PRINT # Print register bitfields in debug > > device cryptodev > > device aesni > > > > I did try without RACK just in case that was the culprit. > > > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org > " > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" > From owner-freebsd-stable@freebsd.org Tue Apr 9 19:01:38 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 182C115694DD for ; Tue, 9 Apr 2019 19:01:38 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 47DD6731C4 for ; Tue, 9 Apr 2019 19:01:37 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id BCDCC211083 for ; Tue, 9 Apr 2019 15:01:04 -0400 (EDT) Received: from [192.168.10.26] (D16.Denninger.Net [192.168.10.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 175118B29 for ; Tue, 9 Apr 2019 14:01:04 -0500 (CDT) From: Karl Denninger Subject: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: Date: Tue, 9 Apr 2019 14:01:03 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms030000080804070202030104" X-Rspamd-Queue-Id: 47DD6731C4 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.50 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: px.denninger.net]; NEURAL_HAM_SHORT(-0.92)[-0.920,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.38)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.74), asn: 14061(1.77), country: US(-0.06)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.988,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 19:01:38 -0000 This is a cryptographically signed message in MIME format. --------------ms030000080804070202030104 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable I've run into something often -- and repeatably -- enough since updating to 12-STABLE that I suspect there may be a code problem lurking in the ZFS stack or in the driver and firmware compatibility with various HBAs based on the LSI/Avago devices. The scenario is this -- I have data sets that are RaidZ2 that are my "normal" working set; one is comprised of SSD volumes and one of spinning rust volumes.=C2=A0 These all are normal and scrubs never show problems.=C2=A0 I've had physical failures with them over the years (alth= ough none since moving to 12-STABLE as of yet) and have never had trouble with resilvers or other misbehavior. I also have a "backup" pool that is a 3-member mirror, to which the volatile (that is, the zfs filesystems not set read-only) has zfs send's done to.=C2=A0 Call them backup-i, backup-e1 and backup-e2. All disks in these pools are geli-encrypted running on top of a freebsd-zfs partition inside a GPT partition table using -s 4096 (4k) geli "sectors". Two of the backup mirror members are always in the machine; backup-i (the base internal drive) is never removed.=C2=A0 The third is in a bank vault.=C2=A0 Every week the vault drive is exchanged with the other, so t= hat the "first" member is never removed from the host, but the other two (-e1 and -e2) alternate.=C2=A0 If the building burns I have a full copy o= f all the volatile data in the vault.=C2=A0 (I also have mirrored copies, 2= each, of all the datasets that are operationally read-only in the vault too; those get updated quarterly if there are changes to the operationally read-only portion of the data store.)=C2=A0 The drive in th= e vault is swapped weekly, so a problem should be detected almost immediately before it can bugger me. Before removing the disk intended to go to the vault I "offline" it, spin it down (camcontrol standby) which issues a standby immediate to the drive insuring that its cache is flushed and the spindle spun down and then pull it.=C2=A0 I go exchange them at the bank, insert the other = one, and "zpool online...." it, which automatically resilvers it. The disk resilvers and all is well -- no errors. Or is it all ok? If I run a scrub on the pool as soon as the resilver completes the disk I just inserted will /invariably /have a few checksum errors on it that the scrub fixes.=C2=A0 It's not a large number, anywhere from a couple do= zen to a hundred or so, but it's not zero -- and it damn well should be as the resilver JUST COMPLETED with no errors which means the ENTIRE DISK'S IN USE AREA was examined, compared, and blocks not on the "new member" or changed copied over.=C2=A0 The "-i" disk (the one that is never pulled= ) NEVER is the one with the checksum errors on it -- it's ALWAYS the one I just inserted and which was resilvered to. If I zpool clear the errors and scrub again all is fine -- no errors.=C2=A0= If I scrub again before pulling the disk the next time to do the swap all is fine as well.=C2=A0 I swap the two, resilver, and I'll get a few m= ore errors on the next scrub, ALWAYS on the disk I just put in. Smartctl shows NO errors on the disk.=C2=A0 No ECC, no reallocated sector= s, no interface errors, no resets, nothing.=C2=A0 Smartd is running and neve= r posts any real-time complaints, other than the expected one a minute or two after I yank the drive to take it to the bank.=C2=A0 There are no CAM-related errors printing on the console either.=C2=A0 So ZFS says ther= e's a *silent* data error (bad checksum; never a read or write error) in a handful of blocks but the disk says there have been no errors, the driver does not report any errors, there have been no power failures as the disk was in a bank vault and thus it COULDN'T have had a write-back cache corruption event or similar occur. I never had trouble with this under 11.1 or before and have been using this paradigm for something on the order of five years running on this specific machine without incident.=C2=A0 Now I'm seeing it repeatedly and= *reliably* under 12.0-STABLE.=C2=A0 I swapped the first disk that did it,= thinking it was physically defective -- the replacement did it on the next swap.=C2=A0 In fact I've yet to record a swap-out on 12-STABLE that *hasn't* done this and yet it NEVER happened under 11.1.=C2=A0 At the sam= e time I can run scrubs until the cows come home on the multiple Raidz2 packs on the same controller and never get any checksum errors on any of them. The firmware in the card was 19.00.00.00 -- again, this firmware *has been stable for years.*=C2=A0 I have just rolled the firmware on the card forward to 20.00.07.00, which is the "latest" available.=C2=A0 I had previously not moved to 20.x= because earlier versions had known issues (some severe and potentially fatal to data integrity) and 19 had been working without problem -- I thus had no reason to move to 20.00.07.00. But there apparently are some fairly significant timing differences between the driver code in 11.1 and 11.2/12.0, as I discovered when the SAS expander I used to have in these boxes started returning timeout errors that were false.=C2=A0 Again -- this same configuration was comple= tely stable under 11.1 and previous over a period of years. With 20.00.07.00 I have yet to have this situation recur -- so far -- but I have limited time with 20.00.07.00 and as such my confidence that the issue is in fact resolved by the card firmware change is only modest at this point.=C2=A0 Over the next month or so, if it doesn't happen agai= n, my confidence will of course improve. Checksum errors on ZFS volumes are extraordinarily uncool for the obvious reason -- they imply the disk thinks the data is fine (since it is not recording any errors on the interface or at the drive level) BUT ZFS thinks the data off that particular record was corrupt as the checksum fails.=C2=A0 Silent corruption is the worst sort in that it can = hide for months or even years before being discovered, long after your redundant copies have been re-used or overwritten. Assuming I do not see a recurrence with the 20.00.07.00 firmware I would suggest that UPDATING, the Release Notes or Errata have an entry added that for 12.x forward card firmware revisions prior to 20.00.07.00 carry *strong* cautions and that those with these HBAs be strongly urged to flash the card forward to 20.00.07.00 before upgrading or installing.=C2=A0= If you get a surprise of this sort and have no second copy that is not impacted you could find yourself severely hosed. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms030000080804070202030104 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDA5MTkwMTAz WjBPBgkqhkiG9w0BCQQxQgRACWGRzU+WR5iZpc8LML8YOsBeOmU4puYh5TSh/jS8dBkRhKq5 4iBkVPLDr5uSqo3wpKMpy7mzNnXwrU67bOtI6zBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAjPTKMymFEqeTA0ySbcoiKp1Xb69tPHjH5EESX7Z+tO3jc43g3v4sCNRvc8Qnsy4vL ZhLfn6yvLKr+IdMNtR0As8IGwD5l9EbHfZrwKiAJjqLEmK2++kW69BSSG1aRpBKNOYzQiQVS LhzBJFjyHOOcb40kSP2aFcA7xvtABDNKvFBLQS1Aje/X7yzlEULCCjcfDUMDba9ulVi+3k2E 08lXA6oC5UIY6oIxf2xdWtfv5de396Q0hVu3WEgaEgPFuOZVJO0nkzsfP5AwsaNlxVwicQKO T4bNVgd3J382F9jEneJztF7jHs2yGQjiND4N/embwt6QyCDqfmlPIvGDzZGs4FYqpFbX/50m Cjtu/vLEuAlJZ8+qtawQKDUxPstAoviJgM6Ggu1VyqQUANEyGFP2/s98CXBQ+ZKEyHE/FQsa adwR5PRGGLsE2bW/Jg8gDk3QoQLsKvzzpqLAOFF0T4Jk0UQBN4+LhWR9QxYj/vD+WlZXUQWP upqmmodmw0LBnnJZ24FsccubO89f5Gvlp80IO+BnVuntE0bmFnOXVvuFTiT1lZWwlQ7XxD2k jfyJ9sx6JnghCCzsRwTrIOXeRZShQ9e9DNGg0BzIR2P83kn5ALTYkgSyevXuuCQ3WKDKToxj i5p6JRD4Z0uGO/vacMbH+UUyBFB0nmMbt3pDS55TTAAAAAAAAA== --------------ms030000080804070202030104-- From owner-freebsd-stable@freebsd.org Tue Apr 9 20:04:33 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 181D4156C0F5 for ; Tue, 9 Apr 2019 20:04:33 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f172.google.com (mail-lj1-f172.google.com [209.85.208.172]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 1C44877C6B for ; Tue, 9 Apr 2019 20:04:32 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f172.google.com with SMTP id j89so15672775ljb.1 for ; Tue, 09 Apr 2019 13:04:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=vR6rs3gKq15WggjWTvh2tGOPeFlw8Ot9WtJwfGxMJn8=; b=NTwGURXV7cpWjfjlxD85fEpcEhREebXZ3K9UacdMUEPQd7ipmXVXX9pvUoK8Tir3cE ySp5YLFbd3RGytd2B+XJKGc39td1MnMnu/qzAfe/CENQNlXc75latxVWlnCR/oa9EI20 qRsLksmEMldulVheEvXHFz2ySTUpuny8rNFO+vtn1t0sQnNupBcFVRAhqXbUN04X0kZd E+o3ZDXSqEPOY23ZFbhsbalDCpZ7Zw2tWqvloZDORDdFSF0UUv7PRphqorzAZKdEvc7d k7T3NC91FpMnree0imrd5UWobVs0dQrzefdNdm0FgSO4ZKJjXmrrqSm3sZik70raFjhi qwaw== X-Gm-Message-State: APjAAAXBpJo8uGei5Ymobtma/f4KYzGO//8ATcFsk4lo+dez4E59cvzC QA35OT3w1D2uvzabPU4j1SxLhOje X-Google-Smtp-Source: APXvYqxoM2otjjpYLWN1IXodwm6krmPk3m4ClzAlIcFqJHBe8YcwETsxvJGnW2tBUNs+uItcrnJR9Q== X-Received: by 2002:a2e:6c0f:: with SMTP id h15mr19957536ljc.155.1554840265278; Tue, 09 Apr 2019 13:04:25 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id v13sm6962799lje.84.2019.04.09.13.04.24 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 13:04:24 -0700 (PDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Karl Denninger , freebsd-stable@freebsd.org References: From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: Date: Tue, 9 Apr 2019 23:04:23 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 1C44877C6B X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.172 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-4.27 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; IP_SCORE(-1.30)[ip: (-0.40), ipnet: 209.85.128.0/17(-3.87), asn: 15169(-2.18), country: US(-0.06)]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_COUNT_THREE(0.00)[3]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[172.208.85.209.list.dnswl.org : 127.0.5.0]; NEURAL_HAM_SHORT(-0.96)[-0.962,0]; RCVD_TLS_LAST(0.00)[]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 20:04:33 -0000 On 09/04/2019 22:01, Karl Denninger wrote: > the resilver JUST COMPLETED with no errors which means the ENTIRE DISK'S > IN USE AREA was examined, compared, and blocks not on the "new member" > or changed copied over. I think that that's not entirely correct. ZFS maintains something called DTL, a dirty-time log, for a missing / offlined / removed device. When the device re-appears and gets resilvered, ZFS walks only those blocks that were born within the TXG range(s) when the device was missing. In any case, I do not have an explanation for what you are seeing. -- Andriy Gapon From owner-freebsd-stable@freebsd.org Tue Apr 9 20:28:31 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4C9AD156C928 for ; Tue, 9 Apr 2019 20:28:31 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8489080860 for ; Tue, 9 Apr 2019 20:28:30 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 2ECAD211083 for ; Tue, 9 Apr 2019 16:27:59 -0400 (EDT) Received: from [192.168.10.26] (D16.Denninger.Net [192.168.10.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 9A9CC8E33 for ; Tue, 9 Apr 2019 15:27:58 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org References: From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> Date: Tue, 9 Apr 2019 15:27:57 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms080500070903000806050602" X-Rspamd-Queue-Id: 8489080860 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.55 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: px.denninger.net]; NEURAL_HAM_SHORT(-0.96)[-0.965,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.98)[-0.985,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.39)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.76), asn: 14061(1.76), country: US(-0.06)]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 20:28:31 -0000 This is a cryptographically signed message in MIME format. --------------ms080500070903000806050602 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/9/2019 15:04, Andriy Gapon wrote: > On 09/04/2019 22:01, Karl Denninger wrote: >> the resilver JUST COMPLETED with no errors which means the ENTIRE DISK= 'S >> IN USE AREA was examined, compared, and blocks not on the "new member"= >> or changed copied over. > I think that that's not entirely correct. > ZFS maintains something called DTL, a dirty-time log, for a missing / o= fflined / > removed device. When the device re-appears and gets resilvered, ZFS wa= lks only > those blocks that were born within the TXG range(s) when the device was= missing. > > In any case, I do not have an explanation for what you are seeing. That implies something much more-serious could be wrong such as given enough time -- a week, say -- that the DTL marker is incorrect and some TXGs that were in fact changed since the OFFLINE are not walked through and synchronized.=C2=A0 That would explain why it gets caught by a scrub = -- the resilver is in fact not actually copying all the blocks that got changed and so when you scrub the blocks are not identical.=C2=A0 Assumin= g the detached disk is consistent that's not catastrophically bad IF CAUGHT; where you'd get screwed HARD is in the situation where (for example) you had a 2-unit mirror, detached one, re-attached it, resilver says all is well, there is no scrub performed and then the *non-detached* disk fails before there is a scrub.=C2=A0 In that case you= will have permanently destroyed or corrupted data since the other disk is allegedly consistent but there are blocks *missing* that were never copied over. Again this just showed up on 12.x; it definitely was *not* at issue in 11.1 at all.=C2=A0 I never ran 11.2 in production for a material amount o= f time (I went from 11.1 to 12.0 STABLE after the IPv6 fixes were posted to 12.x) so I don't know if it is in play on 11.2 or not. I'll see if it shows up again with 20.00.07.00 card firmware. Of note I cannot reproduce this on my test box with EITHER 19.00.00.00 or 20.00.07.00 firmware when I set up a 3-unit mirror, offline one, make a crap-ton of changes, offline the second and reattach the third (in effect mirroring the "take one to the vault" thing) with a couple of hours elapsed time and a synthetic (e.g. "dd if=3D/dev/random of=3Doutfil= e bs=3D1m" sort of thing) "make me some new data that has to be resilvered"= workload.=C2=A0 I don't know if that's because I need more entropy in the= filesystem than I can reasonably generate this way (e.g. more fragmentation of files, etc) or whether it's a time-based issue (e.g. something's wrong with the DTL/TXG thing as you note above in terms of how it functions and it only happens if the time elapsed causes something to be subject to a rollover or similar problem.)=C2=A0 I spent quite a lot of time trying to make reproduce the issue on my "sandbox" machine and was unable -- and of note it is never a large quantity of data that is impacted, it's usually only a couple of dozen checksums that show as bad and fixed.=C2=A0 Of note it's also never just = one; if there was a single random hit on a data block due to ordinary bitrot sort of issues I'd expect only one checksum to be bad.=C2=A0 But generati= ng a realistic synthetic workload over the amount of time involved on a sandbox is not trivial at all; the system on which this is now happening handles a lot of email and routine processing of various sorts including a fair bit of database activity associated with network monitoring and statistical analysis. I'm assuming that using "offline" as a means to do this hasn't become "invalid" as something that's considered "ok" as a means of doing this sort of thing.... it certainly has worked perfectly well for a very long time! --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms080500070903000806050602 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDA5MjAyNzU3 WjBPBgkqhkiG9w0BCQQxQgRAgTj9XLvOvW+k07lRxZgoFoabX1+/C2/9hWYngO1bUfLVEW6f 1y82QZtUUNEYExI56rkXZQ4VwVOLrFh1p9dUYjBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAEErLO2uIehEq4w55sKwJlwL1afcLvWwdLhbsNrkOqZ6NAJ1cTsA+K71rhpleo7fCD VhVzveEr1vSpM36wpwu2IH6JK4GiyK2YIQbUDKOlps1Kg6BeS4AHZL9HOliONM5P3hjaHT22 YQ8m1G56Jn4JEiTHpiib+DRe5mtu5TDMknAnExWn4Ndy6lZSr4mirI3K3rnCfFnbncGXwytf 0aUAFKJVzWsupYEVXvIRVtyl/e/oePSvY7QhZGtQPQndSmJFiUozEGl0YJv6hZ2VMFTinWkz w1/54iEp537C/uc6Ib+34mayuQkd1VIb5YRlSAGCI87ZrlbtXAIcIYtzy09d63NMTylCZnxu lXtFOT4GKSt+7JN4J3aHMB60EI2QYUeZPTPAzaz9gr77mgPppHbznH8U91rxFqvAAi+b0HCj vM2MNHK1QSvm62xir+4NLn6/+4nO5XM6XVWHXlU9Jz+M5sx4Sa2G4V6nGJDDHJzaoP3kK5Fa wDBk+xYs6/SxRNSQ2FoBbuLmf1W1W31Y0xNWgRDQqobJmeFe5Ahv/zm3YXST50f73Cs9VaDx wJQHKpaMXK5Zm5JC/z0wzM+caI6FnP1QTw08lFFlJW4PPNCnhzGjZ3MERuRC62pdiDCo45DW i17yLWYTzWyQ/Q9qbsZt0beq+z1CQg5SAmMPvA0Z2QAAAAAAAA== --------------ms080500070903000806050602-- From owner-freebsd-stable@freebsd.org Tue Apr 9 21:09:54 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8F287156DA92 for ; Tue, 9 Apr 2019 21:09:54 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-vs1-xe30.google.com (mail-vs1-xe30.google.com [IPv6:2607:f8b0:4864:20::e30]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 60B0881E91 for ; Tue, 9 Apr 2019 21:09:53 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-vs1-xe30.google.com with SMTP id f15so66740vsk.9 for ; Tue, 09 Apr 2019 14:09:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:from:subject:date:importance; bh=0PRNWgGJ5R6RLlhM/fMKlBqqvzZy2zg3ZGswzzRMzbE=; b=g02nk1fdcmFy0SX3znkSChIEP5LJ8kha2wU0xzEdmWxMLhprsIQNOkXyCGr94Ix84R pWXR6ChxbALjcUYdpgrVOtQl/9HszI11sWC1ZSjEms1KE6ubspFKWD2Ja2MAL6AUTeBe 7gEnpDpWXrtQAkWw3a+7clUWQQfhTvtYP5XhMZ7z2SrwiTXaDbf1vms718pF5FyIjbnX te6sx4TXO+5nLnZMNBKodzSnVjjgYFzPAeeBvJzZv1Scl8qU/VkNkVbZ7+uwUIPAgZYH V0OX8D17waifDiuuBw1fqxgvfbaHoL4q5ENtGS+Il/RYs20ejGKgSgKOLjsPsQt+bxbl lmPg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:from:subject:date :importance; bh=0PRNWgGJ5R6RLlhM/fMKlBqqvzZy2zg3ZGswzzRMzbE=; b=UhLnuFXk+VDJm0Lk9BuJ/hkJ11FxL/pcaJv0TQQ7vIMFZWzc+WRHU8EHszkQVzv+oS VDhWBHNzfFUJ1MfzgUqEL0YwNDm4NG/sR7ndnvXOj1tIUgRjMrbjjdkYCJqaw6Iory57 1fRcNNVWgMX6GN24nLSaSP2ZUK9BPZiBUB1XlgdgY646LT2fQYd0jWB7g5/70wjSt+DS 7d6vyD9DHmrMTLsrCR+SxcoLFkEOdztGfi8zpgCEjum0Q66wZrevQ/7GORletP+AKcpj 8mIdgKMlEwM4xe9gN4ro/2yd0CTkqv6l6xTPLm9DZhjA+Nkf6YU+1uxULVMjBz6tZ9sX ItyQ== X-Gm-Message-State: APjAAAWrweBVMoHn4D9pWaAfS4CE8Sew/faMuqaHSIOBE+4IY/weqkv0 r3aovBMNNKRXWcttjkVK0D9uugCu X-Google-Smtp-Source: APXvYqyYoG15SZ/fh3WoN4gHJotxzED8ADV7JA0AZZlUCppUEcdh9XMgeSkaCOB1LdrDIT0jxjRkNw== X-Received: by 2002:a67:3093:: with SMTP id w141mr22014885vsw.170.1554844192529; Tue, 09 Apr 2019 14:09:52 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.119? (git.mayberryinv.com. [63.143.111.202]) by smtp.gmail.com with ESMTPSA id z79sm30107301vkd.18.2019.04.09.14.09.51 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 14:09:51 -0700 (PDT) Message-ID: <5cad0a1f.1c69fb81.c5304.4f33@mx.google.com> MIME-Version: 1.0 To: "freebsd-stable@freebsd.org" From: Software Info Subject: Mailx Question Date: Tue, 9 Apr 2019 16:09:52 -0500 Importance: normal X-Priority: 3 X-Rspamd-Queue-Id: 60B0881E91 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=g02nk1fd; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::e30 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-5.98 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.98)[-0.982,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.99)[ip: (-9.75), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[0.3.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_DN_EQ_ADDR_ALL(0.00)[] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 21:09:54 -0000 Hi All Since mailx is built into FreeBSD I decided to try asking this question her= e. I have a text file with about 30 email addresses. The file will change e= very day. I want an easy commandline way to read the file and blind copy se= nd an email to the addresses in the file. So far, I have this working with = just a plain send using the command below. mailx -s "Test Emails" -b `cat mylist.txt` < body.txt -r "No-Reply" Of course, when I use a plain send, everybody sees everybody=E2=80=99s emai= l address so I would love to be able to do a blind copy send. Would anyone = be able to assist me with this? Regards SI Sent from Mail for Windows 10 From owner-freebsd-stable@freebsd.org Tue Apr 9 21:27:38 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E66EE156E66D for ; Tue, 9 Apr 2019 21:27:37 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-it1-x12e.google.com (mail-it1-x12e.google.com [IPv6:2607:f8b0:4864:20::12e]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 068B982FFD for ; Tue, 9 Apr 2019 21:27:37 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-it1-x12e.google.com with SMTP id k64so7496364itb.5 for ; Tue, 09 Apr 2019 14:27:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=gnika/KUM4T2HRI9MKk32EAyvsr/9vmnp+bZ4VoOi2E=; b=hOjSzYv/bpZQmBimqOCOaKsaODQ8sx8m7L69MDqpBneqhJFSMplLcZ5e+MhoB77I48 X44RVTfzpxFRU/41EhnFpSefZAeY6raoaeCzJXJEmv7PUWMAoN1bUCdmVvwc5aP1p6yN 420XwTNs3LCr+cOxl0Lha75J1JbiPqRWJf5pZnLTOsGRlOMJsw0JPzDjl2eLJvFBz2M8 h3OLgpdC7NlG5ovUov/YOI4xx96Ql1NVTXLKsKx8zmI1q1NUFymWTWBeQ5OyTbkyPf3F ygTC/VMiRGdarjLHQvZzaDVbn9GJWlzqAveQhhi3WXHL7xnKVWP6I7lfk1LuK6plMPkO 6FaA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=gnika/KUM4T2HRI9MKk32EAyvsr/9vmnp+bZ4VoOi2E=; b=S8kNUjWT2DS0u1YugxUApFWZiLy05tMdSh0IlMKgr1tlBoAwRPca8L7LDpiE2uNUlj 2MCwzir+xUtKdp48XiYkhJk2GxLXy9tGongO0lcAepTwEVlMCiR7qMBg3OmAVG4/ju0U UsNjwCtFmZJf8KUoq66cP/HKYSwkooHeJC3gDmLCPmNRFm/OAHF/FgR2Ge4QweKgeT3J 4ua6ZJYe37qWwMY7G0o3HmVezhkrIDF3ot8mSsjxZ7HUKwJFD1gtpmKZsLdjkKPHppWa yv5hjpt2k1KSHPwqufwOdY901qmoo0pCLXOt3TL7HFMsmzzQ9MICvCwFNzZTqVhCnh5N f9UQ== X-Gm-Message-State: APjAAAWzPBZIILzjojGrQQE0JValr8BbDxqWaJzkfPvjz9SAXjH775kb +vmLSWJEAScQN++pKDYrOxOxA76W2n4jEFln+7ey X-Google-Smtp-Source: APXvYqySmeMV+tPGPvcg/n72qML9bV1bk7jiDS9G3497AW/NqEnUIs/DIXDtw9fifyTdPXWL1HPoNQzcElHINKLQX9k= X-Received: by 2002:a05:6638:1a:: with SMTP id z26mr26728870jao.99.1554845256237; Tue, 09 Apr 2019 14:27:36 -0700 (PDT) MIME-Version: 1.0 References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> In-Reply-To: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> From: Zaphod Beeblebrox Date: Tue, 9 Apr 2019 17:27:29 -0400 Message-ID: Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Karl Denninger Cc: FreeBSD Stable X-Rspamd-Queue-Id: 068B982FFD X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=hOjSzYv/; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2607:f8b0:4864:20::12e as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-6.78 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.97)[-0.973,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[e.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.80)[ip: (-8.82), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 21:27:38 -0000 I have a "Ghetto" home RAID array. It's built on compromises and makes use of RAID-Z2 to survive. It consists of two plexes of 8x 4T units of "spinning rust". It's been upgraded and upgraded. It started as 8x 2T, then 8x 2T + 8x 4T then the current 16x 4T. The first 8 disks are connected to motherboard SATA. IIRC, there are 10. Two ports are used for a mirror that it boots from. There's also an SSD in there somhow, so it might be 12 ports on the motherboard. The other 8 disks started life in eSATA port multiplier boxes. That was doubleplusungood, so I got a RAID card based on LSI pulled from a fujitsu server in Japan. That's been upgraded a couple of times... not always a good experience. One problem is that cheap or refurbished drives don't always "like" SAS controllers and FreeBSD. YMMV. Anyways, this is all to introduce the fact that I've seen this behaviour multiple times. You have a drive that leaves the array for some amount of time, and after resilvering, a scrub will find a small amount of bad data. 32 k or 40k or somesuch. In my cranial schema of things, I've chalked it up to out-of-order writing of the drives ... or other such behavior s.t. ZFS doesn't know exactly what has been written. I've often wondered if the fix would be to add an amount of fuzz to the transaction range that is resilvered. On Tue, Apr 9, 2019 at 4:32 PM Karl Denninger wrote: > On 4/9/2019 15:04, Andriy Gapon wrote: > > On 09/04/2019 22:01, Karl Denninger wrote: > >> the resilver JUST COMPLETED with no errors which means the ENTIRE DISK'S > >> IN USE AREA was examined, compared, and blocks not on the "new member" > >> or changed copied over. > > I think that that's not entirely correct. > > ZFS maintains something called DTL, a dirty-time log, for a missing / > offlined / > > removed device. When the device re-appears and gets resilvered, ZFS > walks only > > those blocks that were born within the TXG range(s) when the device was > missing. > > > > In any case, I do not have an explanation for what you are seeing. > > That implies something much more-serious could be wrong such as given > enough time -- a week, say -- that the DTL marker is incorrect and some > TXGs that were in fact changed since the OFFLINE are not walked through > and synchronized. That would explain why it gets caught by a scrub -- > the resilver is in fact not actually copying all the blocks that got > changed and so when you scrub the blocks are not identical. Assuming > the detached disk is consistent that's not catastrophically bad IF > CAUGHT; where you'd get screwed HARD is in the situation where (for > example) you had a 2-unit mirror, detached one, re-attached it, resilver > says all is well, there is no scrub performed and then the > *non-detached* disk fails before there is a scrub. In that case you > will have permanently destroyed or corrupted data since the other disk > is allegedly consistent but there are blocks *missing* that were never > copied over. > > Again this just showed up on 12.x; it definitely was *not* at issue in > 11.1 at all. I never ran 11.2 in production for a material amount of > time (I went from 11.1 to 12.0 STABLE after the IPv6 fixes were posted > to 12.x) so I don't know if it is in play on 11.2 or not. > > I'll see if it shows up again with 20.00.07.00 card firmware. > > Of note I cannot reproduce this on my test box with EITHER 19.00.00.00 > or 20.00.07.00 firmware when I set up a 3-unit mirror, offline one, make > a crap-ton of changes, offline the second and reattach the third (in > effect mirroring the "take one to the vault" thing) with a couple of > hours elapsed time and a synthetic (e.g. "dd if=/dev/random of=outfile > bs=1m" sort of thing) "make me some new data that has to be resilvered" > workload. I don't know if that's because I need more entropy in the > filesystem than I can reasonably generate this way (e.g. more > fragmentation of files, etc) or whether it's a time-based issue (e.g. > something's wrong with the DTL/TXG thing as you note above in terms of > how it functions and it only happens if the time elapsed causes > something to be subject to a rollover or similar problem.) > > I spent quite a lot of time trying to make reproduce the issue on my > "sandbox" machine and was unable -- and of note it is never a large > quantity of data that is impacted, it's usually only a couple of dozen > checksums that show as bad and fixed. Of note it's also never just one; > if there was a single random hit on a data block due to ordinary bitrot > sort of issues I'd expect only one checksum to be bad. But generating a > realistic synthetic workload over the amount of time involved on a > sandbox is not trivial at all; the system on which this is now happening > handles a lot of email and routine processing of various sorts including > a fair bit of database activity associated with network monitoring and > statistical analysis. > > I'm assuming that using "offline" as a means to do this hasn't become > "invalid" as something that's considered "ok" as a means of doing this > sort of thing.... it certainly has worked perfectly well for a very long > time! > > -- > Karl Denninger > karl@denninger.net > /The Market Ticker/ > /[S/MIME encrypted email preferred]/ > From owner-freebsd-stable@freebsd.org Tue Apr 9 21:40:11 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 84D46156EDAC for ; Tue, 9 Apr 2019 21:40:11 +0000 (UTC) (envelope-from SRS0=cKrl=SL=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (elsa.codelab.cz [94.124.105.4]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 02D4C837C8 for ; Tue, 9 Apr 2019 21:40:11 +0000 (UTC) (envelope-from SRS0=cKrl=SL=quip.cz=000.fbsd@elsa.codelab.cz) Received: from elsa.codelab.cz (localhost [127.0.0.1]) by elsa.codelab.cz (Postfix) with ESMTP id 4EC2928426; Tue, 9 Apr 2019 23:40:00 +0200 (CEST) Received: from illbsd.quip.test (ip-62-24-76-131.net.upcbroadband.cz [62.24.76.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by elsa.codelab.cz (Postfix) with ESMTPSA id 3922F28423; Tue, 9 Apr 2019 23:39:59 +0200 (CEST) Subject: Re: Mailx Question To: Software Info , "freebsd-stable@freebsd.org" References: <5cad0a1f.1c69fb81.c5304.4f33@mx.google.com> From: Miroslav Lachman <000.fbsd@quip.cz> Message-ID: <0d44d441-3120-c140-40da-7a32baff0cae@quip.cz> Date: Tue, 9 Apr 2019 23:39:59 +0200 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Firefox/52.0 SeaMonkey/2.49.3 MIME-Version: 1.0 In-Reply-To: <5cad0a1f.1c69fb81.c5304.4f33@mx.google.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 02D4C837C8 X-Spamd-Bar: +++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [5.07 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: elsa.codelab.cz]; RCPT_COUNT_TWO(0.00)[2]; FORGED_SENDER(0.30)[000.fbsd@quip.cz,SRS0=cKrl=SL=quip.cz=000.fbsd@elsa.codelab.cz]; FREEMAIL_TO(0.00)[gmail.com]; IP_SCORE(0.92)[ip: (0.43), ipnet: 94.124.104.0/21(0.22), asn: 42000(3.89), country: CZ(0.08)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:42000, ipnet:94.124.104.0/21, country:CZ]; MIME_TRACE(0.00)[0:+]; FROM_NEQ_ENVFROM(0.00)[000.fbsd@quip.cz,SRS0=cKrl=SL=quip.cz=000.fbsd@elsa.codelab.cz]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.97)[0.967,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[quip.cz]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.99)[0.989,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(1.00)[0.999,0]; RCVD_IN_DNSWL_NONE(0.00)[4.105.124.94.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 21:40:11 -0000 Software Info wrote on 2019/04/09 23:09: > Hi All > Since mailx is built into FreeBSD I decided to try asking this question here. I have a text file with about 30 email addresses. The file will change every day. I want an easy commandline way to read the file and blind copy send an email to the addresses in the file. So far, I have this working with just a plain send using the command below. > mailx -s "Test Emails" -b `cat mylist.txt` < body.txt -r "No-Reply" > > Of course, when I use a plain send, everybody sees everybody’s email address so I would love to be able to do a blind copy send. Would anyone be able to assist me with this? It may depend on your MTA (Sendmail, Postfix, Exim etc.) "You must specify direct recipients with -s, -c, or -b." -b bcc-addr Send blind carbon copies to bcc-addr list of users. The bcc-addr argument should be a comma-separated list of names. You should replace newlines with comma: cat mylist.txt | tr "\n" "," Maybe something like this will work for you: mail -s "Test E-mails" -b `cat mylist.txt | tr "\n" ","` my-generic@example.com < body.txt Miroslav Lachman From owner-freebsd-stable@freebsd.org Tue Apr 9 22:21:22 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 36C8A1570591 for ; Tue, 9 Apr 2019 22:21:22 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-vs1-xe43.google.com (mail-vs1-xe43.google.com [IPv6:2607:f8b0:4864:20::e43]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 545C885193 for ; Tue, 9 Apr 2019 22:21:21 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-vs1-xe43.google.com with SMTP id g127so177796vsd.6 for ; Tue, 09 Apr 2019 15:21:21 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:from:subject:date:importance:in-reply-to :references; bh=V5SEA85O5k5/BS4OFK2PxSbLvf8gXoYUCpfvoA08g3g=; b=lTdY2TEqmGsTm+3q4WOzUR8rfcix6331u2m9BX/bCSyeu8X7j4b4FDFIf3MaLL3OrR KPYbNcazIzWtDvQK71LB+1CNF7INtFo7B0ou484O3DUhvkx1d33CcHoVDzrPbUidTgwQ +VF0/i3hTsZSzeNiaG25Kv/EhvsrZu8J+qPJAPeg/QzhHeR1mOzIE/XfVyGmCMml1MaY 2EzTUD/fgh4qp4BEJwThdsGj43uyp6tyH7Voy857M8fRPR/k+Dp/yEcHPNnk3IwMGbJY zNONdfMEZsCr78oy9Z5ozEcGN+FzU8Ug2rXcSK8hNB0NQy5hhpcn3SLL5rU4dJ+VD3Yd bWWw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:from:subject:date :importance:in-reply-to:references; bh=V5SEA85O5k5/BS4OFK2PxSbLvf8gXoYUCpfvoA08g3g=; b=YVdDrmrGojfpe8ZC1qtiVcRM50PxoLsswjBg0TMSTXSgf3QmPamZwePYjNy9oqsyx1 EpAaA/2KoVR/aFQakbCRYNX6LM98tRoZCD3TvRXNRJpJtB6KjlDZ92oh8jfTUgEPIjpA Gm+6apuVGUycAP+eOTcPMqyFRu2bo2CIpUrhECHjwm7iZGTzH5z6cXUDJIGiBV0xV1c+ oT0rN9rG3Alksed4TQXUV7RKBewDCSNZvr7Bua1stt6KtSTxVTnkB0sCPbIvWTf4+54q ugkr8aqDwRzualeJ6LnUv/u3WPPZxPT2kipcHosFdTTbhLJZCJq2RsvHAis+6KYGtj5b ZIRQ== X-Gm-Message-State: APjAAAX3Vx02AQ8k4W3cyxbq49BhMPtvuHITVLWX6lORBWFFfbmFXoVS K2+kLSANEgNl29m1joDO2k0MUWkY X-Google-Smtp-Source: APXvYqwEmYV6/KI8mdvsqqsNqcVzRVKdnMqBt64ATdKFA3bVXGxS0vvzlBzywNUQh2rp+W3xYu97BQ== X-Received: by 2002:a67:c490:: with SMTP id d16mr22231770vsk.93.1554848480937; Tue, 09 Apr 2019 15:21:20 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.119? (git.mayberryinv.com. [63.143.111.202]) by smtp.gmail.com with ESMTPSA id h1sm9988084vsd.2.2019.04.09.15.21.19 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Tue, 09 Apr 2019 15:21:20 -0700 (PDT) Message-ID: <5cad1ae0.1c69fb81.b5b11.08ae@mx.google.com> MIME-Version: 1.0 To: Miroslav Lachman <000.fbsd@quip.cz>, "freebsd-stable@freebsd.org" From: Software Info Subject: RE: Mailx Question Date: Tue, 9 Apr 2019 17:21:20 -0500 Importance: normal X-Priority: 3 In-Reply-To: <0d44d441-3120-c140-40da-7a32baff0cae@quip.cz> References: <5cad0a1f.1c69fb81.c5304.4f33@mx.google.com> <0d44d441-3120-c140-40da-7a32baff0cae@quip.cz> X-Rspamd-Queue-Id: 545C885193 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=lTdY2TEq; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::e43 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-3.31 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.90)[-0.897,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.996,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[3.4.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-0.41)[ip: (3.15), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 09 Apr 2019 22:21:22 -0000 Fantastic. Works like a charm. Thank you very much. Kind Regards SI Sent from Mail for Windows 10 From: Miroslav Lachman Sent: Tuesday, April 9, 2019 4:40 PM To: Software Info; freebsd-stable@freebsd.org Subject: Re: Mailx Question Software Info wrote on 2019/04/09 23:09: > Hi All > Since mailx is built into FreeBSD I decided to try asking this question h= ere. I have a text file with about 30 email addresses. The file will change= every day. I want an easy commandline way to read the file and blind copy = send an email to the addresses in the file. So far, I have this working wit= h just a plain send using the command below. > mailx -s "Test Emails" -b `cat mylist.txt` < body.txt -r "No-Reply" >=20 > Of course, when I use a plain send, everybody sees everybody=E2=80=99s em= ail address so I would love to be able to do a blind copy send. Would anyon= e be able to assist me with this? It may depend on your MTA (Sendmail, Postfix, Exim etc.) "You must specify direct recipients with -s, -c, or -b." -b bcc-addr Send blind carbon copies to bcc-addr list of users. The bcc-addr argument should be a comma-separated list of names. You should replace newlines with comma: cat mylist.txt | tr "\n" "," Maybe something like this will work for you: mail -s "Test E-mails" -b `cat mylist.txt | tr "\n" ","`=20 my-generic@example.com < body.txt Miroslav Lachman From owner-freebsd-stable@freebsd.org Wed Apr 10 01:09:52 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E8D071573796 for ; Wed, 10 Apr 2019 01:09:51 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5701389692 for ; Wed, 10 Apr 2019 01:09:51 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id C05E2211083 for ; Tue, 9 Apr 2019 21:09:49 -0400 (EDT) Received: from [192.168.10.26] (D16.Denninger.Net [192.168.10.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id 3F8639649 for ; Tue, 9 Apr 2019 20:09:49 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> Date: Tue, 9 Apr 2019 20:09:48 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms070207000405010707030509" X-Rspamd-Queue-Id: 5701389692 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.44 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: px.denninger.net]; NEURAL_HAM_SHORT(-0.84)[-0.840,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.40)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.80), asn: 14061(1.75), country: US(-0.06)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.989,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 01:09:52 -0000 This is a cryptographically signed message in MIME format. --------------ms070207000405010707030509 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/9/2019 16:27, Zaphod Beeblebrox wrote: > I have a "Ghetto" home RAID array. It's built on compromises and makes= use > of RAID-Z2 to survive. It consists of two plexes of 8x 4T units of > "spinning rust". It's been upgraded and upgraded. It started as 8x 2T= , > then 8x 2T + 8x 4T then the current 16x 4T. The first 8 disks are > connected to motherboard SATA. IIRC, there are 10. Two ports are used= for > a mirror that it boots from. There's also an SSD in there somhow, so i= t > might be 12 ports on the motherboard. > > The other 8 disks started life in eSATA port multiplier boxes. That wa= s > doubleplusungood, so I got a RAID card based on LSI pulled from a fujit= su > server in Japan. That's been upgraded a couple of times... not always = a > good experience. One problem is that cheap or refurbished drives don't= > always "like" SAS controllers and FreeBSD. YMMV. > > Anyways, this is all to introduce the fact that I've seen this behaviou= r > multiple times. You have a drive that leaves the array for some amount = of > time, and after resilvering, a scrub will find a small amount of bad da= ta. > 32 k or 40k or somesuch. In my cranial schema of things, I've chalked = it > up to out-of-order writing of the drives ... or other such behavior s.t= =2E > ZFS doesn't know exactly what has been written. I've often wondered if= the > fix would be to add an amount of fuzz to the transaction range that is > resilvered. > > > On Tue, Apr 9, 2019 at 4:32 PM Karl Denninger wrot= e: > >> On 4/9/2019 15:04, Andriy Gapon wrote: >>> On 09/04/2019 22:01, Karl Denninger wrote: >>>> the resilver JUST COMPLETED with no errors which means the ENTIRE DI= SK'S >>>> IN USE AREA was examined, compared, and blocks not on the "new membe= r" >>>> or changed copied over. >>> I think that that's not entirely correct. >>> ZFS maintains something called DTL, a dirty-time log, for a missing /= >> offlined / >>> removed device. When the device re-appears and gets resilvered, ZFS >> walks only >>> those blocks that were born within the TXG range(s) when the device w= as >> missing. >>> In any case, I do not have an explanation for what you are seeing. >> That implies something much more-serious could be wrong such as given >> enough time -- a week, say -- that the DTL marker is incorrect and som= e >> TXGs that were in fact changed since the OFFLINE are not walked throug= h >> and synchronized. That would explain why it gets caught by a scrub --= >> the resilver is in fact not actually copying all the blocks that got >> changed and so when you scrub the blocks are not identical. Assuming >> the detached disk is consistent that's not catastrophically bad IF >> CAUGHT; where you'd get screwed HARD is in the situation where (for >> example) you had a 2-unit mirror, detached one, re-attached it, resilv= er >> says all is well, there is no scrub performed and then the >> *non-detached* disk fails before there is a scrub. In that case you >> will have permanently destroyed or corrupted data since the other disk= >> is allegedly consistent but there are blocks *missing* that were never= >> copied over. >> >> Again this just showed up on 12.x; it definitely was *not* at issue in= >> 11.1 at all. I never ran 11.2 in production for a material amount of >> time (I went from 11.1 to 12.0 STABLE after the IPv6 fixes were posted= >> to 12.x) so I don't know if it is in play on 11.2 or not. >> >> I'll see if it shows up again with 20.00.07.00 card firmware. >> >> Of note I cannot reproduce this on my test box with EITHER 19.00.00.00= >> or 20.00.07.00 firmware when I set up a 3-unit mirror, offline one, ma= ke >> a crap-ton of changes, offline the second and reattach the third (in >> effect mirroring the "take one to the vault" thing) with a couple of >> hours elapsed time and a synthetic (e.g. "dd if=3D/dev/random of=3Dout= file >> bs=3D1m" sort of thing) "make me some new data that has to be resilver= ed" >> workload. I don't know if that's because I need more entropy in the >> filesystem than I can reasonably generate this way (e.g. more >> fragmentation of files, etc) or whether it's a time-based issue (e.g. >> something's wrong with the DTL/TXG thing as you note above in terms of= >> how it functions and it only happens if the time elapsed causes >> something to be subject to a rollover or similar problem.) >> >> I spent quite a lot of time trying to make reproduce the issue on my >> "sandbox" machine and was unable -- and of note it is never a large >> quantity of data that is impacted, it's usually only a couple of dozen= >> checksums that show as bad and fixed. Of note it's also never just on= e; >> if there was a single random hit on a data block due to ordinary bitro= t >> sort of issues I'd expect only one checksum to be bad. But generating= a >> realistic synthetic workload over the amount of time involved on a >> sandbox is not trivial at all; the system on which this is now happeni= ng >> handles a lot of email and routine processing of various sorts includi= ng >> a fair bit of database activity associated with network monitoring and= >> statistical analysis. >> >> I'm assuming that using "offline" as a means to do this hasn't become >> "invalid" as something that's considered "ok" as a means of doing this= >> sort of thing.... it certainly has worked perfectly well for a very lo= ng >> time! >> >> -- >> Karl Denninger >> karl@denninger.net >> /The Market Ticker/ >> /[S/MIME encrypted email preferred]/ The problem with the theory you have put forward Zaphod (which is logical) is that in my *specific* case it shouldn't happen -- which implies (strongly) that there's a code problem somewhere and what you're seeing is not a result of random drive write-reordering. Specifically, I *explicitly* OFFLINE the disk in question, which is a controlled operation and *should* result in a cache flush out of the ZFS code into the drive before it is OFFLINE'd. This should result in the "last written" TXG that the remaining online members have, and the one in the offline member, being consistent. Then I "camcontrol standby" the involved drive, which forces a writeback cache flush and a spindown; in other words, re-ordered or not, the on-platter data *should* be consistent with what the system thinks happened before I yank the physical device. But... it's not. And these are not "ghetto" devices in my case either -- they're all either NAS or enterprise drives and all of them are on a LSI/Avago Sata/SAS adapter -- with no expanders involved (any more anyway; under 11.1 the expander was ENTIRELY reliable, but under 11.2 and 12.0 it's not, so I removed it.)=C2=A0 No "grab whatever" sort of disks are involve= d in this and I've seen the same behavior across multiple makes and models, which strongly implies it's a not a random firmware bug in some specific model of disk. In the case of a *random* detach I can see it, because the disk that detaches and goes offline might have some number of transactions in the write cache which are lost when the power fails on said device, and thus the committed TXG number is inconsistent with what the remaining online devices have, and write re-ordering could screw you with no way for the code to know it occurred. But in this case the offline was intentional and thus any queued writes should have been flushed down from the OS to the device, and the standby command should flush any on-drive cache to the physical media before the spindle is spun down -- thus there should be no possible way for an out-of-sync "last written" TXG to happen. Unless there's a code problem somewhere.... --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms070207000405010707030509 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDEwMDEwOTQ4 WjBPBgkqhkiG9w0BCQQxQgRAytdO0YZHl2o6zsCuJjDHqQGyWJ8I2cX82GdURlWwlH4qPU3N +3UOq/OwvbqwphibRqNuDSPKbiGzKxpOI1H1HTBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAMYKhHpTSw1iAttMVRBwwECSxOy0qYFAcihUT30tHPLOkvtjq1bBu7uFB1bp/YzH8I ZFRQM5GKcFUEi0pulsbnAwpvIVoSpjuGFB1d4GGzxDRmofbxLoKScEQtO7qmjwb1DNCXeJ8s x5zF68q1ddBjKf82nI6ID3BcoPT+lOKJLZT7T9D9iOgLVnkbpy2mI+4YynTdgatpa5meb/x0 VJeq/OLkKs6B7xYV8layyTi9s30DpDrgiX0z+kq3akjbvZ/TBosrRFBfkefISM81JLxNl416 mV+sXf84/YZnA9Jq/2XDKRhFwIeBcvBslsTxBI911NBFOOuleF2IUn+Jjrv4aVP/ejmKiHjO tYh11SnI91JJopTNjn4HejGAf3ZHHfj9A0N6wgl1rGFIZYEidW6fvO9EYhX9C9Xw1AFYyXnd HnAfq2RWSThM+3IBgbhn2EwMfJ90Bqmf2CuLiuwZ/JGap47Jwl0o4FLFN+5pyt1L6z1PTMMw 77BYzJ/8bSLTZ+CQ9Bts3kfUYhvbKx4VFAJaTjnqHw/AEdibLQmPCgL7/b6D5Z2UEnvZCCgH 3nlqTqgAMGPoXEAFIZSga8SDRLS42AM14QWIlXyCLK9BKAoASHoAnvZrgawxtchTUXUcynMs ihdC4zVfwp1bpcp1PAmUFfn4+ACGbyp6g6VrddqpqAAAAAAAAA== --------------ms070207000405010707030509-- From owner-freebsd-stable@freebsd.org Wed Apr 10 13:45:50 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9270415834B6 for ; Wed, 10 Apr 2019 13:45:50 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f44.google.com (mail-lf1-f44.google.com [209.85.167.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8618E7179D for ; Wed, 10 Apr 2019 13:45:49 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f44.google.com with SMTP id u17so1893956lfi.3 for ; Wed, 10 Apr 2019 06:45:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:references:from:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=AjnkXKWHzM1u9SqGVdKEXGhI4908C2lHnDMjRf8mkZ8=; b=OADkOzcvfHSJy6CeCfPbApLow+4mhi13IbFMPLMD4WFwy2DwCBGgrtjdiyy+6HYLrz L80ByCGVslcYvRB6X9u0loDOlIAqK13Bor8aLrH95S2RjQ4h4oi7pc6wEtmP0ALbZEQ8 WtHJQpkk7Na01TRgaC66hl+V6Yt6rp4qWwcdmGgMWPMdvmLRqLUAuhg8AxpssQ3aIlDf n716erlixTusqprRXi2OSjNdlDvNbriJVbMuJHa9PmITCiOt0NLNCnWIrg8e0QdbyTgH 1IRneUUq9unxfQLjFojeb9BTrLXuCMvfGH53OHV3pw8l98QeIlqLHXmc5uM2sp/Kt8hd SOCg== X-Gm-Message-State: APjAAAXHhfe5+MHLCMMIkc/v3xiMGbw6Pt8V5P3EOSTxrUXMyImQAz2u bPG7+fgLkIYhVVhHgv3lbkZZBHRi X-Google-Smtp-Source: APXvYqw+A3soQcTP1ABBv8mVLMcLySXPMj5jPFLmmVrTo8WInW4D7TMBkomBnV66uuhEkLHFhXL4VA== X-Received: by 2002:ac2:508c:: with SMTP id f12mr10659065lfm.71.1554903941639; Wed, 10 Apr 2019 06:45:41 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id y10sm7062830lfg.44.2019.04.10.06.45.40 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Apr 2019 06:45:40 -0700 (PDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Karl Denninger , freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> From: Andriy Gapon Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABzR5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz7CwZQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryM7BTQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAcLBfAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> Date: Wed, 10 Apr 2019 16:45:39 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:52.0) Gecko/20100101 Thunderbird/52.7.0 MIME-Version: 1.0 In-Reply-To: <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 8618E7179D X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.44 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-4.32 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.95)[-0.953,0]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; MID_RHS_MATCH_FROM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; MIME_TRACE(0.00)[0:+]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[44.167.85.209.list.dnswl.org : 127.0.5.0]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-1.36)[ip: (-0.67), ipnet: 209.85.128.0/17(-3.87), asn: 15169(-2.18), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 13:45:50 -0000 On 10/04/2019 04:09, Karl Denninger wrote: > Specifically, I *explicitly* OFFLINE the disk in question, which is a > controlled operation and *should* result in a cache flush out of the ZFS > code into the drive before it is OFFLINE'd. > > This should result in the "last written" TXG that the remaining online > members have, and the one in the offline member, being consistent. > > Then I "camcontrol standby" the involved drive, which forces a writeback > cache flush and a spindown; in other words, re-ordered or not, the > on-platter data *should* be consistent with what the system thinks > happened before I yank the physical device. This may not be enough for a specific [RAID] controller and a specific configuration. It should be enough for a dumb HBA. But, for example, mrsas(9) can simply ignore the synchronize cache command (meaning neither the on-board cache is flushed nor the command is propagated to a disk). So, if you use some advanced controller it would make sense to use its own management tool to offline a disk before pulling it. I do not preclude a possibility of an issue in ZFS. But it's not the only possibility either. -- Andriy Gapon From owner-freebsd-stable@freebsd.org Wed Apr 10 14:38:59 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5963E15844F9 for ; Wed, 10 Apr 2019 14:38:59 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id C0C4D7359E for ; Wed, 10 Apr 2019 14:38:50 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 4C1CA211083 for ; Wed, 10 Apr 2019 10:38:19 -0400 (EDT) Received: from [192.168.10.26] (D16.Denninger.Net [192.168.10.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id C07CAAAD5 for ; Wed, 10 Apr 2019 09:38:18 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> Date: Wed, 10 Apr 2019 09:38:19 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms030608080001090807000405" X-Rspamd-Queue-Id: C0C4D7359E X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.58 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: px.denninger.net]; NEURAL_HAM_SHORT(-0.97)[-0.970,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.992,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.41)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.83), asn: 14061(1.75), country: US(-0.06)]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 14:38:59 -0000 This is a cryptographically signed message in MIME format. --------------ms030608080001090807000405 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/10/2019 08:45, Andriy Gapon wrote: > On 10/04/2019 04:09, Karl Denninger wrote: >> Specifically, I *explicitly* OFFLINE the disk in question, which is a >> controlled operation and *should* result in a cache flush out of the Z= FS >> code into the drive before it is OFFLINE'd. >> >> This should result in the "last written" TXG that the remaining online= >> members have, and the one in the offline member, being consistent. >> >> Then I "camcontrol standby" the involved drive, which forces a writeba= ck >> cache flush and a spindown; in other words, re-ordered or not, the >> on-platter data *should* be consistent with what the system thinks >> happened before I yank the physical device. > This may not be enough for a specific [RAID] controller and a specific > configuration. It should be enough for a dumb HBA. But, for example, = mrsas(9) > can simply ignore the synchronize cache command (meaning neither the on= -board > cache is flushed nor the command is propagated to a disk). So, if you = use some > advanced controller it would make sense to use its own management tool = to > offline a disk before pulling it. > > I do not preclude a possibility of an issue in ZFS. But it's not the o= nly > possibility either. In this specific case the adapter in question is... mps0: port 0xc000-0xc0ff mem 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c Which is indeed a "dumb" HBA (in IT mode), and Zeephod says he connects his drives via dumb on-MoBo direct SATA connections. What I don't know (yet) is if the update to firmware 20.00.07.00 in the HBA has fixed it.=C2=A0 The 11.2 and 12.0 revs of FreeBSD through some mechanism changed timing quite materially in the mps driver; prior to 11.2 I ran with a Lenovo SAS expander connected to SATA disks without any problems at all, even across actual disk failures through the years, but in 11.2 and 12.0 doing this resulted in spurious retries out of the CAM layer that allegedly came from timeouts on individual units (which looked very much like a lost command sent to the disk), but only on mirrored volume sets -- yet there were no errors reported by the drive itself, nor did either of my RaidZ2 pools (one spinning rust, one SSD) experience problems of any sort.=C2=A0=C2=A0 Flashing the HBA forward to 20.00.07.00 with the expander in resulted in the=C2=A0 *driver* (mps) tak= ing disconnects and resets instead of the targets, which in turn caused random drive fault events across all of the pools.=C2=A0 For obvious reas= ons that got backed out *fast*. Without the expander 19.00.00.00 has been stable over the last few months *except* for this circumstance, where an intentionally OFFLINE'd disk in a mirror that is brought back online after some reasonably long period of time (days to a week) results in a successful resilver but then a small number of checksum errors on that drive -- always on the one that was OFFLINEd, never on the one(s) not taken OFFLINE -- appear and are corrected when a scrub is subsequently performed.=C2=A0 I am now = on 20.00.07.00 and so far -- no problems.=C2=A0 But I've yet to do the backu= p disk swap on 20.00.07.00 (scheduled for late week or Monday) so I do not know if the 20.00.07.00 roll-forward addresses the scrub issue or not.=C2= =A0 I have no reason to believe it is involved, but given the previous "iffy" nature of 11.2 and 12.0 on 19.0 with the expander it very well might be due to what appear to be timing changes in the driver architectu= re. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms030608080001090807000405 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDEwMTQzODE5 WjBPBgkqhkiG9w0BCQQxQgRAo/0Eo3QU7U0Rt47rVYVCU/7AdRepKOmMiUfpapuV4ykDaTH4 lsdw/cXNuMNKrHT20lTAgPSSef5mCYHbL3bl0zBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgC9r8hGG2fG7K0hTeHKGuSzH0u9PKo+7cyRclAgQs+EKwbnPCHsb9Dtjj4ABFCQ/e+1 OeUKRlV1+o7tBeeln4cIkbXleRDg+p2a7N275F74qvvdm5+x8hP1b5eFpCKHnbbQ+y9blFOd P6+YudwmzAKJJygyZvBkE5LiP7SDp+XzTWJHTpiZ05+V82hmOeX181nu2fOLWFHlunHGlYyu Etq2cKd7y6srf6rkykMXbgkt3tYVN3pge18sg2F3uOasZ6zXhJ3/RZx5Li/iVvkl+RYbHQNE EsFKRRKtfSdD3LWJFLWxX0qevL7IaGF3TQ/t+trrxnHWYphEjYAGLgsYXEV80RrCd2LUmK4y XC+Z5KH8ifYlnWyTtvDAXwJVWxmdCDYvBK5Ek6CEd0DzjslSPcluXcjMROaxJ8Z7Npcr77dh Mo0noZFXst1iOY3BmlPRI/W2M+Li6Ewdo8gHqMm4YueDNvtCgjXxmkYcLJgUCR4F9ue3FzHL dsw3z+J79jpExGSUeY6IBnaQQqMebr8OoKkljcuwTxOwsys66Omb6z769F1HkycQ07XGti1b zEheSjkY1d+QOesKjI9jBpUz4Pe9po3Ga+EKlpVOgLzEqvmN2fUN/SLM2pGrkDwt/CaxxtjK 2BTwBQ5ciBSsKFd3pwNjVUtS3Q2xVow7ydE27kHBhAAAAAAAAA== --------------ms030608080001090807000405-- From owner-freebsd-stable@freebsd.org Wed Apr 10 20:05:37 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id BE327156B11B; Wed, 10 Apr 2019 20:05:37 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: from mail-yb1-f193.google.com (mail-yb1-f193.google.com [209.85.219.193]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 9AEE787FBD; Wed, 10 Apr 2019 20:05:36 +0000 (UTC) (envelope-from lwhsu.freebsd@gmail.com) Received: by mail-yb1-f193.google.com with SMTP id c15so1323730ybk.8; Wed, 10 Apr 2019 13:05:36 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:from:date:message-id:subject:to:cc; bh=a7bT3XRnKun6fSZk7mIObYmJnVFv97fTgaqYvfUoFGA=; b=XuoxjggXc5vzXBbWOLq9HgUT6mrQR92pxpkfajGaXQRoALmplRmjM4vIrkS/nRc9hp IMf6zzdgyrr6J0aViJuuTnjhFKafz/KTXGMIiolDnLjePPsxXfe1SIMwZHZnztyOQH7i mYwL3GEZzUBUMmfo+AYco2c6h//I0QsGql2+1Q9o/J8pEudB8LlP6pm8eYobF57J+W3K Ldf88VtyBskRPDT6etCtWJ5H9DwYeLP7Q9NcwY1qyIgB7TIokyK7Bs5RUYDm9Avr02Jn J12VStymrFe6IYVrEg7Ue+kKLHRisWw7GBMWa3WP76D/CW3rKGnglA+qRYWXm7ZwVgsb HATA== X-Gm-Message-State: APjAAAUwE0NzI30UpOEE/NFQPlfcwax3dmBP8ioThvQGYjteLj6SLesS NXcaZV0zRY48RpUqLTcJHEi+Vhp+bT4Ct/RiXs9/rYWi X-Google-Smtp-Source: APXvYqxsdzBXg+eQhtbrKyXE6OLH8MqD73X92z0nwOTRQ76eP3dHy1JSbetZI7VZsivzPRw/l0sX5IrrGu8Q18K9xhY= X-Received: by 2002:a25:b004:: with SMTP id q4mr36565037ybf.34.1554921439447; Wed, 10 Apr 2019 11:37:19 -0700 (PDT) MIME-Version: 1.0 From: Li-Wen Hsu Date: Thu, 11 Apr 2019 03:37:08 +0900 Message-ID: Subject: FreeBSD CI Weekly Report 2019-04-07 To: freebsd-testing@freebsd.org Cc: FreeBSD Current , freebsd-stable@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 9AEE787FBD X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of lwhsufreebsd@gmail.com designates 209.85.219.193 as permitted sender) smtp.mailfrom=lwhsufreebsd@gmail.com X-Spamd-Result: default: False [-4.07 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; MIME_TRACE(0.00)[0:+]; DMARC_NA(0.00)[freebsd.org]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; RCVD_TLS_LAST(0.00)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.84)[-0.835,0]; RCVD_IN_DNSWL_NONE(0.00)[193.219.85.209.list.dnswl.org : 127.0.5.0]; IP_SCORE(-1.22)[ipnet: 209.85.128.0/17(-3.87), asn: 15169(-2.19), country: US(-0.06)]; FORGED_SENDER(0.30)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[193.219.85.209.rep.mailspike.net : 127.0.0.17]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; TAGGED_FROM(0.00)[]; FROM_NEQ_ENVFROM(0.00)[lwhsu@freebsd.org,lwhsufreebsd@gmail.com]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 20:05:38 -0000 (bcc -current and -stable for more audience) FreeBSD CI Weekly Report 2019-04-07 =================================== Here is a summary of the FreeBSD Continuous Integration results for the period from 2019-04-01 to 2019-04-07. During this period, we have: * 1841 builds (96% passed, 4% failed) were executed on aarch64, amd64, armv6, armv7, i386, mips, mips64, powerpc, powerpc64, powerpcspe, riscv64, sparc64 architectures for head, stable/12, stable/11 branches. * 314 test runs (34.7% passed, 65% unstable, 0.3% exception) were executed on amd64, i386, riscv64 architectures for head, stable/12, stable/11 branches. * 4 doc buils (100% passed) (The statistics from experimental jobs are omitted) If any of the issues found by CI are in your area of interest or expertise please investigate the PRs listed below. The latest web version of this report is available at https://hackmd.io/s/BymrvPI_4 and archive is available at http://hackfoldr.org/freebsd-ci-report/, any help is welcome. ## Failing Tests * https://ci.freebsd.org/job/FreeBSD-head-amd64-test/ * sys.geom.class.eli.online_resize_test.online_resize https://bugs.freebsd.org/237128 * https://ci.freebsd.org/job/FreeBSD-head-i386-test/ * sys.netmap.ctrl-api-test.main https://bugs.freebsd.org/237129 * sys.opencrypto.runtests.main https://bugs.freebsd.org/237130 * sys.kern.coredump_phnum_test.coredump_phnum WIP: https://reviews.freebsd.org/D18495 * lib.libc.sys.sendfile_test.fd_positive_shm_v4 * lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v4 * lib.libc.gen.floatunditf_test.floatunditf * lib.libc.stdio.printfloat_test.hexadecimal_rounding * lib.msun.ctrig_test.test_small_inputs * lib.msun.precision_test.t_precision https://bugs.freebsd.org/236936 * https://ci.freebsd.org/job/FreeBSD-stable-12-i386-test/ * sys.netmap.ctrl-api-test.main https://bugs.freebsd.org/237129 * sys.opencrypto.runtests.main https://bugs.freebsd.org/237130 * sys.kern.coredump_phnum_test.coredump_phnum WIP: https://reviews.freebsd.org/D18495 * https://ci.freebsd.org/job/FreeBSD-stable-11-amd64-test/ * usr.bin.procstat.procstat_test.kernel_stacks * https://ci.freebsd.org/job/FreeBSD-stable-11-i386-test/ * sys.netmap.ctrl-api-test.main https://bugs.freebsd.org/237129 * sys.opencrypto.runtests.main https://bugs.freebsd.org/237130 * usr.bin.procstat.procstat_test.kernel_stacks * local.kyua.* (31 cases) * local.lutok.* (3 cases) * lib.libc.sys.sendfile_test.fd_positive_shm_v4 * lib.libc.sys.sendfile_test.hdtr_negative_bad_pointers_v4 ## Failing Tests (from experimental jobs) * https://ci.freebsd.org/job/FreeBSD-head-amd64-dtrace_test/ * common.ip.t_dtrace_contrib.tst_ipv4localsctp_ksh * common.ip.t_dtrace_contrib.tst_localsctpstate_ksh * https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/ There are ~60 failing cases, including flakey ones, see https://ci.freebsd.org/job/FreeBSD-head-amd64-test_zfs/lastCompletedBuild/testReport/ for more details ## Disabled Tests * lib.libc.sys.mmap_test.mmap_truncate_signal https://bugs.freebsd.org/211924 * sys.fs.tmpfs.mount_test.large https://bugs.freebsd.org/212862 * sys.fs.tmpfs.link_test.kqueue https://bugs.freebsd.org/213662 * sys.kqueue.libkqueue.kqueue_test.main https://bugs.freebsd.org/233586 * usr.bin.procstat.procstat_test.command_line_arguments https://bugs.freebsd.org/233587 * usr.bin.procstat.procstat_test.environment https://bugs.freebsd.org/233588 ## Oepn Issues ### New * https://bugs.freebsd.org/237077 possible race in build: /usr/src/sys/amd64/linux/linux_support.s:38:2: error: expected relocatable expression ### In progress * https://bugs.freebsd.org/236936 4 test cases failing on i386 after r345562 ### Cause build fails * [233735: Possible build race: genoffset.o /usr/src/sys/sys/types.h: error: machine/endian.h: No such file or directory](https://bugs.freebsd.org/233735) * [233769: Possible build race: ld: error: unable to find library -lgcc_s](https://bugs.freebsd.org/233769) ### Others [Tickets related to testing@](https://preview.tinyurl.com/y9maauwg) ## Other News * Some PF tests are filing for a while because of py27-pcap-0.6.5, thanks for kp@'s report and bofh@ committed update of py-pcap-0.6.6. From owner-freebsd-stable@freebsd.org Wed Apr 10 20:13:38 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B3D3A156B6E1 for ; Wed, 10 Apr 2019 20:13:38 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-vs1-xe2c.google.com (mail-vs1-xe2c.google.com [IPv6:2607:f8b0:4864:20::e2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id CD96088764 for ; Wed, 10 Apr 2019 20:13:37 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-vs1-xe2c.google.com with SMTP id i207so2125967vsd.10 for ; Wed, 10 Apr 2019 13:13:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:from:subject:date:importance; bh=By0iu3KOrBEDZeLBPTUjMW2it5l1sPvyaq9EwS6xxio=; b=bzvmf3GEegvij0RNmTqzypKgP7YfLiKvHk1MriupGW3M5rPBimLxkgve1rNKD+i+T4 tCdWfVQ3yWDwlTMb06hxXCEMBEANafwOyimXvCK76UBkCOAt7gxaMWQr2+FFk+dKUYwm /u1NdfnLP1oh+nAOTqHI9d5dwBszg2Usxsdb/JI+yc8/1KfqJ7xRbzuXngf7pX/jdGel WstHrfsUFSVo45IjYy7wnl+IU6nzKBPcu1nBvukMzrVBt25mSn/4aZi88Vdc5oLIrzlS 9f+etityifgXe3JpNYMP4naDOUINQ0+6RXJh5tsENuGtInyy1SfYdFy9meSKQIYfYkph 7Jgg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:from:subject:date :importance; bh=By0iu3KOrBEDZeLBPTUjMW2it5l1sPvyaq9EwS6xxio=; b=HogNAsUpRZFDhX1OfKzXMeOvCJAQY+QIVtqmhn7S5StXHtd5+df06iTnuwh7UcYfM+ tJxbZqelGoF9DGOgZpPSUn2GNdbDhwFlukfvay7yWZKjkESwQlFDrZD4ajI8D5vlrXyv sWcsme7yZkGAvf1ANVtvigi/SrOVPRtpI7Li8hZVpjeYa0Q6OQ3hYLwSDYdWo2OXEKKQ EFUUzHOIvyTF+Ru83vwVqUKeSm+YSNz2DZZlIbJSXt9nnFpE6orqBW8QULiX5ymCEiUm MspVhnlB30ZWKrb1r06bT79ouna50REkZxBKUwD9LJshflTMNjcpUM4C4j9q0NeymdDn Y5hA== X-Gm-Message-State: APjAAAWb6QpS5Ic5GZmBRLRAPQQ1/B/NqcVffXZGu4b9Htff91VX8xSe r2vPXUDYC77UFUSGTP2c3mJnx0eP X-Google-Smtp-Source: APXvYqyiNcmDj+RrPaVQpl1Dv5u5XBKnFGVdtcmdxvJqAlDhHWq9KtkZsYpg8BkTtJ+SnFyuqlZGfw== X-Received: by 2002:a67:fe4f:: with SMTP id m15mr20635962vsr.202.1554927215863; Wed, 10 Apr 2019 13:13:35 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.119? (git.mayberryinv.com. [63.143.111.202]) by smtp.gmail.com with ESMTPSA id d193sm5674467vkd.41.2019.04.10.13.13.34 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Apr 2019 13:13:35 -0700 (PDT) Message-ID: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> MIME-Version: 1.0 To: "freebsd-stable@freebsd.org" From: Software Info Subject: Crontab Question Date: Wed, 10 Apr 2019 15:13:34 -0500 Importance: normal X-Priority: 3 X-Rspamd-Queue-Id: CD96088764 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=bzvmf3GE; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::e2c as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-5.97 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.95)[-0.952,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-3.00)[ip: (-9.82), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.19), country: US(-0.06)]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[c.2.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; TO_DN_EQ_ADDR_ALL(0.00)[] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 20:13:39 -0000 Hi All I am trying to schedule cron to run a script. The script is in my home dire= ctory and so I added my home directory to the path file in /etc/crontab bel= ow. PATH=3D/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:~/bin:/= home:/home/me This is the crontab entry for the scheduled task below. 49 14 * * 1-5 root myscript.sh myscript.sh grabs a file from another directory if it is there. If not, it = says =E2=80=9Cfile not uploaded=E2=80=9D. If the file is there, it copies i= t to my home directory, strips email addresses out of it and uses mailx to = send emai to those users. I keep getting the error that a number of files i= n my home directory are missing but they are not.=20 Please see errors below mv: rename *.csv to listing.csv: No such file or directory grep: listing.csv: No such file or directory /home/me/ipo-script.sh: cannot open body.txt: No such file or directory mv: rename /home/me/listing.txt to /home/me/listing.txt-10-04-19-1435: No s= uch file or directory mv: rename /home/me/listing.csv to /home/me/listing.csv-10-04-19-1435: No s= uch file or directory mv: rename /home/me/listing.txt-10-04-19-1435 to /home/me/IPO-Backup-Files/= listing.txt-10-04-19-1435: No such file or directory mv: rename /home/me/listing.csv-10-04-19-1435 to /home/me/IPO-Backup-Files/= listing.csv-10-04-19-1435: No such file or directory Because I added my home directory to the path in crontab, I am at a loss to= explain why this is still happening. Anyone have any ideas? Would really a= ppreciate some help. Kind Regards SI Sent from Mail for Windows 10 From owner-freebsd-stable@freebsd.org Wed Apr 10 20:50:20 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C975D156C9A0 for ; Wed, 10 Apr 2019 20:50:19 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: from mail-vk1-xa2f.google.com (mail-vk1-xa2f.google.com [IPv6:2607:f8b0:4864:20::a2f]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A5F9F8A509 for ; Wed, 10 Apr 2019 20:50:17 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: by mail-vk1-xa2f.google.com with SMTP id q189so878513vkq.11 for ; Wed, 10 Apr 2019 13:50:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chen-org-nz.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=BFr4PGb8VwU+n3heyfuLUZOEUDer6l2bFS4IyNA6Z30=; b=GUCOIAFVZi6N+Up60pkleJ0d1xn56QCa1TnXw/i7xcFb9X1Xw+gpKhAudxvZUo0k+t aFFo175w1/2ZxsXjqqIl5ESRDw12YSwniuBW+zZOJ7/96IijE51DTTDLQHL98Q//2rN3 8+AYaohtVG5VVNPsDApu6ICX7j4TNRpIneGJExxLEmDJROHyHPz6wk+EvkyyEPHfEb8R S+1FCgylgkXgStoIEyE+A8MraUORVELDYOZ4JwyDBd3cE1112JRt2akphNel3o3oYi70 voKblYy0ebLkG1nMm8n3hx1ZcJkkDq5hqkFCUtKoXoGtoDe2OHlWri1qlza/WB2BdrZ1 VpHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=BFr4PGb8VwU+n3heyfuLUZOEUDer6l2bFS4IyNA6Z30=; b=UPrFjolwoGEnhvjq33/yv6CSfKZ4tSuOxPGYQ9qEnz5aXIpurXjjrlPInDeqTchydY pF3XVyBfWBpuT2t2nK6WycLGFcuscQ/TkEFoLQbZHgcA+DiUwTgOzQG8zsdPaPTUWoIo FYU+nNEvv7x9euqBGQZPuuVQAzrf7unr+Q11Gz55r3BjG/U+K90xvRrMJMlv3ry8Non4 C7VKqLD8caKaOVOwZ6tsm5lz8ktBtijiqQcXM5hJOnj4+spJNAHhcQfbnXE6C7Cg84h0 0BUksCyK69+a3yp7kpk69Ci66LBACacWOcCqiFz1bck5m0hXXtEROnuIh8CEKnYAY0bj p1zg== X-Gm-Message-State: APjAAAXNeIe2UbtOLTRuUTdVqYM2w3yg/2BqV1BCXO9aalKUSvV3pNEo yXT/96ARDIwjjYxYw39cLd2pgbGWxlJFHeoWOT2XxA== X-Google-Smtp-Source: APXvYqwaBdMKbrjIlGgJafKfB+F/sqLP0Qq5oSHICxq1ty5qEjq5byyWttE095gQ5DaP/7MWqPw2p2xnYlsSh5FZTpA= X-Received: by 2002:a1f:b297:: with SMTP id b145mr24835171vkf.74.1554929416765; Wed, 10 Apr 2019 13:50:16 -0700 (PDT) MIME-Version: 1.0 References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> In-Reply-To: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> From: Jonathan Chen Date: Thu, 11 Apr 2019 08:50:00 +1200 Message-ID: Subject: Re: Crontab Question To: Software Info Cc: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: A5F9F8A509 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chen-org-nz.20150623.gappssmtp.com header.s=20150623 header.b=GUCOIAFV X-Spamd-Result: default: False [-6.13 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[chen-org-nz.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.94)[-0.940,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[chen.org.nz]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[chen-org-nz.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[f.2.a.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[alt1.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.88)[ip: (-9.19), ipnet: 2607:f8b0::/32(-2.96), asn: 15169(-2.19), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 20:50:20 -0000 On Thu, 11 Apr 2019 at 08:18, Software Info wro= te: > > Hi All > I am trying to schedule cron to run a script. The script is in my home di= rectory and so I added my home directory to the path file in /etc/crontab b= elow. > PATH=3D/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:~/bin= :/home:/home/me > > This is the crontab entry for the scheduled task below. > 49 14 * * 1-5 root myscript.sh > > myscript.sh grabs a file from another directory if it is there. If not, i= t says =E2=80=9Cfile not uploaded=E2=80=9D. If the file is there, it copies= it to my home directory, strips email addresses out of it and uses mailx t= o send emai to those users. I keep getting the error that a number of files= in my home directory are missing but they are not. > > Please see errors below > mv: rename *.csv to listing.csv: No such file or directory > grep: listing.csv: No such file or directory > /home/me/ipo-script.sh: cannot open body.txt: No such file or directory > mv: rename /home/me/listing.txt to /home/me/listing.txt-10-04-19-1435: No= such file or directory > mv: rename /home/me/listing.csv to /home/me/listing.csv-10-04-19-1435: No= such file or directory > mv: rename /home/me/listing.txt-10-04-19-1435 to /home/me/IPO-Backup-File= s/listing.txt-10-04-19-1435: No such file or directory > mv: rename /home/me/listing.csv-10-04-19-1435 to /home/me/IPO-Backup-File= s/listing.csv-10-04-19-1435: No such file or directory > > Because I added my home directory to the path in crontab, I am at a loss = to explain why this is still happening. Anyone have any ideas? Would really= appreciate some help. You are assuming that the script starts in your home directory. It doesn't, hence: "mv: rename *.csv to listing.csv: No such file or directory" --=20 Jonathan Chen From owner-freebsd-stable@freebsd.org Wed Apr 10 21:14:46 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id C8447156D438 for ; Wed, 10 Apr 2019 21:14:46 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-ua1-x931.google.com (mail-ua1-x931.google.com [IPv6:2607:f8b0:4864:20::931]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C2C08B78C for ; Wed, 10 Apr 2019 21:14:45 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-ua1-x931.google.com with SMTP id v7so1276877uak.13 for ; Wed, 10 Apr 2019 14:14:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:cc:from:subject:date:importance :in-reply-to:references; bh=ZNU4gJYKgVzZGZo5Y9a+P2oNR53dMNWhCbeK/iaFFNY=; b=Pu/EdiNzx9PSir/ujab3coMevh++OIEMkTqgQm4WZ6rxgrA584WnkEy3nCoGrlg3DH eXQZSjRZCCrN2E/ro2NUGxdVxZ9JEeRCvrrFqyGkRwUyvwOSC0Ai1RHZAXJ/NjYkg0wg r9Xk+8jDmwGlaHEMcCSbgc2xm8qgBcTmmzDLuGTl5Q5dvhZIXfBE/0YNdXUq0muKMR6R CinGbkJDzPcdas6NtVaocXcvnx4AWUc2nsfDpnkOzKomRpAm5SXFVEEwgQ3iwicdmSa9 ikVPbuoCc0Xa4p3n/vOUmt4RQ5WJau8HnP2uq8sgq7QAXxxMc/qp0/bz20yodUb88t26 MoaQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:cc:from:subject:date :importance:in-reply-to:references; bh=ZNU4gJYKgVzZGZo5Y9a+P2oNR53dMNWhCbeK/iaFFNY=; b=SLhgCvLcR0Qj0aTxvWre+J5WAP1Et0kISpkElKKWvsbS9LplKzsELO6GS02wrybj/E Z/Sd01qlDuaR6v/aFaAXWSrkLx6zVucIYNeTluFQJr4d95JMyBHXvWTZKQq3CeBLNMNA gYWKjIBThNPDg4D8aGeAh8Ljtta3ycVt9bQEaatVgeYUgpO7UrjmyY/lArG5IBqwWok2 1Z1e89i5z+TOgekcRJTsRLEaWdd4rR5CDGxRW3vNJTyfxnO0kP/muwd6nCfqTsmqjlVq QqjYPNEROFUlftpUKMM7350nGBlbYNNq5EbNgZNryCetxPHG5+gao+Rk7T3QfPL9QQpS evEQ== X-Gm-Message-State: APjAAAUMsIx9VGCDXM031KVOl6h5E0/7TusQO3Z4+vOwC9QFsSj26JXg 672h0ZW4zPG+kroCtP4a+bhcIYG8 X-Google-Smtp-Source: APXvYqyWOjs1tC1ws5uE5TtbTrgdfS1w2Msq2TFulRHrkLT4jPadsHgXRNh9BixX3J9s8cLm1wrsSQ== X-Received: by 2002:ab0:2885:: with SMTP id s5mr24000269uap.39.1554930884499; Wed, 10 Apr 2019 14:14:44 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.119? (mail.mayberryinv.com. [65.183.3.94]) by smtp.gmail.com with ESMTPSA id d191sm4468621vka.39.2019.04.10.14.14.43 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Apr 2019 14:14:43 -0700 (PDT) Message-ID: <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> MIME-Version: 1.0 To: Jonathan Chen Cc: "freebsd-stable@freebsd.org" From: Software Info Subject: RE: Crontab Question Date: Wed, 10 Apr 2019 16:14:43 -0500 Importance: normal X-Priority: 3 In-Reply-To: References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> X-Rspamd-Queue-Id: 2C2C08B78C X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=Pu/EdiNz; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::931 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-5.88 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[3]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.985,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.3.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.88)[ip: (-9.22), ipnet: 2607:f8b0::/32(-2.96), asn: 15169(-2.19), country: US(-0.06)] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 21:14:47 -0000 OK. So although the script is located in my home directory, it doesn=E2=80= =99t start there? Sorry but I don=E2=80=99t quite understand. Could you ex= plain a little further please? Regards SI Sent from Mail for Windows 10 From: Jonathan Chen Sent: Wednesday, April 10, 2019 3:50 PM To: Software Info Cc: freebsd-stable@freebsd.org Subject: Re: Crontab Question On Thu, 11 Apr 2019 at 08:18, Software Info wro= te: > > Hi All > I am trying to schedule cron to run a script. The script is in my home di= rectory and so I added my home directory to the path file in /etc/crontab b= elow. > PATH=3D/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:~/bin= :/home:/home/me > > This is the crontab entry for the scheduled task below. > 49 14 * * 1-5 root myscript.sh > > myscript.sh grabs a file from another directory if it is there. If not, i= t says =E2=80=9Cfile not uploaded=E2=80=9D. If the file is there, it copies= it to my home directory, strips email addresses out of it and uses mailx t= o send emai to those users. I keep getting the error that a number of files= in my home directory are missing but they are not. > > Please see errors below > mv: rename *.csv to listing.csv: No such file or directory > grep: listing.csv: No such file or directory > /home/me/ipo-script.sh: cannot open body.txt: No such file or directory > mv: rename /home/me/listing.txt to /home/me/listing.txt-10-04-19-1435: No= such file or directory > mv: rename /home/me/listing.csv to /home/me/listing.csv-10-04-19-1435: No= such file or directory > mv: rename /home/me/listing.txt-10-04-19-1435 to /home/me/IPO-Backup-File= s/listing.txt-10-04-19-1435: No such file or directory > mv: rename /home/me/listing.csv-10-04-19-1435 to /home/me/IPO-Backup-File= s/listing.csv-10-04-19-1435: No such file or directory > > Because I added my home directory to the path in crontab, I am at a loss = to explain why this is still happening. Anyone have any ideas? Would really= appreciate some help. You are assuming that the script starts in your home directory. It doesn't, hence: "mv: rename *.csv to listing.csv: No such file or directory" --=20 Jonathan Chen From owner-freebsd-stable@freebsd.org Wed Apr 10 21:23:34 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 31B3F156D82B for ; Wed, 10 Apr 2019 21:23:34 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: from mail-vs1-xe2c.google.com (mail-vs1-xe2c.google.com [IPv6:2607:f8b0:4864:20::e2c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 575478BD1C for ; Wed, 10 Apr 2019 21:23:33 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: by mail-vs1-xe2c.google.com with SMTP id g187so2237399vsc.8 for ; Wed, 10 Apr 2019 14:23:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chen-org-nz.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=aVH+8Ra1dN5yydykBq1F95RN3Ljmffc3kuGhBejrNRY=; b=QAWB02GcSlIbYWlqclWWaRol0GmlVCy+3tXgf2fOhYYbRFG+cWIO+XTSAYzTiDzTPP CUG8/ZsTVG6ITeItj7nf1o055bmiIFZfOLTmNyiLz7wIBa45mVDY6voOeWg+Oc68xp3s eMCne6jldIy5CfmN3mZ13UrMMBOJAbDllNhSMvOi3Fk/xZh/NG9bxqL0KaWfDvzoZAY4 iO4MPX0a9NaSxXHH/rv/sVJRJeTd3VLC4FAjR5qXpMa/+W4pRqLGMKBrUQ4c79FTVpGH DJCqC/lXhlMhYhQxhEikMlV/UGrhHDCJ3dpvTFNqxO7x20Rh3+A/VkeKD7PTuejCrR83 T0hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=aVH+8Ra1dN5yydykBq1F95RN3Ljmffc3kuGhBejrNRY=; b=jd/0jOBlpPQlPzLf5OH3s2hW8O0JzJxBiWFjSKUb1M43IOfQlVgCIwvWeNsQeYWsES mtcs3WldsgCETseiQUBKWqZkCH/Ydt+wVy4sEeeoy65jdyaxvnnkS7nzgZvtjriTe6u4 NRK4Bnes5UKuO8I0nOsIkNEOlyBNkTP4EQmWDtwF3v4vwgN6QVtyR2ldvjTWUglxLM/v NZErNOYaIusneiFgAnfCGH/bsSxMBttFnzGwTtzMDE9VWhNeoB4HB4knBsjyqfZEaQxm 3OFU8JfF+P+en1Uy7GhtBA+J/iui+U+07b52YxbZpjm2Nb1kopPpX6Cce8GsenqDi9C8 YA4A== X-Gm-Message-State: APjAAAUcFl5iBWzzd5pitBwaYeObzA1swL0qQPpIb7rIgH9BEt/w0GKf T3jf79xPcoVTp8bIryaFumWnRAiXwwqNdiNuOjuLBg== X-Google-Smtp-Source: APXvYqxXQqYthWA1yBeTrXJMn2Qtt+khShadjKEvJP9eqrivTAWKOm273R0YwKwYYbaqSfyeOodB4VVPsYSXANZZ/V0= X-Received: by 2002:a67:df91:: with SMTP id x17mr21770296vsk.76.1554931412660; Wed, 10 Apr 2019 14:23:32 -0700 (PDT) MIME-Version: 1.0 References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> In-Reply-To: <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> From: Jonathan Chen Date: Thu, 11 Apr 2019 09:23:16 +1200 Message-ID: Subject: Re: Crontab Question To: Software Info Cc: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 575478BD1C X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chen-org-nz.20150623.gappssmtp.com header.s=20150623 header.b=QAWB02Gc X-Spamd-Result: default: False [-6.23 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[chen-org-nz.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.91)[-0.913,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[chen.org.nz]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[chen-org-nz.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[c.2.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-3.00)[ip: (-9.81), ipnet: 2607:f8b0::/32(-2.96), asn: 15169(-2.19), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 21:23:34 -0000 On Thu, 11 Apr 2019 at 09:14, Software Info wro= te: > > OK. So although the script is located in my home directory, it doesn=E2= =80=99t start there? Correct. You cannot make any assumptions about the environment. --=20 Jonathan Chen From owner-freebsd-stable@freebsd.org Wed Apr 10 21:34:52 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 05373156DD6F for ; Wed, 10 Apr 2019 21:34:52 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-ua1-x934.google.com (mail-ua1-x934.google.com [IPv6:2607:f8b0:4864:20::934]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 722268C5C3 for ; Wed, 10 Apr 2019 21:34:50 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-ua1-x934.google.com with SMTP id l17so1316349uar.4 for ; Wed, 10 Apr 2019 14:34:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:cc:from:subject:date:importance :in-reply-to:references; bh=Vau+3KRnbadTGoQ10kGi0KoZH1m8oEbEvHF3Tku67D0=; b=sqqFiySDEQZvHAQWZI9ZRM9IdGMLd9l7CC5HiZ5ORXx/XbHwtuMEVLHuc6kVWW2/mF CrtRnd5pVsqs66Oi321JjoQETfEUrlZfNFurQ19WOPywSEoy9tyE1DEFUU2zGgDEUZV5 dJw2/fXMa0BmHQ0lScC/zNxYrynL9ZTTEPCGZGQZPKgWlzyCyQLv92LbsuocxngdI/oD R7j4v19Qw52uY+eECNqvvpv4d4PfvywurgXmAxedYvVlWCe7i37VUcR4WAat2MszU0Sv p44milMXCE2Z0yD21hSe9uBPfLFWkSuYrUbdShmWngKKol+R+WNhKmr9bT+0rNAKTKSb 9hZA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:cc:from:subject:date :importance:in-reply-to:references; bh=Vau+3KRnbadTGoQ10kGi0KoZH1m8oEbEvHF3Tku67D0=; b=ph1acN7yDnfZ8fm3ni/cfhbSMIHgA/3Jh2ayyBihLowyxZ80RqOblLk/cyB/6Kzfa3 Vfs+3GoHlV1MtCHfFGE+RsdW4TN31y1yJ/zSIhJJLTZLjdo7XwBDqlhPxKAey4wjCm5u pH3iwcr65wpDy5dx+qRDcsjQ4acdSuvl7BLHHOGETjUqk8q498UGCLFdcu4nZPbCx/fH x4UELVJUahPDxexXYPIMFDZJKdHZMO2OVR6PflFhitpXbrXpvCZCnrjbEl5xvEl+fI4e u9DI3LDyFpjTAAuKYy1vtsLuikeEkqM1GsjcKKaAeccew1bJsV6LyWQkkNmd+UPpxX8b vy1w== X-Gm-Message-State: APjAAAXjt0DqMtvqWqP7henuGbdUN8WmImq/5sT6qB+tiesPRDo9TItQ 7R1z2EJIBP23/Rjfea2o4Q02LM/R X-Google-Smtp-Source: APXvYqyiEL9ige235MVfrblmJ/pQOkR/h9E5pyWHEUxBswGW8dvdQBR1/ajSvxqnAoAwt9gH/E6WBQ== X-Received: by 2002:ab0:2341:: with SMTP id h1mr23408865uao.127.1554932089885; Wed, 10 Apr 2019 14:34:49 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.119? (mail.mayberryinv.com. [65.183.3.94]) by smtp.gmail.com with ESMTPSA id u10sm17438627vku.34.2019.04.10.14.34.48 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 10 Apr 2019 14:34:49 -0700 (PDT) Message-ID: <5cae6179.1c69fb81.12661.3a08@mx.google.com> MIME-Version: 1.0 To: Jonathan Chen Cc: "freebsd-stable@freebsd.org" From: Software Info Subject: RE: Crontab Question Date: Wed, 10 Apr 2019 16:34:49 -0500 Importance: normal X-Priority: 3 In-Reply-To: References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> X-Rspamd-Queue-Id: 722268C5C3 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=sqqFiySD; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::934 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-5.90 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[1]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.983,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.3.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.90)[ip: (-9.31), ipnet: 2607:f8b0::/32(-2.96), asn: 15169(-2.19), country: US(-0.06)] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 21:34:52 -0000 I see. I had however copied the output of env to the etc/crontab PATH line.= Wouldn=E2=80=99t that care for an environment issue though? Regards SI Sent from Mail for Windows 10 From: Jonathan Chen Sent: Wednesday, April 10, 2019 4:23 PM To: Software Info Cc: freebsd-stable@freebsd.org Subject: Re: Crontab Question On Thu, 11 Apr 2019 at 09:14, Software Info wro= te: > > OK. So although the script is located in my home directory, it doesn=E2= =80=99t start there? Correct. You cannot make any assumptions about the environment. --=20 Jonathan Chen From owner-freebsd-stable@freebsd.org Wed Apr 10 21:40:49 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 908F6156E1D5 for ; Wed, 10 Apr 2019 21:40:49 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from scully.mintsol.com (scully.mintsol.com [199.182.77.206]) by mx1.freebsd.org (Postfix) with ESMTP id 95EB78CA83 for ; Wed, 10 Apr 2019 21:40:48 +0000 (UTC) (envelope-from wfc@mintsol.com) Received: from mintsol.com (officecc.mintsol.com [96.85.114.33]) by scully.mintsol.com with esmtp; Wed, 10 Apr 2019 17:40:41 -0400 id 00ACDC00.000000005CAE62DA.0000F3D6 Received: from localhost (localhost [127.0.0.1]) (IDENT: uid 1002) by mintsol.com with esmtp; Wed, 10 Apr 2019 17:40:42 -0400 id 0000080E.5CAE62DA.00004781 Date: Wed, 10 Apr 2019 17:40:42 -0400 (EDT) From: Walter Cramer To: Software Info cc: Jonathan Chen , "freebsd-stable@freebsd.org" Subject: RE: Crontab Question In-Reply-To: <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> Message-ID: <20190410172638.C14867@mulder.mintsol.com> References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> MIME-Version: 1.0 Content-ID: <20190410173428.C14867@mulder.mintsol.com> X-Rspamd-Queue-Id: 95EB78CA83 X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of wfc@mintsol.com designates 199.182.77.206 as permitted sender) smtp.mailfrom=wfc@mintsol.com X-Spamd-Result: default: False [-1.43 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; NEURAL_HAM_MEDIUM(-0.94)[-0.939,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+a:scully.mintsol.com]; NEURAL_HAM_LONG(-0.96)[-0.963,0]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; DMARC_NA(0.00)[mintsol.com]; TO_DN_SOME(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[bmx01.pofox.com]; CTYPE_MIXED_BOGUS(1.00)[]; NEURAL_HAM_SHORT(-0.30)[-0.303,0]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:22768, ipnet:199.182.77.0/24, country:US]; IP_SCORE(-0.01)[country: US(-0.06)] Content-Type: TEXT/PLAIN; CHARSET=X-UNKNOWN; format=flowed Content-Transfer-Encoding: QUOTED-PRINTABLE X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 21:40:49 -0000 On Wed, 10 Apr 2019, Software Info wrote: > OK. So although the script is located in my home directory, it doesn=E2= =80=99t=20 > start there? Sorry but I don=E2=80=99t quite understand. Could you expla= in a=20 > little further please? Both 'cp' and 'ls' are located in /bin. But if I run the 'ls' command in= =20 /root, 'ls' can't find 'cp' (unless I tell it where to look) - even though= =20 /bin *is* in my PATH - server7:/root # ls cp ls: cp: No such file or directory server7:/root # ls /bin/cp /bin/cp Where the system looks for *commands*, to execute, is different from where= =20 it looks for other files, which those commands use. The latter is=20 generally only the current directory (unless you tell it otherwise).=20 When cron runs a script as root, "current directory" will be /root. BUT - for security and other reasons, it would be better to have cron run= =20 your script as you (not root), and as '/home/me/myscript' (instead of=20 adding your home directory to PATH in /etc/crontab). -Walter From owner-freebsd-stable@freebsd.org Wed Apr 10 21:43:37 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 59767156E564 for ; Wed, 10 Apr 2019 21:43:37 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: from mail-vs1-xe2b.google.com (mail-vs1-xe2b.google.com [IPv6:2607:f8b0:4864:20::e2b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 6DE9A8CF85 for ; Wed, 10 Apr 2019 21:43:36 +0000 (UTC) (envelope-from jonc@chen.org.nz) Received: by mail-vs1-xe2b.google.com with SMTP id g187so2265307vsc.8 for ; Wed, 10 Apr 2019 14:43:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=chen-org-nz.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=st5EF6Toj4oaftpjqMp60Jwg7uVMLOwxgM2q05BzpoY=; b=BHbrJEJoZ2erusel0A94uhdfWRf+4Ud4FvxI0vJd3v3sYHo1hxG8k8L/5MnirVY9b0 WQV5xZvXUVpbGsNQp82sIhh9Wa0budkXfF5RIVKMbXQSxeIvBKYZ3kLvWFIUOUOkfqb1 h3tY+plgS6zseUnJCZ8eLTONmv8tA5vnYfCoY8lSogzRQ4wTMYXV4OHlqwdqtUldGHnt /ZujFsP+2OIjUTVdXtBa2CQacSd3URCEmqnSTUd/LVfyWAFIGpFNO8SvDLBFh/m7vtmA F9dbTzmdcvUJFRw1WJW8JRSIkU7TuiDcpwL3WB08MELkoAFBhALVASZoEKPQEOqQLGhT IKKQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=st5EF6Toj4oaftpjqMp60Jwg7uVMLOwxgM2q05BzpoY=; b=EYv/2kJhGjT5+bMo3bnlDfyDk/uYJezO5FnoRv76zN6kFbHi01NeDPLnni23ma02Lu +KmkyNWupKerCth8HsBd52jieJH452TsopF536+qm8p/Q0QnzHHPxkQMMWSuDwixGiWu WfEdS0g9/yZsXnGudaJT0+v+X5HUVdn5pMz7cs8S5JDLQtyWLPMXiHQ6x/Gx02PkeF+I oZfq0ohW9EvurH5I3VG6Dfo4wQxomdQwQ2sHMsvmG0X3erHTlcZoL1gSqGfl21sKFCMp k5aJ7kTQJ+lt8NjRz8Vp/PMmQ1hj0/EGOVpVyz6P+Ov0Z5Urfbq0Up0J41BUE5aKCWmn 8+eA== X-Gm-Message-State: APjAAAUFA7vgcv9DYy9CzZIYObCgSMoPO2KMW6eOecylj5Zc/UA7v2Jl GFDCnomMApxhbj4xxst/cd9+TQaFwagJjUUP5v3mEw== X-Google-Smtp-Source: APXvYqwTYBfq8nIPn/8p3EZvyUA+PSOidDp4kKxr+5TcVNRlp0C31mrjo3ztD8+lFb3cQeZptGp3Obd3l2Sjl1NKdcY= X-Received: by 2002:a67:844d:: with SMTP id g74mr26014840vsd.40.1554932615491; Wed, 10 Apr 2019 14:43:35 -0700 (PDT) MIME-Version: 1.0 References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> <5cae6179.1c69fb81.12661.3a08@mx.google.com> In-Reply-To: <5cae6179.1c69fb81.12661.3a08@mx.google.com> From: Jonathan Chen Date: Thu, 11 Apr 2019 09:43:19 +1200 Message-ID: Subject: Re: Crontab Question To: Software Info Cc: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 6DE9A8CF85 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=chen-org-nz.20150623.gappssmtp.com header.s=20150623 header.b=BHbrJEJo X-Spamd-Result: default: False [-6.20 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[chen-org-nz.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.91)[-0.913,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[chen.org.nz]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[chen-org-nz.20150623.gappssmtp.com:+]; MX_GOOD(-0.01)[cached: alt1.aspmx.l.google.com]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[b.2.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-2.98)[ip: (-9.68), ipnet: 2607:f8b0::/32(-2.96), asn: 15169(-2.19), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 21:43:37 -0000 On Thu, 11 Apr 2019 at 09:34, Software Info wro= te: > > I see. I had however copied the output of env to the etc/crontab PATH lin= e. Wouldn=E2=80=99t that care for an environment issue though? When I say "environment", I mean it in the generic sense; including working-directory. However, best practise is to keep /etc/crontab as minimal as possible, and to set up environ(7) within the invoked script-file. Security-wise, you should just keep PATH in /etc/crontab to the standard, and invoke your script with: * * * * * user /path/to/my/script This is especially important when running as root. Cheers. --=20 Jonathan Chen From owner-freebsd-stable@freebsd.org Wed Apr 10 21:44:12 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id F04DE156E5A1 for ; Wed, 10 Apr 2019 21:44:11 +0000 (UTC) (envelope-from merlyn@geeks.org) Received: from mail.geeks.org (mail.geeks.org [IPv6:2001:4980:3333:1::1]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 7FBA78CFB3 for ; Wed, 10 Apr 2019 21:44:11 +0000 (UTC) (envelope-from merlyn@geeks.org) Received: from mail.geeks.org (localhost [127.0.0.1]) by after-clamsmtpd.geeks.org (Postfix) with ESMTP id 1D616110221 for ; Wed, 10 Apr 2019 16:44:04 -0500 (CDT) Received: by mail.geeks.org (Postfix, from userid 1003) id F0A4711020B; Wed, 10 Apr 2019 16:44:03 -0500 (CDT) Date: Wed, 10 Apr 2019 16:44:03 -0500 From: Doug McIntyre To: "freebsd-stable@freebsd.org" Subject: Re: Crontab Question Message-ID: <20190410214403.GA18561@geeks.org> References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> <5cae6179.1c69fb81.12661.3a08@mx.google.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5cae6179.1c69fb81.12661.3a08@mx.google.com> User-Agent: Mutt/1.10.1 (2018-07-13) X-Virus-Scanned: ClamAV using ClamSMTP X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 21:44:12 -0000 No. Your CWD can't be copied to a PATH variable. For cronjobs, assume nothing. Hard code all path names. Assume the only things in the PATH are /bin:/usr/bin, otherwise give full path names to the programs you want to run. Assume no environmental variables are set, assume you are on the most basic setup possible (because you are). On Wed, Apr 10, 2019 at 04:34:49PM -0500, Software Info wrote: > I see. I had however copied the output of env to the etc/crontab PATH line. Wouldn’t that care for an environment issue though? > > > Regards > SI > > Sent from Mail for Windows 10 > > From: Jonathan Chen > Sent: Wednesday, April 10, 2019 4:23 PM > To: Software Info > Cc: freebsd-stable@freebsd.org > Subject: Re: Crontab Question > > On Thu, 11 Apr 2019 at 09:14, Software Info wrote: > > > > OK. So although the script is located in my home directory, it doesn’t start there? > > Correct. You cannot make any assumptions about the environment. > -- > Jonathan Chen > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Wed Apr 10 21:47:22 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CD39E156E921 for ; Wed, 10 Apr 2019 21:47:22 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from mx.catwhisker.org (mx.catwhisker.org [198.144.209.73]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 9CEAF8D753 for ; Wed, 10 Apr 2019 21:47:21 +0000 (UTC) (envelope-from david@catwhisker.org) Received: from albert.catwhisker.org (localhost [127.0.0.1]) by albert.catwhisker.org (8.15.2/8.15.2) with ESMTP id x3ALdDEB052433; Wed, 10 Apr 2019 21:39:13 GMT (envelope-from david@albert.catwhisker.org) Received: (from david@localhost) by albert.catwhisker.org (8.15.2/8.15.2/Submit) id x3ALdDaq052432; Wed, 10 Apr 2019 14:39:13 -0700 (PDT) (envelope-from david) Date: Wed, 10 Apr 2019 14:39:13 -0700 From: David Wolfskill To: Software Info Cc: "freebsd-stable@freebsd.org" Subject: Re: Crontab Question Message-ID: <20190410213913.GS1185@albert.catwhisker.org> Reply-To: stable@freebsd.org Mail-Followup-To: stable@freebsd.org, Software Info , "freebsd-stable@freebsd.org" References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> <5cae6179.1c69fb81.12661.3a08@mx.google.com> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="D80pe+u4shrTMH8N" Content-Disposition: inline In-Reply-To: <5cae6179.1c69fb81.12661.3a08@mx.google.com> User-Agent: Mutt/1.11.4 (2019-03-13) X-Rspamd-Queue-Id: 9CEAF8D753 X-Spamd-Bar: --------- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of david@catwhisker.org designates 198.144.209.73 as permitted sender) smtp.mailfrom=david@catwhisker.org X-Spamd-Result: default: False [-9.05 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_REPLYTO(0.00)[stable@freebsd.org]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:198.144.209.73]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,text/plain]; DMARC_NA(0.00)[catwhisker.org]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: mx.catwhisker.org]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[73.209.144.198.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.98)[-0.984,0]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; SIGNED_PGP(-2.00)[]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:7961, ipnet:198.144.192.0/19, country:US]; RCVD_TLS_LAST(0.00)[]; IP_SCORE(-3.66)[ip: (-9.66), ipnet: 198.144.192.0/19(-4.76), asn: 7961(-3.81), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 Apr 2019 21:47:23 -0000 --D80pe+u4shrTMH8N Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Apr 10, 2019 at 04:34:49PM -0500, Software Info wrote: > I see. I had however copied the output of env to the etc/crontab PATH lin= e. Wouldn=E2=80=99t that care for an environment issue though? >=20 >=20 > Regards > SI > .... The execution search path has no (direct) bearing on the current working directory (and vice versa). Your script cannot assume any particular current working directory. If you need to set it, do so (in the script itself). Peace, david --=20 David H. Wolfskill david@catwhisker.org "The President is a coward." -- Kirsten Gillibrand See http://www.catwhisker.org/~david/publickey.gpg for my public key. --D80pe+u4shrTMH8N Content-Type: application/pgp-signature; name="signature.asc" -----BEGIN PGP SIGNATURE----- iQGTBAEBCgB9FiEE4owz2QxMJyaxAefyQLJg+bY2PckFAlyuYoFfFIAAAAAALgAo aXNzdWVyLWZwckBub3RhdGlvbnMub3BlbnBncC5maWZ0aGhvcnNlbWFuLm5ldEUy OEMzM0Q5MEM0QzI3MjZCMTAxRTdGMjQwQjI2MEY5QjYzNjNEQzkACgkQQLJg+bY2 Pcl3NQf/eJgtQefmLtGaM7fG3N0bhGRyqDyVWdCHNMA3o4lQUecjHM6IP22Y7UJs 8a+j37pFimsd8KulY2H0LrkzXj4JlCBdIA8Y5V7UdKVSF4vvupynm5OW3fg1yvXq Kgj41JKqJ/SRRiv5AKBtOgJUXdAfQN5315/vgvWZnlcoQMl4vV/Nmo0gEs4i8+tK MsJ0KLebd7FKjqeZpCgdlTNKXL18FqhZ8Q+2CFsQygLeH48C8AuC3VZHxwne02eX XSQeOCfR/jmWKdG7xyLwPk3l62aH+GR3h/nZ8el2ARmN/7cXY1QQ0DniD5SId2Tw /Eq3tmp/yuNYi1p3exMbjoX7aE9GAA== =cjip -----END PGP SIGNATURE----- --D80pe+u4shrTMH8N-- From owner-freebsd-stable@freebsd.org Thu Apr 11 03:47:46 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 55EC01576F28 for ; Thu, 11 Apr 2019 03:47:46 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [IPv6:2001:19f0:300:2185:123::1]) (using TLSv1 with cipher DHE-RSA-AES256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id CCF0872278 for ; Thu, 11 Apr 2019 03:47:45 +0000 (UTC) (envelope-from jamie@catflap.org) Received: from donotpassgo.dyslexicfish.net (donotpassgo.dyslexicfish.net [104.207.135.49]) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5) with ESMTP id x3B3liDl034198 for ; Thu, 11 Apr 2019 04:47:44 +0100 (BST) (envelope-from jamie@donotpassgo.dyslexicfish.net) Received: (from jamie@localhost) by donotpassgo.dyslexicfish.net (8.14.5/8.14.5/Submit) id x3B3lhpA034197 for freebsd-stable@freebsd.org; Thu, 11 Apr 2019 04:47:43 +0100 (BST) (envelope-from jamie) From: Jamie Landeg-Jones Message-Id: <201904110347.x3B3lhpA034197@donotpassgo.dyslexicfish.net> Date: Thu, 11 Apr 2019 04:47:43 +0100 Organization: Dyslexic Fish To: freebsd-stable@freebsd.org Subject: Replicable file-system corruption due to fsck/ufs User-Agent: Heirloom mailx 12.4 7/29/08 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.7 (donotpassgo.dyslexicfish.net [104.207.135.49]); Thu, 11 Apr 2019 04:47:44 +0100 (BST) X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 03:47:46 -0000 I've noticed a replicable disk corruption by fsck_ufs/ffs on sparse files. This is on amd/64 12-stable-20190409, but I first noticed it on 12-stable-20190326. I didn't notice it on my previous build of 12-stable-20190107, but I may not have had any relevant sparse files at the time, so I don't know if that version was affected. 12-release worked OK. Here is a simplified replicable example. Thinking about it just now, I suspect it's triggered by files which end in sparseness. Can anyone else replicate this, or has my machine gone nuts? Cheers, Jamie | root@thompson# l | total 12 | 4 drwxr-x--- 2 root wheel - 512 11 Apr 04:08 ./ | 4 drwxr-xr-x 16 root wheel - 1,024 11 Apr 04:08 ../ | 4 -rw-r----- 1 root wheel - 43 11 Apr 04:08 typescript | | root@thompson# dd if=/dev/zero bs=1m count=2048 of=test.img | 2048+0 records in | 2048+0 records out | 2147483648 bytes transferred in 4.127411 secs (520298036 bytes/sec) | | root@thompson# l | total 2097708 | 4 drwxr-x--- 2 root wheel - 512 11 Apr 04:08 ./ | 4 drwxr-xr-x 16 root wheel - 1,024 11 Apr 04:08 ../ | 2097696 -rw-r----- 1 root wheel - 2,147,483,648 11 Apr 04:08 test.img | 4 -rw-r----- 1 root wheel - 43 11 Apr 04:08 typescript | | root@thompson# mdconfig test.img | md1 | | root@thompson# newfs /dev/md1 | /dev/md1: 2048.0MB (4194304 sectors) block size 32768, fragment size 4096 | using 4 cylinder groups of 512.03MB, 16385 blks, 65664 inodes. | super-block backups (for fsck_ffs -b #) at: | 192, 1048832, 2097472, 3146112 | | root@thompson# md mnt | mnt | | root@thompson# mount /dev/md1 mnt | | root@thompson# cd mnt/ | ~/x/mnt ~/x | | root@thompson# df . | Filesystem 1K-blocks Used Avail Capacity Mounted on | /dev/md1 2,031,132 8 1,868,636 0% /root/x/mnt | | root@thompson# l | total 12 | 4 drwxr-xr-x 3 root wheel - 512 11 Apr 04:09 ./ | 4 drwxr-x--- 3 root wheel - 512 11 Apr 04:09 ../ | 4 drwxrwxr-x 2 root operator - 512 11 Apr 04:09 .snap/ | | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test | | root@thompson# l | total 652 | 4 drwxr-xr-x 3 root wheel - 512 11 Apr 04:14 ./ | 4 drwxr-x--- 3 root wheel - 512 11 Apr 04:09 ../ | 4 drwxrwxr-x 2 root operator - 512 11 Apr 04:09 .snap/ | 640 -rw-r----- 1 root wheel - 9,663,676,605 11 Apr 04:14 test | | root@thompson# sha256 -r test > sha256.out | | root@thompson# cd .. | ~/x ~/x/mnt | | root@thompson# umount mnt | | root@thompson# fsck /dev/md1 | ** /dev/md1 | ** Last Mounted on /root/x/mnt | ** Phase 1 - Check Blocks and Sizes | INODE 4: FILE SIZE 9663676605 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 1342210048 | ADJUST? [yn] y | | ** Phase 2 - Check Pathnames | ** Phase 3 - Check Connectivity | ** Phase 4 - Check Reference Counts | ** Phase 5 - Check Cyl groups | 4 files, 163 used, 507620 free (20 frags, 63450 blocks, 0.0% fragmentation) | | ***** FILE SYSTEM IS CLEAN ***** | | ***** FILE SYSTEM WAS MODIFIED ***** | | root@thompson# fsck /dev/md1 | ** /dev/md1 | ** Last Mounted on /root/x/mnt | ** Phase 1 - Check Blocks and Sizes | PARTIALLY TRUNCATED INODE I=4 | SALVAGE? [yn] y | | INCORRECT BLOCK COUNT I=4 (1280 should be 256) | CORRECT? [yn] y | | INODE 4: FILE SIZE 1342210048 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 268468224 | ADJUST? [yn] y | | ** Phase 2 - Check Pathnames | ** Phase 3 - Check Connectivity | ** Phase 4 - Check Reference Counts | ** Phase 5 - Check Cyl groups | FREE BLK COUNT(S) WRONG IN SUPERBLK | SALVAGE? [yn] y | | SUMMARY INFORMATION BAD | SALVAGE? [yn] y | | BLK(S) MISSING IN BIT MAPS | SALVAGE? [yn] y | | 4 files, 35 used, 507748 free (20 frags, 63466 blocks, 0.0% fragmentation) | | ***** FILE SYSTEM IS CLEAN ***** | | ***** FILE SYSTEM WAS MODIFIED ***** | | root@thompson# fsck /dev/md1 | ** /dev/md1 | ** Last Mounted on /root/x/mnt | ** Phase 1 - Check Blocks and Sizes | PARTIALLY TRUNCATED INODE I=4 | SALVAGE? [yn] y | | INCORRECT BLOCK COUNT I=4 (256 should be 128) | CORRECT? [yn] y | | INODE 4: FILE SIZE 268468224 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 134610944 | ADJUST? [yn] y | | ** Phase 2 - Check Pathnames | ** Phase 3 - Check Connectivity | ** Phase 4 - Check Reference Counts | ** Phase 5 - Check Cyl groups | FREE BLK COUNT(S) WRONG IN SUPERBLK | SALVAGE? [yn] y | | SUMMARY INFORMATION BAD | SALVAGE? [yn] y | | BLK(S) MISSING IN BIT MAPS | SALVAGE? [yn] y | | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation) | | ***** FILE SYSTEM IS CLEAN ***** | | ***** FILE SYSTEM WAS MODIFIED ***** | | root@thompson# fsck /dev/md1 | ** /dev/md1 | ** Last Mounted on /root/x/mnt | ** Phase 1 - Check Blocks and Sizes | ** Phase 2 - Check Pathnames | ** Phase 3 - Check Connectivity | ** Phase 4 - Check Reference Counts | ** Phase 5 - Check Cyl groups | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation) | | ***** FILE SYSTEM IS CLEAN ***** | | root@thompson# fsck /dev/md1 | ** /dev/md1 | ** Last Mounted on /root/x/mnt | ** Phase 1 - Check Blocks and Sizes | ** Phase 2 - Check Pathnames | ** Phase 3 - Check Connectivity | ** Phase 4 - Check Reference Counts | ** Phase 5 - Check Cyl groups | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation) | | ***** FILE SYSTEM IS CLEAN ***** | | root@thompson# mount /dev/md1 mnt | | root@thompson# cd mnt/ | ~/x/mnt ~/x | | root@thompson# l | total 80 | 4 drwxr-xr-x 3 root wheel - 512 11 Apr 04:14 ./ | 4 drwxr-x--- 3 root wheel - 512 11 Apr 04:09 ../ | 4 drwxrwxr-x 2 root operator - 512 11 Apr 04:09 .snap/ | 4 -rw-r----- 1 root wheel - 70 11 Apr 04:14 sha256.out | 64 -rw-r----- 1 root wheel - 134,610,944 11 Apr 04:14 test | | root@thompson# cat sha256.out | 76b042e7fbb3ed1914cf600a0b5ed8e10b8d917a006dbbff774a996c9bbce941 test | | root@thompson# sha256 -r test | 6b1a548d057244632b5d2897f8c17177236c262c6af54cc0a9db5ddc8285fbd4 test From owner-freebsd-stable@freebsd.org Thu Apr 11 04:36:34 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0145615784B5 for ; Thu, 11 Apr 2019 04:36:34 +0000 (UTC) (envelope-from pho@holm.cc) Received: from relay05.pair.com (relay05.pair.com [216.92.24.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 91E5674486 for ; Thu, 11 Apr 2019 04:36:33 +0000 (UTC) (envelope-from pho@holm.cc) Received: from x2.osted.lan (87-58-223-204-dynamic.dk.customer.tdc.net [87.58.223.204]) by relay05.pair.com (Postfix) with ESMTP id 50CC31A2D39; Thu, 11 Apr 2019 00:36:31 -0400 (EDT) Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.15.2/8.15.2) with ESMTPS id x3B4aLLX088302 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Thu, 11 Apr 2019 06:36:21 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.15.2/8.15.2/Submit) id x3B4aKLJ088301; Thu, 11 Apr 2019 06:36:20 +0200 (CEST) (envelope-from pho) Date: Thu, 11 Apr 2019 06:36:20 +0200 From: Peter Holm To: Jamie Landeg-Jones Cc: freebsd-stable@freebsd.org, Kirk McKusick Subject: Re: Replicable file-system corruption due to fsck/ufs Message-ID: <20190411043620.GA87473@x2.osted.lan> References: <201904110347.x3B3lhpA034197@donotpassgo.dyslexicfish.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201904110347.x3B3lhpA034197@donotpassgo.dyslexicfish.net> User-Agent: Mutt/1.11.1 (2018-12-01) X-Rspamd-Queue-Id: 91E5674486 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; NEURAL_HAM_SHORT(-0.99)[-0.987,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 04:36:34 -0000 On Thu, Apr 11, 2019 at 04:47:43AM +0100, Jamie Landeg-Jones wrote: > I've noticed a replicable disk corruption by fsck_ufs/ffs on sparse files. > > This is on amd/64 12-stable-20190409, but I first noticed it on > 12-stable-20190326. > > I didn't notice it on my previous build of 12-stable-20190107, but I > may not have had any relevant sparse files at the time, so I don't know > if that version was affected. 12-release worked OK. > > Here is a simplified replicable example. Thinking about it just now, I > suspect it's triggered by files which end in sparseness. > > Can anyone else replicate this, or has my machine gone nuts? > > Cheers, Jamie > > | root@thompson# l > | total 12 > | 4 drwxr-x--- 2 root wheel - 512 11 Apr 04:08 ./ > | 4 drwxr-xr-x 16 root wheel - 1,024 11 Apr 04:08 ../ > | 4 -rw-r----- 1 root wheel - 43 11 Apr 04:08 typescript > | > | root@thompson# dd if=/dev/zero bs=1m count=2048 of=test.img > | 2048+0 records in > | 2048+0 records out > | 2147483648 bytes transferred in 4.127411 secs (520298036 bytes/sec) > | > | root@thompson# l > | total 2097708 > | 4 drwxr-x--- 2 root wheel - 512 11 Apr 04:08 ./ > | 4 drwxr-xr-x 16 root wheel - 1,024 11 Apr 04:08 ../ > | 2097696 -rw-r----- 1 root wheel - 2,147,483,648 11 Apr 04:08 test.img > | 4 -rw-r----- 1 root wheel - 43 11 Apr 04:08 typescript > | > | root@thompson# mdconfig test.img > | md1 > | > | root@thompson# newfs /dev/md1 > | /dev/md1: 2048.0MB (4194304 sectors) block size 32768, fragment size 4096 > | using 4 cylinder groups of 512.03MB, 16385 blks, 65664 inodes. > | super-block backups (for fsck_ffs -b #) at: > | 192, 1048832, 2097472, 3146112 > | > | root@thompson# md mnt > | mnt > | > | root@thompson# mount /dev/md1 mnt > | > | root@thompson# cd mnt/ > | ~/x/mnt ~/x > | > | root@thompson# df . > | Filesystem 1K-blocks Used Avail Capacity Mounted on > | /dev/md1 2,031,132 8 1,868,636 0% /root/x/mnt > | > | root@thompson# l > | total 12 > | 4 drwxr-xr-x 3 root wheel - 512 11 Apr 04:09 ./ > | 4 drwxr-x--- 3 root wheel - 512 11 Apr 04:09 ../ > | 4 drwxrwxr-x 2 root operator - 512 11 Apr 04:09 .snap/ > | > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | root@thompson# echo "testing 1...2...3..." >> test ; truncate -s +1g test > | > | root@thompson# l > | total 652 > | 4 drwxr-xr-x 3 root wheel - 512 11 Apr 04:14 ./ > | 4 drwxr-x--- 3 root wheel - 512 11 Apr 04:09 ../ > | 4 drwxrwxr-x 2 root operator - 512 11 Apr 04:09 .snap/ > | 640 -rw-r----- 1 root wheel - 9,663,676,605 11 Apr 04:14 test > | > | root@thompson# sha256 -r test > sha256.out > | > | root@thompson# cd .. > | ~/x ~/x/mnt > | > | root@thompson# umount mnt > | > | root@thompson# fsck /dev/md1 > | ** /dev/md1 > | ** Last Mounted on /root/x/mnt > | ** Phase 1 - Check Blocks and Sizes > | INODE 4: FILE SIZE 9663676605 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 1342210048 > | ADJUST? [yn] y > | > | ** Phase 2 - Check Pathnames > | ** Phase 3 - Check Connectivity > | ** Phase 4 - Check Reference Counts > | ** Phase 5 - Check Cyl groups > | 4 files, 163 used, 507620 free (20 frags, 63450 blocks, 0.0% fragmentation) > | > | ***** FILE SYSTEM IS CLEAN ***** > | > | ***** FILE SYSTEM WAS MODIFIED ***** > | > | root@thompson# fsck /dev/md1 > | ** /dev/md1 > | ** Last Mounted on /root/x/mnt > | ** Phase 1 - Check Blocks and Sizes > | PARTIALLY TRUNCATED INODE I=4 > | SALVAGE? [yn] y > | > | INCORRECT BLOCK COUNT I=4 (1280 should be 256) > | CORRECT? [yn] y > | > | INODE 4: FILE SIZE 1342210048 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 268468224 > | ADJUST? [yn] y > | > | ** Phase 2 - Check Pathnames > | ** Phase 3 - Check Connectivity > | ** Phase 4 - Check Reference Counts > | ** Phase 5 - Check Cyl groups > | FREE BLK COUNT(S) WRONG IN SUPERBLK > | SALVAGE? [yn] y > | > | SUMMARY INFORMATION BAD > | SALVAGE? [yn] y > | > | BLK(S) MISSING IN BIT MAPS > | SALVAGE? [yn] y > | > | 4 files, 35 used, 507748 free (20 frags, 63466 blocks, 0.0% fragmentation) > | > | ***** FILE SYSTEM IS CLEAN ***** > | > | ***** FILE SYSTEM WAS MODIFIED ***** > | > | root@thompson# fsck /dev/md1 > | ** /dev/md1 > | ** Last Mounted on /root/x/mnt > | ** Phase 1 - Check Blocks and Sizes > | PARTIALLY TRUNCATED INODE I=4 > | SALVAGE? [yn] y > | > | INCORRECT BLOCK COUNT I=4 (256 should be 128) > | CORRECT? [yn] y > | > | INODE 4: FILE SIZE 268468224 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 134610944 > | ADJUST? [yn] y > | > | ** Phase 2 - Check Pathnames > | ** Phase 3 - Check Connectivity > | ** Phase 4 - Check Reference Counts > | ** Phase 5 - Check Cyl groups > | FREE BLK COUNT(S) WRONG IN SUPERBLK > | SALVAGE? [yn] y > | > | SUMMARY INFORMATION BAD > | SALVAGE? [yn] y > | > | BLK(S) MISSING IN BIT MAPS > | SALVAGE? [yn] y > | > | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation) > | > | ***** FILE SYSTEM IS CLEAN ***** > | > | ***** FILE SYSTEM WAS MODIFIED ***** > | > | root@thompson# fsck /dev/md1 > | ** /dev/md1 > | ** Last Mounted on /root/x/mnt > | ** Phase 1 - Check Blocks and Sizes > | ** Phase 2 - Check Pathnames > | ** Phase 3 - Check Connectivity > | ** Phase 4 - Check Reference Counts > | ** Phase 5 - Check Cyl groups > | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation) > | > | ***** FILE SYSTEM IS CLEAN ***** > | > | root@thompson# fsck /dev/md1 > | ** /dev/md1 > | ** Last Mounted on /root/x/mnt > | ** Phase 1 - Check Blocks and Sizes > | ** Phase 2 - Check Pathnames > | ** Phase 3 - Check Connectivity > | ** Phase 4 - Check Reference Counts > | ** Phase 5 - Check Cyl groups > | 4 files, 19 used, 507764 free (20 frags, 63468 blocks, 0.0% fragmentation) > | > | ***** FILE SYSTEM IS CLEAN ***** > | > | root@thompson# mount /dev/md1 mnt > | > | root@thompson# cd mnt/ > | ~/x/mnt ~/x > | > | root@thompson# l > | total 80 > | 4 drwxr-xr-x 3 root wheel - 512 11 Apr 04:14 ./ > | 4 drwxr-x--- 3 root wheel - 512 11 Apr 04:09 ../ > | 4 drwxrwxr-x 2 root operator - 512 11 Apr 04:09 .snap/ > | 4 -rw-r----- 1 root wheel - 70 11 Apr 04:14 sha256.out > | 64 -rw-r----- 1 root wheel - 134,610,944 11 Apr 04:14 test > | > | root@thompson# cat sha256.out > | 76b042e7fbb3ed1914cf600a0b5ed8e10b8d917a006dbbff774a996c9bbce941 test > | > | root@thompson# sha256 -r test > | 6b1a548d057244632b5d2897f8c17177236c262c6af54cc0a9db5ddc8285fbd4 test > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" I see this even with a single truncate on HEAD. $ ./truncate10.sh 96 -rw-r--r-- 1 root wheel 1073741824 11 apr. 06:33 test ** /dev/md10a ** Last Mounted on /mnt ** Phase 1 - Check Blocks and Sizes INODE 3: FILE SIZE 1073741824 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 268435456 ADJUST? yes -- Peter From owner-freebsd-stable@freebsd.org Thu Apr 11 04:45:07 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id B18EA1578AA1 for ; Thu, 11 Apr 2019 04:45:07 +0000 (UTC) (envelope-from jamie@catflap.dyslexicfish.net) Received: from catflap.dyslexicfish.net (catflap.dyslexicfish.net [IPv6:2001:19f0:300:2185::1:1]) by mx1.freebsd.org (Postfix) with ESMTP id 1CFF874C04 for ; Thu, 11 Apr 2019 04:45:06 +0000 (UTC) (envelope-from jamie@catflap.dyslexicfish.net) Received: from jamie (uid 1970) (envelope-from jamie@catflap.dyslexicfish.net) id 75986 by catflap.dyslexicfish.net (DragonFly Mail Agent v0.11+); Thu, 11 Apr 2019 05:45:04 +0100 Date: Thu, 11 Apr 2019 05:45:04 +0100 Organization: Dyslexic Fish To: peter@holm.cc, jamie@catflap.org Cc: mckusick@mckusick.com, freebsd-stable@freebsd.org Subject: Re: Replicable file-system corruption due to fsck/ufs References: <201904110347.x3B3lhpA034197@donotpassgo.dyslexicfish.net> <20190411043620.GA87473@x2.osted.lan> In-Reply-To: <20190411043620.GA87473@x2.osted.lan> User-Agent: Heirloom mailx 12.4 7/29/08 MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Message-Id: <5caec650.75986.3515f2a1@catflap.dyslexicfish.net> From: X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 04:45:07 -0000 Peter Holm wrote: > I see this even with a single truncate on HEAD. > > $ ./truncate10.sh > 96 -rw-r--r-- 1 root wheel 1073741824 11 apr. 06:33 test > ** /dev/md10a > ** Last Mounted on /mnt > ** Phase 1 - Check Blocks and Sizes > INODE 3: FILE SIZE 1073741824 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 268435456 > ADJUST? yes Thanks.. I should have tested that myself.. doh! I was trying to closer replicate my real file that triggered the problem which contained a number of sparse areas. And thanks for adding Kirk to the discussion. I wanted to first be sure it wasn't just me :-) Cheers, Jamie From owner-freebsd-stable@freebsd.org Thu Apr 11 04:51:45 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7AD2C1578FB8 for ; Thu, 11 Apr 2019 04:51:45 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82c.google.com (mail-qt1-x82c.google.com [IPv6:2607:f8b0:4864:20::82c]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 0AA457536B for ; Thu, 11 Apr 2019 04:51:44 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82c.google.com with SMTP id z17so5596752qts.13 for ; Wed, 10 Apr 2019 21:51:44 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=pZiIsZvM1IuWmdSwoRf/+2NbbMbkn1ZNHH+KV7m7oiU=; b=ANOsedlzLJdgsatRPtuUlXNczMP7uoOa9nc1W2iW9m6k/UTXWW8q0T78taJpDzZ8fI WAvILcGNwC3KdvB92BYGArEoayCdaGmlRZeKKElQ/WqeZC6bp0IHoBKf4MMh59IMGmKh v4eEXjxaDi83Va6/t68QVCS/1mjeKpCcLCgTXjwtNZNlxxtNhnUbrS/s60ucH4OQavdq 7+500YuFG5y0SBiYU0kqLfYR1U1bi1UbE8ry98aswucaQDieYgoZ7chSJPpbAOUmGP2Z +oLaweI+i3cJTGGHC200/8dLc1v+QJ064/UHa85IyRR3FDRvXWjuZqM4E+6AjdyM2ajd yuiA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=pZiIsZvM1IuWmdSwoRf/+2NbbMbkn1ZNHH+KV7m7oiU=; b=YelMR2UyITchsFXw+C6DuzCVMi8IzVaHBbXQYnWe5J3renq72h6OxTXqYrsxPrx6ZW xhggAboEA3n4kbI4i1Hz2mxFFM7S3blwDjMVIXUAs/1mAl30jWgXyytlbRwBjhCguPwO DNyolH2s3QU4d5LY+tkeJLP43zMc57NzB2tljBnQI6+ao1JhHlnb+1epNYHOcecPloRW lBJNMHPBCmTnTyFCwPibMS0KQ0Vd2VV+qqg9H8PGEo//zWm4Dk7FpvdzOmtzzLlX8ZSB Z6RRqtt6iif94RKJL3bO4UB/xdbYToKMqlRUbwN0X/wQ/VCAXtK1T9ryyOO/Vgt8jFAA ntzA== X-Gm-Message-State: APjAAAUAVkLCDtKM3u6uYYVgB7wr461b7Yv/SJ3yHXEwCZLtWILpeA61 BoL82oDUobeVoHYqU+EcE4+a7sHtWhtZ2RcXKZqtXTo7 X-Google-Smtp-Source: APXvYqwdi1/CAhIj0J9DmnPI7LGZidyG2sSOSNSlrrc3X3VMYzyMronRSZrCjwATvmgozKBFp+Mi6sh34VVyexnKl74= X-Received: by 2002:aed:3f51:: with SMTP id q17mr38036680qtf.346.1554958304399; Wed, 10 Apr 2019 21:51:44 -0700 (PDT) MIME-Version: 1.0 References: <201904110347.x3B3lhpA034197@donotpassgo.dyslexicfish.net> <20190411043620.GA87473@x2.osted.lan> <5caec650.75986.3515f2a1@catflap.dyslexicfish.net> In-Reply-To: <5caec650.75986.3515f2a1@catflap.dyslexicfish.net> From: Warner Losh Date: Wed, 10 Apr 2019 22:51:32 -0600 Message-ID: Subject: Re: Replicable file-system corruption due to fsck/ufs To: jamie@catflap.dyslexicfish.net Cc: Peter Holm , jamie@catflap.org, Kirk McKusick , FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 0AA457536B X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.96 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.96)[-0.961,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 04:51:45 -0000 On Wed, Apr 10, 2019 at 10:46 PM wrote: > Peter Holm wrote: > > > I see this even with a single truncate on HEAD. > > > > $ ./truncate10.sh > > 96 -rw-r--r-- 1 root wheel 1073741824 11 apr. 06:33 test > > ** /dev/md10a > > ** Last Mounted on /mnt > > ** Phase 1 - Check Blocks and Sizes > > INODE 3: FILE SIZE 1073741824 BEYOND END OF ALLOCATED FILE, SIZE SHOULD > BE 268435456 > > ADJUST? yes > > Thanks.. I should have tested that myself.. doh! I was trying to closer > replicate > my real file that triggered the problem which contained a number of sparse > areas. > > And thanks for adding Kirk to the discussion. I wanted to first be sure it > wasn't > just me :-) > I believe that this was added recently to detect corruption that happens when a file is being appended when the system crashes. Warner From owner-freebsd-stable@freebsd.org Thu Apr 11 15:29:04 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4DE361587227 for ; Thu, 11 Apr 2019 15:29:04 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-vs1-xe34.google.com (mail-vs1-xe34.google.com [IPv6:2607:f8b0:4864:20::e34]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 5FAE86BB6F for ; Thu, 11 Apr 2019 15:29:03 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-vs1-xe34.google.com with SMTP id t78so3745253vsc.1 for ; Thu, 11 Apr 2019 08:29:03 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:cc:from:subject:date:importance :in-reply-to:references; bh=Fs79CkNEr1I44CJGoHcllMpFSbLeSF4v5BxdE+65X7s=; b=uLVjazvJ70u4/ZURizL2DmA6uixvNInFzbC+eVvtDSUVueo80B8T8XiTl0s27CkyXT 2jOvrKM2N4YhPNh/XNKDgqxOggV/r86B0We0a0SocLP1P+aT911P0+2bvDaqk005dJ7U vzEa2mM7oKujqU8U7dnjpgRLnQR3EsbZquQsvI7AyFi9+1dHfDO7fIYjX0+Ej4r5dwPA ZM3HIDIdyYaRA451cGzTaNlGCqdmFXvkEL0x3N9LENvPO+9kLdoKU9SDpeOdHlYatVdN lNXPlxSvpk0BKYO+Jxz0V2v31zsyUbbCfwRQo7IVqxtpj71lgO5k1hq7EaCl+yzuOuyB vzoA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:cc:from:subject:date :importance:in-reply-to:references; bh=Fs79CkNEr1I44CJGoHcllMpFSbLeSF4v5BxdE+65X7s=; b=H1fnRCwg0PcP2Qb9kDUr0VOeYxS/qBy7qOvgVE44svH0bR1osmY/jBzAk5FP+a7Jvb qM89Qy4q5bMzbRwkPCwvP9AHE/08Rd3Aot68jYWs87WQQe6N9Ml7Dd21265d9NP1G1LX WfUIs3OT7pOT16rvoDg7K1BnqfNCF1vfmECR76z1KaxKL9gZZE/9oxTXGHBpsop9Vz4M j8Xiz7PTRym1ceyi7wZm15A++92PWt80lKsUpGJWvTR3uvsd3r8aspCRDM8r7bCN5clO pqB4OTc030fHIMOEQJGlwH7++iq/N4NjZyCF/Qt/ooXCUV/Jb+s7tMrFytmxYaWdbbdy t2yA== X-Gm-Message-State: APjAAAVC4uLNsbhxxfW6wkqFARRIAgT06Z2X1Vp/pOsKysXU9PlRqkKd WtbG9CoI6ez20TQBjM2iKDwc9sE4 X-Google-Smtp-Source: APXvYqwRKe1mH6u/0k8lfpBFPLFadC0rBkqKCTA4HgWQrHFna79GOzZIwoeNCpm5OzXxUszMJjnNtA== X-Received: by 2002:a05:6102:398:: with SMTP id m24mr3613906vsq.23.1554996542610; Thu, 11 Apr 2019 08:29:02 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.119? (git.mayberryinv.com. [63.143.111.202]) by smtp.gmail.com with ESMTPSA id 110sm27880398uaq.7.2019.04.11.08.29.01 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Apr 2019 08:29:01 -0700 (PDT) Message-ID: <5caf5d3d.1c69fb81.63ae1.ffd0@mx.google.com> MIME-Version: 1.0 To: Walter Cramer Cc: Jonathan Chen , "freebsd-stable@freebsd.org" From: Software Info Subject: RE: Crontab Question Date: Thu, 11 Apr 2019 10:29:01 -0500 Importance: normal X-Priority: 3 In-Reply-To: <20190410172638.C14867@mulder.mintsol.com> References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> <20190410172638.C14867@mulder.mintsol.com> X-Rspamd-Queue-Id: 5FAE86BB6F X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=uLVjazvJ; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::e34 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(0.00)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; URI_COUNT_ODD(1.00)[1]; RCVD_COUNT_THREE(0.00)[3]; RBL_SEM_IPV6(1.00)[4.3.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.bl.ipv6.spameatingmonkey.net]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; NEURAL_HAM_SHORT(-0.99)[-0.986,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[4.3.e.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.98)[ip: (-9.71), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.19), country: US(-0.06)] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 15:29:04 -0000 Well thanks for all the input. I just have to tp keep working at it. Again,= much appreciated. Regards SI Sent from Mail for Windows 10 From: Walter Cramer Sent: Wednesday, April 10, 2019 4:40 PM To: Software Info Cc: Jonathan Chen; freebsd-stable@freebsd.org Subject: RE: Crontab Question On Wed, 10 Apr 2019, Software Info wrote: > OK. So although the script is located in my home directory, it doesn=C3= =A2=E2=82=AC=E2=84=A2t=20 > start there? Sorry but I don=C3=A2=E2=82=AC=E2=84=A2t quite understand. = Could you explain a=20 > little further please? Both 'cp' and 'ls' are located in /bin. But if I run the 'ls' command in=20 /root, 'ls' can't find 'cp' (unless I tell it where to look) - even though= =20 /bin *is* in my PATH - server7:/root # ls cp ls: cp: No such file or directory server7:/root # ls /bin/cp /bin/cp Where the system looks for *commands*, to execute, is different from where= =20 it looks for other files, which those commands use. The latter is=20 generally only the current directory (unless you tell it otherwise).=20 When cron runs a script as root, "current directory" will be /root. BUT - for security and other reasons, it would be better to have cron run=20 your script as you (not root), and as '/home/me/myscript' (instead of=20 adding your home directory to PATH in /etc/crontab). -Walter From owner-freebsd-stable@freebsd.org Thu Apr 11 16:53:47 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id ABE0A1588D43 for ; Thu, 11 Apr 2019 16:53:47 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: from mail-ed1-x52d.google.com (mail-ed1-x52d.google.com [IPv6:2a00:1450:4864:20::52d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 968B96E8B9 for ; Thu, 11 Apr 2019 16:53:46 +0000 (UTC) (envelope-from mack63richard@gmail.com) Received: by mail-ed1-x52d.google.com with SMTP id x14so5864070eds.1 for ; Thu, 11 Apr 2019 09:53:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=gqxlvNOWDV9hr2k6wp2sEqvXX+eOInePmGyjXAlesII=; b=tiJueW483pX7oV/f7UE/MSqDX9wbCDh8bCC6RG9XW1IVHNqRQJumeGsu0HxfbAf2yB w/949DzueVCqQ/LT6VoJxdd11adQyJbHmTl/26BoU95uqUGUR1cZ1xTf74rqwE3GeNMo avG0MZMb0NkeUaCcBicwOgdtLDZchE9kXr/N8x/yh/UYbPzEHAOg6mVqnYoUxog1Ns7x ps8zE6VnMHcOQh6O4+sFbYfXmgjKfuukk61iWUcGw1SD0Y/Mg5+1mSj7H7hPSlGFx/OF lPEpsoULELMl6SPzbwrgkoQkUAHOOO0PohFhZooDelmUhleFcml1u/x4tr5dNFrC2Z23 onRg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:subject:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to; bh=gqxlvNOWDV9hr2k6wp2sEqvXX+eOInePmGyjXAlesII=; b=pTID8X7NSqbnhEWZflHAfGvRANmwMSmefqq6d2vK1sTYUv5N7qArdGFC1ZwWTJR8u0 llnYo8Kmbr4dMYDuTCbaKl7gZ7sMwhYvmB39i3G68HjA8ey/Nb8G4GuBjR1rEHiQKznU URVef7DlUFMJEFCr5DBmq0zgsjpmFz9tgM9dDcMNRUDcsE+pavnqgxpZEiMBCkpVoTMx 4J452DSvGtOqqvbMPX+bbYFtdYswMTno4R9+9xiybIKUuUe4gKPirybDCCeNyCibSzDE V1n4ZXv5VyBWMXsgF1XuUFd+tVSVaXtmPn4LYkLSMBLJ8m41wHRLleiBESBN6HaF7DfX i8UQ== X-Gm-Message-State: APjAAAXd193y5j1gj5uoX55b6xCHIwoqwplntoSQzZ7Km08167HKzkYd /++6QcUtovzaW6YhOBnjBCwweOk3lHs= X-Google-Smtp-Source: APXvYqxHG+eaUeeIRJW/N3dTyIvXn8bbWleNM3xzwXbpItGFBo8OuGMwerrGQenxY09D2+vhQ4BJ4w== X-Received: by 2002:a17:906:19db:: with SMTP id h27mr9762389ejd.4.1555001623941; Thu, 11 Apr 2019 09:53:43 -0700 (PDT) Received: from [192.168.0.13] (cpc129954-hawk19-2-0-cust205.know.cable.virginm.net. [92.234.141.206]) by smtp.gmail.com with ESMTPSA id s24sm5096551edq.79.2019.04.11.09.53.42 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Apr 2019 09:53:42 -0700 (PDT) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (1.0) Subject: Re: Crontab Question From: Richard Mackerras X-Mailer: iPad Mail (16E227) In-Reply-To: <5caf5d3d.1c69fb81.63ae1.ffd0@mx.google.com> Date: Thu, 11 Apr 2019 17:53:41 +0100 Cc: Walter Cramer , "freebsd-stable@freebsd.org" , Jonathan Chen Content-Transfer-Encoding: quoted-printable Message-Id: <36054B2F-3456-46C1-BE1A-FB90551E2AE7@gmail.com> References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> <20190410172638.C14867@mulder.mintsol.com> <5caf5d3d.1c69fb81.63ae1.ffd0@mx.google.com> To: Software Info X-Rspamd-Queue-Id: 968B96E8B9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=tiJueW48; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of mack63richard@gmail.com designates 2a00:1450:4864:20::52d as permitted sender) smtp.mailfrom=mack63richard@gmail.com X-Spamd-Result: default: False [-6.22 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36]; MV_CASE(0.50)[]; FREEMAIL_FROM(0.00)[gmail.com]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_SHORT(-0.97)[-0.969,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[d.2.5.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.5.4.1.0.0.a.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.74)[ip: (-9.07), ipnet: 2a00:1450::/32(-2.39), asn: 15169(-2.19), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 16:53:47 -0000 In your script put a few commands outputting to a check file pwd > /tmp/checkfile Add a few more like=20 ENV >> /tmp/checkfile Just to make sure it really is in the directory you expect with the environm= ent you expect.=20 If you want it to be run as you never use the root crontab unless you want r= eally crap security.=20 Cheers Sent from my iPad > On 11 Apr 2019, at 16:29, Software Info wrote:= >=20 > Well thanks for all the input. I just have to tp keep working at it. Again= , much appreciated. >=20 >=20 > Regards > SI >=20 > Sent from Mail for Windows 10 >=20 > From: Walter Cramer > Sent: Wednesday, April 10, 2019 4:40 PM > To: Software Info > Cc: Jonathan Chen; freebsd-stable@freebsd.org > Subject: RE: Crontab Question >=20 >> On Wed, 10 Apr 2019, Software Info wrote: >>=20 >> OK. So although the script is located in my home directory, it doesn=C3=A2= =E2=82=AC=E2=84=A2t=20 >> start there? Sorry but I don=C3=A2=E2=82=AC=E2=84=A2t quite understand. C= ould you explain a=20 >> little further please? >=20 > Both 'cp' and 'ls' are located in /bin. But if I run the 'ls' command in=20= > /root, 'ls' can't find 'cp' (unless I tell it where to look) - even though= =20 > /bin *is* in my PATH - >=20 > server7:/root # ls cp > ls: cp: No such file or directory > server7:/root # ls /bin/cp > /bin/cp >=20 > Where the system looks for *commands*, to execute, is different from where= =20 > it looks for other files, which those commands use. The latter is=20 > generally only the current directory (unless you tell it otherwise).=20 > When cron runs a script as root, "current directory" will be /root. >=20 > BUT - for security and other reasons, it would be better to have cron run=20= > your script as you (not root), and as '/home/me/myscript' (instead of=20 > adding your home directory to PATH in /etc/crontab). >=20 > -Walter >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Thu Apr 11 18:52:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 8DF4B15662F0 for ; Thu, 11 Apr 2019 18:52:41 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: from mail-it1-x12b.google.com (mail-it1-x12b.google.com [IPv6:2607:f8b0:4864:20::12b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id A5FEF721F7 for ; Thu, 11 Apr 2019 18:52:40 +0000 (UTC) (envelope-from zbeeble@gmail.com) Received: by mail-it1-x12b.google.com with SMTP id f22so11309950ita.3 for ; Thu, 11 Apr 2019 11:52:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=p3A0JfbjDgC860HhNTG3zjTJ40nC+sjNhDOYNmPrkt4=; b=N5HRsEriDa6AOACNNJAUiZEomHsc2yLEzruxuRkm+txvfZlQLXHoY+1fgjUHRN0kzK yCk2sy6+9pqX0YFbT+RtQAbmAlLYaDTZgEp51inrs7tL/B36j5cfy5mcvenmjXl2lNlB rAylmhljjdTscLnKV2lfzr3IARgHpon5xnFgZO74gfTAycHquSuPI9MzAAlMSw9zW2uJ Tat2VBLQ3E5DKPAAp0Qpwg06J7GDHVSpvwK05tLE1ADyoxno0GjtTkN3Ren8d97982yR p1ulCiM2bg1adH62oow/XnK0crc6hHfA+NBp3c1Stv1ZkyYkMCx7yEGB3VRvC44SWLc+ PqGQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=p3A0JfbjDgC860HhNTG3zjTJ40nC+sjNhDOYNmPrkt4=; b=KMQXWOdIQJlqr6kLyxJQI71zi9jqiN8hXPJH6TIjSLK3aek/wbdYwrjIpGI9mm56ot 7nrekAaSRugl5LH8WxLwatT40amAcMb+s+2q+GHSZLzV5+kV1biunnxw3lBLxjI7WOCP OmC4WfwFB6pjRQRQh3SR0FQ5M4p4oFYWpxDZTt+LYh2aiko65+ntJokMRCHmiFB1Npsf 2161iH0irXKrmvIAbEM8N4+dvR5nd1BLoSZpYHa5KMgMtANqMcGWt2WhiVEiCS/Wt7BJ +0V6La+fSxatBomslwEbZ0hSrjwkl6NG7SScyGvY95meCXa75s92LyaShTduDgw11dUH Mm7g== X-Gm-Message-State: APjAAAXlRD4H/w7fYFvZ9twDoe/A1CUTxfG3EOYFXqgG+C4kP3X5sHiP ACAHwwlwppHuOOnYbZHQ55sWXurY5pBRCJWzEA== X-Google-Smtp-Source: APXvYqyZHxOXY+ug5J/7/oAGHdzbWSnPd5P/Q56jgch2N371Weuo/sveX0jZxHcntCQlB2kKNmZCr2bQKK9XTsTRaOc= X-Received: by 2002:a24:13c7:: with SMTP id 190mr10199447itz.9.1555008759746; Thu, 11 Apr 2019 11:52:39 -0700 (PDT) MIME-Version: 1.0 References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> In-Reply-To: <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> From: Zaphod Beeblebrox Date: Thu, 11 Apr 2019 14:52:28 -0400 Message-ID: Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: Karl Denninger Cc: FreeBSD Stable X-Rspamd-Queue-Id: A5FEF721F7 X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=N5HRsEri; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of zbeeble@gmail.com designates 2607:f8b0:4864:20::12b as permitted sender) smtp.mailfrom=zbeeble@gmail.com X-Spamd-Result: default: False [-4.40 / 15.00]; R_SPF_ALLOW(0.00)[+ip6:2607:f8b0:4000::/36]; FREEMAIL_FROM(0.00)[gmail.com]; RBL_SEM_IPV6(1.00)[b.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.bl.ipv6.spameatingmonkey.net]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; MIME_BASE64_TEXT(0.10)[]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; NEURAL_HAM_SHORT(-0.94)[-0.938,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[b.2.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.55)[ip: (-7.55), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 18:52:41 -0000 On Wed, Apr 10, 2019 at 10:41 AM Karl Denninger wrote: > In this specific case the adapter in question is... > > mps0: port 0xc000-0xc0ff mem > 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 > mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd > mps0: IOCCapabilities: > 1285c > > Which is indeed a "dumb" HBA (in IT mode), and Zeephod says he connects > his drives via dumb on-MoBo direct SATA connections. > Maybe I'm in good company. My current setup has 8 of the disks connected to: mps0: port 0xb000-0xb0ff mem 0xfe240000-0xfe24ffff,0xfe200000-0xfe23ffff irq 32 at device 0.0 on pci6 mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 5a85c ... just with a cable that breaks out each of the 2 connectors into 4 SATA-style connectors, and the other 8 disks (plus boot disks and SSD cache/log) connected to ports on... - ahci0: port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xfe900000-0xfe9001ff irq 44 at device 0.0 on pci2 - ahci2: port 0xa050-0xa057,0xa040-0xa043,0xa030-0xa037,0xa020-0xa023,0xa000-0xa01f mem 0xfe610000-0xfe6107ff irq 40 at device 0.0 on pci7 - ahci3: port 0xf040-0xf047,0xf030-0xf033,0xf020-0xf027,0xf010-0xf013,0xf000-0xf00f mem 0xfea07000-0xfea073ff irq 19 at device 17.0 on pci0 ... each drive connected to a single port. I can actually reproduce this at will. Because I have 16 drives, when one fails, I need to find it. I pull the sata cable for a drive, determine if it's the drive in question, if not, reconnect, "ONLINE" it and wait for resilver to stop... usually only a minute or two. ... if I do this 4 to 6 odd times to find a drive (I can tell, in general, that a drive is part of the SAS controller or the SATA controllers... so I'm only looking among 8, ever) ... then I "REPLACE" the problem drive. More often than not, the a scrub will find a few problems. In fact, it appears that the most recent scrub is an example: [1:7:306]dgilbert@vr:~> zpool status pool: vr1 state: ONLINE scan: scrub repaired 32K in 47h16m with 0 errors on Mon Apr 1 23:12:03 2019 config: NAME STATE READ WRITE CKSUM vr1 ONLINE 0 0 0 raidz2-0 ONLINE 0 0 0 gpt/v1-d0 ONLINE 0 0 0 gpt/v1-d1 ONLINE 0 0 0 gpt/v1-d2 ONLINE 0 0 0 gpt/v1-d3 ONLINE 0 0 0 gpt/v1-d4 ONLINE 0 0 0 gpt/v1-d5 ONLINE 0 0 0 gpt/v1-d6 ONLINE 0 0 0 gpt/v1-d7 ONLINE 0 0 0 raidz2-2 ONLINE 0 0 0 gpt/v1-e0c ONLINE 0 0 0 gpt/v1-e1b ONLINE 0 0 0 gpt/v1-e2b ONLINE 0 0 0 gpt/v1-e3b ONLINE 0 0 0 gpt/v1-e4b ONLINE 0 0 0 gpt/v1-e5a ONLINE 0 0 0 gpt/v1-e6a ONLINE 0 0 0 gpt/v1-e7c ONLINE 0 0 0 logs gpt/vr1log ONLINE 0 0 0 cache gpt/vr1cache ONLINE 0 0 0 errors: No known data errors ... it doesn't say it now, but there were 5 CKSUM errors on one of the drives that I had trial-removed (and not on the one replaced). From owner-freebsd-stable@freebsd.org Thu Apr 11 18:58:07 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 58ABF15665A9 for ; Thu, 11 Apr 2019 18:58:07 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 8224F723B9 for ; Thu, 11 Apr 2019 18:58:06 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 42E0921109D for ; Thu, 11 Apr 2019 14:58:00 -0400 (EDT) Received: from [192.168.10.26] (D16.Denninger.Net [192.168.10.26]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id AB6ABD7AA for ; Thu, 11 Apr 2019 13:57:59 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> Date: Thu, 11 Apr 2019 13:57:58 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms020004020804090407090706" X-Rspamd-Queue-Id: 8224F723B9 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.52 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[px.denninger.net]; NEURAL_HAM_SHORT(-0.89)[-0.893,0]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-2.42)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.85), asn: 14061(1.71), country: US(-0.06)]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 18:58:07 -0000 This is a cryptographically signed message in MIME format. --------------ms020004020804090407090706 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/11/2019 13:52, Zaphod Beeblebrox wrote: > On Wed, Apr 10, 2019 at 10:41 AM Karl Denninger wr= ote: > > >> In this specific case the adapter in question is... >> >> mps0: port 0xc000-0xc0ff mem >> 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pc= i3 >> mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd >> mps0: IOCCapabilities: >> 1285c >> >> Which is indeed a "dumb" HBA (in IT mode), and Zeephod says he connect= s >> his drives via dumb on-MoBo direct SATA connections. >> > Maybe I'm in good company. My current setup has 8 of the disks connect= ed > to: > > mps0: port 0xb000-0xb0ff mem > 0xfe240000-0xfe24ffff,0xfe200000-0xfe23ffff irq 32 at device 0.0 on pci= 6 > mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd > mps0: IOCCapabilities: > 5a85c > > ... just with a cable that breaks out each of the 2 connectors into 4 > SATA-style connectors, and the other 8 disks (plus boot disks and SSD > cache/log) connected to ports on... > > - ahci0: port > 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f m= em > 0xfe900000-0xfe9001ff irq 44 at device 0.0 on pci2 > - ahci2: port > 0xa050-0xa057,0xa040-0xa043,0xa030-0xa037,0xa020-0xa023,0xa000-0xa01f m= em > 0xfe610000-0xfe6107ff irq 40 at device 0.0 on pci7 > - ahci3: port > 0xf040-0xf047,0xf030-0xf033,0xf020-0xf027,0xf010-0xf013,0xf000-0xf00f m= em > 0xfea07000-0xfea073ff irq 19 at device 17.0 on pci0 > > ... each drive connected to a single port. > > I can actually reproduce this at will. Because I have 16 drives, when = one > fails, I need to find it. I pull the sata cable for a drive, determine= if > it's the drive in question, if not, reconnect, "ONLINE" it and wait for= > resilver to stop... usually only a minute or two. > > ... if I do this 4 to 6 odd times to find a drive (I can tell, in gener= al, > that a drive is part of the SAS controller or the SATA controllers... s= o > I'm only looking among 8, ever) ... then I "REPLACE" the problem drive.= > More often than not, the a scrub will find a few problems. In fact, it= > appears that the most recent scrub is an example: > > [1:7:306]dgilbert@vr:~> zpool status > pool: vr1 > state: ONLINE > scan: scrub repaired 32K in 47h16m with 0 errors on Mon Apr 1 23:12:= 03 > 2019 > config: > > NAME STATE READ WRITE CKSUM > vr1 ONLINE 0 0 0 > raidz2-0 ONLINE 0 0 0 > gpt/v1-d0 ONLINE 0 0 0 > gpt/v1-d1 ONLINE 0 0 0 > gpt/v1-d2 ONLINE 0 0 0 > gpt/v1-d3 ONLINE 0 0 0 > gpt/v1-d4 ONLINE 0 0 0 > gpt/v1-d5 ONLINE 0 0 0 > gpt/v1-d6 ONLINE 0 0 0 > gpt/v1-d7 ONLINE 0 0 0 > raidz2-2 ONLINE 0 0 0 > gpt/v1-e0c ONLINE 0 0 0 > gpt/v1-e1b ONLINE 0 0 0 > gpt/v1-e2b ONLINE 0 0 0 > gpt/v1-e3b ONLINE 0 0 0 > gpt/v1-e4b ONLINE 0 0 0 > gpt/v1-e5a ONLINE 0 0 0 > gpt/v1-e6a ONLINE 0 0 0 > gpt/v1-e7c ONLINE 0 0 0 > logs > gpt/vr1log ONLINE 0 0 0 > cache > gpt/vr1cache ONLINE 0 0 0 > > errors: No known data errors > > ... it doesn't say it now, but there were 5 CKSUM errors on one of the > drives that I had trial-removed (and not on the one replaced). > _______________________________________________ That is EXACTLY what I'm seeing; the "OFFLINE'd" drive is the one that, after a scrub, comes up with the checksum errors.=C2=A0 It does *not* fla= g any errors during the resilver and the drives *not* taken offline do not (ever) show checksum errors either. Interestingly enough you have 19.00.00.00 firmware on your card as well -- which is what was on mine. I have flashed my card forward to 20.00.07.00 -- we'll see if it still does it when I do the next swap of the backup set. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms020004020804090407090706 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDExMTg1NzU5 WjBPBgkqhkiG9w0BCQQxQgRAAnFxWeWzGQ/YQP+19skGMpVQ6SQ4I7utIzSZFYgaBk8vXdqm 0V3moBAFplMKXb5/RH2T0UWSUDgct0aaUeyAETBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgAz/+bf6w2Iy/Eg52GweWqq3XPLh4ol4OxM2f6Y2QYTkfElJgx0gMZU/0hOt3/OOZ8d PG4cJK5oIqketCAUX9t/vKzVMd4D8a7irED/Yds/OjK/3kYJ0mt3bFmh+1hAlcBC9wngBW3w w7B5obh5xAQfoaCdJXim/1F7dAZPCEniKUWxAgWpaJtmTpDfr5aqx3ZbjURNhkgzVEoLwUAo 6IpddkEckjLQEu7vWsF8SCBr+xK9c4QSI4X784gQ6jek/+6cdOH2HEjFEXxUxRD/uC3V1guv 4biSPfMlKaDCmMNgXox+pITneji0glP7tiJdfcUY4orHDzA9tBm8AmR4vk4SJCxo8SV9XooX PDCsAJpXTlzCX6bm5zkNehatxzEt1cno1jH3KFD9Q3UQ1ra5yDlKmOICY4Jj4HRJ1T38xskY J23G2yviX9z/k3PVJ/vXUiJ0EziPQIWeMEcHAeYFRoT7R+kIMAeOs8SPcLaJfQhmsp5kuhMm JT1ZPvgAREEAYojWLkgiQR34O+L9DbPmLk6B131y9PS3jFDIGukD+h5u8BWwCYR3bNkeCOGA YA6w6MIl2kpwnWG0rAPQMBoqW1PyTKy/oQuBDAD5g1yPaTkCXWiSu4qnSPK0HutK/CoADijz V1ouh+Up2+kCEeqoWAi21W8cYQhogSue7LFZWiMPOgAAAAAAAA== --------------ms020004020804090407090706-- From owner-freebsd-stable@freebsd.org Thu Apr 11 22:38:24 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E81F3156C44F for ; Thu, 11 Apr 2019 22:38:23 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: from mail-ua1-x931.google.com (mail-ua1-x931.google.com [IPv6:2607:f8b0:4864:20::931]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 55D318152A for ; Thu, 11 Apr 2019 22:38:22 +0000 (UTC) (envelope-from softwareinfojam@gmail.com) Received: by mail-ua1-x931.google.com with SMTP id l22so2592524uao.8 for ; Thu, 11 Apr 2019 15:38:22 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=message-id:mime-version:to:cc:from:subject:date:importance :in-reply-to:references; bh=1nYLS+Gy4UtqASltsQnfYdWvdSBYZO2kqRfMcFkLL+o=; b=H+vVBaPVP2dnzK+iM/OyGNg+/ubX+zguixjK0CB02GsySMDChEriyQNtZ05cvLTPgW Ox+L5kUQadUG269pTMjf90Jogi/1Ns8neiB+4KLCKP1ceGFNosUKZ7ar0NxtX3XcMhIc 1hytqWNIwX+NkWA1OUipoCs+LG+2UvlDooF2Cf5k6LpSbvJu5pCkAUBBmPvYPyYIstDI 47FVI80VvrLf1Bbvc6bFiWj0SHTzF+R5rb3R+qVjgYw/oq08yB0PBLaL9d2iJy52saDH tQzseSCsc4IV3jjrK+xCRrJMPpmG1ONl4CYTcctIoqughMiwL/nCEw8pEXVGRdBNh0IL 4jLQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:mime-version:to:cc:from:subject:date :importance:in-reply-to:references; bh=1nYLS+Gy4UtqASltsQnfYdWvdSBYZO2kqRfMcFkLL+o=; b=VZzLXab1BkxXz/t/P7L3KlXTJ8zaMoEaLBXftNT25NKQTsxwsNeflDmj6aOf724J+J lkn891B5Cx0c+WXuQVt8AZ5359npuwO+v8A2hspzYxZzcBl1fej7DRvWaRXoMxhRkX3Z u1rIqUfjfImXixm6E3RpVEI6M5WaPv1jsS+x1d7CYEtpaH3GoNrb0ZKyj+Z7zmWidhD3 fpMpYIjQle95ZfhQFEQ9sxaMvudt6eDe3C6V11VaIqe9Fy2SjxybUBtqfdEyNzMHC+HG Y6Z8fXX7iiLI8y5XCqmmdwCUg8lZamx95V/zusmjfJ9lPBRRBh1HQ0W0Mbv74MTu3Dtc fGJg== X-Gm-Message-State: APjAAAUgVGZyBlUzSFza7ptI0RlBQX4rLPIgKAKrgvPO1M2HbVqwqXVD uNP8FX3itgeVrhwo4GkhzzA= X-Google-Smtp-Source: APXvYqw+b7E7TUlizaoFngADGU/ZvzbeGHyT2fPFWDx12AX3HAgvzKubuXhhwyPbfIeqU9iwgXnpZw== X-Received: by 2002:a9f:3c01:: with SMTP id u1mr28547070uah.19.1555022301684; Thu, 11 Apr 2019 15:38:21 -0700 (PDT) Received: from ?IPv6:::ffff:192.10.1.119? (git.mayberryinv.com. [63.143.111.202]) by smtp.gmail.com with ESMTPSA id h78sm4828185vka.48.2019.04.11.15.38.20 (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 11 Apr 2019 15:38:21 -0700 (PDT) Message-ID: <5cafc1dd.1c69fb81.8462d.ffaf@mx.google.com> MIME-Version: 1.0 To: Richard Mackerras Cc: Walter Cramer , "freebsd-stable@freebsd.org" , Jonathan Chen From: Software Info Subject: RE: Crontab Question Date: Thu, 11 Apr 2019 17:38:21 -0500 Importance: normal X-Priority: 3 In-Reply-To: <36054B2F-3456-46C1-BE1A-FB90551E2AE7@gmail.com> References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> <20190410172638.C14867@mulder.mintsol.com> <5caf5d3d.1c69fb81.63ae1.ffd0@mx.google.com> <36054B2F-3456-46C1-BE1A-FB90551E2AE7@gmail.com> X-Rspamd-Queue-Id: 55D318152A X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=H+vVBaPV; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of softwareinfojam@gmail.com designates 2607:f8b0:4864:20::931 as permitted sender) smtp.mailfrom=softwareinfojam@gmail.com X-Spamd-Result: default: False [-3.76 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(0.00)[+ip6:2607:f8b0:4000::/36]; URI_COUNT_ODD(1.00)[1]; RCVD_COUNT_THREE(0.00)[3]; RBL_SEM_IPV6(1.00)[1.3.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.bl.ipv6.spameatingmonkey.net]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(0.00)[gmail.com,none]; HAS_X_PRIO_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: alt3.gmail-smtp-in.l.google.com]; FREEMAIL_TO(0.00)[gmail.com]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_TLS_LAST(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.995,0]; R_DKIM_ALLOW(0.00)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[4]; NEURAL_HAM_SHORT(-0.91)[-0.906,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; BAD_REP_POLICIES(0.10)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[1.3.9.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; IP_SCORE(-2.85)[ip: (-9.05), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.18), country: US(-0.06)] Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 11 Apr 2019 22:38:24 -0000 Thanks so much for all the replies. It was true that I had to hardcode ever= y path but thankfully it is working now. Really appreciate the assistance. Kind Regards SI From: Richard Mackerras Sent: Thursday, April 11, 2019 11:53 AM To: Software Info Cc: Walter Cramer; freebsd-stable@freebsd.org; Jonathan Chen Subject: Re: Crontab Question In your script put a few commands outputting to a check file pwd > /tmp/checkfile Add a few more like=20 ENV >> /tmp/checkfile Just to make sure it really is in the directory you expect with the environ= ment you expect.=20 If you want it to be run as you never use the root crontab unless you want = really crap security.=20 Cheers Sent from my iPad > On 11 Apr 2019, at 16:29, Software Info wrote= : >=20 > Well thanks for all the input. I just have to tp keep working at it. Agai= n, much appreciated. >=20 >=20 > Regards > SI >=20 > Sent from Mail for Windows 10 >=20 > From: Walter Cramer > Sent: Wednesday, April 10, 2019 4:40 PM > To: Software Info > Cc: Jonathan Chen; freebsd-stable@freebsd.org > Subject: RE: Crontab Question >=20 >> On Wed, 10 Apr 2019, Software Info wrote: >>=20 >> OK. So although the script is located in my home directory, it doesn=C3= =A2=E2=82=AC=E2=84=A2t=20 >> start there? Sorry but I don=C3=A2=E2=82=AC=E2=84=A2t quite understand.= Could you explain a=20 >> little further please? >=20 > Both 'cp' and 'ls' are located in /bin. But if I run the 'ls' command in= =20 > /root, 'ls' can't find 'cp' (unless I tell it where to look) - even thoug= h=20 > /bin *is* in my PATH - >=20 > server7:/root # ls cp > ls: cp: No such file or directory > server7:/root # ls /bin/cp > /bin/cp >=20 > Where the system looks for *commands*, to execute, is different from wher= e=20 > it looks for other files, which those commands use. The latter is=20 > generally only the current directory (unless you tell it otherwise).=20 > When cron runs a script as root, "current directory" will be /root. >=20 > BUT - for security and other reasons, it would be better to have cron run= =20 > your script as you (not root), and as '/home/me/myscript' (instead of=20 > adding your home directory to PATH in /etc/crontab). >=20 > -Walter >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Fri Apr 12 05:58:30 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 638161573804 for ; Fri, 12 Apr 2019 05:58:30 +0000 (UTC) (envelope-from Willem@offermans.rompen.nl) Received: from cpsmtpb-ews10.kpnxchange.com (cpsmtpb-ews10.kpnxchange.com [213.75.39.15]) by mx1.freebsd.org (Postfix) with ESMTP id EFFCB8CE6E for ; Fri, 12 Apr 2019 05:58:18 +0000 (UTC) (envelope-from Willem@offermans.rompen.nl) Received: from cpsps-ews13.kpnxchange.com ([10.94.84.180]) by cpsmtpb-ews10.kpnxchange.com with Microsoft SMTPSVC(7.5.7601.17514); Fri, 12 Apr 2019 07:58:11 +0200 X-Brand: 7abm2Q== X-KPN-SpamVerdict: e1=0;e2=0;e3=0;e4=(e4=10;e1=10;e3=10;e2=11);EVW:Whi te;BM:NotScanned;FinalVerdict:Clean X-CMAE-Analysis: v=2.3 cv=CfzZG4jl c=1 sm=1 tr=0 cx=a_idp_e a=4/rmT19p7yX2nqNQQg5uwQ==:117 a=4/rmT19p7yX2nqNQQg5uwQ==:17 a=jpOVt7BSZ2e4Z31A5e1TngXxSK0=:19 a=oexKYjalfGEA:10 a=pGLkceISAAAA:8 a=6I5d2MoRAAAA:8 a=5kunse5LzU4oOerN7HIA:9 a=QEXdDO2ut3YA:10 a=r_UuC3295I9e1owF-gsA:9 a=Nv6spcoC3aJsqdWb:21 a=_W_S_7VecoQA:10 a=IjZwj45LgO3ly-622nXo:22 X-CM-AcctID: kpn@feedback.cloudmark.com Received: from smtp.kpnmail.nl ([195.121.84.44]) by cpsps-ews13.kpnxchange.com over TLS secured channel with Microsoft SMTPSVC(8.5.9600.16384); Fri, 12 Apr 2019 07:58:11 +0200 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kpnmail.nl; s=kpnmail01; h=to:message-id:date:from:subject:mime-version:content-type; bh=lzM935rqCbQp4Sbj+s+UAkDkPdb2X/qeK4LojkZV7pw=; b=tBCjrAKxuK3YM03htRUAwOuuG+EAocz9Ak2K0PRvu7Che4lloM6xlvS2ITFYxlhqQIblgRVYMVleK 3xDNCXOpLMdS8iNqv9zUTo1XaYL5FRf9h9MXNBMwn9rKOdNDtH+ippCxsSUh7SXw9Nz4TDnyXCjIAy ySFd0m0soH7306rU= Received: from donald.offrom.nl (unknown [77.164.21.27]) by smtp.kpnmail.nl (Halon) with ESMTPS id f3c47cfc-5ce7-11e9-871b-005056abf0db; Fri, 12 Apr 2019 07:58:11 +0200 (CEST) Received: from [10.168.0.78] (winnie.vpn.offrom.nl [10.168.0.78]) by donald.offrom.nl (8.15.2/8.15.2) with ESMTPS id x3C5w8Kp067607 (version=TLSv1.2 cipher=DHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Fri, 12 Apr 2019 07:58:08 +0200 (CEST) (envelope-from Willem@Offermans.Rompen.nl) Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: Crontab Question From: Willem Offermans X-Priority: 3 In-Reply-To: <5cafc1dd.1c69fb81.8462d.ffaf@mx.google.com> Date: Fri, 12 Apr 2019 07:58:07 +0200 Cc: Richard Mackerras , Walter Cramer , "freebsd-stable@freebsd.org" , Jonathan Chen Message-Id: <70AAF9FB-A8E9-4E64-9A58-5F316DD7945B@Offermans.Rompen.nl> References: <5cae4e6f.1c69fb81.95785.62bf@mx.google.com> <5cae5cc3.1c69fb81.15e0.dbd1@mx.google.com> <20190410172638.C14867@mulder.mintsol.com> <5caf5d3d.1c69fb81.63ae1.ffd0@mx.google.com> <36054B2F-3456-46C1-BE1A-FB90551E2AE7@gmail.com> <5cafc1dd.1c69fb81.8462d.ffaf@mx.google.com> To: Software Info X-Mailer: Apple Mail (2.3445.104.8) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,HTML_MESSAGE, URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on donald.offrom.nl X-OriginalArrivalTime: 12 Apr 2019 05:58:11.0548 (UTC) FILETIME=[B5ECF9C0:01D4F0F4] X-RcptDomain: freebsd.org X-Rspamd-Queue-Id: EFFCB8CE6E X-Spamd-Bar: ---- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=kpnmail.nl header.s=kpnmail01 header.b=tBCjrAKx X-Spamd-Result: default: False [-4.05 / 15.00]; TO_DN_EQ_ADDR_SOME(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; RCPT_COUNT_FIVE(0.00)[5]; DKIM_TRACE(0.00)[kpnmail.nl:+]; MX_GOOD(-0.01)[donald.offermans.rompen.nl]; HAS_X_PRIO_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.93)[-0.932,0]; FREEMAIL_TO(0.00)[gmail.com]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; IP_SCORE(-1.41)[ipnet: 213.75.0.0/17(-3.91), asn: 8737(-3.13), country: NL(0.01)]; ASN(0.00)[asn:8737, ipnet:213.75.0.0/17, country:NL]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[27.21.164.77.zen.spamhaus.org : 127.0.0.10]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[kpnmail.nl:s=kpnmail01]; RCVD_COUNT_FIVE(0.00)[5]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[Rompen.nl]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[15.39.75.213.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FREEMAIL_CC(0.00)[gmail.com] Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 05:58:30 -0000 Dear FreeBSD friends, Yes, specifying whole directories is a bit counterintuitive, but you get = used to it. To me it became part of crontab, with only a vague understanding of why. Probably all of us went through this process of incorporation once. Wiel Offermans Willem@Offermans.Rompen.nl > On 12 Apr 2019, at 00:38, Software Info = wrote: >=20 > Thanks so much for all the replies. It was true that I had to hardcode = every path but thankfully it is working now. Really appreciate the = assistance. >=20 >=20 > Kind Regards > SI >=20 >=20 >=20 > From: Richard Mackerras > Sent: Thursday, April 11, 2019 11:53 AM > To: Software Info > Cc: Walter Cramer; freebsd-stable@freebsd.org; Jonathan Chen > Subject: Re: Crontab Question >=20 > In your script put a few commands outputting to a check file >=20 > pwd > /tmp/checkfile >=20 > Add a few more like=20 >=20 > ENV >> /tmp/checkfile >=20 > Just to make sure it really is in the directory you expect with the = environment you expect.=20 >=20 > If you want it to be run as you never use the root crontab unless you = want really crap security.=20 >=20 > Cheers >=20 >=20 > Sent from my iPad >=20 >> On 11 Apr 2019, at 16:29, Software Info = wrote: >>=20 >> Well thanks for all the input. I just have to tp keep working at it. = Again, much appreciated. >>=20 >>=20 >> Regards >> SI >>=20 >> Sent from Mail for Windows 10 >>=20 >> From: Walter Cramer >> Sent: Wednesday, April 10, 2019 4:40 PM >> To: Software Info >> Cc: Jonathan Chen; freebsd-stable@freebsd.org >> Subject: RE: Crontab Question >>=20 >>> On Wed, 10 Apr 2019, Software Info wrote: >>>=20 >>> OK. So although the script is located in my home directory, it = doesn=C3=A2=E2=82=AC=E2=84=A2t=20 >>> start there? Sorry but I don=C3=A2=E2=82=AC=E2=84=A2t quite = understand. Could you explain a=20 >>> little further please? >>=20 >> Both 'cp' and 'ls' are located in /bin. But if I run the 'ls' = command in=20 >> /root, 'ls' can't find 'cp' (unless I tell it where to look) - even = though=20 >> /bin *is* in my PATH - >>=20 >> server7:/root # ls cp >> ls: cp: No such file or directory >> server7:/root # ls /bin/cp >> /bin/cp >>=20 >> Where the system looks for *commands*, to execute, is different from = where=20 >> it looks for other files, which those commands use. The latter is=20 >> generally only the current directory (unless you tell it otherwise).=20= >> When cron runs a script as root, "current directory" will be /root. >>=20 >> BUT - for security and other reasons, it would be better to have cron = run=20 >> your script as you (not root), and as '/home/me/myscript' (instead of=20= >> adding your home directory to PATH in /etc/crontab). >>=20 >> -Walter >>=20 >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" >=20 > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to = "freebsd-stable-unsubscribe@freebsd.org" From owner-freebsd-stable@freebsd.org Fri Apr 12 09:31:18 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id DDB901577A0B for ; Fri, 12 Apr 2019 09:31:17 +0000 (UTC) (envelope-from daisy.williams@expand-online-presence.com) Received: from host.gt.onedumb.com (host.gt.onedumb.com [176.107.131.49]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id DCAC96C57F for ; Fri, 12 Apr 2019 09:31:16 +0000 (UTC) (envelope-from daisy.williams@expand-online-presence.com) Received: from WS99 (unknown [106.215.24.161]) by host.gt.onedumb.com (Postfix) with ESMTPA id E757D3D24 for ; Fri, 12 Apr 2019 05:31:13 -0400 (EDT) From: "Daisy Williams" To: Subject: Brand boosting should not be costly! Date: Fri, 12 Apr 2019 14:57:10 +0530 Message-ID: <107b101d4f112$7bfd7320$73f85960$@expand-online-presence.com> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 14.0 Thread-Index: AdTxAKwKiV/4P4D4QgCZdpSRRzFRTg== Content-Language: en-us X-Rspamd-Queue-Id: DCAC96C57F X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org; spf=softfail (mx1.freebsd.org: 176.107.131.49 is neither permitted nor denied by domain of daisy.williams@expand-online-presence.com) smtp.mailfrom=daisy.williams@expand-online-presence.com X-Spamd-Result: default: False [-0.88 / 15.00]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; NEURAL_HAM_MEDIUM(-0.67)[-0.667,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.56)[-0.560,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; TO_DN_NONE(0.00)[]; R_SPF_SOFTFAIL(0.00)[~all]; RCPT_COUNT_ONE(0.00)[1]; MANY_INVISIBLE_PARTS(0.10)[2]; RCVD_TLS_LAST(0.00)[]; NEURAL_SPAM_SHORT(0.03)[0.033,0]; MX_GOOD(-0.01)[cached: mail.expand-online-presence.com]; DMARC_NA(0.00)[expand-online-presence.com]; SUBJECT_ENDS_EXCLAIM(0.00)[]; IP_SCORE(0.32)[ip: (-3.48), ipnet: 176.107.128.0/19(2.74), asn: 205727(2.29), country: PL(0.07)]; RECEIVED_SPAMHAUS_PBL(0.00)[161.24.215.106.zen.spamhaus.org : 127.0.0.11]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:205727, ipnet:176.107.128.0/19, country:PL]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 09:31:18 -0000 Greetings, Site ranking can be tedious and costly, but it doesn't have to be. We have heard you! Our Pay-for-Performance-only Site Ranking Service is crafted for your brand, you pay after you get the desired search-ranking results only. Sounds too good to be true? Well, we have been making our clients happy for the last 10-years with the same service model, and most of them are still working with us today. Why risk your investment without being sure of the ROI? Because, we know Ranking your Site is not luck, but a technique With us, you get; . No contractual Billing . Proven Results (Before you pay!) . Cancel Anytime . 24x7 Prompt support * Minimum one time set up fee No real reason to wait! Turbocharge your brand today! Simply reply to this email with your contact details and our expert will call you back. Thanks & Regards, Daisy Williams Marketing Manager Head Office: San Jose, CA 95120 Disclaimer: We are using this domain for marketing. If you are interested and want to know about us, just reply to this email, if we have offended you by sending this to you by mistake, we apologize. Please reply "NO" or "UNSUBSCRIBE". From owner-freebsd-stable@freebsd.org Fri Apr 12 11:59:07 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0DA17157A5FF for ; Fri, 12 Apr 2019 11:59:07 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5A28770941 for ; Fri, 12 Apr 2019 11:58:59 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x3CBwq94063427 for ; Fri, 12 Apr 2019 13:58:52 +0200 (CEST) Received: from [217.29.44.158] ([217.29.44.158]) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x3CBwqXn055375 for ; Fri, 12 Apr 2019 13:58:52 +0200 (CEST) (envelope-from hausen@punkt.de) From: "Patrick M. Hausen" Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o and controller resets Date: Fri, 12 Apr 2019 13:58:52 +0200 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> To: FreeBSD-STABLE Mailing List In-Reply-To: Message-Id: <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 5A28770941 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.70 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.960,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[punkt.de]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[mailin.pluspunkthosting.de,mailin.pluspunkthosting.de]; NEURAL_HAM_SHORT(-0.63)[-0.632,0]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; TO_MATCH_ENVRCPT_ALL(0.00)[]; IP_SCORE(-0.30)[ipnet: 217.29.32.0/20(-0.82), asn: 16188(-0.65), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 11:59:07 -0000 Hi all, my problems seem not to be TRIM related after all =E2=80=A6 and I can = now quickly reproduce it. =3D=3D=3D=3D=3D root@freenas01[~]# sysctl vfs.zfs.trim.enabled vfs.zfs.trim.enabled: 0 =3D=3D=3D=3D=3D root@freenas01[~]# cd /mnt/zfs root@freenas01[/mnt/zfs]# dd if=3D/dev/urandom of=3Dhurz bs=3D10m ^C =E2=80=94 system freezes temporarily =3D=3D=3D=3D=3D Apr 12 13:42:16 freenas01 nvme6: resetting controller Apr 12 13:42:16 freenas01 nvme6: aborting outstanding i/o Apr 12 13:42:16 freenas01 nvme6: WRITE sqid:1 cid:117 nsid:1 = lba:981825104 len:176 Apr 12 13:42:16 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 = cid:117 cdw0:0 Apr 12 13:42:49 freenas01 nvme6: resetting controller Apr 12 13:42:50 freenas01 nvme6: aborting outstanding i/o Apr 12 13:42:50 freenas01 nvme6: WRITE sqid:1 cid:127 nsid:1 = lba:984107936 len:96 Apr 12 13:42:50 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 = cid:127 cdw0:0 Apr 12 13:43:35 freenas01 nvme6: resetting controller Apr 12 13:43:35 freenas01 nvme6: aborting outstanding i/o Apr 12 13:43:35 freenas01 nvme6: WRITE sqid:1 cid:112 nsid:1 = lba:976172032 len:176 Apr 12 13:43:35 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 = cid:112 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: resetting controller Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:111 nsid:1 = lba:976199176 len:248 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:111 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:102 nsid:1 = lba:976199432 len:248 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:102 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:112 nsid:1 = lba:976199680 len:8 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:112 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:105 nsid:1 = lba:976199752 len:64 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:105 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:122 nsid:1 = lba:976199816 len:64 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:122 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:103 nsid:1 = lba:976199688 len:64 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:103 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:126 nsid:1 = lba:976200136 len:56 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:126 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:106 nsid:1 = lba:976200192 len:8 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:106 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:107 nsid:1 = lba:976200200 len:64 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:107 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:127 nsid:1 = lba:976200264 len:64 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:127 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:113 nsid:1 = lba:976200328 len:120 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:113 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:108 nsid:1 = lba:976200448 len:72 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:108 cdw0:0 Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:116 nsid:1 = lba:976200520 len:64 Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 = cid:116 cdw0:0 =3D=3D=3D=3D=3D root@freenas01[~]# nvmecontrol identify nvme6 Controller Capabilities/Features =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D Vendor ID: 8086 Subsystem Vendor ID: 8086 Serial Number: BTLJ90230EC61P0FGN Model Number: INTEL SSDPE2KX010T8 Firmware Version: VDV10131 Recommended Arb Burst: 0 IEEE OUI Identifier: e4 d2 5c Multi-Interface Cap: 00 Max Data Transfer Size: 131072 Controller ID: 0x00 Admin Command Set Attributes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Security Send/Receive: Not Supported Format NVM: Supported Firmware Activate/Download: Supported Namespace Managment: Supported Abort Command Limit: 4 Async Event Request Limit: 4 Number of Firmware Slots: 1 Firmware Slot 1 Read-Only: No Per-Namespace SMART Log: No Error Log Page Entries: 64 Number of Power States: 1 NVM Command Set Attributes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D Submission Queue Entry Size Max: 64 Min: 64 Completion Queue Entry Size Max: 16 Min: 16 Number of Namespaces: 1 Compare Command: Not Supported Write Uncorrectable Command: Supported Dataset Management Command: Supported Volatile Write Cache: Not Present Namespace Drive Attributes =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D NVM total cap: 1000204886016 NVM unallocated cap: 0 =3D=3D=3D=3D=3D root@freenas01[~]# zpool status pool: freenas-boot state: ONLINE scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Sun Apr 7 = 03:45:03 2019 config: NAME STATE READ WRITE CKSUM freenas-boot ONLINE 0 0 0 mirror-0 ONLINE 0 0 0 nvd0p2 ONLINE 0 0 0 nvd1p2 ONLINE 0 0 0 errors: No known data errors pool: zfs state: ONLINE scan: scrub repaired 0 in 0 days 00:01:53 with 0 errors on Fri Mar 22 = 19:53:37 2019 config: NAME STATE READ = WRITE CKSUM zfs ONLINE 0 = 0 0 raidz2-0 ONLINE 0 = 0 0 gptid/97d0a7ce-44e5-11e9-982e-0025905f99ac ONLINE 0 = 0 0 gptid/98053880-44e5-11e9-982e-0025905f99ac ONLINE 0 = 0 0 gptid/983a9468-44e5-11e9-982e-0025905f99ac ONLINE 0 = 0 0 gptid/987100f2-44e5-11e9-982e-0025905f99ac ONLINE 0 = 0 0 gptid/98aa6e88-44e5-11e9-982e-0025905f99ac ONLINE 0 = 0 0 gptid/98f20b8c-44e5-11e9-982e-0025905f99ac ONLINE 0 = 0 0 errors: No known data errors =3D=3D=3D=3D=3D The problem only appears on the data pool built from 6 INTEL = SSDPE2KX010T8. The system pool built from two KXG50ZNV256G TOSHIBA does not show any problem with = write load. All the Intel drives have the latest firmware according to the Intel = support website. Could it possibly help to tweak dev.nvme.7.ioq0.num_entries and similar = entries? What about switching to the nda device instead of nvd? Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Fri Apr 12 13:45:26 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 03F78157D2F8 for ; Fri, 12 Apr 2019 13:45:26 +0000 (UTC) (envelope-from kelly@rsw-power.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 68614744E9 for ; Fri, 12 Apr 2019 13:45:25 +0000 (UTC) (envelope-from kelly@rsw-power.com) Received: by mailman.ysv.freebsd.org (Postfix) id 1F6C7157D2F6; Fri, 12 Apr 2019 13:45:25 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id EEE98157D2F4 for ; Fri, 12 Apr 2019 13:45:24 +0000 (UTC) (envelope-from kelly@rsw-power.com) Received: from m97188.mail.qiye.163.com (m97188.mail.qiye.163.com [220.181.97.188]) by mx1.freebsd.org (Postfix) with ESMTP id ADE20744BD for ; Fri, 12 Apr 2019 13:45:04 +0000 (UTC) (envelope-from kelly@rsw-power.com) To: Received: from kelly (unknown [10.105.131.43]) by m97188.mail.qiye.163.com (Hmail) with ESMTP id 9CE8B965D6C; Fri, 12 Apr 2019 21:45:00 +0800 (CST) Message-ID: Subject: =?UTF-8?B?UmU6IEhhbmRoZWxkIFByaW50ZXIgTWFudWZhY3R1cmVy?= X-Priority: 3 X-Mailer: HMail Webmail Server V2.0 Copyright (c) 2015-163.com X-Originating-IP: 119.123.73.41 MIME-Version: 1.0 Received: from kelly@rsw-power.com( [119.123.73.41) ] by ajax-webmail ( [127.0.0.1] ) ; Fri, 12 Apr 2019 17:17:14 +0800 (GMT+08:00) From: kelly Date: Fri, 12 Apr 2019 21:45:00 +0800 (GMT+08:00) X-HM-Spam-Status: e1kIGBQJHllBS1VLV1koWUFJQjdXWS1ZQUlXWQkOFx4IWUFZMjUtOjcyP0 FLVUtZBg++ X-HM-Sender-Digest: e1kMHhlZQQ8JDh5XWRIfHhUPWUFZRzo0IjoVDDgJODUQMhcrFk5PKwwf FwoLIlVKVUpOTk5LTE1MS0tDSU5VMxYaEhdVEB4XFwI7CQgMVgsUDB4JVRgUFkVZV1kUFR4JGAsP WUFKV1kSC1lBWUpKQlVKSUhVTEhVT0pZV1kIAVlBQ09LTjcG X-HM-Tid: 0a6a11cb5a6c20bckuqy9ce8b965d6c X-Rspamd-Queue-Id: ADE20744BD X-Spamd-Bar: +++ Authentication-Results: mx1.freebsd.org; dmarc=pass (policy=none) header.from=rsw-power.com; spf=pass (mx1.freebsd.org: domain of kelly@rsw-power.com designates 220.181.97.188 as permitted sender) smtp.mailfrom=kelly@rsw-power.com X-Spamd-Result: default: False [3.21 / 15.00]; HAS_XOIP(0.00)[]; SUBJ_EXCESS_BASE64(1.50)[]; R_SPF_ALLOW(-0.20)[+ip4:220.181.97.0/24]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[cached: qiye163mx02.mxmail.netease.com]; MIME_BASE64_TEXT(0.10)[]; HAS_X_PRIO_THREE(0.00)[3]; NEURAL_HAM_SHORT(-0.46)[-0.462,0]; DMARC_POLICY_ALLOW(-0.50)[rsw-power.com,none]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; ASN(0.00)[asn:23724, ipnet:220.181.96.0/19, country:CN]; IP_SCORE(0.25)[ipnet: 220.181.96.0/19(0.30), asn: 23724(0.90), country: CN(0.02)]; ARC_NA(0.00)[]; FAKE_REPLY(1.00)[]; NEURAL_HAM_MEDIUM(-0.45)[-0.450,0]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_SPAM_LONG(0.99)[0.988,0]; RCVD_IN_DNSWL_NONE(0.00)[188.97.181.220.list.dnswl.org : 127.0.5.0]; MID_CONTAINS_FROM(1.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: base64 X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 13:45:26 -0000 RGVhciBtYW5hZ2VyLAogCldlIHdhbnQgdG8gaW50cm9kdWNlIHlvdSBhIG5ldyBwcm9kdWN0LS0t SGFuZGhlbGQgSW5ramV0IFByaW50ZXIgd2l0aCAzLjUgaW5jaCBUb3VjaCBTY3JlZW4uIEl0J3Mg dmVyeSBob3Qgc2FsZSBub3cuCiAKWW91IGNhbiBwcmludCBkaWZmZXJlbnQgY29udGVudHM6IGxl dHRlcnMsIG51bWJlcnMsIHN5bWJvbHMsIFFSIGNvZGUsIHNjYW4gY29kZSwgYmFyY29kZSwgYmF0 Y2ggY29kZSwgZXhwaXJ5IGRhdGUsIHRpbWUsIGNvdW50ZXIsIGxvdCBudW1iZXIsIGxvZ28gYW5k IG1hcmtzIHZpYSB0b3VjaCBzY3JlZW4gb3IgaW1wb3J0ZWQgdGhlIHByaW50aW5nIGNvbnRlbnQg dmlhIFVTQiBmbGFzayBkaXNrLgogCk91ciBoYW5kaGVsZCBpbmtqZXQgcHJpbnRlciB3aXRoIHNv bHZlbnQgaW5rIGNhbiBwcmludCBvbiBkaWZmZXJlbnQga2luZHMgb2YgbWF0ZXJpYWxzLCBzdWNo IGFzIHBhcGVyLCBjYXJ0b24sIGxhYmVsLCBwbGFzdGljLCBQRSBiYWcsIGdsYXNzLCB3b29kLCBz dGVlbCBwaXBlLCB0dWJlLCBQVkMsIG1ldGFsLCB3YWxsLCBzdG9uZSBib2FyZCwgZmFicmljLCB0 ZXh0aWxlLCBmb2lsLCBmaWxtIG1hdGVyaWFsLCBldGMuCgogCk91ciBoYW5kaGVsZCBpbmtqZXQg cHJpbnRlciBpcyB2ZXJ5IGNyZWF0aXZlLCBwb3J0YWJsZSBhbmQgY2FuIGJlIHdpZGVseSB1c2Vk IGluIGRpZmZlcmVudCBpbmR1c3RyaWVzLgogCkNvbnRhY3QgdXMgaWYgeW91IG5lZWQsIGxldCdz IHRhbGsgZGV0YWlscy4gb3IgcGxlYXNlIGZvcndhcmQgdGhpcyBlbWFpbCB0byB3aG8gaW50ZXJl c3RlZCBpbiBpdC4KIApLaW5kIHJlZ2FyZHMsCi0tCktlbGx5IENoZW4gCk06IDg2LTEzMjY2Njg1 NjY5ClNreXBlOiBrZWxseV8zMTY3CldlYjp3d3cuYmVzaGVuZ3ByaW50ZXJzLmNvbQoKCgoNCg0K From owner-freebsd-stable@freebsd.org Fri Apr 12 14:43:58 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 14DCA157E3EA for ; Fri, 12 Apr 2019 14:43:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x834.google.com (mail-qt1-x834.google.com [IPv6:2607:f8b0:4864:20::834]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 007BC760FE for ; Fri, 12 Apr 2019 14:43:56 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x834.google.com with SMTP id k14so11538333qtb.0 for ; Fri, 12 Apr 2019 07:43:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=PPaEkKifsEsxR2THc1YsOuRdQpQu4pb8stt5pd9IwOA=; b=P60NcOy7R3VEdzIe5ymJv0h+HLqHuJuX8TI6epGIff9T3i9muRX4ZOpmAg8CuLSDQl cLm6nswgdk6itO4x/XJfpCq3iWsCBVXpO8KVBcWtY/+MLbjgGNtakDtH8iYH6wPDoaiz puyUc+vgpwMFFZH3OYEbhr4P1kok9aa3/xX4Tu7UiaUrscxN1FnDR/O1b/eVSXzzBOhY /7IqIMxjDcX6r2Wq/4xOORiZO3ba37MXnikLUThJKocO+5VFCbhFqo7D+HBQXiSNvdjr 3TgJwzb6PjznKhIHZisDWvsmhqKM/ZbKBUY4lKeyhhrrwoXNa6WKLvhcTl9xmibL3hNA i6XQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=PPaEkKifsEsxR2THc1YsOuRdQpQu4pb8stt5pd9IwOA=; b=H0trFBEsUp+e/Pnna9ubUVcbwxWVQ0Gg3lXnlK63wx9QtXyhWWMs0WlbdB6URUhPsf qyXL6glnqGUK56luDTg9ukR/0N9LJ+Jg/yzmc9fOyunb5PBRYc3QXNn5AwUfpI8F/Ert FV4SKhq/lt03f31Aj3v8UsDRzO+Nyq67kPMzCE0QpYPV9+pRAJ6V7YqyYqq4QMQDkL/t g40oN5aPLEcSiogvxf/npyG0XwHzoJN9nFNrUwkF2ZrkJwjjhqULwtnHJVe7/71HjQHY Hjsok04XORnpP5yX4NsD9SPGjYk3ZY6V8a2/X3QuB/WUdnFDbDdvarmsyF5jReopf0mm 61sw== X-Gm-Message-State: APjAAAWKJNFROZwTE2erGPF6fTrzmvU8wQw3XkpaMakRdoTfYWdu7mbb kAeJ+0yIB0IlvWateT3WoxUnT5K6wvRsX/5ylbbxQIAe X-Google-Smtp-Source: APXvYqwgA/b/WwTmZqk2EN8F1SZ+XrgUwIkbWUXgdGcMLygSAmC28gMCh4uNnLERdj7TjOzrmrEqyq9SMnq6jcUsc1Y= X-Received: by 2002:a0c:afa2:: with SMTP id s31mr46307791qvc.118.1555080236302; Fri, 12 Apr 2019 07:43:56 -0700 (PDT) MIME-Version: 1.0 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> In-Reply-To: <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> From: Warner Losh Date: Fri, 12 Apr 2019 08:43:45 -0600 Message-ID: Subject: Re: NVME aborting outstanding i/o and controller resets To: "Patrick M. Hausen" Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 007BC760FE X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=P60NcOy7 X-Spamd-Result: default: False [-5.48 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.71)[-0.708,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[4.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; MX_GOOD(-0.01)[ALT1.aspmx.l.google.com,aspmx.l.google.com,ALT2.aspmx.l.google.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-2.76)[ip: (-8.63), ipnet: 2607:f8b0::/32(-2.94), asn: 15169(-2.18), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 14:43:58 -0000 On Fri, Apr 12, 2019 at 6:00 AM Patrick M. Hausen wrote: > Hi all, > > my problems seem not to be TRIM related after all =E2=80=A6 and I can now > quickly reproduce it. > > =3D=3D=3D=3D=3D > root@freenas01[~]# sysctl vfs.zfs.trim.enabled > vfs.zfs.trim.enabled: 0 > =3D=3D=3D=3D=3D > root@freenas01[~]# cd /mnt/zfs > root@freenas01[/mnt/zfs]# dd if=3D/dev/urandom of=3Dhurz bs=3D10m > ^C =E2=80=94 system freezes temporarily > This does one I/O at a time to the filesystem, which then repackages the I/Os such that multiple I/Os are going on with the NVMe card. > =3D=3D=3D=3D=3D > Apr 12 13:42:16 freenas01 nvme6: resetting controller > OK. This means that whatever I/O workload we've done has caused the NVME card to stop responding for 30s, so we reset it. > Apr 12 13:42:16 freenas01 nvme6: aborting outstanding i/o > Apr 12 13:42:16 freenas01 nvme6: WRITE sqid:1 cid:117 nsid:1 lba:98182510= 4 > len:176 > Apr 12 13:42:16 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 > cid:117 cdw0:0 > But only one request was in flight... And we keep doing it over and over again, but to different LBAs suggesting that we're stuttering: a few go through and then things wedge again. This happens every 30ish seconds. > Apr 12 13:42:49 freenas01 nvme6: resetting controller > Apr 12 13:42:50 freenas01 nvme6: aborting outstanding i/o > Apr 12 13:42:50 freenas01 nvme6: WRITE sqid:1 cid:127 nsid:1 lba:98410793= 6 > len:96 > Apr 12 13:42:50 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 > cid:127 cdw0:0 > Apr 12 13:43:35 freenas01 nvme6: resetting controller > Apr 12 13:43:35 freenas01 nvme6: aborting outstanding i/o > Apr 12 13:43:35 freenas01 nvme6: WRITE sqid:1 cid:112 nsid:1 lba:97617203= 2 > len:176 > Apr 12 13:43:35 freenas01 nvme6: ABORTED - BY REQUEST (00/07) sqid:1 > cid:112 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: resetting controller > And then this one goes wonkies. > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:111 nsid:1 lba:97619917= 6 > len:248 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:111 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:102 nsid:1 lba:97619943= 2 > len:248 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:102 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:112 nsid:1 lba:97619968= 0 > len:8 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:112 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:105 nsid:1 lba:97619975= 2 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:105 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:122 nsid:1 lba:97619981= 6 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:122 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:103 nsid:1 lba:97619968= 8 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:103 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:126 nsid:1 lba:97620013= 6 > len:56 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:126 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:106 nsid:1 lba:97620019= 2 > len:8 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:106 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:107 nsid:1 lba:97620020= 0 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:107 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:127 nsid:1 lba:97620026= 4 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:127 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:113 nsid:1 lba:97620032= 8 > len:120 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:113 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:108 nsid:1 lba:97620044= 8 > len:72 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:108 cdw0:0 > Apr 12 13:44:06 freenas01 nvme7: aborting outstanding i/o > Apr 12 13:44:06 freenas01 nvme7: WRITE sqid:1 cid:116 nsid:1 lba:97620052= 0 > len:64 > Apr 12 13:44:06 freenas01 nvme7: ABORTED - BY REQUEST (00/07) sqid:1 > cid:116 cdw0:0 > =3D=3D=3D=3D=3D > root@freenas01[~]# nvmecontrol identify nvme6 > Controller Capabilities/Features > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D > Vendor ID: 8086 > Subsystem Vendor ID: 8086 > Serial Number: BTLJ90230EC61P0FGN > Model Number: INTEL SSDPE2KX010T8 > So it's an intel card. > Firmware Version: VDV10131 > Recommended Arb Burst: 0 > IEEE OUI Identifier: e4 d2 5c > Multi-Interface Cap: 00 > Max Data Transfer Size: 131072 > Controller ID: 0x00 > > Admin Command Set Attributes > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D > Security Send/Receive: Not Supported > Format NVM: Supported > Firmware Activate/Download: Supported > Namespace Managment: Supported > Abort Command Limit: 4 > Async Event Request Limit: 4 > Number of Firmware Slots: 1 > Firmware Slot 1 Read-Only: No > Per-Namespace SMART Log: No > Error Log Page Entries: 64 > Number of Power States: 1 > > NVM Command Set Attributes > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > Submission Queue Entry Size > Max: 64 > Min: 64 > Completion Queue Entry Size > Max: 16 > Min: 16 > Number of Namespaces: 1 > Compare Command: Not Supported > Write Uncorrectable Command: Supported > Dataset Management Command: Supported > Volatile Write Cache: Not Present > > Namespace Drive Attributes > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D > NVM total cap: 1000204886016 > NVM unallocated cap: 0 > =3D=3D=3D=3D=3D > root@freenas01[~]# zpool status > pool: freenas-boot > state: ONLINE > scan: scrub repaired 0 in 0 days 00:00:03 with 0 errors on Sun Apr 7 > 03:45:03 2019 > config: > > NAME STATE READ WRITE CKSUM > freenas-boot ONLINE 0 0 0 > mirror-0 ONLINE 0 0 0 > nvd0p2 ONLINE 0 0 0 > nvd1p2 ONLINE 0 0 0 > > errors: No known data errors > > pool: zfs > state: ONLINE > scan: scrub repaired 0 in 0 days 00:01:53 with 0 errors on Fri Mar 22 > 19:53:37 2019 > config: > > NAME STATE READ > WRITE CKSUM > zfs ONLINE 0 > 0 0 > raidz2-0 ONLINE 0 > 0 0 > gptid/97d0a7ce-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/98053880-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/983a9468-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/987100f2-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/98aa6e88-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > gptid/98f20b8c-44e5-11e9-982e-0025905f99ac ONLINE 0 > 0 0 > > errors: No known data errors > =3D=3D=3D=3D=3D > > The problem only appears on the data pool built from 6 INTEL > SSDPE2KX010T8. The system > pool built from two KXG50ZNV256G TOSHIBA does not show any problem with > write load. > OK. That suggests Intel has a problem with their firmware. > All the Intel drives have the latest firmware according to the Intel > support website. > > Could it possibly help to tweak dev.nvme.7.ioq0.num_entries and similar > entries? > It might. Maybe. I'm unsure what the intel firmware bug is. > What about switching to the nda device instead of nvd? > I doubt that would have any effect. They both throw as much I/O onto the card as possible in the default config. There's been some minor improvements in -current here. Any chance you could experimentally try that with this test? You won't get as many I/O abort errors (since we don't print those), and we have a few more workarounds for the reset path (though honestly, it's still kinda stinky). Warner From owner-freebsd-stable@freebsd.org Fri Apr 12 19:22:28 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 65984158437E for ; Fri, 12 Apr 2019 19:22:28 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from kagate.punkt.de (kagate.punkt.de [217.29.33.131]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1A09D8954A for ; Fri, 12 Apr 2019 19:22:26 +0000 (UTC) (envelope-from hausen@punkt.de) Received: from hugo10.ka.punkt.de (hugo10.ka.punkt.de [217.29.44.10]) by gate2.intern.punkt.de with ESMTP id x3CJMOhh069266; Fri, 12 Apr 2019 21:22:24 +0200 (CEST) Received: from [217.29.46.66] (unassigned [217.29.46.66] (may be forged)) by hugo10.ka.punkt.de (8.14.2/8.14.2) with ESMTP id x3CJMNv2068370; Fri, 12 Apr 2019 21:22:23 +0200 (CEST) (envelope-from hausen@punkt.de) Content-Type: text/plain; charset=utf-8 Mime-Version: 1.0 (Mac OS X Mail 12.4 \(3445.104.8\)) Subject: Re: NVME aborting outstanding i/o and controller resets From: "Patrick M. Hausen" In-Reply-To: Date: Fri, 12 Apr 2019 21:22:23 +0200 Cc: FreeBSD-STABLE Mailing List Content-Transfer-Encoding: quoted-printable Message-Id: <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> To: Warner Losh X-Mailer: Apple Mail (2.3445.104.8) X-Rspamd-Queue-Id: 1A09D8954A X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; spf=pass (mx1.freebsd.org: domain of hausen@punkt.de designates 217.29.33.131 as permitted sender) smtp.mailfrom=hausen@punkt.de X-Spamd-Result: default: False [-2.63 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.99)[-0.993,0]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:217.29.32.0/20]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[punkt.de]; NEURAL_HAM_LONG(-1.00)[-0.999,0]; RCVD_COUNT_THREE(0.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: mailin.pluspunkthosting.de]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[131.33.29.217.list.dnswl.org : 127.0.10.0]; NEURAL_HAM_SHORT(-0.54)[-0.540,0]; IP_SCORE(-0.29)[ipnet: 217.29.32.0/20(-0.79), asn: 16188(-0.64), country: DE(-0.01)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:16188, ipnet:217.29.32.0/20, country:DE]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 19:22:28 -0000 Hi Warner, thanks for taking the time again =E2=80=A6 > OK. This means that whatever I/O workload we've done has caused the = NVME card to stop responding for 30s, so we reset it. I figured as much ;-) > So it's an intel card. Yes - I already added this info several times. 6 of them, 2.5=E2=80=9C = NVME =E2=80=9Edisk drives=E2=80=9C. > OK. That suggests Intel has a problem with their firmware. I came across this one: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211713 Is it more probable that Intel has got buggy firmware here than that =E2=80=9Ewe=E2=80=9C are missing interrupts? The mainboard is the Supermicro H11SSW-NT. Two NVME drive bays share a connector on the mainboard: NVMe Ports ( NVMe 0~7, 10, 11, 14, 15) The H11SSW-iN/NT has tweleve (12) NVMe ports (2 ports per 1 Slim = SAS connector) on the motherboard. These ports provide high-speed, low-latency PCI-E 3.0 x4 = connections directly from the CPU to NVMe Solid State (SSD) drives. This greatly increases SSD data- throughput = performance and significantly reduces PCI-E latency by simplifying driver/software requirements resulting = from direct PCI-E interface from the CPU to the NVMe SSD drives. Is this purely mechanical or do two drives share PCI-E resources? Which = would explain why the problems always come in pairs (nvme6 and nvme7, for example). This afternoon I set up a system with 4 drives and I was not able to = reproduce the problem. (We just got 3 more machines which happened to have 4 drives each and no = M.2 directly on the mainboard). I will change the config to 6 drives like with the two FreeNAS systems = in our data center. > [=E2=80=A6 nda(4) ...] > I doubt that would have any effect. They both throw as much I/O onto = the card as possible in the default config. I found out - yes, just the same. > There's been some minor improvements in -current here. Any chance you = could experimentally try that with this test? You won't get as many I/O = abort errors (since we don't print those), and we have a few more = workarounds for the reset path (though honestly, it's still kinda = stinky). HEAD or RELENG_12, too? Kind regards, Patrick --=20 punkt.de GmbH Internet - Dienstleistungen - Beratung Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 76133 Karlsruhe info@punkt.de http://punkt.de AG Mannheim 108285 Gf: Juergen Egeling From owner-freebsd-stable@freebsd.org Fri Apr 12 23:37:07 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 0C3C91588EFE for ; Fri, 12 Apr 2019 23:37:07 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 130B391FBB for ; Fri, 12 Apr 2019 23:37:05 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id x3CND02n069663; Fri, 12 Apr 2019 16:13:00 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201904122313.x3CND02n069663@chez.mckusick.com> From: Kirk McKusick To: Peter Holm Subject: Re: Replicable file-system corruption due to fsck/ufs cc: Jamie Landeg-Jones , jamie@catflap.dyslexicfish.net, Warner Losh , freebsd-stable@freebsd.org X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <20190411043620.GA87473@x2.osted.lan> Comments: In-reply-to Peter Holm message dated "Thu, 11 Apr 2019 06:36:20 +0200." MIME-Version: 1.0 Content-ID: <69657.1555110761.0@chez.mckusick.com> Date: Fri, 12 Apr 2019 16:13:00 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-Rspamd-Queue-Id: 130B391FBB X-Spamd-Bar: ++++++++ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [8.96 / 15.00]; HAS_REPLYTO(0.00)[mckusick@mckusick.com]; TO_DN_SOME(0.00)[]; BROKEN_CONTENT_TYPE(1.50)[]; RCPT_COUNT_FIVE(0.00)[5]; MX_GOOD(-0.01)[cached: chez.mckusick.com]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:46375, ipnet:70.36.128.0/19, country:US]; RCVD_TLS_LAST(0.00)[]; MIME_UNKNOWN(0.10)[application/text]; ARC_NA(0.00)[]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; NEURAL_SPAM_SHORT(0.86)[0.863,0]; MIME_GOOD(-0.10)[multipart/mixed,text/plain]; MIME_BAD_ATTACHMENT(4.00)[txt]; DMARC_NA(0.00)[mckusick.com]; AUTH_NA(1.00)[]; NEURAL_SPAM_MEDIUM(0.78)[0.783,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_SPAM_LONG(0.98)[0.985,0]; RCVD_IN_DNSWL_NONE(0.00)[235.157.36.70.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; GREYLIST(0.00)[pass,body]; IP_SCORE(-0.16)[ip: (-0.02), ipnet: 70.36.128.0/19(-0.01), asn: 46375(-0.72), country: US(-0.06)] X-Spam: Yes Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 12 Apr 2019 23:37:07 -0000 > Peter Holm wrote: > = >> I see this even with a single truncate on HEAD. >> >> $ ./truncate10.sh >> 96 -rw-r--r-- 1 root wheel 1073741824 11 apr. 06:33 test >> ** /dev/md10a >> ** Last Mounted on /mnt >> ** Phase 1 - Check Blocks and Sizes >> INODE 3: FILE SIZE 1073741824 BEYOND END OF ALLOCATED FILE, SIZE SHOULD= BE 268435456 >> ADJUST? yes > = > Thanks.. I should have tested that myself.. doh! I was trying to > closer replicate my real file that triggered the problem which > contained a number of sparse areas. > = > And thanks for adding Kirk to the discussion. I wanted to first be > sure it wasn't just me :-) > = > Cheers, Jamie This is indeed a bug in the calculation of the location of the last block of a file. I believe that the following patch to head will fix it. Peter, can you please test and let me know. If Peter confirms that it fixes the bug, I will check it into head and MFC it to 12-stable and 11-stable after a 2-week settle-in time. Kirk McKusick From owner-freebsd-stable@freebsd.org Sat Apr 13 00:38:01 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 35B9B154E22D for ; Sat, 13 Apr 2019 00:38:01 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x833.google.com (mail-qt1-x833.google.com [IPv6:2607:f8b0:4864:20::833]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 8A04993BD1 for ; Sat, 13 Apr 2019 00:37:59 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x833.google.com with SMTP id z17so13255349qts.13 for ; Fri, 12 Apr 2019 17:37:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ER3s+BE1jCThM9SxAVnAyvCl7BwC9WvOrX4igKBoupQ=; b=Ay61AaDpNyV5Jt8qdkPdKmCPane9b8MIrIaJmhMDtD3UPt0AvJeeU1a1SlxuDpTdo+ XzphJDNAkSKpXqfuLnWIlZO3pX9PYk/A9Tt2/u97ydSM9HxczTtB12MZDoraRSGlpOB8 BQxCIlzQmKlna8ntsSxAdxrbwnNvpnWz6mhe8GqKbm6N5QQ6JE4aOc7HD9goi9uSrBtH pkqcCy82r0nKKF06n5kT47cqD8D/YPIFjPIFAgCqrPUt6CaA8N6RlGoaksRfCyobSttD IjLACeiZP0tWJ9ubWc1Xqe00CmHGVZxX7B/IVF0WvsMentolV04KHjyGbbGx/Vbsdlj1 R+aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ER3s+BE1jCThM9SxAVnAyvCl7BwC9WvOrX4igKBoupQ=; b=IAARF1+7ZkC4O48AEaxctLyEt2ToqHPNZ/t+oG3Vm77xxiWE9kOTXCKh2RhX20uGeZ 6o198VVrAZp+H8IQOFIxOCYiod9PETjG2N4TGUCkCWsqR1EWG6Zq913Cvq8gN2O9eZxz vNibVOpTG3gmIM9wSDNgUt8r3GL+J1Kj73hJV8ZbG3v8RWQodkxZyrOeBeFCe4TsN18W l+tpK4zmBKS3NCIKZ9C+Z2QM3YFAT+nJpgqpQ54LJSa3XkE15jYSa+gCmhcFkLvAH8Ov lgQvoUSemyeRE6uNZtAMTIHGNRwAZQVQFWoGKkOGpmJX89D0HEWqb/aIlwq+b9z6vwZW tudQ== X-Gm-Message-State: APjAAAUlSZfK2riA3OlCO1bcUPIPqvrSCpqpWU4xkrkrXQ9mSN+LmI1c V0dLucSnpleydODiUS8Yih9CVU+CgXGC3Ew2b0Nsb5U5 X-Google-Smtp-Source: APXvYqzKdNYTqlyhRdNlW6mA2FUCYgTNrWQ+qWj5p3GV2YYFrj7e2Ox0zP7BjNJdvuHbrRDq83VS2blgB71aYxZykmA= X-Received: by 2002:ac8:5493:: with SMTP id h19mr35899324qtq.23.1555115879008; Fri, 12 Apr 2019 17:37:59 -0700 (PDT) MIME-Version: 1.0 References: <818CF16A-D71C-47C0-8A1B-35C9D8F68F4E@punkt.de> <58E4FC01-D154-42D4-BA0F-EF9A2C60DBF7@punkt.de> <45D98122-7596-4E8A-8A0D-C33E017C1109@punkt.de> <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> In-Reply-To: <92DAD65A-9BFE-4294-9066-977F498300A3@punkt.de> From: Warner Losh Date: Fri, 12 Apr 2019 18:37:47 -0600 Message-ID: Subject: Re: NVME aborting outstanding i/o and controller resets To: "Patrick M. Hausen" Cc: FreeBSD-STABLE Mailing List X-Rspamd-Queue-Id: 8A04993BD1 X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=Ay61AaDp X-Spamd-Result: default: False [-5.71 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_SHORT(-0.92)[-0.915,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; MX_GOOD(-0.01)[cached: ALT1.aspmx.l.google.com]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; RCPT_COUNT_TWO(0.00)[2]; RCVD_IN_DNSWL_NONE(0.00)[3.3.8.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; MIME_TRACE(0.00)[0:+,1:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; IP_SCORE(-2.79)[ip: (-8.73), ipnet: 2607:f8b0::/32(-2.95), asn: 15169(-2.19), country: US(-0.06)]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 00:38:01 -0000 On Fri, Apr 12, 2019, 1:22 PM Patrick M. Hausen wrote: > Hi Warner, > > thanks for taking the time again =E2=80=A6 > > > OK. This means that whatever I/O workload we've done has caused the NVM= E > card to stop responding for 30s, so we reset it. > > I figured as much ;-) > > > So it's an intel card. > > Yes - I already added this info several times. 6 of them, 2.5=E2=80=9C NV= ME =E2=80=9Edisk > drives=E2=80=9C. > Yea, it was more of a knowing sigh... > OK. That suggests Intel has a problem with their firmware. > > I came across this one: > https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D211713 > > Is it more probable that Intel has got buggy firmware here than that > =E2=80=9Ewe=E2=80=9C are missing interrupts? > More probable bad firmware. One of the things I think that is in HEAD is a mitigation for this that looks for completed IO on timeout before doing a reset. The mainboard is the Supermicro H11SSW-NT. Two NVME drive bays share > a connector on the mainboard: > > NVMe Ports ( NVMe 0~7, 10, 11, 14, 15) > > The H11SSW-iN/NT has tweleve (12) NVMe ports (2 ports per 1 Slim > SAS connector) on the motherboard. > These ports provide high-speed, low-latency PCI-E 3.0 x4 > connections directly from the CPU to NVMe Solid > State (SSD) drives. This greatly increases SSD data- throughput > performance and significantly reduces PCI-E > latency by simplifying driver/software requirements resulting fro= m > direct PCI-E interface from the CPU to the NVMe SSD drives. > > Is this purely mechanical or do two drives share PCI-E resources? Which > would explain > why the problems always come in pairs (nvme6 and nvme7, for example). > I'm unfamiliar with this setup, but coming in pairs increases the missed interrupt theory in my mind. Firmware issues usually don't come in pairs. This afternoon I set up a system with 4 drives and I was not able to > reproduce the problem. > (We just got 3 more machines which happened to have 4 drives each and no > M.2 directly > on the mainboard). > I will change the config to 6 drives like with the two FreeNAS systems in > our data center. > > > [=E2=80=A6 nda(4) ...] > > I doubt that would have any effect. They both throw as much I/O onto th= e > card as possible in the default config. > > I found out - yes, just the same. > NDA drives with an iosched kernel will be able to rate limit, which may be useful as a diagnostic tool... > There's been some minor improvements in -current here. Any chance you > could experimentally try that with this test? You won't get as many I/O > abort errors (since we don't print those), and we have a few more > workarounds for the reset path (though honestly, it's still kinda stinky)= . > > HEAD or RELENG_12, too? > HEAD is preferred, but any recent snapshot will do. Warner Kind regards, > Patrick > -- > punkt.de GmbH Internet - Dienstleistungen - Beratung > Kaiserallee 13a Tel.: 0721 9109-0 Fax: -100 > 76133 Karlsruhe info@punkt.de http://punkt.de > AG Mannheim 108285 Gf: Juergen Egeling > > From owner-freebsd-stable@freebsd.org Sat Apr 13 04:41:13 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 3FB32156BE68 for ; Sat, 13 Apr 2019 04:41:13 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from symbion.zaytman.com (symbion.zaytman.com [64.112.176.10]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "symbion", Issuer "Narawntapu" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 561986CA5C for ; Sat, 13 Apr 2019 04:41:12 +0000 (UTC) (envelope-from mi+thun@aldan.algebra.com) Received: from narawntapu.narawntapu (pool-108-53-86-183.nwrknj.fios.verizon.net [108.53.86.183]) by symbion.zaytman.com (8.15.2/8.15.2) with ESMTPS id x3D4DRjN057621 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=OK) for ; Sat, 13 Apr 2019 00:13:27 -0400 (EDT) (envelope-from mi+thun@aldan.algebra.com) X-Authentication-Warning: symbion.zaytman.com: Host pool-108-53-86-183.nwrknj.fios.verizon.net [108.53.86.183] claimed to be narawntapu.narawntapu Received: from aldan.narawntapu (aldan [192.168.1.10]) by narawntapu.narawntapu (8.15.2/8.15.2) with ESMTP id x3D4DQWA007476 for ; Sat, 13 Apr 2019 00:13:26 -0400 (EDT) (envelope-from mi+thun@aldan.algebra.com) X-Authentication-Warning: narawntapu.narawntapu: Host aldan [192.168.1.10] claimed to be aldan.narawntapu To: freebsd-stable@freebsd.org From: "Mikhail T." Subject: old laptop panics in agp_generic_attach Message-ID: Date: Sat, 13 Apr 2019 00:13:26 -0400 MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------A295F6F9C17A9CAB9E7F1CCD" Content-Language: en-US X-DCC-Etherboy-Metrics: narawntapu 1002; Body=1 Fuz1=1 Fuz2=1 X-Rspamd-Queue-Id: 561986CA5C X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [1.29 / 15.00]; HAS_ATTACHMENT(0.00)[]; HAS_XAW(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[aldan.algebra.com]; CTYPE_MIXED_BOGUS(1.00)[]; FROM_EQ_ENVFROM(0.00)[]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:394548, ipnet:64.112.176.0/24, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+,4:+]; TAGGED_FROM(0.00)[thun]; RECEIVED_SPAMHAUS_PBL(0.00)[183.86.53.108.zen.spamhaus.org : 127.0.0.10]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.68)[-0.678,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-0.00)[-0.002,0]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[algebra.com]; NEURAL_SPAM_SHORT(0.09)[0.089,0]; IP_SCORE(-0.01)[country: US(-0.06)]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 04:41:13 -0000 This is a multi-part message in MIME format. --------------A295F6F9C17A9CAB9E7F1CCD Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 8bit Hello! I wanted to take an old laptop with me for a trip, but decided to update the OS on it (it was running 8.1). Long story short, both 11.2 and 12.0 panic on boot... I can boot 8.2 via pxeboot (dmesg attached) off of my server, which is neat, but that's it. When it crashes -- and it does that whether I boot from the local disk or via pxeboot -- it happens so fast, I could not even see, where exactly. Fortunately, an HD-60 video-recording captured the panic: * make_dev_sv * make_dev * agp_generic_attach Are there any boot-time options I can supply to load 12.0 -- and then build custom kernel with a chance of working?     -mi --------------A295F6F9C17A9CAB9E7F1CCD Content-Type: text/plain; charset=UTF-8; name="dmesg-8.2.txt" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="dmesg-8.2.txt" Copyright (c) 1992-2011 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.2-RELEASE #0: Fri Feb 18 02:24:46 UTC 2011 root@almeida.cse.buffalo.edu:/usr/obj/usr/src/sys/GENERIC i386 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Pentium(R) M processor 1000MHz (992.34-MHz 686-class CPU) Origin = "GenuineIntel" Id = 0x695 Family = 6 Model = 9 Stepping = 5 Features=0xa7e9f9bf Features2=0x180 real memory = 805306368 (768 MB) avail memory = 767971328 (732 MB) kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0 cpu0: on acpi0 acpi_ec0: port 0x62,0x66 on acpi0 acpi_lid0: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 pci0: at device 0.1 (no driver attached) pci0: at device 0.3 (no driver attached) vgapci0: port 0x1800-0x1807 mem 0xe8000000-0xefffffff,0xe0000000-0xe007ffff irq 9 at device 2.0 on pci0 agp0: on vgapci0 agp0: detected 3964k stolen memory agp0: aperture size is 128M vgapci1: mem 0xf0000000-0xf7ffffff,0xe0080000-0xe00fffff at device 2.1 on pci0 uhci0: port 0x1820-0x183f irq 9 at device 29.0 on pci0 uhci0: [ITHREAD] usbus0: on uhci0 uhci1: port 0x1840-0x185f irq 9 at device 29.1 on pci0 uhci1: [ITHREAD] usbus1: on uhci1 uhci2: port 0x1860-0x187f at device 29.2 on pci0 uhci2: [ITHREAD] usbus2: on uhci2 ehci0: mem 0xe0100000-0xe01003ff at device 29.7 on pci0 ehci0: [ITHREAD] usbus3: EHCI version 1.0 usbus3: on ehci0 pcib1: at device 30.0 on pci0 pci_link5: BIOS IRQ 3 for 2.5.INTA is invalid pci2: on pcib1 cbb0: irq 9 at device 5.0 on pci2 cardbus0: on cbb0 pccard0: <16-bit PCCard bus> on cbb0 cbb0: [FILTER] fwohci0: mem 0xe0211000-0xe02117ff at device 5.1 on pci2 fwohci0: [ITHREAD] fwohci0: OHCI version 1.0 (ROM=1) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 08:00:46:03:01:7a:95:99 fwohci0: Phy 1394a available S400, 2 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 dcons_crom0: on firewire0 dcons_crom0: bus_addr 0x1068000 fwe0: on firewire0 if_fwe0: Fake Ethernet address: 0a:00:46:7a:95:99 fwe0: Ethernet address: 0a:00:46:7a:95:99 fwip0: on firewire0 fwip0: Firewire address: 08:00:46:03:01:7a:95:99 @ 0xfffe00000000, S400, maxrec 2048 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode fxp0: port 0x3000-0x303f mem 0xe0210000-0xe0210fff irq 9 at device 8.0 on pci2 miibus0: on fxp0 inphy0: PHY 1 on miibus0 inphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto, auto-flow fxp0: Ethernet address: 08:00:46:b7:cc:24 fxp0: [ITHREAD] ath0: mem 0xe0200000-0xe020ffff irq 9 at device 11.0 on pci2 ath0: [ITHREAD] ath0: unable to attach hardware; HAL status 13 device_attach: ath0 attach returned 6 isab0: at device 31.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1810-0x181f at device 31.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 31.3 (no driver attached) pci0: at device 31.5 (no driver attached) pci0: at device 31.6 (no driver attached) acpi_tz0: on acpi0 atrtc0: port 0x70-0x77 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] psm0: irq 12 on atkbdc0 psm0: [GIANT-LOCKED] psm0: [ITHREAD] psm0: model GlidePoint, device ID 0 battery0: on acpi0 acpi_acad0: on acpi0 pmtimer0 on isa0 orm0: at iomem 0xc0000-0xcffff,0xd0000-0xd17ff,0xd8000-0xdbfff,0xdc000-0xdffff pnpid ORM0000 on isa0 sc0: on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 est0: on cpu0 p4tcc0: on cpu0 Timecounter "TSC" frequency 992335582 Hz quality 800 Timecounters tick every 1.000 msec firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 usbus0: 12Mbps Full Speed USB v1.0 usbus1: 12Mbps Full Speed USB v1.0 usbus2: 12Mbps Full Speed USB v1.0 usbus3: 480Mbps High Speed USB v2.0 ugen0.1: at usbus0 uhub0: on usbus0 ugen1.1: at usbus1 uhub1: on usbus1 ugen2.1: at usbus2 uhub2: on usbus2 ugen3.1: at usbus3 uhub3: on usbus3 ad0: 61057MB at ata0-master UDMA100 GEOM: ad0: partition 2 does not start on a track boundary. GEOM: ad0: partition 2 does not end on a track boundary. GEOM: ad0: partition 1 does not end on a track boundary. uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub2: 2 ports with 2 removable, self powered acd0: CDRW at ata1-master UDMA33 Root mount waiting for: usbus3 uhub3: 6 ports with 6 removable, self powered Root mount waiting for: usbus3 ugen3.2: at usbus3 umass0: on usbus3 umass0: 8070i (ATAPI) over Bulk-Only; quirks = 0x0000 umass0: Get Max Lun not supported (USB_ERR_STALLED) Root mount waiting for: usbus3 ugen1.2: at usbus1 umass0:0:0:-1: Attached to scbus0 Trying to mount root from nfs: NFS ROOT: 192.168.1.8:FreeBSD/i386-old (probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0 (probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error (probe0:umass-sim0:0:0:0): SCSI status: Check Condition (probe0:umass-sim0:0:0:0): SCSI sense: NOT READY asc:3a,0 (Medium not present) da0 at umass-sim0 bus 0 scbus0 target 0 lun 0 da0: Removable Direct Access SCSI-0 device da0: 40.000MB/s transfers da0: Attempt to query device size failed: NOT READY, Medium not present fxp0: link state changed to DOWN fxp0: link state changed to UP --------------A295F6F9C17A9CAB9E7F1CCD-- From owner-freebsd-stable@freebsd.org Sat Apr 13 11:00:58 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7E7E51575145 for ; Sat, 13 Apr 2019 11:00:58 +0000 (UTC) (envelope-from karl@denninger.net) Received: from colo1.denninger.net (colo1.denninger.net [104.236.120.189]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 66A287791C for ; Sat, 13 Apr 2019 11:00:57 +0000 (UTC) (envelope-from karl@denninger.net) Received: from denninger.net (ip68-1-57-197.pn.at.cox.net [68.1.57.197]) by colo1.denninger.net (Postfix) with ESMTP id 7552021109C for ; Sat, 13 Apr 2019 07:00:20 -0400 (EDT) Received: from [192.168.10.23] (D13.Denninger.Net [192.168.10.23]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by denninger.net (Postfix) with ESMTPSA id ACCE813AC5 for ; Sat, 13 Apr 2019 06:00:19 -0500 (CDT) Subject: Re: Concern: ZFS Mirror issues (12.STABLE and firmware 19 .v. 20) To: freebsd-stable@freebsd.org References: <9a96b1b5-9337-fcae-1a2a-69d7bb24a5b3@denninger.net> <1866e238-e2a1-ef4e-bee5-5a2f14e35b22@denninger.net> <3d2ad225-b223-e9db-cce8-8250571b92c9@FreeBSD.org> <2bc8a172-6168-5ba9-056c-80455eabc82b@denninger.net> <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> From: Karl Denninger Openpgp: preference=signencrypt Autocrypt: addr=karl@denninger.net; prefer-encrypt=mutual; keydata= mQINBFIX1zsBEADRcJfsQUl9oFeoMfLPJ1kql+3sIaYx0MfJAUhV9LnbWxr0fsWCskM1O4cV tHm5dqPkuPM4Ztc0jLotD1i9ubWvCHOlkLGxFOL+pFbjA+XZ7VKsC/xWmhMwJ3cM8HavK2OV SzEWQ/AEYtMi04IzGSwsxh/5/5R0mPHrsIomV5SbuiI0vjLuDj7fo6146AABI1ULzge4hBYW i/SHrqUrLORmUNBs6bxek79/B0Dzk5cIktD3LOfbT9EAa5J/osVkstMBhToJgQttaMIGv8SG CzpR/HwEokE+7DP+k2mLHnLj6H3kfugOF9pJH8Za4yFmw//s9cPXV8WwtZ2SKfVzn1unpKqf wmJ1PwJoom/d4fGvQDkgkGKRa6RGC6tPmXnqnx+YX4iCOdFfbP8L9rmk2sewDDVzHDU3I3ZZ 8hFIjMYM/QXXYszRatK0LCV0QPZuF7LCf4uQVKw1/oyJInsnH7+6a3c0h21x+CmSja9QJ+y0 yzgEN/nM89d6YTakfR+1xkYgodVmMy/bS8kmXbUUZG/CyeqCqc95RUySjKT2ECrf9GhhoQkl +D8n2MsrAUSMGB4GQSN+TIq9OBTpNuvATGSRuF9wnQcs1iSry+JNCpfRTyWp83uCNApe6oHU EET4Et6KDO3AvjvBMAX0TInTRGW2SQlJMuFKpc7Dg7tHK8zzqQARAQABtCNLYXJsIERlbm5p bmdlciA8a2FybEBkZW5uaW5nZXIubmV0PokCPAQTAQIAJgUCUhfXOwIbIwUJCWYBgAYLCQgH AwIEFQIIAwQWAgMBAh4BAheAAAoJEG6/sivc5s0PLxQP/i6x/QFx9G4Cw7C+LthhLXIm7NSH AtNbz2UjySEx2qkoQQjtsK6mcpEEaky4ky6t8gz0/SifIfJmSmyAx0UhUQ0WBv1vAXwtNrQQ jJd9Bj6l4c2083WaXyHPjt2u2Na6YFowyb4SaQb83hu/Zs25vkPQYJVVE0JX409MFVPUa6E3 zFbd1OTr3T4yNUy4gNeQZfzDqDS8slbIks2sXeoJrZ6qqXVI0ionoivOlaN4T6Q0UYyXtigj dQvvhMt0aNowKFjRqrmSDRpdz+o6yg7Mp7qEZ1V6EZk8KqQTH6htpCTQ8i79ttK4LG6bstSF Re6Fwq52nbrcANrcdmtZXqjo+SGbUqJ8b1ggrxAsJ5MEhRh2peKrCgI/TjQo+ZxfnqEoR4AI 46Cyiz+/lcVvlvmf2iPifS3EEdaH3Itfwt7MxFm6mQORYs6skHDw3tOYB2/AdCW6eRVYs2hB RMAG4uwApZfZDKgRoE95PJmQjeTBiGmRPcsQZtNESe7I7EjHtCDLwtJqvD4HkDDQwpzreT6W XkyIJ7ns7zDfA1E+AQhFR6rsTFGgQZRZKsVeov3SbhYKkCnVDCvb/PKQCAGkSZM9SvYG5Yax 8CMry3AefKktf9fqBFg8pWqtVxDwJr56dhi0GHXRu3jVI995rMGo1fLUG5fSxiZ8L5sAtokh 9WFmQpyl Message-ID: <864062ab-f68b-7e63-c3da-539d1e9714f9@denninger.net> Date: Sat, 13 Apr 2019 06:00:20 -0500 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.6.1 MIME-Version: 1.0 In-Reply-To: <2c23c0de-1802-37be-323e-d390037c6a84@denninger.net> Content-Type: multipart/signed; protocol="application/pkcs7-signature"; micalg=sha-512; boundary="------------ms060607080503070505080707" X-Rspamd-Queue-Id: 66A287791C X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.58 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; HAS_ATTACHMENT(0.00)[]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; MX_GOOD(-0.01)[px.denninger.net]; NEURAL_HAM_SHORT(-0.94)[-0.938,0]; FROM_EQ_ENVFROM(0.00)[]; IP_SCORE(-2.43)[ip: (-9.88), ipnet: 104.236.64.0/18(-3.87), asn: 14061(1.68), country: US(-0.06)]; R_DKIM_NA(0.00)[]; ASN(0.00)[asn:14061, ipnet:104.236.64.0/18, country:US]; MIME_TRACE(0.00)[0:+,1:+,2:+]; MID_RHS_MATCH_FROM(0.00)[]; RECEIVED_SPAMHAUS_PBL(0.00)[197.57.1.68.zen.spamhaus.org : 127.0.0.11]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; FROM_HAS_DN(0.00)[]; SIGNED_SMIME(-2.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.20)[multipart/signed,multipart/alternative,text/plain]; RCVD_TLS_LAST(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-stable@freebsd.org]; AUTH_NA(1.00)[]; RCPT_COUNT_ONE(0.00)[1]; DMARC_NA(0.00)[denninger.net]; R_SPF_NA(0.00)[] X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 11:00:58 -0000 This is a cryptographically signed message in MIME format. --------------ms060607080503070505080707 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable On 4/11/2019 13:57, Karl Denninger wrote: > On 4/11/2019 13:52, Zaphod Beeblebrox wrote: >> On Wed, Apr 10, 2019 at 10:41 AM Karl Denninger w= rote: >> >> >>> In this specific case the adapter in question is... >>> >>> mps0: port 0xc000-0xc0ff mem >>> 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on p= ci3 >>> mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd >>> mps0: IOCCapabilities: >>> 1285c >>> >>> Which is indeed a "dumb" HBA (in IT mode), and Zeephod says he connec= ts >>> his drives via dumb on-MoBo direct SATA connections. >>> >> Maybe I'm in good company. My current setup has 8 of the disks connec= ted >> to: >> >> mps0: port 0xb000-0xb0ff mem >> 0xfe240000-0xfe24ffff,0xfe200000-0xfe23ffff irq 32 at device 0.0 on pc= i6 >> mps0: Firmware: 19.00.00.00, Driver: 21.02.00.00-fbsd >> mps0: IOCCapabilities: >> 5a85c >> >> ... just with a cable that breaks out each of the 2 connectors into 4 >> SATA-style connectors, and the other 8 disks (plus boot disks and SSD >> cache/log) connected to ports on... >> >> - ahci0: port >> 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f = mem >> 0xfe900000-0xfe9001ff irq 44 at device 0.0 on pci2 >> - ahci2: port >> 0xa050-0xa057,0xa040-0xa043,0xa030-0xa037,0xa020-0xa023,0xa000-0xa01f = mem >> 0xfe610000-0xfe6107ff irq 40 at device 0.0 on pci7 >> - ahci3: port >> 0xf040-0xf047,0xf030-0xf033,0xf020-0xf027,0xf010-0xf013,0xf000-0xf00f = mem >> 0xfea07000-0xfea073ff irq 19 at device 17.0 on pci0 >> >> ... each drive connected to a single port. >> >> I can actually reproduce this at will. Because I have 16 drives, when= one >> fails, I need to find it. I pull the sata cable for a drive, determin= e if >> it's the drive in question, if not, reconnect, "ONLINE" it and wait fo= r >> resilver to stop... usually only a minute or two. >> >> ... if I do this 4 to 6 odd times to find a drive (I can tell, in gene= ral, >> that a drive is part of the SAS controller or the SATA controllers... = so >> I'm only looking among 8, ever) ... then I "REPLACE" the problem drive= =2E >> More often than not, the a scrub will find a few problems. In fact, i= t >> appears that the most recent scrub is an example: >> >> [1:7:306]dgilbert@vr:~> zpool status >> pool: vr1 >> state: ONLINE >> scan: scrub repaired 32K in 47h16m with 0 errors on Mon Apr 1 23:12= :03 >> 2019 >> config: >> >> NAME STATE READ WRITE CKSUM >> vr1 ONLINE 0 0 0 >> raidz2-0 ONLINE 0 0 0 >> gpt/v1-d0 ONLINE 0 0 0 >> gpt/v1-d1 ONLINE 0 0 0 >> gpt/v1-d2 ONLINE 0 0 0 >> gpt/v1-d3 ONLINE 0 0 0 >> gpt/v1-d4 ONLINE 0 0 0 >> gpt/v1-d5 ONLINE 0 0 0 >> gpt/v1-d6 ONLINE 0 0 0 >> gpt/v1-d7 ONLINE 0 0 0 >> raidz2-2 ONLINE 0 0 0 >> gpt/v1-e0c ONLINE 0 0 0 >> gpt/v1-e1b ONLINE 0 0 0 >> gpt/v1-e2b ONLINE 0 0 0 >> gpt/v1-e3b ONLINE 0 0 0 >> gpt/v1-e4b ONLINE 0 0 0 >> gpt/v1-e5a ONLINE 0 0 0 >> gpt/v1-e6a ONLINE 0 0 0 >> gpt/v1-e7c ONLINE 0 0 0 >> logs >> gpt/vr1log ONLINE 0 0 0 >> cache >> gpt/vr1cache ONLINE 0 0 0 >> >> errors: No known data errors >> >> ... it doesn't say it now, but there were 5 CKSUM errors on one of the= >> drives that I had trial-removed (and not on the one replaced). >> _______________________________________________ > That is EXACTLY what I'm seeing; the "OFFLINE'd" drive is the one that,= > after a scrub, comes up with the checksum errors.=C2=A0 It does *not* f= lag > any errors during the resilver and the drives *not* taken offline do no= t > (ever) show checksum errors either. > > Interestingly enough you have 19.00.00.00 firmware on your card as well= > -- which is what was on mine. > > I have flashed my card forward to 20.00.07.00 -- we'll see if it still > does it when I do the next swap of the backup set. Verrrrrryyyyy interesting. This drive was last written/read under 19.00.00.00.=C2=A0 Yesterday I swa= pped it back in.=C2=A0 Note that right now I am running: mps0: port 0xc000-0xc0ff mem 0xfbb3c000-0xfbb3ffff,0xfbb40000-0xfbb7ffff irq 30 at device 0.0 on pci3 mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd mps0: IOCCapabilities: 1285c And, after the scrub completed overnight.... [karl@NewFS ~]$ zpool status backup =C2=A0 pool: backup =C2=A0state: DEGRADED status: One or more devices has experienced an unrecoverable error.=C2=A0= An =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 attempt was made to correct th= e error.=C2=A0 Applications are unaffected. action: Determine if the device needs to be replaced, and clear the error= s =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 using 'zpool clear' or replace= the device with 'zpool replace'. =C2=A0=C2=A0 see: http://illumos.org/msg/ZFS-8000-9P =C2=A0 scan: scrub repaired 4K in 0 days 06:30:55 with 0 errors on Sat Ap= r 13 01:42:04 2019 config: =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 NAME=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 STATE=C2=A0=C2=A0=C2=A0=C2=A0 READ WRITE CKSUM =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 backup=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0 DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2= =A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 mirror-0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = DEGRADED=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup61.eli=C2=A0=C2=A0=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 265079= 9076683778414=C2=A0 OFFLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2= =A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0 was /dev/gpt/backup62-1.eli =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gpt/ba= ckup62-2.eli=C2=A0=C2=A0 ONLINE=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 0=C2=A0= =C2=A0=C2=A0=C2=A0 0=C2=A0=C2=A0=C2=A0=C2=A0 1 errors: No known data errors The OTHER interesting data point is that the resilver *also* posted one checksum error, which I cleared before doing the scrub.=C2=A0 Both on the= 62-2 device. That would be one block in both cases.=C2=A0 The expected was several (ma= ybe a half-dozen) checksum errors on 19.00.00.00 during the scrub but *zero* during the resilver. The unit which was put *into* the vault and is now offline was written and scrubbed under 20.00.07.00.=C2=A0 The behavior change certainly impli= es that there are some differences and again, none of these OFFLINE state situations are uncontrolled -- in each case the drive is taken offline intentionally, the geli provider is detached and then the unit has "camcontrol standby" executed against it before it is yanked, so in theory at least there should be no way for a unflushed but write-cached block to be lost or damaged. I smell a rat but it may well be in the 19.00.00.00 firmware on the card.= =2E. --=20 Karl Denninger karl@denninger.net /The Market Ticker/ /[S/MIME encrypted email preferred]/ --------------ms060607080503070505080707 Content-Type: application/pkcs7-signature; name="smime.p7s" Content-Transfer-Encoding: base64 Content-Disposition: attachment; filename="smime.p7s" Content-Description: S/MIME Cryptographic Signature MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCC DdgwggagMIIEiKADAgECAhMA5EiKghDOXrvfxYxjITXYDdhIMA0GCSqGSIb3DQEBCwUAMIGL MQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJTmljZXZpbGxlMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExITAf BgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQTAeFw0xNzA4MTcxNjQyMTdaFw0yNzA4 MTUxNjQyMTdaMHsxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkwFwYDVQQKDBBD dWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5c3RlbXMgQ0ExJTAjBgNVBAMMHEN1 ZGEgU3lzdGVtcyBMTEMgMjAxNyBJbnQgQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIK AoICAQC1aJotNUI+W4jP7xQDO8L/b4XiF4Rss9O0B+3vMH7Njk85fZ052QhZpMVlpaaO+sCI KqG3oNEbuOHzJB/NDJFnqh7ijBwhdWutdsq23Ux6TvxgakyMPpT6TRNEJzcBVQA0kpby1DVD 0EKSK/FrWWBiFmSxg7qUfmIq/mMzgE6epHktyRM3OGq3dbRdOUgfumWrqHXOrdJz06xE9NzY vc9toqZnd79FUtE/nSZVm1VS3Grq7RKV65onvX3QOW4W1ldEHwggaZxgWGNiR/D4eosAGFxn uYeWlKEC70c99Mp1giWux+7ur6hc2E+AaTGh+fGeijO5q40OGd+dNMgK8Es0nDRw81lRcl24 SWUEky9y8DArgIFlRd6d3ZYwgc1DMTWkTavx3ZpASp5TWih6yI8ACwboTvlUYeooMsPtNa9E 6UQ1nt7VEi5syjxnDltbEFoLYcXBcqhRhFETJe9CdenItAHAtOya3w5+fmC2j/xJz29og1KH YqWHlo3Kswi9G77an+zh6nWkMuHs+03DU8DaOEWzZEav3lVD4u76bKRDTbhh0bMAk4eXriGL h4MUoX3Imfcr6JoyheVrAdHDL/BixbMH1UUspeRuqQMQ5b2T6pabXP0oOB4FqldWiDgJBGRd zWLgCYG8wPGJGYgHibl5rFiI5Ix3FQncipc6SdUzOQIDAQABo4IBCjCCAQYwHQYDVR0OBBYE FF3AXsKnjdPND5+bxVECGKtc047PMIHABgNVHSMEgbgwgbWAFBu1oRhUMNEzjODolDka5k4Q EDBioYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UEBwwJ TmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRhIFN5 c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYIJAKxAy1WBo2kY MBIGA1UdEwEB/wQIMAYBAf8CAQAwDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4IC AQCB5686UCBVIT52jO3sz9pKuhxuC2npi8ZvoBwt/IH9piPA15/CGF1XeXUdu2qmhOjHkVLN gO7XB1G8CuluxofOIUce0aZGyB+vZ1ylHXlMeB0R82f5dz3/T7RQso55Y2Vog2Zb7PYTC5B9 oNy3ylsnNLzanYlcW3AAfzZcbxYuAdnuq0Im3EpGm8DoItUcf1pDezugKm/yKtNtY6sDyENj tExZ377cYA3IdIwqn1Mh4OAT/Rmh8au2rZAo0+bMYBy9C11Ex0hQ8zWcvPZBDn4v4RtO8g+K uQZQcJnO09LJNtw94W3d2mj4a7XrsKMnZKvm6W9BJIQ4Nmht4wXAtPQ1xA+QpxPTmsGAU0Cv HmqVC7XC3qxFhaOrD2dsvOAK6Sn3MEpH/YrfYCX7a7cz5zW3DsJQ6o3pYfnnQz+hnwLlz4MK 17NIA0WOdAF9IbtQqarf44+PEyUbKtz1r0KGeGLs+VGdd2FLA0e7yuzxJDYcaBTVwqaHhU2/ Fna/jGU7BhrKHtJbb/XlLeFJ24yvuiYKpYWQSSyZu1R/gvZjHeGb344jGBsZdCDrdxtQQcVA 6OxsMAPSUPMrlg9LWELEEYnVulQJerWxpUecGH92O06wwmPgykkz//UmmgjVSh7ErNvL0lUY UMfunYVO/O5hwhW+P4gviCXzBFeTtDZH259O7TCCBzAwggUYoAMCAQICEwCg0WvVwekjGFiO 62SckFwepz0wDQYJKoZIhvcNAQELBQAwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3Jp ZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBD QTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExMQyAyMDE3IEludCBDQTAeFw0xNzA4MTcyMTIx MjBaFw0yMjA4MTYyMTIxMjBaMFcxCzAJBgNVBAYTAlVTMRAwDgYDVQQIDAdGbG9yaWRhMRkw FwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRswGQYDVQQDDBJrYXJsQGRlbm5pbmdlci5uZXQw ggIiMA0GCSqGSIb3DQEBAQUAA4ICDwAwggIKAoICAQC+HVSyxVtJhy3Ohs+PAGRuO//Dha9A 16l5FPATr6wude9zjX5f2lrkRyU8vhCXTZW7WbvWZKpcZ8r0dtZmiK9uF58Ec6hhvfkxJzbg 96WHBw5Fumd5ahZzuCJDtCAWW8R7/KN+zwzQf1+B3MVLmbaXAFBuKzySKhKMcHbK3/wjUYTg y+3UK6v2SBrowvkUBC+jxNg3Wy12GsTXcUS/8FYIXgVVPgfZZrbJJb5HWOQpvvhILpPCD3xs YJFNKEPltXKWHT7Qtc2HNqikgNwj8oqOb+PeZGMiWapsatKm8mxuOOGOEBhAoTVTwUHlMNTg 6QUCJtuWFCK38qOCyk9Haj+86lUU8RG6FkRXWgMbNQm1mWREQhw3axgGLSntjjnznJr5vsvX SYR6c+XKLd5KQZcS6LL8FHYNjqVKHBYM+hDnrTZMqa20JLAF1YagutDiMRURU23iWS7bA9tM cXcqkclTSDtFtxahRifXRI7Epq2GSKuEXe/1Tfb5CE8QsbCpGsfSwv2tZ/SpqVG08MdRiXxN 5tmZiQWo15IyWoeKOXl/hKxA9KPuDHngXX022b1ly+5ZOZbxBAZZMod4y4b4FiRUhRI97r9l CxsP/EPHuuTIZ82BYhrhbtab8HuRo2ofne2TfAWY2BlA7ExM8XShMd9bRPZrNTokPQPUCWCg CdIATQIDAQABo4IBzzCCAcswPAYIKwYBBQUHAQEEMDAuMCwGCCsGAQUFBzABhiBodHRwOi8v b2NzcC5jdWRhc3lzdGVtcy5uZXQ6ODg4ODAJBgNVHRMEAjAAMBEGCWCGSAGG+EIBAQQEAwIF oDAOBgNVHQ8BAf8EBAMCBeAwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMEMDMGCWCG SAGG+EIBDQQmFiRPcGVuU1NMIEdlbmVyYXRlZCBDbGllbnQgQ2VydGlmaWNhdGUwHQYDVR0O BBYEFLElmNWeVgsBPe7O8NiBzjvjYnpRMIHKBgNVHSMEgcIwgb+AFF3AXsKnjdPND5+bxVEC GKtc047PoYGRpIGOMIGLMQswCQYDVQQGEwJVUzEQMA4GA1UECAwHRmxvcmlkYTESMBAGA1UE BwwJTmljZXZpbGxlMRkwFwYDVQQKDBBDdWRhIFN5c3RlbXMgTExDMRgwFgYDVQQLDA9DdWRh IFN5c3RlbXMgQ0ExITAfBgNVBAMMGEN1ZGEgU3lzdGVtcyBMTEMgMjAxNyBDQYITAORIioIQ zl6738WMYyE12A3YSDAdBgNVHREEFjAUgRJrYXJsQGRlbm5pbmdlci5uZXQwDQYJKoZIhvcN AQELBQADggIBAJXboPFBMLMtaiUt4KEtJCXlHO/3ZzIUIw/eobWFMdhe7M4+0u3te0sr77QR dcPKR0UeHffvpth2Mb3h28WfN0FmJmLwJk+pOx4u6uO3O0E1jNXoKh8fVcL4KU79oEQyYkbu 2HwbXBU9HbldPOOZDnPLi0whi/sbFHdyd4/w/NmnPgzAsQNZ2BYT9uBNr+jZw4SsluQzXG1X lFL/qCBoi1N2mqKPIepfGYF6drbr1RnXEJJsuD+NILLooTNf7PMgHPZ4VSWQXLNeFfygoOOK FiO0qfxPKpDMA+FHa8yNjAJZAgdJX5Mm1kbqipvb+r/H1UAmrzGMbhmf1gConsT5f8KU4n3Q IM2sOpTQe7BoVKlQM/fpQi6aBzu67M1iF1WtODpa5QUPvj1etaK+R3eYBzi4DIbCIWst8MdA 1+fEeKJFvMEZQONpkCwrJ+tJEuGQmjoQZgK1HeloepF0WDcviiho5FlgtAij+iBPtwMuuLiL shAXA5afMX1hYM4l11JXntle12EQFP1r6wOUkpOdxceCcMVDEJBBCHW2ZmdEaXgAm1VU+fnQ qS/wNw/S0X3RJT1qjr5uVlp2Y0auG/eG0jy6TT0KzTJeR9tLSDXprYkN2l/Qf7/nT6Q03qyE QnnKiBXWAZXveafyU/zYa7t3PTWFQGgWoC4w6XqgPo4KV44OMYIFBzCCBQMCAQEwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBglghkgBZQMEAgMFAKCCAkUw GAYJKoZIhvcNAQkDMQsGCSqGSIb3DQEHATAcBgkqhkiG9w0BCQUxDxcNMTkwNDEzMTEwMDIw WjBPBgkqhkiG9w0BCQQxQgRACddloN7/b0/J/l4zoyJyrZZLbh9pUdRICJF6K650HvbZ/B8L arpp2OPodhAY+5X3UXEKkCns+3nkfjHRzKoCyDBsBgkqhkiG9w0BCQ8xXzBdMAsGCWCGSAFl AwQBKjALBglghkgBZQMEAQIwCgYIKoZIhvcNAwcwDgYIKoZIhvcNAwICAgCAMA0GCCqGSIb3 DQMCAgFAMAcGBSsOAwIHMA0GCCqGSIb3DQMCAgEoMIGjBgkrBgEEAYI3EAQxgZUwgZIwezEL MAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lzdGVtcyBM TEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0ZW1zIExM QyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTCBpQYLKoZIhvcNAQkQAgsxgZWg gZIwezELMAkGA1UEBhMCVVMxEDAOBgNVBAgMB0Zsb3JpZGExGTAXBgNVBAoMEEN1ZGEgU3lz dGVtcyBMTEMxGDAWBgNVBAsMD0N1ZGEgU3lzdGVtcyBDQTElMCMGA1UEAwwcQ3VkYSBTeXN0 ZW1zIExMQyAyMDE3IEludCBDQQITAKDRa9XB6SMYWI7rZJyQXB6nPTANBgkqhkiG9w0BAQEF AASCAgA5d6ETonQ89mAmU+xm36T2Uh3r1hQ60x0WqJvnUt3wywuZpyNEfA7z+D+Dc4vqfarE fSxIfOW6icfLOCByYPBY1Fwd2PQTVkBIh9YWpyHob3xt9SZBK5wq08FlVqk0paluhwIXmI3K RM52QcU4kOgcVBSL14C/BScY7ScU27a0nFciSQdxV5eEYIzwjKWYQZ5bkM2DaJ7jMQdHCSPq esrJsplVobgebgO4PM8CmiIfRZZFLbLgHx/nXNaiqGBj4sKlKOXokSlIfsUa9yiwWpWuxKa2 IV31lpK0oVvAUMrR7r7/rwr5LaWRLuZSxwWkobo8Et4e4lBieE1oe0e+i31bokeIA6s4qZWr uM9HqdQcvn3q+TAgG81dvTHdU1u+UGnvp8UmJUvBMfi3AXWI0w5RuhuVcTtCEnJUBY+IhyVl SyXFjT6VwIsaX9Hd68VyrvPPYrj3SgZUprXvSd249RDsXgj3WRJxP6+F4dMkdQmDf8bnLShv JKKJCgsUshSaxqVXwrK5iyRuzizunqZ39L4Nz+PyP1CsDiGsaq3Ey4Q8wf7qlIhTc0GjE8tV BHy+MO9UTTKF3usvXQszHhgdlocRdNuNHbgot1Bcf1W5K3hrLFn1KzWwMhNYj19b4o+XS9nN SLrz3SPS7KqMiNN8STthru7smsN2ginhtzQCIUCYBwAAAAAAAA== --------------ms060607080503070505080707-- From owner-freebsd-stable@freebsd.org Sat Apr 13 12:33:00 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 561C51577BB2 for ; Sat, 13 Apr 2019 12:33:00 +0000 (UTC) (envelope-from pho@holm.cc) Received: from relay05.pair.com (relay05.pair.com [216.92.24.67]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 1F54F82A35 for ; Sat, 13 Apr 2019 12:32:58 +0000 (UTC) (envelope-from pho@holm.cc) Received: from x2.osted.lan (87-58-223-204-dynamic.dk.customer.tdc.net [87.58.223.204]) by relay05.pair.com (Postfix) with ESMTP id DC0211A3598; Sat, 13 Apr 2019 08:32:56 -0400 (EDT) Received: from x2.osted.lan (localhost [127.0.0.1]) by x2.osted.lan (8.15.2/8.15.2) with ESMTPS id x3DCWkEr064645 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Sat, 13 Apr 2019 14:32:46 +0200 (CEST) (envelope-from pho@x2.osted.lan) Received: (from pho@localhost) by x2.osted.lan (8.15.2/8.15.2/Submit) id x3DCWjrW064644; Sat, 13 Apr 2019 14:32:45 +0200 (CEST) (envelope-from pho) Date: Sat, 13 Apr 2019 14:32:45 +0200 From: Peter Holm To: Kirk McKusick Cc: Jamie Landeg-Jones , jamie@catflap.dyslexicfish.net, Warner Losh , freebsd-stable@freebsd.org Subject: Re: Replicable file-system corruption due to fsck/ufs Message-ID: <20190413123245.GA64592@x2.osted.lan> References: <20190411043620.GA87473@x2.osted.lan> <201904122313.x3CND02n069663@chez.mckusick.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <201904122313.x3CND02n069663@chez.mckusick.com> User-Agent: Mutt/1.11.1 (2018-12-01) X-Rspamd-Queue-Id: 1F54F82A35 X-Spamd-Bar: / Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-0.50 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-0.96)[-0.957,0]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.09)[0.088,0]; NEURAL_HAM_LONG(-0.69)[-0.687,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[holm.cc]; AUTH_NA(1.00)[]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[mailwash28.pair.com]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[peter@holm.cc,pho@holm.cc]; RECEIVED_SPAMHAUS_PBL(0.00)[204.223.58.87.zen.spamhaus.org : 127.0.0.11]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:7859, ipnet:216.92.0.0/16, country:US]; FROM_NEQ_ENVFROM(0.00)[peter@holm.cc,pho@holm.cc]; IP_SCORE(-0.14)[asn: 7859(-0.63), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 12:33:00 -0000 On Fri, Apr 12, 2019 at 04:13:00PM -0700, Kirk McKusick wrote: > > Peter Holm wrote: > > > >> I see this even with a single truncate on HEAD. > >> > >> $ ./truncate10.sh > >> 96 -rw-r--r-- 1 root wheel 1073741824 11 apr. 06:33 test > >> ** /dev/md10a > >> ** Last Mounted on /mnt > >> ** Phase 1 - Check Blocks and Sizes > >> INODE 3: FILE SIZE 1073741824 BEYOND END OF ALLOCATED FILE, SIZE SHOULD BE 268435456 > >> ADJUST? yes > > > > Thanks.. I should have tested that myself.. doh! I was trying to > > closer replicate my real file that triggered the problem which > > contained a number of sparse areas. > > > > And thanks for adding Kirk to the discussion. I wanted to first be > > sure it wasn't just me :-) > > > > Cheers, Jamie > > This is indeed a bug in the calculation of the location of the last > block of a file. I believe that the following patch to head will > fix it. > > Peter, can you please test and let me know. > > If Peter confirms that it fixes the bug, I will check it into head > and MFC it to 12-stable and 11-stable after a 2-week settle-in time. > > Kirk McKusick > Yes, this patch works for me. -- Peter From owner-freebsd-stable@freebsd.org Sat Apr 13 13:35:41 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 85D331578BF7 for ; Sat, 13 Apr 2019 13:35:41 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (chez.mckusick.com [70.36.157.235]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id D83D4842C5 for ; Sat, 13 Apr 2019 13:35:39 +0000 (UTC) (envelope-from mckusick@mckusick.com) Received: from chez.mckusick.com (localhost [IPv6:::1]) by chez.mckusick.com (8.15.2/8.15.2) with ESMTP id x3DDfjWf090153; Sat, 13 Apr 2019 06:41:45 -0700 (PDT) (envelope-from mckusick@mckusick.com) Message-Id: <201904131341.x3DDfjWf090153@chez.mckusick.com> From: Kirk McKusick To: Peter Holm Subject: Re: Replicable file-system corruption due to fsck/ufs cc: Jamie Landeg-Jones , jamie@catflap.dyslexicfish.net, Warner Losh , freebsd-stable@freebsd.org X-URL: http://WWW.McKusick.COM/ Reply-To: Kirk McKusick In-reply-to: <20190413123245.GA64592@x2.osted.lan> Comments: In-reply-to Peter Holm message dated "Sat, 13 Apr 2019 14:32:45 +0200." MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <90151.1555162905.1@chez.mckusick.com> Date: Sat, 13 Apr 2019 06:41:45 -0700 X-Spam-Status: No, score=-1.4 required=5.0 tests=BAYES_00,MISSING_MID, UNPARSEABLE_RELAY autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on chez.mckusick.com X-Rspamd-Queue-Id: D83D4842C5 X-Spamd-Bar: + Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [1.08 / 15.00]; ARC_NA(0.00)[]; HAS_REPLYTO(0.00)[mckusick@mckusick.com]; REPLYTO_EQ_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_SHORT(0.88)[0.882,0]; NEURAL_HAM_LONG(-0.75)[-0.755,0]; MIME_GOOD(-0.10)[text/plain]; RCVD_TLS_LAST(0.00)[]; DMARC_NA(0.00)[mckusick.com]; AUTH_NA(1.00)[]; RCPT_COUNT_FIVE(0.00)[5]; NEURAL_SPAM_MEDIUM(0.22)[0.221,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; MX_GOOD(-0.01)[cached: chez.mckusick.com]; RCVD_IN_DNSWL_NONE(0.00)[235.157.36.70.list.dnswl.org : 127.0.10.0]; R_SPF_NA(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:46375, ipnet:70.36.128.0/19, country:US]; RCVD_COUNT_TWO(0.00)[2]; IP_SCORE(-0.16)[ip: (-0.02), ipnet: 70.36.128.0/19(-0.01), asn: 46375(-0.71), country: US(-0.06)] X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 13:35:41 -0000 > Date: Sat, 13 Apr 2019 14:32:45 +0200 > From: Peter Holm > To: Kirk McKusick > Cc: Jamie Landeg-Jones , jamie@catflap.dyslexicfish.net, > Warner Losh , freebsd-stable@freebsd.org > Subject: Re: Replicable file-system corruption due to fsck/ufs > > On Fri, Apr 12, 2019 at 04:13:00PM -0700, Kirk McKusick wrote: > >> This is indeed a bug in the calculation of the location of the last >> block of a file. I believe that the following patch to head will >> fix it. >> >> Peter, can you please test and let me know. >> >> If Peter confirms that it fixes the bug, I will check it into head >> and MFC it to 12-stable and 11-stable after a 2-week settle-in time. >> >> Kirk McKusick > > Yes, this patch works for me. > > -- > Peter Great, thanks for the quick test. Now committed to head as -r346185. Kirk From owner-freebsd-stable@freebsd.org Sat Apr 13 17:19:29 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 1EBF5157E2B0 for ; Sat, 13 Apr 2019 17:19:29 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 87DA28AE16 for ; Sat, 13 Apr 2019 17:19:28 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: by mailman.ysv.freebsd.org (Postfix) id 4840A157E2AD; Sat, 13 Apr 2019 17:19:28 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 2416F157E2AC for ; Sat, 13 Apr 2019 17:19:28 +0000 (UTC) (envelope-from filippomore@yahoo.com) Received: from sonic306-21.consmr.mail.ne1.yahoo.com (sonic306-21.consmr.mail.ne1.yahoo.com [66.163.189.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 032228AE14 for ; Sat, 13 Apr 2019 17:19:26 +0000 (UTC) (envelope-from filippomore@yahoo.com) X-YMail-OSG: wMfm6xEVM1meMYQ7Bv3oPxdFbvgvMHT5NGlaBG_nbEXRhj79d27oyYdfk4KZs9k I8wsbY1V_dJXJlYVPN7VEK_vsOYxuCcfPUCnGyo6i19jf0uLfXFwPnWBkA3LpWlGPJAMpx3uVM0j BaNcW3WErgR.Cou.dklMj2crBRkonEY75mcu71qPvHdjvyK5LpvQef5FO2Q_r33NcDR.nKqSr2CS feMN4_isgZ9Gg1kkSU0qfW9fsRmao0VB5Qi6bYtXsOk9D2lCgxbxCMq8N_6g66skRTbZuQzk.wNK dqRybdNMLXwFIq271aQCGBGhwl9hNNk7XQ2Hw05ijwMseY1P4R5XvGRM3Bctrb2oPbjkNDpRdl5D 6uhgJbTuEAiAtH5KwO7JA98aD0lbIoX5XhD_acX3v4teIm8XmMlC.7rnKiclbZdvQeTQWB8Kw5DZ XzQZUtgsUUesHwoTzSImvK6VWxIVIa82yGYTBtTVcTEtOEvKsURTL6PlT3tXxQmTx4pqzjy9t10J Z48ruZohvBalEr9igO6SrH6ylLuTah0EVOzQXR9yfY3OITpsYwqvkjHcpx25Ur2T3hgohjOGcbO9 Hf3H5UHcrQo57Qb29pro.X0_Trtd2WZ3bjMT5PLycdN5b4V0TGWbV1QKeB.Uc1SE4thw_qmDuBDi K0c2x3lSgQGo1bTWEjaxV9CsYy6pdTwHZcXZ6tQCBPUNYrhdz6_9b8ZrIVNbO.l3RkkWP4_abL26 f5Hj3g.zTH_NuvuPCacgZrfL5d._W1eV8DqqT13DEg.Z1ejZsysXNFtlB5Gfz8dEcf6oDAmEUZtx k_qZNXu0Ng4JzRvga8EmowceQel2CFqxscl6ed8uFx1mOWYpqYB2_OR8ovi8KJii5.cuUCAVny86 Q39C8pTIFIPXx5aJei5qYYeL5SsoFI7Xw0x42iaO1KeUZfsO65boEAYDrGjP6AGEKIxXHZxjnUOz iAESdCyBuREYaE.wDZhSj2IuO5G6lO6SX8MoFGTNGiaKkzExPP8m2EVg3YJwZeV7iYxiGCLeFotF eQ2saXt67QOWYX6fjDZEzyEkteKVQ9YDJ.VbYkcH515ZdDSe.B88jH_Pgpr9ZzSyWjzGthXdBBZF Zl8kF0g-- Received: from sonic.gate.mail.ne1.yahoo.com by sonic306.consmr.mail.ne1.yahoo.com with HTTP; Sat, 13 Apr 2019 17:19:20 +0000 Date: Sat, 13 Apr 2019 17:19:17 +0000 (UTC) From: Filippo Moretti To: FreeBSD Stable ML Message-ID: <1813201497.2559247.1555175957578@mail.yahoo.com> Subject: STABLE-12 system does not boot MIME-Version: 1.0 References: <1813201497.2559247.1555175957578.ref@mail.yahoo.com> X-Mailer: WebService/1.1.13212 YMailNorrin Mozilla/5.0 (X11; FreeBSD amd64; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.103 Safari/537.36 X-Rspamd-Queue-Id: 032228AE14 X-Spamd-Bar: ++ X-Spamd-Result: default: False [2.90 / 15.00]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[yahoo.com:s=s2048]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ptr:yahoo.com]; FREEMAIL_FROM(0.00)[yahoo.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_SPAM_SHORT(0.99)[0.990,0]; NEURAL_SPAM_MEDIUM(0.97)[0.969,0]; RCPT_COUNT_ONE(0.00)[1]; RCVD_TLS_LAST(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[yahoo.com:+]; MX_GOOD(-0.01)[cached: mta6.am0.yahoodns.net]; RCVD_IN_DNSWL_NONE(0.00)[83.189.163.66.list.dnswl.org : 127.0.5.0]; DMARC_POLICY_ALLOW(-0.50)[yahoo.com,reject]; IP_SCORE(1.05)[ip: (2.75), ipnet: 66.163.184.0/21(1.43), asn: 36646(1.15), country: US(-0.06)]; NEURAL_SPAM_LONG(0.90)[0.897,0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+]; FREEMAIL_ENVFROM(0.00)[yahoo.com]; ASN(0.00)[asn:36646, ipnet:66.163.184.0/21, country:US]; RCVD_COUNT_TWO(0.00)[2]; DWL_DNSWL_NONE(0.00)[yahoo.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 17:19:29 -0000 I upgraded my amd64 box to today stable but the system no longer boot it ha= ngs or reboots after=C2=A0Loading kernel modulesAny help appreciated.I can = still access the system via kernel old.I did not get any error during the= =C2=A0the update,sincerelyFilippo From owner-freebsd-stable@freebsd.org Sat Apr 13 18:57:10 2019 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4FD891580566 for ; Sat, 13 Apr 2019 18:57:10 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mailman.ysv.freebsd.org (mailman.ysv.freebsd.org [IPv6:2001:1900:2254:206a::50:5]) by mx1.freebsd.org (Postfix) with ESMTP id 8929D8DF12 for ; Sat, 13 Apr 2019 18:57:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mailman.ysv.freebsd.org (Postfix) id 433DE1580565; Sat, 13 Apr 2019 18:57:09 +0000 (UTC) Delivered-To: stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 035771580564 for ; Sat, 13 Apr 2019 18:57:09 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qt1-x82d.google.com (mail-qt1-x82d.google.com [IPv6:2607:f8b0:4864:20::82d]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 941FE8DF11 for ; Sat, 13 Apr 2019 18:57:08 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qt1-x82d.google.com with SMTP id v20so14940980qtv.12 for ; Sat, 13 Apr 2019 11:57:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ypPy5auYNu6Z54tWqGwkerLA9fN7QFQr+eOQAtapxyM=; b=KN17eyzoiiiW0XPgb5G2lmkAMu7h2eh9QeWAVM4P49HPCFJVgTWDP8W9uX3AGhZoPh 9C/zH7Wccgqsh8uRdEibtxUuNkxjyf6afWXNmqIi4QyPQpESUDpY/lup1ZFclMLyfVpz wgx3CBGMeLO3fEzqFzuqoEDJNhgFJKSZfT3xmIKBESRwOH8kHZaC8371JiJSDll6TBIM vvzeiAwGkJxdmoxq4hXaBVhH1m5WKftiApRWuJaqu8ITfpDbYH7x4wlWHMv+0JOaWfdL XWei6mduycOJukcio5X6g1LFYwCgp9/+tqgbiPp/KazyGfgonqKVHibvAriW3e3yXcM+ nhUQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ypPy5auYNu6Z54tWqGwkerLA9fN7QFQr+eOQAtapxyM=; b=UzNtLafx5ZaT59lUxkr1PcTN36U7bOIposZ81uLlmcAA39LaUDAfcOzgLypJTk3fbG P24tSiYPAPd6NkAMI9/8NBUy/hOJaXQNo7vHtew25uUIKy3FWfoFcdD0VK2s00H8Ch5i cT2xXWNOu1PZ6ufrh8ETeeisx8+bKgOhf/VOiIOQynweYrRxaY2n77IPT3k8gJrtdbQx R3+3yrIlmutuNzZ+kEUS+VbX/mqZ5pXfR/ltwniPVoFA9nS2A4Qr0kxJTd19NM6WLZUF kbG7XQ1/schcIMll/Uoo1E+cPSM30RjLqH53+h4FjYi3CkkWuF76jLnuNmtlSDX2PilA gj1A== X-Gm-Message-State: APjAAAW+I25hIVpG2HAjCvsRfbfETq1DcoLnYOMDzw8sbnM31LgR2lNn dF9QQKWHIHy02lE23xS8e3mLhDt9XZFA3wre/s0JiDAO X-Google-Smtp-Source: APXvYqwdXpUrurvMLCYhCvk02e8yT6WeWxGTZVfWcAoMzAX6b2pCNlXLoa604taW6K38RdaTfZUQFnPLXqCBjEVO6k8= X-Received: by 2002:ac8:3328:: with SMTP id t37mr52972374qta.246.1555181828064; Sat, 13 Apr 2019 11:57:08 -0700 (PDT) MIME-Version: 1.0 References: <1813201497.2559247.1555175957578.ref@mail.yahoo.com> <1813201497.2559247.1555175957578@mail.yahoo.com> In-Reply-To: <1813201497.2559247.1555175957578@mail.yahoo.com> From: Warner Losh Date: Sat, 13 Apr 2019 12:52:16 -0600 Message-ID: Subject: Re: STABLE-12 system does not boot To: Filippo Moretti Cc: FreeBSD Stable ML X-Rspamd-Queue-Id: 941FE8DF11 X-Spamd-Bar: ------ Authentication-Results: mx1.freebsd.org X-Spamd-Result: default: False [-6.99 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_SHORT(-0.99)[-0.986,0]; REPLY(-4.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 13 Apr 2019 18:57:10 -0000 What did you upgrade from? Warner On Sat, Apr 13, 2019, 11:21 AM Filippo Moretti via freebsd-stable < freebsd-stable@freebsd.org> wrote: > I upgraded my amd64 box to today stable but the system no longer boot it > hangs or reboots after Loading kernel modulesAny help appreciated.I can > still access the system via kernel old.I did not get any error during > the the update,sincerelyFilippo > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >