From owner-freebsd-fs@freebsd.org Wed Sep 5 05:00:43 2018 Return-Path: Delivered-To: freebsd-fs@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 7C161FE338E; Wed, 5 Sep 2018 05:00:43 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp.freebsd.org", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 12D8570F70; Wed, 5 Sep 2018 05:00:43 +0000 (UTC) (envelope-from delphij@FreeBSD.org) Received: from odin.corp.delphij.net (unknown [IPv6:2601:646:8882:37a:a954:32ec:5043:1ba5]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client did not present a certificate) (Authenticated sender: delphij/mail) by smtp.freebsd.org (Postfix) with ESMTPSA id 5A45111168; Wed, 5 Sep 2018 05:00:42 +0000 (UTC) (envelope-from delphij@FreeBSD.org) To: cem@freebsd.org, Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Mark R V Murray , re@FreeBSD.org References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> From: Xin Li Openpgp: preference=signencrypt Autocrypt: addr=delphij@FreeBSD.org; prefer-encrypt=mutual; keydata= xsFNBFJNzwQBEACuPNSJjL/AD8oHFuG72vtx5P7Q6dpiEbFABgw/IohS65yDZDd3qFH9ssQv AsFafwB/ofsk6t7dx6zIC05dv5qjhGIOKSJxFC4U1HAot9+QpeUG+8boTKZiiycrMruItj2U JANlv+gN5h0mAsL5f9eNzhRM43kdjN8cQnBIujhO54Derjnrnqz6cQtoonV6SvvVJZUQGxHK 5R1XYJ6wiTuvoEuRYnNObJmPFWZyYOaGZz0qqD6Qe1BhkZuRzv2bZxwJc3Raap/GF6Pm9J/c hlYHUmm2QLaXvmoP8WNosNjla1fup0tgYQE+7MTtHFVxmVj9ZTihN3rEL5IkeEKjQAqcpe1n Db8X2o4K262LRpFl8WtVMW2TfN5Avpj+knZMl3tkYGvYK/nfadCr6Af4co9mkhX6QYgkerg2 mXEGaQzSD/omnsxHCfqMgdphaX3B3eoY2Fv36BMpjSdHmm0rmwqjqZaqlZn89vQ/I6ATvLyx JsdHwTbrj57audl/RKC+OpREOJPaVULp1L+9zdBXslILO8MJaT6YEw1T29bEj5jvLm03Y4rF u/YTruHcMPpsGbpJckDKiy6ISAbMtPvz7/KR91xPHS6KExGiIakIX9xpIXIDKgq+ecEWwkFK PogoKqO6K0/GYkTRoKdXGzsILvIurtbPqSFqWzbRIyNOa82jowARAQABzRZYaW4gTGkgPGRA ZGVscGhpai5uZXQ+wsF9BBMBCgAnBQJTQvBFAhsjBQkJZgGABQsJCAcDBRUKCQgLBRYCAwEA Ah4BAheAAAoJEJW2GBstM+nsha4P/2Roa/REjZLZlIG1TKOxEDqmwc3fynX4w2g7/FXA7f7Z YO5N4vnnnQdJbDZDt4TJtiP1NHHdheQ5+loJrrCXVlU31LuJv1ebM2Ajsuo/0l3tfulEf6Ki GoozmaNZAhwiGJkQVg9DSKsea5xIA31lPnFH4T0SKn8Q6F4HYienmJJtlKVTADvYXA+DRmv0 rNOyVe+V/AuTFuelKg3Ua5a+dY3oqtrQQvFS4n7iIrNjEMUBVx0XTrYLddnF+YjXDg5Phf0D pV/2yJOXiTGiZMK6i7vwHZkJvarACoTSrUrr6OBuZv5Gf87VgifZKLr2Fuf+FePiVCoZTQiL 0hPQyABMzeWa32P6BY2LBMMMFvFiyL5pN5k6nJ0nx4skl8UxZ5ay4yyVg2u3f4aI3+m0XlZ+ iixrjmCTGi1s+d/n6E3eFXdJUUbSOXLZaU4qrbXRzTYCZmZViryv7ibtOHXnG6oWy7BFEHuT rUW6OBvsQDTp5iQ6opENJ5/ZzSA3c5p1WS9Ezv4Bpdqcm7LTQX2j6kXikj8YqICtDF2rkKZ2 Ynjm9se9B0h/T1SOaSpbtRg05UKjsinDq2x8EeX21yFs3UyvwePLrGoNKL45EJM0xwxrnlfr M0ayKJNLoYysY78d54hg7XMmkQD/oZz9I+k4fN6CmZ2i5WGH2BgYs0313JMHxSg7zsFNBFJN zwQBEADPtS+nfTKM6PwgSWLDGVgUYQ/RLaKzCcpQAf4ryLBugXpx3s2BBT1bixX7CpsLXKQi +RRETgSFzDaBL9SEs2ZDV2YT+zGp08aijK/Yl9+RIeezAukI3c+XMHuo8ktUWJmo5/1DX07q G30ckG7uFuTnt31sFzwhh/ZeSuLFyel/fWF48KExLDIVa8DyEUJaYvE9Vfph4T/3LkKuzVTy +iwUBLiSLj5G5N70A+4usbL3eKyYrJqCSaLfrP99/nlgBhMAHVcKcv0uqSuiaH9OMqg1VjQs N8j6NDQug9QrbBTM6U7oZWF/AK+CdFoe+leq5MZfzwCevs0BQgxWm4SHMpXL2vtly67QSPMY dl96fOzw8YbKHv1o0ixhCvc37cI9oUVuSJLXKhEEAvWvLuusiuNeoz+6aPlELvD8h5txJqui tVOzctvJ7ktGZTNiz73tKYVdkKaQVyo8QJFLCNLnUulrQ5wXwteYPg6mrpBxu9VqgDrMp7eB T2kaZ4GRBoMWXXPYSIEe5PM5hhNCsSUfqrKj34UZPijPe+HiWoFJ4S5vIpzutiae11Ctki7u XzeLAhOJQB2raraIqDlFP9I9Zj9JOAZhmiKSEWKfOooCNxQYGiUdPrdYnAe+m7FXRomjF0OO gSepNIESt2gOEIbE5cMxQ0gAueNJc58eHCjWhsNJIwARAQABwsFlBBgBCgAPBQJSTc8EAhsM BQkJZgGAAAoJEJW2GBstM+nsh8EP/1sxZpkJelu+smmqaqdrGHlNrFVLOmeN5yr2IGHBUbmF htjr7fVoU8T0mUnlUU724aKPla4nWhMb4NMu+VxRRFGaT2TYpyR6VIxaStycyUdMGjdXV0Pz TGmxFXhNZXKEITXH9sIxuONBp1czl4AgwN7AAl1MKyV13AaLIyajs58mYmuXtyFn/O+4lxh5 nl2Fa3L9YkL9O7QU2p6WAnDky+L3PgUWp1AzJGfYlLZ8XXCi+KK+pnta+f9yKHt/Oqd/s7OC W4mXgFkBrfuSZZofa4eZckh5u0yBYW3OnEJhClgxRbuOhyYwqQr5oxPrQtjtbMiBzbrOkHhy NnrVCFd9EqlojREGDefHo3V+ZlUOc6OoN3CAYnNa2uLEOm5DCuqOE4z5atBCih5EyITPp7JP J2disEP6ddipcilqbnJdP+TyRQwSv5qRNy8cHahD1Cg9XJJHiC3qr+W3eOtqPkJxhU5biPEr 7dljaLS1Ij771brzqO/x5zW1L9py7muXzYBsW8+keKj8LOYs2242KgjI5Og9YhIJGBFBNddQ wxKBKQpytKQOiXwjhk4Nj77U796bsCd/jIS0r0ZUKBEptPyKso7ncfrm163aEmSaDUkiIjyp 9CEOVT87D+VAVh9PyLGP1niQzWEWFSK36tRGZlF0odP1ZB6wub9zq2DxFouSjHgH Organization: The FreeBSD Project Subject: Re: newfs silently fails if random is not ready (?) Message-ID: Date: Tue, 4 Sep 2018 22:00:34 -0700 User-Agent: Thunderbird MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/signed; micalg=pgp-sha512; protocol="application/pgp-signature"; boundary="duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK" X-BeenThere: freebsd-fs@freebsd.org X-Mailman-Version: 2.1.27 Precedence: list List-Id: Filesystems List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 05 Sep 2018 05:00:43 -0000 This is an OpenPGP/MIME signed message (RFC 4880 and 3156) --duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK Content-Type: multipart/mixed; boundary="nbL0vslebR2my3jPF7fmlKUq0uf8jsbOq"; protected-headers="v1" From: Xin Li To: cem@freebsd.org, Lev Serebryakov Cc: FreeBSD Current , freebsd-fs , Mark R V Murray , re@FreeBSD.org Message-ID: Subject: Re: newfs silently fails if random is not ready (?) References: <609400979.20180904230820@serebryakov.spb.ru> <1942661439.20180904235514@serebryakov.spb.ru> <774228883.20180905001035@serebryakov.spb.ru> In-Reply-To: --nbL0vslebR2my3jPF7fmlKUq0uf8jsbOq Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: quoted-printable On 9/4/18 21:39, Conrad Meyer wrote: > With current libc, I instead see: >=20 > load: 0.10 cmd: blocked_random_poc 1668 [randseed] 1.27r 0.00u 0.00s > 0% 2328k (SIGINFO) >=20 > $ procstat -kk 1668 > PID TID COMM TDNAME KSTACK > 1668 100609 blocked_random_poc - mi_switch+0xd3 > sleepq_catch_signals+0x386 sleepq_timedwait_sig+0x12 _sleep+0x272 > read_random_uio+0xb3 sys_getrandom+0xa3 amd64_syscall+0x940 > fast_syscall_common+0x101 >=20 > and: >=20 > $ truss ./blocked_random_poc > ... > getrandom(0x7fffffffd340,40,0) ERR#35 'Resource > temporarily unavailable' > thr_self(0x7fffffffd310) =3D 0 (0x0) > thr_kill(100609,SIGKILL) =3D 0 (0x0) > SIGNAL 9 (SIGKILL) code=3DSI_NOINFO >=20 > So getrandom(2) (via READ_RANDOM_UIO) is returning a bogus EAGAIN > after we have already slept until random was seeded. This bubbles up > to getentropy(3) -> arc4random(3), which sees a surprising failure > from getentropy(3) and raises KILL against the program. >=20 > I believe the EWOULDBLOCK is just a boring leak of tsleep(9)'s timeout > condition. This may be sufficient to fix the problem: >=20 > --- a/sys/dev/random/randomdev.c > +++ b/sys/dev/random/randomdev.c > @@ -156,6 +156,10 @@ READ_RANDOM_UIO(struct uio *uio, bool nonblock) > error =3D tsleep(&random_alg_context, PCATCH, "randseed= ", hz/10); > if (error =3D=3D ERESTART || error =3D=3D EINTR) > break; > + /* Squash hz/10 timeout condition */ > + if (error =3D=3D EWOULDBLOCK) > + error =3D 0; > + KASSERT(error =3D=3D 0, ("unexpected %d", error)); > } > if (error =3D=3D 0) { > read_rate_increment((uio->uio_resid + > sizeof(uint32_t))/sizeof(uint32_t)); +markm, re I think the proposed change is reasonable (note that I think the same theory applies to the tsleep_sbt() case below as well, which should be handled similarly). > Best, > Conrad >=20 >=20 > On Tue, Sep 4, 2018 at 8:13 PM, Conrad Meyer wrote: >> Hi Lev, >> >> I took a first attempt at reproducing this problem on a fast >> desktop-class system. First steps, give us a way to revert back to >> unseeded status: >> >> --- a/sys/dev/random/fortuna.c >> +++ b/sys/dev/random/fortuna.c >> @@ -39,6 +39,7 @@ __FBSDID("$FreeBSD$"); >> >> #ifdef _KERNEL >> #include >> +#include >> #include >> #include >> #include >> @@ -384,6 +385,17 @@ random_fortuna_pre_read(void) >> return; >> } >> >> + /* >> + * When set, pretend we do not have enough entropy to reseed y= et. >> + */ >> + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_pre_read, { >> + if (RETURN_VALUE !=3D 0) { >> + RANDOM_RESEED_UNLOCK(); >> + return; >> + } >> + }); >> + >> + >> #ifdef _KERNEL >> fortuna_state.fs_lasttime =3D now; >> #endif >> @@ -442,5 +454,11 @@ bool >> random_fortuna_seeded(void) >> { >> >> + /* When set, act as if we are not seeded. */ >> + KFAIL_POINT_CODE(DEBUG_FP, random_fortuna_seeded, { >> + if (RETURN_VALUE !=3D 0) >> + fortuna_state.fs_counter =3D UINT128_ZERO; >> + }); >> + >> return (!uint128_is_zero(fortuna_state.fs_counter)); >> } >> >> >> Second step, enable the failpoints and launch repro program: >> >> $ sudo sysctl debug.fail_point.random_fortuna_pre_read=3D'return(1)' >> debug.fail_point.random_fortuna_pre_read: off -> return(1) >> $ sudo sysctl debug.fail_point.random_fortuna_seeded=3D'return(1)' >> debug.fail_point.random_fortuna_seeded: off -> return(1) >> >> $ cat ./blocked_random_poc.c >> #include >> #include >> #include >> >> int >> main(int argc, char **argv) >> { >> printf("%x\n", arc4random()); >> return (0); >> } >> >> >> $ ./blocked_random_poc >> ... >> >> >> Third step, I looked at what that process was doing: >> >> Curiously, it is not in getrandom() at all, but instead the ARND >> sysctl fallback. I probably need to rebuild world (libc) to test this= >> (new libc arc4random based on Chacha). >> >> $ procstat -kk 1196 >> PID TID COMM TDNAME KSTACK >> 1196 100435 blocked_random_poc - read_random+0x3d >> sysctl_kern_arnd+0x3a sysctl_root_handler_locked+0x89 >> sysctl_root.isra.8+0x167 userland_sysctl+0x126 sys___sysctl+0x7b >> amd64_syscall+0x940 fast_syscall_common+0x101 >> >> >> When I unblocked the failpoints, it completed successfully: >> >> $ sudo sysctl debug.fail_point.random_fortuna_pre_read=3D'off' >> debug.fail_point.random_fortuna_pre_read: return(1) -> off >> $ sudo sysctl debug.fail_point.random_fortuna_seeded=3Doff >> debug.fail_point.random_fortuna_seeded: return(1) -> off >> >> ... >> 9e5eb30f >> >> >> Best, >> Conrad --nbL0vslebR2my3jPF7fmlKUq0uf8jsbOq-- --duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK Content-Type: application/pgp-signature; name="signature.asc" Content-Description: OpenPGP digital signature Content-Disposition: attachment; filename="signature.asc" -----BEGIN PGP SIGNATURE----- iQIcBAEBCgAGBQJbj2L3AAoJEJW2GBstM+nsxVwP/jIyR53g2isbfBVdaseuiiCs Ql9eS1x1xzpIxAHPAndb4bPdROmpZzIgeocZZ1wRM1h/A6Z3isS8AJmtww4D6+W7 Hwm+r1nyGDBv7wgUMqMavQs1JIMpimv/pDScbXD43chlB7n5p1BdSdAcQuu4d3Aq eVrm1eIaVzTldmA5TVS9lBtqkXI9RCx0fwDccDujPB2DNxZoHcp+1h7rNkL31yRg UzF8PtaMLgN1LeDT0BXtYsjtUCZgZtJSZ9PzZWFCjGYVitBYIHYrdrXKbLBjDE00 HEVD/Eyb9dhBhJqFQ9kIprcFJujoY9pAjaDL/qIA8ZPCvyUDt7hbIuaWjxZaC2ep RCAAB5btM9KTRpNAsqt0MhSJC+I/dFmWcgheG4+XOEMSUFlluoIfxVeTFDjgOzt8 OUjc2oLyn8uPCsJQg4q48WwrUGH4hDv8hccFJ1WH7rhfMcR8/51jQHvWt7ObKcx+ mHZUoYBgusePhD1OO/XasBcSwmABviXzpWk/Q6UaFbnFa8BX0uYwFH3dNEeIOmBO Y6ZkoWL2Bg3fdyTHYWe1pnysdQ2DowCxyS8RL1HQgoOAOULnkIHK6MKhNGYVJYn7 bK08/nczyKXz1a2vujPboYLwGfTcyYZb0tLwApRnvnU74jao+nuQO06oZ2TU17uB szrA1pupCukN9iedsogh =auFP -----END PGP SIGNATURE----- --duU4at2AbwwH2ZisV9lm5ZIdFCMgF6cyK--