From owner-freebsd-hackers@freebsd.org Fri Aug 16 13:54:14 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id B8637CD050 for ; Fri, 16 Aug 2019 13:54:14 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) server-signature RSA-PSS (4096 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 4694Yd5mGlz4J28; Fri, 16 Aug 2019 13:54:13 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kib@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id x7GDs4Tm014998 (version=TLSv1.3 cipher=TLS_AES_256_GCM_SHA384 bits=256 verify=NO); Fri, 16 Aug 2019 16:54:07 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua x7GDs4Tm014998 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id x7GDs4Kn014997; Fri, 16 Aug 2019 16:54:04 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Fri, 16 Aug 2019 16:54:04 +0300 From: Konstantin Belousov To: Warner Losh Cc: Andriy Gapon , "freebsd-hackers@freebsd.org" Subject: Re: confused/concerned about ZFS / 64-bit atomics / 32-bit platforms Message-ID: <20190816135404.GB71821@kib.kiev.ua> References: <8bbee981-4f95-22eb-d9ec-00267c8e111d@FreeBSD.org> <20190816081406.GM2738@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.12.1 (2019-06-15) X-Spam-Status: No, score=-1.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FORGED_GMAIL_RCVD,FREEMAIL_FROM, NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.2 X-Spam-Checker-Version: SpamAssassin 3.4.2 (2018-09-13) on tom.home X-Rspamd-Queue-Id: 4694Yd5mGlz4J28 X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=fail reason="No valid SPF, No valid DKIM" header.from=gmail.com (policy=none); spf=softfail (mx1.freebsd.org: 2001:470:d5e7:1::1 is neither permitted nor denied by domain of kostikbel@gmail.com) smtp.mailfrom=kostikbel@gmail.com X-Spamd-Result: default: False [-2.96 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; RCVD_TLS_ALL(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; FREEMAIL_FROM(0.00)[gmail.com]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; HAS_XAW(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_SPF_SOFTFAIL(0.00)[~all]; IP_SCORE_FREEMAIL(0.00)[]; TO_DN_SOME(0.00)[]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.96)[-0.955,0]; IP_SCORE(0.00)[ip: (-2.55), ipnet: 2001:470::/32(-4.47), asn: 6939(-3.04), country: US(-0.05)]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:6939, ipnet:2001:470::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; FREEMAIL_ENVFROM(0.00)[gmail.com]; DMARC_POLICY_SOFTFAIL(0.10)[gmail.com : No valid SPF, No valid DKIM,none] X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2019 13:54:14 -0000 On Fri, Aug 16, 2019 at 07:32:15AM -0600, Warner Losh wrote: > On Fri, Aug 16, 2019 at 2:14 AM Konstantin Belousov > wrote: > > > On Fri, Aug 16, 2019 at 10:23:01AM +0300, Andriy Gapon wrote: > > > > > > I somewhat confused with respect to what guarantees C provides with > > > respect to accessing 64-bit objects on 32-bit platforms, if any. > > > I am also concerned about the use of 64-bit atomic values in ZFS given > > > that FreeBSD seems to support ZFS on 32-bit platforms (powerpc, i386, > > ...). > > > > > > My concerns stems from a failed import of a ZFS change from illumos. > > > That change has this pattern: > > > > > > volatile uint64_t *p; > > > uint64_t x, y; > > > ... > > > x = *p; > > > ... > > > atomic_foo_64(p, y); > > > > > > Specifically, I am concerned that there can be a torn read in x=*p > > > assignment on 32-bit platforms even if they provide a native > > > implementation of atomic_foo_64(). I am even more concerned about > > > platforms where atomic_foo_64() is not available and we need to emulate > > > it in opensolaris_atomic.c with the help from atomic_mtx. > > > > > > In more general terms, I am concerned about plain reads of 64-bit > > > variables that are manipulated atomically elsewhere, on 32-bit platforms. > > > Is my concern justified? > > Yes, your concerns are justified. Plain reads of 64bit variables on 32bit > > arches are not atomic. Also we do not provide e.g. atomic_load_64(9) on > > 32bit machines. > > > > Should we? I mean for those that support it? For i386 the implementation is trivial, I put the patch below, not tested. For arm, apparently there is atomic_load_64/atomic_store_64 already. > > > > On some architectures, there might be specific tricks to ensure > > atomicity of load and store for 64bit vars, e.g. on i386 (actually > > Pentium and newer) with use of of the CMPXCHG8 instruction, using it in > > dumb mode. ARM seems to provide LDREXD/STREXD. But AFAIK 32 ppc does not > > have any means to do it. > > > > My take: On those platforms that can do 64-bit atomics can support this > software, those that can't don't get support for this software. > > i386 and armv[67] are still big enough players that supporting ZFS on them > is desirable (though not mandatory). Since they have the functions, we > should enable building there and fix bugs that are identified. If there's > too many, or they are too elusive, we might want to disable by default > selectively. armv6, if it differs from armv7, is likely OK to mark ZFS as > broken there given how little power / memory the PiB and Pi0 have as well > as a lack of a good way to connect mass storage (just USB which exacerbates > the power situation on systems where power is already fragile). > > PowerPC 32-bit really isn't setup for ZFS either. The systems tend to have > less memory and don't tend to have a lot of disk. While it's nice to run > ZFS in a single disk setup, UFS is adequate for those situations. Don't > build file servers with FreeBSD 13 on this platform. We should mark it as > broken here. We already do this with several drivers. > > mips can do 64-bit atomics on 32-bit platforms, but it's ugly via disabling > interrupts (at least I think I committed those).... But even fewer people > have 32-bit mips boxes that want ZFS too... We don't build ZFS on mips > today, IIRC, so this is kinda moot. > > If ZFS needs this (now or in the future), and there's some atomics that are > possible, but not yet implemented, on these 32-bit platforms, we should > mark ZFS broken on those platforms and let their supporters fill in the > gaps (though it would be nice to ask for help before doing this if its i386 > or armv[67]). diff --git a/sys/i386/include/atomic.h b/sys/i386/include/atomic.h index 0d673af7358..ee2fa421ff8 100644 --- a/sys/i386/include/atomic.h +++ b/sys/i386/include/atomic.h @@ -891,6 +891,8 @@ u_long atomic_swap_long(volatile u_long *p, u_long v); #define atomic_add_rel_64 atomic_add_64 #define atomic_subtract_acq_64 atomic_subtract_64 #define atomic_subtract_rel_64 atomic_subtract_64 +#define atomic_load_64 atomic_load_acq_64 +#define atomic_store_64 atomic_store_rel_64 /* Operations on pointers. */ #define atomic_set_ptr(p, v) \