From owner-freebsd-hackers@freebsd.org Fri Aug 16 13:32:28 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 1D2BCCC800 for ; Fri, 16 Aug 2019 13:32:28 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46944W25ftz4Gnt for ; Fri, 16 Aug 2019 13:32:27 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x743.google.com with SMTP id s145so4679977qke.7 for ; Fri, 16 Aug 2019 06:32:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=ZzCZlSD930PVSdTOQGNkdijNEQWIoUByIpCF06D8ewE=; b=QFhbWYKuc/SUE2Nkakqq3hsoCJhqqs0glShvU0prWiCnlfdOR+Ud4rNQQXHWt9ltcj B5Dv1Z9C1wac8FXZzjE22CgqsF6qZRz1+XzUWp4YKVN3E+w3HPTUNzMWfpMegScZC5xZ KVlkiZsT02vNPdjNycNjWE61yZC452KXguGxBMXUkfVm5ItSSwK+HVYRwI3nD6b/weuy oVJUi5mEhfz21uFEIXj+FtZcGbxm49ljcHm92O//WwTcN1ftZc3ihYtl8ItLhCCmHoQT qS8VNjc3Y21faHtHWWJY3D1MkBVmWJ4KRtKOF1m634umGgqzMl1pMLFvJpPv+8hV5ejA hTTA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=ZzCZlSD930PVSdTOQGNkdijNEQWIoUByIpCF06D8ewE=; b=GwMJFxXwLKVTv/mhbQHmJUXXSybGXFJpev7YYZS1oiwESVAMEhH1VMVDv8xrsUJDDX nlkSTsfvXLd4Wx3jfrB6wbGrRdwqaKeEmtYIw4dQUyEhIKPy+hoYXuL1FFZab11rzCHi PaPruTwSBof2L06LctXxgSjnK1vnUEF/HosCkUitsDXCh/FpfnTH4RHnR1gpmkyn9F+H ghqnEKbMHVn1bX4lwvC7EDtfqNj0GVfU5Rpn5MNt7Y+w0ogVviUV6m7N3Ut9Of6ePcRu B+pO4o846vPw5MBNsr3r5Hstra8vLhiDamL2HYCYx2pxm2ykcUnpqpdEwF8ds7anef9L tK8w== X-Gm-Message-State: APjAAAW0lLftpnJBSfbOUwq0D3fRza7r1bk38PD8funevFUx2Dov20O6 qujobPU57sFML1QTu/6KiJojKUWHwoBHnDGr9MEHxYg5G54= X-Google-Smtp-Source: APXvYqwfZApjaBT8lkI3PIUHg04d+CNEI5lgU9TPNyHe1vOcOq/q0jBpZ/VXsKMESM3WrcK1bkATEWFqp1VhjELpGZs= X-Received: by 2002:a37:9307:: with SMTP id v7mr8666846qkd.495.1565962346030; Fri, 16 Aug 2019 06:32:26 -0700 (PDT) MIME-Version: 1.0 References: <8bbee981-4f95-22eb-d9ec-00267c8e111d@FreeBSD.org> <20190816081406.GM2738@kib.kiev.ua> In-Reply-To: <20190816081406.GM2738@kib.kiev.ua> From: Warner Losh Date: Fri, 16 Aug 2019 07:32:15 -0600 Message-ID: Subject: Re: confused/concerned about ZFS / 64-bit atomics / 32-bit platforms To: Konstantin Belousov Cc: Andriy Gapon , "freebsd-hackers@freebsd.org" X-Rspamd-Queue-Id: 46944W25ftz4Gnt X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=QFhbWYKu; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::743) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.46 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; RCVD_IN_DNSWL_NONE(0.00)[3.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.48)[ip: (2.99), ipnet: 2607:f8b0::/32(-2.96), asn: 15169(-2.38), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2019 13:32:28 -0000 On Fri, Aug 16, 2019 at 2:14 AM Konstantin Belousov wrote: > On Fri, Aug 16, 2019 at 10:23:01AM +0300, Andriy Gapon wrote: > > > > I somewhat confused with respect to what guarantees C provides with > > respect to accessing 64-bit objects on 32-bit platforms, if any. > > I am also concerned about the use of 64-bit atomic values in ZFS given > > that FreeBSD seems to support ZFS on 32-bit platforms (powerpc, i386, > ...). > > > > My concerns stems from a failed import of a ZFS change from illumos. > > That change has this pattern: > > > > volatile uint64_t *p; > > uint64_t x, y; > > ... > > x = *p; > > ... > > atomic_foo_64(p, y); > > > > Specifically, I am concerned that there can be a torn read in x=*p > > assignment on 32-bit platforms even if they provide a native > > implementation of atomic_foo_64(). I am even more concerned about > > platforms where atomic_foo_64() is not available and we need to emulate > > it in opensolaris_atomic.c with the help from atomic_mtx. > > > > In more general terms, I am concerned about plain reads of 64-bit > > variables that are manipulated atomically elsewhere, on 32-bit platforms. > > Is my concern justified? > Yes, your concerns are justified. Plain reads of 64bit variables on 32bit > arches are not atomic. Also we do not provide e.g. atomic_load_64(9) on > 32bit machines. > Should we? I mean for those that support it? > On some architectures, there might be specific tricks to ensure > atomicity of load and store for 64bit vars, e.g. on i386 (actually > Pentium and newer) with use of of the CMPXCHG8 instruction, using it in > dumb mode. ARM seems to provide LDREXD/STREXD. But AFAIK 32 ppc does not > have any means to do it. > My take: On those platforms that can do 64-bit atomics can support this software, those that can't don't get support for this software. i386 and armv[67] are still big enough players that supporting ZFS on them is desirable (though not mandatory). Since they have the functions, we should enable building there and fix bugs that are identified. If there's too many, or they are too elusive, we might want to disable by default selectively. armv6, if it differs from armv7, is likely OK to mark ZFS as broken there given how little power / memory the PiB and Pi0 have as well as a lack of a good way to connect mass storage (just USB which exacerbates the power situation on systems where power is already fragile). PowerPC 32-bit really isn't setup for ZFS either. The systems tend to have less memory and don't tend to have a lot of disk. While it's nice to run ZFS in a single disk setup, UFS is adequate for those situations. Don't build file servers with FreeBSD 13 on this platform. We should mark it as broken here. We already do this with several drivers. mips can do 64-bit atomics on 32-bit platforms, but it's ugly via disabling interrupts (at least I think I committed those).... But even fewer people have 32-bit mips boxes that want ZFS too... We don't build ZFS on mips today, IIRC, so this is kinda moot. If ZFS needs this (now or in the future), and there's some atomics that are possible, but not yet implemented, on these 32-bit platforms, we should mark ZFS broken on those platforms and let their supporters fill in the gaps (though it would be nice to ask for help before doing this if its i386 or armv[67]). Warner > > > > Note that I saw the above access pattern only in the code that is not > > imported yet. I only suspect that that pattern might already be present > > in the current ZFS/FreeBSD code given that it uses 64-bit variables and > > atomics a lot. > > I am not sure if there is a quick way to check that. Maybe > > devel/coccinelle could be used for that. > > > > -- > > Andriy Gapon > > _______________________________________________ > > freebsd-hackers@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > > To unsubscribe, send any mail to " > freebsd-hackers-unsubscribe@freebsd.org" > _______________________________________________ > freebsd-hackers@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers > To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org" >