From owner-freebsd-hackers@freebsd.org Fri Aug 16 13:48:58 2019 Return-Path: Delivered-To: freebsd-hackers@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id BD708CCC81 for ; Fri, 16 Aug 2019 13:48:58 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: from mail-qk1-x743.google.com (mail-qk1-x743.google.com [IPv6:2607:f8b0:4864:20::743]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4694RY6DQsz4HT0 for ; Fri, 16 Aug 2019 13:48:57 +0000 (UTC) (envelope-from wlosh@bsdimp.com) Received: by mail-qk1-x743.google.com with SMTP id m2so4698536qki.12 for ; Fri, 16 Aug 2019 06:48:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bsdimp-com.20150623.gappssmtp.com; s=20150623; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=98kp5uhOb2vtoaf/g5P4gW3b16SQFQWFzirESJByJnk=; b=VB7ngsRblMyfohkcaCRPoAcdz8KsFD8Saat2KMglqKcTa9vJgN5WkPWs1oTtIhgimy 7lahNsOuPwAVL8SvL/2Zj89hsX3XD03hbi5GgbMx+oDdTTcbi3jaXDOlBP1TnygmRt5C eGcbbUpSZU6Xa6N4Su5gHTZsS7sMST79BabeY4Tj/p6YU2ZD+ZWq5w6RdKi2kJFA0Gtc MvCgbx/k1352+EleAQLqds83F1LmATPV9XgJiwVz0MO5Ygu9JWzqVNmTUtcap1j+KL9D TueAbRry6sdIwOktyaZCiAThveED0uHpv8O9oN589SvVedWUUcolFydrDkHj5A0YQ187 uFRw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=98kp5uhOb2vtoaf/g5P4gW3b16SQFQWFzirESJByJnk=; b=faZlfKkgA4TAr/ixRWCo2vgY5TxMRM10ctNcslHlSWUh4t9PVrC0g3LH00awJcbFWc 4wr8ujvSrgVC0qmiK9WOiK8Keaw8VnVjrV/edbKxpFW1knlVROx/UMHrFnSLr+Q5lBBj QRvizNlSXrgvDDPHSS6VDL2ySdJvrwZTvTTbKVoaVSuPsPGYynCsiSqp5ma3Q6tsOlZP hie09qbvraTjIi5L4z7nptQ+9stto4+8g/PrvsOJe6z0XuNfUmBXUR+8fbepMHDvr4u0 wnlLC6UiyBVoRbhYi2dfXhfnjpJx0zlL3kBwE86BdiPxYeuEtiPPaYJdogIqw603ts9p bl6w== X-Gm-Message-State: APjAAAWuZzCVajv973nlTQgrCfxX6DOew21ZW7toVN9vzNfsxB7nAwPq h6yeyCW+50Ft+l4NDwnLbkFNj/tP9AIJ3wXajIMTQSrFQs0= X-Google-Smtp-Source: APXvYqyvtVQEwhkB6u8ZpryNzeU/CS6CLzrfdZd7wM61qSoaHZ1stFKmpW9s5gsoV1sTuWgdmx/7XdT5EJc+FecVYoE= X-Received: by 2002:a37:4b03:: with SMTP id y3mr1140684qka.215.1565963336661; Fri, 16 Aug 2019 06:48:56 -0700 (PDT) MIME-Version: 1.0 References: <8bbee981-4f95-22eb-d9ec-00267c8e111d@FreeBSD.org> <20190816081406.GM2738@kib.kiev.ua> In-Reply-To: From: Warner Losh Date: Fri, 16 Aug 2019 07:48:45 -0600 Message-ID: Subject: Re: confused/concerned about ZFS / 64-bit atomics / 32-bit platforms To: Konstantin Belousov Cc: Andriy Gapon , "freebsd-hackers@freebsd.org" X-Rspamd-Queue-Id: 4694RY6DQsz4HT0 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=bsdimp-com.20150623.gappssmtp.com header.s=20150623 header.b=VB7ngsRb; dmarc=none; spf=none (mx1.freebsd.org: domain of wlosh@bsdimp.com has no SPF policy when checking 2607:f8b0:4864:20::743) smtp.mailfrom=wlosh@bsdimp.com X-Spamd-Result: default: False [-3.46 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; R_DKIM_ALLOW(-0.20)[bsdimp-com.20150623.gappssmtp.com:s=20150623]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; TO_DN_SOME(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-hackers@freebsd.org]; DMARC_NA(0.00)[bsdimp.com]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[bsdimp-com.20150623.gappssmtp.com:+]; NEURAL_HAM_SHORT(-0.98)[-0.980,0]; RCVD_IN_DNSWL_NONE(0.00)[3.4.7.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; R_SPF_NA(0.00)[]; FORGED_SENDER(0.30)[imp@bsdimp.com,wlosh@bsdimp.com]; FREEMAIL_TO(0.00)[gmail.com]; MIME_TRACE(0.00)[0:+,1:+,2:~]; IP_SCORE(-0.48)[ip: (2.98), ipnet: 2607:f8b0::/32(-2.96), asn: 15169(-2.38), country: US(-0.05)]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[imp@bsdimp.com,wlosh@bsdimp.com]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 16 Aug 2019 13:48:58 -0000 [[ sorry to followup, but had a followup thought ]] On Fri, Aug 16, 2019 at 7:32 AM Warner Losh wrote: > > > On Fri, Aug 16, 2019 at 2:14 AM Konstantin Belousov > wrote: > >> On Fri, Aug 16, 2019 at 10:23:01AM +0300, Andriy Gapon wrote: >> > >> > I somewhat confused with respect to what guarantees C provides with >> > respect to accessing 64-bit objects on 32-bit platforms, if any. >> > I am also concerned about the use of 64-bit atomic values in ZFS given >> > that FreeBSD seems to support ZFS on 32-bit platforms (powerpc, i386, >> ...). >> > >> > My concerns stems from a failed import of a ZFS change from illumos. >> > That change has this pattern: >> > >> > volatile uint64_t *p; >> > uint64_t x, y; >> > ... >> > x = *p; >> > ... >> > atomic_foo_64(p, y); >> > >> > Specifically, I am concerned that there can be a torn read in x=*p >> > assignment on 32-bit platforms even if they provide a native >> > implementation of atomic_foo_64(). I am even more concerned about >> > platforms where atomic_foo_64() is not available and we need to emulate >> > it in opensolaris_atomic.c with the help from atomic_mtx. >> > >> > In more general terms, I am concerned about plain reads of 64-bit >> > variables that are manipulated atomically elsewhere, on 32-bit >> platforms. >> > Is my concern justified? >> Yes, your concerns are justified. Plain reads of 64bit variables on 32bit >> arches are not atomic. Also we do not provide e.g. atomic_load_64(9) on >> 32bit machines. >> > > Should we? I mean for those that support it? > Or is the practice pervasive enough that we can't hope to catch them all and we should just mark ZFS broken on all 32-bit platforms because even though it compiles, the torn-write and other cases are too hard to detect and too pervasive to support? > On some architectures, there might be specific tricks to ensure >> atomicity of load and store for 64bit vars, e.g. on i386 (actually >> Pentium and newer) with use of of the CMPXCHG8 instruction, using it in >> dumb mode. ARM seems to provide LDREXD/STREXD. But AFAIK 32 ppc does not >> have any means to do it. >> > > My take: On those platforms that can do 64-bit atomics can support this > software, those that can't don't get support for this software. > > i386 and armv[67] are still big enough players that supporting ZFS on them > is desirable (though not mandatory). Since they have the functions, we > should enable building there and fix bugs that are identified. If there's > too many, or they are too elusive, we might want to disable by default > selectively. armv6, if it differs from armv7, is likely OK to mark ZFS as > broken there given how little power / memory the PiB and Pi0 have as well > as a lack of a good way to connect mass storage (just USB which exacerbates > the power situation on systems where power is already fragile). > > PowerPC 32-bit really isn't setup for ZFS either. The systems tend to have > less memory and don't tend to have a lot of disk. While it's nice to run > ZFS in a single disk setup, UFS is adequate for those situations. Don't > build file servers with FreeBSD 13 on this platform. We should mark it as > broken here. We already do this with several drivers. > > mips can do 64-bit atomics on 32-bit platforms, but it's ugly via > disabling interrupts (at least I think I committed those).... But even > fewer people have 32-bit mips boxes that want ZFS too... We don't build ZFS > on mips today, IIRC, so this is kinda moot. > > If ZFS needs this (now or in the future), and there's some atomics that > are possible, but not yet implemented, on these 32-bit platforms, we > should mark ZFS broken on those platforms and let their supporters fill in > the gaps (though it would be nice to ask for help before doing this if its > i386 or armv[67]). > > Warner > > >> > >> > Note that I saw the above access pattern only in the code that is not >> > imported yet. I only suspect that that pattern might already be present >> > in the current ZFS/FreeBSD code given that it uses 64-bit variables and >> > atomics a lot. >> > I am not sure if there is a quick way to check that. Maybe >> > devel/coccinelle could be used for that. >> > >> > -- >> > Andriy Gapon >> > _______________________________________________ >> > freebsd-hackers@freebsd.org mailing list >> > https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> > To unsubscribe, send any mail to " >> freebsd-hackers-unsubscribe@freebsd.org" >> _______________________________________________ >> freebsd-hackers@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-hackers >> To unsubscribe, send any mail to "freebsd-hackers-unsubscribe@freebsd.org >> " >> >