From owner-freebsd-arch@freebsd.org Tue Oct 8 15:22:12 2019 Return-Path: Delivered-To: freebsd-arch@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C1F0612CFDD for ; Tue, 8 Oct 2019 15:22:12 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f52.google.com (mail-lf1-f52.google.com [209.85.167.52]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46nh0g74TMz463x for ; Tue, 8 Oct 2019 15:22:11 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f52.google.com with SMTP id y127so12312626lfc.0 for ; Tue, 08 Oct 2019 08:22:11 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:subject:openpgp:autocrypt:message-id :date:user-agent:mime-version:content-language :content-transfer-encoding; bh=lwWomSUFaf+5RqXqs1b3HEV9eMcuwJnGMACcZFRQr3M=; b=IOkBPPUTV208N/6d+W+R3kZm+MEvtA5B7wdEekv7HzuqsKACzOUefmDZYaZatt1EUA HXWOGG/dEl3HkLpWQcMtL9XmPIH8JlaVISlOkfNsk/UcxP+woIFXgwycrnsREVctDb19 lGLCeVwdv01shlNKj0I7V98CvK4dIpE58qGxpOfBjsJNhbi7lKQX/IT9U9bljJ4+c6as tkUGjtwmvOaop2ItUbQ2cF8pRju8iznVfsxS1r+3K3zNAeOMY113XbOhAtz/akXMojk2 3BGwMXUU/tZVKmnQHoQuesk4B6bJSdeVpzWzShznK/I5phpbTg/afH21VHbPzj+ZNFON JLQg== X-Gm-Message-State: APjAAAWeMB+ysyN64xBRHr4/KiOvE6C6xlcdvNMXTnN8PPeRC+b4y3lu FziKCGfXI0MPRFKH0FeeAdRZ5qf3/0w= X-Google-Smtp-Source: APXvYqzVfg1gvvFTgbfi9zPtJ+XI2i5yEQsdzA9rYftWVr/gNXRf044H+NiV4R2s+N/KrwdBOlRSKw== X-Received: by 2002:a19:2313:: with SMTP id j19mr20623129lfj.138.1570548129714; Tue, 08 Oct 2019 08:22:09 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id 207sm4742796lfn.0.2019.10.08.08.22.08 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 08 Oct 2019 08:22:08 -0700 (PDT) To: freebsd-arch@FreeBSD.ORG From: Andriy Gapon Subject: illumos membar_producer emulation (zfs, dtrace) Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <28c5d9c1-740f-7832-873f-3cf03c6d94a4@FreeBSD.org> Date: Tue, 8 Oct 2019 18:22:07 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 46nh0g74TMz463x X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.52 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.20 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; TO_DN_NONE(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-arch@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.20)[ip: (-0.55), ipnet: 209.85.128.0/17(-3.26), asn: 15169(-2.14), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[52.167.85.209.list.dnswl.org : 127.0.5.0]; RWL_MAILSPIKE_POSSIBLE(0.00)[52.167.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 08 Oct 2019 15:22:12 -0000 I have got a passing interest in cleaning up of the atomic and related illumos compatibility (and contrib) code. At the moment I am looking at membar_producer. >From Oracle Solaris documentation it follows that membar_producer is a store to store barrier. https://docs.oracle.com/cd/E88353_01/html/E37855/membar-producer-9f.html I do not think that we have an MI abstraction for that, so it seems that atomic_thread_fence_rel could be the closest thing that we have. It also affects loads which is redundant but not a big deal (IMO). As of now, we have membar_producer implemented in assembly for several platforms (aarch64, amd64, i386, powerpc64, sparc64) and it's a no-op for the rest. I think that the latter is a problem as it can affect correctness of the code. Also, the assembly implementations seem to be closer to wmb() although not always the same. E.g., on amd64 both are sfence, while on sparc64 membar_producer is "membar #StoreStore" while wmb is "membar #MemIssue". I think that those implementations, maybe except for sparc64, are a bit of an overkill as the code that uses membar_producer only deals with "normal" memory. Maybe illumos uses membar_producer for access ordering of both normal memory and memory (or instructions) with special caching properties. The relevant code is in: - sys/cddl/compat/opensolaris/sys/atomic.h - sys/cddl/compat/opensolaris/kern/opensolaris_atomic.c - sys/cddl/contrib/opensolaris/common/atomic/*/opensolaris_atomic.S So, what do you think, would it be safe to emulate membar_producer with atomic_thread_fence_rel? Or is it better to leave the current code (copied from illumos for amd64, i386, sparc64 and home-grown for aarch64, powerpc64) alone? In that case, we still need to add something for the rest of the platforms (arm-s, mips-es, riscv). P.S. A bit offtopic. I noticed a couple of uses of wmb() with stores to regular memory. I think that they are either not needed or better be "release barriers". vm_set_rendezvous_func() in sys/amd64/vmm/vmm.c rendezvous_func is checked and modified under rendezvous_mtx anyway. cpu_reset() in sys/x86/x86/cpu_machdep.c I do not see a need for a store-store barrier here. pmc_initialize() in sys/dev/hwpmc/hwpmc_mod.c I think that a store_rel or fence_rel would be more appropriate. Note sure about all uses of wmb in xen code, seems like at least some of them should be release operations or fences as well. -- Andriy Gapon