From owner-freebsd-mips@FreeBSD.ORG Mon Jun 3 11:06:48 2013 Return-Path: Delivered-To: freebsd-mips@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 62AE95D8 for ; Mon, 3 Jun 2013 11:06:48 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id 53E9F13B8 for ; Mon, 3 Jun 2013 11:06:48 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.7/8.14.7) with ESMTP id r53B6mxA015082 for ; Mon, 3 Jun 2013 11:06:48 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.7/8.14.7/Submit) id r53B6lYN015080 for freebsd-mips@FreeBSD.org; Mon, 3 Jun 2013 11:06:47 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 3 Jun 2013 11:06:47 GMT Message-Id: <201306031106.r53B6lYN015080@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-mips@FreeBSD.org Subject: Current problem reports assigned to freebsd-mips@FreeBSD.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 11:06:48 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/178319 mips [patch] [arge] arge_stop() doesn't clean the tx ring p o kern/178318 mips [patch] [arge] if_arge/bootp race under some circunsta o kern/177876 mips [mips] kernel stack overflow panic on mips64, EdgeRout o kern/177832 mips [mips] [gpio] [patch] GPIO and RF LED do not function o kern/177032 mips [arge] arge1 fails to attach on UBNT Routerstation o kern/165951 mips [ar913x] [ath] DDR flush isn't being done for the WMAC p kern/163670 mips [mips][arge] arge can't allocate ring buffer on multip 7 problems total. From owner-freebsd-mips@FreeBSD.ORG Mon Jun 3 14:04:24 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 10259199; Mon, 3 Jun 2013 14:04:24 +0000 (UTC) (envelope-from edschouten@gmail.com) Received: from mail-ve0-x230.google.com (mail-ve0-x230.google.com [IPv6:2607:f8b0:400c:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id BA5861ED6; Mon, 3 Jun 2013 14:04:23 +0000 (UTC) Received: by mail-ve0-f176.google.com with SMTP id c13so2783512vea.7 for ; Mon, 03 Jun 2013 07:04:23 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:date:x-google-sender-auth:message-id:subject :from:to:cc:content-type; bh=5mOuUcOvJhlE7fzQiapnilkp6TFn7nTM41OAxUuOws8=; b=S4u1MiTYxZLIYpMI9pEq+Jgnv0tsImRWnKaIG7avhfuJiYtqN434gO4X4HJ737gdlE nYSAMAXZa28xvfZhYPwssZoOxY8xnQaurLMJ5eEteOXGQKZdj0pOs00Vu6b4n5LFzRYw fNY7ww7/hXYiIXD8OA8T62vMQ9acNHZ+EkxbPg+b4Nz8pjVX16FARtb/glX4rPMClYu4 m7bZFQX8UhcshntZ+L9N/9cuAA4bbTTSOztLacr+COUDC3oJ8LMm5FAEUZ/PU2DKiH+F 0nTmgR2pgUu8ZjeoyHZCWEzBz/ukncgtLMIkbbrsb8zDIThl0k3UKJ6BvV/OzR9MZxPL ddeA== MIME-Version: 1.0 X-Received: by 10.52.183.170 with SMTP id en10mr14491274vdc.5.1370268263228; Mon, 03 Jun 2013 07:04:23 -0700 (PDT) Sender: edschouten@gmail.com Received: by 10.220.107.139 with HTTP; Mon, 3 Jun 2013 07:04:23 -0700 (PDT) Date: Mon, 3 Jun 2013 16:04:23 +0200 X-Google-Sender-Auth: Inb7_DbHLzfo8UoJdPCaDKalzAk Message-ID: Subject: Kernelspace C11 atomics for MIPS From: Ed Schouten To: freebsd-mips@freebsd.org Content-Type: text/plain; charset=UTF-8 Cc: freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 14:04:24 -0000 Hi, As of r251230, it should be possible to use C11 atomics in kernelspace, by including ! Even when not using Clang (but GCC 4.2), it is possible to use quite a large portion of the API. A couple of limitations: - The memory order argument is simply ignored, making all the calls do a full memory barrier. - At least Clang allows you to do arithmetic on C11 atomics directly (e.g. "a += 5" == "atomic_fetch_add(&a, 5)"), which is of course not possible to mimick. - The atomic functions only work on 1,2,4,8-byte types, which is probably a good thing. Amazingly, it turns out that it most of the architectures, with the exception of ARM and MIPS. To make MIPS work, we need to implement some of the __sync_* functions that are described here: http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html Some time ago I already added some of these functions to our libcompiler-rt in userspace, to make atomics work there. Unfortunately, these functions were quite horribly implemented, as I tried to build them on top of , which is far from trivial/efficient. It is also restricted to 4 and 8-byte types. That's why I thought: why not spend some time learning MIPS assembly and write some decent implementations for these functions? The result: http://80386.nl/pub/mips-stdatomic.txt For now, please focus on sys/mips/mips/stdatomic.c. It implements all the __sync_* functions called by for 1, 2, 4 and 8 byte types. There is some testing code in there as well, which can be ignored. This code disassembles to the following: http://80386.nl/pub/mips-stdatomic-disasm.txt As I don't own a MIPS system myself, I was thinking about tinkering a bit with qemu to see whether these functions work properly. My questions are: - Does anyone have any comments on the C code and/or the machine code generated? Are there some nifty tricks I can apply to make the machine code more efficient that I am unaware o? - Is there anyone interested in testing this code a bit more thoroughly on physical hardware? - Would anyone mind if I committed this to HEAD? Thanks, -- Ed Schouten From owner-freebsd-mips@FreeBSD.ORG Mon Jun 3 17:53:14 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 413E56D3 for ; Mon, 3 Jun 2013 17:53:14 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ve0-x22f.google.com (mail-ve0-x22f.google.com [IPv6:2607:f8b0:400c:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id F33231CFB for ; Mon, 3 Jun 2013 17:53:13 +0000 (UTC) Received: by mail-ve0-f175.google.com with SMTP id da11so3079467veb.34 for ; Mon, 03 Jun 2013 10:53:13 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=zZ9JiRMvGJMFntbqO8iJk2bKrHzgv6TJ/3Bb1fz2rI4=; b=M/C59X8um5PrD8xuYSoM5rKrgvtyEQRo6Y/oRhsiE0+tORGe/BOsSrhWpDuakB+rp1 Fh7b+F7E+AY361H7T7sbpf65WlsPoqvrVOWqE3ARycYICBpnzuXYKt/1bibOOERTB41O ViazIxGPqkRlqtM0AZgKwgY/sr7nTB005ajiIiiN1Swids+NsdLwEHverMsNnVSBaH3v UwPAkI3U3OTpvXOP4MSfRoErzYcc5rVn57gT/3Vy/9TKfBgU21BW9s3ktbMLsJawrNkC 3zfDvAzBEVNlz1ap/V87pE/QF15zSkQHe9ADPpCXuY8lvlYXK5FN12eKdSLUi4/6KiAZ Z4eA== X-Received: by 10.52.30.14 with SMTP id o14mr13771998vdh.106.1370281993273; Mon, 03 Jun 2013 10:53:13 -0700 (PDT) Received: from monkey-bot.int.fusionio.com ([209.117.142.2]) by mx.google.com with ESMTPSA id s6sm37041931vdj.5.2013.06.03.10.53.11 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Jun 2013 10:53:12 -0700 (PDT) Sender: Warner Losh Subject: Re: Kernelspace C11 atomics for MIPS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Mon, 3 Jun 2013 11:53:09 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Ed Schouten X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQnYyk6dBKqoS8a5Yov8o8MUgpyEFEKE1YSR/npPNc2UXTWm6m4RpYnH9GzAZJAliXzgOSx3 Cc: freebsd-mips@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 17:53:14 -0000 On Jun 3, 2013, at 8:04 AM, Ed Schouten wrote: > Hi, >=20 > As of r251230, it should be possible to use C11 atomics in > kernelspace, by including ! Even when not using Clang > (but GCC 4.2), it is possible to use quite a large portion of the API. > A couple of limitations: >=20 > - The memory order argument is simply ignored, making all the calls do > a full memory barrier. > - At least Clang allows you to do arithmetic on C11 atomics directly > (e.g. "a +=3D 5" =3D=3D "atomic_fetch_add(&a, 5)"), which is of course = not > possible to mimick. > - The atomic functions only work on 1,2,4,8-byte types, which is > probably a good thing. >=20 > Amazingly, it turns out that it most of the architectures, with the > exception of ARM and MIPS. To make MIPS work, we need to implement > some of the __sync_* functions that are described here: >=20 > http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html >=20 > Some time ago I already added some of these functions to our > libcompiler-rt in userspace, to make atomics work there. > Unfortunately, these functions were quite horribly implemented, as I > tried to build them on top of , which is far from > trivial/efficient. It is also restricted to 4 and 8-byte types. That's > why I thought: why not spend some time learning MIPS assembly and > write some decent implementations for these functions? >=20 > The result: >=20 > http://80386.nl/pub/mips-stdatomic.txt The number of necessary syncs varies by processor type. There's also = newer synchronization instructions that make this as efficient as = possible for all mips32r2 and mips64r2-based machines. Older Caviums, at = least and maybe newer ones, also have their own variants. What you have = will mostly work for the processors we have to support. mips_sync could = therefore be better. Doing it before AND after seems like overkill as = well. Since sync is a fairly performance killing assembler instruction, = how would you feel about allowing optimizations? This is my biggest single concern about the patch, but it also my = current biggest concern about the MIPS atomic operators in general. > For now, please focus on sys/mips/mips/stdatomic.c. It implements all > the __sync_* functions called by for 1, 2, 4 and 8 byte > types. There is some testing code in there as well, which can be > ignored. This code disassembles to the following: >=20 > http://80386.nl/pub/mips-stdatomic-disasm.txt >=20 > As I don't own a MIPS system myself, I was thinking about tinkering a > bit with qemu to see whether these functions work properly. My > questions are: >=20 > - Does anyone have any comments on the C code and/or the machine code > generated? Are there some nifty tricks I can apply to make the machine > code more efficient that I am unaware o? > - Is there anyone interested in testing this code a bit more > thoroughly on physical hardware? > - Would anyone mind if I committed this to HEAD? I have some cavium gear I can easily test on, and some other stuff I can = less-easily test on. It wouldn't be horrible to commit to head, but it would affect = performance in many places. Don't commit the kern/bla.c standard change to conf/files, it looks to = be bogus :) Warner From owner-freebsd-mips@FreeBSD.ORG Mon Jun 3 20:16:00 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2E47FFA; Mon, 3 Jun 2013 20:16:00 +0000 (UTC) (envelope-from edschouten@gmail.com) Received: from mail-ve0-x230.google.com (mail-ve0-x230.google.com [IPv6:2607:f8b0:400c:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id CFDC91368; Mon, 3 Jun 2013 20:15:59 +0000 (UTC) Received: by mail-ve0-f176.google.com with SMTP id c13so3133867vea.7 for ; Mon, 03 Jun 2013 13:15:59 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=epO2QtREq85Qirwj2jzCOL7EyD4dKSnfZsna/f1A2Y4=; b=YeX3NKgJ1qIhGHuQRp0KfESFzmdD2puMnlPCvs9E5M9yawAO+yH1kYeJ/ulEayyEdS m4LsNbphuwT+oJGwdyCOc7EQ01W+RJReGApnLRQ1p/nGnLGgPFu6MEoCV9wYLEH0Y5Rs NISPaMrZCYMtRlrBdnKaKO6xVAw9DDMeERxkDYYhihRJi6AaRghF/6lyzGKh7rc4FQZS zJYjQCGWBmuBBH7nGFMTi7YPXUK5skC+8rDwEshjwKvQX9kB42f99XI70WJJxVcpApD+ Me2ozlt6VGuSbw+U69dxdhPxM3jTT9fhVrMswYPRRYI6dAw0mhzWLw2oXniy4ubno8BG XwwA== MIME-Version: 1.0 X-Received: by 10.52.183.170 with SMTP id en10mr14965864vdc.5.1370290559285; Mon, 03 Jun 2013 13:15:59 -0700 (PDT) Sender: edschouten@gmail.com Received: by 10.220.107.139 with HTTP; Mon, 3 Jun 2013 13:15:59 -0700 (PDT) In-Reply-To: References: Date: Mon, 3 Jun 2013 22:15:59 +0200 X-Google-Sender-Auth: FeVDz4OIqufmn0UG2b3qrapNfk0 Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Ed Schouten To: Warner Losh Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Cc: freebsd-mips@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Jun 2013 20:16:00 -0000 Hey Warner, After playing around a bit I managed to get qemu user mode emulation for mips64 working on my workstation. Apart from a stupid thinko (__sync_fetch_and_sub() did B - A instead of A - B), the code managed to survive the following test, which calls the stdatomic library functions with random parameters: http://80386.nl/pub/stdatomic-fuzzer.txt I'll see if I can push this tool into the tree after polishing it up a bit. Unfortunately it only tests the logical aspects of the routines -- not whether the routines are actually atomic. 2013/6/3 Warner Losh : > The number of necessary syncs varies by processor type. There's also newe= r synchronization instructions that make this as efficient as possible for = all mips32r2 and mips64r2-based machines. Older Caviums, at least and maybe= newer ones, also have their own variants. What you have will mostly work f= or the processors we have to support. mips_sync could therefore be better. = Doing it before AND after seems like overkill as well. Since sync is a fair= ly performance killing assembler instruction, how would you feel about allo= wing optimizations? > > This is my biggest single concern about the patch, but it also my current= biggest concern about the MIPS atomic operators in general. I have to confess, that's exactly the part I know very little about. The code I wrote is largely based on the code in . The mips_sync() function has been copied over almost literally. I think tuning this could be done separately. Regarding calling mips_sync() before and after. I think we always have to call it before we perform the action (for example if the atomic call is used to implement an unlock). But indeed, afterwards makes little sense. We would only need to perform a barrier at the compiler level -- not the memory level. It wouldn't make sense to add this explicitly, because these are separate functions in a separate compilation unit anyway. Thoughts? > I have some cavium gear I can easily test on, and some other stuff I can = less-easily test on. Awesome! At least testing it on Cavium would be nice. I've updated the diff. Please refresh. http://80386.nl/pub/mips-stdatomic.txt One easy way to test this, would be to link the fuzzer source file against the stdatomic.c file added by my patch and run that. The source file should compile both in user+kernel now. Thanks, -- Ed Schouten From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 02:45:50 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 5F1F2E25; Tue, 4 Jun 2013 02:45:50 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x231.google.com (mail-qc0-x231.google.com [IPv6:2607:f8b0:400d:c01::231]) by mx1.freebsd.org (Postfix) with ESMTP id 0EB2A13BF; Tue, 4 Jun 2013 02:45:49 +0000 (UTC) Received: by mail-qc0-f177.google.com with SMTP id e1so2570765qcy.8 for ; Mon, 03 Jun 2013 19:45:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=bnGVtp2/EgnS27TtyDbKsZ87JYvZoL8p61vYDb7erdA=; b=Hf5pBfMTlR4UlJsOlMuWk3cuYUfFaGBoMd9thKut04fmfqvUvJoTQBlPorQcuxzm5u yBBm7uol/quWIH50jxHXcAOk5KSRCE2tPbXiWKMLt6x/vHLoo2OTcRU3WJSItbM/7edG D0K9RH/ITfZG1AhrOcenss6vnLwshrW84Ki6WNKvH3lWtdjeYjMZC46VGIdSDH9vOqrN 7fbGIF0Ozd78M20cMEO2RaOCr9B4zLkxs6l6zFJc2DxMLY9wzW/W7mBpCy79ADGGg9Hi giq/9S+RL4DmnnymScFWOZj+eGPGz3jlYkRTfQZrxqikmm+s3ovCchMAN03dZ/c+aExV vy4Q== MIME-Version: 1.0 X-Received: by 10.49.38.169 with SMTP id h9mr24294063qek.54.1370313948960; Mon, 03 Jun 2013 19:45:48 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.71.12 with HTTP; Mon, 3 Jun 2013 19:45:48 -0700 (PDT) In-Reply-To: References: Date: Mon, 3 Jun 2013 19:45:48 -0700 X-Google-Sender-Auth: kt6s0ZfAuLunSnQESDzNm8hulqU Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: Ed Schouten , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 02:45:50 -0000 Speaking of this; any idea why the SYNC operators have 8 NOPs following the= m? I noticed that when going through disassemblies of various mips24k .o files= . Adrian On 3 June 2013 10:53, Warner Losh wrote: > > On Jun 3, 2013, at 8:04 AM, Ed Schouten wrote: > >> Hi, >> >> As of r251230, it should be possible to use C11 atomics in >> kernelspace, by including ! Even when not using Clang >> (but GCC 4.2), it is possible to use quite a large portion of the API. >> A couple of limitations: >> >> - The memory order argument is simply ignored, making all the calls do >> a full memory barrier. >> - At least Clang allows you to do arithmetic on C11 atomics directly >> (e.g. "a +=3D 5" =3D=3D "atomic_fetch_add(&a, 5)"), which is of course n= ot >> possible to mimick. >> - The atomic functions only work on 1,2,4,8-byte types, which is >> probably a good thing. >> >> Amazingly, it turns out that it most of the architectures, with the >> exception of ARM and MIPS. To make MIPS work, we need to implement >> some of the __sync_* functions that are described here: >> >> http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html >> >> Some time ago I already added some of these functions to our >> libcompiler-rt in userspace, to make atomics work there. >> Unfortunately, these functions were quite horribly implemented, as I >> tried to build them on top of , which is far from >> trivial/efficient. It is also restricted to 4 and 8-byte types. That's >> why I thought: why not spend some time learning MIPS assembly and >> write some decent implementations for these functions? >> >> The result: >> >> http://80386.nl/pub/mips-stdatomic.txt > > The number of necessary syncs varies by processor type. There's also newe= r synchronization instructions that make this as efficient as possible for = all mips32r2 and mips64r2-based machines. Older Caviums, at least and maybe= newer ones, also have their own variants. What you have will mostly work f= or the processors we have to support. mips_sync could therefore be better. = Doing it before AND after seems like overkill as well. Since sync is a fair= ly performance killing assembler instruction, how would you feel about allo= wing optimizations? > > This is my biggest single concern about the patch, but it also my current= biggest concern about the MIPS atomic operators in general. > >> For now, please focus on sys/mips/mips/stdatomic.c. It implements all >> the __sync_* functions called by for 1, 2, 4 and 8 byte >> types. There is some testing code in there as well, which can be >> ignored. This code disassembles to the following: >> >> http://80386.nl/pub/mips-stdatomic-disasm.txt >> >> As I don't own a MIPS system myself, I was thinking about tinkering a >> bit with qemu to see whether these functions work properly. My >> questions are: >> >> - Does anyone have any comments on the C code and/or the machine code >> generated? Are there some nifty tricks I can apply to make the machine >> code more efficient that I am unaware o? >> - Is there anyone interested in testing this code a bit more >> thoroughly on physical hardware? >> - Would anyone mind if I committed this to HEAD? > > I have some cavium gear I can easily test on, and some other stuff I can = less-easily test on. > > It wouldn't be horrible to commit to head, but it would affect performanc= e in many places. > > Don't commit the kern/bla.c standard change to conf/files, it looks to be= bogus :) > > Warner > > _______________________________________________ > freebsd-mips@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mips > To unsubscribe, send any mail to "freebsd-mips-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 03:55:26 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A6DA8152 for ; Tue, 4 Jun 2013 03:55:26 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-la0-x22e.google.com (mail-la0-x22e.google.com [IPv6:2a00:1450:4010:c03::22e]) by mx1.freebsd.org (Postfix) with ESMTP id 2A7E41901 for ; Tue, 4 Jun 2013 03:55:25 +0000 (UTC) Received: by mail-la0-f46.google.com with SMTP id eg20so833150lab.33 for ; Mon, 03 Jun 2013 20:55:24 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :x-gm-message-state; bh=wOTHXKDbCzxubNM6SPoOBUamwHGn0dQNYctqIK7v5SQ=; b=RayRgNO3bA3YdxqLcEKTWcROgdfbcyEKn4qpRaf7QHAfPkWklxn2dYS6A46J71PPXF XifBkMXVErv7ymkSbbPYLR0WawTSOTdP6kZs45FxSkXqcn6mjbOsNNKwk8i7Fdg1B8Tx a/eZ4wIe8Y+e8rHL3GDvf9Tl1ZE5lnM9+TiH+nacSaHUZqr1c2orNIkfaBV2Oy/hx9vm mM1Zxo/hot3LkPvyAf3fNSxU0EbujAdbaRfQ9pRHkk4C6FGBnH66SJGLEfllkq2a9v93 C2B2ymzsutq3yX2ZGaEEnmkGQqPi7by4ZKv/bnLb+cOYgdpS5iTqpxP0YVRi9YgDbI/X Vhug== X-Received: by 10.112.89.8 with SMTP id bk8mr12168340lbb.73.1370318124329; Mon, 03 Jun 2013 20:55:24 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.152.129.195 with HTTP; Mon, 3 Jun 2013 20:55:04 -0700 (PDT) In-Reply-To: References: From: Juli Mallett Date: Mon, 3 Jun 2013 20:55:04 -0700 X-Google-Sender-Auth: 6Wr2OOY81fqP7rhfs45eOMWDZbw Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS To: Adrian Chadd X-Gm-Message-State: ALoCoQnwwtBnMWmZzwxmJSub7jVkedPyhq2FdqHuFyhzS4IIb851rLjROJXQuulHqRdR4GMlXca9 Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 03:55:26 -0000 On Mon, Jun 3, 2013 at 7:45 PM, Adrian Chadd wrote: > Speaking of this; any idea why the SYNC operators have 8 NOPs following > them? > > I noticed that when going through disassemblies of various mips24k .o > files. > To drain the pipeline on certain deficient (and mostly older) CPUs by way of guesswork and a little vague magic. Most CPUs we support, I would guess, do not need this, and it continues to exist solely for hysterical reasons. I've certainly gotten rid of them and some other cargo cult synchronization on Octeon for testing and had it survive under considerable load, and occasionally with some slight speedups (for some more commonly-used or slower things than Just a Bunch Of NOPs.) The trouble is that proving they aren't necessary requires being rigorous and careful in understanding documentation and errata, and FUD about their possible necessity is somewhat-intimidating. It's not an easy kind of corruption/unreliability/etc., to prove the lack of empirically. Juli. From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 03:57:52 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 77B1F235; Tue, 4 Jun 2013 03:57:52 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qe0-f48.google.com (mail-qe0-f48.google.com [209.85.128.48]) by mx1.freebsd.org (Postfix) with ESMTP id B56F01918; Tue, 4 Jun 2013 03:57:51 +0000 (UTC) Received: by mail-qe0-f48.google.com with SMTP id 2so1352401qea.7 for ; Mon, 03 Jun 2013 20:57:45 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=wzZzWKsOzqixlVirXSFfSMCMvKvuBGyXjgtmRkySlFw=; b=hFV5iuxDYtq+KoiaW1e0xIeGfbw9XHuf3LEaSnSIjsPZsTWiuidnYpA3dYo37sjW7/ gD1ZMDJsJ1qPdJ7GhsTSkGu92GPVSirmKYTCsrn7ZPJ9n/9Do5S7zTzy8Smc6sCDW967 kYvpZUTnNpdzHqUSLYTlpNHDGNTqIIol7WjBuzQCrxqxOeQ2n4te3okfJl8Rom0GPU1S Fg4Fe7NVkyW28dhLZ2AY91suKYzGEuguC4g5YeqsIFuCJVlTuo2Ys6rXyoS4MdwsOugv 5pcnRQwQxhZkKM7f2VvzLul4lOM/vXi+ZJ+SY+ZJyoDFKkNxBMaTStYfJpKyHibzeg6Q leiA== MIME-Version: 1.0 X-Received: by 10.229.149.14 with SMTP id r14mr5802450qcv.59.1370318265612; Mon, 03 Jun 2013 20:57:45 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.71.12 with HTTP; Mon, 3 Jun 2013 20:57:45 -0700 (PDT) In-Reply-To: References: Date: Mon, 3 Jun 2013 20:57:45 -0700 X-Google-Sender-Auth: 4aJjyK5RjzXSjOg7ZrQc72CBkFU Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Adrian Chadd To: Juli Mallett Content-Type: text/plain; charset=ISO-8859-1 Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 03:57:52 -0000 On 3 June 2013 20:55, Juli Mallett wrote: > To drain the pipeline on certain deficient (and mostly older) CPUs by way of > guesswork and a little vague magic. Most CPUs we support, I would guess, do > not need this, and it continues to exist solely for hysterical reasons. How can I turn it off for my compiles? > I've certainly gotten rid of them and some other cargo cult synchronization > on Octeon for testing and had it survive under considerable load, and > occasionally with some slight speedups (for some more commonly-used or > slower things than Just a Bunch Of NOPs.) Right. Well, since it's happening on every inlined lock, it's a bit silly. > The trouble is that proving they aren't necessary requires being rigorous > and careful in understanding documentation and errata, and FUD about their > possible necessity is somewhat-intimidating. It's not an easy kind of > corruption/unreliability/etc., to prove the lack of empirically. I've checked the diassembly from gcc-4.mumble on linux; it doesn't include NOPs like this as far as I can tell. Adrian From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 04:05:18 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E1533363 for ; Tue, 4 Jun 2013 04:05:18 +0000 (UTC) (envelope-from juli@clockworksquid.com) Received: from mail-la0-x22c.google.com (mail-la0-x22c.google.com [IPv6:2a00:1450:4010:c03::22c]) by mx1.freebsd.org (Postfix) with ESMTP id 66B97194D for ; Tue, 4 Jun 2013 04:05:18 +0000 (UTC) Received: by mail-la0-f44.google.com with SMTP id er20so2100740lab.31 for ; Mon, 03 Jun 2013 21:05:17 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date :x-google-sender-auth:message-id:subject:to:cc:content-type :x-gm-message-state; bh=WVG2tjYb43vhRNx/JWGJd6FCPwewo2DPNIzuj6sE36g=; b=kBnGhyW837Sev/NNTyimX7n38pfUGhrgVSgcm2K7X/8hDN648tssAYH0iQTXiHmsgC 67jQ6LAyP6b6ucLT+8B90ift3+abSCMk74X7o0zJL4F0WB9InF59xLrfdcLaEYJa4jqS YxOzGd9JhrTwMhWIayWFVgIuyq+SpwuZ/d9mdCbR6+7Jpe50+nr/T7+nbFO/Ol3aVHGp N0ztcgqmI0NL8GJjFBIVmfb+gQFI2lfeySRsHxqVljabfpbUHeK/qRrzcmJy+VEIEuLa OS+B7bU5jBEwoo9jiP5nOhIiSemPm1NVvhXpNvBRtFXX6a3ieRswxbE4yUpHt8IbFmyC d3rw== X-Received: by 10.112.180.232 with SMTP id dr8mr12138264lbc.67.1370318717158; Mon, 03 Jun 2013 21:05:17 -0700 (PDT) MIME-Version: 1.0 Sender: juli@clockworksquid.com Received: by 10.152.129.195 with HTTP; Mon, 3 Jun 2013 21:04:57 -0700 (PDT) In-Reply-To: References: From: Juli Mallett Date: Mon, 3 Jun 2013 21:04:57 -0700 X-Google-Sender-Auth: XobjzOOSHy3MIjEHOo-pP8uINW0 Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS To: Adrian Chadd X-Gm-Message-State: ALoCoQl2YZj1xbdZs9i1t69Lf4YweAurkg1eQGxddNgc/nO/Kl4O7PvC3EYFFU7P9wCwJHUwO74m Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 04:05:18 -0000 On Mon, Jun 3, 2013 at 8:57 PM, Adrian Chadd wrote: > On 3 June 2013 20:55, Juli Mallett wrote: > > > To drain the pipeline on certain deficient (and mostly older) CPUs by > way of > > guesswork and a little vague magic. Most CPUs we support, I would > guess, do > > not need this, and it continues to exist solely for hysterical reasons. > > How can I turn it off for my compiles? Edit the source. This is not and this kind of thing must not be a user-visible go-faster knob. I'm anticipating that someone might want to respond to "edit the source" by saying that users don't have to edit the source, without understanding the kind of change this is. > > I've certainly gotten rid of them and some other cargo cult > synchronization > > on Octeon for testing and had it survive under considerable load, and > > occasionally with some slight speedups (for some more commonly-used or > > slower things than Just a Bunch Of NOPs.) > > Right. Well, since it's happening on every inlined lock, it's a bit silly. Yes. > > The trouble is that proving they aren't necessary requires being > rigorous > > and careful in understanding documentation and errata, and FUD about > their > > possible necessity is somewhat-intimidating. It's not an easy kind of > > corruption/unreliability/etc., to prove the lack of empirically. > > I've checked the diassembly from gcc-4.mumble on linux; it doesn't > include NOPs like this as far as I can tell. > Neat. You might also like to look at usage of 'sync' (and its variants, or the lack of use of its variants) and the possibility of using newer mips32/64 instructions to change whether interrupts are enabled, and a number of other things, at least for certain CPU types. And excessive use of all kinds of memory barriers (including simple memory-clobber barriers) and and and. There's a lot of small changes that can be made that add up, but building confidence across the range of hardware we support is genuinely-hard. Juli. From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 04:08:49 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 4610B442; Tue, 4 Jun 2013 04:08:49 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: from mail-bk0-x22f.google.com (mail-bk0-x22f.google.com [IPv6:2a00:1450:4008:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 4EEDF1962; Tue, 4 Jun 2013 04:08:48 +0000 (UTC) Received: by mail-bk0-f47.google.com with SMTP id jg9so1957392bkc.6 for ; Mon, 03 Jun 2013 21:08:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=to6zmT8ADvmoITeNfrPJrkIAgVgpUlojSrM7V1oplNY=; b=vfVG2Ao4eQcZms5aWRvBqZYTDeKRXgkS2OA6qJpHg2VHczR9YkOu+OYdx05gb/Q8e5 dk6ePR21/IRMzOk2avpWoj/g+4JzHRcxmJ5RnY73xq684q5nCF9yqDsr/lmd51TqA4YV 3l6Zxifi3a8GD7a+Qsm9RXVmz4fSzqqdKp0sE/V/ihxcr17pk3YP+NFqj60XC4RyDXLH KV+nWt7OJFanvMv2oZTGywI1R0Agvc7m3TPcHdxCYzmRHYMv1tqCrFTC/JSFhWjOdZg/ HqRZMoGr315Qm91sbBgjXsKUrdGc/RI+nZOnBpkroDrKi9MjIm3GqEK4iZ8dFluVIGqy 13Cg== MIME-Version: 1.0 X-Received: by 10.204.63.1 with SMTP id z1mr7346196bkh.148.1370318927404; Mon, 03 Jun 2013 21:08:47 -0700 (PDT) Sender: pkelsey@gmail.com Received: by 10.205.141.68 with HTTP; Mon, 3 Jun 2013 21:08:47 -0700 (PDT) In-Reply-To: References: Date: Tue, 4 Jun 2013 00:08:47 -0400 X-Google-Sender-Auth: m26drlMzlt_pTRy98rdfaB80jr0 Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Patrick Kelsey To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 04:08:49 -0000 On Mon, Jun 3, 2013 at 11:57 PM, Adrian Chadd wrote: > On 3 June 2013 20:55, Juli Mallett wrote: > >> To drain the pipeline on certain deficient (and mostly older) CPUs by way of >> guesswork and a little vague magic. Most CPUs we support, I would guess, do >> not need this, and it continues to exist solely for hysterical reasons. > > How can I turn it off for my compiles? > >> I've certainly gotten rid of them and some other cargo cult synchronization >> on Octeon for testing and had it survive under considerable load, and >> occasionally with some slight speedups (for some more commonly-used or >> slower things than Just a Bunch Of NOPs.) > > Right. Well, since it's happening on every inlined lock, it's a bit silly. > >> The trouble is that proving they aren't necessary requires being rigorous >> and careful in understanding documentation and errata, and FUD about their >> possible necessity is somewhat-intimidating. It's not an easy kind of >> corruption/unreliability/etc., to prove the lack of empirically. > > I've checked the diassembly from gcc-4.mumble on linux; it doesn't > include NOPs like this as far as I can tell. > The sync + 8 nops is coming from the definition of mips_sync() in sys/mips/include/atomic.h. I agree with Juli that it appears to be a manual pipeline-flush holdover from earlier days - I'm guessing there's 8 nops because the R4000/4400 had both the sync instruction and an 8-stage pipeline. I'm further guessing this was an attempt at providing stronger ordering semantics than the sync instruction itself for the following mb()/wmb()/rmb() definitions that use it, as the sync instruction definition doesn't restrict execution of the before/after loads/stores with respect to the sync instruction itself. From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 04:15:20 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 68DF862D; Tue, 4 Jun 2013 04:15:20 +0000 (UTC) (envelope-from pkelsey@gmail.com) Received: from mail-bk0-x233.google.com (mail-bk0-x233.google.com [IPv6:2a00:1450:4008:c01::233]) by mx1.freebsd.org (Postfix) with ESMTP id 710E4198F; Tue, 4 Jun 2013 04:15:19 +0000 (UTC) Received: by mail-bk0-f51.google.com with SMTP id ji2so1022881bkc.10 for ; Mon, 03 Jun 2013 21:15:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=H1WND2Yc9hbvyisbZ6h5Xe9U6SOzd4cTJmVloti6WtU=; b=GUnkZGY18aWJ3BmoAuE6ShVUJC8El4GStrAUmMGJvkOxJQkQshhyzJrpirC8MwBKdI OK1fZ6UOvQfqDM8ArR1Vc/GhV0yqVWW1G3JeCNXjx/f6JhBAembg4s2V//1KfrC4Swr5 7XNSiZo0692hDkt54w4bRJEoekSD6g80IsuQlaNKO8MmSCpT6BG6v1AvZcpaWaudOaRC meGmU3Abrd5XyWdFNL8HKGZXuNjAzbMTDfyMomQckxwxs+FYtq86qGIh8ZyMGSCUc8WN 5StdB4LWUmt4OpndBNNh7bqowg5fetBtztj+uOrecWEvQdrQtndNhFR4MN8zNC6s+v2x 6iHA== MIME-Version: 1.0 X-Received: by 10.205.107.202 with SMTP id dz10mr7284751bkc.180.1370319317940; Mon, 03 Jun 2013 21:15:17 -0700 (PDT) Sender: pkelsey@gmail.com Received: by 10.205.141.68 with HTTP; Mon, 3 Jun 2013 21:15:17 -0700 (PDT) In-Reply-To: References: Date: Tue, 4 Jun 2013 00:15:17 -0400 X-Google-Sender-Auth: GsvU6Ysz54D3_WzBTuuZdPLpqiM Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Patrick Kelsey To: Adrian Chadd Content-Type: text/plain; charset=ISO-8859-1 Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 04:15:20 -0000 On Tue, Jun 4, 2013 at 12:08 AM, Patrick Kelsey wrote: > On Mon, Jun 3, 2013 at 11:57 PM, Adrian Chadd wrote: >> On 3 June 2013 20:55, Juli Mallett wrote: >> >>> To drain the pipeline on certain deficient (and mostly older) CPUs by way of >>> guesswork and a little vague magic. Most CPUs we support, I would guess, do >>> not need this, and it continues to exist solely for hysterical reasons. >> >> How can I turn it off for my compiles? >> >>> I've certainly gotten rid of them and some other cargo cult synchronization >>> on Octeon for testing and had it survive under considerable load, and >>> occasionally with some slight speedups (for some more commonly-used or >>> slower things than Just a Bunch Of NOPs.) >> >> Right. Well, since it's happening on every inlined lock, it's a bit silly. >> >>> The trouble is that proving they aren't necessary requires being rigorous >>> and careful in understanding documentation and errata, and FUD about their >>> possible necessity is somewhat-intimidating. It's not an easy kind of >>> corruption/unreliability/etc., to prove the lack of empirically. >> >> I've checked the diassembly from gcc-4.mumble on linux; it doesn't >> include NOPs like this as far as I can tell. >> > > The sync + 8 nops is coming from the definition of mips_sync() in > sys/mips/include/atomic.h. > > I agree with Juli that it appears to be a manual pipeline-flush > holdover from earlier days - I'm guessing there's 8 nops because the > R4000/4400 had both the sync instruction and an 8-stage pipeline. I'm > further guessing this was an attempt at providing stronger ordering > semantics than the sync instruction itself for the following > mb()/wmb()/rmb() definitions that use it, as the sync instruction > definition doesn't restrict execution of the before/after loads/stores > with respect to the sync instruction itself. Forgot to emphasize that this particular bit of old-school nop-counting is either pointless or a latent hazard - 8 does not cover the deepest MIPS pipeline around, then there's superscalar issue to consider - so I think it's either unnecessary or insufficient. So far, that's all criticism and no solution :/ From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 04:32:04 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 2F6927FD for ; Tue, 4 Jun 2013 04:32:04 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) by mx1.freebsd.org (Postfix) with ESMTP id F28B41A05 for ; Tue, 4 Jun 2013 04:32:03 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id 10so11178131ied.0 for ; Mon, 03 Jun 2013 21:32:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=0sGsWd94t071+T76PYAZNXbdQAwu+/CjisI7Kru0qj4=; b=hoYAl4LIVsNR+nabzh4HxAZHV+DMa2speYh1xY23ISnufEJcwwyRS2sU6c8bTPfmvr PoH3NR4j6FBc/IH/gaTUOFUfWeHy6Q+2t7H7O7qocD6CvQOR6epmwPyPh0LrYY7k1w1R GmAcZD+II6TG95uNsj+xC20pvTjKD/SOcWEb2nveV63E71YDhY/TmrOFD36o3uKEWvI0 NievA2WZJqlNKzvYtZGEtoRt/KtcCivKN0ztBunV8Tnq9Gt/+t9G9g8wlc0meEomF7WW oHYFiaCuXl22JXTPoGLmHX/JNNWu7kxGjJl6O/0N+Zaya5zK9EfkBO4BuUazCdU/UMsj TkCg== X-Received: by 10.50.73.226 with SMTP id o2mr672932igv.22.1370320323385; Mon, 03 Jun 2013 21:32:03 -0700 (PDT) Received: from 53.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id ot10sm16729268igb.9.2013.06.03.21.32.01 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Jun 2013 21:32:02 -0700 (PDT) Sender: Warner Losh Subject: Re: Kernelspace C11 atomics for MIPS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Mon, 3 Jun 2013 22:32:00 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Juli Mallett X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQkS85D3GOiLZuypQgooaiYO7hoIjs6xRtdDogBGaDRMQbUORG9QQklg3fap7sktwXV2I+ge Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 04:32:04 -0000 On Jun 3, 2013, at 10:04 PM, Juli Mallett wrote: > On Mon, Jun 3, 2013 at 8:57 PM, Adrian Chadd = wrote: > On 3 June 2013 20:55, Juli Mallett wrote: >=20 > > To drain the pipeline on certain deficient (and mostly older) CPUs = by way of > > guesswork and a little vague magic. Most CPUs we support, I would = guess, do > > not need this, and it continues to exist solely for hysterical = reasons. >=20 > How can I turn it off for my compiles? >=20 > Edit the source. This is not and this kind of thing must not be a = user-visible go-faster knob. I'm anticipating that someone might want = to respond to "edit the source" by saying that users don't have to edit = the source, without understanding the kind of change this is. This isn't a user-visible knob. If you don't know what you are doing, = then don't do it. In the absence of errata, they aren't called out as being required. =46rom Linux, I could find them in the following contexts: One of the places where sync was used in au1000 (at the end of the = DO_SLEEP macro) On octeon, syncw; syncw is used to work around CN3xxx core bugs bmips_read_zscm_reg has a bunch of _ssnops after it. But it is unused = (perhaps in retired CPUs) au1k_wait() has them And that's it. I think they can safely go if Linux doesn't have them :) > > I've certainly gotten rid of them and some other cargo cult = synchronization > > on Octeon for testing and had it survive under considerable load, = and > > occasionally with some slight speedups (for some more commonly-used = or > > slower things than Just a Bunch Of NOPs.) >=20 > Right. Well, since it's happening on every inlined lock, it's a bit = silly. >=20 > Yes. Yes. > > The trouble is that proving they aren't necessary requires being = rigorous > > and careful in understanding documentation and errata, and FUD about = their > > possible necessity is somewhat-intimidating. It's not an easy kind = of > > corruption/unreliability/etc., to prove the lack of empirically. >=20 > I've checked the diassembly from gcc-4.mumble on linux; it doesn't > include NOPs like this as far as I can tell. >=20 > Neat. >=20 > You might also like to look at usage of 'sync' (and its variants, or = the lack of use of its variants) and the possibility of using newer = mips32/64 instructions to change whether interrupts are enabled, and a = number of other things, at least for certain CPU types. =20 Yes, that would be awesome... > And excessive use of all kinds of memory barriers (including simple = memory-clobber barriers) and and and. There's a lot of small changes = that can be made that add up, but building confidence across the range = of hardware we support is genuinely-hard. Yes, we need read barrier, write barrier and general memory barrier = better in the tree. And MIPS' implementation needs to improve. I think they date to the very earliest days of the port (and predate the = Juniper MIPS merge)... If you look at svn blame, it says: 178172 imp "nop\n\t" for all of them. I have no clue where they came from. Looking at the svn = log, it appears they came from the mips2 branch, which may or may not = have been before or after the Juniper code base was merged in. I think they can safely be relegated to the dustbin of history. Warner= From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 04:42:42 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B7928A52 for ; Tue, 4 Jun 2013 04:42:42 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ie0-x229.google.com (mail-ie0-x229.google.com [IPv6:2607:f8b0:4001:c03::229]) by mx1.freebsd.org (Postfix) with ESMTP id 855341A66 for ; Tue, 4 Jun 2013 04:42:42 +0000 (UTC) Received: by mail-ie0-f169.google.com with SMTP id 10so11196452ied.0 for ; Mon, 03 Jun 2013 21:42:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=1vTRJLmUnOFfInZqxD6Y4bkeBW+s05zLyd8c9y47FSo=; b=pOw6NbFdBmRH05560fJP6CsQDUZ4EgQPc3Ew9N2slpqrFQk1ZnVoXENxRyDTAwlZ/V yBDxvdN/J2ez3BsJBBV5XQGnR5IxLeOv6yd5TudjVuBURAoWf7FdLRisPw4rYk/DgZnA U9UaNtoaxHd/tvF4QBhjnwEoiyQ5+pbZUVatqS/7jV2oa5davVWEHI8pFrumfboqw4xt 5Wma3dYmlahyJovCPknhTDLxtPDI1mhrfOO4zy98dpUFT8j3gU0657Shz7lsY7YrxrHg AL7XLPi6OPmfE03yFi0aPjIs5CbavJvXL2p30k8036RI3c74PXbEHtamBOTXGHViqB41 JjaQ== X-Received: by 10.50.136.138 with SMTP id qa10mr671216igb.53.1370320962060; Mon, 03 Jun 2013 21:42:42 -0700 (PDT) Received: from 53.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id l14sm22938417igf.9.2013.06.03.21.42.40 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Jun 2013 21:42:41 -0700 (PDT) Sender: Warner Losh Subject: Re: Kernelspace C11 atomics for MIPS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Mon, 3 Jun 2013 22:42:39 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <05C98B6B-1531-4E09-80D7-4F3B1A88FF01@bsdimp.com> References: To: Patrick Kelsey X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQkVUt/s1cUq/Cnkb/RG2YlsejDaZxeAis2GNtNZj88EH7LYCKbhyEoJZfcJ6pOHC6SdLCZ+ Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 04:42:42 -0000 On Jun 3, 2013, at 10:15 PM, Patrick Kelsey wrote: > On Tue, Jun 4, 2013 at 12:08 AM, Patrick Kelsey = wrote: >> On Mon, Jun 3, 2013 at 11:57 PM, Adrian Chadd = wrote: >>> On 3 June 2013 20:55, Juli Mallett wrote: >>>=20 >>>> To drain the pipeline on certain deficient (and mostly older) CPUs = by way of >>>> guesswork and a little vague magic. Most CPUs we support, I would = guess, do >>>> not need this, and it continues to exist solely for hysterical = reasons. >>>=20 >>> How can I turn it off for my compiles? >>>=20 >>>> I've certainly gotten rid of them and some other cargo cult = synchronization >>>> on Octeon for testing and had it survive under considerable load, = and >>>> occasionally with some slight speedups (for some more commonly-used = or >>>> slower things than Just a Bunch Of NOPs.) >>>=20 >>> Right. Well, since it's happening on every inlined lock, it's a bit = silly. >>>=20 >>>> The trouble is that proving they aren't necessary requires being = rigorous >>>> and careful in understanding documentation and errata, and FUD = about their >>>> possible necessity is somewhat-intimidating. It's not an easy kind = of >>>> corruption/unreliability/etc., to prove the lack of empirically. >>>=20 >>> I've checked the diassembly from gcc-4.mumble on linux; it doesn't >>> include NOPs like this as far as I can tell. >>>=20 >>=20 >> The sync + 8 nops is coming from the definition of mips_sync() in >> sys/mips/include/atomic.h. They came from the old mips2 branch, which may have been from the = Juniper code merge, or maybe not. >> I agree with Juli that it appears to be a manual pipeline-flush >> holdover from earlier days - I'm guessing there's 8 nops because the >> R4000/4400 had both the sync instruction and an 8-stage pipeline. = I'm >> further guessing this was an attempt at providing stronger ordering >> semantics than the sync instruction itself for the following >> mb()/wmb()/rmb() definitions that use it, as the sync instruction >> definition doesn't restrict execution of the before/after = loads/stores >> with respect to the sync instruction itself. >=20 > Forgot to emphasize that this particular bit of old-school > nop-counting is either pointless or a latent hazard - 8 does not cover > the deepest MIPS pipeline around, then there's superscalar issue to > consider - so I think it's either unnecessary or insufficient. So > far, that's all criticism and no solution :/ Yes, there's new nops for these situations starting in mips32r2 and = mips64r2 ISAs. I think that this originated in the mips-jnpr merge, but can't find the = old branches anywhere... Warner From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 04:43:21 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id E1E71B19 for ; Tue, 4 Jun 2013 04:43:21 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ie0-x22a.google.com (mail-ie0-x22a.google.com [IPv6:2607:f8b0:4001:c03::22a]) by mx1.freebsd.org (Postfix) with ESMTP id B2A171A6F for ; Tue, 4 Jun 2013 04:43:21 +0000 (UTC) Received: by mail-ie0-f170.google.com with SMTP id e14so13211917iej.29 for ; Mon, 03 Jun 2013 21:43:21 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=Hi/EKc4snZ+PBS/UVEUSxyh7IB7PY1jEqB3WBPP6uYE=; b=NgsWb1sbGn6pH27OP/WoibPoy9FHfq84gfNfZMXnrLZWN2NO4gmJQa4jel+aF56Yaa 7C1YNyxSQG4sdhDvkaDnoTT3Ob7HkpbRi2aInTuo7i4goyGYkUxJScViTPbcRlXg/ZkE 3BCTamv6PTw8HemCXacE/oNEWbm3vrPmna8yadq0wOYMqBabzQ8N/y7f2NTccjRnLXCp 56T2iZe1XHs++ftohhPBYe7feknZtrD/Z8aEHMrGgSaM7w+t+MRWrEazDYr4lSnhpTtF rWJF9G9dDogiLB3OvLJbZZ/ktZoi9bAbUbl9hRzNW3WORsALkI0hOIXwB8dMdkrDVm3I 0FgA== X-Received: by 10.50.87.4 with SMTP id t4mr648650igz.76.1370321001338; Mon, 03 Jun 2013 21:43:21 -0700 (PDT) Received: from 53.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id l14sm22938417igf.9.2013.06.03.21.43.19 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Jun 2013 21:43:20 -0700 (PDT) Sender: Warner Losh Subject: Re: Kernelspace C11 atomics for MIPS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: Date: Mon, 3 Jun 2013 22:43:19 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: References: To: Adrian Chadd X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQm9sFwf0CtF4EWDNK4ia1/EjN1lUquAAIPKpAqDM7TZTy0muaxBcK5qm0ZwyNJbap/7Zg6X Cc: Ed Schouten , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 04:43:21 -0000 On Jun 3, 2013, at 8:45 PM, Adrian Chadd wrote: > Speaking of this; any idea why the SYNC operators have 8 NOPs = following them? Yes, that's the exact issue that I've had with them, but have never had = time to sort it out... Warner > I noticed that when going through disassemblies of various mips24k .o = files. >=20 >=20 >=20 > Adrian >=20 > On 3 June 2013 10:53, Warner Losh wrote: >>=20 >> On Jun 3, 2013, at 8:04 AM, Ed Schouten wrote: >>=20 >>> Hi, >>>=20 >>> As of r251230, it should be possible to use C11 atomics in >>> kernelspace, by including ! Even when not using = Clang >>> (but GCC 4.2), it is possible to use quite a large portion of the = API. >>> A couple of limitations: >>>=20 >>> - The memory order argument is simply ignored, making all the calls = do >>> a full memory barrier. >>> - At least Clang allows you to do arithmetic on C11 atomics directly >>> (e.g. "a +=3D 5" =3D=3D "atomic_fetch_add(&a, 5)"), which is of = course not >>> possible to mimick. >>> - The atomic functions only work on 1,2,4,8-byte types, which is >>> probably a good thing. >>>=20 >>> Amazingly, it turns out that it most of the architectures, with the >>> exception of ARM and MIPS. To make MIPS work, we need to implement >>> some of the __sync_* functions that are described here: >>>=20 >>> http://gcc.gnu.org/onlinedocs/gcc-4.1.2/gcc/Atomic-Builtins.html >>>=20 >>> Some time ago I already added some of these functions to our >>> libcompiler-rt in userspace, to make atomics work there. >>> Unfortunately, these functions were quite horribly implemented, as I >>> tried to build them on top of , which is far from >>> trivial/efficient. It is also restricted to 4 and 8-byte types. = That's >>> why I thought: why not spend some time learning MIPS assembly and >>> write some decent implementations for these functions? >>>=20 >>> The result: >>>=20 >>> http://80386.nl/pub/mips-stdatomic.txt >>=20 >> The number of necessary syncs varies by processor type. There's also = newer synchronization instructions that make this as efficient as = possible for all mips32r2 and mips64r2-based machines. Older Caviums, at = least and maybe newer ones, also have their own variants. What you have = will mostly work for the processors we have to support. mips_sync could = therefore be better. Doing it before AND after seems like overkill as = well. Since sync is a fairly performance killing assembler instruction, = how would you feel about allowing optimizations? >>=20 >> This is my biggest single concern about the patch, but it also my = current biggest concern about the MIPS atomic operators in general. >>=20 >>> For now, please focus on sys/mips/mips/stdatomic.c. It implements = all >>> the __sync_* functions called by for 1, 2, 4 and 8 = byte >>> types. There is some testing code in there as well, which can be >>> ignored. This code disassembles to the following: >>>=20 >>> http://80386.nl/pub/mips-stdatomic-disasm.txt >>>=20 >>> As I don't own a MIPS system myself, I was thinking about tinkering = a >>> bit with qemu to see whether these functions work properly. My >>> questions are: >>>=20 >>> - Does anyone have any comments on the C code and/or the machine = code >>> generated? Are there some nifty tricks I can apply to make the = machine >>> code more efficient that I am unaware o? >>> - Is there anyone interested in testing this code a bit more >>> thoroughly on physical hardware? >>> - Would anyone mind if I committed this to HEAD? >>=20 >> I have some cavium gear I can easily test on, and some other stuff I = can less-easily test on. >>=20 >> It wouldn't be horrible to commit to head, but it would affect = performance in many places. >>=20 >> Don't commit the kern/bla.c standard change to conf/files, it looks = to be bogus :) >>=20 >> Warner >>=20 >> _______________________________________________ >> freebsd-mips@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-mips >> To unsubscribe, send any mail to = "freebsd-mips-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 05:07:46 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 85C5F74 for ; Tue, 4 Jun 2013 05:07:46 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-pb0-x230.google.com (mail-pb0-x230.google.com [IPv6:2607:f8b0:400e:c01::230]) by mx1.freebsd.org (Postfix) with ESMTP id 59C951B16 for ; Tue, 4 Jun 2013 05:07:46 +0000 (UTC) Received: by mail-pb0-f48.google.com with SMTP id md4so6689367pbc.35 for ; Mon, 03 Jun 2013 22:07:45 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :message-id:references:to:x-mailer:x-gm-message-state; bh=ZT2hy3S9lWNrMxgh/knBaazXir6HH2gE2yZYYehORdo=; b=W5O23I3t1cO8GS2TJsLYHNXl1ayoGg/cakDaVK9iMjDd6/kId1Xmn7DMe34uea0AmR PWwVi1zqRfahvmfysYgRQnilCHbRpibEbcIFbVlTHX7TVSJrokBgOaB3CD9EybunNozc /2+RClFvUxHeZ1gpvsCw7KhV/moxkbSYbJGEj+ZjVnC9cNIcaRWmZgTKL1KExKf8d6lv AvE8ibzD3XwE70bi6+h7Jddn5+qwzphWh0jVvuOhCd23nYPkfUdBPfAJAZHIX5n/6ek2 cvVTD+Nqbvy8Vy3YSkzE/xe6BQU2T9c5z6oQUwyjXIJK1rXNosCDZ0jqbzwwQfP6CLOA Hoag== X-Received: by 10.68.71.129 with SMTP id v1mr27218958pbu.136.1370322465563; Mon, 03 Jun 2013 22:07:45 -0700 (PDT) Received: from 53.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id b10sm14925302pag.22.2013.06.03.22.07.43 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Mon, 03 Jun 2013 22:07:44 -0700 (PDT) Sender: Warner Losh Subject: Re: Kernelspace C11 atomics for MIPS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: multipart/mixed; boundary=Apple-Mail-8--974245673 From: Warner Losh In-Reply-To: Date: Mon, 3 Jun 2013 23:07:41 -0600 Message-Id: <232DBBD8-3F32-4D42-85AB-AC5647EEA768@bsdimp.com> References: To: Patrick Kelsey X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQn4icn/OV7ajlYFn6ON4MyEBIdb+QThNUWA5HMRL5kK5nE+SXqzvrTzGKKtE1OuSza13iSC Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 05:07:46 -0000 --Apple-Mail-8--974245673 Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset=us-ascii Please find attached a simple patch that I'd like all MIPS users to try. Warner --Apple-Mail-8--974245673 Content-Disposition: attachment; filename=P Content-Type: application/octet-stream; x-unix-mode=0664; name="P" Content-Transfer-Encoding: 7bit Index: atomic.h =================================================================== --- atomic.h (revision 250753) +++ atomic.h (working copy) @@ -44,20 +44,16 @@ * do not have atomic operations defined for them, but generally shouldn't * need atomic operations. */ +#ifndef __MIPS_PLATFORM_SYNC_NOPS +#define __MIPS_PLATFORM_SYNC_NOPS "" +#endif static __inline void mips_sync(void) { - __asm __volatile (".set noreorder\n\t" - "sync\n\t" - "nop\n\t" - "nop\n\t" - "nop\n\t" - "nop\n\t" - "nop\n\t" - "nop\n\t" - "nop\n\t" - "nop\n\t" + __asm __volatile (".set noreorder\n" + "\tsync\n" + __MIPS_PLATFORM_SYNC_NOPS ".set reorder\n" : : : "memory"); } --Apple-Mail-8--974245673 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii On Jun 3, 2013, at 10:15 PM, Patrick Kelsey wrote: > On Tue, Jun 4, 2013 at 12:08 AM, Patrick Kelsey = wrote: >> On Mon, Jun 3, 2013 at 11:57 PM, Adrian Chadd = wrote: >>> On 3 June 2013 20:55, Juli Mallett wrote: >>>=20 >>>> To drain the pipeline on certain deficient (and mostly older) CPUs = by way of >>>> guesswork and a little vague magic. Most CPUs we support, I would = guess, do >>>> not need this, and it continues to exist solely for hysterical = reasons. >>>=20 >>> How can I turn it off for my compiles? >>>=20 >>>> I've certainly gotten rid of them and some other cargo cult = synchronization >>>> on Octeon for testing and had it survive under considerable load, = and >>>> occasionally with some slight speedups (for some more commonly-used = or >>>> slower things than Just a Bunch Of NOPs.) >>>=20 >>> Right. Well, since it's happening on every inlined lock, it's a bit = silly. >>>=20 >>>> The trouble is that proving they aren't necessary requires being = rigorous >>>> and careful in understanding documentation and errata, and FUD = about their >>>> possible necessity is somewhat-intimidating. It's not an easy kind = of >>>> corruption/unreliability/etc., to prove the lack of empirically. >>>=20 >>> I've checked the diassembly from gcc-4.mumble on linux; it doesn't >>> include NOPs like this as far as I can tell. >>>=20 >>=20 >> The sync + 8 nops is coming from the definition of mips_sync() in >> sys/mips/include/atomic.h. >>=20 >> I agree with Juli that it appears to be a manual pipeline-flush >> holdover from earlier days - I'm guessing there's 8 nops because the >> R4000/4400 had both the sync instruction and an 8-stage pipeline. = I'm >> further guessing this was an attempt at providing stronger ordering >> semantics than the sync instruction itself for the following >> mb()/wmb()/rmb() definitions that use it, as the sync instruction >> definition doesn't restrict execution of the before/after = loads/stores >> with respect to the sync instruction itself. >=20 > Forgot to emphasize that this particular bit of old-school > nop-counting is either pointless or a latent hazard - 8 does not cover > the deepest MIPS pipeline around, then there's superscalar issue to > consider - so I think it's either unnecessary or insufficient. So > far, that's all criticism and no solution :/ > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to = "freebsd-arch-unsubscribe@freebsd.org" --Apple-Mail-8--974245673-- From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 07:08:57 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id B0938D9E; Tue, 4 Jun 2013 07:08:57 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qc0-x22f.google.com (mail-qc0-x22f.google.com [IPv6:2607:f8b0:400d:c01::22f]) by mx1.freebsd.org (Postfix) with ESMTP id 5105B1EAC; Tue, 4 Jun 2013 07:08:57 +0000 (UTC) Received: by mail-qc0-f175.google.com with SMTP id a1so2729186qcx.34 for ; Tue, 04 Jun 2013 00:08:56 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=D9i+6Bc7wu79/ZOiNtq4M0IHPsQlgoNuxMY225E8AVY=; b=b55dNU1W/NM6yyJ/HpvMIBr9FBce57JVxpNrZ+SRUzhPclQqxESSZ0uG7xZMHZ+YBh lkN6PsMjWAl1H6rn4ms/bogJ+rwlWLfnifAJWd90pCf/t15gFcBrVXmucOycYJ7t1sMT SpHdQwG5uyL/ZiZSPMmujP6PxTSoe9s0BpXgkJHegq4uTOgSHJKLxEbX/WptfdPNl7fv ixOJ6ZFYJTWusJSnWozF8RZf880DyBE4g7NPZSO+bG41WaodZ2yF5zZoFLnKBCWEdTFV swRoIPFMmeh/+sh/DbtLzPTY4iHNfL8ltdV1P1zPaajYZp3vbs6YbdUmhANERmEWwV2T 4yDA== MIME-Version: 1.0 X-Received: by 10.49.120.198 with SMTP id le6mr25760086qeb.59.1370329736693; Tue, 04 Jun 2013 00:08:56 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.71.12 with HTTP; Tue, 4 Jun 2013 00:08:56 -0700 (PDT) In-Reply-To: <232DBBD8-3F32-4D42-85AB-AC5647EEA768@bsdimp.com> References: <232DBBD8-3F32-4D42-85AB-AC5647EEA768@bsdimp.com> Date: Tue, 4 Jun 2013 00:08:56 -0700 X-Google-Sender-Auth: jb8X0a2919fvzPwcpjzjGNgZAGU Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD-arch , "freebsd-mips@FreeBSD.org" , Ed Schouten X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 07:08:57 -0000 I'm running this particular patch on my test AR71xx AP. Everything seems to be ok so far. Adrian From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 08:19:24 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id D704480B for ; Tue, 4 Jun 2013 08:19:24 +0000 (UTC) (envelope-from andre@freebsd.org) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.freebsd.org (Postfix) with ESMTP id 4EF4110F7 for ; Tue, 4 Jun 2013 08:19:23 +0000 (UTC) Received: (qmail 96941 invoked from network); 4 Jun 2013 09:16:43 -0000 Received: from c00l3r.networx.ch (HELO [127.0.0.1]) ([62.48.2.2]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 4 Jun 2013 09:16:43 -0000 Message-ID: <51ADA308.6040904@freebsd.org> Date: Tue, 04 Jun 2013 10:19:20 +0200 From: Andre Oppermann User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130509 Thunderbird/17.0.6 MIME-Version: 1.0 To: Ed Schouten Subject: Re: Kernelspace C11 atomics for MIPS References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-mips@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 08:19:24 -0000 On 03.06.2013 16:04, Ed Schouten wrote: > Hi, > > As of r251230, it should be possible to use C11 atomics in > kernelspace, by including ! Even when not using Clang > (but GCC 4.2), it is possible to use quite a large portion of the API. I'm a bit wary of *kernel* developers using C11-native atomics as opposed to our own atomic API. This could lead to a proliferation of home-grown, more or less correctly working, locks and variants thereof (mostly less correct). Atomics and locks are difficult enough to get right and reason about even with our rather good API and I scream in fear thinking about everyone(tm) doing their own "optimized" lock or even forgoing it because "it's atomic". I would even propose to go as far as disbarring the use of C11 atomics in the kernel other than inside the officially supported lock API. -- Andre From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 12:53:03 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CC94DF18; Tue, 4 Jun 2013 12:53:03 +0000 (UTC) (envelope-from aduane@juniper.net) Received: from tx2outboundpool.messaging.microsoft.com (tx2ehsobe001.messaging.microsoft.com [65.55.88.11]) by mx1.freebsd.org (Postfix) with ESMTP id 7B4971013; Tue, 4 Jun 2013 12:53:03 +0000 (UTC) Received: from mail142-tx2-R.bigfish.com (10.9.14.226) by TX2EHSOBE015.bigfish.com (10.9.40.35) with Microsoft SMTP Server id 14.1.225.23; Tue, 4 Jun 2013 12:22:44 +0000 Received: from mail142-tx2 (localhost [127.0.0.1]) by mail142-tx2-R.bigfish.com (Postfix) with ESMTP id 99B2A240091; Tue, 4 Jun 2013 12:22:44 +0000 (UTC) X-Forefront-Antispam-Report: CIP:66.129.224.51; KIP:(null); UIP:(null); IPV:NLI; H:P-EMHUB02-HQ.jnpr.net; RD:none; EFVD:NLI X-SpamScore: -4 X-BigFish: PS-4(zz98dI9371I542I1432Izz1f42h1ee6h1de0h1fdah1202h1e76h1d1ah1d2ah1fc6hzz8275ch17326ah8275dhz31h2a8h683h839h944hd25hf0ah1220h1288h12a5h12a9h12bdh137ah13b6h1441h1504h1537h153bh15d0h162dh1631h1758h18e1h1946h19b5h19ceh1ad9h1b0ah1d07h1d0ch1d2eh1d3fh1de9h1dfeh1dffh1155h) Received-SPF: softfail (mail142-tx2: transitioning domain of juniper.net does not designate 66.129.224.51 as permitted sender) client-ip=66.129.224.51; envelope-from=aduane@juniper.net; helo=P-EMHUB02-HQ.jnpr.net ; -HQ.jnpr.net ; X-Forefront-Antispam-Report-Untrusted: CIP:157.56.244.213; KIP:(null); UIP:(null); (null); H:CH1PRD0510HT002.namprd05.prod.outlook.com; R:internal; EFV:INT Received: from mail142-tx2 (localhost.localdomain [127.0.0.1]) by mail142-tx2 (MessageSwitch) id 13703485634494_30393; Tue, 4 Jun 2013 12:22:43 +0000 (UTC) Received: from TX2EHSMHS008.bigfish.com (unknown [10.9.14.234]) by mail142-tx2.bigfish.com (Postfix) with ESMTP id E7B10440050; Tue, 4 Jun 2013 12:22:42 +0000 (UTC) Received: from P-EMHUB02-HQ.jnpr.net (66.129.224.51) by TX2EHSMHS008.bigfish.com (10.9.99.108) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 4 Jun 2013 12:22:40 +0000 Received: from P-CLDFE02-HQ.jnpr.net (172.24.192.60) by P-EMHUB02-HQ.jnpr.net (172.24.192.36) with Microsoft SMTP Server (TLS) id 8.3.213.0; Tue, 4 Jun 2013 05:22:39 -0700 Received: from o365mail.juniper.net (207.17.137.149) by o365mail.juniper.net (172.24.192.60) with Microsoft SMTP Server id 14.1.355.2; Tue, 4 Jun 2013 05:22:38 -0700 Received: from ch1outboundpool.messaging.microsoft.com (216.32.181.183) by o365mail.juniper.net (207.17.137.149) with Microsoft SMTP Server (TLS) id 14.1.355.2; Tue, 4 Jun 2013 05:25:48 -0700 Received: from mail57-ch1-R.bigfish.com (10.43.68.240) by CH1EHSOBE019.bigfish.com (10.43.70.76) with Microsoft SMTP Server id 14.1.225.23; Tue, 4 Jun 2013 12:22:38 +0000 Received: from mail57-ch1 (localhost [127.0.0.1]) by mail57-ch1-R.bigfish.com (Postfix) with ESMTP id 41C33E0161; Tue, 4 Jun 2013 12:22:38 +0000 (UTC) Received: from mail57-ch1 (localhost.localdomain [127.0.0.1]) by mail57-ch1 (MessageSwitch) id 1370348556289141_8751; Tue, 4 Jun 2013 12:22:36 +0000 (UTC) Received: from CH1EHSMHS014.bigfish.com (snatpool1.int.messaging.microsoft.com [10.43.68.244]) by mail57-ch1.bigfish.com (Postfix) with ESMTP id 43FFE2C00F9; Tue, 4 Jun 2013 12:22:36 +0000 (UTC) Received: from CH1PRD0510HT002.namprd05.prod.outlook.com (157.56.244.213) by CH1EHSMHS014.bigfish.com (10.43.70.14) with Microsoft SMTP Server (TLS) id 14.1.225.23; Tue, 4 Jun 2013 12:22:34 +0000 Received: from CH1PRD0510MB392.namprd05.prod.outlook.com ([169.254.6.109]) by CH1PRD0510HT002.namprd05.prod.outlook.com ([10.255.150.37]) with mapi id 14.16.0311.000; Tue, 4 Jun 2013 12:22:34 +0000 From: Andrew Duane To: Juli Mallett , Adrian Chadd Subject: RE: Kernelspace C11 atomics for MIPS Thread-Topic: Kernelspace C11 atomics for MIPS Thread-Index: AQHOYINHGC9M8xF/6kiJf8fnnz6uaJkk2jUAgAATWwCAAIsx4A== Date: Tue, 4 Jun 2013 12:22:33 +0000 Message-ID: <477C1270D3E5484DA2303CEBE274C9E13210A1C0@CH1PRD0510MB392.namprd05.prod.outlook.com> References: In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [66.129.232.2] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-FOPE-CONNECTOR: Id%12219$Dn%FREEBSD.ORG$RO%2$TLS%5$FQDN%onpremiseedge-1018244.customer.frontbridge.com$TlsDn%o365mail.juniper.net X-FOPE-CONNECTOR: Id%12219$Dn%80386.NL$RO%2$TLS%5$FQDN%onpremiseedge-1018244.customer.frontbridge.com$TlsDn%o365mail.juniper.net X-OriginatorOrg: juniper.net Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 12:53:03 -0000 > -----Original Message----- > From: owner-freebsd-mips@freebsd.org [mailto:owner-freebsd- > mips@freebsd.org] On Behalf Of Juli Mallett > Sent: Monday, June 03, 2013 11:55 PM > To: Adrian Chadd > Cc: Ed Schouten; freebsd-mips@FreeBSD.org; FreeBSD-arch > Subject: Re: Kernelspace C11 atomics for MIPS >=20 > On Mon, Jun 3, 2013 at 7:45 PM, Adrian Chadd wrote: >=20 > > Speaking of this; any idea why the SYNC operators have 8 NOPs following > > them? > > > > I noticed that when going through disassemblies of various mips24k .o > > files. > > >=20 > To drain the pipeline on certain deficient (and mostly older) CPUs by way > of guesswork and a little vague magic. Most CPUs we support, I would > guess, do not need this, and it continues to exist solely for > hysterical reasons. >=20 > I've certainly gotten rid of them and some other cargo cult synchronizati= on > on Octeon for testing and had it survive under considerable load, and > occasionally with some slight speedups (for some more commonly-used or > slower things than Just a Bunch Of NOPs.) >=20 > The trouble is that proving they aren't necessary requires being rigorous > and careful in understanding documentation and errata, and FUD about thei= r > possible necessity is somewhat-intimidating. It's not an easy kind of > corruption/unreliability/etc., to prove the lack of empirically. The various CPU types are supposed to specify exactly how many NOPs are nee= ded, what kind of barrier is needed and where, and which type of NOP is nee= ded (Alpha had at least two). The barriers are designed to insure correct o= peration ordering across the memory architectures including write buffers, = DMA hardware, L1/L2[/L3] caches and their connections to the cores. The CPU= should specify exactly what is needed where, and why. It should never be "= superstition". There is an exact hardware reason for every sync/barrier ope= ration, and every NOP needed, just like the COP0 hazards. Given that, Juli's last paragraph is right on point. The documentation can = be dense and difficult to understand, since it's usually written by hardwar= e engineers :-) And since getting it wrong can make for some really subtle,= intermittent, and incredibly hard to diagnose problems, it's easier to err= on the side of caution. It also happens that different CPUs included in a = certain compile switch may have different requirements, so you have to use = worst case. >=20 > Juli. > _______________________________________________ > freebsd-mips@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-mips > To unsubscribe, send any mail to "freebsd-mips-unsubscribe@freebsd.org" >=20 .................................... Andrew L. Duane Resident Architect - AT&T Technical Lead m +1 603.770.7088 o +1 408.933.6944 (2-6944) skype: andrewlduane aduane@juniper.net From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 13:12:37 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 0963F4F1; Tue, 4 Jun 2013 13:12:37 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-ob0-x22c.google.com (mail-ob0-x22c.google.com [IPv6:2607:f8b0:4003:c01::22c]) by mx1.freebsd.org (Postfix) with ESMTP id B2EB01113; Tue, 4 Jun 2013 13:12:36 +0000 (UTC) Received: by mail-ob0-f172.google.com with SMTP id wo10so310488obc.17 for ; Tue, 04 Jun 2013 06:12:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=BVokO6MxSP+o73X/DQF48jeYhdomuymc4EyA8LVGtL8=; b=c8D/2l+kO9thxT0NpY8hUGAEDgFrxYHGam+k8cyFRMLhjP5UJ9ITURXtSHzPeSlc/f /rN/vsmIg8wiwr7Ll5Yz/vQpDK1j40FDJ9zcjiGYpxQ0HvHhZAk300Gs8HUHV7ndFqBG cO1bfcv99K3UAfFZle80rLt1VX0pQ0KOVRme06xxuhiy6L6/+Jg29KWl7cuFFsI/p6CS thHC5aybbKBY3PcUQliWm+I8FFasGpOUYcSt3+deN3CHPBjP1e/DXeA00FgBp23hNMij Sn7QxJzTAwtlscjeBcEQX2NXeCniJoKmcu1FWV+c+i+5xySwVXqRTPDDssemRhD0+++U /YQQ== MIME-Version: 1.0 X-Received: by 10.60.33.102 with SMTP id q6mr12334202oei.111.1370351556336; Tue, 04 Jun 2013 06:12:36 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.182.237.100 with HTTP; Tue, 4 Jun 2013 06:12:36 -0700 (PDT) In-Reply-To: <51ADA308.6040904@freebsd.org> References: <51ADA308.6040904@freebsd.org> Date: Tue, 4 Jun 2013 06:12:36 -0700 X-Google-Sender-Auth: UKda3lyyqPEvZiuB93DLOxsQkOM Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: mdf@FreeBSD.org To: Andre Oppermann Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Ed Schouten , freebsd-mips@freebsd.org, FreeBSD Arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 13:12:37 -0000 On Tue, Jun 4, 2013 at 1:19 AM, Andre Oppermann wrote: > On 03.06.2013 16:04, Ed Schouten wrote: > >> Hi, >> >> As of r251230, it should be possible to use C11 atomics in >> kernelspace, by including ! Even when not using Clang >> (but GCC 4.2), it is possible to use quite a large portion of the API. >> > > I'm a bit wary of *kernel* developers using C11-native atomics as opposed > to our own atomic API. This could lead to a proliferation of home-grown, > more or less correctly working, locks and variants thereof (mostly less > correct). > I don't understand this. > Atomics and locks are difficult enough to get right and reason about even > with our rather good API and I scream in fear thinking about everyone(tm) > doing their own "optimized" lock or even forgoing it because "it's atomic". > Why would we replace primitives that work? Meanwhile, the C11 atomics are at least as well documented as FreeBSD's, and they're standardized. Why should a future programmer, who understands C11 atomics, need to learn all new names to work on our OS? I would even propose to go as far as disbarring the use of C11 atomics in > the kernel other than inside the officially supported lock API. So compare and swap is hard to reason about? The C11 atomics should be no harder to reason about than our own -- their effects are documented. And we expect kernel programmers, of all people, to actually be careful about choosing their instructions. Personally, I find both the C11 atomics and FreeBSD's annoying, since "acquire" and "release" semantics are basically an x86 ism. PPC has no notion of this; it has sync and isync and lwsync instructions which are separate from the atomic set, but can be combined to create the same effect. Except the PPC manual is exceptionally explicit about what guarantees sync provides; it gives a mathematical ordering on loads/stores i, j and which effects can be seen when. "Acquire" and "Release" seem to be named because you kinda need one to acquire a lock and kinda need one to release it. But the effect of ordering loads or stores or both doesn't need to be dependent on the store/load, so putting the two together is just an x86 convenience (and an annoyance on at least PPC). Anyways, that aside, I see no reason to use a home-grown solution when the C standard finally provides one. It seems akin to preferring u_int64_t over uint64_t because we had it first and they changed the spelling on us. Thanks, matthew From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 15:09:21 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 2926D8A4 for ; Tue, 4 Jun 2013 15:09:21 +0000 (UTC) (envelope-from imp@bsdimp.com) Received: from mail-ie0-x22d.google.com (mail-ie0-x22d.google.com [IPv6:2607:f8b0:4001:c03::22d]) by mx1.freebsd.org (Postfix) with ESMTP id EBB6F1823 for ; Tue, 4 Jun 2013 15:09:20 +0000 (UTC) Received: by mail-ie0-f173.google.com with SMTP id k13so683138iea.18 for ; Tue, 04 Jun 2013 08:09:20 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=sender:subject:mime-version:content-type:from:in-reply-to:date:cc :content-transfer-encoding:message-id:references:to:x-mailer :x-gm-message-state; bh=mQjEh5F3H8ANLTXokocrCCP3P7W6+e2Rh72FQ4MymME=; b=JBae+/UGPed6ATeLAcKRB8enbFTxt2kyezUOznfNxLrKboAc9wMLatZiPdfUF/3vDP 0AqemfVCLkSAd7dO9UAShO3ETHWBVJLKqJvp2aWcM4kadgGFETXpNkR1Akovcnl0Bp0u wdWiC6b+S9Bhkx9raAnT4w2qLLlihd+FR1x7IJ8UDoh+UItMIdyqkTd81vKUFtCPpQYX ILsQL2WpFhl7eXH2dx6bD3M1zgXkJQSoQvrLSzTQPp1WLgnOrf3ErLd9i8XXU05XWZd1 tyLTY4eiCocRRoHYB6AhQxYESQlmfBXKYkSmpUuPhr71c4wT3x5+GfIX9TPdtFpU75dM M06w== X-Received: by 10.50.18.17 with SMTP id s17mr1027482igd.80.1370358560526; Tue, 04 Jun 2013 08:09:20 -0700 (PDT) Received: from 53.imp.bsdimp.com (50-78-194-198-static.hfc.comcastbusiness.net. [50.78.194.198]) by mx.google.com with ESMTPSA id f6sm2261850igz.1.2013.06.04.08.09.18 for (version=TLSv1 cipher=ECDHE-RSA-RC4-SHA bits=128/128); Tue, 04 Jun 2013 08:09:19 -0700 (PDT) Sender: Warner Losh Subject: Re: Kernelspace C11 atomics for MIPS Mime-Version: 1.0 (Apple Message framework v1085) Content-Type: text/plain; charset=us-ascii From: Warner Losh In-Reply-To: <477C1270D3E5484DA2303CEBE274C9E13210A1C0@CH1PRD0510MB392.namprd05.prod.outlook.com> Date: Tue, 4 Jun 2013 09:09:17 -0600 Content-Transfer-Encoding: quoted-printable Message-Id: <2EE5CF35-1E46-44B8-83B3-6923FC6FA854@bsdimp.com> References: <477C1270D3E5484DA2303CEBE274C9E13210A1C0@CH1PRD0510MB392.namprd05.prod.outlook.com> To: Andrew Duane X-Mailer: Apple Mail (2.1085) X-Gm-Message-State: ALoCoQlqzWjkMsjebV8yLDzbCjqysOPQh3bAwgnDHfDBW2ZE9ANXp3Az8e4GAHGeVg56fHOp2+Mu Cc: Ed Schouten , "freebsd-mips@FreeBSD.org" , FreeBSD-arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 15:09:21 -0000 On Jun 4, 2013, at 6:22 AM, Andrew Duane wrote: >=20 >> -----Original Message----- >> From: owner-freebsd-mips@freebsd.org [mailto:owner-freebsd- >> mips@freebsd.org] On Behalf Of Juli Mallett >> Sent: Monday, June 03, 2013 11:55 PM >> To: Adrian Chadd >> Cc: Ed Schouten; freebsd-mips@FreeBSD.org; FreeBSD-arch >> Subject: Re: Kernelspace C11 atomics for MIPS >>=20 >> On Mon, Jun 3, 2013 at 7:45 PM, Adrian Chadd = wrote: >>=20 >>> Speaking of this; any idea why the SYNC operators have 8 NOPs = following >>> them? >>>=20 >>> I noticed that when going through disassemblies of various mips24k = .o >>> files. >>>=20 >>=20 >> To drain the pipeline on certain deficient (and mostly older) CPUs by = way >> of guesswork and a little vague magic. Most CPUs we support, I would >> guess, do not need this, and it continues to exist solely for >> hysterical reasons. >>=20 >> I've certainly gotten rid of them and some other cargo cult = synchronization >> on Octeon for testing and had it survive under considerable load, and >> occasionally with some slight speedups (for some more commonly-used = or >> slower things than Just a Bunch Of NOPs.) >>=20 >> The trouble is that proving they aren't necessary requires being = rigorous >> and careful in understanding documentation and errata, and FUD about = their >> possible necessity is somewhat-intimidating. It's not an easy kind = of >> corruption/unreliability/etc., to prove the lack of empirically. >=20 > The various CPU types are supposed to specify exactly how many NOPs = are needed, what kind of barrier is needed and where, and which type of = NOP is needed (Alpha had at least two). The barriers are designed to = insure correct operation ordering across the memory architectures = including write buffers, DMA hardware, L1/L2[/L3] caches and their = connections to the cores. The CPU should specify exactly what is needed = where, and why. It should never be "superstition". There is an exact = hardware reason for every sync/barrier operation, and every NOP needed, = just like the COP0 hazards. Except that none of the examples in the ISA manual have them, and = there's no mention of them at all, unlike COP0 hazards. I know of only = one case in the Linux tree where it is done (for the au1xxxx cores). = There's another place where it is defined in a function, but that = function is never called for older Broadcom MIPS. In NetBSD, extra NOPs are not inserted at all. They do do two syncs for = the SB1250 PASS 1. None of the docs I've seen for latter-day MIPS CPUs document the need = for NOPs. In fact, the only place I think that I've seen them was on one = or two of the older R8000, R10000 and R120000 errata that stated they = were needed there. Since these were the first complicated multi-issue = designs, it wasn't surprising that the NOPs were needed as work arounds. = I can't find these errata with a google search now, so I can't confirm = this dim memory that I have. The reason the NOPs are there today likely is superstition. A cargo-cult = workaround from the past whose days have come and gone, leaving nothing = but an echo in the code. > Given that, Juli's last paragraph is right on point. The documentation = can be dense and difficult to understand, since it's usually written by = hardware engineers :-) And since getting it wrong can make for some = really subtle, intermittent, and incredibly hard to diagnose problems, = it's easier to err on the side of caution. It also happens that = different CPUs included in a certain compile switch may have different = requirements, so you have to use worst case. I'll agree about the dense documentation. But usually that's around all = the crazy cache effects that one must understand to cope with the design = that puts half of the cache management in software. I'm all for ditching them unless a specific reason for keeping them can = be found. >> Juli. >> _______________________________________________ >> freebsd-mips@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-mips >> To unsubscribe, send any mail to = "freebsd-mips-unsubscribe@freebsd.org" >>=20 >=20 >=20 > .................................... > Andrew L. Duane > Resident Architect - AT&T Technical Lead > m +1 603.770.7088 > o +1 408.933.6944 (2-6944) > skype: andrewlduane > aduane@juniper.net >=20 >=20 >=20 > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to = "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 16:03:59 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id F28566C3; Tue, 4 Jun 2013 16:03:58 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id CFB2F1B17; Tue, 4 Jun 2013 16:03:58 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id A82C9B94F; Tue, 4 Jun 2013 12:03:57 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: Kernelspace C11 atomics for MIPS Date: Tue, 4 Jun 2013 09:52:51 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <51ADA308.6040904@freebsd.org> In-Reply-To: <51ADA308.6040904@freebsd.org> MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201306040952.51513.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 04 Jun 2013 12:03:57 -0400 (EDT) Cc: Ed Schouten , Andre Oppermann , freebsd-mips@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 16:03:59 -0000 On Tuesday, June 04, 2013 4:19:20 am Andre Oppermann wrote: > On 03.06.2013 16:04, Ed Schouten wrote: > > Hi, > > > > As of r251230, it should be possible to use C11 atomics in > > kernelspace, by including ! Even when not using Clang > > (but GCC 4.2), it is possible to use quite a large portion of the API. > > I'm a bit wary of *kernel* developers using C11-native atomics as opposed > to our own atomic API. This could lead to a proliferation of home-grown, > more or less correctly working, locks and variants thereof (mostly less > correct). I think this is not a big deal to worry about as developers have already been free to do this via and haven't gone super crazy. Replacing with is probably fine and should be a simple drop-in replacement for our lock implementations. -- John Baldwin From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 16:03:59 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id A5D776C5; Tue, 4 Jun 2013 16:03:59 +0000 (UTC) (envelope-from jhb@freebsd.org) Received: from bigwig.baldwin.cx (bigwig.baldwin.cx [IPv6:2001:470:1f11:75::1]) by mx1.freebsd.org (Postfix) with ESMTP id 818CC1B18; Tue, 4 Jun 2013 16:03:59 +0000 (UTC) Received: from jhbbsd.localnet (unknown [209.249.190.124]) by bigwig.baldwin.cx (Postfix) with ESMTPSA id C88B1B953; Tue, 4 Jun 2013 12:03:58 -0400 (EDT) From: John Baldwin To: freebsd-arch@freebsd.org Subject: Re: Kernelspace C11 atomics for MIPS Date: Tue, 4 Jun 2013 09:56:00 -0400 User-Agent: KMail/1.13.5 (FreeBSD/8.2-CBSD-20110714-p25; KDE/4.5.5; amd64; ; ) References: <51ADA308.6040904@freebsd.org> In-Reply-To: MIME-Version: 1.0 Content-Type: Text/Plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <201306040956.01065.jhb@freebsd.org> X-Greylist: Sender succeeded SMTP AUTH, not delayed by milter-greylist-4.2.7 (bigwig.baldwin.cx); Tue, 04 Jun 2013 12:03:58 -0400 (EDT) Cc: mdf@freebsd.org, Andre Oppermann , freebsd-mips@freebsd.org, Ed Schouten X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 16:03:59 -0000 On Tuesday, June 04, 2013 9:12:36 am mdf@freebsd.org wrote: > Personally, I find both the C11 atomics and FreeBSD's annoying, > since "acquire" and "release" semantics are basically an x86 ism. PPC has > no notion of this; it has sync and isync and lwsync instructions which are > separate from the atomic set, but can be combined to create the same > effect. Except the PPC manual is exceptionally explicit about what > guarantees sync provides; it gives a mathematical ordering on loads/stores > i, j and which effects can be seen when. "Acquire" and "Release" seem to > be named because you kinda need one to acquire a lock and kinda need one to > release it. But the effect of ordering loads or stores or both doesn't > need to be dependent on the store/load, so putting the two together is just > an x86 convenience (and an annoyance on at least PPC). Actually, it came from ia64 (at least for FreeBSD's), not x86. :) However, it is still useful to think about, and they are barriers with respect to the load/store of the lock cookie. The requirement that the "acquire" blocks any subsequent loads/stores in program order from occurring until after the operation on the lock cookie succeeds and that "release" prevents any loads/stores frmo moving past the operation on the lock cookie is not quite the same as a traditional read or write barrier. acquire and release only require a barrier in one direction and enforce ordering on both reads and writes. -- John Baldwin From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 16:46:57 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D73DCCB5; Tue, 4 Jun 2013 16:46:57 +0000 (UTC) (envelope-from mdf356@gmail.com) Received: from mail-oa0-x236.google.com (mail-oa0-x236.google.com [IPv6:2607:f8b0:4003:c02::236]) by mx1.freebsd.org (Postfix) with ESMTP id 7DD731DA2; Tue, 4 Jun 2013 16:46:57 +0000 (UTC) Received: by mail-oa0-f54.google.com with SMTP id o17so332216oag.27 for ; Tue, 04 Jun 2013 09:46:57 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=U321lJJP6z/A5sq5VfRFdvgFCaN/7LN+d3tIChXOn44=; b=H4T8tvGYva69xqc7cwInlVqQiZDqGTrTUY0L7df2COt5iEdFU2kcc//5leLt/pVmiS WQF9UNIqJLwtdYB6C1pgAN5o+CzBvec08NtgY+9X5Av1MkUpJL0SMQForPlAJcnHwFsY tofa+gGnTylaETpTjusWlX4J9D5HRvFcMwaRwq2jldD6JaK0RORx+7WYCiUMLRGJMwxF NPv+51mCe7AfGM5jmlI+Be+AwUi4fQv2i4plYxbdiusdYOYizZGfGAlmyVkt+w9Jw0Xn wxJ1fKJGnaWf3+HoxBkb9pyw1UhSg2aYU1pMCAo/tE/hT6NWm2Xmbs5q2PqV2sSavQyI MOrA== MIME-Version: 1.0 X-Received: by 10.60.79.231 with SMTP id m7mr12868216oex.105.1370364417114; Tue, 04 Jun 2013 09:46:57 -0700 (PDT) Sender: mdf356@gmail.com Received: by 10.182.237.100 with HTTP; Tue, 4 Jun 2013 09:46:57 -0700 (PDT) In-Reply-To: <201306040956.01065.jhb@freebsd.org> References: <51ADA308.6040904@freebsd.org> <201306040956.01065.jhb@freebsd.org> Date: Tue, 4 Jun 2013 09:46:57 -0700 X-Google-Sender-Auth: 4yIQmTIH3ETY_Ag5xxf_lyy1qnE Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: mdf@FreeBSD.org To: John Baldwin Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Ed Schouten , Andre Oppermann , freebsd-mips@freebsd.org, FreeBSD Arch X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 16:46:57 -0000 On Tue, Jun 4, 2013 at 6:56 AM, John Baldwin wrote: > On Tuesday, June 04, 2013 9:12:36 am mdf@freebsd.org wrote: > > Personally, I find both the C11 atomics and FreeBSD's > annoying, > > since "acquire" and "release" semantics are basically an x86 ism. PPC > has > > no notion of this; it has sync and isync and lwsync instructions which > are > > separate from the atomic set, but can be combined to create the same > > effect. Except the PPC manual is exceptionally explicit about what > > guarantees sync provides; it gives a mathematical ordering on > loads/stores > > i, j and which effects can be seen when. "Acquire" and "Release" seem to > > be named because you kinda need one to acquire a lock and kinda need one > to > > release it. But the effect of ordering loads or stores or both doesn't > > need to be dependent on the store/load, so putting the two together is > just > > an x86 convenience (and an annoyance on at least PPC). > > Actually, it came from ia64 (at least for FreeBSD's), not x86. :) However, > it is still useful to think about, and they are barriers with respect to > the > load/store of the lock cookie. The requirement that the "acquire" blocks > any > subsequent loads/stores in program order from occurring until after the > operation on the lock cookie succeeds and that "release" prevents any > loads/stores frmo moving past the operation on the lock cookie is not quite > the same as a traditional read or write barrier. acquire and release only > require a barrier in one direction and enforce ordering on both reads and > writes. Yeah, thinking more I feel sorry for those CISC architectures that need so many C primitives because it's less efficient to emit a memory fence then the load (or fence then store). It is less elegant, though, and the C standard had to add all the fenced variants of atomics to support it. Thanks, matthew From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 16:56:59 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 13400848; Tue, 4 Jun 2013 16:56:59 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) by mx1.freebsd.org (Postfix) with ESMTP id A81711E34; Tue, 4 Jun 2013 16:56:58 +0000 (UTC) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.14.7/8.14.7) with ESMTP id r54Guqs4088526; Tue, 4 Jun 2013 19:56:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.8.3 kib.kiev.ua r54Guqs4088526 Received: (from kostik@localhost) by tom.home (8.14.7/8.14.7/Submit) id r54GuqYt088525; Tue, 4 Jun 2013 19:56:52 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Tue, 4 Jun 2013 19:56:52 +0300 From: Konstantin Belousov To: John Baldwin Subject: Re: Kernelspace C11 atomics for MIPS Message-ID: <20130604165652.GT3047@kib.kiev.ua> References: <51ADA308.6040904@freebsd.org> <201306040952.51513.jhb@freebsd.org> MIME-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jS6B5y1SGyDdwoJD" Content-Disposition: inline In-Reply-To: <201306040952.51513.jhb@freebsd.org> User-Agent: Mutt/1.5.21 (2010-09-15) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no version=3.3.2 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on tom.home Cc: Ed Schouten , Andre Oppermann , freebsd-mips@freebsd.org, freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 16:56:59 -0000 --jS6B5y1SGyDdwoJD Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Tue, Jun 04, 2013 at 09:52:51AM -0400, John Baldwin wrote: > On Tuesday, June 04, 2013 4:19:20 am Andre Oppermann wrote: > > On 03.06.2013 16:04, Ed Schouten wrote: > > > Hi, > > > > > > As of r251230, it should be possible to use C11 atomics in > > > kernelspace, by including ! Even when not using Clang > > > (but GCC 4.2), it is possible to use quite a large portion of the API. > >=20 > > I'm a bit wary of *kernel* developers using C11-native atomics as oppos= ed > > to our own atomic API. This could lead to a proliferation of home-grow= n, > > more or less correctly working, locks and variants thereof (mostly less > > correct). >=20 > I think this is not a big deal to worry about as developers have already = been=20 > free to do this via and haven't gone super crazy. =20 > Replacing with is probably fine and= =20 > should be a simple drop-in replacement for our lock implementations. I do not think so. The compilers are free to use whatever means to implement the stdatomic. In particular, they are allowed to use simple global lock to protect the 'atomic' access, see ATOMIC_type_LOCK_FREE documented macros. IMO using our machine/atomic.h gives us the desirable exact control over the semantic of locks both kernel and usermode C runtime implement. Practically speaking, I think most people there are capable of fixing bugs and extending functionality of machine/atomic.h, but have no desire or time digging into to change something. --jS6B5y1SGyDdwoJD Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.20 (FreeBSD) iQIcBAEBAgAGBQJRrhxUAAoJEJDCuSvBvK1BVckP/1ZYaOksy4B8GrcRo2vxCUzp cKoWJJ1hqTmygBMfM0umWi0cyOIQvn15GGzKKwpuNWuUKpZSKGaVDrR9PzPeUwoP N249KNgun+jIEE145WkXaP5DtPg3SEsH1KBWDRjIGHXFDMtBUT6msBf1BToCzKTP 0mMyLA3gG53xuPU6sISQT7pNKEf9WXnua33kEM4j7bdlyKYatvL3EgrbOfS1Gwc6 UDJch3jedXDhPJbD26m5AYQm/oLBEbZfkVgquIa5pDslB4w+7bNK0FimQfd+RrpT BI99dZmhLjS5ra3Udwcn8q+05G02HBfHmNmfmizgVVUwQC/8b64KWmyO6orynE2V QsGsFZWzuE6fxfOkQMFCM0nQQW/jk7zvStEL12wOR/UYIEUdvMHDre02/JOkPNQV qEIupfnyDzjI2cFGVO2ZW/2zbrzaE++bhHhwhq3HDs6s4ZLtZV6joCKYZmDNsfQl vrODQLVpHmiMOJ5BRllk4JXKb3N6pv49HQ0J5oXKjJPDrolAOOrb+FMICEqd0Uuu aTYgaDA0zKbMp4VwBqz4r4QiUC+2tDftW/X+Rs06MWSJIfQF53jyhpyVMO8a2f7t MmhQf2iTMVZzgtEf3+oWNSKrDQkPwcoef09aXmBD5VowFX4MtZa5Dvc5D9OI+r7u e0krjCO8WjPF7INZL43S =BxK5 -----END PGP SIGNATURE----- --jS6B5y1SGyDdwoJD-- From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 17:23:35 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3AB6276A; Tue, 4 Jun 2013 17:23:35 +0000 (UTC) (envelope-from edschouten@gmail.com) Received: from mail-vc0-f180.google.com (mail-vc0-f180.google.com [209.85.220.180]) by mx1.freebsd.org (Postfix) with ESMTP id BB3A3103D; Tue, 4 Jun 2013 17:23:34 +0000 (UTC) Received: by mail-vc0-f180.google.com with SMTP id gd11so390049vcb.25 for ; Tue, 04 Jun 2013 10:23:28 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=IkW8ji4gMuhRowwT1AgRR6Wx2nP9CdEAmVAmPGh+zzI=; b=Yslk64yT7w6C3YN6aDro68PEvdUSdrVvg04zhrOTR/fDbO5bquJItSSpXsBJZlETJw C93ke1nlkjJ8wFZlZKo2YNgcf7M9M+Um6dxXrpEB8ndrWbn7sajKzcT/GLPLa+uCYotJ dp/RwBzrogAS7JM8WcO09U9+0faplHuCdfvlec3IJc+jHEp2cTWsudwRDMdpA1OgbuAU sIPCMlo61I3Q/OneI86ZKOTHHFHJzUIqBS1wdj5W4yYGhLzHnuldVn/WrdIF4BVcWpID o3p4pcWBdWdHK9RFG7837Hl6an1LsdFl0KVB0gY5UVll2BiijKP+4icjofPsLfhCVKrr L5vA== MIME-Version: 1.0 X-Received: by 10.221.4.131 with SMTP id oc3mr18813611vcb.49.1370366608682; Tue, 04 Jun 2013 10:23:28 -0700 (PDT) Sender: edschouten@gmail.com Received: by 10.220.107.139 with HTTP; Tue, 4 Jun 2013 10:23:28 -0700 (PDT) In-Reply-To: <20130604165652.GT3047@kib.kiev.ua> References: <51ADA308.6040904@freebsd.org> <201306040952.51513.jhb@freebsd.org> <20130604165652.GT3047@kib.kiev.ua> Date: Tue, 4 Jun 2013 19:23:28 +0200 X-Google-Sender-Auth: 3h2-NMjqZ9YUhjcB7SUcOmt4FZc Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Ed Schouten To: Konstantin Belousov Content-Type: text/plain; charset=UTF-8 Cc: Andre Oppermann , freebsd-mips@freebsd.org, John Baldwin , freebsd-arch@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 17:23:35 -0000 2013/6/4 Konstantin Belousov : > I do not think so. The compilers are free to use whatever means to > implement the stdatomic. In particular, they are allowed to use simple > global lock to protect the 'atomic' access, see ATOMIC_type_LOCK_FREE > documented macros. Well, yes, no, it's complicated. The fact is, we are still free to implement without using those compiler's features. For example, we could have also decided to implement using only code we provide ourselves, as follows: static inline int8_t our_8bit_atomic_store(...) { ... } #define atomic_store(...) _Generic( \ int8_t: our_8bit_atomic_store(....), \ ... \ ) Also, it is extremely unlikely that compilers implement handlers for non-lock-free atomics themselves. Both Clang and GCC 4.7+, for example, will call into __atomic_*_{1,2,4,8,16,c}() whenever it does not know built-in CPU instructions to perform the operation. More details: http://gcc.gnu.org/wiki/Atomic/GCCMM/LIbrary So in my opinion there are tons of ways we can still influence how the atomic operations are performed. The patch I sent out already demonstrates this, as we are free to implement the GCC intrinsics the way we like. -- Ed Schouten From owner-freebsd-mips@FreeBSD.ORG Tue Jun 4 18:07:12 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id F20E229D; Tue, 4 Jun 2013 18:07:11 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qe0-f45.google.com (mail-qe0-f45.google.com [209.85.128.45]) by mx1.freebsd.org (Postfix) with ESMTP id 5A4681210; Tue, 4 Jun 2013 18:07:11 +0000 (UTC) Received: by mail-qe0-f45.google.com with SMTP id q19so409787qeb.4 for ; Tue, 04 Jun 2013 11:07:05 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=ye+/DK9B8RSq2OPcPcACrpKg94AVyIf7MLo8RohwV+U=; b=TuaeoYMffceTzPIq49hkvisUPY+x9S1kLU+YaiuOCEejtxHYz4bvlV01iecsIwfz+c Qa5wxXbAcyvpzt1i1Kf4UZtID5H7NYPM7Bl06C6IFYpnIgi5GWn3ivFYkq/0K3MPuyW/ a24xwmS4kqa2DBhSaNzKGxmN+3Zz65Oj5bxDxA7CWJBbqC8K1hOG3uRBBlh/z1Scf998 b4MXneK7Pq/9lQXyGJ/EJGlXvD5KJbn0MsF+C3Y4S2fO/iMgtusCBqgXk6kDiCCWAquY VEvFc8n8PI3bo8mZyW8X7yncW81MEAQRhQYyt5Xm1LHQRxcSrugsH15rzI6UStpNJ1H7 oUOA== MIME-Version: 1.0 X-Received: by 10.229.149.14 with SMTP id r14mr6819349qcv.59.1370369225122; Tue, 04 Jun 2013 11:07:05 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.71.12 with HTTP; Tue, 4 Jun 2013 11:07:05 -0700 (PDT) In-Reply-To: <232DBBD8-3F32-4D42-85AB-AC5647EEA768@bsdimp.com> References: <232DBBD8-3F32-4D42-85AB-AC5647EEA768@bsdimp.com> Date: Tue, 4 Jun 2013 11:07:05 -0700 X-Google-Sender-Auth: VI61RTyvzr8ndvlahzEoCAnACaw Message-ID: Subject: Re: Kernelspace C11 atomics for MIPS From: Adrian Chadd To: Warner Losh Content-Type: text/plain; charset=ISO-8859-1 Cc: FreeBSD-arch , "freebsd-mips@FreeBSD.org" , Ed Schouten X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 04 Jun 2013 18:07:12 -0000 Hi, It survived an overnight thrashing on my AR7161 (mips24k.) Adrian On 3 June 2013 22:07, Warner Losh wrote: > Please find attached a simple patch that I'd like all MIPS users to try. > > Warner > > > > > On Jun 3, 2013, at 10:15 PM, Patrick Kelsey wrote: > >> On Tue, Jun 4, 2013 at 12:08 AM, Patrick Kelsey wrote: >>> On Mon, Jun 3, 2013 at 11:57 PM, Adrian Chadd wrote: >>>> On 3 June 2013 20:55, Juli Mallett wrote: >>>> >>>>> To drain the pipeline on certain deficient (and mostly older) CPUs by way of >>>>> guesswork and a little vague magic. Most CPUs we support, I would guess, do >>>>> not need this, and it continues to exist solely for hysterical reasons. >>>> >>>> How can I turn it off for my compiles? >>>> >>>>> I've certainly gotten rid of them and some other cargo cult synchronization >>>>> on Octeon for testing and had it survive under considerable load, and >>>>> occasionally with some slight speedups (for some more commonly-used or >>>>> slower things than Just a Bunch Of NOPs.) >>>> >>>> Right. Well, since it's happening on every inlined lock, it's a bit silly. >>>> >>>>> The trouble is that proving they aren't necessary requires being rigorous >>>>> and careful in understanding documentation and errata, and FUD about their >>>>> possible necessity is somewhat-intimidating. It's not an easy kind of >>>>> corruption/unreliability/etc., to prove the lack of empirically. >>>> >>>> I've checked the diassembly from gcc-4.mumble on linux; it doesn't >>>> include NOPs like this as far as I can tell. >>>> >>> >>> The sync + 8 nops is coming from the definition of mips_sync() in >>> sys/mips/include/atomic.h. >>> >>> I agree with Juli that it appears to be a manual pipeline-flush >>> holdover from earlier days - I'm guessing there's 8 nops because the >>> R4000/4400 had both the sync instruction and an 8-stage pipeline. I'm >>> further guessing this was an attempt at providing stronger ordering >>> semantics than the sync instruction itself for the following >>> mb()/wmb()/rmb() definitions that use it, as the sync instruction >>> definition doesn't restrict execution of the before/after loads/stores >>> with respect to the sync instruction itself. >> >> Forgot to emphasize that this particular bit of old-school >> nop-counting is either pointless or a latent hazard - 8 does not cover >> the deepest MIPS pipeline around, then there's superscalar issue to >> consider - so I think it's either unnecessary or insufficient. So >> far, that's all criticism and no solution :/ >> _______________________________________________ >> freebsd-arch@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-arch >> To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > From owner-freebsd-mips@FreeBSD.ORG Thu Jun 6 11:57:37 2013 Return-Path: Delivered-To: mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id C1546409; Thu, 6 Jun 2013 11:57:37 +0000 (UTC) (envelope-from tinderbox@freebsd.org) Received: from freebsd-current.sentex.ca (freebsd-current.sentex.ca [64.7.128.98]) by mx1.freebsd.org (Postfix) with ESMTP id 8E1BE1BE0; Thu, 6 Jun 2013 11:57:37 +0000 (UTC) Received: from freebsd-current.sentex.ca (localhost [127.0.0.1]) by freebsd-current.sentex.ca (8.14.5/8.14.5) with ESMTP id r56BvalM047022; Thu, 6 Jun 2013 07:57:36 -0400 (EDT) (envelope-from tinderbox@freebsd.org) Received: (from tinderbox@localhost) by freebsd-current.sentex.ca (8.14.5/8.14.5/Submit) id r56BvaMv047018; Thu, 6 Jun 2013 11:57:36 GMT (envelope-from tinderbox@freebsd.org) Date: Thu, 6 Jun 2013 11:57:36 GMT Message-Id: <201306061157.r56BvaMv047018@freebsd-current.sentex.ca> X-Authentication-Warning: freebsd-current.sentex.ca: tinderbox set sender to FreeBSD Tinderbox using -f Sender: FreeBSD Tinderbox From: FreeBSD Tinderbox To: FreeBSD Tinderbox , , Subject: [head tinderbox] failure on mips/mips Precedence: bulk X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 Jun 2013 11:57:37 -0000 TB --- 2013-06-06 11:02:04 - tinderbox 2.10 running on freebsd-current.sentex.ca TB --- 2013-06-06 11:02:04 - FreeBSD freebsd-current.sentex.ca 8.3-PRERELEASE FreeBSD 8.3-PRERELEASE #0: Mon Mar 26 13:54:12 EDT 2012 des@freebsd-current.sentex.ca:/usr/obj/usr/src/sys/GENERIC amd64 TB --- 2013-06-06 11:02:04 - starting HEAD tinderbox run for mips/mips TB --- 2013-06-06 11:02:04 - cleaning the object tree TB --- 2013-06-06 11:02:04 - /usr/local/bin/svn stat /src TB --- 2013-06-06 11:02:08 - At svn revision 251452 TB --- 2013-06-06 11:02:09 - building world TB --- 2013-06-06 11:02:09 - CROSS_BUILD_TESTING=YES TB --- 2013-06-06 11:02:09 - MAKEOBJDIRPREFIX=/obj TB --- 2013-06-06 11:02:09 - PATH=/usr/bin:/usr/sbin:/bin:/sbin TB --- 2013-06-06 11:02:09 - SRCCONF=/dev/null TB --- 2013-06-06 11:02:09 - TARGET=mips TB --- 2013-06-06 11:02:09 - TARGET_ARCH=mips TB --- 2013-06-06 11:02:09 - TZ=UTC TB --- 2013-06-06 11:02:09 - __MAKE_CONF=/dev/null TB --- 2013-06-06 11:02:09 - cd /src TB --- 2013-06-06 11:02:09 - /usr/bin/make -B buildworld >>> Building an up-to-date make(1) >>> World build started on Thu Jun 6 11:02:15 UTC 2013 >>> Rebuilding the temporary build tree >>> stage 1.1: legacy release compatibility shims >>> stage 1.2: bootstrap tools >>> stage 2.1: cleaning up the object tree >>> stage 2.2: rebuilding the object tree >>> stage 2.3: build tools >>> stage 3: cross tools >>> stage 4.1: building includes >>> stage 4.2: building libraries >>> stage 4.3: make dependencies >>> stage 4.4: building everything [...] *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop. make: stopped in /src *** Error code 1 Stop in /src. TB --- 2013-06-06 11:57:36 - WARNING: /usr/bin/make returned exit code 1 TB --- 2013-06-06 11:57:36 - ERROR: failed to build world TB --- 2013-06-06 11:57:36 - 2375.20 user 535.11 system 3331.77 real http://tinderbox.freebsd.org/tinderbox-head-build-HEAD-mips-mips.full From owner-freebsd-mips@FreeBSD.ORG Sat Jun 8 21:57:52 2013 Return-Path: Delivered-To: freebsd-mips@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 71047D0 for ; Sat, 8 Jun 2013 21:57:52 +0000 (UTC) (envelope-from adrian.chadd@gmail.com) Received: from mail-qa0-x230.google.com (mail-qa0-x230.google.com [IPv6:2607:f8b0:400d:c00::230]) by mx1.freebsd.org (Postfix) with ESMTP id 371621682 for ; Sat, 8 Jun 2013 21:57:52 +0000 (UTC) Received: by mail-qa0-f48.google.com with SMTP id cm16so1696448qab.14 for ; Sat, 08 Jun 2013 14:57:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type; bh=k1p7X9rbNQ17kF3dS/2fHE6k/PyjDcLbV2wJ5vN21k0=; b=EX7aswfYYvHeIn7pxqXxIvAoXZITLv377lYMoTL5wqmEflqMQJMC6fJT9Jt9jxS5xV iuWd3fv+Dyj405fgnVPa4IEd9aVDlyB9DEyfMBiigUNKr6JS5wHsoWQJ+GaFBgDUYrWJ U86CQnDN5h+drdIfZdu46ni7n/WZxhAQ87+imMpQNvMxtv3Gn4p0xvSXQjz8U9yZB/47 Nr2r0qHR8vS61ivX9wGHOgEYWQky1QtjfypB+UI1yAf4ZWxG8tj6/L40Zg6qKGjrQlQW 9UW256PPGJT2G5Im7ExZQpO8iZuwQZzdy4PcEyZu7yPCRhrojNEczgdFNBcVZUtUa3kY 2Cwg== MIME-Version: 1.0 X-Received: by 10.224.59.200 with SMTP id m8mr8192095qah.43.1370728671648; Sat, 08 Jun 2013 14:57:51 -0700 (PDT) Sender: adrian.chadd@gmail.com Received: by 10.224.5.65 with HTTP; Sat, 8 Jun 2013 14:57:51 -0700 (PDT) In-Reply-To: References: <201306081319.r58DJBS0061186@svn.freebsd.org> Date: Sat, 8 Jun 2013 14:57:51 -0700 X-Google-Sender-Auth: GCnmhDsRSKRb9yQHC5tBFL_BYD8 Message-ID: Subject: Re: svn commit: r251524 - in head/sys: conf mips/mips From: Adrian Chadd To: Ed Schouten Content-Type: text/plain; charset=ISO-8859-1 Cc: freebsd-mips@freebsd.org X-BeenThere: freebsd-mips@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Porting FreeBSD to MIPS List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 08 Jun 2013 21:57:52 -0000 Hm, On 8 June 2013 10:35, Ed Schouten wrote: > Hello Adrian, > > 2013/6/8 Adrian Chadd : >> .. have we decided whether to nuke the NOP flush from orbit? > > I thought we didn't make a clear decision on that yet, so I decided to > leave it as it is for now. I'll make sure the sync routine in this > file remains in sync with the one in . Can we just commit bsdimp's change and have your macro include his "extra sync ops" macro, in case that running system requires it? Adrian