From owner-freebsd-drivers@freebsd.org Sun Sep 6 13:54:03 2015 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 9134B9CC70E for ; Sun, 6 Sep 2015 13:54:03 +0000 (UTC) (envelope-from leonardofogel@yahoo.com.br) Received: from nm18-vm4.bullet.mail.ne1.yahoo.com (nm18-vm4.bullet.mail.ne1.yahoo.com [98.138.91.178]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 542D0D20 for ; Sun, 6 Sep 2015 13:54:02 +0000 (UTC) (envelope-from leonardofogel@yahoo.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.br; s=s2048; t=1441547636; bh=hVaKTLHPssS6m+8TfOc3+fdRWi82HxiM9qQaCQ+r+q8=; h=Date:From:Subject:To:In-Reply-To:From:Subject; b=FaF5AG0Hl275oSnto7sBqhCe50kC2xGSy7uHx+p0ABCTiF8xA6/PL9ezk52WRQxx+4MARkGSZPruyz8lgkif/+x+xaAXKxagjHcfuSToXuTFQ4DUkUBEKfgFDxX6jnSNRTRUI8qGXEY/UecbCQf5mBkc/CpKbl6p5WAQQ+35SGAFpG/xeG/HWwHguCf8Nmp9JpqQqhp1Dr0U29D93YlxHX6tCdbeVYv3zm2pldDivazVpandKZ79eslKtdBiVzRZ6R1j1N9AcjrMy9KxqrOhu9DVcFMB2EjEL/I4WRIs7Z4+AeGZvdLgImOOcNAJVrQ3HvaLXJ/DcYyMwrp+5Yc9ww== Received: from [98.138.100.102] by nm18.bullet.mail.ne1.yahoo.com with NNFMP; 06 Sep 2015 13:53:56 -0000 Received: from [98.138.89.245] by tm101.bullet.mail.ne1.yahoo.com with NNFMP; 06 Sep 2015 13:53:56 -0000 Received: from [127.0.0.1] by omp1059.mail.ne1.yahoo.com with NNFMP; 06 Sep 2015 13:53:56 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 765148.14349.bm@omp1059.mail.ne1.yahoo.com Received: (qmail 87774 invoked by uid 60001); 6 Sep 2015 13:53:56 -0000 X-YMail-OSG: zOfNEQkVM1m9qG5rBQ9VKjfWC.6NWOH8gWa7YmQ_AlLeGSr KB7FSyUPyZmnMXSNw2NQm8U7LLZCV3TCj22Wb_jj_JpAl9WbaKLYmy_IFfpl 1OdqyxZl2xVK8iwnMa1N4ICfBExPYrXYSwuNtT1c0.dnTKMmZot6ZnL5sNv2 rdJ.wP3pM2mcEHVxWJl42y26MPeesBM8o3vfAFCigYowjbvoshLvPgP1u_P0 hV13xEJdBHwCE9Nd4BymNPU.SRl9h_6CZJPngb.f4nqIAKtoo_.5mbMd5zDR o75lgOdhKfmFIWE1fXpVUoGndKUQnT60nwVI2TptKLYY3Mbcawss3jGkcQW1 ntueVPTmAIn_escw4xi1RFchdMTGT5C_QN8yWD3ubHdwlu8YzJ.fEt.ch386 yCyFli2nwZnvIPNvDiraMSuTEdvhqli8g66h9B1idXWAHgDCcM5Dkqeh.WdD S7ZVenWIKJB.7RwFvMwFB5nJ1yLYg.jZyaHGupkyTHcixqe8D_1mlipwrBEI P_FBk5e5P8NwxKIbj6q5zRLMg7AO_jsehgFaUVJCVEvsLG88QTSsfNF1tC53 Og0MzOSKvVN5wsT0- Received: from [189.60.224.8] by web120801.mail.ne1.yahoo.com via HTTP; Sun, 06 Sep 2015 06:53:56 PDT X-Rocket-MIMEInfo: 002.001, SGkuDQpQbGVhc2UsIHdoYXQgYXJlIHRoZSBjb3JyZWN0IGJhcnJpZXJzIGZvciB0aGUgZm9sbG93aW5nIGNhc2VzPw0KDQpDYXNlIDE6DQogICBidXNfd3JpdGVfMShyZWdpb25fMCwgLi4uKTsNCiAgIC8qIGJhcnJpZXIgaGVyZSAqLw0KICAgREVMQVkoc29tZV90aW1lKTsNCg0KQ2FzZSAyOg0KICAgYnVzX3dyaXRlXzEocmVnaW9uXzAsIC4uLik7DQogICAvKiBiYXJyaWVyIGhlcmUgKi8NCiAgIGJ1c193cml0ZV8xKHJlZ2lvbl8yLCAuLi4pOw0KDQpJbiB0aGUgZmlyc3Qgb25lLCBJIHdhbnQgdGhlIHdyaXQBMAEBAQE- X-Mailer: YahooMailBasic/651 YahooMailWebService/0.8.203.813 Message-ID: <1441547636.78646.YahooMailBasic@web120801.mail.ne1.yahoo.com> Date: Sun, 6 Sep 2015 06:53:56 -0700 From: Leonardo Fogel Subject: Memory barrier To: freebsd-drivers@freebsd.org In-Reply-To: <2149458.2Hg3JbBXY3@ralph.baldwin.cx> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 13:54:03 -0000 Hi. Please, what are the correct barriers for the following cases? Case 1: bus_write_1(region_0, ...); /* barrier here */ DELAY(some_time); Case 2: bus_write_1(region_0, ...); /* barrier here */ bus_write_1(region_2, ...); In the first one, I want the write to reach the device before the thread bu= sy-waits. As I understand it, bus_space_barrier(9) is not adequate, because= it does not prevent the processor from executing instructions (or load/sto= re to RAM) before the write completes. In the second one, I want the write to a device (e.g. power management) to = complete before the write to another starts/completes. Again, bus_space_bar= rier() seems not to be adequate because it can not cover two regions. Thank you for your time. Leonardo From owner-freebsd-drivers@freebsd.org Sun Sep 6 18:03:16 2015 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id E75839CC1CC for ; Sun, 6 Sep 2015 18:03:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from kib.kiev.ua (kib.kiev.ua [IPv6:2001:470:d5e7:1::1]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 5B037792 for ; Sun, 6 Sep 2015 18:03:16 +0000 (UTC) (envelope-from kostikbel@gmail.com) Received: from tom.home (kostik@localhost [127.0.0.1]) by kib.kiev.ua (8.15.2/8.15.2) with ESMTPS id t86I3B8v006620 (version=TLSv1 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Sun, 6 Sep 2015 21:03:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) DKIM-Filter: OpenDKIM Filter v2.10.3 kib.kiev.ua t86I3B8v006620 Received: (from kostik@localhost) by tom.home (8.15.2/8.15.2/Submit) id t86I3BUI006619; Sun, 6 Sep 2015 21:03:11 +0300 (EEST) (envelope-from kostikbel@gmail.com) X-Authentication-Warning: tom.home: kostik set sender to kostikbel@gmail.com using -f Date: Sun, 6 Sep 2015 21:03:11 +0300 From: Konstantin Belousov To: Leonardo Fogel Cc: freebsd-drivers@freebsd.org Subject: Re: Memory barrier Message-ID: <20150906180311.GS2072@kib.kiev.ua> References: <2149458.2Hg3JbBXY3@ralph.baldwin.cx> <1441547636.78646.YahooMailBasic@web120801.mail.ne1.yahoo.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1441547636.78646.YahooMailBasic@web120801.mail.ne1.yahoo.com> User-Agent: Mutt/1.5.23 (2014-03-12) X-Spam-Status: No, score=-2.0 required=5.0 tests=ALL_TRUSTED,BAYES_00, DKIM_ADSP_CUSTOM_MED,FREEMAIL_FROM,NML_ADSP_CUSTOM_MED autolearn=no autolearn_force=no version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on tom.home X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Sep 2015 18:03:17 -0000 On Sun, Sep 06, 2015 at 06:53:56AM -0700, Leonardo Fogel wrote: > Hi. > Please, what are the correct barriers for the following cases? > > Case 1: > bus_write_1(region_0, ...); > /* barrier here */ > DELAY(some_time); > > Case 2: > bus_write_1(region_0, ...); > /* barrier here */ > bus_write_1(region_2, ...); > > In the first one, I want the write to reach the device before the thread busy-waits. As I understand it, bus_space_barrier(9) is not adequate, because it does not prevent the processor from executing instructions (or load/store to RAM) before the write completes. > > In the second one, I want the write to a device (e.g. power management) to complete before the write to another starts/completes. Again, bus_space_barrier() seems not to be adequate because it can not cover two regions. I believe that the bus_write semantic includes the required serialization. E.g., on x86 all CPU write buffers are flushed before the write instruction is declared completed, because this is the semantic of the uncacheable memory. For powerpc, the system automatically inserts powerpc_iomb() after the write, which is full sync. I am not aware of other architectures. From owner-freebsd-drivers@freebsd.org Mon Sep 7 19:55:34 2015 Return-Path: Delivered-To: freebsd-drivers@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 64DE59CDB65 for ; Mon, 7 Sep 2015 19:55:34 +0000 (UTC) (envelope-from leonardofogel@yahoo.com.br) Received: from nm4-vm6.bullet.mail.ne1.yahoo.com (nm4-vm6.bullet.mail.ne1.yahoo.com [98.138.91.97]) (using TLSv1 with cipher ECDHE-RSA-RC4-SHA (128/128 bits)) (Client did not present a certificate) by mx1.freebsd.org (Postfix) with ESMTPS id 2FBA512B4 for ; Mon, 7 Sep 2015 19:55:33 +0000 (UTC) (envelope-from leonardofogel@yahoo.com.br) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com.br; s=s2048; t=1441655727; bh=5iV4XDLzxY9nfwtGGiiNcBIRtZRH1Reqg2GbJCNn5po=; h=Date:From:Subject:To:Cc:In-Reply-To:From:Subject; b=q6bpmX6DRwI/xg50Qcah+uex1bEafEwVz6BFE2YHBe1033+rhfKk2itJ+Evrpcbg/sjhNqQolcgpncgLZcVr8uBuh5tpuXnb6AnF5uVt5wrneUzePIEWvb/aTyw21PaK/fKzgiYi6LJ/XDyoVmbnj44cH7gpWhqB1cwPlO5VBj5hcqq7hS1+Uog4MzzMqv40N3dhr4G40myQFQn/Ilwl0jcQWaTarJUXmFogqXoXpLaM4MeOQUZph4z6M1GKLfPOJ251qfsuVFJw4whiywL940BBuXgpeHr39eDJ9ffNvkuay02czXVcxUJVlHn37YAlUmJMrYdAho7VfVZII2C+tg== Received: from [98.138.100.112] by nm4.bullet.mail.ne1.yahoo.com with NNFMP; 07 Sep 2015 19:55:27 -0000 Received: from [98.138.87.5] by tm103.bullet.mail.ne1.yahoo.com with NNFMP; 07 Sep 2015 19:55:27 -0000 Received: from [127.0.0.1] by omp1005.mail.ne1.yahoo.com with NNFMP; 07 Sep 2015 19:55:27 -0000 X-Yahoo-Newman-Property: ymail-3 X-Yahoo-Newman-Id: 483469.1119.bm@omp1005.mail.ne1.yahoo.com Received: (qmail 76921 invoked by uid 60001); 7 Sep 2015 19:55:27 -0000 X-YMail-OSG: oqATtNAVM1lU0pejSQ978aNuHWoEA9BBCdDmywl.vo2rQlP zcUbD_g6ugMGZzHYPxc.AgjUvsc8sZxhWWVolpbZCEkbRo8HOsIjAn.UEQAr aAEqobvKmtdTzJicrgY24E.MUSCi0FRysvTINvWS.GTTODDZIkYOjtyqcRDp PJ0duF03rJ06lckLI7oFC940oFChl.lEDXE._a5e.SOFlRjYp2g3oVRT6k_d PYXy.D4sfjS8ZCRJVjf4Zbv8.yEwZHghexkt4O3T63lsOlcnIcF98byT9e9v iLI0wPWJHMMv_qY_YmMioecaebxDqhJoO0FcRDCnaX7reg8QzxRyUJZc7uT0 9C7p.8l8wCvw5IycDTBjkA0jzHKrIFZxkFg3czBhzrk6fsNItgNgeJWSFst_ BCiV.nEqwE3Fi7Th2n6FsQZUTb3bOAHEbdXQMtMqSXGFi0NzLdZAGHFtI7eT PJzDcc7wLG8k7_NMuL71dPcKgR4J5Y1JxeI3Krqe3UucLAQfeKjTxYe86Sv4 8mM0fhckxPF6ti.efwvJK6TkYRbWXp5ekgN6G38He9K7N6M5qNCCpadVrYpH Ylih9UCz9BIhhuaSAYg-- Received: from [189.60.224.236] by web120802.mail.ne1.yahoo.com via HTTP; Mon, 07 Sep 2015 12:55:27 PDT X-Rocket-MIMEInfo: 002.001, PiA.IENhc2UgMToNCj4gPiAgICBidXNfd3JpdGVfMShyZWdpb25fMCwgLi4uKTsNCj4gPiAgICAvKiBiYXJyaWVyIGhlcmUgKi8NCj4gPiAgICBERUxBWShzb21lX3RpbWUpOw0KPiA.DQo.ID4gQ2FzZSAyOg0KPiA.ICAgIGJ1c193cml0ZV8xKHJlZ2lvbl8wLCAuLi4pOw0KPiA.ICAgIC8qIGJhcnJpZXIgaGVyZSAqLw0KPiA.ICAgIGJ1c193cml0ZV8xKHJlZ2lvbl8yLCAuLi4pOw0KPiA.DQo.ID4gSW4gdGhlIGZpcnN0IG9uZSwgSSB3YW50IHRoZSB3cml0ZSB0byByZWFjaCB0aGUgZGV2aWNlIGJlZm9yZSABMAEBAQE- X-Mailer: YahooMailBasic/651 YahooMailWebService/0.8.203.813 Message-ID: <1441655727.36257.YahooMailBasic@web120802.mail.ne1.yahoo.com> Date: Mon, 7 Sep 2015 12:55:27 -0700 From: Leonardo Fogel Subject: Re: Memory barrier To: freebsd-drivers@freebsd.org In-Reply-To: <20150906180311.GS2072@kib.kiev.ua> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-BeenThere: freebsd-drivers@freebsd.org X-Mailman-Version: 2.1.20 Precedence: list List-Id: Writing device drivers for FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Sep 2015 19:55:34 -0000 > > Case 1: > > bus_write_1(region_0, ...); > > /* barrier here */ > > DELAY(some_time); > > > > Case 2: > > bus_write_1(region_0, ...); > > /* barrier here */ > > bus_write_1(region_2, ...); > > > > In the first one, I want the write to reach the device before the threa= d busy-waits. > > > > In the second one, I want the write to a device (e.g. power management)= to > > complete before the write to another starts/completes. >=20 > I believe that the bus_write semantic includes the required serialization= . > E.g., on x86 all CPU write buffers are flushed before the write instructi= on > is declared completed, because this is the semantic of the uncacheable > memory. For powerpc, the system automatically inserts powerpc_iomb() aft= er > the write, which is full sync. I am not aware of other architectures. I've found the implementation of the bus_space_barrier for the ARM architec= ture (the one in which I'm interested): generic_bs_barrier(bus_space_tag_t t, bus_space_handle_t bsh, bus_size_t= offset, bus_size_t len, int flags) { /* * dsb() will drain the L1 write buffer and establish a memory a= ccess * barrier point on platforms where that has meaning. On a writ= e we * also need to drain the L2 write buffer, because most on-chip = memory * mapped devices are downstream of the L2 cache. Note that thi= s needs * to be done even for memory mapped as Device type, because whi= le * Device memory is not cached, writes to it are still buffered. */ dsb(); if (flags & BUS_SPACE_BARRIER_WRITE) { cpu_l2cache_drain_writebuf(); } } The ARM architecture specifies two _data_ barrier instructions: DMB and DSB= . The first synchronizes memory accesses, and the second synchronizes both = memory accesses and instruction execution. So, DSB is the answer to Case 1,= and DMB or DSB is the answer to Case 2. The implementation above brings something of which I was not aware: it also= drains the L2 write buffer. Older implementations of the "PL310 Store Buff= er did not have any automatic draining mechanism." (ARM CoreLink Level 2 Ca= che Controller (L2C-310 or PL310), r3 releases, Software Developers Errata = Notice.) In newer implementations, the writes to device memory are "Put in = store buffer, not merged, immediately drained to L3." (CoreLink Level 2 Cac= he Controller L2C-310 Technical Reference Manual=09Revision: r3p3.) Leonardo