From owner-freebsd-hackers@FreeBSD.ORG Sat Jun 4 03:27:04 2005 Return-Path: X-Original-To: freebsd-hackers@freebsd.org Delivered-To: freebsd-hackers@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0107316A41C for ; Sat, 4 Jun 2005 03:27:04 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from pooker.samsco.org (pooker.samsco.org [168.103.85.57]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7335B43D48 for ; Sat, 4 Jun 2005 03:27:03 +0000 (GMT) (envelope-from scottl@samsco.org) Received: from [192.168.254.11] (junior.samsco.home [192.168.254.11]) (authenticated bits=0) by pooker.samsco.org (8.13.3/8.13.3) with ESMTP id j543XWEF037152; Fri, 3 Jun 2005 21:33:32 -0600 (MDT) (envelope-from scottl@samsco.org) Message-ID: <42A11F4A.40502@samsco.org> Date: Fri, 03 Jun 2005 21:26:02 -0600 From: Scott Long User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.7.5) Gecko/20050218 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Matthew Dillon References: <200506032057.j53KvOFw062012@apollo.backplane.com> <20050604021812.GG594@funkthat.com> <200506040257.j542veCm063487@apollo.backplane.com> In-Reply-To: <200506040257.j542veCm063487@apollo.backplane.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Status: No, score=-2.8 required=3.8 tests=ALL_TRUSTED autolearn=failed version=3.0.2 X-Spam-Checker-Version: SpamAssassin 3.0.2 (2004-11-16) on pooker.samsco.org Cc: freebsd-hackers@freebsd.org, John-Mark Gurney Subject: Re: Possible instruction pipelining problem between HT's on the same die ? X-BeenThere: freebsd-hackers@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Technical Discussions relating to FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 04 Jun 2005 03:27:04 -0000 Matthew Dillon wrote: [...] > It seems so unlikely that this could occur across physical cpus that > I was not surprised at all by this. But 16 instructions seemed unlikely > to me. The only scenario I can come up with is that the READ SIDE on > the HT cpu (logical cpu #1) did a speculative read of B before logical > cpu #0 wrote to it, then somehow held that speculative read for 16 > whole instructions on logical cpu #1. > > Is that even possible ? holding speculative read data across > 16 instructions ? Yes > > The only other possibility is that there are major interactions in the > instruction pipeline and cpu #1 is reading e.g. the index B from the > pipeline or write buffer and data A from memory prior to data A being > retired to memory by cpu #0. That seems ridiculous to me, but I > wonder if it's possible without an SFENCE. > > This crash occurs fairly rarely. It takes a lot of packets for it to > occur... perhaps a million or more. > > In anycase, we are now testing a kernel with a locked bus cycle inbetwen > the READ B and the READ A to see if that fixes the problem. If that > doesn't work I will put an SFENCE between the WRITE A and the WRITE B. > And if that doesn't work then I'm shooting up the wrong alley and it > isn't an instruction/memory ordering issue. I would expect that putting the fence on the write side will solve the problem. As Stephen discussed, the writes will land in a store buffer for a period of time, during which a fence on the write CPU will flush it out and make it visible to the other CPUs. Doing a fence on the read CPU will have no effect on the store buffers of the write CPU and will be a waste of time. Another thing to keep in mind is that there is no difference here between HT and non HT SMP protocol. While HT cores share execution units, they DO NOT share registers, store buffers, or cache (at least, not in a way that is visible outside of the low-level implementation of the chip). Scott