From owner-freebsd-bugs@FreeBSD.ORG Tue Feb 28 02:40:17 2006 Return-Path: X-Original-To: freebsd-bugs@hub.freebsd.org Delivered-To: freebsd-bugs@hub.freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A643A16A420 for ; Tue, 28 Feb 2006 02:40:17 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [216.136.204.21]) by mx1.FreeBSD.org (Postfix) with ESMTP id E99F043D6A for ; Tue, 28 Feb 2006 02:40:11 +0000 (GMT) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (gnats@localhost [127.0.0.1]) by freefall.freebsd.org (8.13.4/8.13.4) with ESMTP id k1S2eBcI019579 for ; Tue, 28 Feb 2006 02:40:11 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.13.4/8.13.4/Submit) id k1S2eBxd019578; Tue, 28 Feb 2006 02:40:11 GMT (envelope-from gnats) Date: Tue, 28 Feb 2006 02:40:11 GMT Message-Id: <200602280240.k1S2eBxd019578@freefall.freebsd.org> To: freebsd-bugs@FreeBSD.org From: Arthur Hartwig Cc: Subject: Re: misc/93887: cpu_spinwait calls missing X-BeenThere: freebsd-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Arthur Hartwig List-Id: Bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 28 Feb 2006 02:40:17 -0000 The following reply was made to PR kern/93887; it has been noted by GNATS. From: Arthur Hartwig To: ext Kris Kennaway Cc: freebsd-gnats-submit@FreeBSD.org Subject: Re: misc/93887: cpu_spinwait calls missing Date: Tue, 28 Feb 2006 12:32:08 +1000 This is a multi-part message in MIME format. --------------090009090003010006010508 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit ext Kris Kennaway wrote: >On Mon, Feb 27, 2006 at 06:42:08AM +0000, Arthur Hartwig wrote: > > >>>Number: 93887 >>>Category: misc >>>Synopsis: cpu_spinwait calls missing >>>Confidential: no >>>Severity: non-critical >>>Priority: low >>>Responsible: freebsd-bugs >>>State: open >>>Quarter: >>>Keywords: >>>Date-Required: >>>Class: sw-bug >>>Submitter-Id: current-users >>>Arrival-Date: Mon Feb 27 06:50:04 GMT 2006 >>>Closed-Date: >>>Last-Modified: >>>Originator: Arthur Hartwig >>>Release: 6.1-BETA1 >>>Organization: >>> >>> >>Nokia >> >> >>>Environment: >>> >>> >>oz-net-11# uname -a >>FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006 root@oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11 amd64 >> >> >> >> >>>Description: >>> >>> >>There are five busy wait loops in kern/subr_smp.c which would be more "multi-processor friendly" on i386 and amd64 architectures if they included a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes a pause instruction which the Intel IA32 architecture manual says "improves the performance of spin wait loops." >> >>Loops which could include the call to cpu_spinwait() occur in stop_cpus(), stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action() >> >> > >Why would you want to do this in stop_cpus(), stop_cpus_nmi() and >restart_cpus()? They're used e.g. as preparation to break into/out of >the debugger, and AFAICT they're not in the critical path for >anything. > >Not sure about smp_rendezvous_action(), maybe that one is OK. > >Kris > Good question which didn't occur to me because I was coming from another context. I was looking at using these functions to support a particular piece of "hotswap" hardware. The hardware generates an interrupt to say a PCI device has been removed (or plugged in). In the removal case some work needs to be done to remap to main memory the memory address space assigned to the device being removed so that if a device driver is accessing the device's memory space it won't cause the system to die. On a UP system this is "easy" in that because the hotswap interrupt is running we know the driver for the removed device is not currently running and even if it was running before the interrupt by the time it runs again the memory remapping will have occurred and it will be safe for the driver to access the memory space previously assigned to the removed device.However on a MP system the driver could be executing concurrently on another CPU and cause the system to die by accessing memory that is no longer present. I thought of using these functions (stop_cpus() etc) to momentarily "pause" all other CPUs while adjusting the memory mapping. It would be of advantage to cause other CPUs to pause as quickly as possible to minimise the likelihood that the driver for the removed device will attempt to access memory on the removed device. I have not been able to find any data on how long I have between the interrupt to say a device is being removed and the actual removal of the device. Nor have I any data on how long a busy CPU of a hyperthreaded pair (for example) is able to "lock out" its hyperthreaded mate. The referenced functions are clearly adequate for their existing use by the debugger so I accept that I might be better off making a customised copy. I still have doubts about the effectiveness of the approach using stop_cpus() and co and I'm inclined to see if I can get away with saying "you can't pull out a board untill you have disabled all the devices on that board" but this is somewhat restricted compared with what users have become accustomed to. Arthur --------------090009090003010006010508 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit ext Kris Kennaway wrote: 
On Mon, Feb 27, 2006 at 06:42:08AM +0000, Arthur Hartwig wrote:
   
Number:         93887
 Category:       misc
 Synopsis:       cpu_spinwait calls missing
 Confidential:   no
 Severity:       non-critical
 Priority:       low
 Responsible:    freebsd-bugs
 State:          open
 Quarter:        
 Keywords:       
 Date-Required:
 Class:          sw-bug
 Submitter-Id:   current-users
 Arrival-Date:   Mon Feb 27 06:50:04 GMT 2006
 Closed-Date:
 Last-Modified:
 Originator:     Arthur Hartwig
 Release:        6.1-BETA1
 Organization:
       
Nokia
     
Environment:
       
oz-net-11# uname -a
 FreeBSD oz-net-11.nes.nokia.com 6.1-BETA1 FreeBSD 6.1-BETA1 #4: Tue Feb 21 14:01:31 EST 2006     root@oz-net-11.nes.nokia.com:/usr/src/sys/amd64/compile/oz-net-11  amd64
 
 
     
Description:
       
There are five busy wait loops in kern/subr_smp.c which would be more "multi-processor friendly" on i386 and amd64 architectures if they included a call to cpu_spinwait(). On these architectures the cpu_spinwait() executes a pause instruction which the Intel IA32 architecture manual says "improves the performance of spin wait loops."
 
 Loops which could include the call to cpu_spinwait() occur in stop_cpus(), stop_cpus_nmi(), restart_cpus(), and two in smp_rendezvous_action()
     

 Why would you want to do this in stop_cpus(), stop_cpus_nmi() and
 restart_cpus()?  They're used e.g. as preparation to break into/out of
 the debugger, and AFAICT they're not in the critical path for
 anything.
 
 Not sure about smp_rendezvous_action(), maybe that one is OK.
 
 Kris
Good question which didn't occur to me because I was coming from another context. I was looking at using these functions to support a particular piece of "hotswap" hardware. The hardware generates an interrupt to say a PCI device has been removed (or plugged in). In the removal case some work needs to be done to remap to main memory the memory address space assigned to the device being removed so that if a device driver is accessing the device's memory space it won't cause the system to die. On a UP system this is "easy" in that because the hotswap interrupt is running we know the driver for the removed device is not currently running and even if it was running before the interrupt by the time it runs again the memory remapping will have occurred and it will be safe for the driver to access the memory space previously assigned to the removed device.However on a MP system the driver could be executing concurrently on another CPU and cause the system to die by accessing memory that is no longer present.

I thought of using these functions (stop_cpus() etc) to momentarily "pause" all other CPUs while adjusting the memory mapping. It would be of advantage to cause other CPUs to pause as quickly as possible to minimise the likelihood that the driver for the removed device will attempt to access memory on the removed device. I have not been able to find any data on how long I have between the interrupt to say a device is being removed and the actual removal of the device. Nor have I any data on how long a busy CPU of a hyperthreaded pair (for example) is able to "lock out" its hyperthreaded mate.

The referenced functions are clearly adequate for their existing use by the debugger so I accept that I might be better off making a customised copy.

I still have doubts about the effectiveness of the approach using stop_cpus() and co and I'm inclined to see if I can get away with saying "you can't pull out a board untill you have disabled all the devices on that board" but this is somewhat restricted compared with what users have become accustomed to.

Arthur

--------------090009090003010006010508--