From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 02:44:17 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E733A16A4CE for ; Sun, 3 Oct 2004 02:44:17 +0000 (GMT) Received: from ylpvm29.prodigy.net (ylpvm29-ext.prodigy.net [207.115.57.60]) by mx1.FreeBSD.org (Postfix) with ESMTP id DAB2243D1D for ; Sun, 3 Oct 2004 02:44:15 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (adsl-67-124-49-205.dsl.snfc21.pacbell.net [67.124.49.205])i932hsIV026274; Sat, 2 Oct 2004 22:43:57 -0400 Message-ID: <415F677B.5030108@elischer.org> Date: Sat, 02 Oct 2004 19:44:11 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4b) Gecko/20030524 X-Accept-Language: en, hu MIME-Version: 1.0 To: Stephan Uphoff References: <1096489576.3733.1868.camel@palm.tree.com> <200409291652.29990.jhb@FreeBSD.org> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <20041002053351.GA6259@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <1096760257.34527.14.camel@palm.tree.com> In-Reply-To: <1096760257.34527.14.camel@palm.tree.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 02:44:18 -0000 Stephan Uphoff wrote: > On Sat, 2004-10-02 at 14:31, Peter Holm wrote: > >>OK, right now I'm testing with all of Stephan's patches + the >>MUTEX_WAKE_ALL flag. Uptime is 3 3/4 hour and looking good. I've just resurfaced after a week of hell at work and home not BAD hell but more like "busy as hell". I'm just integrating your fixes into my tree to understand what they do.. expect more mail from me later. > > > Great. > > Your attached diff contained all the fixes needed and I don't see the > need to post a cumulative patch. > > The only thing left to do is migrate a critical sections from > kern_mutex.c to subr_turnstile.c for readability. > (no functional changes) > > Maybe it would also better to just force MUTEX_WAKE_ALL in > kern_mutex.c (#ifndef MUTEX_WAKE_ALL \n#define MUTEX_WAKE_ALL\n#endif) > to avoid temporary configuration file pollution? > > Stephan > > From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 05:31:45 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 04E5216A4CF for ; Sun, 3 Oct 2004 05:31:45 +0000 (GMT) Received: from mail4.speakeasy.net (mail4.speakeasy.net [216.254.0.204]) by mx1.FreeBSD.org (Postfix) with ESMTP id B511443D3F for ; Sun, 3 Oct 2004 05:31:44 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: (qmail 22424 invoked from network); 3 Oct 2004 05:31:44 -0000 Received: from gate.funkthat.com (HELO hydrogen.funkthat.com) ([69.17.45.168]) (envelope-sender ) by mail4.speakeasy.net (qmail-ldap-1.03) with SMTP for ; 3 Oct 2004 05:31:44 -0000 Received: from hydrogen.funkthat.com (vqlwxk@localhost.funkthat.com [127.0.0.1])i935Vhlb035388 for ; Sat, 2 Oct 2004 22:31:43 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.12.10/8.12.10/Submit) id i935Vgx6035387 for freebsd-arch@freebsd.org; Sat, 2 Oct 2004 22:31:42 -0700 (PDT) Date: Sat, 2 Oct 2004 22:31:42 -0700 From: John-Mark Gurney To: freebsd-arch@freebsd.org Message-ID: <20041003053142.GI22681@funkthat.com> Mail-Followup-To: freebsd-arch@freebsd.org Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="ftEhullJWpWg/VHq" Content-Disposition: inline User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD 4.2-RELEASE i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html Subject: pci device table update... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 05:31:45 -0000 --ftEhullJWpWg/VHq Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Ok, I figured I might as well put my recently purchased PCI book to good use and update our device types for unrecognized devices... If you have any problems with the wording of the devices, let me know, otherwise it'll go into the tree in a few days... (with a respective copy to pciconf)... Thanks. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." --ftEhullJWpWg/VHq Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="pci.diff" Index: pci.c =================================================================== RCS file: /home/ncvs/src/sys/dev/pci/pci.c,v retrieving revision 1.265 diff -u -r1.265 pci.c --- pci.c 2004/09/23 22:58:43 1.265 +++ pci.c 2004/10/03 05:28:09 @@ -1150,12 +1150,15 @@ {PCIC_NETWORK, PCIS_NETWORK_TOKENRING, "token ring"}, {PCIC_NETWORK, PCIS_NETWORK_FDDI, "fddi"}, {PCIC_NETWORK, PCIS_NETWORK_ATM, "ATM"}, + {PCIC_NETWORK, PCIS_NETWORK_ISDN, "ISDN"}, {PCIC_DISPLAY, -1, "display"}, {PCIC_DISPLAY, PCIS_DISPLAY_VGA, "VGA"}, {PCIC_DISPLAY, PCIS_DISPLAY_XGA, "XGA"}, + {PCIC_DISPLAY, PCIS_DISPLAY_3D, "3D"}, {PCIC_MULTIMEDIA, -1, "multimedia"}, {PCIC_MULTIMEDIA, PCIS_MULTIMEDIA_VIDEO, "video"}, {PCIC_MULTIMEDIA, PCIS_MULTIMEDIA_AUDIO, "audio"}, + {PCIC_MULTIMEDIA, PCIS_MULTIMEDIA_TEL, "telephony"}, {PCIC_MEMORY, -1, "memory"}, {PCIC_MEMORY, PCIS_MEMORY_RAM, "RAM"}, {PCIC_MEMORY, PCIS_MEMORY_FLASH, "flash"}, @@ -1168,19 +1171,24 @@ {PCIC_BRIDGE, PCIS_BRIDGE_PCMCIA, "PCI-PCMCIA"}, {PCIC_BRIDGE, PCIS_BRIDGE_NUBUS, "PCI-NuBus"}, {PCIC_BRIDGE, PCIS_BRIDGE_CARDBUS, "PCI-CardBus"}, - {PCIC_BRIDGE, PCIS_BRIDGE_OTHER, "PCI-unknown"}, + {PCIC_BRIDGE, PCIS_BRIDGE_RACEWAY, "PCI-RACEway"}, {PCIC_SIMPLECOMM, -1, "simple comms"}, {PCIC_SIMPLECOMM, PCIS_SIMPLECOMM_UART, "UART"}, /* could detect 16550 */ {PCIC_SIMPLECOMM, PCIS_SIMPLECOMM_PAR, "parallel port"}, + {PCIC_SIMPLECOMM, PCIS_SIMPLECOMM_MULSER, "multiport serial"}, + {PCIC_SIMPLECOMM, PCIS_SIMPLECOMM_MODEM, "generic modem"}, {PCIC_BASEPERIPH, -1, "base peripheral"}, {PCIC_BASEPERIPH, PCIS_BASEPERIPH_PIC, "interrupt controller"}, {PCIC_BASEPERIPH, PCIS_BASEPERIPH_DMA, "DMA controller"}, {PCIC_BASEPERIPH, PCIS_BASEPERIPH_TIMER, "timer"}, {PCIC_BASEPERIPH, PCIS_BASEPERIPH_RTC, "realtime clock"}, + {PCIC_BASEPERIPH, PCIS_BASEPERIPH_PCIHOT, "PCI hot-plug controller"}, {PCIC_INPUTDEV, -1, "input device"}, {PCIC_INPUTDEV, PCIS_INPUTDEV_KEYBOARD, "keyboard"}, {PCIC_INPUTDEV, PCIS_INPUTDEV_DIGITIZER,"digitizer"}, {PCIC_INPUTDEV, PCIS_INPUTDEV_MOUSE, "mouse"}, + {PCIC_INPUTDEV, PCIS_INPUTDEV_SCANNER, "scanner"}, + {PCIC_INPUTDEV, PCIS_INPUTDEV_GAMEPORT, "gameport"}, {PCIC_DOCKING, -1, "docking station"}, {PCIC_PROCESSOR, -1, "processor"}, {PCIC_SERIALBUS, -1, "serial bus"}, @@ -1190,6 +1198,22 @@ {PCIC_SERIALBUS, PCIS_SERIALBUS_USB, "USB"}, {PCIC_SERIALBUS, PCIS_SERIALBUS_FC, "Fibre Channel"}, {PCIC_SERIALBUS, PCIS_SERIALBUS_SMBUS, "SMBus"}, + {PCIC_WIRELESS, -1, "wireless controller"}, + {PCIC_WIRELESS, PCIS_WIRELESS_IRDA, "iRDA"}, + {PCIC_WIRELESS, PCIS_WIRELESS_IR, "IR"}, + {PCIC_WIRELESS, PCIS_WIRELESS_RF, "RF"}, + {PCIC_INTELLIIO, -1, "intelligent I/O controller"}, + {PCIC_INTELLIIO, PCIS_INTELLIIO_I2O, "I2O"}, + {PCIC_SATCOM, -1, "satellite communication"}, + {PCIC_SATCOM, PCIS_SATCOM_TV, "sat TV"}, + {PCIC_SATCOM, PCIS_SATCOM_TV, "sat audio"}, + {PCIC_SATCOM, PCIS_SATCOM_TV, "sat voice"}, + {PCIC_SATCOM, PCIS_SATCOM_TV, "sat data"}, + {PCIC_CRYPTO, -1, "encrypt/decrypt"}, + {PCIC_CRYPTO, PCIS_CRYPTO_NETCOMP, "network/computer crypto"}, + {PCIC_CRYPTO, PCIS_CRYPTO_NETCOMP, "entertainment crypto"}, + {PCIC_DASP, -1, "dasp"}, + {PCIC_DASP, PCIS_DASP_DPIO, "DPIO module"}, {0, 0, NULL} }; Index: pcireg.h =================================================================== RCS file: /home/ncvs/src/sys/dev/pci/pcireg.h,v retrieving revision 1.39 diff -u -r1.39 pcireg.h --- pcireg.h 2003/09/14 19:30:00 1.39 +++ pcireg.h 2004/10/03 05:28:09 @@ -207,16 +207,19 @@ #define PCIS_NETWORK_TOKENRING 0x01 #define PCIS_NETWORK_FDDI 0x02 #define PCIS_NETWORK_ATM 0x03 +#define PCIS_NETWORK_ISDN 0x04 #define PCIS_NETWORK_OTHER 0x80 #define PCIC_DISPLAY 0x03 #define PCIS_DISPLAY_VGA 0x00 #define PCIS_DISPLAY_XGA 0x01 +#define PCIS_DISPLAY_3D 0x02 #define PCIS_DISPLAY_OTHER 0x80 #define PCIC_MULTIMEDIA 0x04 #define PCIS_MULTIMEDIA_VIDEO 0x00 #define PCIS_MULTIMEDIA_AUDIO 0x01 +#define PCIS_MULTIMEDIA_TELE 0x02 #define PCIS_MULTIMEDIA_OTHER 0x80 #define PCIC_MEMORY 0x05 @@ -233,12 +236,15 @@ #define PCIS_BRIDGE_PCMCIA 0x05 #define PCIS_BRIDGE_NUBUS 0x06 #define PCIS_BRIDGE_CARDBUS 0x07 +#define PCIS_BRIDGE_RACEWAY 0x08 #define PCIS_BRIDGE_OTHER 0x80 #define PCIC_SIMPLECOMM 0x07 #define PCIS_SIMPLECOMM_UART 0x00 #define PCIP_SIMPLECOMM_UART_16550A 0x02 #define PCIS_SIMPLECOMM_PAR 0x01 +#define PCIS_SIMPLECOMM_MULSER 0x02 +#define PCIS_SIMPLECOMM_MODEM 0x03 #define PCIS_SIMPLECOMM_OTHER 0x80 #define PCIC_BASEPERIPH 0x08 @@ -246,12 +252,15 @@ #define PCIS_BASEPERIPH_DMA 0x01 #define PCIS_BASEPERIPH_TIMER 0x02 #define PCIS_BASEPERIPH_RTC 0x03 +#define PCIS_BASEPERIPH_PCIHOT 0x04 #define PCIS_BASEPERIPH_OTHER 0x80 #define PCIC_INPUTDEV 0x09 #define PCIS_INPUTDEV_KEYBOARD 0x00 #define PCIS_INPUTDEV_DIGITIZER 0x01 #define PCIS_INPUTDEV_MOUSE 0x02 +#define PCIS_INPUTDEV_SCANNER 0x02 +#define PCIS_INPUTDEV_GAMEPORT 0x02 #define PCIS_INPUTDEV_OTHER 0x80 #define PCIC_DOCKING 0x0a @@ -264,6 +273,7 @@ #define PCIS_PROCESSOR_PENTIUM 0x02 #define PCIS_PROCESSOR_ALPHA 0x10 #define PCIS_PROCESSOR_POWERPC 0x20 +#define PCIS_PROCESSOR_MIPS 0x20 #define PCIS_PROCESSOR_COPROC 0x40 #define PCIC_SERIALBUS 0x0c @@ -276,6 +286,30 @@ #define PCIP_SERIALBUS_USB_EHCI 0x20 #define PCIS_SERIALBUS_FC 0x04 #define PCIS_SERIALBUS_SMBUS 0x05 + +#define PCIC_WIRELESS 0x0d +#define PCIS_WIRELESS_IRDA 0x00 +#define PCIS_WIRELESS_IR 0x01 +#define PCIS_WIRELESS_RF 0x10 +#define PCIS_WIRELESS_OTHER 0x80 + +#define PCIC_INTELLIIO 0x0e +#define PCIS_INTELLIIO_I2O 0x00 + +#define PCIC_SATCOM 0x0f +#define PCIS_SATCOM_TV 0x01 +#define PCIS_SATCOM_AUDIO 0x02 +#define PCIS_SATCOM_VOICE 0x03 +#define PCIS_SATCOM_DATA 0x04 + +#define PCIC_CRYPTO 0x10 +#define PCIS_CRYPTO_NETCOMP 0x00 +#define PCIS_CRYPTO_ENTERTAIN 0x10 +#define PCIS_CRYPTO_OTHER 0x80 + +#define PCIC_DASP 0x11 +#define PCIS_DASP_DPIO 0x00 +#define PCIS_DASP_OTHER 0x80 #define PCIC_OTHER 0xff --ftEhullJWpWg/VHq-- From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 07:04:56 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D927E16A4CE for ; Sun, 3 Oct 2004 07:04:56 +0000 (GMT) Received: from ylpvm15.prodigy.net (ylpvm15-ext.prodigy.net [207.115.57.46]) by mx1.FreeBSD.org (Postfix) with ESMTP id 92F8A43D1D for ; Sun, 3 Oct 2004 07:04:56 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (adsl-67-124-49-205.dsl.snfc21.pacbell.net [67.124.49.205])i93752qM004216; Sun, 3 Oct 2004 03:05:03 -0400 Message-ID: <415FA496.6000902@elischer.org> Date: Sun, 03 Oct 2004 00:04:54 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4b) Gecko/20030524 X-Accept-Language: en, hu MIME-Version: 1.0 To: Stephan Uphoff References: <1096489576.3733.1868.camel@palm.tree.com> <200409291652.29990.jhb@FreeBSD.org> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <20041002053351.GA6259@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <1096760257.34527.14.camel@palm.tree.com> In-Reply-To: <1096760257.34527.14.camel@palm.tree.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 07:04:57 -0000 Stephan Uphoff wrote: > On Sat, 2004-10-02 at 14:31, Peter Holm wrote: > >>OK, right now I'm testing with all of Stephan's patches + the >>MUTEX_WAKE_ALL flag. Uptime is 3 3/4 hour and looking good. > > > Great. > > Your attached diff contained all the fixes needed and I don't see the > need to post a cumulative patch. > > The only thing left to do is migrate a critical sections from > kern_mutex.c to subr_turnstile.c for readability. > (no functional changes) > > Maybe it would also better to just force MUTEX_WAKE_ALL in > kern_mutex.c (#ifndef MUTEX_WAKE_ALL \n#define MUTEX_WAKE_ALL\n#endif) > to avoid temporary configuration file pollution? > > Stephan > > Hi stephan. Just reading your patch(es).. What's the logic behind adding the critical sections around the priority changes in the turnstyle code? can you give an example of the timeline you are avoidning? I presume it involves an interrupt setting another thread running. Why do you think that it is unacceptible to allow a thread with a lower priority to continue to run on another cpu (or this cpu) (maybe_preempt_in_ksegrp()) Can you show the reason that this is "critical" and not just suboptimal? If you are going to preempt threads from the same KSEGRP why not preempt threads from other ksegrps? BTW an available slot count going -ve might not be a disaster.. it's just a count.. I guess you must have KSEG_PEEMPT_BEST_CPU always set because preempting the "not best" cpu would lead to teh same inversion problem that you are trying to avoid, when you leave a thread running on another CPU that has less priority (numerically greater than) than the one that you are preempting.. Where is the critical_enter that matches the extra critical_exit() you put in sched_switch()? I haven' been able to yet figure out how you don't get a double exit. but I've only looked for a few minutes. regarding the ksegrp runq. For some time I have wanted to split it into 3 queues and the threads will move between them. (in the ksegrp sched-private section) queue 1.. threads that are running queue 2.. threads that are on the system run queue. queue 3.. threads that do not yet have a slot. what do you think? it should simplify the slot counting stuff. From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 07:24:42 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 1543E16A4CF for ; Sun, 3 Oct 2004 07:24:42 +0000 (GMT) Received: from relay.pair.com (relay.pair.com [209.68.1.20]) by mx1.FreeBSD.org (Postfix) with SMTP id 965DC43D46 for ; Sun, 3 Oct 2004 07:24:41 +0000 (GMT) (envelope-from pho@holm.cc) Received: (qmail 21261 invoked from network); 3 Oct 2004 07:24:40 -0000 Received: from 0x50a43fc7.hknxx1.adsl-dhcp.tele.dk (HELO peter.osted.lan) (80.164.63.199) by relay.pair.com with SMTP; 3 Oct 2004 07:24:40 -0000 X-pair-Authenticated: 80.164.63.199 Received: from peter.osted.lan (localhost.osted.lan [127.0.0.1]) by peter.osted.lan (8.12.10/8.12.10) with ESMTP id i937OdXh004818; Sun, 3 Oct 2004 09:24:39 +0200 (CEST) (envelope-from pho@peter.osted.lan) Received: (from pho@localhost) by peter.osted.lan (8.12.10/8.12.10/Submit) id i937OchI004817; Sun, 3 Oct 2004 09:24:38 +0200 (CEST) (envelope-from pho) Date: Sun, 3 Oct 2004 09:24:38 +0200 From: Peter Holm To: Julian Elischer Message-ID: <20041003072438.GA4734@peter.osted.lan> References: <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <20041002053351.GA6259@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <1096760257.34527.14.camel@palm.tree.com> <415F677B.5030108@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <415F677B.5030108@elischer.org> User-Agent: Mutt/1.4.1i cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 07:24:42 -0000 On Sat, Oct 02, 2004 at 07:44:11PM -0700, Julian Elischer wrote: > Stephan Uphoff wrote: > >On Sat, 2004-10-02 at 14:31, Peter Holm wrote: > > I have now stress tested for more than 16 hours without seeing any freezes. This is *good* news. However, I did get a panic. I'll try to recreate it and hopefully get a dump: kernel trap 12 with interrupts disabled Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x14c fault code = supervisor read, page not present instruction pointer = 0x8:0xc0618ade stack pointer = 0x10:0xcfcbabfc frame pointer = 0x10:0xcfcbac0c code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, def32 1, gran 1 processor eflags = resume, IOPL = 0 current process = 30600 (java_vm) [thread 100801] Stopped at sched_add+0x16: movl 0x14c(%esi),%ebx db> where sched_add(0,0) at sched_add+0x16 setrunqueue(c3545a80,0,c181ca80,c3545a80,cfcbac54) at setrunqueue+0x199 sched_wakeup(c3545a80) at sched_wakeup+0x43 setrunnable(c3545a80,c3545a80,cfcbac78,cfcbac8c,c0625cbf) at setrunnable+0x92 sleepq_resume_thread(c3545a80,ffffffff) at sleepq_resume_thread+0x82 sleepq_broadcast(c370be40,0,ffffffff,cfcbaccc,c05fae01) at sleepq_broadcast+0xf7 wakeup(c370be40) at wakeup+0xf thread_userret(c1a93d80,cfcbad48) at thread_userret+0x121 userret(c1a93d80,cfcbad48,1,3,1) at userret+0x57 syscall(837002f,80d002f,bf69002f,8215000,2850c6e0) at syscall+0x2d9 Xint0x80_syscall() at Xint0x80_syscall+0x1f --- syscall (1, FreeBSD ELF32, sys_exit), eip = 0x280e667f, esp = 0xbf8e5b50, ebp = 0xbf8e5b7c --- db> call doadump Dumping 255 MB panic: blockable sleep lock (sleep mutex) taskqueue @ kern/subr_taskqueue.c:132 cpuid = 0 Uptime: 16h5m17s (kgdb) l *0xc0618ade 0xc0618ade is in sched_add (../../../kern/sched_4bsd.c:960). 955 #ifdef SMP 956 int forwarded = 0; 957 int cpu; 958 #endif 959 960 ke = td->td_kse; 961 mtx_assert(&sched_lock, MA_OWNED); 962 KASSERT(ke->ke_state != KES_ONRUNQ, 963 ("sched_add: kse %p (%s) already in run queue", ke, 964 ke->ke_proc->p_comm)); > >>OK, right now I'm testing with all of Stephan's patches + the > >>MUTEX_WAKE_ALL flag. Uptime is 3 3/4 hour and looking good. > > I've just resurfaced after a week of hell at work and home > not BAD hell but more like "busy as hell". > > I'm just integrating your fixes into my tree to understand what they do.. > expect more mail from me later. > > > > > > >Great. > > > >Your attached diff contained all the fixes needed and I don't see the > >need to post a cumulative patch. > > > >The only thing left to do is migrate a critical sections from > >kern_mutex.c to subr_turnstile.c for readability. > >(no functional changes) > > > >Maybe it would also better to just force MUTEX_WAKE_ALL in > >kern_mutex.c (#ifndef MUTEX_WAKE_ALL \n#define MUTEX_WAKE_ALL\n#endif) > >to avoid temporary configuration file pollution? > > > > Stephan > > > > -- Peter Holm From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 08:04:42 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 715C116A4CE; Sun, 3 Oct 2004 08:04:42 +0000 (GMT) Received: from mailout2.pacific.net.au (mailout2.pacific.net.au [61.8.0.85]) by mx1.FreeBSD.org (Postfix) with ESMTP id CA2AC43D39; Sun, 3 Oct 2004 08:04:41 +0000 (GMT) (envelope-from bde@zeta.org.au) Received: from mailproxy1.pacific.net.au (mailproxy1.pacific.net.au [61.8.0.86])i9384dKP026440; Sun, 3 Oct 2004 18:04:39 +1000 Received: from katana.zip.com.au (katana.zip.com.au [61.8.7.246]) i9384cCL007778; Sun, 3 Oct 2004 18:04:38 +1000 Date: Sun, 3 Oct 2004 18:04:38 +1000 (EST) From: Bruce Evans X-X-Sender: bde@delplex.bde.org To: John Baldwin In-Reply-To: <200410011100.42302.jhb@FreeBSD.org> Message-ID: <20041003180329.G6058@delplex.bde.org> References: <200410011100.42302.jhb@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII cc: arch@FreeBSD.org Subject: Re: [PATCH] Rework how we store process times in the kernel and deferring calcru() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 08:04:42 -0000 On Fri, 1 Oct 2004, John Baldwin wrote: > I'll commit this soonish unless there are any objections. The basic idea is > to store process times resource usage as raw data (i.e. as bintimes and tick > counts) for both process usage and child usage and only calculate the timeval > style times if they are explicitly asked for. This lets us avoid always > calling calcru() to calculate the timeval values in exit1() for example. A > more detailed listing of the changes follows: > ... > The patch is at http://www.freebsd.org/~jhb/patches/rusage_ext.patch and is > largely based on a patch given to me by bde@. Thanks for finishing this. I will send private mail about one of the details. Hope you don't get too much mail :-). Bruce From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 18:42:21 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 3EA6B16A4CE for ; Sun, 3 Oct 2004 18:42:21 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id BC48343D3F for ; Sun, 3 Oct 2004 18:42:20 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 27898 invoked by uid 89); 3 Oct 2004 18:42:18 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 3 Oct 2004 18:42:18 -0000 Received: (qmail 27891 invoked by uid 89); 3 Oct 2004 18:42:18 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 3 Oct 2004 18:42:18 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i93IgHmt038963; Sun, 3 Oct 2004 14:42:18 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: Julian Elischer In-Reply-To: <415FA496.6000902@elischer.org> References: <1096489576.3733.1868.camel@palm.tree.com> <200409291652.29990.jhb@FreeBSD.org> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <415FA496.6000902@elischer.org> Content-Type: text/plain Message-Id: <1096828937.38592.52.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Sun, 03 Oct 2004 14:42:17 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 18:42:21 -0000 On Sun, 2004-10-03 at 03:04, Julian Elischer wrote: > Hi stephan. > > Just reading your patch(es).. > > What's the logic behind adding the critical sections around the > priority changes in the turnstyle code? > > can you give an example of the timeline you are avoidning? > I presume it involves an interrupt setting another thread running. John asked the same question - the answer should already be in your inbox ;-) ( Re: scheduler (sched_4bsd) questions | Wed, 29 Sep 2004 18:14:17 -0400) > > Why do you think that it is unacceptible to allow a thread with a lower > priority to continue to run on another cpu (or this cpu) (maybe_preempt_in_ksegrp()) > Can you show the reason that this is "critical" and not just suboptimal? > If you are going to preempt threads from the same KSEGRP why not preempt > threads from other ksegrps? BTW an available slot count going > -ve might not be a disaster.. it's just a count.. This may preempt other ksegrps once the thread gets a slot and adds itself to the right runtime queue. I kind of liked the idea of being able to limit the concurrency and decided to honor slot count. Normal cases will have as many slot openings as cpus and we can just preempt the current cpu. (if necessary) maybe_preempt_in_ksegrp is in general needed to allow priority inheritance to work correctly: Example: A ksegrp has two threads A and B. A owns mutex x. B owns mutex y and is blocked on mutex x. The clock thread C is blocked on mutex y held by B. C donates the priority to B and A. Thread A runs and releases x. Without maybe_preempt_in_ksegrp B does not get a slot and won't be added to the system run queue. Thread A exits the kernel and continues to run in userspace. C is still blocked (and maybe other interrupt threads) and A can potentially loop in userspace forever. ( Or until Peter sends a ping ;-) > I guess you must have KSEG_PEEMPT_BEST_CPU always set because > preempting the "not best" cpu would lead to teh same inversion problem > that you are trying to avoid, when you leave a thread running on another CPU > that has less priority (numerically greater than) than the one that you > are preempting.. Actually I don't have KSEG_PEEMPT_BEST_CPU defined. I just had the feeling that it will reduce switching when/once FULL_PREEMTION is defined. > > Where is the critical_enter that matches the extra critical_exit() > you put in sched_switch()? I haven' been able to yet figure out how > you don't get a double exit. but I've only looked for a few minutes. sched_switch calls setrunqueue() or slot_fill(). Both can call sched_add. sched_add can call maybe_preempt. maybe_preempt may call mi_switch without the extra critical_enter. > regarding the ksegrp runq. > For some time I have wanted to split it into 3 queues and the > threads will move between them. (in the ksegrp sched-private section) > queue 1.. threads that are running > queue 2.. threads that are on the system run queue. > queue 3.. threads that do not yet have a slot. > > what do you think? > > it should simplify the slot counting stuff. > This would be neat. Queue 3 needs to be sorted..... ... what is the maximum size of this queue ? Stephan From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 20:03:36 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C2A1816A4CE for ; Sun, 3 Oct 2004 20:03:36 +0000 (GMT) Received: from ylpvm29.prodigy.net (ylpvm29-ext.prodigy.net [207.115.57.60]) by mx1.FreeBSD.org (Postfix) with ESMTP id 703F243D1D for ; Sun, 3 Oct 2004 20:03:36 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (adsl-67-124-50-20.dsl.snfc21.pacbell.net [67.124.50.20])i93K3HIV011590; Sun, 3 Oct 2004 16:03:17 -0400 Message-ID: <41605B16.50103@elischer.org> Date: Sun, 03 Oct 2004 13:03:34 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4b) Gecko/20030524 X-Accept-Language: en, hu MIME-Version: 1.0 To: Stephan Uphoff References: <1096489576.3733.1868.camel@palm.tree.com> <200409291652.29990.jhb@FreeBSD.org> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <415FA496.6000902@elischer.org> <1096828937.38592.52.camel@palm.tree.com> In-Reply-To: <1096828937.38592.52.camel@palm.tree.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 20:03:36 -0000 Stephan Uphoff wrote: > On Sun, 2004-10-03 at 03:04, Julian Elischer wrote: > >>regarding the ksegrp runq. >>For some time I have wanted to split it into 3 queues and the >>threads will move between them. (in the ksegrp sched-private section) >>queue 1.. threads that are running >>queue 2.. threads that are on the system run queue. >>queue 3.. threads that do not yet have a slot. >> >>what do you think? >> >>it should simplify the slot counting stuff. >> > > > This would be neat. > > Queue 3 needs to be sorted..... > ... what is the maximum size of this queue ? about 8000 on i386 :-/ one for every runnable thread in the kernel for that ksegrp. Usually just a few. I've been considerring using the standard run-queue code and using a system runqueue (i.e. 64 queues in prioity order) per ksegrp but it's a lot of overhead. I guess we'll need some sort of tunable syste that is efficient for just a few threads and switches if it gets a lot. > > > Stephan > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 21:39:33 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C006016A4CE for ; Sun, 3 Oct 2004 21:39:33 +0000 (GMT) Received: from ylpvm43.prodigy.net (ylpvm43-ext.prodigy.net [207.115.57.74]) by mx1.FreeBSD.org (Postfix) with ESMTP id 60D1243D39 for ; Sun, 3 Oct 2004 21:39:33 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (adsl-67-124-50-20.dsl.snfc21.pacbell.net [67.124.50.20])i93LdeCE021380; Sun, 3 Oct 2004 17:39:41 -0400 Message-ID: <41607192.8020809@elischer.org> Date: Sun, 03 Oct 2004 14:39:30 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4b) Gecko/20030524 X-Accept-Language: en, hu MIME-Version: 1.0 To: Stephan Uphoff References: <1096489576.3733.1868.camel@palm.tree.com> <200409291652.29990.jhb@FreeBSD.org> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <20041002053351.GA6259@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <1096760257.34527.14.camel@palm.tree.com> <415FA496.6000902@elischer.org> <1096828937.38592.52.camel@palm.tree.com> In-Reply-To: <1096828937.38592.52.camel@palm.tree.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 21:39:33 -0000 Stephan Uphoff wrote: you didn't answer this question.. > > >>Where is the critical_enter that matches the extra critical_exit() >>you put in sched_switch()? I haven' been able to yet figure out how >>you don't get a double exit. but I've only looked for a few minutes. > ==== //depot/projects/nsched/sys/kern/sched_4bsd.c#58 - /home/julian/p4/nsched/sys/kern/sched_4bsd.c ==== @@ -844,6 +844,8 @@ if (newtd == NULL || newtd->td_ksegrp != td->td_ksegrp) slot_fill(td->td_ksegrp); } + critical_exit(); + td->td_pflags &= ~TDP_OWEPREEMPT; } if (newtd) { /* where is the matching critical_enter()? From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 22:43:42 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5880516A4CE for ; Sun, 3 Oct 2004 22:43:42 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id BD44243D55 for ; Sun, 3 Oct 2004 22:43:41 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 31878 invoked by uid 89); 3 Oct 2004 22:43:40 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 3 Oct 2004 22:43:40 -0000 Received: (qmail 31871 invoked by uid 89); 3 Oct 2004 22:43:40 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 3 Oct 2004 22:43:40 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i93Mhdmt039850; Sun, 3 Oct 2004 18:43:40 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: Julian Elischer In-Reply-To: <41607192.8020809@elischer.org> References: <1096489576.3733.1868.camel@palm.tree.com> <200409291652.29990.jhb@FreeBSD.org> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <415FA496.6000902@elischer.org><41607192.8020809@elischer.org> Content-Type: text/plain Message-Id: <1096843419.38592.76.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Sun, 03 Oct 2004 18:43:39 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 22:43:42 -0000 On Sun, 2004-10-03 at 17:39, Julian Elischer wrote: > Stephan Uphoff wrote: > you didn't answer this question.. > > > > > > >>Where is the critical_enter that matches the extra critical_exit() > >>you put in sched_switch()? I haven' been able to yet figure out how > >>you don't get a double exit. but I've only looked for a few minutes. > > > > ==== //depot/projects/nsched/sys/kern/sched_4bsd.c#58 - > /home/julian/p4/nsched/sys/kern/sched_4bsd.c ==== > @@ -844,6 +844,8 @@ > if (newtd == NULL || newtd->td_ksegrp != td->td_ksegrp) > slot_fill(td->td_ksegrp); > } > + critical_exit(); > + td->td_pflags &= ~TDP_OWEPREEMPT; > } > if (newtd) { > /* > > > where is the matching critical_enter()? > >From Peter's unified diff: RCS file: /home/ncvs/src/sys/kern/sched_4bsd.c,v retrieving revision 1.65 diff -u -r1.65 sched_4bsd.c --- sys/kern/sched_4bsd.c 16 Sep 2004 07:12:59 -0000 1.65 +++ sys/kern/sched_4bsd.c 2 Oct 2004 14:46:29 -0000 @@ -823,6 +823,7 @@ TD_SET_CAN_RUN(td); else { td->td_ksegrp->kg_avail_opennings++; + critical_enter(); if (TD_IS_RUNNING(td)) { /* Put us back on the run queue (kse and all). */ setrunqueue(td, SRQ_OURSELF|SRQ_YIELDING); @@ -834,6 +835,8 @@ */ slot_fill(td->td_ksegrp); } + critical_exit(); + td->td_pflags &= ~TDP_OWEPREEMPT; } if (newtd == NULL) newtd = choosethread(); From owner-freebsd-arch@FreeBSD.ORG Sun Oct 3 23:55:17 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5434B16A4CE for ; Sun, 3 Oct 2004 23:55:17 +0000 (GMT) Received: from ylpvm43.prodigy.net (ylpvm43-ext.prodigy.net [207.115.57.74]) by mx1.FreeBSD.org (Postfix) with ESMTP id F12E443D31 for ; Sun, 3 Oct 2004 23:55:16 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (adsl-67-124-50-20.dsl.snfc21.pacbell.net [67.124.50.20])i93NtOCE007355; Sun, 3 Oct 2004 19:55:25 -0400 Message-ID: <41609162.9090502@elischer.org> Date: Sun, 03 Oct 2004 16:55:14 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.4b) Gecko/20030524 X-Accept-Language: en, hu MIME-Version: 1.0 To: Stephan Uphoff References: <1096489576.3733.1868.camel@palm.tree.com> <200409291652.29990.jhb@FreeBSD.org> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <1096608201.21577.203.camel@palm.tree.com> <20041001141040.GA1556@peter.osted.lan> <1096647194.27811.12.camel@palm.tree.com> <20041001192551.GA3381@peter.osted.lan> <20041002053351.GA6259@peter.osted.lan> <415EEFFE.5080309@elischer.org> <20041002183120.GA1202@peter.osted.lan> <1096760257.34527.14.camel@palm.tree.com> <415FA496.6000902@elischer.org> <1096828937.38592.52.camel@palm.tree.com> <41607192.8020809@elischer.org> <1096843419.38592.76.camel@palm.tree.com> In-Reply-To: <1096843419.38592.76.camel@palm.tree.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 03 Oct 2004 23:55:17 -0000 Stephan Uphoff wrote: > On Sun, 2004-10-03 at 17:39, Julian Elischer wrote: > >>Stephan Uphoff wrote: >> you didn't answer this question.. >> >> >>> >>>>Where is the critical_enter that matches the extra critical_exit() >>>>you put in sched_switch()? I haven' been able to yet figure out how >>>>you don't get a double exit. but I've only looked for a few minutes. >>> >>==== //depot/projects/nsched/sys/kern/sched_4bsd.c#58 - >>/home/julian/p4/nsched/sys/kern/sched_4bsd.c ==== >>@@ -844,6 +844,8 @@ >> if (newtd == NULL || newtd->td_ksegrp != td->td_ksegrp) >> slot_fill(td->td_ksegrp); >> } >>+ critical_exit(); >>+ td->td_pflags &= ~TDP_OWEPREEMPT; >> } >> if (newtd) { >> /* >> >> >>where is the matching critical_enter()? interesting.. somehow when I applied that diff, the critical_enter disappeared? >> > > >>From Peter's unified diff: > > RCS file: /home/ncvs/src/sys/kern/sched_4bsd.c,v > retrieving revision 1.65 > diff -u -r1.65 sched_4bsd.c > --- sys/kern/sched_4bsd.c 16 Sep 2004 07:12:59 -0000 1.65 > +++ sys/kern/sched_4bsd.c 2 Oct 2004 14:46:29 -0000 > @@ -823,6 +823,7 @@ > TD_SET_CAN_RUN(td); > else { > td->td_ksegrp->kg_avail_opennings++; > + critical_enter(); > if (TD_IS_RUNNING(td)) { > /* Put us back on the run queue (kse and all). > */ > setrunqueue(td, SRQ_OURSELF|SRQ_YIELDING); > @@ -834,6 +835,8 @@ > */ > slot_fill(td->td_ksegrp); > } > + critical_exit(); > + td->td_pflags &= ~TDP_OWEPREEMPT; > } > if (newtd == NULL) > newtd = choosethread(); From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 01:25:19 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2C1CD16A4D0 for ; Mon, 4 Oct 2004 01:25:19 +0000 (GMT) Received: from dhcp77.rapids.campusvillage.com (dhcp77.rapids.campusvillage.com [63.76.42.77]) by mx1.FreeBSD.org (Postfix) with SMTP id 1BD4C43D5A for ; Mon, 4 Oct 2004 01:25:09 +0000 (GMT) (envelope-from reddenedgeair@hostin.com) From: =?utf-8?q?Isaiah Wapdtega?= To: =?utf-8?q?Leslie Rlin?= Date: Mon, 4 Oct 2004 01:29:51 +0000 MIME-Version: 1.0 Message-Id: <20041004012509.1BD4C43D5A@mx1.FreeBSD.org> Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.1 Subject: Coddled and dissolvable lozenges for literal chaps X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 01:25:19 -0000 Our lozenges are only similar usual tablets but they are especially formulated to be supple and dissolvable under the glossa. The lozenges is took up at the oral fissure and moves into the blood straight instead of rising through with the tummytum. This results in a speedy more potent result which run up to 33 hours! http://drbilly.info/ct/index.php?pid=3Deph5653 From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 10:15:00 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2611316A4CE for ; Mon, 4 Oct 2004 10:15:00 +0000 (GMT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7F01243D55 for ; Mon, 4 Oct 2004 10:14:59 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.1/8.13.1) with ESMTP id i94AEvxw001689 for ; Mon, 4 Oct 2004 12:14:57 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: arch@freebsd.org From: Poul-Henning Kamp Date: Mon, 04 Oct 2004 12:14:57 +0200 Message-ID: <1688.1096884897@critter.freebsd.dk> Sender: phk@critter.freebsd.dk Subject: /sys/conf is getting unwieldy to handle... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 10:15:00 -0000 I think somebody® should start to look at our kernel configuration mechanism now so that we can have something better for 6.x. I am sure that there are 257 ways to do this much better and that at least 5% of these solutions involve XML and 10% involves java and graphical userinterfaces, and at least a 24 bit colorspace for the bikeshed. I think a lot can be done with some very simple changes, and I think more userfriendly tools can be written on top of any sensible "run this ascii-file through this too" model, so lets try to leave the rocket science out of this for now and just improve on the scheme we all know and love. I would love to see a dedicated band of merry men go off in a corner somewhere and "just do it"® Any takers ? Here is a strawman to think about: Split GENERIC into a number of files: PROD.conf standard stuff for a production environment kernel. Stuff like UFS, CD9660 etc goes here. DEBUG.conf Adds the current canonical debug options. ATA.conf Adds ata drivers USB.conf Adds usb stuff FIREWIRE.conf IPV6.conf SCSI.conf NETGRAPH.conf IPFW.conf IPF.conf you get the idea. The actual GENERIC file would then look like: ident GENERIC include PROD.conf include ATA.conf include USB.conf include FIREWIRE.conf include SCSI.conf ... And people can create kernel config files which will be much more resistant to feature addition: ident MYKERNEL include PROD.conf include ATA.conf nooption ataraid include IPFW.conf -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 10:41:46 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5CF2216A4CE for ; Mon, 4 Oct 2004 10:41:46 +0000 (GMT) Received: from moutng.kundenserver.de (moutng.kundenserver.de [212.227.126.186]) by mx1.FreeBSD.org (Postfix) with ESMTP id E893043D2D for ; Mon, 4 Oct 2004 10:41:45 +0000 (GMT) (envelope-from max@love2party.net) Received: from [212.227.126.206] (helo=mrelayng.kundenserver.de) by moutng.kundenserver.de with esmtp (Exim 3.35 #1) id 1CEQHp-0007qd-00; Mon, 04 Oct 2004 12:41:45 +0200 Received: from [217.83.8.81] (helo=donor.laier.local) by mrelayng.kundenserver.de with asmtp (TLSv1:RC4-MD5:128) (Exim 3.35 #1) id 1CEQHo-0004vX-00; Mon, 04 Oct 2004 12:41:45 +0200 From: Max Laier To: freebsd-arch@freebsd.org Date: Mon, 4 Oct 2004 12:40:56 +0200 User-Agent: KMail/1.7 References: <1688.1096884897@critter.freebsd.dk> In-Reply-To: <1688.1096884897@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: multipart/signed; boundary="nextPart1691403.cshXRPRAU9"; protocol="application/pgp-signature"; micalg=pgp-sha1 Content-Transfer-Encoding: 7bit Message-Id: <200410041241.04608.max@love2party.net> X-Provags-ID: kundenserver.de abuse@kundenserver.de auth:61c499deaeeba3ba5be80f48ecc83056 cc: Poul-Henning Kamp Subject: Re: /sys/conf is getting unwieldy to handle... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 10:41:46 -0000 --nextPart1691403.cshXRPRAU9 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Content-Disposition: inline On Monday 04 October 2004 12:14, Poul-Henning Kamp wrote: > I think somebody=AE should start to look at our kernel configuration > mechanism now so that we can have something better for 6.x. > > I am sure that there are 257 ways to do this much better and that > at least 5% of these solutions involve XML and 10% involves java > and graphical userinterfaces, and at least a 24 bit colorspace for > the bikeshed. > > I think a lot can be done with some very simple changes, and I think > more userfriendly tools can be written on top of any sensible "run > this ascii-file through this too" model, so lets try to leave the > rocket science out of this for now and just improve on the scheme > we all know and love. Well spoken! > I would love to see a dedicated band of merry men go off in a corner > somewhere and "just do it"=AE Any takers ? Seem like a good EuroBSDCon BoF/Devsummit topic etc. ... > Here is a strawman to think about: [...] Yes, this would create something I liked to call "meta-options" in former=20 discussions on this topic. Yet, there is a bit more to it. It would (imo) also involve some thinking=20 about the way we build modules (there is a TODO item "revised kld build=20 infrastructure" that seems to cover that part) and maybe also a bit of=20 reorganization/coalescence of the opt_*.h include mechanism ... I have thought about this a bit already, so I'd like to help (if my babblin= g=20 makes sense to you ;) =2D-=20 /"\ Best regards, | mlaier@freebsd.org \ / Max Laier | ICQ #67774661 X http://pf4freebsd.love2party.net/ | mlaier@EFnet / \ ASCII Ribbon Campaign | Against HTML Mail and News --nextPart1691403.cshXRPRAU9 Content-Type: application/pgp-signature -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.6 (FreeBSD) iD8DBQBBYSjAXyyEoT62BG0RAnepAJsEXH9KJDKsJbkR6BKTQv/T7c8OjACfeLLP SKJB9h6WxwnOmUp4cEAJ4tE= =5L1O -----END PGP SIGNATURE----- --nextPart1691403.cshXRPRAU9-- From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 10:48:25 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id CF93A16A4CE for ; Mon, 4 Oct 2004 10:48:25 +0000 (GMT) Received: from mail.auriga.ru (mail.auriga.ru [80.240.102.102]) by mx1.FreeBSD.org (Postfix) with ESMTP id CEA3F43D46 for ; Mon, 4 Oct 2004 10:48:20 +0000 (GMT) (envelope-from alex.neyman@auriga.ru) Received: from localhost ([127.0.0.1] helo=vagabond.auriga.ru) by mail.auriga.ru with esmtp (Exim 4.30) id 1CEQOA-0004DN-1e for freebsd-arch@freebsd.org; Mon, 04 Oct 2004 14:48:18 +0400 From: Alexey Neyman Organization: Auriga To: freebsd-arch@freebsd.org Date: Mon, 4 Oct 2004 14:48:17 +0400 User-Agent: KMail/1.6.2 References: <20041003053142.GI22681@funkthat.com> In-Reply-To: <20041003053142.GI22681@funkthat.com> MIME-Version: 1.0 Content-Disposition: inline Message-Id: <200410041448.17766.alex.neyman@auriga.ru> X-SA-Exim-Mail-From: alex.neyman@auriga.ru Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Spam-Checker-Version: SpamAssassin 2.61 (1.212.2.1-2003-12-09-exp) on mail.auriga.ru X-Spam-Level: X-Spam-Status: No, hits=-4.9 required=5.0 tests=BAYES_00 autolearn=ham version=2.61 X-Spam-Report: * -4.9 BAYES_00 BODY: Bayesian spam probability is 0 to 1% * [score: 0.0000] X-SA-Exim-Version: 3.1 (built Wed Dec 31 15:51:03 MSK 2003) X-SA-Exim-Scanned: Yes Subject: Re: pci device table update... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 10:48:25 -0000 On Sunday 03 October 2004 09:31, John-Mark Gurney wrote: + {PCIC_SATCOM, PCIS_SATCOM_TV, "sat audio"}, ^^^^^^^^^^^^^^- PCIS_SATCOM_AUDIO + {PCIC_SATCOM, PCIS_SATCOM_TV, "sat voice"}, + {PCIC_SATCOM, PCIS_SATCOM_TV, "sat data"}, + {PCIC_CRYPTO, PCIS_CRYPTO_NETCOMP, "entertainment crypto"}, And the other values above, seemingly, have the same cut&paste syndrome ;-) Regards, Alexey. > Ok, I figured I might as well put my recently purchased PCI book to > good use and update our device types for unrecognized devices... > > If you have any problems with the wording of the devices, let me know, > otherwise it'll go into the tree in a few days... (with a respective > copy to pciconf)... > > Thanks. > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > -- I set the thrusters to maximum, close my eye, and begin pressing the fire stud wildly. -- Spathi captain Fwiffo, SC2 From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 11:06:41 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 454CE16A4F7 for ; Mon, 4 Oct 2004 11:06:41 +0000 (GMT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9B97043D48 for ; Mon, 4 Oct 2004 11:06:40 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.1/8.13.1) with ESMTP id i94B6YSb002654; Mon, 4 Oct 2004 13:06:39 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Max Laier From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 04 Oct 2004 12:40:56 +0200." <200410041241.04608.max@love2party.net> Date: Mon, 04 Oct 2004 13:06:34 +0200 Message-ID: <2653.1096887994@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: freebsd-arch@freebsd.org Subject: Re: /sys/conf is getting unwieldy to handle... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 11:06:41 -0000 In message <200410041241.04608.max@love2party.net>, Max Laier writes: >Yes, this would create something I liked to call "meta-options" in former=20 >discussions on this topic. > >Yet, there is a bit more to it. It would (imo) also involve some thinking=20 >about the way we build modules (there is a TODO item "revised kld build=20 >infrastructure" that seems to cover that part) and maybe also a bit of=20 >reorganization/coalescence of the opt_*.h include mechanism ... > >I have thought about this a bit already, so I'd like to help (if my babblin= >g=20 >makes sense to you ;) Well, I didn't really plan to get involved in this apart from using it, this was merely an attempt to encourage people with more spare time than me to pick up the ball. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 12:29:43 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8317F16A4CE for ; Mon, 4 Oct 2004 12:29:43 +0000 (GMT) Received: from av12-1-sn2.hy.skanova.net (av12-1-sn2.hy.skanova.net [81.228.8.185]) by mx1.FreeBSD.org (Postfix) with ESMTP id F3BD343D45 for ; Mon, 4 Oct 2004 12:29:42 +0000 (GMT) (envelope-from martin@mullet.se) Received: by av12-1-sn2.hy.skanova.net (Postfix, from userid 502) id C9B47380F1; Mon, 4 Oct 2004 14:29:41 +0200 (CEST) Received: from smtp4-2-sn2.hy.skanova.net (smtp4-2-sn2.hy.skanova.net [81.228.8.93]) by av12-1-sn2.hy.skanova.net (Postfix) with ESMTP id B8ADD37E5B; Mon, 4 Oct 2004 14:29:41 +0200 (CEST) Received: from [192.168.2.10] (h118n1fls31o985.telia.com [213.65.16.118]) by smtp4-2-sn2.hy.skanova.net (Postfix) with ESMTP id 878C937E43; Mon, 4 Oct 2004 14:29:41 +0200 (CEST) Message-ID: <41614235.5080704@mullet.se> Date: Mon, 04 Oct 2004 14:29:41 +0200 From: Martin Nilsson User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.0; en-US; rv:1.7.3) Gecko/20040910 X-Accept-Language: sv, en-us, en MIME-Version: 1.0 To: Max Laier References: <1688.1096884897@critter.freebsd.dk> <200410041241.04608.max@love2party.net> In-Reply-To: <200410041241.04608.max@love2party.net> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 8bit cc: Poul-Henning Kamp cc: freebsd-arch@freebsd.org Subject: Re: /sys/conf is getting unwieldy to handle... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 12:29:43 -0000 Max Laier wrote: > Yet, there is a bit more to it. It would (imo) also involve some thinking > about the way we build modules (there is a TODO item "revised kld build > infrastructure" that seems to cover that part) Are there any really good reasons why we build lots of modules and most users uses kernels with nearly everything compiled in. Isn't it time to make GENERIC really small, just include what can't be loaded as modules and make loader.conf and the rc scripts load the rest? Of course this requres a stable module ABI and no use of options that break this. /Martin -- Martin Nilsson, CTO & Founder, Mullet Scandinavia AB, Malmö, SWEDEN E-mail: martin@mullet.se, Phone: +46-(0)708-606170, Web: www.mullet.se Our business is well engineered servers optimised for FreeBSD & Linux From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 15:33:31 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 84D5C16A4CE for ; Mon, 4 Oct 2004 15:33:31 +0000 (GMT) Received: from mail6.speakeasy.net (mail6.speakeasy.net [216.254.0.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id C50B443D6D for ; Mon, 4 Oct 2004 15:33:30 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 3041 invoked from network); 4 Oct 2004 15:33:29 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 4 Oct 2004 15:33:29 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i94FXOo2056806; Mon, 4 Oct 2004 11:33:25 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-arch@FreeBSD.org Date: Mon, 4 Oct 2004 11:31:35 -0400 User-Agent: KMail/1.6.2 References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> In-Reply-To: <1096603981.21577.195.camel@palm.tree.com> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410041131.35387.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Julian Elischer cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 15:33:31 -0000 On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > I was looking at the MUTEX_WAKE_ALL undefined case when I used the > > critical section for turnstile_claim(). > > However there are bigger problems with MUTEX_WAKE_ALL undefined > > so you are right - the critical section for turnstile_claim is pretty > > useless. > > Arghhh !!! > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > I recall verifying that it is defined twice. Guess I must have looked at > the wrong source tree :-( > This means yes - we have bigger problems! > > Example: > > Thread A holds a mutex x contested by Thread B and C and has priority > pri(A). > > Thread C holds a mutex y and pri(B) < pri(C) > > Thread A releases the lock wakes thread B but lets C on the turnstile > wait queue. > > An interrupt thread I tries to lock mutex y owned by C. > > However priority inheritance does not work since B needs to run first to > take ownership of the lock. > > I is blocked :-( Ermm, if the interrupt happens after x is released then I's priority should propagate from I to C to B. If the interrupt happens before x is released, then the final bit of propagate_priority() should handle it since it resorts the turnstile's thread queue so that C will be awakened rather than B. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 15:47:49 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5EC5016A4CE for ; Mon, 4 Oct 2004 15:47:49 +0000 (GMT) Received: from harmony.village.org (rover.village.org [168.103.84.182]) by mx1.FreeBSD.org (Postfix) with ESMTP id EC51A43D4C for ; Mon, 4 Oct 2004 15:47:48 +0000 (GMT) (envelope-from imp@bsdimp.com) Received: from localhost (harmony.village.org [10.0.0.6]) by harmony.village.org (8.13.1/8.13.1) with ESMTP id i94FkKgs091470; Mon, 4 Oct 2004 09:46:20 -0600 (MDT) (envelope-from imp@bsdimp.com) Date: Mon, 04 Oct 2004 09:47:53 -0600 (MDT) Message-Id: <20041004.094753.106215007.imp@bsdimp.com> To: martin@mullet.se From: "M. Warner Losh" In-Reply-To: <41614235.5080704@mullet.se> References: <1688.1096884897@critter.freebsd.dk> <200410041241.04608.max@love2party.net> <41614235.5080704@mullet.se> X-Mailer: Mew version 3.3 on Emacs 21.3 / Mule 5.0 (SAKAKI) Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: max@love2party.net cc: phk@phk.freebsd.dk cc: freebsd-arch@FreeBSD.ORG Subject: Re: /sys/conf is getting unwieldy to handle... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 15:47:49 -0000 In message: <41614235.5080704@mullet.se> Martin Nilsson writes: : Max Laier wrote: : > Yet, there is a bit more to it. It would (imo) also involve some thinking : > about the way we build modules (there is a TODO item "revised kld build : > infrastructure" that seems to cover that part) : : Are there any really good reasons why we build lots of modules and most : users uses kernels with nearly everything compiled in. Isn't it time to : make GENERIC really small, just include what can't be loaded as modules : and make loader.conf and the rc scripts load the rest? Of course this : requres a stable module ABI and no use of options that break this. I've been running this way for about 18-20 months. Except for sometimes forgetting to include acpi in the module list, it works great. Well, as long as I don't take short-cuts like 'NO_MODULES' when I'm developing a new kernel. However, I'm not building third-party binary packages at the same time, which is the real kicker. I believe that we already have an 'include' statement, so someone could easily try the include stuff on a prototype basis w/o any real hassle. That would let us get experience with how good or bad phk's ideas are. I suspect they are overly simplistic because they are too lumpy: they include too many drivers that a typical custom kernel would want to get rid of. They just make GENERIC look nicer, but I'm not sure how well it will help things. Warner From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 15:53:51 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6347916A4D1 for ; Mon, 4 Oct 2004 15:53:51 +0000 (GMT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id 7F81A43D3F for ; Mon, 4 Oct 2004 15:53:50 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.1/8.13.1) with ESMTP id i94FrjWC007906; Mon, 4 Oct 2004 17:53:45 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: "M. Warner Losh" From: "Poul-Henning Kamp" In-Reply-To: Your message of "Mon, 04 Oct 2004 09:47:53 MDT." <20041004.094753.106215007.imp@bsdimp.com> Date: Mon, 04 Oct 2004 17:53:45 +0200 Message-ID: <7905.1096905225@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: martin@mullet.se cc: max@love2party.net cc: freebsd-arch@freebsd.org Subject: Re: /sys/conf is getting unwieldy to handle... X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 15:53:51 -0000 In message <20041004.094753.106215007.imp@bsdimp.com>, "M. Warner Losh" writes: I believe that we already have an 'include' statement, so someone >could easily try the include stuff on a prototype basis w/o any real >hassle. That would let us get experience with how good or bad phk's >ideas are. I suspect they are overly simplistic because they are too >lumpy: they include too many drivers that a typical custom kernel >would want to get rid of. They just make GENERIC look nicer, but I'm >not sure how well it will help things. I'm all for higher granularity, but absent that I'll take big chunks I can easily avoid. My take on the issue is that the stuff you want to avoid follow the big lines ("no USB", "no SCSI", "no IPv6") whereas the stuff you want usually is a subset ("scsi, da and ahc"). Just because we segregate all the scsi stuff into one file doesn't mean you have to take it all, you can still pick and choose the bits you want, but it means that it is easy to get rid of it all. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 17:24:11 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BCC3D16A4CE for ; Mon, 4 Oct 2004 17:24:11 +0000 (GMT) Received: from mail3.speakeasy.net (mail3.speakeasy.net [216.254.0.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id 8058F43D5A for ; Mon, 4 Oct 2004 17:24:11 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 8279 invoked from network); 4 Oct 2004 17:24:10 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 4 Oct 2004 17:24:09 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i94HNxY1057634; Mon, 4 Oct 2004 13:24:06 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: Peter Holm Date: Mon, 4 Oct 2004 13:20:22 -0400 User-Agent: KMail/1.6.2 References: <1095468747.31297.241.camel@palm.tree.com> <200409301017.54350.jhb@FreeBSD.org> <20040930203826.GA55153@peter.osted.lan> In-Reply-To: <20040930203826.GA55153@peter.osted.lan> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410041320.22267.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Stephan Uphoff cc: Julian Elischer cc: freebsd-arch@FreeBSD.org Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 17:24:12 -0000 On Thursday 30 September 2004 04:38 pm, Peter Holm wrote: > On Thu, Sep 30, 2004 at 10:17:54AM -0400, John Baldwin wrote: > > On Wednesday 29 September 2004 06:14 pm, Stephan Uphoff wrote: > > > On Wed, 2004-09-29 at 16:52, John Baldwin wrote: > > > > > > OK - here is a crude patch to fix some problems with mutex > > > > > > priority inheritance. My theory is that the clock thread gets > > > > > > stuck waiting on GIANT. > > > > > > > > > > > > During release/acquisition of a contested sleep mutex there are a > > > > > > few windows where a task can be preempted when actions (waking up > > > > > > blocked threads, ownership of the mutex, ..) need to be atomic as > > > > > > far as scheduling is concerned. Otherwise priority inheritance > > > > > > may fail. The patch uses critical_enter/critical_exit to protect > > > > > > these regions against preemption. > > > > > > > > > > > > It would be great if could run this in addition to the other > > > > > > patches. > > > > > > > > turnstile_claim() doesn't make any threads runnable and thus can't > > > > preempt. The other place is supposed to preempt, and it should be ok > > > > to do so. Note that since the turnstile chain lock is held, that > > > > includes a nested critical section and any preemption will be > > > > deferred until the turnstile lock is released via turnstile_release > > > > which happens in the middle of > > > > turnstile_unpend() after it has finished building a list of all the > > > > threads to be made runnable so that the turnstile object can be > > > > re-used safely. I don't think this patch will make much of a > > > > difference (if any). Can you provide a description of a case where > > > > you think the priority inheritance can fail if turnstile_unpend() > > > > doesn't run in a nested critical section? > > > > > > This is a bit of a mind bender. > > > I hope you have some aspirins close by ;-) > > > > > > Thread A holds a mutex x contested by Thread B and has priority pri(A). > > > Thread B holds a mutex y. > > > There is a thread C with priority pri(C) with pri(C) < pri(A). > > > > > > Thread A is in the process of releasing x. > > > It removes thread B from the turnstile and holds a pointer to B in a > > > private list. > > > Thread A sets the owner of the turnstile to NULL and releases all spin > > > locks. ( mtx_unlock_spin(&tc->tc_lock); line 148) > > > This means interrupts are now enabled. > > > > > > An interrupt occurs (or is already pending) and the interrupt handler > > > puts the associated interrupt thread I on the run queue. > > > This causes a preemption from A to I. > > > The interrupt thread I tries to acquire mutex y owned by B and blocks. > > > I donates its priority to B - but inheritance stops at B. > > > The next thread with the best priority is C and the cpu switches to C. > > > However B needs A to run to make it to the run-queue. > > > > > > If y is GIANT and I is the clock thread C could run forever in > > > userspace without being interrupted. > > > > Fair enough. The right place to fix this is in turnstile_unpend() though > > I think. I have had these patches that try to "clump" setrunqueue's > > before preempting lying around (but not thoroughly tested yet) that might > > fix this as well but in the turnstile code itself: > > > > --- //depot/projects/smpng/sys/kern/kern_thread.c 2004/09/22 15:31:15 > > +++ //depot/user/jhb/preemption/kern/kern_thread.c 2004/09/22 16:59:47 > > @@ -954,6 +954,7 @@ > > p->p_suspcount++; > > TD_SET_SUSPENDED(td); > > TAILQ_INSERT_TAIL(&p->p_suspended, td, td_runq); > > +#if 0 > > /* > > * Hack: If we are suspending but are on the sleep queue > > * then we are in msleep or the cv equivalent. We > > @@ -962,6 +963,7 @@ > > */ > > if (TD_ON_SLEEPQ(td)) > > TD_SET_SLEEPING(td); > > +#endif > > } > > > > void > > @@ -988,9 +990,11 @@ > > mtx_assert(&sched_lock, MA_OWNED); > > PROC_LOCK_ASSERT(p, MA_OWNED); > > if (!P_SHOULDSTOP(p)) { > > + critical_enter(); > > while ((td = TAILQ_FIRST(&p->p_suspended))) { > > thread_unsuspend_one(td); > > } > > + critical_exit(); > > } else if ((P_SHOULDSTOP(p) == P_STOPPED_SINGLE) && > > (p->p_numthreads == p->p_suspcount)) { > > /* > > @@ -1025,9 +1029,11 @@ > > * to continue however as this is a bad place to stop. > > */ > > if ((p->p_numthreads != 1) && (!P_SHOULDSTOP(p))) { > > - while (( td = TAILQ_FIRST(&p->p_suspended))) { > > + critical_enter(); > > + while ((td = TAILQ_FIRST(&p->p_suspended))) { > > thread_unsuspend_one(td); > > } > > + critical_exit(); > > } > > mtx_unlock_spin(&sched_lock); > > } > > --- //depot/projects/smpng/sys/kern/subr_sleepqueue.c 2004/08/20 17:10:02 > > +++ //depot/user/jhb/preemption/kern/subr_sleepqueue.c 2004/09/10 > > 21:36:10 @@ -400,9 +400,10 @@ > > * just return. > > */ > > if (td->td_sleepqueue != NULL) { > > - MPASS(!TD_ON_SLEEPQ(td)); > > mtx_unlock_spin(&sc->sc_lock); > > mtx_lock_spin(&sched_lock); > > + MPASS(!TD_ON_SLEEPQ(td)); > > + MPASS(!TD_IS_SLEEPING(td)); > > return; > > } > > > > @@ -709,11 +710,13 @@ > > sleepq_release(wchan); > > > > /* Resume all the threads on the temporary list. */ > > + critical_enter(); > > while (!TAILQ_EMPTY(&list)) { > > td = TAILQ_FIRST(&list); > > TAILQ_REMOVE(&list, td, td_slpq); > > sleepq_resume_thread(td, pri); > > } > > + critical_exit(); > > } > > > > /* > > --- //depot/projects/smpng/sys/kern/subr_turnstile.c 2004/09/03 14:14:21 > > +++ //depot/user/jhb/preemption/kern/subr_turnstile.c 2004/09/10 21:36:10 > > @@ -727,6 +726,7 @@ > > * in turnstile_wait(). Set a flag to force it to try to acquire > > * the lock again instead of blocking. > > */ > > + critical_enter(); > > while (!TAILQ_EMPTY(&pending_threads)) { > > td = TAILQ_FIRST(&pending_threads); > > TAILQ_REMOVE(&pending_threads, td, td_lockq); > > @@ -742,6 +742,7 @@ > > MPASS(TD_IS_RUNNING(td) || TD_ON_RUNQ(td)); > > } > > } > > + critical_exit(); > > mtx_unlock_spin(&sched_lock); > > } > > > > --- //depot/projects/smpng/sys/vm/vm_glue.c 2004/09/22 15:31:15 > > +++ //depot/user/jhb/preemption/vm/vm_glue.c 2004/09/22 16:59:47 > > @@ -753,6 +753,7 @@ > > vm_thread_swapin(td); > > > > PROC_LOCK(p); > > + critical_enter(); > > mtx_lock_spin(&sched_lock); > > p->p_sflag &= ~PS_SWAPPINGIN; > > p->p_sflag |= PS_INMEM; > > @@ -767,6 +768,7 @@ > > > > /* Allow other threads to swap p out now. */ > > --p->p_lock; > > + critical_exit(); > > } > > #endif /* NO_SWAPPING */ > > } > > > > > > I.e., you could just move the critical_enter() in subr_turnstile.c > > earlier so it is before the mtx_unlock_spin() of the turnstile chain > > lock. > > > > -- > > John Baldwin <>< http://www.FreeBSD.org/~jhb/ > > "Power Users Use the Power to Serve" = http://www.FreeBSD.org > > This patch did not seem to make the freeze problem go away. It requires tweaking to get Stephan's fix for turnstile_unpend(). -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 17:24:12 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C7FB916A4CE for ; Mon, 4 Oct 2004 17:24:12 +0000 (GMT) Received: from mail6.speakeasy.net (mail6.speakeasy.net [216.254.0.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9CD0543D49 for ; Mon, 4 Oct 2004 17:24:12 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 14629 invoked from network); 4 Oct 2004 17:24:12 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 4 Oct 2004 17:24:11 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i94HNxY2057634; Mon, 4 Oct 2004 13:24:08 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: Stephan Uphoff Date: Mon, 4 Oct 2004 13:21:45 -0400 User-Agent: KMail/1.6.2 References: <1096610130.21577.219.camel@palm.tree.com> In-Reply-To: <1096610130.21577.219.camel@palm.tree.com> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410041321.45142.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: sched_switch (sched_4bsd) may be preempted X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 17:24:12 -0000 On Friday 01 October 2004 01:55 am, Stephan Uphoff wrote: > sched_switch (sched_4bsd) may be preempted in setrunqueue or slot_fill. > This could be ugly. > Wrapping it into a critical section and resetting TDP_OWEPREEMPT should > work. > > Hand trimmed patch: > > RCS file: /cvsroot/src/sys/kern/sched_4bsd.c,v > retrieving revision 1.65 > diff -u -r1.65 sched_4bsd.c > --- sys/kern/sched_4bsd.c 16 Sep 2004 07:12:59 -0000 1.65 > +++ sys/kern/sched_4bsd.c 1 Oct 2004 05:35:28 -0000 > @@ -823,6 +823,7 @@ > TD_SET_CAN_RUN(td); > else { > td->td_ksegrp->kg_avail_opennings++; > + critical_enter(); > if (TD_IS_RUNNING(td)) { > /* Put us back on the run queue (kse and all). > */ > setrunqueue(td, SRQ_OURSELF|SRQ_YIELDING); > @@ -834,6 +835,8 @@ > */ > slot_fill(td->td_ksegrp); > } > + critical_exit(); > + td->td_pflags &= ~TDP_OWEPREEMPT; > } > if (newtd == NULL) > newtd = choosethread(); I thought that SRQ_YIELDING turned preempting off already. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 17:34:42 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6D2A316A4CF for ; Mon, 4 Oct 2004 17:34:42 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id DCAE943D48 for ; Mon, 4 Oct 2004 17:34:41 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 29366 invoked by uid 89); 4 Oct 2004 17:34:40 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 4 Oct 2004 17:34:40 -0000 Received: (qmail 29334 invoked by uid 89); 4 Oct 2004 17:34:40 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 4 Oct 2004 17:34:40 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i94HYcmt044759; Mon, 4 Oct 2004 13:34:39 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: John Baldwin In-Reply-To: <200410041131.35387.jhb@FreeBSD.org> References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> Content-Type: text/plain Message-Id: <1096911278.44307.17.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Mon, 04 Oct 2004 13:34:38 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 17:34:42 -0000 On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > I was looking at the MUTEX_WAKE_ALL undefined case when I used the > > > critical section for turnstile_claim(). > > > However there are bigger problems with MUTEX_WAKE_ALL undefined > > > so you are right - the critical section for turnstile_claim is pretty > > > useless. > > > > Arghhh !!! > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > I recall verifying that it is defined twice. Guess I must have looked at > > the wrong source tree :-( > > This means yes - we have bigger problems! > > > > Example: > > > > Thread A holds a mutex x contested by Thread B and C and has priority > > pri(A). > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > Thread A releases the lock wakes thread B but lets C on the turnstile > > wait queue. > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > However priority inheritance does not work since B needs to run first to > > take ownership of the lock. > > > > I is blocked :-( > > Ermm, if the interrupt happens after x is released then I's priority should > propagate from I to C to B. There is a hole after the mutex x is released by A - but before B can claim the mutex. The turnstile for mutex x is unowned and interrupt thread I when trying to donate its priority will run into: if (td == NULL) { /* * This really isn't quite right. Really * ought to bump priority of thread that * next acquires the lock. */ return; } So B needs to run and acquire the mutex before priority inheritance works again and does not get a priority boost to do so. This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time - but my time budget is limited and Peter has an interesting bug left that has priority. > If the interrupt happens before x is released, > then the final bit of propagate_priority() should handle it since it resorts > the turnstile's thread queue so that C will be awakened rather than B. Agreed. Stephan From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 18:26:25 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C58AB16A4CE for ; Mon, 4 Oct 2004 18:26:25 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 6EE1B43D2D for ; Mon, 4 Oct 2004 18:26:25 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 19133 invoked by uid 89); 4 Oct 2004 18:26:23 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 4 Oct 2004 18:26:23 -0000 Received: (qmail 19102 invoked by uid 89); 4 Oct 2004 18:26:23 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 4 Oct 2004 18:26:23 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i94IQNmt045033; Mon, 4 Oct 2004 14:26:23 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: John Baldwin In-Reply-To: <200410041321.45142.jhb@FreeBSD.org> References: <1096610130.21577.219.camel@palm.tree.com> <200410041321.45142.jhb@FreeBSD.org> Content-Type: text/plain Message-Id: <1096914383.44307.25.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Mon, 04 Oct 2004 14:26:23 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: sched_switch (sched_4bsd) may be preempted X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 18:26:25 -0000 On Mon, 2004-10-04 at 13:21, John Baldwin wrote: > On Friday 01 October 2004 01:55 am, Stephan Uphoff wrote: > > sched_switch (sched_4bsd) may be preempted in setrunqueue or slot_fill. > > This could be ugly. > > Wrapping it into a critical section and resetting TDP_OWEPREEMPT should > > work. > > > > Hand trimmed patch: > > > > RCS file: /cvsroot/src/sys/kern/sched_4bsd.c,v > > retrieving revision 1.65 > > diff -u -r1.65 sched_4bsd.c > > --- sys/kern/sched_4bsd.c 16 Sep 2004 07:12:59 -0000 1.65 > > +++ sys/kern/sched_4bsd.c 1 Oct 2004 05:35:28 -0000 > > @@ -823,6 +823,7 @@ > > TD_SET_CAN_RUN(td); > > else { > > td->td_ksegrp->kg_avail_opennings++; > > + critical_enter(); > > if (TD_IS_RUNNING(td)) { > > /* Put us back on the run queue (kse and all). > > */ > > setrunqueue(td, SRQ_OURSELF|SRQ_YIELDING); > > @@ -834,6 +835,8 @@ > > */ > > slot_fill(td->td_ksegrp); > > } > > + critical_exit(); > > + td->td_pflags &= ~TDP_OWEPREEMPT; > > } > > if (newtd == NULL) > > newtd = choosethread(); > > I thought that SRQ_YIELDING turned preempting off already. Thanks - I missed that for setrunqueue. However it is needed for maybe_preempt_in_ksegrp and slot_fill. Guess adding a flag parameter and checking for SRQ_YIELDING for both would be and easy way to get rid of the critical section hack. Stephan From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 18:38:46 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A901E16A4CE; Mon, 4 Oct 2004 18:38:46 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 96E7943D2F; Mon, 4 Oct 2004 18:38:46 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id 5E4997A403; Mon, 4 Oct 2004 11:38:46 -0700 (PDT) Message-ID: <416198B6.4030801@elischer.org> Date: Mon, 04 Oct 2004 11:38:46 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: John Baldwin References: <1096610130.21577.219.camel@palm.tree.com> <200410041321.45142.jhb@FreeBSD.org> In-Reply-To: <200410041321.45142.jhb@FreeBSD.org> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Stephan Uphoff Subject: Re: sched_switch (sched_4bsd) may be preempted X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 18:38:46 -0000 John Baldwin wrote: >On Friday 01 October 2004 01:55 am, Stephan Uphoff wrote: > > >>sched_switch (sched_4bsd) may be preempted in setrunqueue or slot_fill. >>This could be ugly. >>Wrapping it into a critical section and resetting TDP_OWEPREEMPT should >>work. >> >>Hand trimmed patch: >> >>RCS file: /cvsroot/src/sys/kern/sched_4bsd.c,v >>retrieving revision 1.65 >>diff -u -r1.65 sched_4bsd.c >>--- sys/kern/sched_4bsd.c 16 Sep 2004 07:12:59 -0000 1.65 >>+++ sys/kern/sched_4bsd.c 1 Oct 2004 05:35:28 -0000 >>@@ -823,6 +823,7 @@ >> TD_SET_CAN_RUN(td); >> else { >> td->td_ksegrp->kg_avail_opennings++; >>+ critical_enter(); >> if (TD_IS_RUNNING(td)) { >> /* Put us back on the run queue (kse and all). >>*/ >> setrunqueue(td, SRQ_OURSELF|SRQ_YIELDING); >>@@ -834,6 +835,8 @@ >> */ >> slot_fill(td->td_ksegrp); >> } >>+ critical_exit(); >>+ td->td_pflags &= ~TDP_OWEPREEMPT; >> } >> if (newtd == NULL) >> newtd = choosethread(); >> >> > >I thought that SRQ_YIELDING turned preempting off already. > > well, that was the intention.. I wonder if there can be more nesting than we expect? From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 18:49:44 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D8AA916A4CF for ; Mon, 4 Oct 2004 18:49:44 +0000 (GMT) Received: from relay.pair.com (relay.pair.com [209.68.1.20]) by mx1.FreeBSD.org (Postfix) with SMTP id 533D843D53 for ; Mon, 4 Oct 2004 18:49:44 +0000 (GMT) (envelope-from pho@holm.cc) Received: (qmail 50983 invoked from network); 4 Oct 2004 18:49:40 -0000 Received: from 0x50a43fc7.hknxx1.adsl-dhcp.tele.dk (HELO peter.osted.lan) (80.164.63.199) by relay.pair.com with SMTP; 4 Oct 2004 18:49:40 -0000 X-pair-Authenticated: 80.164.63.199 Received: from peter.osted.lan (localhost.osted.lan [127.0.0.1]) by peter.osted.lan (8.12.10/8.12.10) with ESMTP id i94IndEe008279; Mon, 4 Oct 2004 20:49:39 +0200 (CEST) (envelope-from pho@peter.osted.lan) Received: (from pho@localhost) by peter.osted.lan (8.12.10/8.12.10/Submit) id i94IndYK008278; Mon, 4 Oct 2004 20:49:39 +0200 (CEST) (envelope-from pho) Date: Mon, 4 Oct 2004 20:49:39 +0200 From: Peter Holm To: Stephan Uphoff Message-ID: <20041004184939.GA8178@peter.osted.lan> References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <1096911278.44307.17.camel@palm.tree.com> User-Agent: Mutt/1.4.1i cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 18:49:45 -0000 On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: > On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > > I was looking at the MUTEX_WAKE_ALL undefined case when I used the > > > > critical section for turnstile_claim(). > > > > However there are bigger problems with MUTEX_WAKE_ALL undefined > > > > so you are right - the critical section for turnstile_claim is pretty > > > > useless. > > > > > > Arghhh !!! > > > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > > I recall verifying that it is defined twice. Guess I must have looked at > > > the wrong source tree :-( > > > This means yes - we have bigger problems! > > > > > > Example: > > > > > > Thread A holds a mutex x contested by Thread B and C and has priority > > > pri(A). > > > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > > > Thread A releases the lock wakes thread B but lets C on the turnstile > > > wait queue. > > > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > > > However priority inheritance does not work since B needs to run first to > > > take ownership of the lock. > > > > > > I is blocked :-( > > > > Ermm, if the interrupt happens after x is released then I's priority should > > propagate from I to C to B. > > There is a hole after the mutex x is released by A - but before B can > claim the mutex. The turnstile for mutex x is unowned and interrupt > thread I when trying to donate its priority will run into: > > if (td == NULL) { > /* > * This really isn't quite right. Really > * ought to bump priority of thread that > * next acquires the lock. > */ > return; > } > > So B needs to run and acquire the mutex before priority inheritance > works again and does not get a priority boost to do so. > > This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time > - but my time budget is limited and Peter has an interesting bug left > that has priority. I'm not closer to being able to create this panic in a controlled way. After a whole day of different tests I finally got this panic: http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one particular Java applet, but it is not easily reproduceable. - Peter > > > If the interrupt happens before x is released, > > then the final bit of propagate_priority() should handle it since it resorts > > the turnstile's thread queue so that C will be awakened rather than B. > > Agreed. > > Stephan From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 18:57:46 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 8E19E16A4CE for ; Mon, 4 Oct 2004 18:57:46 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0040243D3F for ; Mon, 4 Oct 2004 18:57:45 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id D48557A403; Mon, 4 Oct 2004 11:57:45 -0700 (PDT) Message-ID: <41619D29.1000704@elischer.org> Date: Mon, 04 Oct 2004 11:57:45 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: Peter Holm References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> In-Reply-To: <20041004184939.GA8178@peter.osted.lan> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: "freebsd-arch@freebsd.org" cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 18:57:46 -0000 can you run ktrdump against teh corefile and get the ktr output? (you do have it enabled right?) Peter Holm wrote: >On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: > > >>On Mon, 2004-10-04 at 11:31, John Baldwin wrote: >> >> >>>On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: >>> >>> >>>>On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: >>>> >>>> >>>>>I was looking at the MUTEX_WAKE_ALL undefined case when I used the >>>>>critical section for turnstile_claim(). >>>>>However there are bigger problems with MUTEX_WAKE_ALL undefined >>>>>so you are right - the critical section for turnstile_claim is pretty >>>>>useless. >>>>> >>>>> >>>>Arghhh !!! >>>> >>>>MUTEX_WAKE_ALL is NOT an option in GENERIC. >>>>I recall verifying that it is defined twice. Guess I must have looked at >>>>the wrong source tree :-( >>>>This means yes - we have bigger problems! >>>> >>>>Example: >>>> >>>>Thread A holds a mutex x contested by Thread B and C and has priority >>>>pri(A). >>>> >>>>Thread C holds a mutex y and pri(B) < pri(C) >>>> >>>>Thread A releases the lock wakes thread B but lets C on the turnstile >>>>wait queue. >>>> >>>>An interrupt thread I tries to lock mutex y owned by C. >>>> >>>>However priority inheritance does not work since B needs to run first to >>>>take ownership of the lock. >>>> >>>>I is blocked :-( >>>> >>>> >>>Ermm, if the interrupt happens after x is released then I's priority should >>>propagate from I to C to B. >>> >>> >>There is a hole after the mutex x is released by A - but before B can >>claim the mutex. The turnstile for mutex x is unowned and interrupt >>thread I when trying to donate its priority will run into: >> >> if (td == NULL) { >> /* >> * This really isn't quite right. Really >> * ought to bump priority of thread that >> * next acquires the lock. >> */ >> return; >> } >> >>So B needs to run and acquire the mutex before priority inheritance >>works again and does not get a priority boost to do so. >> >>This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time >>- but my time budget is limited and Peter has an interesting bug left >>that has priority. >> >> > >I'm not closer to being able to create this panic in a controlled way. >After a whole day of different tests I finally got this panic: >http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one >particular Java applet, but it is not easily reproduceable. > >- Peter > > > >>>If the interrupt happens before x is released, >>>then the final bit of propagate_priority() should handle it since it resorts >>>the turnstile's thread queue so that C will be awakened rather than B. >>> >>> >>Agreed. >> >> Stephan >> >> >_______________________________________________ >freebsd-arch@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-arch >To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 19:14:13 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DD3FA16A4CE for ; Mon, 4 Oct 2004 19:14:12 +0000 (GMT) Received: from relay.pair.com (relay.pair.com [209.68.1.20]) by mx1.FreeBSD.org (Postfix) with SMTP id 7D89943D41 for ; Mon, 4 Oct 2004 19:14:12 +0000 (GMT) (envelope-from pho@holm.cc) Received: (qmail 78180 invoked from network); 4 Oct 2004 19:14:11 -0000 Received: from 0x50a43fc7.hknxx1.adsl-dhcp.tele.dk (HELO peter.osted.lan) (80.164.63.199) by relay.pair.com with SMTP; 4 Oct 2004 19:14:11 -0000 X-pair-Authenticated: 80.164.63.199 Received: from peter.osted.lan (localhost.osted.lan [127.0.0.1]) by peter.osted.lan (8.12.10/8.12.10) with ESMTP id i94JEAEe008457; Mon, 4 Oct 2004 21:14:10 +0200 (CEST) (envelope-from pho@peter.osted.lan) Received: (from pho@localhost) by peter.osted.lan (8.12.10/8.12.10/Submit) id i94JEAd1008456; Mon, 4 Oct 2004 21:14:10 +0200 (CEST) (envelope-from pho) Date: Mon, 4 Oct 2004 21:14:10 +0200 From: Peter Holm To: Julian Elischer Message-ID: <20041004191410.GA8423@peter.osted.lan> References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> <41619D29.1000704@elischer.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <41619D29.1000704@elischer.org> User-Agent: Mutt/1.4.1i cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 19:14:13 -0000 On Mon, Oct 04, 2004 at 11:57:45AM -0700, Julian Elischer wrote: > can you run ktrdump against teh corefile and get the ktr output? > (you do have it enabled right?) > No, that's one of the problems: doadump() fails with this specific panic. - Peter > > Peter Holm wrote: > > >On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: > > > > > >>On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > >> > >> > >>>On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > >>> > >>> > >>>>On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > >>>> > >>>> > >>>>>I was looking at the MUTEX_WAKE_ALL undefined case when I used the > >>>>>critical section for turnstile_claim(). > >>>>>However there are bigger problems with MUTEX_WAKE_ALL undefined > >>>>>so you are right - the critical section for turnstile_claim is pretty > >>>>>useless. > >>>>> > >>>>> > >>>>Arghhh !!! > >>>> > >>>>MUTEX_WAKE_ALL is NOT an option in GENERIC. > >>>>I recall verifying that it is defined twice. Guess I must have looked at > >>>>the wrong source tree :-( > >>>>This means yes - we have bigger problems! > >>>> > >>>>Example: > >>>> > >>>>Thread A holds a mutex x contested by Thread B and C and has priority > >>>>pri(A). > >>>> > >>>>Thread C holds a mutex y and pri(B) < pri(C) > >>>> > >>>>Thread A releases the lock wakes thread B but lets C on the turnstile > >>>>wait queue. > >>>> > >>>>An interrupt thread I tries to lock mutex y owned by C. > >>>> > >>>>However priority inheritance does not work since B needs to run first to > >>>>take ownership of the lock. > >>>> > >>>>I is blocked :-( > >>>> > >>>> > >>>Ermm, if the interrupt happens after x is released then I's priority > >>>should propagate from I to C to B. > >>> > >>> > >>There is a hole after the mutex x is released by A - but before B can > >>claim the mutex. The turnstile for mutex x is unowned and interrupt > >>thread I when trying to donate its priority will run into: > >> > >> if (td == NULL) { > >> /* > >> * This really isn't quite right. Really > >> * ought to bump priority of thread that > >> * next acquires the lock. > >> */ > >> return; > >> } > >> > >>So B needs to run and acquire the mutex before priority inheritance > >>works again and does not get a priority boost to do so. > >> > >>This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time > >>- but my time budget is limited and Peter has an interesting bug left > >>that has priority. > >> > >> > > > >I'm not closer to being able to create this panic in a controlled way. > >After a whole day of different tests I finally got this panic: > >http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one > >particular Java applet, but it is not easily reproduceable. > > > >- Peter > > > > > > > >>>If the interrupt happens before x is released, > >>>then the final bit of propagate_priority() should handle it since it > >>>resorts the turnstile's thread queue so that C will be awakened rather > >>>than B. > >>> > >>> > >>Agreed. > >> > >> Stephan > >> > >> > >_______________________________________________ > >freebsd-arch@freebsd.org mailing list > >http://lists.freebsd.org/mailman/listinfo/freebsd-arch > >To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > > > > -- Peter Holm From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 19:42:55 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id F059916A4CE for ; Mon, 4 Oct 2004 19:42:54 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 9DC1843D2F for ; Mon, 4 Oct 2004 19:42:54 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id 28FA27A403; Mon, 4 Oct 2004 12:42:54 -0700 (PDT) Message-ID: <4161A7BD.3040706@elischer.org> Date: Mon, 04 Oct 2004 12:42:53 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: Peter Holm References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> <41619D29.1000704@elischer.org> <20041004191410.GA8423@peter.osted.lan> In-Reply-To: <20041004191410.GA8423@peter.osted.lan> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit cc: Stephan Uphoff cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 19:42:55 -0000 ok, then if it happens again, from ddb, run show ktr after you've done the 'ps' and go back a couple of hundred events.. thanks. Peter Holm wrote: >On Mon, Oct 04, 2004 at 11:57:45AM -0700, Julian Elischer wrote: > > >>can you run ktrdump against teh corefile and get the ktr output? >>(you do have it enabled right?) >> >> >> > >No, that's one of the problems: doadump() fails with this specific panic. > >- Peter > > > >>Peter Holm wrote: >> >> >> >>>On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: >>> >>> >>> >>> >>>>On Mon, 2004-10-04 at 11:31, John Baldwin wrote: >>>> >>>> >>>> >>>> >>>>>On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: >>>>> >>>>> >>>>> >>>>> >>>>>>On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>I was looking at the MUTEX_WAKE_ALL undefined case when I used the >>>>>>>critical section for turnstile_claim(). >>>>>>>However there are bigger problems with MUTEX_WAKE_ALL undefined >>>>>>>so you are right - the critical section for turnstile_claim is pretty >>>>>>>useless. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>Arghhh !!! >>>>>> >>>>>>MUTEX_WAKE_ALL is NOT an option in GENERIC. >>>>>>I recall verifying that it is defined twice. Guess I must have looked at >>>>>>the wrong source tree :-( >>>>>>This means yes - we have bigger problems! >>>>>> >>>>>>Example: >>>>>> >>>>>>Thread A holds a mutex x contested by Thread B and C and has priority >>>>>>pri(A). >>>>>> >>>>>>Thread C holds a mutex y and pri(B) < pri(C) >>>>>> >>>>>>Thread A releases the lock wakes thread B but lets C on the turnstile >>>>>>wait queue. >>>>>> >>>>>>An interrupt thread I tries to lock mutex y owned by C. >>>>>> >>>>>>However priority inheritance does not work since B needs to run first to >>>>>>take ownership of the lock. >>>>>> >>>>>>I is blocked :-( >>>>>> >>>>>> >>>>>> >>>>>> >>>>>Ermm, if the interrupt happens after x is released then I's priority >>>>>should propagate from I to C to B. >>>>> >>>>> >>>>> >>>>> >>>>There is a hole after the mutex x is released by A - but before B can >>>>claim the mutex. The turnstile for mutex x is unowned and interrupt >>>>thread I when trying to donate its priority will run into: >>>> >>>> if (td == NULL) { >>>> /* >>>> * This really isn't quite right. Really >>>> * ought to bump priority of thread that >>>> * next acquires the lock. >>>> */ >>>> return; >>>> } >>>> >>>>So B needs to run and acquire the mutex before priority inheritance >>>>works again and does not get a priority boost to do so. >>>> >>>>This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time >>>>- but my time budget is limited and Peter has an interesting bug left >>>>that has priority. >>>> >>>> >>>> >>>> >>>I'm not closer to being able to create this panic in a controlled way. >>>After a whole day of different tests I finally got this panic: >>>http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one >>>particular Java applet, but it is not easily reproduceable. >>> >>>- Peter >>> >>> >>> >>> >>> >>>>>If the interrupt happens before x is released, >>>>>then the final bit of propagate_priority() should handle it since it >>>>>resorts the turnstile's thread queue so that C will be awakened rather >>>>>than B. >>>>> >>>>> >>>>> >>>>> >>>>Agreed. >>>> >>>> Stephan >>>> >>>> >>>> >>>> >>>_______________________________________________ >>>freebsd-arch@freebsd.org mailing list >>>http://lists.freebsd.org/mailman/listinfo/freebsd-arch >>>To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >>> >>> >>> >>> > > > From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 20:28:42 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EC91F16A4D3 for ; Mon, 4 Oct 2004 20:28:41 +0000 (GMT) Received: from mail6.speakeasy.net (mail6.speakeasy.net [216.254.0.206]) by mx1.FreeBSD.org (Postfix) with ESMTP id B8E1F43D48 for ; Mon, 4 Oct 2004 20:28:41 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 14822 invoked from network); 4 Oct 2004 20:28:41 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 4 Oct 2004 20:28:40 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i94KSU5n058953; Mon, 4 Oct 2004 16:28:37 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-arch@FreeBSD.org Date: Mon, 4 Oct 2004 14:03:06 -0400 User-Agent: KMail/1.6.2 References: <1095468747.31297.241.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> In-Reply-To: <1096911278.44307.17.camel@palm.tree.com> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410041403.06187.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Julian Elischer cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 20:28:42 -0000 On Monday 04 October 2004 01:34 pm, Stephan Uphoff wrote: > On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > > I was looking at the MUTEX_WAKE_ALL undefined case when I used the > > > > critical section for turnstile_claim(). > > > > However there are bigger problems with MUTEX_WAKE_ALL undefined > > > > so you are right - the critical section for turnstile_claim is pretty > > > > useless. > > > > > > Arghhh !!! > > > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > > I recall verifying that it is defined twice. Guess I must have looked > > > at the wrong source tree :-( > > > This means yes - we have bigger problems! > > > > > > Example: > > > > > > Thread A holds a mutex x contested by Thread B and C and has priority > > > pri(A). > > > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > > > Thread A releases the lock wakes thread B but lets C on the turnstile > > > wait queue. > > > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > > > However priority inheritance does not work since B needs to run first > > > to take ownership of the lock. > > > > > > I is blocked :-( > > > > Ermm, if the interrupt happens after x is released then I's priority > > should propagate from I to C to B. > > There is a hole after the mutex x is released by A - but before B can > claim the mutex. The turnstile for mutex x is unowned and interrupt > thread I when trying to donate its priority will run into: > > if (td == NULL) { > /* > * This really isn't quite right. Really > * ought to bump priority of thread that > * next acquires the lock. > */ > return; > } > > So B needs to run and acquire the mutex before priority inheritance > works again and does not get a priority boost to do so. > > This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time > - but my time budget is limited and Peter has an interesting bug left > that has priority. Isn't this handled by the mtx_lock == MTX_CONTESTED case that calls into turnstile_claim() which bumps the priority of the new owner to the highest priority waiting thread? I guess this won't happen until B gets to run again which is the problem. You don't know which thread is going to get the lock, so what do you do? You don't even have a way to get to the threads that you might have just woken up. BTW, Solaris uses MUTEX_WAKE_ALL by default, but for performance reasons. It is a kernel option because the idea was to benchmark it both ways and then choose the default based on those numbers. It's off by default as the wake-one was the original behavior. I'm pretty sure BSD/OS has this same issue. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 20:40:29 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 350AA16A4D5 for ; Mon, 4 Oct 2004 20:40:29 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 64B1A43D46 for ; Mon, 4 Oct 2004 20:40:17 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 315 invoked by uid 89); 4 Oct 2004 20:40:16 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 4 Oct 2004 20:40:16 -0000 Received: (qmail 301 invoked by uid 89); 4 Oct 2004 20:40:16 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 4 Oct 2004 20:40:16 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i94KeFmt045718; Mon, 4 Oct 2004 16:40:15 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: John Baldwin In-Reply-To: <200410041403.06187.jhb@FreeBSD.org> References: <1095468747.31297.241.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <200410041403.06187.jhb@FreeBSD.org> Content-Type: text/plain Message-Id: <1096922414.45640.6.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Mon, 04 Oct 2004 16:40:15 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 20:40:29 -0000 On Mon, 2004-10-04 at 14:03, John Baldwin wrote: > On Monday 04 October 2004 01:34 pm, Stephan Uphoff wrote: > > On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > > > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > > > I was looking at the MUTEX_WAKE_ALL undefined case when I used the > > > > > critical section for turnstile_claim(). > > > > > However there are bigger problems with MUTEX_WAKE_ALL undefined > > > > > so you are right - the critical section for turnstile_claim is pretty > > > > > useless. > > > > > > > > Arghhh !!! > > > > > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > > > I recall verifying that it is defined twice. Guess I must have looked > > > > at the wrong source tree :-( > > > > This means yes - we have bigger problems! > > > > > > > > Example: > > > > > > > > Thread A holds a mutex x contested by Thread B and C and has priority > > > > pri(A). > > > > > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > > > > > Thread A releases the lock wakes thread B but lets C on the turnstile > > > > wait queue. > > > > > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > > > > > However priority inheritance does not work since B needs to run first > > > > to take ownership of the lock. > > > > > > > > I is blocked :-( > > > > > > Ermm, if the interrupt happens after x is released then I's priority > > > should propagate from I to C to B. > > > > There is a hole after the mutex x is released by A - but before B can > > claim the mutex. The turnstile for mutex x is unowned and interrupt > > thread I when trying to donate its priority will run into: > > > > if (td == NULL) { > > /* > > * This really isn't quite right. Really > > * ought to bump priority of thread that > > * next acquires the lock. > > */ > > return; > > } > > > > So B needs to run and acquire the mutex before priority inheritance > > works again and does not get a priority boost to do so. > > > > This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time > > - but my time budget is limited and Peter has an interesting bug left > > that has priority. > > Isn't this handled by the mtx_lock == MTX_CONTESTED case that calls into > turnstile_claim() which bumps the priority of the new owner to the highest > priority waiting thread? I guess this won't happen until B gets to run again > which is the problem. You don't know which thread is going to get the lock, > so what do you do? You don't even have a way to get to the threads that you > might have just woken up. The solution is for A not to release the lock but to re-assign it to B. However I have the feeling there will be some (bad?) interaction with adaptive mutexes and did not have time to think about it. > BTW, Solaris uses MUTEX_WAKE_ALL by default, but for performance reasons. It > is a kernel option because the idea was to benchmark it both ways and then > choose the default based on those numbers. It's off by default as the > wake-one was the original behavior. I'm pretty sure BSD/OS has this same > issue. From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 20:58:26 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 0C07516A4D2 for ; Mon, 4 Oct 2004 20:58:25 +0000 (GMT) Received: from mail1.speakeasy.net (mail1.speakeasy.net [216.254.0.201]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8D6F43D49 for ; Mon, 4 Oct 2004 20:58:25 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 7301 invoked from network); 4 Oct 2004 20:58:25 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 4 Oct 2004 20:58:24 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i94KwJdk059159; Mon, 4 Oct 2004 16:58:21 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-arch@FreeBSD.org Date: Mon, 4 Oct 2004 16:57:49 -0400 User-Agent: KMail/1.6.2 References: <1095468747.31297.241.camel@palm.tree.com> <200410041403.06187.jhb@FreeBSD.org> <1096922414.45640.6.camel@palm.tree.com> In-Reply-To: <1096922414.45640.6.camel@palm.tree.com> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410041657.49278.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Julian Elischer cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 20:58:26 -0000 On Monday 04 October 2004 04:40 pm, Stephan Uphoff wrote: > On Mon, 2004-10-04 at 14:03, John Baldwin wrote: > > On Monday 04 October 2004 01:34 pm, Stephan Uphoff wrote: > > > On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > > > > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > > > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > > > > I was looking at the MUTEX_WAKE_ALL undefined case when I used > > > > > > the critical section for turnstile_claim(). > > > > > > However there are bigger problems with MUTEX_WAKE_ALL undefined > > > > > > so you are right - the critical section for turnstile_claim is > > > > > > pretty useless. > > > > > > > > > > Arghhh !!! > > > > > > > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > > > > I recall verifying that it is defined twice. Guess I must have > > > > > looked at the wrong source tree :-( > > > > > This means yes - we have bigger problems! > > > > > > > > > > Example: > > > > > > > > > > Thread A holds a mutex x contested by Thread B and C and has > > > > > priority pri(A). > > > > > > > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > > > > > > > Thread A releases the lock wakes thread B but lets C on the > > > > > turnstile wait queue. > > > > > > > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > > > > > > > However priority inheritance does not work since B needs to run > > > > > first to take ownership of the lock. > > > > > > > > > > I is blocked :-( > > > > > > > > Ermm, if the interrupt happens after x is released then I's priority > > > > should propagate from I to C to B. > > > > > > There is a hole after the mutex x is released by A - but before B can > > > claim the mutex. The turnstile for mutex x is unowned and interrupt > > > thread I when trying to donate its priority will run into: > > > > > > if (td == NULL) { > > > /* > > > * This really isn't quite right. Really > > > * ought to bump priority of thread that > > > * next acquires the lock. > > > */ > > > return; > > > } > > > > > > So B needs to run and acquire the mutex before priority inheritance > > > works again and does not get a priority boost to do so. > > > > > > This is easy to fix and MUTEX_WAKE_ALL can be removed again at that > > > time - but my time budget is limited and Peter has an interesting bug > > > left that has priority. > > > > Isn't this handled by the mtx_lock == MTX_CONTESTED case that calls into > > turnstile_claim() which bumps the priority of the new owner to the > > highest priority waiting thread? I guess this won't happen until B gets > > to run again which is the problem. You don't know which thread is going > > to get the lock, so what do you do? You don't even have a way to get to > > the threads that you might have just woken up. > > The solution is for A not to release the lock but to re-assign it to B. > However I have the feeling there will be some (bad?) interaction with > adaptive mutexes and did not have time to think about it. Yes, for adaptive mutexes this breaks because you don't know who is adaptively spinning on it to compare their priorities to know who to give the lock to. Threads on other CPUs that are trying to acquire the lock at the same time have the same problem even w/o adaptive mutexes. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 21:18:21 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D532F16A4CF for ; Mon, 4 Oct 2004 21:18:21 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 800BF43D45 for ; Mon, 4 Oct 2004 21:18:21 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 4101 invoked by uid 89); 4 Oct 2004 21:18:20 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 4 Oct 2004 21:18:20 -0000 Received: (qmail 4080 invoked by uid 89); 4 Oct 2004 21:18:19 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 4 Oct 2004 21:18:19 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i94LIImt045876; Mon, 4 Oct 2004 17:18:18 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: John Baldwin In-Reply-To: <200410041657.49278.jhb@FreeBSD.org> References: <1095468747.31297.241.camel@palm.tree.com> <200410041403.06187.jhb@FreeBSD.org> <1096922414.45640.6.camel@palm.tree.com> <200410041657.49278.jhb@FreeBSD.org> Content-Type: text/plain Message-Id: <1096924698.45640.22.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Mon, 04 Oct 2004 17:18:18 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 21:18:22 -0000 On Mon, 2004-10-04 at 16:57, John Baldwin wrote: > On Monday 04 October 2004 04:40 pm, Stephan Uphoff wrote: > > On Mon, 2004-10-04 at 14:03, John Baldwin wrote: > > > On Monday 04 October 2004 01:34 pm, Stephan Uphoff wrote: > > > > On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > > > > > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > > > > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > > > > > I was looking at the MUTEX_WAKE_ALL undefined case when I used > > > > > > > the critical section for turnstile_claim(). > > > > > > > However there are bigger problems with MUTEX_WAKE_ALL undefined > > > > > > > so you are right - the critical section for turnstile_claim is > > > > > > > pretty useless. > > > > > > > > > > > > Arghhh !!! > > > > > > > > > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > > > > > I recall verifying that it is defined twice. Guess I must have > > > > > > looked at the wrong source tree :-( > > > > > > This means yes - we have bigger problems! > > > > > > > > > > > > Example: > > > > > > > > > > > > Thread A holds a mutex x contested by Thread B and C and has > > > > > > priority pri(A). > > > > > > > > > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > > > > > > > > > Thread A releases the lock wakes thread B but lets C on the > > > > > > turnstile wait queue. > > > > > > > > > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > > > > > > > > > However priority inheritance does not work since B needs to run > > > > > > first to take ownership of the lock. > > > > > > > > > > > > I is blocked :-( > > > > > > > > > > Ermm, if the interrupt happens after x is released then I's priority > > > > > should propagate from I to C to B. > > > > > > > > There is a hole after the mutex x is released by A - but before B can > > > > claim the mutex. The turnstile for mutex x is unowned and interrupt > > > > thread I when trying to donate its priority will run into: > > > > > > > > if (td == NULL) { > > > > /* > > > > * This really isn't quite right. Really > > > > * ought to bump priority of thread that > > > > * next acquires the lock. > > > > */ > > > > return; > > > > } > > > > > > > > So B needs to run and acquire the mutex before priority inheritance > > > > works again and does not get a priority boost to do so. > > > > > > > > This is easy to fix and MUTEX_WAKE_ALL can be removed again at that > > > > time - but my time budget is limited and Peter has an interesting bug > > > > left that has priority. > > > > > > Isn't this handled by the mtx_lock == MTX_CONTESTED case that calls into > > > turnstile_claim() which bumps the priority of the new owner to the > > > highest priority waiting thread? I guess this won't happen until B gets > > > to run again which is the problem. You don't know which thread is going > > > to get the lock, so what do you do? You don't even have a way to get to > > > the threads that you might have just woken up. > > > > The solution is for A not to release the lock but to re-assign it to B. > > However I have the feeling there will be some (bad?) interaction with > > adaptive mutexes and did not have time to think about it. > > Yes, for adaptive mutexes this breaks because you don't know who is adaptively > spinning on it to compare their priorities to know who to give the lock to. Yep. > > Threads on other CPUs that are trying to acquire the lock at the same time > have the same problem even w/o adaptive mutexes. Not with the right locking. They will just donate their priority to B. We could improve this by implement "lock stealing" where another thread with higher priority steals the lock before B becomes aware that he owns it. Stephan From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 21:37:23 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 82D0C16A4CE for ; Mon, 4 Oct 2004 21:37:23 +0000 (GMT) Received: from mail3.speakeasy.net (mail3.speakeasy.net [216.254.0.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C75743D3F for ; Mon, 4 Oct 2004 21:37:23 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 13144 invoked from network); 4 Oct 2004 21:37:22 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 4 Oct 2004 21:37:22 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i94LbH8w059379; Mon, 4 Oct 2004 17:37:18 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-arch@FreeBSD.org Date: Mon, 4 Oct 2004 17:35:39 -0400 User-Agent: KMail/1.6.2 References: <1095468747.31297.241.camel@palm.tree.com> <200410041657.49278.jhb@FreeBSD.org> <1096924698.45640.22.camel@palm.tree.com> In-Reply-To: <1096924698.45640.22.camel@palm.tree.com> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410041735.39472.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Julian Elischer cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 21:37:23 -0000 On Monday 04 October 2004 05:18 pm, Stephan Uphoff wrote: > On Mon, 2004-10-04 at 16:57, John Baldwin wrote: > > On Monday 04 October 2004 04:40 pm, Stephan Uphoff wrote: > > > On Mon, 2004-10-04 at 14:03, John Baldwin wrote: > > > > On Monday 04 October 2004 01:34 pm, Stephan Uphoff wrote: > > > > > On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > > > > > > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > > > > > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > > > > > > I was looking at the MUTEX_WAKE_ALL undefined case when I > > > > > > > > used the critical section for turnstile_claim(). > > > > > > > > However there are bigger problems with MUTEX_WAKE_ALL > > > > > > > > undefined so you are right - the critical section for > > > > > > > > turnstile_claim is pretty useless. > > > > > > > > > > > > > > Arghhh !!! > > > > > > > > > > > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > > > > > > I recall verifying that it is defined twice. Guess I must have > > > > > > > looked at the wrong source tree :-( > > > > > > > This means yes - we have bigger problems! > > > > > > > > > > > > > > Example: > > > > > > > > > > > > > > Thread A holds a mutex x contested by Thread B and C and has > > > > > > > priority pri(A). > > > > > > > > > > > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > > > > > > > > > > > Thread A releases the lock wakes thread B but lets C on the > > > > > > > turnstile wait queue. > > > > > > > > > > > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > > > > > > > > > > > However priority inheritance does not work since B needs to run > > > > > > > first to take ownership of the lock. > > > > > > > > > > > > > > I is blocked :-( > > > > > > > > > > > > Ermm, if the interrupt happens after x is released then I's > > > > > > priority should propagate from I to C to B. > > > > > > > > > > There is a hole after the mutex x is released by A - but before B > > > > > can claim the mutex. The turnstile for mutex x is unowned and > > > > > interrupt thread I when trying to donate its priority will run > > > > > into: > > > > > > > > > > if (td == NULL) { > > > > > /* > > > > > * This really isn't quite right. Really > > > > > * ought to bump priority of thread that > > > > > * next acquires the lock. > > > > > */ > > > > > return; > > > > > } > > > > > > > > > > So B needs to run and acquire the mutex before priority inheritance > > > > > works again and does not get a priority boost to do so. > > > > > > > > > > This is easy to fix and MUTEX_WAKE_ALL can be removed again at that > > > > > time - but my time budget is limited and Peter has an interesting > > > > > bug left that has priority. > > > > > > > > Isn't this handled by the mtx_lock == MTX_CONTESTED case that calls > > > > into turnstile_claim() which bumps the priority of the new owner to > > > > the highest priority waiting thread? I guess this won't happen until > > > > B gets to run again which is the problem. You don't know which > > > > thread is going to get the lock, so what do you do? You don't even > > > > have a way to get to the threads that you might have just woken up. > > > > > > The solution is for A not to release the lock but to re-assign it to B. > > > However I have the feeling there will be some (bad?) interaction with > > > adaptive mutexes and did not have time to think about it. > > > > Yes, for adaptive mutexes this breaks because you don't know who is > > adaptively spinning on it to compare their priorities to know who to give > > the lock to. > > Yep. Actually, note that they will stop spinning when they notice the owner change currently, so you could still use the lock == MTX_CONTESTED trick to get the adaptive waiters to stop spinning. Actually, giving the lock away explicitly will also make them stop spinning. You just might not pick the absolute best candidate to give it to. > > Threads on other CPUs that are trying to acquire the lock at the same > > time have the same problem even w/o adaptive mutexes. > > Not with the right locking. Err, if there is a solution for this case then it will fix the adaptive case as well; they are the same problem. > They will just donate their priority to B. > We could improve this by implement "lock stealing" where another thread > with higher priority steals the lock before B becomes aware that he owns > it. The problem is that there is still a window before they can donate their priority to B, which is the same window as you mentioned before, though there is a slight difference in that these threads are executing and might have a shot at the lock. If they don't get it they will be waiting on the turnstile lock before they can lend their priority to the new owner. So I guess what you would like to do is in propigate_priority(), when the thread pointer is NULL, you want to take the highest priority waiter and give the lock to it. Of course, there might not be any waiters in the adaptive case, so you can never fully eliminate the race. I wouldn't mind turning MUTEX_WAKE_ALL on by default instead if that makes a difference. -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Mon Oct 4 22:27:03 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 676DC16A4D8 for ; Mon, 4 Oct 2004 22:27:03 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 9863C43D53 for ; Mon, 4 Oct 2004 22:27:02 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 13329 invoked by uid 89); 4 Oct 2004 22:27:01 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 4 Oct 2004 22:27:01 -0000 Received: (qmail 13316 invoked by uid 89); 4 Oct 2004 22:27:01 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 4 Oct 2004 22:27:01 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i94MQxmt046158; Mon, 4 Oct 2004 18:26:59 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: John Baldwin In-Reply-To: <200410041735.39472.jhb@FreeBSD.org> References: <1095468747.31297.241.camel@palm.tree.com> <200410041657.49278.jhb@FreeBSD.org> <1096924698.45640.22.camel@palm.tree.com> <200410041735.39472.jhb@FreeBSD.org> Content-Type: text/plain Message-Id: <1096928819.45640.65.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Mon, 04 Oct 2004 18:26:59 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 04 Oct 2004 22:27:03 -0000 On Mon, 2004-10-04 at 17:35, John Baldwin wrote: > On Monday 04 October 2004 05:18 pm, Stephan Uphoff wrote: > > On Mon, 2004-10-04 at 16:57, John Baldwin wrote: > > > On Monday 04 October 2004 04:40 pm, Stephan Uphoff wrote: > > > > On Mon, 2004-10-04 at 14:03, John Baldwin wrote: > > > > > On Monday 04 October 2004 01:34 pm, Stephan Uphoff wrote: > > > > > > On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > > > > > > > On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > > > > > > > > On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > > > > > > > > > I was looking at the MUTEX_WAKE_ALL undefined case when I > > > > > > > > > used the critical section for turnstile_claim(). > > > > > > > > > However there are bigger problems with MUTEX_WAKE_ALL > > > > > > > > > undefined so you are right - the critical section for > > > > > > > > > turnstile_claim is pretty useless. > > > > > > > > > > > > > > > > Arghhh !!! > > > > > > > > > > > > > > > > MUTEX_WAKE_ALL is NOT an option in GENERIC. > > > > > > > > I recall verifying that it is defined twice. Guess I must have > > > > > > > > looked at the wrong source tree :-( > > > > > > > > This means yes - we have bigger problems! > > > > > > > > > > > > > > > > Example: > > > > > > > > > > > > > > > > Thread A holds a mutex x contested by Thread B and C and has > > > > > > > > priority pri(A). > > > > > > > > > > > > > > > > Thread C holds a mutex y and pri(B) < pri(C) > > > > > > > > > > > > > > > > Thread A releases the lock wakes thread B but lets C on the > > > > > > > > turnstile wait queue. > > > > > > > > > > > > > > > > An interrupt thread I tries to lock mutex y owned by C. > > > > > > > > > > > > > > > > However priority inheritance does not work since B needs to run > > > > > > > > first to take ownership of the lock. > > > > > > > > > > > > > > > > I is blocked :-( > > > > > > > > > > > > > > Ermm, if the interrupt happens after x is released then I's > > > > > > > priority should propagate from I to C to B. > > > > > > > > > > > > There is a hole after the mutex x is released by A - but before B > > > > > > can claim the mutex. The turnstile for mutex x is unowned and > > > > > > interrupt thread I when trying to donate its priority will run > > > > > > into: > > > > > > > > > > > > if (td == NULL) { > > > > > > /* > > > > > > * This really isn't quite right. Really > > > > > > * ought to bump priority of thread that > > > > > > * next acquires the lock. > > > > > > */ > > > > > > return; > > > > > > } > > > > > > > > > > > > So B needs to run and acquire the mutex before priority inheritance > > > > > > works again and does not get a priority boost to do so. > > > > > > > > > > > > This is easy to fix and MUTEX_WAKE_ALL can be removed again at that > > > > > > time - but my time budget is limited and Peter has an interesting > > > > > > bug left that has priority. > > > > > > > > > > Isn't this handled by the mtx_lock == MTX_CONTESTED case that calls > > > > > into turnstile_claim() which bumps the priority of the new owner to > > > > > the highest priority waiting thread? I guess this won't happen until > > > > > B gets to run again which is the problem. You don't know which > > > > > thread is going to get the lock, so what do you do? You don't even > > > > > have a way to get to the threads that you might have just woken up. > > > > > > > > The solution is for A not to release the lock but to re-assign it to B. > > > > However I have the feeling there will be some (bad?) interaction with > > > > adaptive mutexes and did not have time to think about it. > > > > > > Yes, for adaptive mutexes this breaks because you don't know who is > > > adaptively spinning on it to compare their priorities to know who to give > > > the lock to. > > > > Yep. > > Actually, note that they will stop spinning when they notice the owner change > currently, so you could still use the lock == MTX_CONTESTED trick to get the > adaptive waiters to stop spinning. Actually, giving the lock away explicitly > will also make them stop spinning. You just might not pick the absolute best > candidate to give it to. > > > > Threads on other CPUs that are trying to acquire the lock at the same > > > time have the same problem even w/o adaptive mutexes. > > > > Not with the right locking. > > Err, if there is a solution for this case then it will fix the adaptive case > as well; they are the same problem. > > > They will just donate their priority to B. > > We could improve this by implement "lock stealing" where another thread > > with higher priority steals the lock before B becomes aware that he owns > > it. > > The problem is that there is still a window before they can donate their > priority to B, which is the same window as you mentioned before, though there > is a slight difference in that these threads are executing and might have a > shot at the lock. If they don't get it they will be waiting on the turnstile > lock before they can lend their priority to the new owner. > > So I guess what you would like to do is in propigate_priority(), when the > thread pointer is NULL, you want to take the highest priority waiter and give > the lock to it. Of course, there might not be any waiters in the adaptive > case, so you can never fully eliminate the race. I wouldn't mind turning > MUTEX_WAKE_ALL on by default instead if that makes a difference. No - I would like to eliminate the case where propagate_priority finds a NULL pointer by giving the thread A that is releasing the mutex the responsibility to assign both mutex and turnstile to B. ( and even recalculate B's priority should the implementation requires - but I doubt it) A runs in a critical region while this is going on and the relevant turnstile locks are held. B would just wake up and find out that it owns the mutex. As an optimization one could for a higher priority thread later steal the mutex from B before it gets a chance to run. Stephan From owner-freebsd-arch@FreeBSD.ORG Tue Oct 5 03:12:43 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 2681D16A4CF for ; Tue, 5 Oct 2004 03:12:43 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 966AF43D4C for ; Tue, 5 Oct 2004 03:12:42 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 4806 invoked by uid 89); 5 Oct 2004 03:12:41 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 5 Oct 2004 03:12:41 -0000 Received: (qmail 4772 invoked by uid 89); 5 Oct 2004 03:12:41 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 5 Oct 2004 03:12:41 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i953Ccmt047364; Mon, 4 Oct 2004 23:12:38 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: Peter Holm In-Reply-To: <20041004184939.GA8178@peter.osted.lan> References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> Content-Type: text/plain Message-Id: <1096945958.46385.44.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Mon, 04 Oct 2004 23:12:38 -0400 Content-Transfer-Encoding: 7bit cc: Julian Elischer cc: John Baldwin cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Oct 2004 03:12:43 -0000 On Mon, 2004-10-04 at 14:49, Peter Holm wrote: > On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: ---- snip ---- > > - but my time budget is limited and Peter has an interesting bug left > > that has priority. > > I'm not closer to being able to create this panic in a controlled way. > After a whole day of different tests I finally got this panic: > http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one > particular Java applet, but it is not easily reproduceable. > > - Peter ---- snip ---- I found a race condition in sleepq_catch_signals / sleepq_resume_thread that may cause sleepq_resume_thread to add a thread to the run queue that is already there. This could explain your crash. Shouldn't be to hard to fix but I am burned out today :-(. Stephan From owner-freebsd-arch@FreeBSD.ORG Tue Oct 5 13:03:18 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id DB1BC16A4D3 for ; Tue, 5 Oct 2004 13:03:18 +0000 (GMT) Received: from relay.pair.com (relay.pair.com [209.68.1.20]) by mx1.FreeBSD.org (Postfix) with SMTP id DBA8143D46 for ; Tue, 5 Oct 2004 13:03:13 +0000 (GMT) (envelope-from pho@holm.cc) Received: (qmail 2737 invoked from network); 5 Oct 2004 13:03:11 -0000 Received: from 0x50a43fc7.hknxx1.adsl-dhcp.tele.dk (HELO peter.osted.lan) (80.164.63.199) by relay.pair.com with SMTP; 5 Oct 2004 13:03:11 -0000 X-pair-Authenticated: 80.164.63.199 Received: from peter.osted.lan (localhost.osted.lan [127.0.0.1]) by peter.osted.lan (8.12.10/8.12.10) with ESMTP id i95D3AXc002744; Tue, 5 Oct 2004 15:03:10 +0200 (CEST) (envelope-from pho@peter.osted.lan) Received: (from pho@localhost) by peter.osted.lan (8.12.10/8.12.10/Submit) id i95D39sw002743; Tue, 5 Oct 2004 15:03:09 +0200 (CEST) (envelope-from pho) Date: Tue, 5 Oct 2004 15:03:08 +0200 From: Peter Holm To: Julian Elischer Message-ID: <20041005130308.GA2586@peter.osted.lan> References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> <41619D29.1000704@elischer.org> <20041004191410.GA8423@peter.osted.lan> <4161A7BD.3040706@elischer.org> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="NzB8fVQJ5HfG6fxh" Content-Disposition: inline In-Reply-To: <4161A7BD.3040706@elischer.org> User-Agent: Mutt/1.4.1i cc: Peter Holm cc: Stephan Uphoff cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Oct 2004 13:03:19 -0000 --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii Content-Disposition: inline On Mon, Oct 04, 2004 at 12:42:53PM -0700, Julian Elischer wrote: OK, I got a crash dump now, after a few modifications to kern_shutdown.c There are however a few strange things worth noticing: 1) The are no panic string: Mounted root from ufs:/dev/ad0s1a. pid 1146: corrected slot count (2->1) [thread 100796] Stopped at sched_add+0x13: movl 0x14c(%esi),%ebx 2) The gdb stack trace gets a bit weird at: #8 0xc07812da in calltrap () at ../../../i386/i386/exception.s:140 #9 0xc05f0018 in flock (td=0x0, uap=0x0) at ../../../kern/kern_descrip.c:2138 #10 0xc0619fd1 in setrunqueue (td=0xc2319180, flags=0x0) at kern_switch.c:521 #11 0xc061921f in sched_wakeup (td=0xc2319180) at ../../../kern/sched_4bsd.c:859 Where did flock() come from? The full console output is at http://www.holm.cc/stress/log/cons82.html - Peter > ok, then if it happens again, from ddb, run > show ktr > after you've done the 'ps' and go back a couple of hundred events.. > > thanks. > > > Peter Holm wrote: > > >On Mon, Oct 04, 2004 at 11:57:45AM -0700, Julian Elischer wrote: > > > > > >>can you run ktrdump against teh corefile and get the ktr output? > >>(you do have it enabled right?) > >> > >> > >> > > > >No, that's one of the problems: doadump() fails with this specific panic. > > > >- Peter > > > > > > > >>Peter Holm wrote: > >> > >> > >> > >>>On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: > >>> > >>> > >>> > >>> > >>>>On Mon, 2004-10-04 at 11:31, John Baldwin wrote: > >>>> > >>>> > >>>> > >>>> > >>>>>On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: > >>>>> > >>>>> > >>>>> > >>>>> > >>>>>>On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>>I was looking at the MUTEX_WAKE_ALL undefined case when I used the > >>>>>>>critical section for turnstile_claim(). > >>>>>>>However there are bigger problems with MUTEX_WAKE_ALL undefined > >>>>>>>so you are right - the critical section for turnstile_claim is pretty > >>>>>>>useless. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>Arghhh !!! > >>>>>> > >>>>>>MUTEX_WAKE_ALL is NOT an option in GENERIC. > >>>>>>I recall verifying that it is defined twice. Guess I must have looked > >>>>>>at > >>>>>>the wrong source tree :-( > >>>>>>This means yes - we have bigger problems! > >>>>>> > >>>>>>Example: > >>>>>> > >>>>>>Thread A holds a mutex x contested by Thread B and C and has priority > >>>>>>pri(A). > >>>>>> > >>>>>>Thread C holds a mutex y and pri(B) < pri(C) > >>>>>> > >>>>>>Thread A releases the lock wakes thread B but lets C on the turnstile > >>>>>>wait queue. > >>>>>> > >>>>>>An interrupt thread I tries to lock mutex y owned by C. > >>>>>> > >>>>>>However priority inheritance does not work since B needs to run first > >>>>>>to > >>>>>>take ownership of the lock. > >>>>>> > >>>>>>I is blocked :-( > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>Ermm, if the interrupt happens after x is released then I's priority > >>>>>should propagate from I to C to B. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>There is a hole after the mutex x is released by A - but before B can > >>>>claim the mutex. The turnstile for mutex x is unowned and interrupt > >>>>thread I when trying to donate its priority will run into: > >>>> > >>>> if (td == NULL) { > >>>> /* > >>>> * This really isn't quite right. Really > >>>> * ought to bump priority of thread that > >>>> * next acquires the lock. > >>>> */ > >>>> return; > >>>> } > >>>> > >>>>So B needs to run and acquire the mutex before priority inheritance > >>>>works again and does not get a priority boost to do so. > >>>> > >>>>This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time > >>>>- but my time budget is limited and Peter has an interesting bug left > >>>>that has priority. > >>>> > >>>> > >>>> > >>>> > >>>I'm not closer to being able to create this panic in a controlled way. > >>>After a whole day of different tests I finally got this panic: > >>>http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one > >>>particular Java applet, but it is not easily reproduceable. > >>> > >>>- Peter > >>> > >>> > >>> > >>> > >>> > >>>>>If the interrupt happens before x is released, > >>>>>then the final bit of propagate_priority() should handle it since it > >>>>>resorts the turnstile's thread queue so that C will be awakened rather > >>>>>than B. > >>>>> > >>>>> > >>>>> > >>>>> > >>>>Agreed. > >>>> > >>>> Stephan > >>>> > >>>> > >>>> > >>>> > >>>_______________________________________________ > >>>freebsd-arch@freebsd.org mailing list > >>>http://lists.freebsd.org/mailman/listinfo/freebsd-arch > >>>To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" > >>> > >>> > >>> > >>> > > > > > > -- Peter Holm --NzB8fVQJ5HfG6fxh Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="kern_shutdown.diff" Index: kern_shutdown.c =================================================================== RCS file: /home/ncvs/src/sys/kern/kern_shutdown.c,v retrieving revision 1.166 diff -u -r1.166 kern_shutdown.c --- kern_shutdown.c 2 Sep 2004 18:59:15 -0000 1.166 +++ kern_shutdown.c 5 Oct 2004 12:23:45 -0000 @@ -230,10 +230,14 @@ return; } + if (panicstr == NULL) + panicstr = "In doadump()"; /* Major hack XXX pho */ savectx(&dumppcb); dumptid = curthread->td_tid; dumping++; dumpsys(&dumper); + if (!strcmp(panicstr, "In doadump()")) + panicstr = NULL; /* Major hack XXX pho */ } /* @@ -519,6 +523,8 @@ #endif #ifdef KDB + if (panicstr == NULL) + panicstr = "(NULL)"; /* XXX pho */ if (newpanic && trace_on_panic) kdb_backtrace(); if (debugger_on_panic) --NzB8fVQJ5HfG6fxh-- From owner-freebsd-arch@FreeBSD.ORG Tue Oct 5 16:26:29 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id EBC7E16A4CE for ; Tue, 5 Oct 2004 16:26:29 +0000 (GMT) Received: from mail3.speakeasy.net (mail3.speakeasy.net [216.254.0.203]) by mx1.FreeBSD.org (Postfix) with ESMTP id C2EEE43D39 for ; Tue, 5 Oct 2004 16:26:29 +0000 (GMT) (envelope-from jhb@FreeBSD.org) Received: (qmail 9851 invoked from network); 5 Oct 2004 16:26:29 -0000 Received: from dsl027-160-063.atl1.dsl.speakeasy.net (HELO server.baldwin.cx) ([216.27.160.63]) (envelope-sender ) encrypted SMTP for ; 5 Oct 2004 16:26:28 -0000 Received: from [10.50.40.210] (gw1.twc.weather.com [216.133.140.1]) (authenticated bits=0) by server.baldwin.cx (8.12.11/8.12.11) with ESMTP id i95GQN0l065355; Tue, 5 Oct 2004 12:26:24 -0400 (EDT) (envelope-from jhb@FreeBSD.org) From: John Baldwin To: freebsd-arch@FreeBSD.org Date: Tue, 5 Oct 2004 10:00:15 -0400 User-Agent: KMail/1.6.2 References: <1095468747.31297.241.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> <1096945958.46385.44.camel@palm.tree.com> In-Reply-To: <1096945958.46385.44.camel@palm.tree.com> MIME-Version: 1.0 Content-Disposition: inline Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Message-Id: <200410051000.15379.jhb@FreeBSD.org> X-Spam-Checker-Version: SpamAssassin 2.63 (2004-01-11) on server.baldwin.cx cc: Peter Holm cc: "freebsd-arch@freebsd.org" cc: Julian Elischer cc: Stephan Uphoff Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Oct 2004 16:26:30 -0000 On Monday 04 October 2004 11:12 pm, Stephan Uphoff wrote: > On Mon, 2004-10-04 at 14:49, Peter Holm wrote: > > On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: > > ---- snip ---- > > > > - but my time budget is limited and Peter has an interesting bug left > > > that has priority. > > > > I'm not closer to being able to create this panic in a controlled way. > > After a whole day of different tests I finally got this panic: > > http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one > > particular Java applet, but it is not easily reproduceable. > > > > - Peter > > ---- snip ---- > > I found a race condition in sleepq_catch_signals / sleepq_resume_thread > that may cause sleepq_resume_thread to add a thread to the run queue > that is already there. Note that setrunnable() checks the thread state and doesn't call setrunqueue if the thread is already on the runqueue (see the TDS_RUNQ case in the switch statement). Is this what you are referring to? -- John Baldwin <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org From owner-freebsd-arch@FreeBSD.ORG Tue Oct 5 18:32:18 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id E7EF716A4CE for ; Tue, 5 Oct 2004 18:32:17 +0000 (GMT) Received: from mail.vicor-nb.com (bigwoop.vicor-nb.com [208.206.78.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id A8E8943D1D for ; Tue, 5 Oct 2004 18:32:17 +0000 (GMT) (envelope-from julian@elischer.org) Received: from elischer.org (julian.vicor-nb.com [208.206.78.97]) by mail.vicor-nb.com (Postfix) with ESMTP id 223EF7A457; Tue, 5 Oct 2004 11:32:17 -0700 (PDT) Message-ID: <4162E8B1.90803@elischer.org> Date: Tue, 05 Oct 2004 11:32:17 -0700 From: Julian Elischer User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.3.1) Gecko/20030516 X-Accept-Language: en, hu MIME-Version: 1.0 To: Peter Holm References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> <41619D29.1000704@elischer.org> <20041004191410.GA8423@peter.osted.lan> <4161A7BD.3040706@elischer.org> <20041005130308.GA2586@peter.osted.lan> In-Reply-To: <20041005130308.GA2586@peter.osted.lan> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit cc: Stephan Uphoff cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Oct 2004 18:32:18 -0000 Peter Holm wrote: >On Mon, Oct 04, 2004 at 12:42:53PM -0700, Julian Elischer wrote: > >OK, I got a crash dump now, after a few modifications to kern_shutdown.c > >There are however a few strange things worth noticing: > >1) The are no panic string: > >Mounted root from ufs:/dev/ad0s1a. >pid 1146: corrected slot count (2->1) >[thread 100796] >Stopped at sched_add+0x13: movl 0x14c(%esi),%ebx > >2) The gdb stack trace gets a bit weird at: > >#8 0xc07812da in calltrap () at ../../../i386/i386/exception.s:140 >#9 0xc05f0018 in flock (td=0x0, uap=0x0) at ../../../kern/kern_descrip.c:2138 >#10 0xc0619fd1 in setrunqueue (td=0xc2319180, flags=0x0) at kern_switch.c:521 >#11 0xc061921f in sched_wakeup (td=0xc2319180) at ../../../kern/sched_4bsd.c:859 > >Where did flock() come from? > probably just a partially initialised frame.. ddb seems to have a good trace, starting at setrunqueue(). there are two things to notice.. firstly the "corrected slot count (2->1)" messge is still there. (grumble). this is hapenning when a threade dprocess moves back to be ing an unthreaded preocess. for some reason, the number of openning s is not being set back to 1 but rather to 2. I believe it is because while in thhe threaded mode it is already too high by some amount (sometimes equivalent to NTHREAD) but I can not see why. Hopefully it is not a fatal problem (as it would be if it were too LOW, but I hope to figure it out soon (maybe another one for Stephan :-) On the topic of the crash. the ktr shows no unexpected activity in the time before the crash.... no preemption, or similar.. it might be possible that there was an interrupt, but there is nothing htath the ktr mask used shows.. maybe you could compile in and use a few more bits in the ktr masks to show process events and interrupts In the absence of unexpected happennings we must assume the kseg runq is in an odd state before it gets used in setrunqueue, leading to the panic.. I think I will check in some debug and cleanup stuff I have here.. maybe it will shake out something.. > >The full console output is at http://www.holm.cc/stress/log/cons82.html > >- Peter > > > >>ok, then if it happens again, from ddb, run >>show ktr >>after you've done the 'ps' and go back a couple of hundred events.. >> >>thanks. >> >> >>Peter Holm wrote: >> >> >> >>>On Mon, Oct 04, 2004 at 11:57:45AM -0700, Julian Elischer wrote: >>> >>> >>> >>> >>>>can you run ktrdump against teh corefile and get the ktr output? >>>>(you do have it enabled right?) >>>> >>>> >>>> >>>> >>>> >>>No, that's one of the problems: doadump() fails with this specific panic. >>> >>>- Peter >>> >>> >>> >>> >>> >>>>Peter Holm wrote: >>>> >>>> >>>> >>>> >>>> >>>>>On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>On Mon, 2004-10-04 at 11:31, John Baldwin wrote: >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>>>On Friday 01 October 2004 12:13 am, Stephan Uphoff wrote: >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>>>On Wed, 2004-09-29 at 18:14, Stephan Uphoff wrote: >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>I was looking at the MUTEX_WAKE_ALL undefined case when I used the >>>>>>>>>critical section for turnstile_claim(). >>>>>>>>>However there are bigger problems with MUTEX_WAKE_ALL undefined >>>>>>>>>so you are right - the critical section for turnstile_claim is pretty >>>>>>>>>useless. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>Arghhh !!! >>>>>>>> >>>>>>>>MUTEX_WAKE_ALL is NOT an option in GENERIC. >>>>>>>>I recall verifying that it is defined twice. Guess I must have looked >>>>>>>>at >>>>>>>>the wrong source tree :-( >>>>>>>>This means yes - we have bigger problems! >>>>>>>> >>>>>>>>Example: >>>>>>>> >>>>>>>>Thread A holds a mutex x contested by Thread B and C and has priority >>>>>>>>pri(A). >>>>>>>> >>>>>>>>Thread C holds a mutex y and pri(B) < pri(C) >>>>>>>> >>>>>>>>Thread A releases the lock wakes thread B but lets C on the turnstile >>>>>>>>wait queue. >>>>>>>> >>>>>>>>An interrupt thread I tries to lock mutex y owned by C. >>>>>>>> >>>>>>>>However priority inheritance does not work since B needs to run first >>>>>>>>to >>>>>>>>take ownership of the lock. >>>>>>>> >>>>>>>>I is blocked :-( >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>Ermm, if the interrupt happens after x is released then I's priority >>>>>>>should propagate from I to C to B. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>There is a hole after the mutex x is released by A - but before B can >>>>>>claim the mutex. The turnstile for mutex x is unowned and interrupt >>>>>>thread I when trying to donate its priority will run into: >>>>>> >>>>>> if (td == NULL) { >>>>>> /* >>>>>> * This really isn't quite right. Really >>>>>> * ought to bump priority of thread that >>>>>> * next acquires the lock. >>>>>> */ >>>>>> return; >>>>>> } >>>>>> >>>>>>So B needs to run and acquire the mutex before priority inheritance >>>>>>works again and does not get a priority boost to do so. >>>>>> >>>>>>This is easy to fix and MUTEX_WAKE_ALL can be removed again at that time >>>>>>- but my time budget is limited and Peter has an interesting bug left >>>>>>that has priority. >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>I'm not closer to being able to create this panic in a controlled way. >>>>>After a whole day of different tests I finally got this panic: >>>>>http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one >>>>>particular Java applet, but it is not easily reproduceable. >>>>> >>>>>- Peter >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>>>If the interrupt happens before x is released, >>>>>>>then the final bit of propagate_priority() should handle it since it >>>>>>>resorts the turnstile's thread queue so that C will be awakened rather >>>>>>>than B. >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>Agreed. >>>>>> >>>>>> Stephan >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> >>>>>_______________________________________________ >>>>>freebsd-arch@freebsd.org mailing list >>>>>http://lists.freebsd.org/mailman/listinfo/freebsd-arch >>>>>To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>> >>> >>> > > > >------------------------------------------------------------------------ > >Index: kern_shutdown.c >=================================================================== >RCS file: /home/ncvs/src/sys/kern/kern_shutdown.c,v >retrieving revision 1.166 >diff -u -r1.166 kern_shutdown.c >--- kern_shutdown.c 2 Sep 2004 18:59:15 -0000 1.166 >+++ kern_shutdown.c 5 Oct 2004 12:23:45 -0000 >@@ -230,10 +230,14 @@ > return; > } > >+ if (panicstr == NULL) >+ panicstr = "In doadump()"; /* Major hack XXX pho */ > savectx(&dumppcb); > dumptid = curthread->td_tid; > dumping++; > dumpsys(&dumper); >+ if (!strcmp(panicstr, "In doadump()")) >+ panicstr = NULL; /* Major hack XXX pho */ > } > > /* >@@ -519,6 +523,8 @@ > #endif > > #ifdef KDB >+ if (panicstr == NULL) >+ panicstr = "(NULL)"; /* XXX pho */ > if (newpanic && trace_on_panic) > kdb_backtrace(); > if (debugger_on_panic) > > From owner-freebsd-arch@FreeBSD.ORG Tue Oct 5 20:41:26 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 50FB816A4EE for ; Tue, 5 Oct 2004 20:41:26 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 7EDAE43D3F for ; Tue, 5 Oct 2004 20:41:18 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 9425 invoked by uid 89); 5 Oct 2004 20:41:17 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 5 Oct 2004 20:41:17 -0000 Received: (qmail 9413 invoked by uid 89); 5 Oct 2004 20:41:17 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 5 Oct 2004 20:41:17 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i95KfGmt051683; Tue, 5 Oct 2004 16:41:16 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: John Baldwin In-Reply-To: <200410051000.15379.jhb@FreeBSD.org> References: <1095468747.31297.241.camel@palm.tree.com> <20041004184939.GA8178@peter.osted.lan> <1096945958.46385.44.camel@palm.tree.com> <200410051000.15379.jhb@FreeBSD.org> Content-Type: text/plain Message-Id: <1097008876.50232.476.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Tue, 05 Oct 2004 16:41:16 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: Julian Elischer cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 Oct 2004 20:41:26 -0000 On Tue, 2004-10-05 at 10:00, John Baldwin wrote: > On Monday 04 October 2004 11:12 pm, Stephan Uphoff wrote: > > On Mon, 2004-10-04 at 14:49, Peter Holm wrote: > > > On Mon, Oct 04, 2004 at 01:34:38PM -0400, Stephan Uphoff wrote: > > > > ---- snip ---- > > > > > > - but my time budget is limited and Peter has an interesting bug left > > > > that has priority. > > > > > > I'm not closer to being able to create this panic in a controlled way. > > > After a whole day of different tests I finally got this panic: > > > http://www.holm.cc/stress/log/cons81.html. The trigger seems to be one > > > particular Java applet, but it is not easily reproduceable. > > > > > > - Peter > > > > ---- snip ---- > > > > I found a race condition in sleepq_catch_signals / sleepq_resume_thread > > that may cause sleepq_resume_thread to add a thread to the run queue > > that is already there. > > Note that setrunnable() checks the thread state and doesn't call setrunqueue > if the thread is already on the runqueue (see the TDS_RUNQ case in the switch > statement). Is this what you are referring to? Yes - but I misread the TD_CLR_SLEEPING(td) macro and missed that it only makes the thread runable if the bit had been inhibited before. :-( Thanks Stephan From owner-freebsd-arch@FreeBSD.ORG Wed Oct 6 00:05:47 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from green.homeunix.org (freefall.freebsd.org [216.136.204.21]) by hub.freebsd.org (Postfix) with ESMTP id D51FD16A4CE; Wed, 6 Oct 2004 00:05:46 +0000 (GMT) Received: from green.homeunix.org (green@localhost [127.0.0.1]) by green.homeunix.org (8.13.1/8.13.1) with ESMTP id i9605km6057006; Tue, 5 Oct 2004 20:05:46 -0400 (EDT) (envelope-from green@green.homeunix.org) Received: (from green@localhost) by green.homeunix.org (8.13.1/8.13.1/Submit) id i9605kjP057005; Tue, 5 Oct 2004 20:05:46 -0400 (EDT) (envelope-from green) Date: Tue, 5 Oct 2004 20:05:45 -0400 From: Brian Fundakowski Feldman To: arch@FreeBSD.org Message-ID: <20041006000545.GG47017@green.homeunix.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.5.6i cc: phk@FreeBSD.org Subject: GEOM modules don't wait for "init" completion X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Oct 2004 00:05:47 -0000 In src/sys/geom/geom_subr.c, g_modevent(MOD_LOAD) does not wait for completion of the class's initialization before proceeding to return success to the caller (kldload(8)). This is especially problematic for mount_mfs(8)'s use of mdctl(8) -- the first mount hardly ever succeeds. Does it not seem like this line: g_post_event(g_load_class, hh, M_WAITOK, NULL); should really be: g_waitfor_event(g_load_class, hh, M_WAITOK, NULL); ? I can't really envision situations where you would at least not want to have your control device exist before returning from a kldload(2) system call. -- Brian Fundakowski Feldman \'[ FreeBSD ]''''''''''\ <> green@FreeBSD.org \ The Power to Serve! \ Opinions expressed are my own. \,,,,,,,,,,,,,,,,,,,,,,\ From owner-freebsd-arch@FreeBSD.ORG Wed Oct 6 04:09:48 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from green.homeunix.org (freefall.freebsd.org [216.136.204.21]) by hub.freebsd.org (Postfix) with ESMTP id D860016A4CE; Wed, 6 Oct 2004 04:09:47 +0000 (GMT) Received: from green.homeunix.org (green@localhost [127.0.0.1]) by green.homeunix.org (8.13.1/8.13.1) with ESMTP id i9649ilA058473; Wed, 6 Oct 2004 00:09:44 -0400 (EDT) (envelope-from green@green.homeunix.org) Received: (from green@localhost) by green.homeunix.org (8.13.1/8.13.1/Submit) id i9649dEP058472; Wed, 6 Oct 2004 00:09:39 -0400 (EDT) (envelope-from green) Date: Wed, 6 Oct 2004 00:09:38 -0400 From: Brian Fundakowski Feldman To: arch@FreeBSD.org Message-ID: <20041006040938.GH47017@green.homeunix.org> References: <20041006000545.GG47017@green.homeunix.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20041006000545.GG47017@green.homeunix.org> User-Agent: Mutt/1.5.6i cc: phk@FreeBSD.org Subject: Re: GEOM modules don't wait for "init" completion X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Oct 2004 04:09:48 -0000 On Tue, Oct 05, 2004 at 08:05:45PM -0400, Brian Fundakowski Feldman wrote: > In src/sys/geom/geom_subr.c, g_modevent(MOD_LOAD) does not wait for > completion of the class's initialization before proceeding to return > success to the caller (kldload(8)). This is especially problematic > for mount_mfs(8)'s use of mdctl(8) -- the first mount hardly ever > succeeds. Does it not seem like this line: if (cold) > g_post_event(g_load_class, hh, M_WAITOK, NULL); > should really be: if (!cold) > g_waitfor_event(g_load_class, hh, M_WAITOK, NULL); > ? > > I can't really envision situations where you would at least not want to > have your control device exist before returning from a kldload(2) system > call. Okay, that was the easy answer why not to do it... but changing it so that it does g_waitfor_event() when it's !cold seems to work fine, even if I'd prefer the logic to go somewhere deeper than that... -- Brian Fundakowski Feldman \'[ FreeBSD ]''''''''''\ <> green@FreeBSD.org \ The Power to Serve! \ Opinions expressed are my own. \,,,,,,,,,,,,,,,,,,,,,,\ From owner-freebsd-arch@FreeBSD.ORG Wed Oct 6 22:03:10 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id A385B16A4CE for ; Wed, 6 Oct 2004 22:03:10 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.136]) by mx1.FreeBSD.org (Postfix) with ESMTP id 4C29243D49 for ; Wed, 6 Oct 2004 22:03:10 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 7D696ACAFB; Thu, 7 Oct 2004 00:03:08 +0200 (CEST) Date: Thu, 7 Oct 2004 00:03:08 +0200 From: Pawel Jakub Dawidek To: freebsd-arch@freebsd.org Message-ID: <20041006220308.GP73767@darkness.comp.waw.pl> Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="jx/LfW4V5TfZLeq7" Content-Disposition: inline User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 Subject: swapoff_all() X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Oct 2004 22:03:10 -0000 --jx/LfW4V5TfZLeq7 Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Hi. As you know, or not, some GEOM classes (gmirror, graid3) set DIRTY flag on devices which are open for writing to be able to detect power failures, panics, etc. and rebuild device when needed. DIRTY flag is removed on last device close. (BTW. UFS uses simlar mechanism to detect dirty file systems.) When only file systems are mounted on those devices, there is no problem, because file systems are unmounted on shutdown. When one is using swap on mirror/raid3 things are worse, because to remove DRITY flag, he has to turn off swap devices. I introduced a workaround some time ago, which allows to remove swap partition on shutdown via rc.shutdown script. This is a workaround, so there are some issues: - One has to use shutdown(8) command, reboot(8) is not enough. - If there are still processes which use swap, this could be problematic to remove swap devices. This patch: http://people.freebsd.org/~pjd/patches/swapoff_all.patch removes all swap devices after all file systems are unmounted, so there should be no processes which use swap and it of course works even for reboot(8). I'll be grateful for reviews/tests/comments. --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --jx/LfW4V5TfZLeq7 Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBZGucForvXbEpPzQRAlSMAKCOzFeVLlOuXd20bYe+lcfI0QqMkgCgw/TI fnaoHnQQtvp/RoUrcXaZCbU= =jO5L -----END PGP SIGNATURE----- --jx/LfW4V5TfZLeq7-- From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 03:35:24 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BE29416A4CE for ; Thu, 7 Oct 2004 03:35:24 +0000 (GMT) Received: from athena.softcardsystems.com (mail.softcardsystems.com [12.34.136.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1644C43D54 for ; Thu, 7 Oct 2004 03:35:24 +0000 (GMT) (envelope-from sah@softcardsystems.com) Received: from athena (athena [12.34.136.114])i974Y5p5023088 for ; Wed, 6 Oct 2004 23:34:05 -0500 Date: Wed, 6 Oct 2004 23:34:05 -0500 (EST) From: Sam X-X-Sender: sah@athena To: freebsd-arch@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: fdisk/geom, AoE 5.2.1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 03:35:24 -0000 Hello again, I've got the AoE driver ported and running on 5.2.1. Much faster than I anticipated; the geom interface is delightful. I can read and write to the disk. I cannot, however, partition it and I'm soliciting the list for ideas as to why. If I fdisk -u an AoE device, fdisk claims all was successful, but in reality nothing gets written. Further calls to fdisk support this. I'm probably not tickling something right. Is it possible I need to grease the path for the disk partition table in geom? I didn't see anything of the like. Maybe it's a tasting issue? BTB - Is the geom(4) manpage finished in 5.3? Thoughts much appreciated. Cheers, Sam From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 05:04:08 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 4352016A4CE for ; Thu, 7 Oct 2004 05:04:08 +0000 (GMT) Received: from mail5.speakeasy.net (mail5.speakeasy.net [216.254.0.205]) by mx1.FreeBSD.org (Postfix) with ESMTP id 142AF43D45 for ; Thu, 7 Oct 2004 05:04:08 +0000 (GMT) (envelope-from jmg@hydrogen.funkthat.com) Received: (qmail 6557 invoked from network); 7 Oct 2004 05:04:07 -0000 Received: from gate.funkthat.com (HELO hydrogen.funkthat.com) ([69.17.45.168]) (envelope-sender ) by mail5.speakeasy.net (qmail-ldap-1.03) with SMTP for ; 7 Oct 2004 05:04:07 -0000 Received: from hydrogen.funkthat.com (ohwnsp@localhost.funkthat.com [127.0.0.1])i97546lb063677; Wed, 6 Oct 2004 22:04:06 -0700 (PDT) (envelope-from jmg@hydrogen.funkthat.com) Received: (from jmg@localhost) by hydrogen.funkthat.com (8.12.10/8.12.10/Submit) id i97545wT063676; Wed, 6 Oct 2004 22:04:05 -0700 (PDT) Date: Wed, 6 Oct 2004 22:04:05 -0700 From: John-Mark Gurney To: Sam Message-ID: <20041007050405.GP22681@funkthat.com> Mail-Followup-To: Sam , freebsd-arch@freebsd.org, freebsd-geom@freebsd.org References: Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.1i X-Operating-System: FreeBSD 4.2-RELEASE i386 X-PGP-Fingerprint: B7 EC EF F8 AE ED A7 31 96 7A 22 B3 D8 56 36 F4 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html cc: freebsd-geom@freebsd.org cc: freebsd-arch@freebsd.org Subject: Re: fdisk/geom, AoE 5.2.1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list Reply-To: John-Mark Gurney List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 05:04:08 -0000 Sam wrote this message on Wed, Oct 06, 2004 at 23:34 -0500: > I've got the AoE driver ported and running on > 5.2.1. Much faster than I anticipated; the geom > interface is delightful. > > I can read and write to the disk. I cannot, > however, partition it and I'm soliciting the > list for ideas as to why. > > If I fdisk -u an AoE device, fdisk claims all > was successful, but in reality nothing gets written. > Further calls to fdisk support this. > > I'm probably not tickling something right. Is it > possible I need to grease the path for the disk > partition table in geom? I didn't see anything of > the like. Maybe it's a tasting issue? On last close (from fdisk hopefully), the provider will be passed back for tasting since the meta data could possibly have changed.. Have you verified that the writes are hitting the disk? and that the fdisk table can be seen from another machine? Verified that you can read what you write to the disk? /me notes that these questions should probably end up on -geom and so cc'd it. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 09:40:00 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 76F0216A4CE; Thu, 7 Oct 2004 09:40:00 +0000 (GMT) Received: from darkness.comp.waw.pl (darkness.comp.waw.pl [195.117.238.136]) by mx1.FreeBSD.org (Postfix) with ESMTP id F0D0543D1F; Thu, 7 Oct 2004 09:39:59 +0000 (GMT) (envelope-from pjd@darkness.comp.waw.pl) Received: by darkness.comp.waw.pl (Postfix, from userid 1009) id 33321ACABF; Thu, 7 Oct 2004 11:39:52 +0200 (CEST) Date: Thu, 7 Oct 2004 11:39:52 +0200 From: Pawel Jakub Dawidek To: Sam Message-ID: <20041007093952.GZ73767@darkness.comp.waw.pl> References: Mime-Version: 1.0 Content-Type: multipart/signed; micalg=pgp-sha1; protocol="application/pgp-signature"; boundary="utG8zEvBNE7w4yrA" Content-Disposition: inline In-Reply-To: User-Agent: Mutt/1.4.2i X-PGP-Key-URL: http://people.freebsd.org/~pjd/pjd.asc X-OS: FreeBSD 5.2.1-RC2 i386 cc: freebsd-geom@freebsd.org cc: freebsd-arch@freebsd.org Subject: Re: fdisk/geom, AoE 5.2.1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 09:40:00 -0000 --utG8zEvBNE7w4yrA Content-Type: text/plain; charset=iso-8859-2 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable On Wed, Oct 06, 2004 at 11:34:05PM -0500, Sam wrote: +> I can read and write to the disk. I cannot, +> however, partition it and I'm soliciting the +> list for ideas as to why. You can set kern.geom.debugflags to 5 and see what's going on there. +> I'm probably not tickling something right. Is it +> possible I need to grease the path for the disk +> partition table in geom? I didn't see anything of +> the like. Maybe it's a tasting issue? Show me the source!:) +> BTB - Is the geom(4) manpage finished in 5.3? Nope, but for developing GEOM classes there are many useful (I hope) manual pages: DECLARE_GEOM_CLASS(9) g_access(9) g_attach(9) g_bio(9) g_consumer(9) g_data(9) g_event(9) g_geom(9) g_provider(9) g_provider_by_name(9) g_wither_geom(9) --=20 Pawel Jakub Dawidek http://www.FreeBSD.org pjd@FreeBSD.org http://garage.freebsd.pl FreeBSD committer Am I Evil? Yes, I Am! --utG8zEvBNE7w4yrA Content-Type: application/pgp-signature Content-Disposition: inline -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.2.4 (FreeBSD) iD8DBQFBZQ7oForvXbEpPzQRAjJ/AJ9EivbqZMqgIu17W0fliFsWqRNGFACgiFKv 7xUvq/s5Zcr/IcpPIiHOins= =s+yo -----END PGP SIGNATURE----- --utG8zEvBNE7w4yrA-- From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 13:56:44 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 422FE16A4CE; Thu, 7 Oct 2004 13:56:44 +0000 (GMT) Received: from athena.softcardsystems.com (mail.softcardsystems.com [12.34.136.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id DDD5943D5F; Thu, 7 Oct 2004 13:56:43 +0000 (GMT) (envelope-from sah@softcardsystems.com) Received: from athena (athena [12.34.136.114])i97EtPKP025964; Thu, 7 Oct 2004 09:55:25 -0500 Date: Thu, 7 Oct 2004 09:55:25 -0500 (EST) From: Sam X-X-Sender: sah@athena To: John-Mark Gurney In-Reply-To: <20041007050405.GP22681@funkthat.com> Message-ID: References: <20041007050405.GP22681@funkthat.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: freebsd-geom@freebsd.org cc: freebsd-arch@freebsd.org Subject: Re: fdisk/geom, AoE 5.2.1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 13:56:44 -0000 > Sam wrote this message on Wed, Oct 06, 2004 at 23:34 -0500: >> I've got the AoE driver ported and running on >> 5.2.1. Much faster than I anticipated; the geom >> interface is delightful. >> >> I can read and write to the disk. I cannot, >> however, partition it and I'm soliciting the >> list for ideas as to why. >> >> If I fdisk -u an AoE device, fdisk claims all >> was successful, but in reality nothing gets written. >> Further calls to fdisk support this. >> >> I'm probably not tickling something right. Is it >> possible I need to grease the path for the disk >> partition table in geom? I didn't see anything of >> the like. Maybe it's a tasting issue? > > On last close (from fdisk hopefully), the provider will be passed > back for tasting since the meta data could possibly have changed.. > > Have you verified that the writes are hitting the disk? and that the > fdisk table can be seen from another machine? Verified that you can > read what you write to the disk? Arg. My goof. I wrote zeros, read zeros and assumed i could write to the disk. In fact I could - even when I wanted to read. I won't go into the details, but suffice it to say you can't do: if (bp->bio_flags & BIO_READ) // do read else // do write and expect the correct behaviour. :) Thanks - Sam From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 14:25:17 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 9D00F16A4CE; Thu, 7 Oct 2004 14:25:17 +0000 (GMT) Received: from critter.freebsd.dk (critter.freebsd.dk [212.242.86.163]) by mx1.FreeBSD.org (Postfix) with ESMTP id F2EC043D39; Thu, 7 Oct 2004 14:25:16 +0000 (GMT) (envelope-from phk@critter.freebsd.dk) Received: from critter.freebsd.dk (localhost [127.0.0.1]) by critter.freebsd.dk (8.13.1/8.13.1) with ESMTP id i97EP4Bg024465; Thu, 7 Oct 2004 16:25:09 +0200 (CEST) (envelope-from phk@critter.freebsd.dk) To: Sam From: "Poul-Henning Kamp" In-Reply-To: Your message of "Thu, 07 Oct 2004 09:55:25 CDT." Date: Thu, 07 Oct 2004 16:25:04 +0200 Message-ID: <24464.1097159104@critter.freebsd.dk> Sender: phk@critter.freebsd.dk cc: John-Mark Gurney cc: freebsd-arch@freebsd.org cc: freebsd-geom@freebsd.org Subject: Re: fdisk/geom, AoE 5.2.1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 14:25:17 -0000 In message , Sam writes: >Arg. My goof. I wrote zeros, read zeros and assumed i could >write to the disk. In fact I could - even when I wanted to >read. I won't go into the details, but suffice it to say you >can't do: > >if (bp->bio_flags & BIO_READ) > // do read >else > // do write > >and expect the correct behaviour. :) Please be aware that we have _three_ I/O operations under 5.x and later: BIO_READ, BIO_WRITE and BIO_DELETE. Just return EOPNOTSUPP for BIO_DELETE. -- Poul-Henning Kamp | UNIX since Zilog Zeus 3.20 phk@FreeBSD.ORG | TCP/IP since RFC 956 FreeBSD committer | BSD since 4.3-tahoe Never attribute to malice what can adequately be explained by incompetence. From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 14:32:37 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id C3F0F16A4CE; Thu, 7 Oct 2004 14:32:37 +0000 (GMT) Received: from athena.softcardsystems.com (mail.softcardsystems.com [12.34.136.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6CF3043D55; Thu, 7 Oct 2004 14:32:37 +0000 (GMT) (envelope-from sah@softcardsystems.com) Received: from athena (athena [12.34.136.114])i97FVJ7o026196; Thu, 7 Oct 2004 10:31:19 -0500 Date: Thu, 7 Oct 2004 10:31:19 -0500 (EST) From: Sam X-X-Sender: sah@athena To: Poul-Henning Kamp In-Reply-To: <24464.1097159104@critter.freebsd.dk> Message-ID: References: <24464.1097159104@critter.freebsd.dk> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed cc: John-Mark Gurney cc: freebsd-geom@freebsd.org cc: freebsd-arch@freebsd.org Subject: Re: fdisk/geom, AoE 5.2.1 X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 14:32:37 -0000 > Please be aware that we have _three_ I/O operations under 5.x and > later: BIO_READ, BIO_WRITE and BIO_DELETE. Just return EOPNOTSUPP > for BIO_DELETE. I was filtering out anything but R/W before I got there. Thanks for the tip on EOPNOTSUPP, though. I was returning EIO. Sam From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 19:30:43 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 5D99D16A4CE for ; Thu, 7 Oct 2004 19:30:43 +0000 (GMT) Received: from athena.softcardsystems.com (mail.softcardsystems.com [12.34.136.114]) by mx1.FreeBSD.org (Postfix) with ESMTP id 1C4A243D1D for ; Thu, 7 Oct 2004 19:30:43 +0000 (GMT) (envelope-from sah@softcardsystems.com) Received: from athena (athena [12.34.136.114])i97KTOoF028006 for ; Thu, 7 Oct 2004 15:29:24 -0500 Date: Thu, 7 Oct 2004 15:29:24 -0500 (EST) From: Sam X-X-Sender: sah@athena To: freebsd-arch@freebsd.org Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Subject: sys/net/netisr.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 19:30:43 -0000 Hello - I think I've found a bug in -- and have a question about the overall stability of -- sys/net/netisr.c (5.2.1 branch). My AoE module calls netisr_register on load, netisr_unregister on unload. Netisr_unregister fails to clear the ni->ni_queue pointer and the next received frame gets queued up to a page fault. Pretty easy to fix: --- src/sys/net/netisr.c Sat Nov 8 17:28:39 2003 +++ src2/sys/net/netisr.c Thu Oct 7 15:03:39 2004 @@ -103,6 +103,7 @@ ni->ni_handler = NULL; if (ni->ni_queue != NULL) IF_DRAIN(ni->ni_queue); + ni->ni_queue = NULL; } struct isrstat { Looking at the code, though, I don't see why I can't cause something just as ugly to happen anyway. Suppose the following: cpu0 starts processing an inbound frame while cpu1 unloads module (calling netisr_unregister). It *seems* possible for cpu0 to get a pointer to the queue, then cpu1 unload the module completely, causing cpu0 to page fault on the queue address. I don't claim to understand the context in which netisr_dispatch is called, so perhaps I'm off base, but shouldn't there be a mutex protecting against this? Please prove me wrong. Sam From owner-freebsd-arch@FreeBSD.ORG Thu Oct 7 20:57:40 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id BAF7A16A4CE for ; Thu, 7 Oct 2004 20:57:40 +0000 (GMT) Received: from c00l3r.networx.ch (c00l3r.networx.ch [62.48.2.2]) by mx1.FreeBSD.org (Postfix) with ESMTP id 0029243D2F for ; Thu, 7 Oct 2004 20:57:39 +0000 (GMT) (envelope-from andre@freebsd.org) Received: (qmail 97057 invoked from network); 7 Oct 2004 20:48:56 -0000 Received: from unknown (HELO freebsd.org) ([62.48.0.53]) (envelope-sender ) by c00l3r.networx.ch (qmail-ldap-1.03) with SMTP for ; 7 Oct 2004 20:48:56 -0000 Message-ID: <4165ADC2.70D2F0C1@freebsd.org> Date: Thu, 07 Oct 2004 22:57:38 +0200 From: Andre Oppermann X-Mailer: Mozilla 4.8 [en] (Windows NT 5.0; U) X-Accept-Language: en MIME-Version: 1.0 To: Sam References: Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit cc: freebsd-arch@freebsd.org Subject: Re: sys/net/netisr.c X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Oct 2004 20:57:40 -0000 Sam wrote: > > Hello - > > I think I've found a bug in -- and have a question about > the overall stability of -- sys/net/netisr.c (5.2.1 branch). This particular bug has already been fixed in 5.3 and 6.0-current. You should do your development on either RELENG_5 or MAIN. The 5.2.1 branch was only a technology demo and the areas you are concerned with have changed significantly (read: really a lot). RELENG_5 (5.3) will be the next stable branch which features future binary compatibility within further > 5.x releases. -- Andre > My AoE module calls netisr_register on load, netisr_unregister > on unload. Netisr_unregister fails to clear the ni->ni_queue > pointer and the next received frame gets queued up to a page > fault. Pretty easy to fix: > > --- src/sys/net/netisr.c Sat Nov 8 17:28:39 2003 > +++ src2/sys/net/netisr.c Thu Oct 7 15:03:39 2004 > @@ -103,6 +103,7 @@ > ni->ni_handler = NULL; > if (ni->ni_queue != NULL) > IF_DRAIN(ni->ni_queue); > + ni->ni_queue = NULL; > } > > struct isrstat { > > Looking at the code, though, I don't see why I can't > cause something just as ugly to happen anyway. Suppose > the following: cpu0 starts processing an inbound frame > while cpu1 unloads module (calling netisr_unregister). > It *seems* possible for cpu0 to get a pointer to the > queue, then cpu1 unload the module completely, causing > cpu0 to page fault on the queue address. > > I don't claim to understand the context in which > netisr_dispatch is called, so perhaps I'm off base, > but shouldn't there be a mutex protecting against this? > > Please prove me wrong. > > Sam > > _______________________________________________ > freebsd-arch@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-arch > To unsubscribe, send any mail to "freebsd-arch-unsubscribe@freebsd.org" From owner-freebsd-arch@FreeBSD.ORG Sat Oct 9 20:21:33 2004 Return-Path: Delivered-To: freebsd-arch@freebsd.org Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id D5D7816A4CE for ; Sat, 9 Oct 2004 20:21:33 +0000 (GMT) Received: from duchess.speedfactory.net (duchess.speedfactory.net [66.23.201.84]) by mx1.FreeBSD.org (Postfix) with SMTP id 628E243D2D for ; Sat, 9 Oct 2004 20:21:33 +0000 (GMT) (envelope-from ups@tree.com) Received: (qmail 9957 invoked by uid 89); 9 Oct 2004 20:21:31 -0000 Received: from duchess.speedfactory.net (66.23.201.84) by duchess.speedfactory.net with SMTP; 9 Oct 2004 20:21:31 -0000 Received: (qmail 9947 invoked by uid 89); 9 Oct 2004 20:21:31 -0000 Received: from unknown (HELO palm.tree.com) (66.23.216.49) by duchess.speedfactory.net with SMTP; 9 Oct 2004 20:21:31 -0000 Received: from [127.0.0.1] (localhost.tree.com [127.0.0.1]) by palm.tree.com (8.12.10/8.12.10) with ESMTP id i99KLUmt075542; Sat, 9 Oct 2004 16:21:31 -0400 (EDT) (envelope-from ups@tree.com) From: Stephan Uphoff To: Julian Elischer In-Reply-To: <4162E8B1.90803@elischer.org> References: <1095468747.31297.241.camel@palm.tree.com> <1096496057.3733.2163.camel@palm.tree.com> <1096603981.21577.195.camel@palm.tree.com> <200410041131.35387.jhb@FreeBSD.org> <1096911278.44307.17.camel@palm.tree.com> <41619D29.1000704@elischer.org><4161A7BD.3040706@elischer.org> <4162E8B1.90803@elischer.org> Content-Type: text/plain Message-Id: <1097353290.70165.1589.camel@palm.tree.com> Mime-Version: 1.0 X-Mailer: Ximian Evolution 1.4.6 Date: Sat, 09 Oct 2004 16:21:30 -0400 Content-Transfer-Encoding: 7bit cc: Peter Holm cc: "freebsd-arch@freebsd.org" Subject: Re: scheduler (sched_4bsd) questions X-BeenThere: freebsd-arch@freebsd.org X-Mailman-Version: 2.1.1 Precedence: list List-Id: Discussion related to FreeBSD architecture List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 Oct 2004 20:21:33 -0000 On Tue, 2004-10-05 at 14:32, Julian Elischer wrote: > -- snip -- > there are two things to notice.. > firstly the "corrected slot count (2->1)" messge is still there. (grumble). > > this is hapenning when a threade dprocess moves back to be ing an > unthreaded preocess. > for some reason, the number of openning s is not being set back to 1 but > rather to 2. > I believe it is because while in thhe threaded mode it is already too > high by some amount > (sometimes equivalent to NTHREAD) but I can not see why. > Hopefully it is not a fatal problem (as it would be if it were too LOW, > but I hope to figure it > out soon (maybe another one for Stephan :-) > -- snip -- There is an extra SLOT_RELEASE - sched_rem already released the slot. RCS file: /cvsroot/src/sys/kern/kern_switch.c,v retrieving revision 1.97 diff -u -r1.97 kern_switch.c --- kern_switch.c 5 Oct 2004 22:03:10 -0000 1.97 +++ kern_switch.c 9 Oct 2004 20:18:53 -0000 @@ -372,7 +372,6 @@ sched_rem(tda); tda = kg->kg_last_assigned = TAILQ_PREV(tda, threadqueue, td_runq); - SLOT_RELEASE(kg); } /*