Date: Thu, 1 Sep 2005 13:48:49 -0400 From: John Baldwin <jhb@FreeBSD.org> To: freebsd-hackers@freebsd.org Cc: Koen Martens <fbsd@metro.cx> Subject: Re: panic in propagate_priority w/ postgresql under heavy load Message-ID: <200509011348.50159.jhb@FreeBSD.org> In-Reply-To: <4317340E.2050208@metro.cx> References: <4317340E.2050208@metro.cx>
next in thread | previous in thread | raw e-mail | index | archive | help
On Thursday 01 September 2005 01:02 pm, Koen Martens wrote: > Hi Hackers, > > I've had a little chat with neologism on ircnet/#freebsd about this > already, and done as he suggested: compile a debug kernel to obtain > a stack trace. Can you reproduce it with a kernel that has INVARIANTS and INVARIANT_SUPPORT on? I see that you had WITNESS on, can you check to see if there were any witness messages about sleepign with non-sleepable locks held before the crash? > Anyway, what is happening is that there is a crash when running > postgresql 8.0.3 with a very large database and doing heavy queries. > > Kernel is 5.4-RELEASE-p6 (5.4-RELENG i checked out tuesday a week > ago). Kernel conf is down below. > > Here is the message i get on the console: > > ==================== > kernel trap 12 with interrupts disabled > > > Fatal trap 12: page fault while in kernel mode > cpuid = 1; apic id = 06 > fault virtual address = 0x24 > fault code = supervisor read, page not present > instruction pointer = 0x8:0xc050cff7 > stack pointer = 0x10:0xe99c2b0c > frame pointer = 0x10:0xe99c2b20 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, def32 1, gran 1 > processor eflags = resume, IOPL = 0 > current process = 4571 (postgres) > ==================== > > It has been a postgres process in all of the observed cases. > > I've looked it up with gdk on my kernel.debug, here's what i get: > > ===================== > yin# gdb kernel.debug > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and > you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for > details. > This GDB was configured as "i386-marcel-freebsd"... > (gdb) l *propagate_priority+0x7f > 0xc050cff7 is in propagate_priority > (/usr/src/sys/kern/subr_turnstile.c:245). > 240 /* > 241 * Pick up the lock that td is blocked on. > 242 */ > 243 ts = td->td_blocked; > 244 MPASS(ts != NULL); > 245 tc = TC_LOOKUP(ts->ts_lockobj); > 246 mtx_lock_spin(&tc->tc_lock); > 247 > 248 /* > 249 * This thread may not be blocked on this > turnstile anymore > (gdb) > ===================== > > > So the next thing you'll ask for is a stack trace, but i haven't > been able to obtain one. According to the freebsd handbook > (http://www.freebsd.org/doc/en_US.ISO8859-1/books/developers-handbook/kerne >ldebug-gdb.html) there should be a core dump in /var/crash, but there is > none and the handbook chapter seems outdated anyway judging by it > mentioning kgdb... > > Anyway, it seems the dump should've gone to the swap partition, but > i'm into multi-user mode again so i guess i'll have to wait for > another panic to obtain it? > > I'm not very knowledgeable about the freebsd kernel internals (yet), > so i'm not sure what could be wrong here.. I hope some of you can > provide some insight, and ideally a fix :) > > ======================[ kernel config: > > # > # GENERIC -- Generic kernel configuration file for FreeBSD/i386 > # > # For more information on this file, please read the handbook section on > # Kernel Configuration Files: > # > # > http://www.FreeBSD.org/doc/en_US.ISO8859-1/books/handbook/kernelconfig-conf >ig.html # > # The handbook is also available locally in /usr/share/doc/handbook > # if you've installed the doc distribution, otherwise always see the > # FreeBSD World Wide Web server (http://www.FreeBSD.org/) for the > # latest information. > # > # An exhaustive list of options and more detailed explanations of the > # device lines is also present in the ../../conf/NOTES and NOTES files. > # If you are in doubt as to the purpose or necessity of a line, > check first > # in NOTES. > # > # $FreeBSD: src/sys/i386/conf/GENERIC,v 1.413.2.13 2005/04/02 > 16:37:58 scottl Exp $ > > machine i386 > cpu I486_CPU > cpu I586_CPU > cpu I686_CPU > ident YIN-YANG > > # debug > options WITNESS > options KDB > options DDB > # > makeoptions DEBUG=-g > # debug > > > # To statically compile in device wiring instead of /boot/device.hints > #hints "GENERIC.hints" # Default places to look for devices. > > options SCHED_4BSD # 4BSD scheduler > options INET # InterNETworking > options INET6 # IPv6 communications protocols > options FFS # Berkeley Fast Filesystem > options SOFTUPDATES # Enable FFS soft updates support > options UFS_ACL # Support for access control lists > options UFS_DIRHASH # Improve performance on big directories > options MD_ROOT # MD is a potential root device > options NFSCLIENT # Network Filesystem Client > options NFSSERVER # Network Filesystem Server > options NFS_ROOT # NFS usable as /, requires NFSCLIENT > options MSDOSFS # MSDOS Filesystem > options CD9660 # ISO 9660 Filesystem > options PROCFS # Process filesystem (requires PSEUDOFS) > options PSEUDOFS # Pseudo-filesystem framework > options GEOM_GPT # GUID Partition Tables. > options COMPAT_43 # Compatible with BSD 4.3 [KEEP THIS!] > options COMPAT_FREEBSD4 # Compatible with FreeBSD4 > options SCSI_DELAY=15000 # Delay (in ms) before probing SCSI > options KTRACE # ktrace(1) support > options SYSVSHM # SYSV-style shared memory > options SYSVMSG # SYSV-style message queues > options SYSVSEM # SYSV-style semaphores > options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time > extensions > options KBD_INSTALL_CDEV # install a CDEV entry in /dev > options AHC_REG_PRETTY_PRINT # Print register bitfields in debug > # output. Adds ~128k to driver. > options AHD_REG_PRETTY_PRINT # Print register bitfields in debug > # output. Adds ~215k to driver. > options ADAPTIVE_GIANT # Giant mutex is adaptive. > > > options IPFILTER > options IPFILTER_LOG > options IPFILTER_DEFAULT_BLOCK > > options IPFILTER > options IPFILTER_LOG > > options IPFIREWALL #firewall > options IPFIREWALL_VERBOSE #enable logging to syslogd(8) > options IPFIREWALL_FORWARD #enable transparent proxy > support > options IPFIREWALL_VERBOSE_LIMIT=10 #limit verbosity > options IPFIREWALL_DEFAULT_TO_ACCEPT #allow everything by > default > options IPV6FIREWALL #firewall for IPv6 > options IPV6FIREWALL_VERBOSE > options IPV6FIREWALL_VERBOSE_LIMIT=10 > options IPV6FIREWALL_DEFAULT_TO_ACCEPT > > options SHMMAXPGS=262144 > options SHMSEG=512 > options SHMMNI=512 > options SEMMNI=512 > options SEMMNS=1024 > options SEMMNU=512 > options SEMMAP=512 > options NMBCLUSTERS=32768 > > device apic # I/O APIC > > options SMP # Symmetric MultiProcessor > Kernel > > # Bus support. Do not remove isa, even if you have no isa slots > device isa > device eisa > device pci > > # Floppy drives > device fdc > > # ATA and ATAPI devices > device ata > device atadisk # ATA disk drives > device ataraid # ATA RAID drives > device atapicd # ATAPI CDROM drives > #device atapifd # ATAPI floppy drives > #device atapist # ATAPI tape drives > options ATA_STATIC_ID # Static device numbering > > # SCSI Controllers > #device ahb # EISA AHA1742 family > #device ahc # AHA2940 and onboard AIC7xxx devices > #device ahd # AHA39320/29320 and onboard AIC79xx devices > #device amd # AMD 53C974 (Tekram DC-390(T)) > #device isp # Qlogic family > #device mpt # LSI-Logic MPT-Fusion > ##device ncr # NCR/Symbios Logic > #device sym # NCR/Symbios Logic (newer chipsets + those of `ncr') > #device trm # Tekram DC395U/UW/F DC315U adapters > > #device adv # Advansys SCSI adapters > #device adw # Advansys wide SCSI adapters > #device aha # Adaptec 154x SCSI adapters > #device aic # Adaptec 15[012]x SCSI adapters, AIC-6[23]60. > #device bt # Buslogic/Mylex MultiMaster SCSI adapters > > #device ncv # NCR 53C500 > #device nsp # Workbit Ninja SCSI-3 > #device stg # TMC 18C30/18C50 > > # SCSI peripherals > device scbus # SCSI bus (required for SCSI) > device ch # SCSI media changers > device da # Direct Access (disks) > device sa # Sequential Access (tape etc) > device cd # CD > device pass # Passthrough device (direct SCSI access) > device ses # SCSI Environmental Services (and SAF-TE) > > # RAID controllers interfaced to the SCSI subsystem > #device amr # AMI MegaRAID > #device arcmsr # Areca SATA II RAID > #device asr # DPT SmartRAID V, VI and Adaptec SCSI RAID > #device ciss # Compaq Smart RAID 5* > #device dpt # DPT Smartcache III, IV - See NOTES for options > #device hptmv # Highpoint RocketRAID 182x > #device iir # Intel Integrated RAID > #device ips # IBM (Adaptec) ServeRAID > #device mly # Mylex AcceleRAID/eXtremeRAID > device twa # 3ware 9000 series PATA/SATA RAID > > # RAID controllers > #device aac # Adaptec FSA RAID > #device aacp # SCSI passthrough for aac (requires CAM) > #device ida # Compaq Smart RAID > #device mlx # Mylex DAC960 family > #device pst # Promise Supertrak SX6000 > device twe # 3ware ATA RAID > > # atkbdc0 controls both the keyboard and the PS/2 mouse > device atkbdc # AT keyboard controller > device atkbd # AT keyboard > device psm # PS/2 mouse > > device vga # VGA video card driver > > #device splash # Splash screen and screen saver support > > # syscons is the default console driver, resembling an SCO console > device sc > > # Enable this for the pcvt (VT220 compatible) console driver > #device vt > #options XSERVER # support for X server on a vt console > #options FAT_CURSOR # start with block cursor > > device agp # support several AGP chipsets > > # Floating point support - do not disable. > device npx > > # Power management support (see NOTES for more options) > #device apm > # Add suspend/resume support for the i8254. > device pmtimer > > # PCCARD (PCMCIA) support > # PCMCIA and cardbus bridge support > #device cbb # cardbus (yenta) bridge > #device pccard # PC Card (16-bit) bus > #device cardbus # CardBus (32-bit) bus > > # Serial (COM) ports > device sio # 8250, 16[45]50 based serial ports > > # Parallel port > device ppc > device ppbus # Parallel port bus (required) > device lpt # Printer > device plip # TCP/IP over parallel > device ppi # Parallel port interface device > #device vpo # Requires scbus and da > > # If you've got a "dumb" serial or parallel PCI card that is > # supported by the puc(4) glue driver, uncomment the following > # line to enable it (connects to the sio and/or ppc drivers): > #device puc > > # PCI Ethernet NICs. > #device de # DEC/Intel DC21x4x (``Tulip'') > device em # Intel PRO/1000 adapter Gigabit Ethernet Card > #device ixgb # Intel PRO/10GbE Ethernet Card > #device txp # 3Com 3cR990 (``Typhoon'') > #device vx # 3Com 3c590, 3c595 (``Vortex'') > > # PCI Ethernet NICs that use the common MII bus controller code. > # NOTE: Be sure to keep the 'device miibus' line in order to use > these NICs! > device miibus # MII bus support > #device bfe # Broadcom BCM440x 10/100 Ethernet > #device bge # Broadcom BCM570xx Gigabit Ethernet > #device dc # DEC/Intel 21143 and various workalikes > device fxp # Intel EtherExpress PRO/100B (82557, 82558) > #device lge # Level 1 LXT1001 gigabit ethernet > #device nge # NatSemi DP83820 gigabit ethernet > #evice pcn # AMD Am79C97x PCI 10/100 (precedence over 'lnc') > #device re # RealTek 8139C+/8169/8169S/8110S > #device rl # RealTek 8129/8139 > #device sf # Adaptec AIC-6915 (``Starfire'') > #device sis # Silicon Integrated Systems SiS 900/SiS 7016 > #device sk # SysKonnect SK-984x & SK-982x gigabit Ethernet > #device ste # Sundance ST201 (D-Link DFE-550TX) > #device ti # Alteon Networks Tigon I/II gigabit Ethernet > #device tl # Texas Instruments ThunderLAN > #device tx # SMC EtherPower II (83c170 ``EPIC'') > #device vge # VIA VT612x gigabit ethernet > #device vr # VIA Rhine, Rhine II > #device wb # Winbond W89C840F > #device xl # 3Com 3c90x (``Boomerang'', ``Cyclone'') > > # ISA Ethernet NICs. pccard NICs included. > #device cs # Crystal Semiconductor CS89x0 NIC > # 'device ed' requires 'device miibus' > #device ed # NE[12]000, SMC Ultra, 3c503, DS8390 cards > #device ex # Intel EtherExpress Pro/10 and Pro/10+ > #device ep # Etherlink III based cards > #device fe # Fujitsu MB8696x based cards > #device ie # EtherExpress 8/16, 3C507, StarLAN 10 etc. > #device lnc # NE2100, NE32-VL Lance Ethernet cards > #device sn # SMC's 9000 series of Ethernet chips > #device xe # Xircom pccard Ethernet > > # ISA devices that use the old ISA shims > #device le > > # Wireless NIC cards > #device wlan # 802.11 support > #device an # Aironet 4500/4800 802.11 wireless NICs. > #device awi # BayStack 660 and others > #device wi # WaveLAN/Intersil/Symbol 802.11 wireless NICs. > #device wl # Older non 802.11 Wavelan wireless NIC. > > # Pseudo devices. > device loop # Network loopback > device mem # Memory and kernel memory devices > device io # I/O device > device random # Entropy device > device ether # Ethernet support > device sl # Kernel SLIP > device ppp # Kernel PPP > device tun # Packet tunnel. > device pty # Pseudo-ttys (telnet etc) > device md # Memory "disks" > device gif # IPv6 and IPv4 tunneling > device faith # IPv6-to-IPv4 relaying (translation) > > # The `bpf' device enables the Berkeley Packet Filter. > # Be aware of the administrative consequences of enabling this! > # Note that 'bpf' is required for DHCP. > device bpf # Berkeley packet filter > > # USB support > #device uhci # UHCI PCI->USB interface > #device ohci # OHCI PCI->USB interface > #device ehci # EHCI PCI->USB interface (USB 2.0) > #device usb # USB Bus (required) > #device udbp # USB Double Bulk Pipe devices > #device ugen # Generic > #device uhid # "Human Interface Devices" > #device ukbd # Keyboard > #device ulpt # Printer > #device umass # Disks/Mass storage - Requires scbus and da > #device ums # Mouse > #device urio # Diamond Rio 500 MP3 player > #device uscanner # Scanners > # USB Ethernet, requires mii > #device aue # ADMtek USB Ethernet > #device axe # ASIX Electronics USB Ethernet > #device cdce # Generic USB over Ethernet > #device cue # CATC USB Ethernet > #device kue # Kawasaki LSI USB Ethernet > #device rue # RealTek RTL8150 USB Ethernet > > # FireWire support > #device firewire # FireWire bus code > #device sbp # SCSI over FireWire (Requires scbus and da) > #device fwe # Ethernet over FireWire (non-standard!) > > > -- > K.F.J. Martens, Sonologic, http://www.sonologic.nl/ Networking, > hosting, embedded systems, unix, artificial intelligence. Public PGP > key: http://www.metro.cx/pubkey-gmc.asc Wondering about the funny > attachment your mail program can't read? Visit http://www.openpgp.org/ -- John Baldwin <jhb@FreeBSD.org> <>< http://www.FreeBSD.org/~jhb/ "Power Users Use the Power to Serve" = http://www.FreeBSD.org
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?200509011348.50159.jhb>