From owner-freebsd-amd64@FreeBSD.ORG Wed Jan 18 07:13:23 2006 Return-Path: X-Original-To: amd64@freebsd.org Delivered-To: freebsd-amd64@FreeBSD.ORG Received: from mx1.FreeBSD.org (mx1.freebsd.org [216.136.204.125]) by hub.freebsd.org (Postfix) with ESMTP id 6040C16A41F for ; Wed, 18 Jan 2006 07:13:23 +0000 (GMT) (envelope-from jp@tns.cz) Received: from bns.tns.cz (bns.tns.cz [212.71.178.249]) by mx1.FreeBSD.org (Postfix) with ESMTP id 6DC3043D46 for ; Wed, 18 Jan 2006 07:13:21 +0000 (GMT) (envelope-from jp@tns.cz) Received: from bns.tns.cz (localhost [127.0.0.1]) by bns.tns.cz (Postfix) with ESMTP id C93D29595B for ; Wed, 18 Jan 2006 08:13:19 +0100 (CET) Received: from bertik.tns.cz (bertik [192.168.144.14]) by bns.tns.cz with ESMTP id 47JFAH4000EW6YQGHBK; Wed, 18 Jan 2006 08:13:13 +0100 (CET) Received: by bertik.tns.cz (Postfix, from userid 1000) id 67A9659192; Wed, 18 Jan 2006 08:13:24 +0100 (CET) Date: Wed, 18 Jan 2006 08:13:23 +0100 From: Josef Pojsl To: amd64@freebsd.org Message-ID: <20060118071323.GB795@bertik.tns.cz> MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="x+6KMIRAuhnl3hBn" Content-Disposition: inline User-Agent: Mutt/1.5.6i X-Kernun-Virus-Status: OK Cc: Subject: 5.4/amd64 not stable X-BeenThere: freebsd-amd64@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Porting FreeBSD to the AMD64 platform List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Jan 2006 07:13:23 -0000 --x+6KMIRAuhnl3hBn Content-Type: text/plain; charset=us-ascii Content-Disposition: inline Hello list, I am observing panics of FreeBSD 5.4/amd64 on a web server. We tested it heavily with Apache bench. Load was over 250, without problems. However, when we put the server in production, it crashes time after time. It takes some time between 1 and 8 hours. The panics look like this: Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0x48 fault code = supervisor read, page not present instruction pointer = 0x8:0xffffffff80271a83 stack pointer = 0x10:0xffffffffc1043990 frame pointer = 0x10:0x0 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = resume, IOPL = 0 current process = 10567 (httpd) trap number = 12 panic: page fault cpuid = 0 boot() called on cpu#0 The instruction pointer is always the same, and process is always httpd. Does it mean anything? We are not able to produce crash dump. Dumping memory into swap starts after crash, but it stops after 4GB and no further progress is apparent. If we break it, savecore does not recognize the data in swap space as crash dump during boot. BTW, it is HP ProLiant DL385, dmesg.boot and kernel configuration are attached. It has got: 2 Opterons, 16GB RAM, it runs Apache with mod_ssl and php, a mysql server, a nfs server and a jail. I am attaching pkg_info as well. Both memtest and HP diagnostic tools show no hardware problems. I would appreciate any ideas or comments. Thanks in advance, -- Josef --x+6KMIRAuhnl3hBn Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename="dmesg.boot" Copyright (c) 1992-2005 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD 5.4-RELEASE-p8 #0: Wed Nov 16 10:50:18 CET 2005 root@amd64.tns.cz:/usr/obj/usr/src/sys/PLDL385 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: AMD Opteron(tm) Processor 252 (2605.92-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x20f51 Stepping = 1 Features=0x78bfbff Features2=0x1 AMD Features=0xe2500800,LM,3DNow+,3DNow> real memory = 17179865088 (16383 MB) avail memory = 16480096256 (15716 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 2 CPUs cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-23 on motherboard ioapic1 irqs 24-27 on motherboard ioapic2 irqs 28-31 on motherboard ioapic3 irqs 32-35 on motherboard ioapic4 irqs 36-39 on motherboard acpi0: on motherboard acpi0: Power Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 1000 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x908-0x90b on acpi0 cpu0: on acpi0 cpu1: on acpi0 pcib0: on acpi0 pci0: on pcib0 pcib1: at device 3.0 on pci0 pci1: on pcib1 pci1: at device 2.0 (no driver attached) pci1: at device 2.2 (no driver attached) pci1: at device 3.0 (no driver attached) isab0: at device 4.0 on pci0 isa0: on isab0 atapci0: port 0x2000-0x200f,0x376,0x170-0x177,0x3f6,0x1f0-0x1f7 at device 4.1 on pci0 ata0: channel #0 on atapci0 ata1: channel #1 on atapci0 pci0: at device 4.3 (no driver attached) pcib2: at device 7.0 on pci0 pci2: on pcib2 ciss0: port 0x5000-0x50ff mem 0xf7c80000-0xf7cbffff,0xf7cf0000-0xf7cf1fff irq 24 at device 4.0 on pci2 pci0: at device 7.1 (no driver attached) pcib3: at device 8.0 on pci0 pci3: on pcib3 bge0: mem 0xf7df0000-0xf7dfffff irq 28 at device 6.0 on pci3 miibus0: on bge0 brgphy0: on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge0: Ethernet address: 00:14:c2:3d:d3:0c bge1: mem 0xf7de0000-0xf7deffff irq 29 at device 6.1 on pci3 miibus1: on bge1 brgphy1: on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseTX, 1000baseTX-FDX, auto bge1: Ethernet address: 00:14:c2:3d:d3:0b pci0: at device 8.1 (no driver attached) pcib4: on acpi0 pci4: on pcib4 pcib5: at device 9.0 on pci4 pci5: on pcib5 mpt0: port 0x6000-0x60ff mem 0xf7fc0000-0xf7fdffff,0xf7fe0000-0xf7ffffff irq 34 at device 7.0 on pci5 mpt1: port 0x6400-0x64ff mem 0xf7f80000-0xf7f9ffff,0xf7fa0000-0xf7fbffff irq 35 at device 7.1 on pci5 em0: port 0x6800-0x683f mem 0xf7f00000-0xf7f3ffff,0xf7f60000-0xf7f7ffff irq 32 at device 8.0 on pci5 em0: Ethernet address: 00:11:0a:53:f2:c2 em0: Speed:N/A Duplex:N/A em1: port 0x6840-0x687f mem 0xf7ee0000-0xf7efffff irq 33 at device 8.1 on pci5 em1: Ethernet address: 00:11:0a:53:f2:c3 em1: Speed:N/A Duplex:N/A pci4: at device 9.1 (no driver attached) pcib6: at device 10.0 on pci4 pci6: on pcib6 pci4: at device 10.1 (no driver attached) atkbdc0: port 0x64,0x60 irq 1 on acpi0 sio0: port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 sio0: type 16550A fdc0: port 0x3f2-0x3f5 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 fdc0: port 0x3f2-0x3f5 irq 6 drq 2 on acpi0 fdc0: does not respond device_attach: fdc0 attach returned 6 orm0: at iomem 0xee000-0xeffff,0xcc800-0xcdfff,0xcc000-0xcc7ff,0xc8000-0xcbfff,0xc0000-0xc7fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> sio1: configured irq 3 not in bitmap of probed irqs 0 sio1: port may not be enabled vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec IP Filter: v3.4.35 initialized. Default = pass all, Logging = enabled acd0: DVDROM at ata0-master PIO4 Waiting 15 seconds for SCSI devices to settle da0 at ciss0 bus 0 target 0 lun 0 da0: Fixed Direct Access SCSI-0 device da0: 135.168MB/s transfers da0: 138919MB (284506560 512 byte sectors: 255H 32S/T 34866C) SMP: AP CPU #1 Launched! Mounting root from ufs:/dev/da0s1a --x+6KMIRAuhnl3hBn Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=PLDL385 machine amd64 cpu HAMMER ident PLDL385 # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. options SCHED_4BSD # 4BSD scheduler options INET # InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options MD_ROOT # MD is a potential root device options NFSCLIENT # Network Filesystem Client options NFSSERVER # Network Filesystem Server options NFS_ROOT # NFS usable as /, requires NFSCLIENT options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_GPT # GUID Partition Tables. options COMPAT_43 # Needed by COMPAT_LINUX32 options COMPAT_IA32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_LINUX32 # Compatible with i386 linux binaries options SCSI_DELAY=15000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options KBD_INSTALL_CDEV # install a CDEV entry in /dev options ADAPTIVE_GIANT # Giant mutex is adaptive. # Workarounds for some known-to-be-broken chipsets (nVidia nForce3-Pro150) device atpic # 8259A compatability options NO_MIXED_MODE # Don't penalize working chipsets # Bus support. Do not remove isa, even if you have no isa slots device acpi device isa device pci # Floppy drives device fdc # ATA and ATAPI devices device ata device atadisk device atapicd # ATAPI CDROM drives options ATA_STATIC_ID # Static device numbering # SCSI Controllers device mpt # LSI-Logic MPT-Fusion # SCSI peripherals device scbus # SCSI bus (required for SCSI) device ch # SCSI media changers device da # Direct Access (disks) device sa # Sequential Access (tape etc) device cd # CD device pass # Passthrough device (direct SCSI access) device ses # SCSI Environmental Services (and SAF-TE) # RAID controllers interfaced to the SCSI subsystem device ciss # Compaq Smart RAID 5* # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc # Serial (COM) ports device sio # 8250, 16[45]50 based serial ports # PCI Ethernet NICs. device em # Intel PRO/1000 adapter Gigabit Ethernet Card # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device bge # Broadcom BCM570xx Gigabit Ethernet # Pseudo devices. device loop # Network loopback device mem # Memory and kernel memory devices device io # I/O device device random # Entropy device device ether # Ethernet support device tun # Packet tunnel. device tap # Virtual ethernet device pty # Pseudo-ttys (telnet etc) device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter options IPFILTER options IPFILTER_LOG options PQ_CACHESIZE=1024 options SMP --x+6KMIRAuhnl3hBn Content-Type: text/plain; charset=us-ascii Content-Disposition: attachment; filename=pkg_info abcache-0.14 A tool to cache applications written in PHP apache+mod_ssl-1.3.34+2.8.25_1 The Apache 1.3 webserver with SSL/TLS functionality autoconf-2.59_2 Automatically configure source code on many Un*x platforms bash-3.0.16_1 The GNU Project's Bourne Again SHell cvsup-without-gui-16.1h_2 General network file distribution system optimized for CVS expat-1.95.8_3 XML 1.0 parser written in C freetype2-2.1.10_1 A free and portable TrueType font rendering engine gd-2.0.33_3,1 A graphics library for fast creation of images gettext-0.14.5 GNU gettext package gmake-3.80_2 GNU version of 'make' utility help2man-1.36.3 Automatically generating simple manual pages from program o jpeg-6b_3 IJG's jpeg compression utilities libiconv-1.9.2_1 A character set conversion library libtool-1.5.20 Generic shared library support script (1.5) m4-1.4.4 GNU m4 mm-1.4.0 Shared memory allocation library for pre-forked process mod mod_security-1.8.7 An intrusion detection and prevention engine mysql-client-4.0.26 Multithreaded SQL database (client) mysql-scripts-4.0.26 Multithreaded SQL database (scripts) mysql-server-4.0.26 Multithreaded SQL database (server) p5-DBD-mysql40-3.0002 MySQL 4.0 driver for the Perl5 Database Interface (DBI) p5-DBI-1.48 The perl5 Database Interface. Required for DBD::* modules p5-gettext-1.05_1 Message handling functions pcre-6.4 Perl Compatible Regular Expressions library perl-5.8.6_2 Practical Extraction and Report Language php4-4.4.1_2 PHP Scripting Language (Apache Module and CLI) php4-extensions-1.0 A "meta-port" to install PHP extensions php4-gd-4.4.1_2 The gd shared extension for php php4-iconv-4.4.1_2 The iconv shared extension for php php4-mysql-4.4.1_2 The mysql shared extension for php php4-pcre-4.4.1_2 The pcre shared extension for php php4-session-4.4.1_2 The session shared extension for php php4-xml-4.4.1_2 The xml shared extension for php pkgconfig-0.20 A utility to retrieve information about installed libraries png-1.2.8_2 Library for manipulating PNG images portaudit-0.5.10 Checks installed ports against a list of security vulnerabi postfix-2.2.5_1,1 A secure alternative to widely-used Sendmail rsync-2.6.6 A network file distribution/synchronization utility scponly-4.3 A tiny shell that only permits scp and sftp t1lib-5.1.0,1 A Type 1 Rasterizer Library for UNIX/X11 vsftpd-2.0.3_1 A FTP daemon that aims to be "very secure" webalizer-2.1.10_5 A web server log file analysis program wget-1.10.2 Retrieve files from the Net via HTTP and FTP --x+6KMIRAuhnl3hBn--