From owner-freebsd-stable@FreeBSD.ORG Thu Jan 19 14:03:25 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B263A1065677 for ; Thu, 19 Jan 2012 14:03:25 +0000 (UTC) (envelope-from laszlo@karolyi.hu) Received: from flix.hu (flix.hu [81.94.178.130]) by mx1.freebsd.org (Postfix) with ESMTP id 245E68FC18 for ; Thu, 19 Jan 2012 14:03:24 +0000 (UTC) Received: from [192.168.0.103] (pd95c31e4.dip0.t-ipconnect.de [217.92.49.228]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: laszlo@karolyi.hu) by flix.hu (Postfix) with ESMTPSA id 3C9C5CC25 for ; Thu, 19 Jan 2012 14:51:42 +0100 (CET) Message-ID: <4F181FE7.30904@karolyi.hu> Date: Thu, 19 Jan 2012 14:51:35 +0100 From: =?ISO-8859-2?Q?L=E1szl=F3_K=C1ROLYI?= User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:9.0) Gecko/20111222 Thunderbird/9.0.1 MIME-Version: 1.0 To: freebsd-stable@freebsd.org Content-Type: text/plain; charset=ISO-8859-2 Content-Transfer-Encoding: 8bit Subject: Strange 'hangs' with RELENG_9 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 19 Jan 2012 14:03:25 -0000 Hello, Recently I updated my RELENG_8 to RELENG_9. Since then, the server hangs from time to time for 5 minutes. When I run a top in a remote terminal, I can see that it hangs so strong, that the clock hangs too. When it continues to run , the time continues from the when it 'hanged'. TCP connections are also dropped with timeout at that time. However, no kernel panic, and i can't see anything in the dmesg log too. A strange thing is, the server continues working when I press a key at the physical console (I'm doing this with a remote IP console). More strange thing is, when I do a reboot, the server flushes all its disks, and then does a panic, instead of rebooting. I have to revert to the RELENG_8 kernel (userland is RELENG_9 now), I have no other choice. I hardly can get the configuration and log out from it these times, because of the hangs. Hardware details: This has 4 SAMSUNG disk (1.5TB each) array, driven by a 3ware Raid controller, each disk exported as is. It also has an OCZ Revodrive as a disk cache (zfs L2ARC cache) limited to SATA1 speed (strange kernel panics because of disk timeouts when using at full speed), 8GB RAM, AMD64 processor. FreeBSD details: The server runs on the 4-disk zfs array, boots from it and uses the zfs array also as root media. It has 4 jails, connections handled by pf. Kernel configuration: cpu HAMMER ident MYSERVER machine amd64 options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options INET6 # IPv6 communications protocols options SCTP # Stream Control Transmission Protocol options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options NFSCLIENT # Network Filesystem Client options NFSLOCKD # Network Lock Manager options MSDOSFS # MSDOS Filesystem options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options FLOWTABLE # per-cpu routing cache options INCLUDE_CONFIG_FILE # Include this file in kernel options SMP # Symmetric MultiProcessor Kernel device cpufreq device acpi device pci device ata device atadisk # ATA disk drives device ataraid # ATA RAID drives options ATA_STATIC_ID # Static device numbering device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device twa # 3ware 9000 series PATA/SATA RAID device atkbdc # AT keyboard controller device atkbd # AT keyboard device psm # PS/2 mouse device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support device sc device agp # support several AGP chipsets device uart # Generic UART driver device ppc device ppbus # Parallel port bus (required) device lpt # Printer device miibus # MII bus support device re # RealTek 8139C+/8169/8169S/8110S device loop # Network loopback device random # Entropy device device ether # Ethernet support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device gif # IPv6 and IPv4 tunneling device faith # IPv6-to-IPv4 relaying (translation) device firmware # firmware assist module device bpf # Berkeley packet filter options USB_DEBUG # enable debug msgs device uhci # UHCI PCI->USB interface device ohci # OHCI PCI->USB interface device ehci # EHCI PCI->USB interface (USB 2.0) device xhci # XHCI PCI->USB interface (USB 3.0) device usb # USB Bus (required) device uhid # "Human Interface Devices" device ukbd # Keyboard device ulpt # Printer device umass # Disks/Mass storage - Requires scbus and da device ums # Mouse device firewire # FireWire bus code device pf device pflog device pfsync device siis device ahci device pass device ada dmesg log, from boot to now: Copyright (c) 1992-2012 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 9.0-STABLE #20: Thu Jan 19 12:26:03 CET 2012 root@tilos.hu:/usr/obj/usr/src/sys/MYKERNEL amd64 CPU: AMD Phenom(tm) II X4 945 Processor (3013.69-MHz K8-class CPU) Origin = "AuthenticAMD" Id = 0x100f42 Family = 10 Model = 4 Stepping = 2 Features=0x178bfbff Features2=0x802009 AMD Features=0xee500800 AMD Features2=0x37ff TSC: P-state invariant real memory = 8589934592 (8192 MB) avail memory = 8118177792 (7742 MB) Event timer "LAPIC" quality 400 ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 1 cpu2 (AP): APIC ID: 2 cpu3 (AP): APIC ID: 3 ioapic0: Changing APIC ID to 2 ioapic0 irqs 0-23 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: Power Button (fixed) acpi0: reservation of 0, a0000 (3) failed acpi0: reservation of 100000, d7ce0000 (3) failed Timecounter "ACPI-fast" frequency 3579545 Hz quality 900 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 acpi_button0: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pcib0: Length mismatch for 3 range: 26d00000 vs 26e10000 pci0: on pcib0 pcib1: at device 1.0 on pci0 pci1: on pcib1 vgapci0: port 0xce00-0xceff mem 0xd8000000-0xdfffffff,0xfdbf0000-0xfdbfffff,0xfda00000-0xfdafffff irq 18 at device 5.0 on pci1 pcib2: irq 18 at device 2.0 on pci0 pci2: on pcib2 3ware device driver for 9000 series storage controllers, version: 3.80.06.003 twa0: <3ware 9000 series Storage Controller> port 0xbe00-0xbeff mem 0xfa000000-0xfbffffff,0xfd7ff000-0xfd7fffff irq 18 at device 0.0 on pci2 twa0: INFO: (0x15: 0x1300): Controller details:: Model 9650SE-8LPML, 8 ports, Firmware FE9X 3.08.00.016, BIOS BE9X 3.08.00.004 pcib3: irq 16 at device 4.0 on pci0 pci3: on pcib3 pcib4: irq 16 at device 0.0 on pci3 pci4: on pcib4 siis0: port 0xef00-0xef0f mem 0xfdfff000-0xfdfff07f,0xfdff0000-0xfdff7fff irq 16 at device 0.0 on pci4 siisch0: at channel 0 on siis0 siisch1: at channel 1 on siis0 siisch2: at channel 2 on siis0 siisch3: at channel 3 on siis0 pcib5: irq 18 at device 10.0 on pci0 pci5: on pcib5 re0: port 0xde00-0xdeff mem 0xfdcff000-0xfdcfffff,0xfdce0000-0xfdceffff irq 18 at device 0.0 on pci5 re0: Using 1 MSI-X message re0: Chip rev. 0x3c000000 re0: MAC rev. 0x00400000 miibus0: on re0 rgephy0: PHY 1 on miibus0 rgephy0: none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow re0: Ethernet address: 00:24:1d:d5:9e:b0 ahci0: port 0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f mem 0xfe02f000-0xfe02f3ff irq 22 at device 17.0 on pci0 ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported ahcich0: at channel 0 on ahci0 ahcich1: at channel 1 on ahci0 ahcich2: at channel 2 on ahci0 ahcich3: at channel 3 on ahci0 pci0: at device 18.0 (no driver attached) pci0: at device 18.1 (no driver attached) pci0: at device 18.2 (no driver attached) pci0: at device 19.0 (no driver attached) pci0: at device 19.1 (no driver attached) pci0: at device 19.2 (no driver attached) pci0: at device 20.0 (no driver attached) atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1 on pci0 ata0: at channel 0 on atapci0 ata1: at channel 1 on atapci0 isab0: at device 20.3 on pci0 isa0: on isab0 pcib6: at device 20.4 on pci0 pci6: on pcib6 fwohci0: mem 0xfd9ff000-0xfd9ff7ff,0xfd9f8000-0xfd9fbfff irq 22 at device 14.0 on pci6 fwohci0: OHCI version 1.10 (ROM=0) fwohci0: No. of Isochronous channels is 4. fwohci0: EUI64 00:4f:4d:c7:00:00:24:1d fwohci0: Phy 1394a available S400, 3 ports. fwohci0: Link S400, max_rec 2048 bytes. firewire0: on fwohci0 fwohci0: Initiate bus reset fwohci0: fwohci_intr_core: BUS reset fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode pci0: at device 20.5 (no driver attached) uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 ppc0: port 0x378-0x37f,0x778-0x77b irq 7 drq 3 on acpi0 ppc0: SMC-like chipset (ECP/EPP/PS2/NIBBLE) in COMPATIBLE mode ppc0: FIFO with 16/16/16 bytes threshold ppbus0: on ppc0 lpt0: on ppbus0 lpt0: Interrupt-driven port atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] attimer0: port 0x40-0x43 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 450 Event timer "HPET1" frequency 14318180 Hz quality 450 Event timer "HPET2" frequency 14318180 Hz quality 450 atrtc0: port 0x70-0x73 on acpi0 Event timer "RTC" frequency 32768 Hz quality 0 orm0: at iomem 0xd0000-0xd4fff,0xd5000-0xd6fff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 hwpstate0: on cpu0 firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0) (me) firewire0: bus manager 0 (noperiph:siisch0:0:-1:-1): rescan already queued (noperiph:siisch1:0:-1:-1): rescan already queued ZFS filesystem version 5 ZFS storage pool version 28 Timecounters tick every 1.000 msec da0 at twa0 bus 0 scbus0 target 0 lun 0 da0: Fixed Direct Access SCSI-5 device da0: 100.000MB/s transfers da0: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C) da1 at twa0 bus 0 scbus0 target 1 lun 0 da1: Fixed Direct Access SCSI-5 device da1: 100.000MB/s transfers da1: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C) da2 at twa0 bus 0 scbus0 target 2 lun 0 da2: Fixed Direct Access SCSI-5 device da2: 100.000MB/s transfers da2: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C) da3 at twa0 bus 0 scbus0 target 3 lun 0 da3: Fixed Direct Access SCSI-5 device da3: 100.000MB/s transfers da3: 1430501MB (2929666048 512 byte sectors: 255H 63S/T 182363C) ada0 at siisch0 bus 0 scbus1 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 38164MB (78161328 512 byte sectors: 16H 63S/T 16383C) ada1 at siisch1 bus 0 scbus2 target 0 lun 0 ada1: ATA-8 SATA 2.x device ada1: 150.000MB/s transfers (SATA 1.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 38164MB (78161328 512 byte sectors: 16H 63S/T 16383C) SMP: AP CPU #1 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #3 Launched! Timecounter "TSC-low" frequency 11772208 Hz quality 800 GEOM_RAID: SiI-110316131611: Array SiI-110316131611 created. GEOM_RAID: SiI-110316131611: Disk ada0 state changed from NONE to ACTIVE. GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:0-ada0 state changed from NONE to STALE. GEOM_RAID: SiI-110316131611: Disk ada1 state changed from NONE to ACTIVE. GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:1-ada1 state changed from NONE to STALE. GEOM_RAID: SiI-110316131611: Array started. GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:0-ada0 state changed from STALE to ACTIVE. GEOM_RAID: SiI-110316131611: Subdisk SiI Raid0 Set:1-ada1 state changed from STALE to ACTIVE. GEOM_RAID: SiI-110316131611: Volume SiI Raid0 Set state changed from STARTING to OPTIMAL. GEOM_RAID: SiI-110316131611: Provider raid/r0 for volume SiI Raid0 Set created. Trying to mount root from zfs:pool/root []... /boot/loader.conf: zfs_load="YES" vfs.root.mountfrom="zfs:pool/root" vfs.zfs.vdev.max_pending="8" geom_raid_load="YES" hint.siisch.0.sata_rev=1 hint.siisch.1.sata_rev=1 /etc/sysctl.conf: vfs.zfs.l2arc_noprefetch=0 /etc/make.conf, the kernel was compiled with this settings: CPUTYPE?=athlon64 I'd highly appreciate any help, as I am clueless with this one. -- László KÁROLYI http://linkedin.com/in/karolyi