From owner-freebsd-xen@FreeBSD.ORG Mon Dec 20 10:17:30 2010 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B05611065670 for ; Mon, 20 Dec 2010 10:17:30 +0000 (UTC) (envelope-from joovke@joovke.com) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id ECD648FC17 for ; Mon, 20 Dec 2010 10:17:29 +0000 (UTC) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id oBK8iITD015414 for ; Mon, 20 Dec 2010 19:44:19 +1100 Received: from [172.16.0.242] (c122-106-61-108.rivrw1.nsw.optusnet.com.au [122.106.61.108]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id oBK8iD5P001390 for ; Mon, 20 Dec 2010 19:44:15 +1100 Message-ID: <4D0F175D.8070701@joovke.com> Date: Mon, 20 Dec 2010 19:44:13 +1100 From: Alex User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: lockups/freezes/pauses - 8.1-release/amd64 + XEN X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2010 10:17:30 -0000 Hi Guys, I am having rather odd issue where the system locks up, I have purchased a XEN based VPS and decided to go with FreeBSD as the OS. I frequently get cases where there seem to be noticeable pauses, ie: I could be SSH'd in and issue an "ls" command or "mc" to start midnight commander, as soon as i press enter to run the command, there is often a good pause for 20 seconds or so. Load averages are almost non existent because it's a new install, almost no network traffic hitting it. if i ping the IP address of the VPS during the pauses, there is abnormally high latency (instead of 200ms, 1500-2000ms). A traceroute during these times shows the high latency is at the final hop. I noticed some oddities, here is a full DMESG and my kernel config: -------------------------------------------------------------------- Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE-p2 #0: Sat Dec 18 02:49:02 EST 2010 alex@svr.mydomain.net:/usr/obj/usr/src/sys/custom amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2398.80-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c Stepping = 2 Features=0x1781fbff Features2=0x80982201> AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant real memory = 1572864000 (1500 MB) avail memory = 1508638720 (1438 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 ioapic0: Changing APIC ID to 1 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-47 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x1f48-0x1f4b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc220-0xc22f at device 1.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 1.2 (no driver attached) pci0: at device 1.3 (no driver attached) vgapci0: mem 0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0 pci0: at device 3.0 (no driver attached) re0: port 0xc100-0xc1ff mem 0xf3001000-0xf30010ff irq 32 at device 4.0 on pci0 re0: Chip rev. 0x74800000 re0: MAC rev. 0x00000000 miibus0: on re0 rlphy0: PHY 0 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto re0: Ethernet address: 00:16:3e:f0:8b:6c re0: [FILTER] atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] orm0: at iomem 0xc9000-0xc97ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ad0: 46080MB at ata0-master WDMA2 acd0: CDROM at ata1-master WDMA2 SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/ad0s1a calcru: runtime went backwards from 23875254 usec to 22130573 usec for pid 1414 (csh) calcru: runtime went backwards from 5675 usec to 5054 usec for pid 1413 (su) calcru: runtime went backwards from 19908 usec to 17728 usec for pid 1395 (bash) calcru: runtime went backwards from 74003 usec to 65901 usec for pid 1395 (bash) calcru: runtime went backwards from 27707 usec to 24673 usec for pid 1390 (sshd) calcru: runtime went backwards from 16882 usec to 15033 usec for pid 1390 (sshd) calcru: runtime went backwards from 2960 usec to 2636 usec for pid 1377 (getty) calcru: runtime went backwards from 4201 usec to 3741 usec for pid 1376 (getty) calcru: runtime went backwards from 2334 usec to 2078 usec for pid 1375 (getty) calcru: runtime went backwards from 3360 usec to 2992 usec for pid 1374 (getty) calcru: runtime went backwards from 4099 usec to 3650 usec for pid 1373 (getty) calcru: runtime went backwards from 3052 usec to 2718 usec for pid 1372 (getty) calcru: runtime went backwards from 3001 usec to 2673 usec for pid 1371 (getty) calcru: runtime went backwards from 3503 usec to 3120 usec for pid 1370 (getty) calcru: runtime went backwards from 2561 usec to 2280 usec for pid 1293 (sshd) calcru: runtime went backwards from 20635 usec to 18376 usec for pid 1171 (sh) calcru: runtime went backwards from 88696 usec to 78985 usec for pid 1171 (sh) calcru: runtime went backwards from 231 usec to 205 usec for pid 675 (dhclient) calcru: runtime went backwards from 2342 usec to 2086 usec for pid 659 (dhclient) calcru: runtime went backwards from 25186 usec to 22429 usec for pid 659 (dhclient) calcru: runtime went backwards from 464 usec to 413 usec for pid 110 (adjkerntz) calcru: runtime went backwards from 10422 usec to 9861 usec for pid 1 (init) calcru: runtime went backwards from 61599028 usec to 56035925 usec for pid 1906 (httpd) calcru: runtime went backwards from 2636 usec to 2063 usec for pid 1377 (getty) calcru: runtime went backwards from 3741 usec to 2928 usec for pid 1376 (getty) calcru: runtime went backwards from 2078 usec to 1626 usec for pid 1375 (getty) calcru: runtime went backwards from 2992 usec to 2341 usec for pid 1374 (getty) calcru: runtime went backwards from 3650 usec to 2857 usec for pid 1373 (getty) calcru: runtime went backwards from 2718 usec to 2127 usec for pid 1372 (getty) calcru: runtime went backwards from 2673 usec to 2092 usec for pid 1371 (getty) calcru: runtime went backwards from 3120 usec to 2441 usec for pid 1370 (getty) calcru: runtime went backwards from 2511 usec to 1965 usec for pid 1337 (inetd) calcru: runtime went backwards from 88691784 usec to 69414995 usec for pid 1337 (inetd) calcru: runtime went backwards from 96115 usec to 76173 usec for pid 1293 (sshd) calcru: runtime went backwards from 18376 usec to 14382 usec for pid 1171 (sh) calcru: runtime went backwards from 78985 usec to 61818 usec for pid 1171 (sh) calcru: runtime went backwards from 1542 usec to 1306 usec for pid 676 (devd) calcru: runtime went backwards from 2086 usec to 1632 usec for pid 659 (dhclient) calcru: runtime went backwards from 22429 usec to 17554 usec for pid 659 (dhclient) calcru: runtime went backwards from 413 usec to 323 usec for pid 110 (adjkerntz) calcru: runtime went backwards from 10170 usec to 8126 usec for pid 1 (init) calcru: runtime went backwards from 111215702 usec to 89125025 usec for pid 1 (init) ---------------------------------------------------------------------- [alex@svr ~]$ cat /usr/src/sys/amd64/conf/custom # cpu HAMMER ident custom # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. # Use the following to compile in values accessible to the kernel # through getenv() (or kenv(1) in userland). The format of the file # is 'variable=value', see kenv(1) # # env "GENERIC.env" makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options COMPAT_FREEBSD32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # CPU frequency control device cpufreq # Bus support. device acpi device pci # ATA and ATAPI devices device ata device atadisk # ATA disk drives device atapicd # ATAPI CDROM drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI peripherals device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct SCSI access) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device re # RealTek 8139C+/8169/8169S/8110S # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter device pf device pflog device pfsync # Xen HVM support #options XENHVM #device xenpci ------------------------------------------- The XENHVM options commented out causes the VM not to boot (seems to hang after saying it's found a generic block device attached to the xenpci bus (virtual hard disk) and that it's adding it as ad0. The calcru: runtime messages are rather puzzling, perhaps this is related to the hangs I am seeing regularly. Any ideas on where to start looking? Thanks! Alex.