From owner-freebsd-xen@FreeBSD.ORG Sun Dec 19 11:42:44 2010 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B0F86106564A for ; Sun, 19 Dec 2010 11:42:44 +0000 (UTC) (envelope-from tobiasztworek@gmail.com) Received: from mail-fx0-f49.google.com (mail-fx0-f49.google.com [209.85.161.49]) by mx1.freebsd.org (Postfix) with ESMTP id 436A88FC17 for ; Sun, 19 Dec 2010 11:42:44 +0000 (UTC) Received: by fxm19 with SMTP id 19so1904894fxm.36 for ; Sun, 19 Dec 2010 03:42:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:message-id:date:from :user-agent:mime-version:to:subject:content-type :content-transfer-encoding; bh=qOg/M6Spp/tjxppSM7T0p4JnCL3FjlQ+bUHWD5iRl7c=; b=SClW1CHggTbTCpYO96X/C38jn26DYOIRX8fjsWPnED4Bxoaa3EyGhPd7G7Y/9xEo5m DFutoyqRKLWx4kc9+LBWhE8pl91yobpxlvNGlSzDlNCYsqKUlrdf23c/+ICnLk984dgI efKGBDmlv2fwoAiDdRtas/BY1dkDc/bf4Ohcs= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=message-id:date:from:user-agent:mime-version:to:subject :content-type:content-transfer-encoding; b=KXc7q3aBixPsSkexQaFexOKKdx708kMOuyxY/S303af+btSL0mzBh94t7VvL+unJOH QoH2sRFXJa8WORbeQwmtoECNcubZNTfHhK4OJtmPNTpHqYwP7ONhwfs19Knw0PGnhpgX 0KY0q13XylFChULQE4SicwDbossjs9awpZ1HA= Received: by 10.223.107.141 with SMTP id b13mr3340601fap.86.1292758963327; Sun, 19 Dec 2010 03:42:43 -0800 (PST) Received: from [10.1.1.4] (87-205-171-54.adsl.inetia.pl [87.205.171.54]) by mx.google.com with ESMTPS id e17sm600961fak.10.2010.12.19.03.42.41 (version=TLSv1/SSLv3 cipher=RC4-MD5); Sun, 19 Dec 2010 03:42:42 -0800 (PST) Message-ID: <4D0DEFB0.8000304@tobnet.pl> Date: Sun, 19 Dec 2010 12:42:40 +0100 From: Admin TobNet User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; pl; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Subject: Sysinstall problem since FreeBSD Curent snapshot 201010 on XenServer 5.6 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 19 Dec 2010 11:42:44 -0000 Hellow I have sysinstall problem since FreeBSD Curent snapshot 201010 on XenServer 5.6. On snapshot 201009 sysinstall is working good. Since 201010 kernel hanging on "Timecounter "TSC frequency ... guality 800"" and sysinstall is never run. This bug is on amd64 and i386 platform :-( I test sanshots 201010,11,12 and bug is in all. My procesor is AMD Athlon(tm) II X4 640. Tobiasz Tworek From owner-freebsd-xen@FreeBSD.ORG Mon Dec 20 10:17:30 2010 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id B05611065670 for ; Mon, 20 Dec 2010 10:17:30 +0000 (UTC) (envelope-from joovke@joovke.com) Received: from fallbackmx10.syd.optusnet.com.au (fallbackmx10.syd.optusnet.com.au [211.29.132.251]) by mx1.freebsd.org (Postfix) with ESMTP id ECD648FC17 for ; Mon, 20 Dec 2010 10:17:29 +0000 (UTC) Received: from mail09.syd.optusnet.com.au (mail09.syd.optusnet.com.au [211.29.132.190]) by fallbackmx10.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id oBK8iITD015414 for ; Mon, 20 Dec 2010 19:44:19 +1100 Received: from [172.16.0.242] (c122-106-61-108.rivrw1.nsw.optusnet.com.au [122.106.61.108]) by mail09.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id oBK8iD5P001390 for ; Mon, 20 Dec 2010 19:44:15 +1100 Message-ID: <4D0F175D.8070701@joovke.com> Date: Mon, 20 Dec 2010 19:44:13 +1100 From: Alex User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.2.13) Gecko/20101208 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: lockups/freezes/pauses - 8.1-release/amd64 + XEN X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2010 10:17:30 -0000 Hi Guys, I am having rather odd issue where the system locks up, I have purchased a XEN based VPS and decided to go with FreeBSD as the OS. I frequently get cases where there seem to be noticeable pauses, ie: I could be SSH'd in and issue an "ls" command or "mc" to start midnight commander, as soon as i press enter to run the command, there is often a good pause for 20 seconds or so. Load averages are almost non existent because it's a new install, almost no network traffic hitting it. if i ping the IP address of the VPS during the pauses, there is abnormally high latency (instead of 200ms, 1500-2000ms). A traceroute during these times shows the high latency is at the final hop. I noticed some oddities, here is a full DMESG and my kernel config: -------------------------------------------------------------------- Copyright (c) 1992-2010 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 8.1-RELEASE-p2 #0: Sat Dec 18 02:49:02 EST 2010 alex@svr.mydomain.net:/usr/obj/usr/src/sys/custom amd64 Timecounter "i8254" frequency 1193182 Hz quality 0 CPU: Intel(R) Xeon(R) CPU E5620 @ 2.40GHz (2398.80-MHz K8-class CPU) Origin = "GenuineIntel" Id = 0x206c2 Family = 6 Model = 2c Stepping = 2 Features=0x1781fbff Features2=0x80982201> AMD Features=0x28100800 AMD Features2=0x1 TSC: P-state invariant real memory = 1572864000 (1500 MB) avail memory = 1508638720 (1438 MB) ACPI APIC Table: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs FreeBSD/SMP: 1 package(s) x 4 core(s) cpu0 (BSP): APIC ID: 0 cpu1 (AP): APIC ID: 2 cpu2 (AP): APIC ID: 4 cpu3 (AP): APIC ID: 6 ioapic0: Changing APIC ID to 1 MADT: Forcing active-low polarity and level trigger for SCI ioapic0 irqs 0-47 on motherboard kbd1 at kbdmux0 acpi0: on motherboard acpi0: [ITHREAD] acpi0: Power Button (fixed) acpi0: Sleep Button (fixed) Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <32-bit timer at 3.579545MHz> port 0x1f48-0x1f4b on acpi0 cpu0: on acpi0 cpu1: on acpi0 cpu2: on acpi0 cpu3: on acpi0 pcib0: port 0xcf8-0xcff on acpi0 pci0: on pcib0 isab0: at device 1.0 on pci0 isa0: on isab0 atapci0: port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc220-0xc22f at device 1.1 on pci0 ata0: on atapci0 ata0: [ITHREAD] ata1: on atapci0 ata1: [ITHREAD] pci0: at device 1.2 (no driver attached) pci0: at device 1.3 (no driver attached) vgapci0: mem 0xf0000000-0xf1ffffff,0xf3000000-0xf3000fff at device 2.0 on pci0 pci0: at device 3.0 (no driver attached) re0: port 0xc100-0xc1ff mem 0xf3001000-0xf30010ff irq 32 at device 4.0 on pci0 re0: Chip rev. 0x74800000 re0: MAC rev. 0x00000000 miibus0: on re0 rlphy0: PHY 0 on miibus0 rlphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto re0: Ethernet address: 00:16:3e:f0:8b:6c re0: [FILTER] atrtc0: port 0x70-0x71 irq 8 on acpi0 atkbdc0: port 0x60,0x64 irq 1 on acpi0 atkbd0: irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] atkbd0: [ITHREAD] orm0: at iomem 0xc9000-0xc97ff on isa0 sc0: at flags 0x100 on isa0 sc0: VGA <16 virtual consoles, flags=0x300> vga0: at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0 Timecounters tick every 1.000 msec ad0: 46080MB at ata0-master WDMA2 acd0: CDROM at ata1-master WDMA2 SMP: AP CPU #2 Launched! SMP: AP CPU #1 Launched! SMP: AP CPU #3 Launched! Trying to mount root from ufs:/dev/ad0s1a calcru: runtime went backwards from 23875254 usec to 22130573 usec for pid 1414 (csh) calcru: runtime went backwards from 5675 usec to 5054 usec for pid 1413 (su) calcru: runtime went backwards from 19908 usec to 17728 usec for pid 1395 (bash) calcru: runtime went backwards from 74003 usec to 65901 usec for pid 1395 (bash) calcru: runtime went backwards from 27707 usec to 24673 usec for pid 1390 (sshd) calcru: runtime went backwards from 16882 usec to 15033 usec for pid 1390 (sshd) calcru: runtime went backwards from 2960 usec to 2636 usec for pid 1377 (getty) calcru: runtime went backwards from 4201 usec to 3741 usec for pid 1376 (getty) calcru: runtime went backwards from 2334 usec to 2078 usec for pid 1375 (getty) calcru: runtime went backwards from 3360 usec to 2992 usec for pid 1374 (getty) calcru: runtime went backwards from 4099 usec to 3650 usec for pid 1373 (getty) calcru: runtime went backwards from 3052 usec to 2718 usec for pid 1372 (getty) calcru: runtime went backwards from 3001 usec to 2673 usec for pid 1371 (getty) calcru: runtime went backwards from 3503 usec to 3120 usec for pid 1370 (getty) calcru: runtime went backwards from 2561 usec to 2280 usec for pid 1293 (sshd) calcru: runtime went backwards from 20635 usec to 18376 usec for pid 1171 (sh) calcru: runtime went backwards from 88696 usec to 78985 usec for pid 1171 (sh) calcru: runtime went backwards from 231 usec to 205 usec for pid 675 (dhclient) calcru: runtime went backwards from 2342 usec to 2086 usec for pid 659 (dhclient) calcru: runtime went backwards from 25186 usec to 22429 usec for pid 659 (dhclient) calcru: runtime went backwards from 464 usec to 413 usec for pid 110 (adjkerntz) calcru: runtime went backwards from 10422 usec to 9861 usec for pid 1 (init) calcru: runtime went backwards from 61599028 usec to 56035925 usec for pid 1906 (httpd) calcru: runtime went backwards from 2636 usec to 2063 usec for pid 1377 (getty) calcru: runtime went backwards from 3741 usec to 2928 usec for pid 1376 (getty) calcru: runtime went backwards from 2078 usec to 1626 usec for pid 1375 (getty) calcru: runtime went backwards from 2992 usec to 2341 usec for pid 1374 (getty) calcru: runtime went backwards from 3650 usec to 2857 usec for pid 1373 (getty) calcru: runtime went backwards from 2718 usec to 2127 usec for pid 1372 (getty) calcru: runtime went backwards from 2673 usec to 2092 usec for pid 1371 (getty) calcru: runtime went backwards from 3120 usec to 2441 usec for pid 1370 (getty) calcru: runtime went backwards from 2511 usec to 1965 usec for pid 1337 (inetd) calcru: runtime went backwards from 88691784 usec to 69414995 usec for pid 1337 (inetd) calcru: runtime went backwards from 96115 usec to 76173 usec for pid 1293 (sshd) calcru: runtime went backwards from 18376 usec to 14382 usec for pid 1171 (sh) calcru: runtime went backwards from 78985 usec to 61818 usec for pid 1171 (sh) calcru: runtime went backwards from 1542 usec to 1306 usec for pid 676 (devd) calcru: runtime went backwards from 2086 usec to 1632 usec for pid 659 (dhclient) calcru: runtime went backwards from 22429 usec to 17554 usec for pid 659 (dhclient) calcru: runtime went backwards from 413 usec to 323 usec for pid 110 (adjkerntz) calcru: runtime went backwards from 10170 usec to 8126 usec for pid 1 (init) calcru: runtime went backwards from 111215702 usec to 89125025 usec for pid 1 (init) ---------------------------------------------------------------------- [alex@svr ~]$ cat /usr/src/sys/amd64/conf/custom # cpu HAMMER ident custom # To statically compile in device wiring instead of /boot/device.hints #hints "GENERIC.hints" # Default places to look for devices. # Use the following to compile in values accessible to the kernel # through getenv() (or kenv(1) in userland). The format of the file # is 'variable=value', see kenv(1) # # env "GENERIC.env" makeoptions DEBUG=-g # Build kernel with gdb(1) debug symbols options SCHED_ULE # ULE scheduler options PREEMPTION # Enable kernel thread preemption options INET # InterNETworking options FFS # Berkeley Fast Filesystem options SOFTUPDATES # Enable FFS soft updates support options UFS_ACL # Support for access control lists options UFS_DIRHASH # Improve performance on big directories options UFS_GJOURNAL # Enable gjournal-based UFS journaling options MD_ROOT # MD is a potential root device options MSDOSFS # MSDOS Filesystem options CD9660 # ISO 9660 Filesystem options PROCFS # Process filesystem (requires PSEUDOFS) options PSEUDOFS # Pseudo-filesystem framework options GEOM_PART_GPT # GUID Partition Tables. options GEOM_LABEL # Provides labelization options COMPAT_43TTY # BSD 4.3 TTY compat (sgtty) options COMPAT_FREEBSD32 # Compatible with i386 binaries options COMPAT_FREEBSD4 # Compatible with FreeBSD4 options COMPAT_FREEBSD5 # Compatible with FreeBSD5 options COMPAT_FREEBSD6 # Compatible with FreeBSD6 options COMPAT_FREEBSD7 # Compatible with FreeBSD7 options SCSI_DELAY=5000 # Delay (in ms) before probing SCSI options KTRACE # ktrace(1) support options STACK # stack(9) support options SYSVSHM # SYSV-style shared memory options SYSVMSG # SYSV-style message queues options SYSVSEM # SYSV-style semaphores options P1003_1B_SEMAPHORES # POSIX-style semaphores options _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions options PRINTF_BUFR_SIZE=128 # Prevent printf output being interspersed. options KBD_INSTALL_CDEV # install a CDEV entry in /dev options HWPMC_HOOKS # Necessary kernel hooks for hwpmc(4) options AUDIT # Security event auditing options MAC # TrustedBSD MAC Framework options FLOWTABLE # per-cpu routing cache #options KDTRACE_FRAME # Ensure frames are compiled in #options KDTRACE_HOOKS # Kernel DTrace hooks options INCLUDE_CONFIG_FILE # Include this file in kernel # Make an SMP-capable kernel by default options SMP # Symmetric MultiProcessor Kernel # CPU frequency control device cpufreq # Bus support. device acpi device pci # ATA and ATAPI devices device ata device atadisk # ATA disk drives device atapicd # ATAPI CDROM drives device atapist # ATAPI tape drives options ATA_STATIC_ID # Static device numbering # SCSI peripherals device scbus # SCSI bus (required for SCSI) device da # Direct Access (disks) device cd # CD device pass # Passthrough device (direct SCSI access) # atkbdc0 controls both the keyboard and the PS/2 mouse device atkbdc # AT keyboard controller device atkbd # AT keyboard device kbdmux # keyboard multiplexer device vga # VGA video card driver device splash # Splash screen and screen saver support # syscons is the default console driver, resembling an SCO console device sc # PCI Ethernet NICs that use the common MII bus controller code. # NOTE: Be sure to keep the 'device miibus' line in order to use these NICs! device miibus # MII bus support device re # RealTek 8139C+/8169/8169S/8110S # Pseudo devices. device loop # Network loopback device random # Entropy device device ether # Ethernet support device vlan # 802.1Q VLAN support device tun # Packet tunnel. device pty # BSD-style compatibility pseudo ttys device md # Memory "disks" device firmware # firmware assist module # The `bpf' device enables the Berkeley Packet Filter. # Be aware of the administrative consequences of enabling this! # Note that 'bpf' is required for DHCP. device bpf # Berkeley packet filter device pf device pflog device pfsync # Xen HVM support #options XENHVM #device xenpci ------------------------------------------- The XENHVM options commented out causes the VM not to boot (seems to hang after saying it's found a generic block device attached to the xenpci bus (virtual hard disk) and that it's adding it as ad0. The calcru: runtime messages are rather puzzling, perhaps this is related to the hangs I am seeing regularly. Any ideas on where to start looking? Thanks! Alex. From owner-freebsd-xen@FreeBSD.ORG Mon Dec 20 11:07:10 2010 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id A75FD10656CB for ; Mon, 20 Dec 2010 11:07:10 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 94B478FC1D for ; Mon, 20 Dec 2010 11:07:10 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id oBKB7AIe024732 for ; Mon, 20 Dec 2010 11:07:10 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id oBKB7A33024730 for freebsd-xen@FreeBSD.org; Mon, 20 Dec 2010 11:07:10 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 20 Dec 2010 11:07:10 GMT Message-Id: <201012201107.oBKB7A33024730@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-xen@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-xen@FreeBSD.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2010 11:07:10 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/153150 xen [xen] xen/ec2: disable checksum offloading on interfac o kern/152228 xen [xen] [panic] Xen/PV panic with machdep.idle_mwait=1 o kern/144629 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143398 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143340 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143069 xen [xen] [panic] Xen Kernel Panic - Memory modified after o kern/141328 xen [xen] [panic] gstat exit causes kernel panic from unma o kern/140313 xen [xen] [panic] FreeBSD8 RC2 as PV domU crashes during c o kern/135421 xen [xen] FreeBSD Xen PVM DomU network failure - netfronc. f kern/135178 xen [xen] Xen domU outgoing data transfer stall when TSO i o kern/135069 xen [xen] FreeBSD-current/Xen SMP doesn't function at all f i386/124516 xen [xen] FreeBSD-CURRENT Xen Kernel Segfaults when config 12 problems total. From owner-freebsd-xen@FreeBSD.ORG Mon Dec 20 21:35:20 2010 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx2.freebsd.org (mx2.freebsd.org [IPv6:2001:4f8:fff6::35]) by hub.freebsd.org (Postfix) with ESMTP id 7A36D106564A for ; Mon, 20 Dec 2010 21:35:20 +0000 (UTC) (envelope-from cperciva@freebsd.org) Received: from xps.daemonology.net (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx2.freebsd.org (Postfix) with SMTP id 420D614F16A for ; Mon, 20 Dec 2010 21:35:14 +0000 (UTC) Received: (qmail 41320 invoked from network); 20 Dec 2010 21:35:13 -0000 Received: from unknown (HELO xps.daemonology.net) (127.0.0.1) by localhost with SMTP; 20 Dec 2010 21:35:13 -0000 Message-ID: <4D0FCC11.2020306@freebsd.org> Date: Mon, 20 Dec 2010 13:35:13 -0800 From: Colin Percival User-Agent: Mozilla/5.0 (X11; U; FreeBSD amd64; en-US; rv:1.9.1.11) Gecko/20100803 Thunderbird/3.0.6 MIME-Version: 1.0 To: Collin Forbes References: <4D066D52.2080100@freebsd.org> <1292328289.26377.48.camel@pow> <4D07A52A.1080208@freebsd.org> <20101220212118.GA17665@gmail.com> In-Reply-To: <20101220212118.GA17665@gmail.com> X-Enigmail-Version: 1.0.1 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: "freebsd-xen@freebsd.org" Subject: Re: FreeBSD/EC2 lives! X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2010 21:35:20 -0000 On 12/20/10 13:21, Collin Forbes wrote: > the last line I see before the panic is: > > checking whether mlock is broken... Yep, known issue -- this is PR kern/140313. > I'm a debugging n00b however. What do you need in terms of a debugging > report? I have files in /var/crash as a result of the two panics at > exactly the same place. > > -rw-r--r-- 1 root wheel 2 Dec 20 21:16 bounds > -rw------- 1 root wheel 149546 Dec 20 20:45 core.txt.0 > -rw------- 1 root wheel 140384 Dec 20 21:16 core.txt.1 > -rw------- 1 root wheel 519 Dec 20 20:45 info.0 > -rw------- 1 root wheel 518 Dec 20 21:16 info.1 > -rw-r--r-- 1 root wheel 5 Dec 11 21:52 minfree > -rw------- 1 root wheel 118460416 Dec 20 20:45 vmcore.0 > -rw------- 1 root wheel 102764544 Dec 20 21:16 vmcore.1 Can you look in the core.txt.[01] files and make sure that the backtraces match the backtrace at http://www.freebsd.org/cgi/query-pr.cgi?pr=kern/140313 Assuming they're the same, I don't think I need any more data... of course, if you want to dig through the FreeBSD kernel source code and figure out why the vm page queue mutex is not owned, that would be great. :-) (My guess is that the VM code was adjusted to push that lock down (i.e., hold it for less time) and the xen code wasn't modified at the same time, but I haven't had time to investigate this fully yet.) -- Colin Percival Security Officer, FreeBSD | freebsd.org | The power to serve Founder / author, Tarsnap | tarsnap.com | Online backups for the truly paranoid From owner-freebsd-xen@FreeBSD.ORG Mon Dec 20 21:53:34 2010 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id F2565106564A for ; Mon, 20 Dec 2010 21:53:34 +0000 (UTC) (envelope-from collin.forbes@gmail.com) Received: from mail-gy0-f182.google.com (mail-gy0-f182.google.com [209.85.160.182]) by mx1.freebsd.org (Postfix) with ESMTP id A6B408FC0C for ; Mon, 20 Dec 2010 21:53:34 +0000 (UTC) Received: by gyf3 with SMTP id 3so1504865gyf.13 for ; Mon, 20 Dec 2010 13:53:34 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:received:received:date:from:to:cc:subject :message-id:references:mime-version:content-type:content-disposition :in-reply-to:x-message-flag:user-agent; bh=XqNQLGVkbSbmgGfpmWTxyDW2TOTp32NzMIV9qxe73x0=; b=tOY2dxxcGOwldP/5KneltjLhFI9vBemn6J7gm6yp/8ZuFPLE7e/5pzSyJOaETMW2TC BlLOGWXKNBBoqS/I650J5rCVAkiulZah/GH190ljVIYzQdlBUK2E4ILYUDci8h1fVR8d /Ek3Nr3vQQnDXpXsM1OqpwhWAqq/+MTafxOZM= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=date:from:to:cc:subject:message-id:references:mime-version :content-type:content-disposition:in-reply-to:x-message-flag :user-agent; b=G9Or6ifpUc0tmHmwHmfyJDvlGSZ9SivGSZsDBY/+7GL8RoAmwzqt1c/8zwi2dR+aGi 240WbdtAxgkpDAbRQHywXQqmgpEU8bgx2fl0m97Oqvv4PBfRV0UHDHoA8QhXyYO1Pe8Q nAwn/y7TDc4StvwEZ39QU2FlncBSwjfJwkuaE= Received: by 10.150.53.18 with SMTP id b18mr7173527yba.351.1292880200837; Mon, 20 Dec 2010 13:23:20 -0800 (PST) Received: from gmail.com (71-214-187-87.tcso.qwest.net [71.214.187.87]) by mx.google.com with ESMTPS id 72sm2312075yhl.38.2010.12.20.13.23.18 (version=TLSv1/SSLv3 cipher=RC4-MD5); Mon, 20 Dec 2010 13:23:20 -0800 (PST) Date: Mon, 20 Dec 2010 14:21:20 -0700 From: Collin Forbes To: Colin Percival Message-ID: <20101220212118.GA17665@gmail.com> References: <4D066D52.2080100@freebsd.org> <1292328289.26377.48.camel@pow> <4D07A52A.1080208@freebsd.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4D07A52A.1080208@freebsd.org> X-Message-Flag: Beep Beep! User-Agent: Mutt/1.5.20 (2009-06-14) Cc: "freebsd-xen@freebsd.org" Subject: Re: FreeBSD/EC2 lives! X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 20 Dec 2010 21:53:35 -0000 On Tue, Dec 14, 2010 at 09:11:06AM -0800, Colin Percival wrote: > > Probably the biggest thing you can do to help right now is to spin up a bunch > of instances, throw some load at them, and see if you can get them to panic. :-) I signed up for Amazon EC2 last night and managed to panic my FreeBSD instance a couple times today while compiling security/libgcrypt out of /usr/ports. The panics happen during the configure phase, and the last line I see before the panic is: checking whether mlock is broken... I'm a debugging n00b however. What do you need in terms of a debugging report? I have files in /var/crash as a result of the two panics at exactly the same place. -rw-r--r-- 1 root wheel 2 Dec 20 21:16 bounds -rw------- 1 root wheel 149546 Dec 20 20:45 core.txt.0 -rw------- 1 root wheel 140384 Dec 20 21:16 core.txt.1 -rw------- 1 root wheel 519 Dec 20 20:45 info.0 -rw------- 1 root wheel 518 Dec 20 21:16 info.1 -rw-r--r-- 1 root wheel 5 Dec 11 21:52 minfree -rw------- 1 root wheel 118460416 Dec 20 20:45 vmcore.0 -rw------- 1 root wheel 102764544 Dec 20 21:16 vmcore.1 And finally my "uname -a" is: FreeBSD ip-10-196-90-178 9.0-CURRENT FreeBSD 9.0-CURRENT #68: Sun Dec 12 03:52:29 UTC 2010 root@chch.daemonology.net:/usr/obj/i386.i386/usr/src/sys/XEN i386 (Sent to the freebsd-xen mailing list as well) -- Collin Forbes collin.forbes@gmail.com From owner-freebsd-xen@FreeBSD.ORG Tue Dec 21 04:21:38 2010 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8DB7A106564A for ; Tue, 21 Dec 2010 04:21:38 +0000 (UTC) (envelope-from joovke@joovke.com) Received: from one.digitalpacific.com.au (one.digitalpacific.com.au [203.19.59.9]) by mx1.freebsd.org (Postfix) with ESMTP id 1D3858FC15 for ; Tue, 21 Dec 2010 04:21:37 +0000 (UTC) X-ClientAddr: 150.101.205.12 Received: from [127.0.0.1] (eth525.nsw.adsl.internode.on.net [150.101.205.12]) (authenticated bits=0) by one.digitalpacific.com.au (8.13.8/8.13.8) with ESMTP id oBL3eKcI024929 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NO) for ; Tue, 21 Dec 2010 14:40:22 +1100 Message-ID: <4D0F78A6.4070204@joovke.com> Date: Tue, 21 Dec 2010 02:39:18 +1100 From: Alex User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.9.2.13) Gecko/20101207 Thunderbird/3.1.7 MIME-Version: 1.0 To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DigitalPacific-MailScanner-Information: Please contact support@digitalpacific.com.au for more information X-DigitalPacific-MailScanner-ID: oBL3eKcI024929 X-DigitalPacific-MailScanner: Found to be clean X-DigitalPacific-MailScanner-From: joovke@joovke.com X-Spam-Status: No Subject: Re: lockups/freezes/pauses - 8.1-release/amd64 + XEN X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 21 Dec 2010 04:21:38 -0000 Found something interesting from an email sent by the daily cron: Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll re0 1500 00:16:3e:f0:8b:6c 160763 0 0 12053 0 12024 Uptime is about a day, 12024 collisions. That couldnt be good! Houston we have a problem. Now whether this is a bug with the re() driver or a side effect of something else, I dont know. From owner-freebsd-xen@FreeBSD.ORG Sat Dec 25 10:10:10 2010 Return-Path: Delivered-To: freebsd-xen@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 546A51065675 for ; Sat, 25 Dec 2010 10:10:10 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 431CF8FC0C for ; Sat, 25 Dec 2010 10:10:10 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id oBPAAAr8003098 for ; Sat, 25 Dec 2010 10:10:10 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id oBPAAALO003095; Sat, 25 Dec 2010 10:10:10 GMT (envelope-from gnats) Date: Sat, 25 Dec 2010 10:10:10 GMT Message-Id: <201012251010.oBPAAALO003095@freefall.freebsd.org> To: freebsd-xen@FreeBSD.org From: Colin Percival Cc: Subject: Re: kern/140313: [xen] [panic] FreeBSD8 RC2 as PV domU crashes during configure of security/libgcrypt X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: Colin Percival List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 25 Dec 2010 10:10:10 -0000 The following reply was made to PR kern/140313; it has been noted by GNATS. From: Colin Percival To: bug-followup@FreeBSD.org, mister.olli@googlemail.com Cc: Subject: Re: kern/140313: [xen] [panic] FreeBSD8 RC2 as PV domU crashes during configure of security/libgcrypt Date: Sat, 25 Dec 2010 02:06:45 -0800 Test case: #include int main() { int i; mlock(&i, 4); return (0); } This appears to have been partially fixed by SVN r204160, but pmap_pte_release needs to be updated the same way as pmap_pte. I'll commit this once I've had a chance to test further. -- Colin Percival Security Officer, FreeBSD | freebsd.org | The power to serve Founder / author, Tarsnap | tarsnap.com | Online backups for the truly paranoid