Date: Thu, 23 May 2013 11:06:18 +0200 From: =?ISO-8859-1?Q?Roger_Pau_Monn=E9?= <roger.pau@citrix.com> To: Colin Percival <cperciva@freebsd.org> Cc: "freebsd-xen@freebsd.org" <freebsd-xen@freebsd.org>, xen-devel <xen-devel@lists.xen.org>, "freebsd-virtualization@freebsd.org" <freebsd-virtualization@freebsd.org>, "xen-users@lists.xen.org" <xen-users@lists.xen.org> Subject: Re: FreeBSD PVHVM call for testing Message-ID: <519DDC0A.9000201@citrix.com> In-Reply-To: <519D24A9.3050407@freebsd.org> References: <519131D8.9010307@citrix.com> <51952BAE.6010609@freebsd.org> <51957D42.9060801@citrix.com> <51959ED9.6040405@freebsd.org> <51974EC9.9030204@citrix.com> <5197A1EA.2040404@freebsd.org> <519CAFC7.1070908@citrix.com> <519D24A9.3050407@freebsd.org>
next in thread | previous in thread | raw e-mail | index | archive | help
--------------000309090701090501040108
Content-Type: text/plain; charset="ISO-8859-1"
Content-Transfer-Encoding: 8bit
On 22/05/13 22:03, Colin Percival wrote:
> On 05/22/13 04:45, Roger Pau Monné wrote:
>> On 18/05/13 17:44, Colin Percival wrote:
>>> That seems to work. dmesg is attached. Are there any particular tests
>>> you'd like me to run?
>>
>> I have not tested ZFS, that might be a good one. If you are running this
>> on Xen 3.4 the behaviour should be the same as without this patches, so
>> there shouldn't be many differences.
>
> I don't use ZFS personally, so I'm not sure exactly what tests to run on it;
> hopefully someone else can take care of that.
>
>> If you could try that on Xen 4.0 at least (if I remember correctly
>> that's when the vector callback was introduced), you should see the PV
>> timer getting attached, and a performance increase.
>
> Testing on a cr1.8xlarge EC2 instance, I get Xen 4.2, but it ends up with
> a panic -- console output below. I can get a backtrace and possibly even
> a dump if those would help.
Hello Colin,
Thanks for the test, I've been using Xen 4.2 (and 4.3) without problems
so far. By looking at the Xen code, the only reason the timer setup
could return -22 (EINVAL), is that we try to set the timer for a
different vCPU than the one we are running on.
I've been able to boot a 32 vCPU DomU on my 8way box using Xen 4.2.1
(using both qemu-xen and qemu-xen-traditional device models), so I'm
unsure if this could be due to some patch Amazon applies to Xen. Could
you try the following patch and post the error message? I would like to
see if the cpuid reported by kdb and the vCPU that we are trying to set
the timer are the same.
Booting...
GDB: no debug ports present
KDB: debugger backends: ddb
KDB: current backend: ddb
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 10.0-CURRENT #68: Wed May 22 19:00:14 CEST 2013
root@:/usr/obj/usr/src/sys/XENHVM amd64
FreeBSD clang version 3.3 (trunk 178860) 20130405
WARNING: WITNESS option enabled, expect reduced performance.
XEN: Hypervisor version 4.2 detected.
CPU: Intel(R) Xeon(R) CPU W3550 @ 3.07GHz (3066.83-MHz K8-class CPU)
Origin = "GenuineIntel" Id = 0x106a5 Family = 0x6 Model = 0x1a Stepping = 5
Features=0x1783fbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE,SSE2,HTT>
Features2=0x81b82201<SSE3,SSSE3,CX16,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,HV>
AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
AMD Features2=0x1<LAHF>
real memory = 4286578688 (4088 MB)
avail memory = 3961323520 (3777 MB)
Event timer "LAPIC" quality 400
ACPI APIC Table: <Xen HVM>
FreeBSD/SMP: Multiprocessor System Detected: 32 CPUs
FreeBSD/SMP: 2 package(s) x 16 core(s)
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 2
cpu2 (AP): APIC ID: 4
cpu3 (AP): APIC ID: 6
cpu4 (AP): APIC ID: 8
cpu5 (AP): APIC ID: 10
cpu6 (AP): APIC ID: 12
cpu7 (AP): APIC ID: 14
cpu8 (AP): APIC ID: 16
cpu9 (AP): APIC ID: 18
cpu10 (AP): APIC ID: 20
cpu11 (AP): APIC ID: 22
cpu12 (AP): APIC ID: 24
cpu13 (AP): APIC ID: 26
cpu14 (AP): APIC ID: 28
cpu15 (AP): APIC ID: 30
cpu16 (AP): APIC ID: 32
cpu17 (AP): APIC ID: 34
cpu18 (AP): APIC ID: 36
cpu19 (AP): APIC ID: 38
cpu20 (AP): APIC ID: 40
cpu21 (AP): APIC ID: 42
cpu22 (AP): APIC ID: 44
cpu23 (AP): APIC ID: 46
cpu24 (AP): APIC ID: 48
cpu25 (AP): APIC ID: 50
cpu26 (AP): APIC ID: 52
cpu27 (AP): APIC ID: 54
cpu28 (AP): APIC ID: 56
cpu29 (AP): APIC ID: 58
cpu30 (AP): APIC ID: 60
cpu31 (AP): APIC ID: 62
random device not loaded; using insecure entropy
ioapic0: Changing APIC ID to 1
MADT: Forcing active-low polarity and level trigger for SCI
ioapic0 <Version 1.1> irqs 0-47 on motherboard
kbd1 at kbdmux0
xen_et0: <Xen PV Clock> on motherboard
Event timer "XENTIMER" frequency 1000000000 Hz quality 950
Timecounter "XENTIMER" frequency 1000000000 Hz quality 950
acpi0: <Xen> on motherboard
acpi0: Power Button (fixed)
acpi0: Sleep Button (fixed)
acpi0: reservation of 0, a0000 (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
cpu16: <ACPI CPU> on acpi0
cpu17: <ACPI CPU> on acpi0
cpu18: <ACPI CPU> on acpi0
cpu19: <ACPI CPU> on acpi0
cpu20: <ACPI CPU> on acpi0
cpu21: <ACPI CPU> on acpi0
cpu22: <ACPI CPU> on acpi0
cpu23: <ACPI CPU> on acpi0
cpu24: <ACPI CPU> on acpi0
cpu25: <ACPI CPU> on acpi0
cpu26: <ACPI CPU> on acpi0
cpu27: <ACPI CPU> on acpi0
cpu28: <ACPI CPU> on acpi0
cpu29: <ACPI CPU> on acpi0
cpu30: <ACPI CPU> on acpi0
cpu31: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 62500000 Hz quality 950
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <32-bit timer at 3.579545MHz> port 0xb008-0xb00b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
isab0: <PCI-ISA bridge> at device 1.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel PIIX3 WDMA2 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xc300-0xc30f at device 1.1 on pci0
ata0: <ATA channel> at channel 0 on atapci0
ata1: <ATA channel> at channel 1 on atapci0
pci0: <bridge> at device 1.3 (no driver attached)
vgapci0: <VGA-compatible display> mem 0xf0000000-0xf1ffffff,0xf3030000-0xf3030fff at device 2.0 on pci0
xenpci0: <Xen Platform Device> port 0xc000-0xc0ff mem 0xf2000000-0xf2ffffff irq 28 at device 3.0 on pci0
xenstore0: <XenStore> on xenpci0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
psm0: <PS/2 Mouse> irq 12 on atkbdc0
psm0: [GIANT-LOCKED]
psm0: model IntelliMouse Explorer, device ID 4
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (9600,n,8,1)
ppc0: <Parallel port> port 0x378-0x37f irq 7 on acpi0
ppc0: Generic chipset (NIBBLE-only) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
lpt0: <Printer> on ppbus0
lpt0: Interrupt-driven port
ppi0: <Parallel I/O> on ppbus0
qpi0: <QPI system bus> on motherboard
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
fdc0: No FDOUT register!
Timecounters tick every 10.000 msec
xenbusb_front0: <Xen Frontend Devices> on xenstore0
xenbusb_add_device: Device device/suspend/event-channel ignored. State 6
cd0 at ata1 bus 0 scbus1 target 0 lun 0
cd0: <QEMU QEMU DVD-ROM 1.0.> Removable CD-ROM SCSI-0 device
cd0: 16.700MB/s transfers (WDMA2, ATAPI 12bytes, PIO 65534bytes)
cd0: Attempt to query device size failed: NOT READY, Medium not present
xn0: <Virtual Network Interface> at device/vif/0 on xenbusb_front0
xn0: Ethernet address: 00:16:3e:47:d4:52
xn1: <Virtual Network Interface> at device/vif/1 on xenbusb_front0
xn1: Ethernet address: 00:16:3e:47:d4:53
xenbusb_back0: <Xen Backend Devices> on xenstore0
xctrl0: <Xen Control Device> on xenstore0
xn0: backend features: feature-sg feature-gso-tcp4
xn1: backend features: feature-sg feature-gso-tcp4
xbd0: 20480MB <Virtual Block Device> at device/vbd/768 on xenbusb_front0
xbd0: attaching as ad0
xbd0: disk supports cache flush using: flush
SMP: AP CPU #25 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #30 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #21 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
Trying to mount root from ufs:/dev/ad0p2 [rw]...
Setting hostuuid: c9230f36-1a54-489e-877c-1d15b8f463e9.
Setting hostid: 0xd52252c7.
Entropy harvesting: interrupts ethernet point_to_point kickstart.
Starting file system checks:
/dev/ad0p2: FILE SYSTEM CLEAN; SKIPPING CHECKS
/dev/ad0p2: clean, 1690868 free (11852 frags, 209877 blocks, 0.2% fragmentation)
Mounting local file systems:.
Writing entropy file:.
xn0: link state changed to DOWN
xn0: link state changed to UP
Starting Network: lo0 xn0 xn1.
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
xn0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether 00:16:3e:47:d4:52
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet manual
status: active
xn1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether 00:16:3e:47:d4:53
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet manual
status: active
Starting devd.
Starting Network: xn1.
xn1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=503<RXCSUM,TXCSUM,TSO4,LRO>
ether 00:16:3e:47:d4:53
nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL>
media: Ethernet manual
status: active
Starting dhclient.
DHCPREQUEST on xn0 to 255.255.255.255 port 67
DHCPACK from 192.168.1.1
bound to 192.168.1.107 -- renewal in 43200 seconds.
add net ::ffff:0.0.0.0: gateway ::1
add net ::0.0.0.0: gateway ::1
add net fe80::: gateway ::1
add net ff02::: gateway ::1
ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib
32-bit compatibility ldconfig path: /usr/lib32
Creating and/or trimming log files.
Starting syslogd.
Starting watchdogd.
watchdogd: patting the dog: Operation not supported
/etc/rc: WARNING: failed to start watchdogd
No core dumps found.
lock order reversal:
1st 0xffffff80f7627d80 bufwait (bufwait) @ /usr/src/sys/kern/vfs_bio.c:3070
2nd 0xfffffe0027c18000 dirhash (dirhash) @ /usr/src/sys/ufs/ufs/ufs_dirhash.c:284
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xffffff8121720450
kdb_backtrace() at kdb_backtrace+0x39/frame 0xffffff8121720500
witness_checkorder() at witness_checkorder+0xc3f/frame 0xffffff8121720580
_sx_xlock() at _sx_xlock+0x75/frame 0xffffff81217205c0
ufsdirhash_add() at ufsdirhash_add+0x3b/frame 0xffffff8121720600
ufs_direnter() at ufs_direnter+0x688/frame 0xffffff81217206c0
ufs_mkdir() at ufs_mkdir+0x863/frame 0xffffff81217208c0
VOP_MKDIR_APV() at VOP_MKDIR_APV+0xf0/frame 0xffffff81217208f0
kern_mkdirat() at kern_mkdirat+0x21a/frame 0xffffff8121720ae0
amd64_syscall() at amd64_syscall+0x265/frame 0xffffff8121720bf0
Xfast_syscall() at Xfast_syscall+0xfb/frame 0xffffff8121720bf0
--- syscall (136, FreeBSD ELF64, sys_mkdir), rip = 0x80092faaa, rsp = 0x7fffffffd788, rbp = 0x7fffffffdc70 ---
Clearing /tmp (X related).
Updating motd:.
Configuring syscons: keymap blanktime.
Performing sanity check on sshd configuration.
Starting sshd.
Starting cron.
Starting background file system checks in 60 seconds.
Thu May 23 10:08:04 CEST 2013
FreeBSD/amd64 (Amnesiac) (ttyu0)
--------------000309090701090501040108
Content-Type: text/plain; charset="UTF-8"; x-mac-type=0; x-mac-creator=0;
name="timer_panic.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename="timer_panic.patch"
diff --git a/sys/dev/xen/timer/timer.c b/sys/dev/xen/timer/timer.c
index 7083e46..4cb49b0 100644
--- a/sys/dev/xen/timer/timer.c
+++ b/sys/dev/xen/timer/timer.c
@@ -418,8 +418,8 @@ xentimer_et_start(struct eventtimer *et,
} while (error == -ETIME);
if (error)
- panic("%s: Error %d setting singleshot timer to %"PRIu64"\n",
- device_get_nameunit(sc->dev), error, next_time);
+ panic("%s: Error %d setting singleshot timer to %"PRIu64" for vCPU#%d\n",
+ device_get_nameunit(sc->dev), error, next_time, cpu);
pcpu->timer = next_time;
return (error);
--------------000309090701090501040108--
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?519DDC0A.9000201>