Date: Tue, 17 Feb 2009 21:16:38 +0100 (CET) From: xen@komadev.de To: Kip Macy <kmacy@freebsd.org> Cc: freebsd-xen <freebsd-xen@freebsd.org> Subject: Re: domU page faults under heavy network load Message-ID: <14830020.49681234901798034.JavaMail.root@zim.freshx.de> In-Reply-To: <13712113.49661234900587705.JavaMail.root@zim.freshx.de>
next in thread | previous in thread | raw e-mail | index | archive | help
Hi,
in the kernel i disabled:
WITNESS
WITNESS_SKIPSPIN
in /etc/make.conf i enabled
DEBUG_FLAGS=-ggdb
To make sure that user/kernel are in sync i just rebuilt/installed both. I furthermore tracked down the problem,
since it seems to occur only between the dom0 and its domU, but not between the dom0 and a "real" remote host.
FreeBSD 8.0-CURRENT #0 r188718: Tue Feb 17 18:29:43 CET 2009
# dd if=/dev/zero bs=1024k| cpipe -vt | nc dom0 12345
thru: 5500.000ms at 23.3kB/s ( 23.3kB/s avg) 128.0kB
thru: 3200.000ms at 40.0kB/s ( 29.4kB/s avg) 256.0kB
...
thru: 0.000ms at infGB/s ( 398.8kB/s avg) 5.4MB
thru: 0.000ms at infGB/s ( 408.1kB/s avg) 5.5MB
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0xd1d9a500
fault code = supervisor read, page not present
instruction pointer = 0x21:0xc02e59b0
stack pointer = 0x29:0xc332aba8
frame pointer = 0x29:0xc332abfc
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 1, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (irq137: xn)
[thread pid 12 tid 100025 ]
Stopped at xlvbd_add+0x2f70: movl 0x900(%edx,%eax,4),%edx
Second System (AMD64, CentOS5.2, Xen 3.3.1):
....
thru: 0.000ms at infGB/s ( 598.1kB/s avg) 82.9MB
thru: 0.000ms at infGB/s ( 599.0kB/s avg) 83.0MB
Fatal trap 12: page fault while in kernel mode
cpuid = 0; apic id = 00
fault virtual address = 0xc433a140
fault code = supervisor read, page not present
instruction pointer = 0x21:0xc02db38e
stack pointer = 0x29:0xc332ab7c
frame pointer = 0x29:0xc332ab84
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 1, pres 1, def32 1, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 12 (irq137: xn)
[thread pid 12 tid 100025 ]
Stopped at gnttab_claim_grant_reference+0x2e: movl 0(%eax,%edx,4),%eax
Best Kai
PS : Between the dom0 and some random system via Gigabit Ethernet i get only 1/4 of the real machine speed
# dd if=/dev/zero bs=1024k | cpipe -vt -b 65535 | nc other 12345
thru: 7003.500ms at 9.1MB/s ( 9.1MB/s avg) 64.0MB
thru: 3301.650ms at 19.4MB/s ( 12.4MB/s avg) 128.0MB
thru: 1600.800ms at 40.0MB/s ( 16.1MB/s avg) 192.0MB
thru: 1800.900ms at 35.5MB/s ( 18.7MB/s avg) 256.0MB
thru: 2601.300ms at 24.6MB/s ( 19.6MB/s avg) 320.0MB
thru: 1400.700ms at 45.7MB/s ( 21.7MB/s avg) 384.0MB
thru: 1200.600ms at 53.3MB/s ( 23.7MB/s avg) 448.0MB
thru: 1600.800ms at 40.0MB/s ( 25.0MB/s avg) 512.0MB
thru: 1500.750ms at 42.6MB/s ( 26.2MB/s avg) 576.0MB
thru: 1900.682ms at 33.7MB/s ( 26.8MB/s avg) 640.0MB
thru: 2000.035ms at 32.0MB/s ( 27.2MB/s avg) 704.0MB
thru: 2300.000ms at 27.8MB/s ( 27.2MB/s avg) 768.0MB
thru: 1700.000ms at 37.6MB/s ( 27.8MB/s avg) 832.0MB
----- "Kip Macy" <kmacy@freebsd.org> schrieb:
> Is this a recent version of HEAD? I'm not able to reproduce.
>
> <...>
> thru: 1947.346ms at 32.9MB/s ( 27.8MB/s avg) 3.3GB
> thru: 1946.356ms at 32.9MB/s ( 27.9MB/s avg) 3.4GB
> thru: 1946.910ms at 32.9MB/s ( 27.9MB/s avg) 3.4GB
> <...>
>
> Maybe some variation in the options?
>
> Thanks,
> Kip
>
> On Tue, Feb 10, 2009 at 11:40 AM, Kai Mosebach <xen@komadev.de> wrote:
> > 1024meg
> >
> >
> > On 2/10/09 8:30 PM, "Kip Macy" <kmacy@freebsd.org> wrote:
> >
> >> How big is the memory allocation to the VM?
> >>
> >> Thanks,
> >> Kip
> >>
> >>
> >> On Tue, Feb 10, 2009 at 7:59 AM, Kai Mosebach <xen@komadev.de>
> wrote:
> >>> Hi,
> >>>
> >>> running r188436M on Xen3.3.1 (Centos5, x86_64). The reason is not
> to clear
> >>> but seems related to heavy load on pipes + memory + network?
> >>> It is repeatable with :
> >>>
> >>> bsd8# dd if=/dev/zero | nc otherhost 12345
> >>>
> >>> Fatal trap 12: page fault while in kernel mode
> >>> cpuid = 0; apic id = 00
> >>> fault virtual address = 0xd216dd00
> >>> fault code = supervisor read, page not present
> >>> instruction pointer = 0x21:0xc02e4d40
> >>> stack pointer = 0x29:0xc332aba8
> >>> frame pointer = 0x29:0xc332abfc
> >>> code segment = base 0x0, limit 0xfffff, type 0x1b
> >>> = DPL 1, pres 1, def32 1, gran 1
> >>> processor eflags = interrupt enabled, resume, IOPL = 0
> >>> current process = 12 (irq137: xn)
> >>> [thread pid 12 tid 100025 ]
> >>> Stopped at xlvbd_add+0x2f70: movl
> 0x900(%edx,%eax,4),%edx
> >>> db> bt
> >>> <console hangs>
> >>>
> >>>
> >>> On otherhost i get some stuff (between 19 and 62 MB), after nc
> closes down
> >>> eventually
> >>>
> >>> otherhost# nc -l 12345 | cpipe -vt -b 65535 >/dev/null
> >>> ...
> >>> thru: 108930.791ms at 184.3kB/s ( 184.3kB/s avg) 19.6MB
> >>>
> >>> best Kai
> >>> _______________________________________________
> >>> freebsd-xen@freebsd.org mailing list
> >>> http://lists.freebsd.org/mailman/listinfo/freebsd-xen
> >>> To unsubscribe, send any mail to
> "freebsd-xen-unsubscribe@freebsd.org"
> >>>
> >
> >
> >
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?14830020.49681234901798034.JavaMail.root>