From owner-freebsd-xen@FreeBSD.ORG Sun Jan 6 00:48:42 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A9990627 for ; Sun, 6 Jan 2013 00:48:42 +0000 (UTC) (envelope-from moto@kawasaki3.org) Received: from kawasaki3.org (blackpearl.kawasaki3.org [173.230.157.78]) by mx1.freebsd.org (Postfix) with ESMTP id 83285C19 for ; Sun, 6 Jan 2013 00:48:42 +0000 (UTC) Received: from localhost (s253.HtokyoFL10.vectant.ne.jp [222.228.92.253]) (Authenticated sender: moto) by kawasaki3.org (Postfix) with ESMTPSA id CF8A61CD42 for ; Sat, 5 Jan 2013 19:39:57 -0500 (EST) Date: Sun, 06 Jan 2013 09:39:41 +0900 (JST) Message-Id: <20130106.093941.323021822222343593.moto@kawasaki3.org> To: freebsd-xen@freebsd.org Subject: XENHVM hang-up w/ "network_alloc_rx_buffers: m_cljget failed" From: moto kawasaki X-Mailer: Mew version 6.5 on Emacs 24.2.50 / Mule 6.0 (HANACHIRUSATO) X-Face: )._4~w!_D$r6qNS0+; nS|]WNeI4f3o)QnH[ItB[esXuc$~hQ$.,?}$SnLe/[24Hao%^q/Is 'SJtZe#21h;7z;q+iyj[^%7\46.Gg-t7.px<}L-f_:P+6i4-a{DIL[ Mime-Version: 1.0 Content-Type: Text/Plain; charset=us-ascii Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jan 2013 00:48:42 -0000 Hello, Thank you very much for your effort to porting Xen to FreeBSD. I am so happy I can use FreeBSD instead of Linux on XenServer!! Today, I have encountered hang-ups as follows; - XenServer 6.1 * 3 hosts (HP DL360G7) - FreeBSD 9.0-RELEASE-p5 (customized XENHVM) * 2 VMs - Those VMs hang up with the following messages on console: network_alloc_rx_buffers: m_cjlget failed This message is repeatedly shown so quickly, and I cannot get login prompt. - During this issue, I cannot login via ssh over network, although TCP connection is established (I tried with telnet to ssh port). - I guess the trigger of this issue was something like broadcast storm, which was made by failed network configuration change. - This issue seems similar to exhausting mbuf(rx). Could you please advice me of countermeasure/prevention against this issue ? Also, I would like to know how to make it self-healing, I mean I want to make those VMs to recover automatically after the trigger had gone. Any suggestion/hint/reference is welcomed. Thank you very much. -- moto kawasaki From owner-freebsd-xen@FreeBSD.ORG Sun Jan 6 10:35:58 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 47FF34D8 for ; Sun, 6 Jan 2013 10:35:58 +0000 (UTC) (envelope-from kalessin@kalessin.fr) Received: from glofish.kalessin.fr (glofish.kalessin.fr [88.190.38.78]) by mx1.freebsd.org (Postfix) with ESMTP id 0CE8B9FE for ; Sun, 6 Jan 2013 10:35:57 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=kalessin.fr; s=glofish; h=To:References:Message-Id:Content-Transfer-Encoding:Cc:Date:In-Reply-To:From:Subject:Mime-Version:Content-Type; bh=rbMLJCOW2tZpR+tdEyc4q5vb7A8OStBi1ENYDiKtb0U=; b=BO3DC4xyk4TdLHYvof6XPa1e85HDI7NKrhSq4X9ryhOxWv/+VEklQpVAHzipvGVngEX9h5zwFe6K339yRJ9HHWlu0zFING+YpcUcEmNkAJkivUmIDpWDY3TZJkBlkx75KAyXH9bSDxp8OqHZnNgFLlXGtWqC8RTrZ7U2M1v/cC0=; Received: from did75-3-82-66-148-158.fbx.proxad.net ([82.66.148.158] helo=[192.168.1.102]) by glofish.kalessin.fr with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16) (Exim 4.72) (envelope-from ) id 1Trn1h-00027l-Pr; Sun, 06 Jan 2013 11:00:21 +0100 Content-Type: text/plain; charset=windows-1252 Mime-Version: 1.0 (Mac OS X Mail 6.2 \(1499\)) Subject: Re: XENHVM hang-up w/ "network_alloc_rx_buffers: m_cljget failed" From: Louis Opter In-Reply-To: <20130106.093941.323021822222343593.moto@kawasaki3.org> Date: Sun, 6 Jan 2013 11:00:16 +0100 Content-Transfer-Encoding: quoted-printable Message-Id: <3897E6E6-91A7-406D-9FA1-62E9104F3319@kalessin.fr> References: <20130106.093941.323021822222343593.moto@kawasaki3.org> To: moto kawasaki X-Mailer: Apple Mail (2.1499) Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jan 2013 10:35:58 -0000 On Jan 6, 2013, at 1:39 AM, moto kawasaki wrote: >=20 > - Those VMs hang up with the following messages on console: >=20 > network_alloc_rx_buffers: m_cjlget failed >=20 > This message is repeatedly shown so quickly, and I cannot get > login prompt. I got this too (by starting too much torrents at once=85) My understanding is that FreeBSD is not really dynamic concerning memory allocations for the network, so I bumped up the limits in /boot/loader.conf: kern.ipc.nmbclusters=3D"51200" kern.ipc.nmbjumbop=3D"51200" netstat -m is really useful to see what's going on. My dom0 is Linux 3.2, but I don't think that this problem is specific to Xen in any way. Best --=20 Louis Opter From owner-freebsd-xen@FreeBSD.ORG Sun Jan 6 13:19:58 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id A36A0313 for ; Sun, 6 Jan 2013 13:19:58 +0000 (UTC) (envelope-from moto@kawasaki3.org) Received: from kawasaki3.org (blackpearl.kawasaki3.org [173.230.157.78]) by mx1.freebsd.org (Postfix) with ESMTP id 85A84FA3 for ; Sun, 6 Jan 2013 13:19:58 +0000 (UTC) Received: from localhost (s253.HtokyoFL10.vectant.ne.jp [222.228.92.253]) (Authenticated sender: moto) by kawasaki3.org (Postfix) with ESMTPSA id 30D941CD42 for ; Sun, 6 Jan 2013 08:19:57 -0500 (EST) Date: Sun, 06 Jan 2013 22:19:35 +0900 (JST) Message-Id: <20130106.221935.1391230821526186583.moto@kawasaki3.org> To: freebsd-xen@freebsd.org Subject: Re: XENHVM hang-up w/ "network_alloc_rx_buffers: m_cljget failed" From: moto kawasaki In-Reply-To: <3897E6E6-91A7-406D-9FA1-62E9104F3319@kalessin.fr> References: <20130106.093941.323021822222343593.moto@kawasaki3.org> <3897E6E6-91A7-406D-9FA1-62E9104F3319@kalessin.fr> X-Mailer: Mew version 6.5 on Emacs 24.2.50 / Mule 6.0 (HANACHIRUSATO) X-Face: )._4~w!_D$r6qNS0+; nS|]WNeI4f3o)QnH[ItB[esXuc$~hQ$.,?}$SnLe/[24Hao%^q/Is 'SJtZe#21h;7z;q+iyj[^%7\46.Gg-t7.px<}L-f_:P+6i4-a{DIL[ Mime-Version: 1.0 Content-Type: Text/Plain; charset=iso-2022-jp Content-Transfer-Encoding: 7bit X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 06 Jan 2013 13:19:58 -0000 Hi Louis, Thank you very much for your response. kalessin> On Jan 6, 2013, at 1:39 AM, moto kawasaki wrote: kalessin> > kalessin> > - Those VMs hang up with the following messages on console: kalessin> > kalessin> > network_alloc_rx_buffers: m_cjlget failed kalessin> > kalessin> > This message is repeatedly shown so quickly, and I cannot get kalessin> > login prompt. kalessin> kalessin> I got this too (by starting too much torrents at once…) kalessin> kalessin> My understanding is that FreeBSD is not really dynamic concerning kalessin> memory allocations for the network, so I bumped up the limits in kalessin> /boot/loader.conf: kalessin> kalessin> kern.ipc.nmbclusters="51200" kalessin> kern.ipc.nmbjumbop="51200" In my case, kern.ipc.nmbclusters: 262144 <-- set in /etc/sysctl.conf kern.ipc.nmbjumbop: 12800 <-- OS default so, 'nmbjumbop' seems the key point. I will try to increase it. kalessin> netstat -m is really useful to see what's going on. Yes it really is. But it is not so well-documented, isn't it ? I'd really appreciate some documents/pointers describing it :-) kalessin> My dom0 is Linux 3.2, but I don't think that this problem is specific to kalessin> Xen in any way. I agree. Thank you very much -- moto kawasaki From owner-freebsd-xen@FreeBSD.ORG Mon Jan 7 11:06:58 2013 Return-Path: Delivered-To: freebsd-xen@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 1CECD110 for ; Mon, 7 Jan 2013 11:06:58 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:1900:2254:206c::16:87]) by mx1.freebsd.org (Postfix) with ESMTP id E5742FAB for ; Mon, 7 Jan 2013 11:06:57 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.6/8.14.6) with ESMTP id r07B6vbL088111 for ; Mon, 7 Jan 2013 11:06:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.6/8.14.6/Submit) id r07B6v2T088109 for freebsd-xen@FreeBSD.org; Mon, 7 Jan 2013 11:06:57 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 7 Jan 2013 11:06:57 GMT Message-Id: <201301071106.r07B6v2T088109@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-xen@FreeBSD.org Subject: Current problem reports assigned to freebsd-xen@FreeBSD.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 07 Jan 2013 11:06:58 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o kern/171873 xen [xen] xn network device floods warning in dmesg o kern/171138 xen [xen] [panic] Deactivating network interface produces o kern/171118 xen [xen] FreeBSD XENHVM guest doesn't shutdown cleanly o kern/166174 xen [xen] Problems ROOT MOUNT ERROR o kern/165418 xen [xen] Problems mounting root filesystem from XENHVM o kern/164630 xen [xen] XEN HVM kernel: run_interrupt_driven_hooks: stil o kern/164450 xen [xen] Failed to install FreeeBSD 9.0-RELEASE from CD i o kern/162677 xen [xen] FreeBSD not compatible with "Current Stable Xen" o kern/161318 xen [xen] sysinstall crashes with floating point exception o kern/155468 xen [xen] Xen PV i386 multi-kernel CPU system is not worki o kern/155353 xen [xen] [patch] put "nudging TOD" message under boot_ver o kern/154833 xen [xen]: xen 4.0 - DomU freebsd8.2RC3 i386, XEN kernel. o kern/154473 xen [xen] xen 4.0 - DomU freebsd8.1 i386, XEN kernel. Not o kern/154472 xen [xen] xen 4.0 - DomU freebsd8.1 i386 xen kernel reboot o kern/154428 xen [xen] xn0 network interface and PF - Massive performan o kern/153674 xen [xen] i386/XEN idle thread shows wrong percentages o kern/153672 xen [xen] [panic] i386/XEN panics under heavy fork load o kern/153620 xen [xen] Xen guest system clock drifts in AWS EC2 (FreeBS o kern/153477 xen [xen] XEN pmap code abuses vm page queue lock o kern/153150 xen [xen] xen/ec2: disable checksum offloading on interfac o kern/152228 xen [xen] [panic] Xen/PV panic with machdep.idle_mwait=1 o kern/144629 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143398 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor o kern/143340 xen [xen] FreeBSD 8-RELEASE XEN pvm networking doesn't wor f kern/143069 xen [xen] [panic] Xen Kernel Panic - Memory modified after f kern/135667 xen ufs filesystem corruption on XEN DomU system f kern/135421 xen [xen] FreeBSD Xen PVM DomU network failure - netfronc. f kern/135178 xen [xen] Xen domU outgoing data transfer stall when TSO i p kern/135069 xen [xen] FreeBSD-current/Xen SMP doesn't function at all f i386/124516 xen [xen] FreeBSD-CURRENT Xen Kernel Segfaults when config o kern/118734 xen [xen] FreeBSD 6.3-RC1 and FreeBSD 7.0-BETA 4 fail to b 31 problems total. From owner-freebsd-xen@FreeBSD.ORG Thu Jan 10 12:56:20 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 78DA1988 for ; Thu, 10 Jan 2013 12:56:20 +0000 (UTC) (envelope-from feld@feld.me) Received: from feld.me (feld.me [66.170.3.2]) by mx1.freebsd.org (Postfix) with ESMTP id 56A4CCC5 for ; Thu, 10 Jan 2013 12:56:20 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=feld.me; s=blargle; h=Content-Transfer-Encoding:Content-Type:Mime-Version:References:In-Reply-To:Message-Id:Subject:Cc:To:From:Date; bh=F+dKGL6RnLbPGfz/bgxFcdvKvVGeIaVsQPqEWWPl96g=; b=Hn2fkFXGr1PlLhhusgBMFFg8oKjdI79AKsTFTBoWrrciMqY75YE6mXjYcts9ofKn6+0Ko4INRP7XD1GXLJWCPuEMjQ7X9r6XCUxni+TgUEuoVhVgP56D+5DSA3giHf//; Received: from localhost ([127.0.0.1] helo=mwi1.coffeenet.org) by feld.me with esmtp (Exim 4.80.1 (FreeBSD)) (envelope-from ) id 1TtHg9-000E0B-3Q; Thu, 10 Jan 2013 06:56:17 -0600 Received: from feld@feld.me by mwi1.coffeenet.org (Archiveopteryx 3.1.4) with esmtpsa id 1357822571-49025-49023/5/1; Thu, 10 Jan 2013 12:56:11 +0000 Date: Thu, 10 Jan 2013 06:56:11 -0600 From: Mark Felder To: moto kawasaki Subject: Re: 9.1R? Message-Id: <20130110065611.03e66265@tech304.office.supranet.net> In-Reply-To: <20121223.183654.1876624675013089283.moto@kawasaki3.org> References: <000301cddfa8$b6af8b70$240ea250$@ezwind.net> <20121222012933.5694d482@tech304> <20121223.183654.1876624675013089283.moto@kawasaki3.org> X-Mailer: Claws Mail 3.9.0 (GTK+ 2.24.6; amd64-portbld-freebsd9.0) Mime-Version: 1.0 Content-Type: text/plain Content-Transfer-Encoding: quoted-printable X-SA-Report: ALL_TRUSTED=-1, KHOP_THREADED=-0.5 X-SA-Score: -1.5 Cc: freebsd-xen@freebsd.org X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2013 12:56:20 -0000 On Sun, 23 Dec 2012 18:36:54 +0900 moto kawasaki wrote: > It might be better to disable rxcsum, txcsum, and lro, too. > My understanding is the point is the 'NIC hardware' is not the real > one but emulation by Xen Host, so that those 'off loading' makes > context switch storm between domU and dom0. > # please kindly correct me if wrong. You're actually quite correct as this is a source of network problems on = many different domU OSes. From owner-freebsd-xen@FreeBSD.ORG Thu Jan 10 21:09:57 2013 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id D4B356B4 for ; Thu, 10 Jan 2013 21:09:57 +0000 (UTC) (envelope-from dcherednik@roshianokatachi.com) Received: from smtp.nanocore.sportcomitet.org (unknown [IPv6:2a01:4f8:d13:2941::1:3]) by mx1.freebsd.org (Postfix) with ESMTP id 8F7B5936 for ; Thu, 10 Jan 2013 21:09:57 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by smtp.nanocore.sportcomitet.org (Postfix) with SMTP id 1D7DFC048E for ; Fri, 11 Jan 2013 01:09:56 +0400 (MSK) Received: from [192.168.11.85] (ppp91-76-109-188.pppoe.mtu-net.ru [91.76.109.188]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) (Authenticated sender: dcherednik@roshianokatachi.com) by smtp.nanocore.sportcomitet.org (Postfix) with ESMTPSA id 84266C028E for ; Fri, 11 Jan 2013 01:09:55 +0400 (MSK) Message-ID: <50EF2E21.1010106@roshianokatachi.com> Date: Fri, 11 Jan 2013 01:09:53 +0400 From: Daniil Cherednik User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:17.0) Gecko/17.0 Thunderbird/17.0 MIME-Version: 1.0 To: freebsd-xen@freebsd.org Subject: i386 XEN PV kernel and SMP Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-DSPAM-Result: Innocent X-DSPAM-Processed: Fri Jan 11 01:09:55 2013 X-DSPAM-Confidence: 1.0000 X-DSPAM-Improbability: 1 in 98689409 chance of being spam X-DSPAM-Probability: 0.0023 X-DSPAM-Signature: 49,50ef2e2313021517633544 X-DSPAM-Factors: 27, EINVAL+#+#+understand, 0.40000, I+#+#+#+fixing, 0.40000, information+#+#+#+Best, 0.40000, User-Agent*Thunderbird+17.0, 0.40000, Received*Postfix+with, 0.40000, fix+#+#+#+has, 0.40000, Received*with+#+id, 0.40000, can+#+#+Did, 0.40000, Received*Authenticated+sender, 0.40000, Date*09+53, 0.40000, Received*using+TLSv1, 0.40000, VCPUOP+#+#+ctxt, 0.40000, could+#+#+we, 0.40000, Received*2013+01, 0.40000, User-Agent*X11+#+#+rv, 0.40000, Date*Fri+11, 0.40000, Content-Type*text+plain, 0.40000, Received*freebsd-xen+#+#+#+Jan, 0.40000, values+#+vcpu, 0.40000, op+#+intialize, 0.40000, User-Agent*rv+#+#+17.0, 0.40000, User-Agent*Mozilla+#+X11, 0.40000, about+this, 0.40000, I+#+#+sure, 0.40000, Received*certificate+requested, 0.40000, Received*from+#+ppp91-76-109-188.pppoe.mtu-net.ru, 0.40000, Received*client+certificate, 0.40000 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 10 Jan 2013 21:09:57 -0000 Hi! I am thinking about fixing bug 155468 and see that HYPERVISOR_vcpu_op(VCPUOP_intialize, cpu, &ctxt) returns -EINVAL. If I understand XEN code correctly, it could happen if we had wrong vcpu_info. But I am not sure, wrong values in vcpu_guest_context_t also can be involved. Did anyone try to fix it or just has any additional information about this issue? -- Best Regards Daniil Cherednik