From owner-freebsd-xen@FreeBSD.ORG Wed Nov 21 17:27:11 2012 Return-Path: Delivered-To: freebsd-xen@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 859E72C0 for ; Wed, 21 Nov 2012 17:27:11 +0000 (UTC) (envelope-from james.mcjones@gmail.com) Received: from mail-lb0-f196.google.com (mail-lb0-f196.google.com [209.85.217.196]) by mx1.freebsd.org (Postfix) with ESMTP id 02CA68FC0C for ; Wed, 21 Nov 2012 17:27:10 +0000 (UTC) Received: by mail-lb0-f196.google.com with SMTP id m4so715196lbo.7 for ; Wed, 21 Nov 2012 09:27:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=q1mIhEmK4bbDl7JQ1AbUE5ustLdcYISSlrilmifxq2k=; b=jXE/+7xuS6ek73L5Slse7HZOoA3mtLg/OH10eePXMdEKevn1SjLrBSX3xZ5v4oZ8p/ UdNaSc9xHnm9cV40UGsj5KCPO1G+j8x/6qpRYDpZiLxRpU3ZEEcroJepOKXGZGdNT+rm +smUoQ0hZnNNv/rRjEmulBGiJUMHOz1aDXnCCljMfoWXyRKWCpsPzcLIzWJas6E9sErm j0V/6MDzXX0Bonodf3y9zf8YF0g4Jgb3SM3syYupk8lMmivgQO3DUlmCDDiZaAekoQHS 5r2RkNCT/3eMPoBDSPprOfbw05Z8fXnmdEoWY1GYx1vlUo56JIn4XBgD+C8xfFHp1zaY 7CRA== MIME-Version: 1.0 Received: by 10.112.102.130 with SMTP id fo2mr8236355lbb.35.1353518829528; Wed, 21 Nov 2012 09:27:09 -0800 (PST) Received: by 10.112.46.137 with HTTP; Wed, 21 Nov 2012 09:27:09 -0800 (PST) Date: Wed, 21 Nov 2012 18:27:09 +0100 Message-ID: Subject: 8.3-RELEASE-p4 with pfsense 2.1-BETA0 patches running as PV xen domU randomly crashes From: James McJones To: freebsd-xen@freebsd.org Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 X-BeenThere: freebsd-xen@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: Discussion of the freebsd port to xen - implementation and usage List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Nov 2012 17:27:11 -0000 Hi We have a significant number of pfsense 2.0 firewalls (with freebsd 8.1-RELEASE-p6 under the hood) all running in HVM mode on xen 4.1, with which we've been experiencing performance issues recently. In essence, the (virtual) firewall load goes through the roof as we reach a certain throughput level. The xen host server's CPU only runs at 20-30% when this happens. A first improvement was noticed by switching from the default realtek 8139 drivers (re$x) to the e1000 drivers (em$x). As the results were still not acceptable, we've attempted to build a xen PV-aware freebsd 8.3-RELEASE-p4 kernel with pfsense 2.1-BETA0 patches, in which we've eventually succeeded. The performance gain was massive. Firewalls that were previously becoming unresponsive at a throughput of 20mbit, would now do 200mbit with a load of max 4. The issue we're facing with the 2.1-BETA running in PV mode however, is that it randomly crashes and reboots. This does not appear to be load related at all. Happens roughly once a day. The output on the console when it crashes can be found here: https://gist.github.com/d943d0c97fd4d9fba1e9 The headline of this crash is: panic: HYPERVISOR_update_va_mapping(((unsigned long)(sysmaps->CADDR2)), (0x001 | 0x002 | xpmap_ptom(((dst)->phys_addr)) | 0x020 | 0x040), UVMF_INVLPG| UVMF_ALL) < 0: /usr/pfSensesrc/src/sys/i386/xen/pmap.c:342 Has anyone else been successful in running pfsense in PV mode? Any experiences to share? Or any pointers towards what the issue could be with our PV kernel? Thanks in advance James