From owner-freebsd-hardware@FreeBSD.ORG Mon Sep 3 03:13:32 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AE299106566B for ; Mon, 3 Sep 2012 03:13:32 +0000 (UTC) (envelope-from jose.amengual@gmail.com) Received: from mail-wi0-f172.google.com (mail-wi0-f172.google.com [209.85.212.172]) by mx1.freebsd.org (Postfix) with ESMTP id 3C3B88FC0C for ; Mon, 3 Sep 2012 03:13:31 +0000 (UTC) Received: by wicr5 with SMTP id r5so2684274wic.13 for ; Sun, 02 Sep 2012 20:13:31 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=9oLJv2MnAq2XFRH3Ob4VRb6ATWHW4RHDnZjTnSDKaT4=; b=J7plbKT5Y8qRszvOfcRgJs9mevb5a+ajeLeRKF3udTC05WfS6yZKl6knP7sHHhli6f IUMYsxBfbCvP+qIKrx/Kc0EuqhJGYq9PQ/1GcrL3wm+H9V/4Ui55iseiao2Uqlpf9zKf g0MmHkHeBl+fWW9FJDl359s/YkAOlFZQelKckXe3MdLycno7tnfLjmxKji6s5XVnJyLW Cv2nyFGjwVoj8K3haWYWJ5GMz7a4eVRp4X50/zBpnrrVBYWsSgnsAP9crU18r1qUgskr OYgoTYbirU7i9OcaFvQaKdw1GX2Pi30XwQn7z4w6qa1nprUAT6pExFlZ8ZKnOT9UiFND eKeQ== MIME-Version: 1.0 Received: by 10.180.103.136 with SMTP id fw8mr20086287wib.20.1346642010949; Sun, 02 Sep 2012 20:13:30 -0700 (PDT) Received: by 10.227.92.24 with HTTP; Sun, 2 Sep 2012 20:13:30 -0700 (PDT) Received: by 10.227.92.24 with HTTP; Sun, 2 Sep 2012 20:13:30 -0700 (PDT) In-Reply-To: <50431E04.5050207@gatorhole.com> References: <50431E04.5050207@gatorhole.com> Date: Sun, 2 Sep 2012 20:13:30 -0700 Message-ID: From: "Pepe (Jose) Amengual" To: Ragnar Lonn Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hardware@freebsd.org Subject: Re: Load testing knocks out network X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Sep 2012 03:13:32 -0000 Maybe you should check vmstat -z while running the load testing to see if you get any errors. On Sep 2, 2012 1:58 AM, "Ragnar Lonn" wrote: > Hi Andy, > > I work for an online load testing service (loadimpact.com) and what we > see is that the most common cause when a server crashes during a load test, > is that it runs out of some vital system resource. Usually system memory, > but network connections (sockets/file descriptors) is also a likely cause. > > You should have gotten some kind of error messages in the system log, but > if the problem is easily repeatable I would set up monitoring of at least > memory and file descriptors, and see if you are near the limits when the > machine freezes. > > Regards, > > /Ragnar > > > On 09/01/2012 10:14 PM, Andy Young wrote: > >> Last night one our servers went offline while I was load testing it. When >> I >> got to the datacenter to check on it, the server seemed perfectly fine. >> Everything was running on it, there were no panics or any other sign of a >> hard crash. The only problem is the network was unreachable. I couldn't >> connect to the box even from a laptop directly attached to the ethernet >> port. I couldn't connect to anything from the box either. It was if the >> network controller had seized up. I restarted netif and it didn't make a >> difference. Rebooting the machine however, solved the issue and everything >> went back to working great. I restarted the load testing and reproduced >> the >> problem twice more this morning so at least its repeatable. It feels like >> a >> network controller / driver issue to me for a couple reasons. First, the >> problem affects the entire system. We're running FreeBSD 9 with about a >> half dozen jails. Most of the jails are running Apache but the one I was >> load testing was running Jetty. However, if it was my application code >> crashing I would expect the problem to at least be isolated to the jail >> that hosts it. Instead, the entire machine and all jails in it lose access >> to the network. >> >> Apart from not being able to access the network, I don't see any other >> signs of problems. This is the first major problem I've had to debug in >> FreeBSD so I'm not a debugging expert by any means. There are no error >> messages in /var/log/messages or dmesg apart from syslogd not being able >> to >> reach the network. If anyone has ideas on where I can look for more >> evidence of what is going wrong, I would really appreciate it. >> >> We're running FreeBSD 9.0-RELEASE-p3. The network controller is a Intel(R) >> PRO/1000 Network Connection version - 2.2.5 configured with 6 ips using >> aliases, five of which are used for jails. >> >> Thank you for the help!! >> >> Andy >> ______________________________**_________________ >> freebsd-hardware@freebsd.org mailing list >> http://lists.freebsd.org/**mailman/listinfo/freebsd-**hardware >> To unsubscribe, send any mail to "freebsd-hardware-unsubscribe@** >> freebsd.org " >> > > ______________________________**_________________ > freebsd-hardware@freebsd.org mailing list > http://lists.freebsd.org/**mailman/listinfo/freebsd-**hardware > To unsubscribe, send any mail to "freebsd-hardware-unsubscribe@** > freebsd.org " >