From owner-freebsd-hardware@FreeBSD.ORG Mon Sep 3 04:04:03 2012 Return-Path: Delivered-To: freebsd-hardware@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1D8AF106566C for ; Mon, 3 Sep 2012 04:04:03 +0000 (UTC) (envelope-from ayoung@mosaicarchive.com) Received: from mail-ob0-f182.google.com (mail-ob0-f182.google.com [209.85.214.182]) by mx1.freebsd.org (Postfix) with ESMTP id C80648FC17 for ; Mon, 3 Sep 2012 04:04:02 +0000 (UTC) Received: by obbun3 with SMTP id un3so11128809obb.13 for ; Sun, 02 Sep 2012 21:04:02 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20120113; h=mime-version:x-originating-ip:in-reply-to:references:date :message-id:subject:from:to:cc:content-type:x-gm-message-state; bh=9xVfCdLR9qbsSyhrLAFAfWEugegSq/LKfbyzr/nqTZA=; b=cpMdkMO8reQgVBkK87bduPPuyRPZ2D/c4culu8z01Vxu+ps2y0HSR7y53q5mZcLx7s i76WmYfxyKv8oR3oK4gMBygAzDuR+MBjwD9zDLnXvnOQPUPiBTkcRZ85+kM5FcWHll6Z l1bejnNar3RkYXenMsk0OQtj6g8XWhLzMZ2eG7nnwR4Tc5dLY14bDLNiCMoWkBJG7kQH tYgaen1a246yFLgbusgWDnJCP13/Rhg1rqUAA+Zw58mOzBLb2Zk1bhU8qRdUFnjVrQ5k 2YuxpBdXNYxptsWaH49YSHClW0yJL7l8mtJKGYG3zaCuV1OD0jsROrvX6+HbcRyvLbUH suhg== MIME-Version: 1.0 Received: by 10.182.111.74 with SMTP id ig10mr13457782obb.14.1346645042146; Sun, 02 Sep 2012 21:04:02 -0700 (PDT) Received: by 10.76.174.38 with HTTP; Sun, 2 Sep 2012 21:04:02 -0700 (PDT) X-Originating-IP: [96.237.242.243] In-Reply-To: References: <50431E04.5050207@gatorhole.com> Date: Mon, 3 Sep 2012 00:04:02 -0400 Message-ID: From: Andy Young To: "Pepe (Jose) Amengual" X-Gm-Message-State: ALoCoQnryYsA1/InB7crfa2UMo2ZoQIWXUhmy6LXmpQJKrSem5n04ZSKhPyXoRdsRZUB6HVz/csP Content-Type: text/plain; charset=ISO-8859-1 X-Content-Filtered-By: Mailman/MimeDel 2.1.5 Cc: freebsd-hardware@freebsd.org Subject: Re: Load testing knocks out network X-BeenThere: freebsd-hardware@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: General discussion of FreeBSD hardware List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 Sep 2012 04:04:03 -0000 Hi Pepe, Thank you for the tip. I don't know how to interpret any of the output but I will dig into the documentation. Andy On Sun, Sep 2, 2012 at 11:13 PM, Pepe (Jose) Amengual < jose.amengual@gmail.com> wrote: > Maybe you should check vmstat -z while running the load testing to see if > you get any errors. > On Sep 2, 2012 1:58 AM, "Ragnar Lonn" wrote: > > > Hi Andy, > > > > I work for an online load testing service (loadimpact.com) and what we > > see is that the most common cause when a server crashes during a load > test, > > is that it runs out of some vital system resource. Usually system memory, > > but network connections (sockets/file descriptors) is also a likely > cause. > > > > You should have gotten some kind of error messages in the system log, but > > if the problem is easily repeatable I would set up monitoring of at least > > memory and file descriptors, and see if you are near the limits when the > > machine freezes. > > > > Regards, > > > > /Ragnar > > > > > > On 09/01/2012 10:14 PM, Andy Young wrote: > > > >> Last night one our servers went offline while I was load testing it. > When > >> I > >> got to the datacenter to check on it, the server seemed perfectly fine. > >> Everything was running on it, there were no panics or any other sign of > a > >> hard crash. The only problem is the network was unreachable. I couldn't > >> connect to the box even from a laptop directly attached to the ethernet > >> port. I couldn't connect to anything from the box either. It was if the > >> network controller had seized up. I restarted netif and it didn't make a > >> difference. Rebooting the machine however, solved the issue and > everything > >> went back to working great. I restarted the load testing and reproduced > >> the > >> problem twice more this morning so at least its repeatable. It feels > like > >> a > >> network controller / driver issue to me for a couple reasons. First, the > >> problem affects the entire system. We're running FreeBSD 9 with about a > >> half dozen jails. Most of the jails are running Apache but the one I was > >> load testing was running Jetty. However, if it was my application code > >> crashing I would expect the problem to at least be isolated to the jail > >> that hosts it. Instead, the entire machine and all jails in it lose > access > >> to the network. > >> > >> Apart from not being able to access the network, I don't see any other > >> signs of problems. This is the first major problem I've had to debug in > >> FreeBSD so I'm not a debugging expert by any means. There are no error > >> messages in /var/log/messages or dmesg apart from syslogd not being able > >> to > >> reach the network. If anyone has ideas on where I can look for more > >> evidence of what is going wrong, I would really appreciate it. > >> > >> We're running FreeBSD 9.0-RELEASE-p3. The network controller is a > Intel(R) > >> PRO/1000 Network Connection version - 2.2.5 configured with 6 ips using > >> aliases, five of which are used for jails. > >> > >> Thank you for the help!! > >> > >> Andy > >> ______________________________**_________________ > >> freebsd-hardware@freebsd.org mailing list > >> http://lists.freebsd.org/**mailman/listinfo/freebsd-**hardware< > http://lists.freebsd.org/mailman/listinfo/freebsd-hardware> > >> To unsubscribe, send any mail to "freebsd-hardware-unsubscribe@** > >> freebsd.org " > >> > > > > ______________________________**_________________ > > freebsd-hardware@freebsd.org mailing list > > http://lists.freebsd.org/**mailman/listinfo/freebsd-**hardware< > http://lists.freebsd.org/mailman/listinfo/freebsd-hardware> > > To unsubscribe, send any mail to "freebsd-hardware-unsubscribe@** > > freebsd.org " > > > _______________________________________________ > freebsd-hardware@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-hardware > To unsubscribe, send any mail to "freebsd-hardware-unsubscribe@freebsd.org > " > -- Andrew Young Mosaic Storage Systems, Inc http://www.mosaicarchive.com/ Follow us on: Twitter , Facebook , Google Plus , Pinterest