From owner-freebsd-stable@FreeBSD.ORG Mon Jan 2 18:17:12 2012 Return-Path: Delivered-To: freebsd-stable@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 59789106566C for ; Mon, 2 Jan 2012 18:17:12 +0000 (UTC) (envelope-from artemb@gmail.com) Received: from mail-gx0-f182.google.com (mail-gx0-f182.google.com [209.85.161.182]) by mx1.freebsd.org (Postfix) with ESMTP id 164E88FC12 for ; Mon, 2 Jan 2012 18:17:11 +0000 (UTC) Received: by ggnp1 with SMTP id p1so13178104ggn.13 for ; Mon, 02 Jan 2012 10:17:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=mime-version:sender:in-reply-to:references:date :x-google-sender-auth:message-id:subject:from:to:cc:content-type :content-transfer-encoding; bh=L55H8GOFgpbZqmlSA9x+oC73lRwk11k0Tbv52mfK32A=; b=BAWF7ek94BJI5uUf8xpMVsJleC0MUtYeV7Nk3s2lE4AMWrhY/nSka+pjPy53GyV31T uoIJTY30KP49jWyvYzi/TDaoHd2klyZLTLG4NbdWB6LnRpUaOnpbtErCbD65NJHSTDxJ PsIwJTrOHHW1w38rU4Dr6Ler5GbxKh6rVJeWw= MIME-Version: 1.0 Received: by 10.100.86.8 with SMTP id j8mr19745885anb.66.1325528231524; Mon, 02 Jan 2012 10:17:11 -0800 (PST) Sender: artemb@gmail.com Received: by 10.147.52.27 with HTTP; Mon, 2 Jan 2012 10:17:11 -0800 (PST) In-Reply-To: <20120102134150.GC17398@equilibrium.bsdes.net> References: <20120102122549.GB17398@equilibrium.bsdes.net> <20120102125929.GA89966@icarus.home.lan> <20120102134150.GC17398@equilibrium.bsdes.net> Date: Mon, 2 Jan 2012 10:17:11 -0800 X-Google-Sender-Auth: 0Pa--D_6xpe-THef57Xu2csu1Bk Message-ID: From: Artem Belevich To: Victor Balada Diaz Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable Cc: freebsd-stable@freebsd.org, Jeremy Chadwick Subject: Re: Performance problems with pagedaemon X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 02 Jan 2012 18:17:12 -0000 On Mon, Jan 2, 2012 at 5:41 AM, Victor Balada Diaz wrote= : ... > System wide totals computed every five seconds: (values in kilobytes) > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > Processes: =A0 =A0 =A0 =A0 =A0 =A0 =A0(RUNQ: 2 Disk Wait: 0 Page Wait: 0 = Sleep: 51) > Virtual Memory: =A0 =A0 =A0 =A0 (Total: 1098017100K, Active 24065448K) > Real Memory: =A0 =A0 =A0 =A0 =A0 =A0(Total: 21157668K Active 20971144K) > Shared Virtual Memory: =A0(Total: 27740K Active: 8632K) > Shared Real Memory: =A0 =A0 (Total: 6128K Active: 4928K) > Free Memory Pages: =A0 =A0 =A0315636K On a system with 24GB of RAM you seem to have almost all of it active. It appears that you're simply on the edge of running out of memory and thus page daemon wakes up constantly trying to find more pages... ... > Top: > > last pid: 24777; =A0load averages: =A03.26, =A04.07, =A04.49 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0= =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 = =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 =A0 up 34+19:43:58 =A01= 4:32:05 > 66 processes: =A05 running, 61 sleeping > CPU: =A00.2% user, =A00.0% nice, 37.6% system, =A00.0% interrupt, 62.2% i= dle > Mem: 18G Active, 1908M Inact, 3008M Wired, 73M Cache, 2465M Buf, 232M Fre= e > Swap: 4096M Total, 4096M Free > > =A0PID USERNAME =A0THR PRI NICE =A0 SIZE =A0 =A0RES STATE =A0 C =A0 TIME = =A0 WCPU COMMAND > =A01059 mysql =A0 =A0 =A022 =A044 =A0 =A00 21004M 19741M ucond =A0 3 =A07= 1.2H 124.51% mysqld Mysql uses more than 20G of RAM. You may want to tune it down a bit so that there is a bit of free RAM around. Page daemon is trying to maintain v_free_target + v_cache_min. >vm.v_free_target: 161771 >vm.v_cache_min: 161771 In your case that would be about 1.2GB. If 'v_free_count + v_cache_count' are below that page daemon will periodically wake up and will start scanning active/inactive lists trying to find the pages it could use. In your case, when most of the memory is in active use, page daemon's job will be almost pointless and would just waste CPU time. On large memory systems default tuning for v_free/cache_min/target is probably somewhat conservative. You may try setting them somewhat lower via sysctl and see if you can find an equilibrium with mysql happy, pagedaemon sleeping and the system up and running. The danger of tuning these parameters too low is that is you don't have enough memory available for allocation without having to sleep, things will start falling apart and will eventually hang or crash your box. --Artem