From owner-freebsd-stable@freebsd.org Sun Jun 17 23:04:37 2018 Return-Path: Delivered-To: freebsd-stable@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 4D80F100E214 for ; Sun, 17 Jun 2018 23:04:37 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: from mail-it0-x231.google.com (mail-it0-x231.google.com [IPv6:2607:f8b0:4001:c0b::231]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id D3B58840F6 for ; Sun, 17 Jun 2018 23:04:36 +0000 (UTC) (envelope-from amvandemore@gmail.com) Received: by mail-it0-x231.google.com with SMTP id j135-v6so9582760itj.1 for ; Sun, 17 Jun 2018 16:04:36 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=DGl5qb2Klbqts1gLgGrahYlFpV3vpYKevXc1oK+Q8o8=; b=reFKOvAB4wAGhSnN1SmCCz7DHjU48Rz3S4V9+NH6ENHw1prJz9zxi9R95jHh77CYFB TQpFvCZxEfqswDjNnRFfKbZGX/l4MRqLXYlrpUYPzwsAbLuxYo2aDcXY1bMa2QkUZnTa QiUamlV5KYfgjZEN8Vsbl5w66SZAFxq6dlnY6YwYoApoXvCUeTxX58T+2Ybobimycb3y PYT2I1Vc1u1D/zevVGUdW+BvgZeu4cnsRszIfTN7NnGt2L852y9bDBnf0u9MseWm7GCh G9eO8wqSVaLGZJOLQqw79L77l210zbxaVbqVBb0ndMrov+9OK0CIBk26f1pMg42Pwx0/ 02aA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=DGl5qb2Klbqts1gLgGrahYlFpV3vpYKevXc1oK+Q8o8=; b=DcyCskpXZarGLwAfMBWFRlctmPlGdhTf6TJkOAx31PhievL6y1aZaKTKO6+tNfovlo t8zX9szRxC71UI+QnRaiDc7rCAh64RcTFAQLTRSeqKEBBR5ViCDmioADRy+uU63LbtM4 HDO6O01BUb/hCahlPmO+EoZ8SZcdXcf2oxE1W6oDehh423KU1wL16Ez3Z2+aQcbzNuJl Oxzy8qQ5wLukuBYkyNJt1mJQ2tgrlcmtHCLS2McXorXV0tgdpS13JI3Fn6OdCdEHMrMX DJ9uHBr8GB+6tMIFcTHEI4Jvq4o6/gfVB45c93Jb4gljbV2HKAyBxlRYa6aS0bMReWIX ZXZw== X-Gm-Message-State: APt69E1GPviGMdhe/BZWYQTA8rQc0ht+kEVZpWXOfdfxwCuGD8Poksre mJ8sX6lTRKClj3g+886WVxWCCScmLUplBc4x4YJEJQ== X-Google-Smtp-Source: ADUXVKJ+K06cKwdUsAKynrSlO4FfVZwl9/KfJboTofrdHNQI1ySAhGdcOOEfXrY4SsU+QQ9ZUw8HEFZV3OyAO6KiMYs= X-Received: by 2002:a24:ec44:: with SMTP id g65-v6mr8343067ith.18.1529276676168; Sun, 17 Jun 2018 16:04:36 -0700 (PDT) MIME-Version: 1.0 Received: by 2002:a02:8d5c:0:0:0:0:0 with HTTP; Sun, 17 Jun 2018 16:04:35 -0700 (PDT) In-Reply-To: References: From: Adam Date: Sun, 17 Jun 2018 18:04:35 -0500 Message-ID: Subject: Re: lightly loaded system eats swap space To: tech-lists Cc: "freebsd-stable@freebsd.org" Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 17 Jun 2018 23:04:37 -0000 On Sun, Jun 17, 2018 at 5:19 PM, tech-lists wrote: > Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, E5-2630 > @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, 4123M > Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and how > would I debug it? > That seems to be shown in the output you provided: ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > > A server of similar spec which is running freebsd-current with seven bhyve > instances doesn't have this issue: > Based upon the output neither ram nor swap seems like similar spec so I wonder if you could say what you mean by that. -- Adam