From owner-freebsd-questions@freebsd.org Mon Jun 18 04:47:44 2018 Return-Path: Delivered-To: freebsd-questions@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id CEFC0101658B; Mon, 18 Jun 2018 04:47:43 +0000 (UTC) (envelope-from michaelsprivate@gmail.com) Received: from mail-wr0-x22b.google.com (mail-wr0-x22b.google.com [IPv6:2a00:1450:400c:c0c::22b]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (Client CN "smtp.gmail.com", Issuer "Google Internet Authority G2" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 2C0016DEC8; Mon, 18 Jun 2018 04:47:43 +0000 (UTC) (envelope-from michaelsprivate@gmail.com) Received: by mail-wr0-x22b.google.com with SMTP id d2-v6so15253577wrm.10; Sun, 17 Jun 2018 21:47:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=FIpIzt2fCLYBjQDP4IzSgyZWUMxU9Zd4asZZe8s1FnI=; b=jex3euNIB+mFSVQvv+CNV0ouk8Y1OdqsHr+p5GgvBBciTwM/zKAuERRgwQknc9a5jT dyy3+inezWmDLLKKpRYnLe3L3Ae2fNMC2bQrIkGZiBYONvl6qiiZE0NCmyaDKTklE9ev o9Hcwji1g9aOv8+luLucFnj007HRqh/seN7ZX0lCnxmI2KhrpFobq8WO/wUj/QTdW756 2yfHz0iaAiXghndn2emHFMfGl3A08tR2dF1SpVD5iwpkaGPloZJYgP9pFldv76LSN8dl W4xyZ1ki1QoaSbeRjJar2iqA5pAYiHDT2pdRMOFMB02x6ijKFJs6ooO55c8LYERkxMzD 4IBA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=FIpIzt2fCLYBjQDP4IzSgyZWUMxU9Zd4asZZe8s1FnI=; b=GayWcqCxRsFuU8iS3E2ng10uOKpP6pFsq14ZAcSw9q0j30WBpNGEJpwmjqiYNoFaIo jHv78tS1ksserct+/D4zDaCwtPZJ4skV67m7Ahg6zdwdIhtyD4Mc/0MbAP3u/cHyfSLb w5HEA4yKGefxrpdoQnMj0ja6G0GRNFGeKL5hDBsTRtcBjvBLnvO7gX5aRJZJFQFd7fhK B0AjC8L08IlEEFtvMmqRbwXtZrkc8JdF8ijeewMf94azkCAqUbQBJ7yidH6txzguZpJc KGzVcqXPiaDZT+l6rt0YJewHG4Gds4cX5ToCcNR+0U91NvZiE7rC+M+8HNZwtoOjzyZl eI5A== X-Gm-Message-State: APt69E3aht6TM7LD6cl6A4azFiR5nlUkKOaXAyuVNfDadrtEcXcuY2zC JlrXiwoF60ZfQg0HpWZQmS2Xc30DgyyBCbGmtDs+mA== X-Google-Smtp-Source: ADUXVKI4dKlpGO5UojqaiE/U10Sr3muuOPh5/4ir+DZjPI6dYmLxf+eJ5G9LRMHM1vxd9NTF0ZzoSCqMS4Oelasfbac= X-Received: by 2002:adf:efcf:: with SMTP id i15-v6mr9458643wrp.195.1529297261899; Sun, 17 Jun 2018 21:47:41 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Michael Schuster Date: Mon, 18 Jun 2018 06:47:30 +0200 Message-ID: Subject: Re: lightly loaded system eats swap space To: tech-lists@zyxst.net Cc: freebsd-stable@freebsd.org, freeBSD Mailing List Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.26 X-BeenThere: freebsd-questions@freebsd.org X-Mailman-Version: 2.1.26 Precedence: list List-Id: User questions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 18 Jun 2018 04:47:44 -0000 I'd suggest - if you haven't done so yet - you familiarize yourself with DTrace and write a probe that fires when swap_pager_getswapspace() fails, to print execname and both kernel and userland stacks (or aggregations thereof). That should give you a starting idea of what's going on. HTH Michael On Mon, Jun 18, 2018 at 12:20 AM tech-lists wrote: > Hello list, > > context is (server) > freebsd-11-stable r333874, ZFS raidz1-0 (3x4TB disks), 128GB RAM, > E5-2630 @2.3GHz, generic kernel. > > There's one bhyve guest on this server (using 4x cpu and 16GB RAM, also > freebsd-11-stable) > > There have been no special options for zfs configuration on the server, > apart from several datasets having the compressed property set (lz4). > > The server runs nothing else really apart from sshd and it uses ntpd to > sync local time. > > How come such a lightly loaded server with plenty of resources is eating > up swap? If I run two bhyve instances, i.e. two of the same size as > indicated above, so 32GB used for the bhyves, I'll get out-of-swapspace > errors in the daily logs: > > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > +swap_pager_getswapspace(24): failed > > Here's top, with one bhyve instance running: > > last pid: 49494; load averages: 0.12, 0.13, 0.88 > > > up 29+11:36:06 22:52:45 > 54 processes: 1 running, 53 sleeping > CPU: 0.4% user, 0.0% nice, 0.4% system, 0.3% interrupt, 98.9% idle > Mem: 8664K Active, 52M Inact, 4797M Laundry, 116G Wired, 1391M Buf, > 4123M Free > ARC: 108G Total, 1653M MFU, 105G MRU, 32K Anon, 382M Header, 632M Other > 103G Compressed, 104G Uncompressed, 1.00:1 Ratio > Swap: 4096M Total, 3502M Used, 594M Free, 85% Inuse > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 49491 root 1 4 0 16444K 12024K select 9 0:12 6.49% ssh > 32868 root 12 20 0 9241M 4038M kqread 2 23.2H 1.30% bhyve > 49490 root 1 20 0 10812K 6192K sbwait 5 0:02 0.88% sftp > > From the looks of it, a huge amount of ram is wired. Why is that, and > how would I debug it? > > A server of similar spec which is running freebsd-current with seven > bhyve instances doesn't have this issue: > > last pid: 41904; load averages: 0.26, 0.19, 0.15 > > > up 17+01:06:11 23:14:13 > 27 processes: 1 running, 26 sleeping > CPU: 0.1% user, 0.0% nice, 0.3% system, 0.0% interrupt, 99.6% idle > Mem: 17G Active, 6951M Inact, 41G Laundry, 59G Wired, 1573M Buf, 1315M Free > ARC: 53G Total, 700M MFU, 52G MRU, 512K Anon, 182M Header, 958K Other > 53G Compressed, 69G Uncompressed, 1.30:1 Ratio, 122M Overhead > Swap: 35G Total, 2163M Used, 33G Free, 6% Inuse > > thanks, > -- > J. > _______________________________________________ > freebsd-questions@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-questions > To unsubscribe, send any mail to " > freebsd-questions-unsubscribe@freebsd.org" > -- Michael Schuster http://recursiveramblings.wordpress.com/ recursion, n: see 'recursion'