From owner-freebsd-stable@freebsd.org Fri Nov 15 23:24:14 2019 Return-Path: Delivered-To: freebsd-stable@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 472D01B3613 for ; Fri, 15 Nov 2019 23:24:14 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-ot1-x32b.google.com (mail-ot1-x32b.google.com [IPv6:2607:f8b0:4864:20::32b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 47FDvJ5tzdz4SQy; Fri, 15 Nov 2019 23:24:12 +0000 (UTC) (envelope-from delphij@gmail.com) Received: by mail-ot1-x32b.google.com with SMTP id n23so9422372otr.13; Fri, 15 Nov 2019 15:24:12 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=lgvFQ6sU9rL7Irwa38LLlwWqaqrWNvwLHtQ3kxyawtM=; b=q60beAl8H7wTC/UaGE+rmN7ZIIJ9Edw21OaFcfJJOUV3QuqxmtycjyJ+wE1C188QFk BuMAQ6Kozza6M6dcWRQoY7ocoAOS+FJaXZeg5TbHCLdP8gBbjLrvjEq3LKkqpGteoDdy rkZUTl5/51jTCf1ZsfbbHkIyL8sElCdNEnJmaOKANQmWuNjoWmy0ZgHignssL9x9oMC0 StbbVwz7GBsleJ7aqE5krBK+FBcpu/S8OnMkog5M5uw8BJvqLD8K2rW1v5qXcWDfF9UR OWyY9cqXPw1Qfpnju1i2W/Ip9rFKtZiPqmTC9hzjgkKq9Bv/QoGaQc78DfD7fxTKZyCY SH7Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=lgvFQ6sU9rL7Irwa38LLlwWqaqrWNvwLHtQ3kxyawtM=; b=my6rvr21aB44qSqJFZn02gF9xjUpNQb+tbCfR1EB/tsQH+KEmKntSbgG6Hf6gcTutN DFPzJDOfrAWyW/GR3orpXdL5uAs3mYuY9mZwrQcWNo7qgaxw6huEi0rlfnd0V/jbqmXu L4rmWqcY6J8CPibtg4MrWz4bO6I7EU8pOaG2a8qD77FYgWz5KHxoURd+vrFwtTdFAiDg CKCrLZkum8CQv92Qix4S/qvEvZhaSRXjy1SMKe3IATqgkYtNhZ6ZwQMJkgXsoggAdjyu LamcM9Dw7k7lK8Yw8udYPLxNhq39cb9kCU89u6p/CkW4dpp2hoZ5BgP3W29SNS3O4pWb IjIQ== X-Gm-Message-State: APjAAAUOuZ/LRTwq+Yu5Oad9jxoBiLkzS5Slanc7oGLxs6zzd3QQx9rt KvcPuvaAzeUgDvZAKEc5zRKo1InOd27Q/CRxdtf8P4j7cfU= X-Google-Smtp-Source: APXvYqwBJlYiQQnC3KIySTp4XPCwx5DUSEmCpiM01hXLmPduVAuKSGzdRkK8i+BOer7j0VrY6KvV+YYPpTpDD2kYNyQ= X-Received: by 2002:a9d:12a3:: with SMTP id g32mr13821309otg.61.1573860251004; Fri, 15 Nov 2019 15:24:11 -0800 (PST) MIME-Version: 1.0 References: <7fddcea5-2188-afe1-3ea9-a53dffdbec32@grosbein.net> In-Reply-To: <7fddcea5-2188-afe1-3ea9-a53dffdbec32@grosbein.net> From: Xin LI Date: Fri, 15 Nov 2019 15:24:00 -0800 Message-ID: Subject: Re: bhyve memory leak in stable/11 To: Eugene Grosbein Cc: FreeBSD stable , Alexander Motin X-Rspamd-Queue-Id: 47FDvJ5tzdz4SQy X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=q60beAl8; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of delphij@gmail.com designates 2607:f8b0:4864:20::32b as permitted sender) smtp.mailfrom=delphij@gmail.com X-Spamd-Result: default: False [-2.00 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; IP_SCORE(0.00)[ip: (-8.12), ipnet: 2607:f8b0::/32(-2.32), asn: 15169(-1.98), country: US(-0.05)]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE_FREEMAIL(0.00)[]; URI_COUNT_ODD(1.00)[3]; TO_MATCH_ENVRCPT_SOME(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; RCVD_IN_DNSWL_NONE(0.00)[b.2.3.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.0.4.6.8.4.0.b.8.f.7.0.6.2.list.dnswl.org : 127.0.5.0]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com.dwl.dnswl.org : 127.0.5.0] Content-Type: text/plain; charset="UTF-8" X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 15 Nov 2019 23:24:14 -0000 There are several leaks fixed recently, but not all of them were merged back to stable/11 (e.g. r341705 which may or may not be related to your issue). Personally, I'd recommend updating to 12.x for new features as they are being actively developed and would be supported for about 3 more years compared to 11.x series. On Thu, Nov 14, 2019 at 7:25 AM Eugene Grosbein wrote: > Hi! > > After 1 day and 11 hours of host's uptime I observe significant leak of > bhyve instances RSS: > > PID USERNAME THR PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > 66668 root 34 20 0 60320M 58866M kqread 21 62.0H 692.35% > bhyve: sappdev (bhyve) > 5546 root 26 20 0 22573M 22088M kqread 12 310:39 6.41% > bhyve: srdapp01 (bhyve) > 3842 root 20 20 0 15946M 15862M kqread 13 340:36 14.03% > bhyve: sdc01 (bhyve) > 4729 root 20 20 0 13792M 13455M kqread 8 79:40 1.02% > bhyve: skms01 (bhyve) > 4283 root 28 20 0 12721M 12682M kqread 29 92:23 0.80% > bhyve: sfile01 (bhyve) > 8057 root 34 20 0 8512M 8183M kqread 25 186:52 17.02% > bhyve: ssql01 (bhyve) > 23336 root 34 20 0 16418M 7716M kqread 14 114:23 17.62% > bhyve: solap (bhyve) > > The package vm-bhyve-1.2.3 is used here to manage bhyve instances > and sappdev.conf has memory=40G limit. srdapp01 is limited to 12G, > sdc01 to 4G, skms01 to 2G, sfile01 to 2G, ssql01 to 8G and solap to 16G. > > Most of are using much more memory than they should. All of them run > different versions of Windows servers. > > The host is 11.3-STABLE/amd64 r354667 with 360G RAM and guest systems use > ZFS ZVOLs and vfs.zfs.arc_max=160g. > > How do I debug this? > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >