From nobody Tue Apr 5 08:25:33 2022 X-Original-To: stable@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id BDA2F1A81224 for ; Tue, 5 Apr 2022 08:25:38 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from elf.hq.norma.perm.ru (mail.norman-retail.ru [128.127.146.8]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "mail.norma.perm.ru", Issuer "Let's Encrypt Authority X3" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KXgh14Xy9z4d1n for ; Tue, 5 Apr 2022 08:25:37 +0000 (UTC) (envelope-from eugene@zhegan.in) Received: from [192.168.243.7] ([192.168.243.7]) by elf.hq.norma.perm.ru (8.16.1/8.15.2) with ESMTP id 2358Jvcu035812 for ; Tue, 5 Apr 2022 13:19:57 +0500 (+05) (envelope-from eugene@zhegan.in) DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=vivat-retail.ru; s=key; t=1649146797; bh=b4BEXztZLzqfT4PR9iw8N4oGIZSvy5ZJ28S02qPVw8o=; h=Date:Subject:To:References:From:In-Reply-To; b=F3ku92pJ9hUjE3rKcW1wszs9Q6PSm3CHZ/SiiUBpF5i7fwNqmLFDw7mc/J/qNNeHN EJK3GVgqXDJIgcsW52k28ZsQ852UEp/YLvBoOh9UfjCNsK5x5TND39E7Dlygi1dTPY U7hK3+FyxkWu1uSN7g8iVRqTRH8XwjQTGG/wAZOY= Message-ID: Date: Tue, 5 Apr 2022 13:25:33 +0500 List-Id: Production branch of FreeBSD source code List-Archive: https://lists.freebsd.org/archives/freebsd-stable List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-stable@freebsd.org X-BeenThere: freebsd-stable@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Thunderbird/91.7.0 Subject: Re: FreeBSD on KVM, Q35 and kernel structure leak Content-Language: ru To: stable@freebsd.org References: From: "Eugene M. Zheganin" In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4KXgh14Xy9z4d1n X-Spamd-Bar: -- Authentication-Results: mx1.freebsd.org; dkim=fail ("headers rsa verify failed") header.d=vivat-retail.ru header.s=key header.b=F3ku92pJ; dmarc=none; spf=pass (mx1.freebsd.org: domain of eugene@zhegan.in designates 128.127.146.8 as permitted sender) smtp.mailfrom=eugene@zhegan.in X-Spamd-Result: default: False [-2.30 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; R_DKIM_REJECT(1.00)[vivat-retail.ru:s=key]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[stable@freebsd.org]; TO_DN_NONE(0.00)[]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000]; DMARC_NA(0.00)[zhegan.in]; R_SPF_ALLOW(-0.20)[+a]; DKIM_TRACE(0.00)[vivat-retail.ru:-]; NEURAL_HAM_SHORT(-1.00)[-1.000]; MLMMJ_DEST(0.00)[stable]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; RCVD_TLS_LAST(0.00)[]; ASN(0.00)[asn:212494, ipnet:128.127.146.0/24, country:RU]; RCVD_COUNT_TWO(0.00)[2]; MID_RHS_MATCH_FROM(0.00)[] X-ThisMailContainsUnwantedMimeParts: N Follow-up: 13.0-RELEASE also has this problem (I upgraded to a recent version trying to see if it solves the issue), and of course when running similar setup (heavily loaded nginx + grpc application) on the baremetal server everything is just fine (regardless of the version - I have several 12.x and 13.x). This also may be related to the fact that this particular VM is using vtnet(4) driver; but I'm using the same cloud VM type in Hetzner to run nginx + prometheus, it has the uptime of 19 days and shows no signs of the leakage. 05.04.2022 3:19, Eugene M. Zheganin пишет: > Hello, > > I'm trying to run FreeBSD 13.1-RC1 (releng/13.1-n250066-aef190f298a) > on KVM/Qemu, presumably Q35 type VM in Hetzner. > > After some uptime and considerable amount of connections, VM leaks > tcp_inpcb to the limit: > > > ITEM                   SIZE  LIMIT     USED     FREE      REQ > FAILSLEEP XDOMAIN > tcp_inpcb:              496, 510671,  510281,     391, 617269,11530,   > 0,   0 > > > Causing multiple dmesg errors > > sonewconn: pcb 0xfffff8001ac8bd90: pru_attach() failed > sonewconn: pcb 0xfffff8000ab625d0: pru_attach() failed > sonewconn: pcb 0xfffff8000af999b0: pru_attach() failed > sonewconn: pcb 0xfffff8000ab621f0: pru_attach() failed > sonewconn: pcb 0xfffff8000ab62000: pru_attach() failed > sonewconn: pcb 0xfffff8000ab625d0: pru_attach() failed > sonewconn: pcb 0xfffff8000af999b0: pru_attach() failed > sonewconn: pcb 0xfffff8000af999b0: pru_attach() failed > sonewconn: pcb 0xfffff8000af993e0: pru_attach() failed > sonewconn: pcb 0xfffff8000af999b0: pru_attach() failed > sonewconn: pcb 0xfffff8000ab627c0: pru_attach() failed > > > console errors (even on low number of kern.ipc.numopensockets: 1447) > > > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > [zone: tcp_inpcb] kern.ipc.maxsockets limit reached > > > and inability to connect to just anything: > > > # telnet 127.0.0.1 4080 > Trying 127.0.0.1... > telnet: socket: No buffer space available > > > Even when the number of connections id dropped (for instance I remove > this VM from balancing) it cannot recover (not freeing tcp_inpcb). > > > is there any hope considering this ? > > > Thanks. > > Eugene. > >