From nobody Thu Mar 10 14:31:33 2022 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 18A7B19F804A for ; Thu, 10 Mar 2022 14:31:39 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: from mail-wr1-x434.google.com (mail-wr1-x434.google.com [IPv6:2a00:1450:4864:20::434]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4KDs2L1Ywhz3LQV; Thu, 10 Mar 2022 14:31:38 +0000 (UTC) (envelope-from joh.hendriks@gmail.com) Received: by mail-wr1-x434.google.com with SMTP id k24so8263722wrd.7; Thu, 10 Mar 2022 06:31:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=iW5D5lszLNFWJrx8dKZAtdR7kxPxHp0WQxfdxtIBHC0=; b=IsFPD1N9gG4S0a6wBN8moINqYURjEkqHbh6aVESOb21VewOMmoE8HpgZx3NRSD8APD HMTbFGWeU7b0XCrA8oNDzDt3/1mgWS91aByhuunpeEV7mHiHPKt+qges+JUWSIn4dUmC PLxQrq5PZUMT6/gJtop7G/McYxYoBnF+00QpIsdSOuKO6Dztqic+WYOzKj7RtaAM5+yc f8kjmNJHULnCPRJY0bPK622htK+ulP+uLAAO3JAFG9hcO7uCfV9o9/E4BwHBdEi+O6QV /7Xm726ce6QRSIHjyHRHjujJjVVYKvQlp/W4pCGqYyfi73zZZTdzr/bBpMuMn0XaMsik +u4Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=iW5D5lszLNFWJrx8dKZAtdR7kxPxHp0WQxfdxtIBHC0=; b=fytPEWGYOTlGclQcY8xEreRCs8UmuJpyRixlwXfQ99zJK3TnZ6nhnW2W96XrwmYjIn M5iwH16WFrqF7AArWJ57qfwulSTZuJqLxy/i5S2pyf1Mccbf40nmuzT5SSMW3g7PoMbl Z8D8IbUlNujjo9W2EzCL7GiUAfoYHdWi9ASv2UnFtZfB4O1n167cvVXGyB7cprHaBeIc FiG4F7AJ+dUGdXgoBYQeinmVWwh5gvGR29eYP8xng76ClIXULpNtohM8zYOUzMh3YRnb U95rnalTUBbVGp5TNGppuNr6X6/jkoRU714vdy9AdyxJdgXOqCuPmMm+nq2y++h1Bhx6 r77A== X-Gm-Message-State: AOAM533sx9h8yohHJU7j0bN/4LXkw1S8ETH5ahkuugjQqby43N/8ddoW p81XfNYAzuOmRAmnPN7O0vzYJvYxiFUhrQ== X-Google-Smtp-Source: ABdhPJwqwbT8bK1jlb5Lc+2p6AJeElXN8ZMd8fCyuzaVqkMlGcvd3dwxQ5qsD31N0YP3YHwlhGb+bw== X-Received: by 2002:a05:6000:178c:b0:203:86a7:e49 with SMTP id e12-20020a056000178c00b0020386a70e49mr2980738wrg.640.1646922697016; Thu, 10 Mar 2022 06:31:37 -0800 (PST) Received: from [192.168.1.18] (85-147-130-226.cable.dynamic.v4.ziggo.nl. [85.147.130.226]) by smtp.gmail.com with ESMTPSA id n65-20020a1c2744000000b003862bfb509bsm7652271wmn.46.2022.03.10.06.31.36 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 10 Mar 2022 06:31:36 -0800 (PST) Message-ID: <981aae9a-b31f-a4c0-216a-f448884fce2d@gmail.com> Date: Thu, 10 Mar 2022 15:31:33 +0100 List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@freebsd.org MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.6.2 Subject: Re: epair and vnet jail loose connection. Content-Language: en-US To: Wolfgang Zenker , Kristof Provost Cc: "Bjoern A. Zeeb" , mops@punkt.de, FreeBSD Net References: <051d51b6-2a07-fbc6-7b4d-13947e7fcdbb@gmail.com> <65a18f1b-ea22-a3d2-b4ad-41fd52b7fbae@gmail.com> From: Johan Hendriks In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4KDs2L1Ywhz3LQV X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20210112 header.b=IsFPD1N9; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of johhendriks@gmail.com designates 2a00:1450:4864:20::434 as permitted sender) smtp.mailfrom=johhendriks@gmail.com X-Spamd-Result: default: False [-3.98 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2a00:1450:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; RCPT_COUNT_FIVE(0.00)[5]; RCVD_COUNT_THREE(0.00)[3]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.98)[-0.984]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2a00:1450::/32, country:US]; MID_RHS_MATCH_FROM(0.00)[]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-0.998]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20210112]; FROM_HAS_DN(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[text/plain]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2a00:1450:4864:20::434:from]; MLMMJ_DEST(0.00)[freebsd-net]; RCVD_TLS_ALL(0.00)[] X-ThisMailContainsUnwantedMimeParts: N On 10/03/2022 13:37, Wolfgang Zenker wrote: > Hi Kristof, > > Am Thu, Mar 10, 2022 at 12:44:00PM +0100 schrieb Kristof Provost: >> On 10 Mar 2022, at 10:13, Johan Hendriks wrote: >>> On 10/03/2022 08:54, Patrick M. Hausen wrote: >>>> Hi Johan, >>>> >>>> we experience the same on 13.1-PRERELEASE. Currently trying to collect some evidence >>>> (dtrace) to send to Kristof Provost who was so kind to assist. We are hit by the problem >>>> in production in 12-24 hour intervals. Have not done any artificial load tests, yet. >>>> >>>> May I ask you to run this dtrace script while at least one jail is disconnected and while >>>> traffic is present that is trying to reach the jail? If you can afford to do that in production (?) >>>> that would be great. Forward to Kristof (kp@), please. >>>> >>>> Thanks and kind regards >>>> Patrick >>>> ---------- >>>> #!/usr/sbin/dtrace -s >>>> >>>> BEGIN >>>> { >>>> self->in_menq = 0; >>>> } >>>> >>>> fbt:if_epair:epair_menq:entry >>>> { >>>> self->in_menq = 1; >>>> printf("In epair_menq"); >>>> } >>>> >>>> fbt:if_epair:epair_menq:return >>>> / self->in_menq == 1 / >>>> { >>>> self->in_menq = 0; >>>> printf("Leave epair_menq"); >>>> } >>>> >>>> fbt:kernel:taskqueue_enqueue:entry >>>> / self->in_menq == 1 / >>>> { >>>> printf("Enqueue task"); >>>> >>>> } >>>> >>>> fbt:if_epair:epair_tx_start_deferred:entry >>>> { >>>> printf("epair_tx_start_deferred"); >>>> } >>>> ---------- >>>> >>> I was asked the above, so hereby the output of that command. >>> I did do a  hey -h2 -n 10 -c 10 -z 60s https://wp.test.nl to that machine and in the 60 seconds the jail became unresponsive. Then i did run the dtrace.sh script above like so /root/bin/dtrace.sh > /root/dtrace_output >>> >>> I hope this helps, if you need anything please let me know. Also root access is possible if you want. That way you do not have to create a test environment. >> Were there other epair interfaces running at this time, with active traffic? >> The dtrace output appears to show that the appropriate callouts (to epair_tx_start_deferred()) are getting through, so I’d expect traffic to be flowing. > There is one second jail using epair on that system, using the same > bridge as well. This second jail is a low-traffic system, it is unlikely > but possible that there was some traffic during that time. > In all previous cases this second jail continued to be reachable all > the time. > > Wolfgang > I use 13-STABLE from 01-02-2022 this year and i can not replicate this, i step ahead a week and do a rebuild and try again.