From owner-freebsd-net@freebsd.org Mon May 3 19:31:15 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id ABA1F63B13F for ; Mon, 3 May 2021 19:31:15 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: from mail-qk1-x72d.google.com (mail-qk1-x72d.google.com [IPv6:2607:f8b0:4864:20::72d]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FYtPV1XC4z3PWV for ; Mon, 3 May 2021 19:31:10 +0000 (UTC) (envelope-from markjdb@gmail.com) Received: by mail-qk1-x72d.google.com with SMTP id o5so6310211qkb.0 for ; Mon, 03 May 2021 12:31:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=Hi5AR/OQxdxnvT0o78potHzW9DwEtx/4Gvpu0uG9vrI=; b=UzzgqkAKbyZZhFLyYz98WCw52RfHUQCsedmSktBdo7oW+Hbt9w4nUXRZc3ZvC3AylE OC0X0wjsfNWTzlmnkJY2Qar7WmCcLwSG446VArA/g4EgcnT/Y70bL9MUrxcM1qDCry0B Miyw9nGlangPxiGWG/Z4yEh6nwPpouHuIq7t0kiECa2sAUS2Oov9fNn3tS809IH9c7E9 Vmp+NbU0KV67E4MXMVCK+jP6cikQKYDV61Ng+sRQGRoBJRzwKl6SOmNvFkVPnHJKxn/F cPhmVdlarGucULbJ6kOKUFniLyWR9S6De6kOTfAOY7iuvBh6Gcv5BCnD6JppF/QkTFvp gqJA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition :content-transfer-encoding:in-reply-to; bh=Hi5AR/OQxdxnvT0o78potHzW9DwEtx/4Gvpu0uG9vrI=; b=TFIwrZ8IZgiQOzF+ZRzm7lH5MKkgGeVGXuLjBGKKc2wG+bfsnHszbZW7lrL3pr+0wP aivPrwt7mrGkqJUCuJ3niSA4ou7o+yYu0LdtxK4Q+eUYgcxF0ASnc1JWqMd+jHaggxJZ 1jhbz7ZLSP7N5FWsmKPtgWiWfcl5owv7h74DPNKNKpQY7QoubF8y5tZWS35Je9c5y0qz i2xD5KZQOTKLj/7QmYvaYDzRMAeK3qqWKvpq/wwK/oxQbiNtFEDSMPRyJt4xqESIMfYM n7bEi77paL2ijyPrf8+dKJYhNqiaxeM4AN+oUyESHfL7F4PmzkmyfAFoYmCX+gs59hjU MDZg== X-Gm-Message-State: AOAM530rEQdpJ5+gVgqvP28n5Z510uTvxrMp3Rcp5RyKBIAp+SnmCm/h lDgEi0prh7Ee2hJksIVn8yJO/MYCy1z1VA== X-Google-Smtp-Source: ABdhPJyUQdEwSGfb0mahgOsMGudNcEdaYG5+SffN9kkpRIeNu6rCSiMTpJHC7TzqiynCBkBOYj+OVA== X-Received: by 2002:ae9:df82:: with SMTP id t124mr20472338qkf.267.1620070269300; Mon, 03 May 2021 12:31:09 -0700 (PDT) Received: from nuc ([142.126.164.150]) by smtp.gmail.com with ESMTPSA id t18sm9333469qkj.75.2021.05.03.12.31.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 03 May 2021 12:31:08 -0700 (PDT) Sender: Mark Johnston Date: Mon, 3 May 2021 15:31:09 -0400 From: Mark Johnston To: "Andrey V. Elsukov" Cc: =?iso-8859-1?Q?=D6zkan?= KIRIK , FreeBSD Net Subject: Re: IPsec performace - netisr hits %100 Message-ID: References: <50cfc0e6-5cc6-7004-2566-bc06428d4394@yandex.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <50cfc0e6-5cc6-7004-2566-bc06428d4394@yandex.ru> X-Rspamd-Queue-Id: 4FYtPV1XC4z3PWV X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UzzgqkAK; dmarc=none; spf=pass (mx1.freebsd.org: domain of markjdb@gmail.com designates 2607:f8b0:4864:20::72d as permitted sender) smtp.mailfrom=markjdb@gmail.com X-Spamd-Result: default: False [-1.75 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; RCVD_COUNT_THREE(0.00)[3]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; NEURAL_HAM_SHORT(-0.05)[-0.046]; FREEMAIL_TO(0.00)[yandex.ru]; FORGED_SENDER(0.30)[markj@freebsd.org,markjdb@gmail.com]; MIME_TRACE(0.00)[0:+]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; FROM_NEQ_ENVFROM(0.00)[markj@freebsd.org,markjdb@gmail.com]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::72d:from]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; TAGGED_RCPT(0.00)[]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::72d:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; BLOCKLISTDE_FAIL(0.00)[142.126.164.150:query timed out,2607:f8b0:4864:20::72d:query timed out]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::72d:from]; MID_RHS_NOT_FQDN(0.50)[]; FREEMAIL_CC(0.00)[gmail.com,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 03 May 2021 19:31:15 -0000 On Sun, May 02, 2021 at 04:08:18PM +0300, Andrey V. Elsukov wrote: > 30.04.2021 23:32, Mark Johnston пишет: > > Second, netipsec unconditionally hands rx processing off to netisr > > threads for some reason, that's why changing the dispatch policy doesn't > > help. Maybe it's to help avoid running out of kernel stack space or to > > somehow avoid packet reordering in some case that is not clear to me. I > > tried a patch (see below) which eliminates this and it helped somewhat. > > If anyone can provide an explanation for the current behaviour I'd > > appreciate it. > > Previously we have reports about kernel stack overflow during IPsec > processing. In your example there is only one IPsec transform is > configured, but it is possible to configure several in the bundle, > AFAIR, it is limited to 4 transforms. E.g. if you configure ESP+AH - it > is bundle of two transforms and this will grow kernel stack requirements. Is it only a problem for synchronous crypto ops? With hardware drivers I'd expect the stack usage to be reset after each transform, since completions are handled by a dedicated thread. There is also the net.inet.ipsec.async_crypto knob, which has a similar effect I think.