From owner-freebsd-net@freebsd.org Thu May 6 14:07:44 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 41235622DFD for ; Thu, 6 May 2021 14:07:44 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: from mail-vs1-xe2b.google.com (mail-vs1-xe2b.google.com [IPv6:2607:f8b0:4864:20::e2b]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4Fbb4v3ZFJz4jXc; Thu, 6 May 2021 14:07:43 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: by mail-vs1-xe2b.google.com with SMTP id h18so2951968vsp.8; Thu, 06 May 2021 07:07:43 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=/aXhl6NO/ivNXF0NXI10VKc+a+6SdRWxZ7L11tzh1m8=; b=cyPyD2FwcGycagRdIOYJfEx2RkqiuazvFEdkJg+3R/ax6Qs732tfTL3c0Z/Cugb1X3 eqbqPFZ2r4Plpw1Tk4WEjVXtadOLx3DIQy9M6xOxpBLP+btwGWE6rPrDAhPaMN9to6vE +5x5KMPLM9Y5tFkEuTXarduAfphgHIjLF1Oi/H8/n+h9x2tuNL/TUubcC53dG6Sn/ZW+ D1kuvwbi4VXFoFxhBWWsWxnd57+CN1fSh+sIvC+m3IF4bCuoVnU3vwTMO4yFkSwXrshj 0BTEJ9a4+MwtfnzBo2ZNGH30BfXsKlq8nwRlvqXDBrpd0k/0wK7skwrmNR7ZjcLf42oL k2ww== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=/aXhl6NO/ivNXF0NXI10VKc+a+6SdRWxZ7L11tzh1m8=; b=fnYqLnuLf4Gefwh6olxnEIWqKqhcUiCqXwUjex75GzVetJeVjMMJI/pXg29Lcqile9 ayJR4d+B7yCEc4H4o6agPAu5D7wjIctbxsyCM6Al3Qrz1MyC/EYA8rxl2qaIxiIu4BfI X1kqN/n/pkbqV1wNNDQTUGsEk+1zxf+79Vw1U8EpCaE0xNCtOMqQ9CnQ9uWyjbUc4avR 3T80/rEmnhqz/+pAXxGRPsv/YxrAqq9skWSsP3RXIMuFolfB/2K5ou9EPfnojWNkW8CW 3D79sYs3P5FpKJ4fFhUvBvBQmunftDZhoOGXoz2YMDJ7pNKn6R9Xz4SqpxbntAoohidH w4Cw== X-Gm-Message-State: AOAM530oP3I0SPrOZai//WztVOMrBdOatEMPNatyUm6qCIF6cc0nIkAR g2O4JgBEk6E6rrBMcTR4nDyfPhXfp3fzDiUO1yTyuH6s7No= X-Google-Smtp-Source: ABdhPJzlq5CdRWBvSU3ccNBpbdgzJHhcUu3M87EavUYjJL392Eu+1D/eLqfwkgPZQrdKTk0h/JV8vWOOFHs0IqesSiI= X-Received: by 2002:a67:c406:: with SMTP id c6mr3313166vsk.33.1620310061595; Thu, 06 May 2021 07:07:41 -0700 (PDT) MIME-Version: 1.0 References: <50cfc0e6-5cc6-7004-2566-bc06428d4394@yandex.ru> In-Reply-To: From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= Date: Thu, 6 May 2021 17:07:30 +0300 Message-ID: Subject: Re: IPsec performace - netisr hits %100 To: Mark Johnston Cc: "Andrey V. Elsukov" , FreeBSD Net X-Rspamd-Queue-Id: 4Fbb4v3ZFJz4jXc X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=cyPyD2Fw; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ozkankirik@gmail.com designates 2607:f8b0:4864:20::e2b as permitted sender) smtp.mailfrom=ozkankirik@gmail.com X-Spamd-Result: default: False [-3.36 / 15.00]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36:c]; FREEMAIL_FROM(0.00)[gmail.com]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-1.00)[-1.000]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_MIXED_CHARSET(0.62)[subject]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::e2b:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-0.98)[-0.984]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::e2b:from:127.0.2.255]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e2b:from]; FREEMAIL_CC(0.00)[yandex.ru,freebsd.org]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net]; RCVD_COUNT_TWO(0.00)[2] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 06 May 2021 14:07:44 -0000 I wonder that if you received the flame graphs ? I also tested system with multiple if_ipsec interfaces using different source-dst tunnel address. By this way, system can utilize all cpu cores. But for single if_ipsec interface, is there a way to speed up transfer ? Thanks! On Mon, May 3, 2021 at 10:31 PM Mark Johnston wrote: > On Sun, May 02, 2021 at 04:08:18PM +0300, Andrey V. Elsukov wrote: > > 30.04.2021 23:32, Mark Johnston =D0=BF=D0=B8=D1=88=D0=B5=D1=82: > > > Second, netipsec unconditionally hands rx processing off to netisr > > > threads for some reason, that's why changing the dispatch policy > doesn't > > > help. Maybe it's to help avoid running out of kernel stack space or = to > > > somehow avoid packet reordering in some case that is not clear to me. > I > > > tried a patch (see below) which eliminates this and it helped somewha= t. > > > If anyone can provide an explanation for the current behaviour I'd > > > appreciate it. > > > > Previously we have reports about kernel stack overflow during IPsec > > processing. In your example there is only one IPsec transform is > > configured, but it is possible to configure several in the bundle, > > AFAIR, it is limited to 4 transforms. E.g. if you configure ESP+AH - it > > is bundle of two transforms and this will grow kernel stack requirement= s. > > Is it only a problem for synchronous crypto ops? With hardware drivers > I'd expect the stack usage to be reset after each transform, since > completions are handled by a dedicated thread. There is also the > net.inet.ipsec.async_crypto knob, which has a similar effect I think. >