From owner-freebsd-net@freebsd.org Fri Apr 30 22:26:53 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 3D6C262C2A5 for ; Fri, 30 Apr 2021 22:26:53 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: from mail-ua1-x930.google.com (mail-ua1-x930.google.com [IPv6:2607:f8b0:4864:20::930]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FX6Rc1fgvz3jpP; Fri, 30 Apr 2021 22:26:51 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: by mail-ua1-x930.google.com with SMTP id x22so4273018uav.8; Fri, 30 Apr 2021 15:26:51 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=sVZaF7JjSGjeU5Ac0KB+ycsxiedF0a3R5Reo5qwpZF4=; b=UgOdb0W/VnnN1kO9AV9iY4Uf797phfpETO8Y+UP30Y7kBPMhdoH4c85+342jJu1KBM MzGo0kI8mfS6+WTcN2x8PuIvNHJ0rb92LYJQiwiIwJL60NACIFTHJah2RnMdZhrDeFtN X049lWdgillxOtKqjGLryTwisbH0+Zd1E76RgeOF010asYm6BcUTP4GhD446vG7rFf8P 2XZIck8n/tQ6i83iObZzkiwcVe5PDnmDD1aDoQs8lY3hNOi1fR6ypGpk1wObsj3VYmkd SUEuXMsEacMeKQuVzawznhPQritAhQE+amtWefVacwBARCI9us32fhN4pTodcSueOFpu FDHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=sVZaF7JjSGjeU5Ac0KB+ycsxiedF0a3R5Reo5qwpZF4=; b=r547oSee6nOlAyJvegE9vJOJ1nsZ5EGHlACpRDJ9CuoO0BhZRFCUmWp1UycM5fPD2S aJB8h1cjnMpN8EU4A3WnMnldOUBmcln92iC/C9TRDlPKVL6GopA9kOPh6gXtiig82Ena vZMAqif0lBG5g8JX7GIGz+RS4oqQy8nMFe3lk3EPd7iCCOFBR27qDVolkKzvNRU64XHI C+txaE42p7aDSqzvX5P5TFMFsQV0PkotIBNnFsx8gsLtEdTUUThWPldvR10W/RiTFXwD cJbQ8SAAI9wmMY7uvmGlQzqfHpvDxcQ84sUloff41R8Sk2835WJ5dhMGz6pTr126htwI L4Bg== X-Gm-Message-State: AOAM533+ek52pLzAs40G6hGaXDjkNswxzjFs6BRGB3AVw7g+fEBw0gl0 Ca+gAbOAjw0rfKDhWD5QAeaO9DnEqa5DFOMPpVVNbkJDlNrLFw== X-Google-Smtp-Source: ABdhPJxC/OVrrSg8qv7Z6Hk1awq3SMa7l+gQpCUtP47hNmbw5C2ByFB+0L1Hz52w+4+LtfC5Mzw/0wmXo8Zhduip9NE= X-Received: by 2002:a9f:2404:: with SMTP id 4mr7658505uaq.48.1619821610884; Fri, 30 Apr 2021 15:26:50 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= Date: Sat, 1 May 2021 01:26:40 +0300 Message-ID: Subject: Re: IPsec performace - netisr hits %100 To: Mark Johnston Cc: FreeBSD Net X-Rspamd-Queue-Id: 4FX6Rc1fgvz3jpP X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=UgOdb0W/; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ozkankirik@gmail.com designates 2607:f8b0:4864:20::930 as permitted sender) smtp.mailfrom=ozkankirik@gmail.com X-Spamd-Result: default: False [-3.28 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; MIME_BASE64_TEXT(0.10)[]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-1.00)[-1.000]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; R_MIXED_CHARSET(0.62)[subject]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::930:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::930:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::930:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Apr 2021 22:26:53 -0000 Here is fresh results; a bit performance gain achieved. But strangely QAT is better now. - with CCR - 2.14 Gbps PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 15 root -16 - 0B 16K CPU7 7 0:11 100.00% [crypto returns 9] 11 root -92 - 0B 1120K CPU4 4 0:10 98.74% [intr{irq295: t6nex0:0a0}] 14 root -16 - 0B 16K CPU11 11 0:09 95.32% [crypto returns 8] 11 root -92 - 0B 1120K CPU8 8 0:07 63.49% [intr{irq297: t6nex0:0a2}] 11 root -92 - 0B 1120K WAIT 12 0:05 42.73% [intr{irq307: t6nex0:1a2}] 16 root -16 - 0B 16K CPU15 15 0:04 33.82% [crypto returns 10] 13 root -16 - 0B 16K RUN 3 0:02 21.78% [crypto returns 7] 4317 root 22 0 23M 4348K sbwait 14 0:01 7.24% iperf -B 172.16.70.6 -c 172.16.68.1 -P 2 -t 20{iperf} 4329 root 23 0 23M 4348K sbwait 14 0:01 7.18% iperf -B 172.16.70.10 -c 172.16.68.1 -P 2 -t 20{iperf} ... 11 root -92 - 0B 1120K WAIT 3 0:00 0.31% [intr{irq294: t6nex0:evt}] 0 root -92 - 0B 2144K - 14 0:00 0.30% [kernel{t6nex0 tq0}] ... - with QAT - 2.63 Gbps PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU COMMAND 8 root -16 - 0B 16K CPU7 7 0:31 92.05% [crypto returns 5] 7 root -16 - 0B 16K CPU0 0 0:29 86.03% [crypto returns 4] 9 root -16 - 0B 16K RUN 15 0:15 43.32% [crypto returns 6] 11 root -92 - 0B 1664K RUN 4 0:15 43.03% [intr{irq297: t6nex0:0a2}] 6 root -16 - 0B 16K crypto 4 0:09 26.01% [crypto returns 3] 11 root -92 - 0B 1664K WAIT 12 0:07 19.88% [intr{irq307: t6nex0:1a2}] 11 root -92 - 0B 1664K WAIT 8 0:02 7.22% [intr{irq368: qat1}] 4313 root 22 0 23M 4348K sbwait 2 0:01 6.51% iperf -B 172.16.70.8 -c 172.16.68.1 -P 2 -t 20{iperf} 4316 root 22 0 23M 4348K sbwait 3 0:01 6.18% iperf -B 172.16.70.9 -c 172.16.68.1 -P 2 -t 20{iperf} ... 11 root -92 - 0B 1664K WAIT 12 0:02 4.52% [intr{irq355: qat0}] ... 11 root -92 - 0B 1664K WAIT 12 0:01 1.89% [intr{irq372: qat1}] 11 root -92 - 0B 1664K WAIT 0 0:00 1.12% [intr{irq360: qat1}] 11 root -92 - 0B 1664K WAIT 2 0:00 1.11% [intr{irq362: qat1}] 11 root -92 - 0B 1664K WAIT 14 0:00 1.10% [intr{irq374: qat1}] 11 root -92 - 0B 1664K WAIT 4 0:00 1.10% [intr{irq364: qat1}] 11 root -92 - 0B 1664K WAIT 10 0:00 1.10% [intr{irq370: qat1}] 11 root -92 - 0B 1664K WAIT 7 0:00 1.09% [intr{irq367: qat1}] 11 root -92 - 0B 1664K WAIT 11 0:00 1.07% [intr{irq371: qat1}] 11 root -92 - 0B 1664K WAIT 5 0:00 1.04% [intr{irq365: qat1}] 11 root -92 - 0B 1664K WAIT 15 0:00 1.04% [intr{irq375: qat1}] 11 root -92 - 0B 1664K WAIT 1 0:00 1.04% [intr{irq361: qat1}] 11 root -92 - 0B 1664K WAIT 6 0:00 1.03% [intr{irq366: qat1}] On Sat, May 1, 2021 at 12:45 AM Mark Johnston wrote: > On Sat, May 01, 2021 at 12:31:57AM +0300, =C3=96zkan KIRIK wrote: > > Hello again, > > > > patch is applied, now netisr is not eating CPU. but performance drops > > around 0.2Gbps according to previous kernel. > > > > I tried also both net.isr.maxthreads=3D1 and net.isr.maxthreads=3D4 . r= esults > > are same > > > > Results are: > > > > - with CCR - 1.8Gbps > > top: > > PID USERNAME PRI NICE SIZE RES STATE C TIME WCPU > COMMAND > > 14 root -16 - 0B 16K CPU5 5 1:38 100.00% > [crypto > > returns 8] > > 3 root -16 - 0B 16K CPU1 1 0:58 77.83% > [crypto > > returns 0] > > Could you also try this patch? It won't help with aesni. > > diff --git a/sys/opencrypto/crypto.c b/sys/opencrypto/crypto.c > index dfd22662e87f..bc23056e86ad 100644 > --- a/sys/opencrypto/crypto.c > +++ b/sys/opencrypto/crypto.c > @@ -94,6 +94,7 @@ struct crypto_session { > void *softc; > uint32_t hid; > uint32_t capabilities; > + uint64_t id; > }; > > SDT_PROVIDER_DEFINE(opencrypto); > @@ -572,6 +573,7 @@ crypto_select_driver(const struct cryptoini *cri, int > flags) > int > crypto_newsession(crypto_session_t *cses, struct cryptoini *cri, int cri= d) > { > + static uint64_t sessid =3D 0; > crypto_session_t res; > void *softc_mem; > struct cryptocap *cap; > @@ -616,6 +618,7 @@ crypto_newsession(crypto_session_t *cses, struct > cryptoini *cri, int crid) > softc_mem =3D malloc(softc_size, M_CRYPTO_DATA, M_WAITOK | M_ZERO= ); > res =3D uma_zalloc(cryptoses_zone, M_WAITOK | M_ZERO); > res->softc =3D softc_mem; > + res->id =3D atomic_fetchadd_64(&sessid, 1); > > CRYPTO_DRIVER_LOCK(); > cap =3D crypto_checkdriver(hid); > @@ -1016,7 +1019,7 @@ crypto_dispatch(struct cryptop *crp) > binuptime(&crp->crp_tstamp); > #endif > > - crp->crp_retw_id =3D ((uintptr_t)crp->crp_session) % > crypto_workers_num; > + crp->crp_retw_id =3D crp->crp_session->id % crypto_workers_num; > > if (CRYPTOP_ASYNC(crp)) { > if (crp->crp_flags & CRYPTO_F_ASYNC_KEEPORDER) { >