From owner-freebsd-net@freebsd.org Sat May 1 21:18:18 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 58F13638B41 for ; Sat, 1 May 2021 21:18:18 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: from mail-vs1-xe34.google.com (mail-vs1-xe34.google.com [IPv6:2607:f8b0:4864:20::e34]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FXht101xbz4vG6; Sat, 1 May 2021 21:18:16 +0000 (UTC) (envelope-from ozkan.kirik@gmail.com) Received: by mail-vs1-xe34.google.com with SMTP id j13so1052800vsf.2; Sat, 01 May 2021 14:18:16 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=qF0ily5V+WxUp3u7uIyKNpvpwZJID4EBCwR+BMhdYXU=; b=dD/048Ql+Z6FJoncMAWbmaiKLJlS5c3HKd0RN9k3lFY8AOtFKEKMbK6NHOIqCwqP2J CYHc+v0AfZ0RqBTO+F0ERak04BHXTIbs+9dHsKISE8SvwZUOsanmcKhh5Fw2fjWmfY4m 1lArhtKU/sl6OWipPGiqaklXGHu23pancUa/fZ15T4x86VH2ABN6v8T+tUPfdtutsTlP H00No5T08tbPRD+SsYnDGLLRg5Bl519JSpg9X4RukagR6NL4bn1MWacFDDFTNfm0tD13 FpMnNfWv6cT3AWUHXg15F+wMLHKq+cpPbFPVt6KUE9SErcBP4cArU2h2wvRs02ju7dgB STyg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qF0ily5V+WxUp3u7uIyKNpvpwZJID4EBCwR+BMhdYXU=; b=hda5r+BsbAv/6fv9+3UpHb5FLa9HxHcuEjuu0pM3pbV27EuILLpYCy+SAPw/gZGW9k TqauVSBtHu2g4K4yWGy2EhN65eNijlH9fomdaBbKxMPB1FCf5xFzwNAQDXiZwSu1BxjA YvhEGHkCczjUlbQ6wZfihr+I0arIAeymaQ9EbJowVwwDhnPsOxQnAsPwrwCg5IgWnx2M 0GLPJgFiwxzMjSXVGQDSFVQIqsgaC84exbGB5FYoIYihH4yRVzYl8WoqaFtLAgu6F6WA dxjkjH4p5a1n3/GQIAnF65Vd1kGPVYz9O1+/n1Ungh1XoMqemZSGwVa5RGDLTFOnMV56 DX2A== X-Gm-Message-State: AOAM532ty4OpRa1TMOzE8HFtDdEvQfQQSUf47x++BuUlPoOFXkPDVH5p wO+oTPetNTcsphKX3rTCg9wrFCdSBTUZ/pHFYB1kZ1ZZ X-Google-Smtp-Source: ABdhPJy+LX7AYwlgRyJj2KGrcnGl2CXDQYL2dqsu26bY+nCumaHfupPIvXdD9+jn1mfp5P7rQ6mebsWzvWGf58qEILw= X-Received: by 2002:a67:f693:: with SMTP id n19mr11460689vso.55.1619903895247; Sat, 01 May 2021 14:18:15 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: =?UTF-8?B?w5Z6a2FuIEtJUklL?= Date: Sun, 2 May 2021 00:18:04 +0300 Message-ID: Subject: Re: IPsec performace - netisr hits %100 To: Mark Johnston Cc: FreeBSD Net X-Rspamd-Queue-Id: 4FXht101xbz4vG6 X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=pass header.d=gmail.com header.s=20161025 header.b=dD/048Ql; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (mx1.freebsd.org: domain of ozkankirik@gmail.com designates 2607:f8b0:4864:20::e34 as permitted sender) smtp.mailfrom=ozkankirik@gmail.com X-Spamd-Result: default: False [-3.35 / 15.00]; FREEMAIL_FROM(0.00)[gmail.com]; R_SPF_ALLOW(-0.20)[+ip6:2607:f8b0:4000::/36]; HAS_ATTACHMENT(0.00)[]; TO_DN_ALL(0.00)[]; DKIM_TRACE(0.00)[gmail.com:+]; RCPT_COUNT_TWO(0.00)[2]; DMARC_POLICY_ALLOW(-0.50)[gmail.com,none]; NEURAL_HAM_SHORT(-0.97)[-0.974]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:+,3:~,4:+]; R_MIXED_CHARSET(0.62)[subject]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:2607:f8b0::/32, country:US]; TAGGED_FROM(0.00)[]; DWL_DNSWL_NONE(0.00)[gmail.com:dkim]; ARC_NA(0.00)[]; RBL_DBL_DONT_QUERY_IPS(0.00)[2607:f8b0:4864:20::e34:from]; R_DKIM_ALLOW(-0.20)[gmail.com:s=20161025]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/mixed,multipart/alternative,text/plain,image/svg+xml]; SPAMHAUS_ZRD(0.00)[2607:f8b0:4864:20::e34:from:127.0.2.255]; RCVD_IN_DNSWL_NONE(0.00)[2607:f8b0:4864:20::e34:from]; RCVD_COUNT_TWO(0.00)[2]; RCVD_TLS_ALL(0.00)[]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 01 May 2021 21:18:18 -0000 the previous flamegraph is captured while iperf client (jail) sends to iperf server. the attached flamegraph to this mail is captured while iperf configured full-duplex mode. Throughput is about up: 1.5 Gbps down: 1.5 Gbps total: 3 Gbps On Sat, May 1, 2021 at 11:57 PM =C3=96zkan KIRIK wr= ote: > Hello, > The flamegraph is attached. > > # netstat -s > ... > ipsec: > 0 inbound packets violated process security policy > 0 inbound packets failed due to insufficient memory > 0 invalid inbound packets > 0 outbound packets violated process security policy > 0 outbound packets with no SA available > 0 outbound packets failed due to insufficient memory > 0 outbound packets with no route available > 0 invalid outbound packets > 0 outbound packets with bundled SAs > 0 spd cache hits > 0 spd cache misses > 0 clusters copied during clone > 0 mbufs inserted during makespace > ah: > 0 packets shorter than header shows > 0 packets dropped; protocol family not supported > 0 packets dropped; no TDB > 0 packets dropped; bad KCR > 0 packets dropped; queue full > 0 packets dropped; no transform > 0 replay counter wraps > 0 packets dropped; bad authentication detected > 0 packets dropped; bad authentication length > 0 possible replay packets detected > 0 packets in > 0 packets out > 0 packets dropped; invalid TDB > 0 bytes in > 0 bytes out > 0 packets dropped; larger than IP_MAXPACKET > 0 packets blocked due to policy > 0 crypto processing failures > 0 tunnel sanity check failures > AH output histogram: > aes-gmac-128: 35517864 > esp: > 0 packets shorter than header shows > 0 packets dropped; protocol family not supported > 0 packets dropped; no TDB > 0 packets dropped; bad KCR > 0 packets dropped; queue full > 20 packets dropped; no transform > 0 packets dropped; bad ilen > 0 replay counter wraps > 0 packets dropped; bad encryption detected > 0 packets dropped; bad authentication detected > 0 possible replay packets detected > 23598941 packets in > 11918943 packets out > 0 packets dropped; invalid TDB > 32247932688 bytes in > 630318292 bytes out > 0 packets dropped; larger than IP_MAXPACKET > 0 packets blocked due to policy > 0 crypto processing failures > 0 tunnel sanity check failures > ESP output histogram: > aes-gcm-16: 35517864 > > dev.qat.1.stats.sym_alloc_failures: 0 > dev.qat.1.stats.ring_full: 1267 > dev.qat.1.stats.gcm_aad_updates: 0 > dev.qat.1.stats.gcm_aad_restarts: 0 > dev.qat.1.%domain: 0 > dev.qat.1.%parent: pci16 > dev.qat.1.%pnpinfo: vendor=3D0x8086 device=3D0x37c8 subvendor=3D0x8086 > subdevice=3D0x0000 class=3D0x0b4000 > dev.qat.1.%location: slot=3D0 function=3D0 dbsf=3Dpci0:182:0:0 > dev.qat.1.%driver: qat > dev.qat.1.%desc: Intel C620/Xeon D-2100 QuickAssist PF > dev.qat.0.stats.sym_alloc_failures: 0 > dev.qat.0.stats.ring_full: 0 > dev.qat.0.stats.gcm_aad_updates: 0 > dev.qat.0.stats.gcm_aad_restarts: 0 > dev.qat.0.%domain: 0 > dev.qat.0.%parent: pci15 > dev.qat.0.%pnpinfo: vendor=3D0x8086 device=3D0x37c8 subvendor=3D0x8086 > subdevice=3D0x0000 class=3D0x0b4000 > dev.qat.0.%location: slot=3D0 function=3D0 dbsf=3Dpci0:181:0:0 > dev.qat.0.%driver: qat > dev.qat.0.%desc: Intel C620/Xeon D-2100 QuickAssist PF > dev.qat.%parent: > > > > > On Sat, May 1, 2021 at 5:51 PM Mark Johnston wrote: > >> On Sat, May 01, 2021 at 04:30:59PM +0300, =C3=96zkan KIRIK wrote: >> > This bug is related to CCR. @Navdeep Parhar , @John >> Baldwin >> > if you are interested to fix this bug related with >> CCR, I >> > can test if you provide patches. Test environment is explained in my >> first >> > email on this thread. >> > >> > @Mark Johnston Now again on stable/13, >> > - with aesni, without netipsec/ipsec_input.c patch - 1.44Gbps - single >> > netisr thread eats %100 cpu >> > - with qat, without netipsec/ipsec_input.c patch - 1.88Gbps - single >> netisr >> > thread eats %100 cpu >> > - with aesni, with netipsec/ipsec_input.c patch - 1.33Gbps >> > - with qat, with netipsec/ipsec_input.c patch - 2.85Gbps - >> > >> > stable/13 results are better then stable/12 but not enough fast. There >> is >> > something makes bottleneck for IPsec. >> >> So with these results it looks like we have 4 crypto threads running, >> which is what I'd expect for two pairs of IP addresses. There is still >> a single-threaded bottleneck. I would suggest generating a flame graph >> using DTrace and https://github.com/brendangregg/FlameGraph to see where >> we're spending CPU time. It would also be useful to know if we're >> getting errors or drops anywhere. The QAT (sysctl dev.qat.*.stats) and >> ESP/AH (netstat -s -p (esp|ah)) counters would be a useful start, in >> addition to counters from cxgbe. >> >