From owner-freebsd-net@freebsd.org Thu Sep 12 03:56:41 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C9D43E951F for ; Thu, 12 Sep 2019 03:56:41 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: from mail-wm1-x331.google.com (mail-wm1-x331.google.com [IPv6:2a00:1450:4864:20::331]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46TQ1j4xVrz3Gpy; Thu, 12 Sep 2019 03:56:41 +0000 (UTC) (envelope-from akgupt3@gmail.com) Received: by mail-wm1-x331.google.com with SMTP id n10so5833506wmj.0; Wed, 11 Sep 2019 20:56:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=86r827DMVsHVbXybV4Sx+Vo2C+huDUB6U6+yt/f+h1k=; b=gka8i4N4SU2uRkGlV637u7PNNpddqfofLcoRcmtGad4x9kN0cJgA1O5UJplVN8+kyi Gd78ApNa5e50RGnbfYUaDHxMnNDKsweb4vVCinO5+3aAg+Xp++DEC3ZFXOVGOtkVNiFf b6kZFy3VLNiUV3LRP6YpxSZjsUmlMtqmousg5BYYZOSt10DA5EvoYSdVGo6TzQFhdc7S x6tCbumllv8DjngP1GvtfDIOvvUbrbuzJfEMnSiwhLSQrSlZKqIOPZkgGabmUQ0P/dy3 DxHGeUjURr3cldakKYDko+WImpaQBA3XK2p6Az8MY4SenwOc0rM8yqUU/53Y7dnsG4R3 WkzA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=86r827DMVsHVbXybV4Sx+Vo2C+huDUB6U6+yt/f+h1k=; b=nIhnTXLPXMilSrjiqbDEqVZa4jyNzapo7Xas8n8JEml+DxwTDwMy6xa/x10phompvE s+DesIy2mQI2tu6pytXM9Qffw2AB7km+8/I0ThLytQfyvedm0qRCF2PH4TxuRgLOHxpt DYCwWbZs47V3iyrvP6R+LoD1zn3DbvmlVK9ed9LBjtjN3xDVFnAiDx1C56EbPTR2Fhgt jYllp2qJSGq+OS2JMrFjwfiu/3KjGTTXPvpfv8Gsisga82LmcoO1RHrr5R+S/UZd7vTm dy5uvXrdRetuXLaBXKwRk93PFtoBNJUohTFdo6l0ogquJcaypXRn4LI+Z+zghvlFRZ89 WL+A== X-Gm-Message-State: APjAAAWzTqdsAxjyQnXvuuYVZBZDrwfsqtiGpLKy0ZoacwurNJXToZ8E /DYMf4OwAJjcglkBfF6VR3VFZ1U7kVtpu96D4PwiKWlmaG0= X-Google-Smtp-Source: APXvYqwL5vdsy3bqHgdudI/2vV3t/unO943yUb1Eae1tOEfN13w14niBtxrVvQyIDXciZXc3uMuSmdvJUcZ3vtsFtLc= X-Received: by 2002:a7b:c5ca:: with SMTP id n10mr3714294wmk.138.1568260600175; Wed, 11 Sep 2019 20:56:40 -0700 (PDT) MIME-Version: 1.0 References: <29ed38c7-5835-b354-368a-f9ebaddc9a7d@FreeBSD.org> In-Reply-To: <29ed38c7-5835-b354-368a-f9ebaddc9a7d@FreeBSD.org> From: Anish Date: Wed, 11 Sep 2019 20:56:28 -0700 Message-ID: Subject: Re: ixv + PCBGROUP + RSS: problem establishing an outgoing TCP connection To: Andriy Gapon Cc: "freebsd-net@freebsd.org" X-Rspamd-Queue-Id: 46TQ1j4xVrz3Gpy X-Spamd-Bar: ----- Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-6.00 / 15.00]; NEURAL_HAM_MEDIUM(-1.00)[-0.999,0]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; REPLY(-4.00)[] Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 12 Sep 2019 03:56:41 -0000 >if_ixv.c does not include opt_rss.h. Because of this IXGBE_FEATURE_RSS gets defined to zero (in ixgbe_features.h). So, instead of of using rss_getkey() to get the RSS key, the driver just generates a random one. I have not looked at ixv but as far as I understand our build process, opt_rss.h need to be included. -Anish On Tue, Sep 10, 2019 at 11:16 PM Andriy Gapon wrote: > On 10/09/2019 12:14, Andriy Gapon wrote: > > > > This happens on an EC2 instance with ixv driver. > > I wonder if anyone ever tested ixv with PCBGROUP... > I see a trivial but severe bug. > if_ixv.c does not include opt_rss.h. Because of this IXGBE_FEATURE_RSS > gets > defined to zero (in ixgbe_features.h). So, instead of of using > rss_getkey() to > get the RSS key, the driver just generates a random one. > No surprise then that the hardware (VF) produces totally different hashes= . > But maybe that's not all. > > On top of that, I wonder why the driver enables RSS in the hardware when > feat_en > does not have IXGBE_FEATURE_RSS. > Could anyone please explain the logic behind that? > Please see ixv_initialize_rss_mapping. > For example: > if (adapter->feat_en & IXGBE_FEATURE_RSS) { > /* Fetch the configured RSS key */ > rss_getkey((uint8_t *)&rss_key); > } else { > /* set up random bits */ > arc4rand(&rss_key, sizeof(rss_key), 0); > } > And so on. > > Additionally, I found this bit of information: > The limitation for VF RSS on Intel=C2=AE 82599 10 Gigabit Ethernet Contro= ller > is: The > hash and key are shared among PF and all VF, the RETA table with 128 > entries is > also shared among PF and all VF; So it could not to provide a method to > query > the hash and reta content per VF on guest, while, if possible, please > query them > on host for the shared RETA information. > > And my "hardware" is exactly 82599 VF. > I hacked the driver to not call ixv_initialize_rss_mapping() at all, but > even > with that change the packet descriptors had IXGBE_RXDADV_RSSTYPE_IPV4_TCP > in > pkt_info. Maybe it's because of how PF was configured. > So, I wonder if ixgbe_isc_rxd_pkt_get() should be modified to not set > iri_flowid > and iri_rsstype under some conditions. > > > When I try to establish an outgoing TCP connection I see the following > exchange. > > Local side sends SYN, it receives SYN+ACK and immediately sends RST. > > I tracked this down to in_pcblookup_mbuf() failing to find the > corresponding inpcb. > > > > I dug a bit deeper and this is my understanding of the issue. > > > > When tcp_connect() calls in_pcbrehash() the inpcb gets placed into a > group > > determined by in_pcbgroup_bytuple() [see in_pcbgroup_update and > > in_pcbgroup_byinpcb]. The inpcb does not have INP_RSS_BUCKET_SET. Bot= h > > addresses and ports are populated at that time. > > > > When the reply packet is received, in_pcblookup_mbuf() uses > in_pcbgroup_byhash() > > to look up the group because the packet has M_HASHTYPE_RSS_TCP_IPV4. > > The problem is that in_pcbgroup_byhash() returns a different group and > the inpcb > > cannot be found. > > > > I am very new to this code, so I would appreciate any help with further > > debugging and root causing the problem. > > > > > -- > Andriy Gapon > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org" >