From owner-freebsd-net@freebsd.org Wed Sep 18 12:45:58 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 5956BFCAAD for ; Wed, 18 Sep 2019 12:45:58 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lj1-f170.google.com (mail-lj1-f170.google.com [209.85.208.170]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46YKTd2h6bz4Gsc for ; Wed, 18 Sep 2019 12:45:57 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lj1-f170.google.com with SMTP id m13so7020209ljj.11 for ; Wed, 18 Sep 2019 05:45:57 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=P1l4FaofJ7NBqnFzhErkLDRujRpeUXvODiFsTAek2Dk=; b=R5LLizIoEAcimerBP9aHOZr2JwM8i1baVzdgSs0xhsZSLHMSm21obP9L0NsSC7j3eZ 3HdB4eHMP9p5DXe+qp7Mf1dSohVeamMiVKoHSDrj6zcZUsnzJ4ZGuK4pv14pN+TjJ3f+ gX5UzkbByxAmWduxqlOH6fhyywY5tTwHzRvzWK88r6iCajKlvis5Rc06CZYh5TAjsm5M g3EWYobnFn3W3vpMP9mYnRTte3iZ5AT52jaCEJ+oGLkAbNGFP6EHzQSPp9Wuykqawogp 5B5YUx98YJF5/HRqWXpYbSHoCsFNK25Df807TyBS3o4ZlnuGXo68p3UUyuMee92hbF4K F3uQ== X-Gm-Message-State: APjAAAWSSAeaGqFnwXAc8DtL0hG+L2UX45WtkPwmkDoqrBh8BX65EUlj VaScgFn7eJ4EFrrjJz1mmxEIMcxt X-Google-Smtp-Source: APXvYqwYlqQCZs1a/zRWz6OT50pKeOTBbfesnRxeFICgEe+7x1wNvRtRtcIfmJlGSkaovfoi3Yf5xA== X-Received: by 2002:a05:651c:292:: with SMTP id b18mr2179114ljo.131.1568810755191; Wed, 18 Sep 2019 05:45:55 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id x76sm1185043ljb.81.2019.09.18.05.45.53 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 18 Sep 2019 05:45:53 -0700 (PDT) Subject: Re: ixv + PCBGROUP + RSS: problem establishing an outgoing TCP connection From: Andriy Gapon To: "freebsd-net@freebsd.org" References: <29ed38c7-5835-b354-368a-f9ebaddc9a7d@FreeBSD.org> Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <5196d3e2-5237-df5b-713c-1e0f84908238@FreeBSD.org> Date: Wed, 18 Sep 2019 15:45:52 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <29ed38c7-5835-b354-368a-f9ebaddc9a7d@FreeBSD.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 46YKTd2h6bz4Gsc X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.208.170 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.23 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; RCVD_COUNT_THREE(0.00)[3]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.23)[ip: (-0.53), ipnet: 209.85.128.0/17(-3.31), asn: 15169(-2.23), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[170.208.85.209.list.dnswl.org : 127.0.5.0]; TO_DN_EQ_ADDR_ALL(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[170.208.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 18 Sep 2019 12:45:58 -0000 Here is a change that I would like to propose based on the earlier observations and findings: https://reviews.freebsd.org/D21705 On 11/09/2019 09:15, Andriy Gapon wrote: > On 10/09/2019 12:14, Andriy Gapon wrote: >> >> This happens on an EC2 instance with ixv driver. > > I wonder if anyone ever tested ixv with PCBGROUP... > I see a trivial but severe bug. > if_ixv.c does not include opt_rss.h. Because of this IXGBE_FEATURE_RSS gets > defined to zero (in ixgbe_features.h). So, instead of of using rss_getkey() to > get the RSS key, the driver just generates a random one. > No surprise then that the hardware (VF) produces totally different hashes. > But maybe that's not all. > > On top of that, I wonder why the driver enables RSS in the hardware when feat_en > does not have IXGBE_FEATURE_RSS. > Could anyone please explain the logic behind that? > Please see ixv_initialize_rss_mapping. > For example: > if (adapter->feat_en & IXGBE_FEATURE_RSS) { > /* Fetch the configured RSS key */ > rss_getkey((uint8_t *)&rss_key); > } else { > /* set up random bits */ > arc4rand(&rss_key, sizeof(rss_key), 0); > } > And so on. > > Additionally, I found this bit of information: > The limitation for VF RSS on Intel® 82599 10 Gigabit Ethernet Controller is: The > hash and key are shared among PF and all VF, the RETA table with 128 entries is > also shared among PF and all VF; So it could not to provide a method to query > the hash and reta content per VF on guest, while, if possible, please query them > on host for the shared RETA information. > > And my "hardware" is exactly 82599 VF. > I hacked the driver to not call ixv_initialize_rss_mapping() at all, but even > with that change the packet descriptors had IXGBE_RXDADV_RSSTYPE_IPV4_TCP in > pkt_info. Maybe it's because of how PF was configured. > So, I wonder if ixgbe_isc_rxd_pkt_get() should be modified to not set iri_flowid > and iri_rsstype under some conditions. > >> When I try to establish an outgoing TCP connection I see the following exchange. >> Local side sends SYN, it receives SYN+ACK and immediately sends RST. >> I tracked this down to in_pcblookup_mbuf() failing to find the corresponding inpcb. >> >> I dug a bit deeper and this is my understanding of the issue. >> >> When tcp_connect() calls in_pcbrehash() the inpcb gets placed into a group >> determined by in_pcbgroup_bytuple() [see in_pcbgroup_update and >> in_pcbgroup_byinpcb]. The inpcb does not have INP_RSS_BUCKET_SET. Both >> addresses and ports are populated at that time. >> >> When the reply packet is received, in_pcblookup_mbuf() uses in_pcbgroup_byhash() >> to look up the group because the packet has M_HASHTYPE_RSS_TCP_IPV4. >> The problem is that in_pcbgroup_byhash() returns a different group and the inpcb >> cannot be found. >> >> I am very new to this code, so I would appreciate any help with further >> debugging and root causing the problem. > > > > -- Andriy Gapon