From owner-freebsd-net@freebsd.org Wed Sep 11 06:16:05 2019 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 2E57DF0183 for ; Wed, 11 Sep 2019 06:16:05 +0000 (UTC) (envelope-from agapon@gmail.com) Received: from mail-lf1-f43.google.com (mail-lf1-f43.google.com [209.85.167.43]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) server-signature RSA-PSS (4096 bits) client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1O1" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 46Ss901qp8z4TtD for ; Wed, 11 Sep 2019 06:16:03 +0000 (UTC) (envelope-from agapon@gmail.com) Received: by mail-lf1-f43.google.com with SMTP id j4so15439983lfh.8 for ; Tue, 10 Sep 2019 23:16:03 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:references:openpgp:autocrypt :message-id:date:user-agent:mime-version:in-reply-to :content-language:content-transfer-encoding; bh=osmcxkFJuRiTceGL08X7+vVc2up3+Onegfd96PiCXuI=; b=W9BDB/be5oOyIECOG2WZYp7GzJihIY1TcmyZ651c9WEEOehEun2xG59ZZKzeZLEin1 G+inLStnYYp3/KoHLj4n7+vmcExvKW+2YBpyshlp0t7JRo95dr+z4VBSTOLXpms1zb++ piyy/0xuhLdlV0N4z3R445Sf1U5/kmVu9sy8/NjSSkOyiOaMX2MhAuLVlaO7NriLzVOm NSHb4yo8t3JUc0dVATxLEBitxsot7JYKqeZiKUMaftpkEu4QTiiZ5/grRoK9tRhfRDKu FHGxXQiRn82yAyCdFyAZJBtHzP6wSSzMuDuAvqt3bQwdM3SwCRmdDwFt1jlcPpub9U6t UhIA== X-Gm-Message-State: APjAAAXvxjylfG4SFQYGvCLuvPZbVDdd8F918ztKlX/5xL/zg4NN1aV/ yRdbVbDvaas/og8f+hY/XtGIKVTU X-Google-Smtp-Source: APXvYqwniSKtNo5hTVJpc1Cjmmak0qREoZvUjuuXU4uVtkh/Gc+Z/dPY5vtgVrHFe0DopM6enaJQAg== X-Received: by 2002:a05:6512:c3:: with SMTP id c3mr22344293lfp.190.1568182561959; Tue, 10 Sep 2019 23:16:01 -0700 (PDT) Received: from [192.168.0.88] (east.meadow.volia.net. [93.72.151.96]) by smtp.googlemail.com with ESMTPSA id t6sm4448906lji.3.2019.09.10.23.16.00 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Tue, 10 Sep 2019 23:16:01 -0700 (PDT) Subject: ixv + PCBGROUP + RSS: problem establishing an outgoing TCP connection From: Andriy Gapon To: "freebsd-net@freebsd.org" References: Openpgp: preference=signencrypt Autocrypt: addr=avg@FreeBSD.org; prefer-encrypt=mutual; keydata= mQINBFm4LIgBEADNB/3lT7f15UKeQ52xCFQx/GqHkSxEdVyLFZTmY3KyNPQGBtyvVyBfprJ7 mAeXZWfhat6cKNRAGZcL5EmewdQuUfQfBdYmKjbw3a9GFDsDNuhDA2QwFt8BmkiVMRYyvI7l N0eVzszWCUgdc3qqM6qqcgBaqsVmJluwpvwp4ZBXmch5BgDDDb1MPO8AZ2QZfIQmplkj8Y6Z AiNMknkmgaekIINSJX8IzRzKD5WwMsin70psE8dpL/iBsA2cpJGzWMObVTtCxeDKlBCNqM1i gTXta1ukdUT7JgLEFZk9ceYQQMJJtUwzWu1UHfZn0Fs29HTqawfWPSZVbulbrnu5q55R4PlQ /xURkWQUTyDpqUvb4JK371zhepXiXDwrrpnyyZABm3SFLkk2bHlheeKU6Yql4pcmSVym1AS4 dV8y0oHAfdlSCF6tpOPf2+K9nW1CFA8b/tw4oJBTtfZ1kxXOMdyZU5fiG7xb1qDgpQKgHUX8 7Rd2T1UVLVeuhYlXNw2F+a2ucY+cMoqz3LtpksUiBppJhw099gEXehcN2JbUZ2TueJdt1FdS ztnZmsHUXLxrRBtGwqnFL7GSd6snpGIKuuL305iaOGODbb9c7ne1JqBbkw1wh8ci6vvwGlzx rexzimRaBzJxlkjNfMx8WpCvYebGMydNoeEtkWldtjTNVsUAtQARAQABtB5BbmRyaXkgR2Fw b24gPGF2Z0BGcmVlQlNELm9yZz6JAlQEEwEIAD4WIQS+LEO7ngQnXA4Bjr538m7TUc1yjwUC WbgsiAIbIwUJBaOagAULCQgHAgYVCAkKCwIEFgIDAQIeAQIXgAAKCRB38m7TUc1yj+JAEACV l9AK/nOWAt/9cufV2fRj0hdOqB1aCshtSrwHk/exXsDa4/FkmegxXQGY+3GWX3deIyesbVRL rYdtdK0dqJyT1SBqXK1h3/at9rxr9GQA6KWOxTjUFURsU7ok/6SIlm8uLRPNKO+yq0GDjgaO LzN+xykuBA0FlhQAXJnpZLcVfPJdWv7sSHGedL5ln8P8rxR+XnmsA5TUaaPcbhTB+mG+iKFj GghASDSfGqLWFPBlX/fpXikBDZ1gvOr8nyMY9nXhgfXpq3B6QCRYKPy58ChrZ5weeJZ29b7/ QdEO8NFNWHjSD9meiLdWQaqo9Y7uUxN3wySc/YUZxtS0bhAd8zJdNPsJYG8sXgKjeBQMVGuT eCAJFEYJqbwWvIXMfVWop4+O4xB+z2YE3jAbG/9tB/GSnQdVSj3G8MS80iLS58frnt+RSEw/ psahrfh0dh6SFHttE049xYiC+cM8J27Aaf0i9RflyITq57NuJm+AHJoU9SQUkIF0nc6lfA+o JRiyRlHZHKoRQkIg4aiKaZSWjQYRl5Txl0IZUP1dSWMX4s3XTMurC/pnja45dge/4ESOtJ9R 8XuIWg45Oq6MeIWdjKddGhRj3OohsltKgkEU3eLKYtB6qRTQypHHUawCXz88uYt5e3w4V16H lCpSTZV/EVHnNe45FVBlvK7k7HFfDDkryLkCDQRZuCyIARAAlq0slcsVboY/+IUJdcbEiJRW be9HKVz4SUchq0z9MZPX/0dcnvz/gkyYA+OuM78dNS7Mbby5dTvOqfpLJfCuhaNYOhlE0wY+ 1T6Tf1f4c/uA3U/YiadukQ3+6TJuYGAdRZD5EqYFIkreARTVWg87N9g0fT9BEqLw9lJtEGDY EWUE7L++B8o4uu3LQFEYxcrb4K/WKmgtmFcm77s0IKDrfcX4doV92QTIpLiRxcOmCC/OCYuO jB1oaaqXQzZrCutXRK0L5XN1Y1PYjIrEzHMIXmCDlLYnpFkK+itlXwlE2ZQxkfMruCWdQXye syl2fynAe8hvp7Mms9qU2r2K9EcJiR5N1t1C2/kTKNUhcRv7Yd/vwusK7BqJbhlng5ZgRx0m WxdntU/JLEntz3QBsBsWM9Y9wf2V4tLv6/DuDBta781RsCB/UrU2zNuOEkSixlUiHxw1dccI 6CVlaWkkJBxmHX22GdDFrcjvwMNIbbyfQLuBq6IOh8nvu9vuItup7qemDG3Ms6TVwA7BD3j+ 3fGprtyW8Fd/RR2bW2+LWkMrqHffAr6Y6V3h5kd2G9Q8ZWpEJk+LG6Mk3fhZhmCnHhDu6CwN MeUvxXDVO+fqc3JjFm5OxhmfVeJKrbCEUJyM8ESWLoNHLqjywdZga4Q7P12g8DUQ1mRxYg/L HgZY3zfKOqcAEQEAAYkCPAQYAQgAJhYhBL4sQ7ueBCdcDgGOvnfybtNRzXKPBQJZuCyIAhsM BQkFo5qAAAoJEHfybtNRzXKPBVwQAKfFy9P7N3OsLDMB56A4Kf+ZT+d5cIx0Yiaf4n6w7m3i ImHHHk9FIetI4Xe54a2IXh4Bq5UkAGY0667eIs+Z1Ea6I2i27Sdo7DxGwq09Qnm/Y65ADvXs 3aBvokCcm7FsM1wky395m8xUos1681oV5oxgqeRI8/76qy0hD9WR65UW+HQgZRIcIjSel9vR XDaD2HLGPTTGr7u4v00UeTMs6qvPsa2PJagogrKY8RXdFtXvweQFz78NbXhluwix2Tb9ETPk LIpDrtzV73CaE2aqBG/KrboXT2C67BgFtnk7T7Y7iKq4/XvEdDWscz2wws91BOXuMMd4c/c4 OmGW9m3RBLufFrOag1q5yUS9QbFfyqL6dftJP3Zq/xe+mr7sbWbhPVCQFrH3r26mpmy841ym dwQnNcsbIGiBASBSKksOvIDYKa2Wy8htPmWFTEOPRpFXdGQ27awcjjnB42nngyCK5ukZDHi6 w0qK5DNQQCkiweevCIC6wc3p67jl1EMFY5+z+zdTPb3h7LeVnGqW0qBQl99vVFgzLxchKcl0 R/paSFgwqXCZhAKMuUHncJuynDOP7z5LirUeFI8qsBAJi1rXpQoLJTVcW72swZ42IdPiboqx NbTMiNOiE36GqMcTPfKylCbF45JNX4nF9ElM0E+Y8gi4cizJYBRr2FBJgay0b9Cp Message-ID: <29ed38c7-5835-b354-368a-f9ebaddc9a7d@FreeBSD.org> Date: Wed, 11 Sep 2019 09:15:59 +0300 User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 46Ss901qp8z4TtD X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of agapon@gmail.com designates 209.85.167.43 as permitted sender) smtp.mailfrom=agapon@gmail.com X-Spamd-Result: default: False [-3.21 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; RCVD_COUNT_THREE(0.00)[3]; FORGED_SENDER(0.30)[avg@FreeBSD.org,agapon@gmail.com]; RECEIVED_SPAMHAUS_PBL(0.00)[96.151.72.93.khpj7ygk5idzvmvt5x4ziurxhy.zen.dq.spamhaus.net : 127.0.0.10]; MIME_TRACE(0.00)[0:+]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; FROM_NEQ_ENVFROM(0.00)[avg@FreeBSD.org,agapon@gmail.com]; TO_DOM_EQ_FROM_DOM(0.00)[]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[FreeBSD.org]; RCPT_COUNT_ONE(0.00)[1]; IP_SCORE(-1.21)[ip: (-0.43), ipnet: 209.85.128.0/17(-3.32), asn: 15169(-2.26), country: US(-0.05)]; RCVD_IN_DNSWL_NONE(0.00)[43.167.85.209.list.dnswl.org : 127.0.5.0]; TO_DN_EQ_ADDR_ALL(0.00)[]; RWL_MAILSPIKE_POSSIBLE(0.00)[43.167.85.209.rep.mailspike.net : 127.0.0.17]; RCVD_TLS_ALL(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 11 Sep 2019 06:16:05 -0000 On 10/09/2019 12:14, Andriy Gapon wrote: > > This happens on an EC2 instance with ixv driver. I wonder if anyone ever tested ixv with PCBGROUP... I see a trivial but severe bug. if_ixv.c does not include opt_rss.h. Because of this IXGBE_FEATURE_RSS gets defined to zero (in ixgbe_features.h). So, instead of of using rss_getkey() to get the RSS key, the driver just generates a random one. No surprise then that the hardware (VF) produces totally different hashes. But maybe that's not all. On top of that, I wonder why the driver enables RSS in the hardware when feat_en does not have IXGBE_FEATURE_RSS. Could anyone please explain the logic behind that? Please see ixv_initialize_rss_mapping. For example: if (adapter->feat_en & IXGBE_FEATURE_RSS) { /* Fetch the configured RSS key */ rss_getkey((uint8_t *)&rss_key); } else { /* set up random bits */ arc4rand(&rss_key, sizeof(rss_key), 0); } And so on. Additionally, I found this bit of information: The limitation for VF RSS on Intel® 82599 10 Gigabit Ethernet Controller is: The hash and key are shared among PF and all VF, the RETA table with 128 entries is also shared among PF and all VF; So it could not to provide a method to query the hash and reta content per VF on guest, while, if possible, please query them on host for the shared RETA information. And my "hardware" is exactly 82599 VF. I hacked the driver to not call ixv_initialize_rss_mapping() at all, but even with that change the packet descriptors had IXGBE_RXDADV_RSSTYPE_IPV4_TCP in pkt_info. Maybe it's because of how PF was configured. So, I wonder if ixgbe_isc_rxd_pkt_get() should be modified to not set iri_flowid and iri_rsstype under some conditions. > When I try to establish an outgoing TCP connection I see the following exchange. > Local side sends SYN, it receives SYN+ACK and immediately sends RST. > I tracked this down to in_pcblookup_mbuf() failing to find the corresponding inpcb. > > I dug a bit deeper and this is my understanding of the issue. > > When tcp_connect() calls in_pcbrehash() the inpcb gets placed into a group > determined by in_pcbgroup_bytuple() [see in_pcbgroup_update and > in_pcbgroup_byinpcb]. The inpcb does not have INP_RSS_BUCKET_SET. Both > addresses and ports are populated at that time. > > When the reply packet is received, in_pcblookup_mbuf() uses in_pcbgroup_byhash() > to look up the group because the packet has M_HASHTYPE_RSS_TCP_IPV4. > The problem is that in_pcbgroup_byhash() returns a different group and the inpcb > cannot be found. > > I am very new to this code, so I would appreciate any help with further > debugging and root causing the problem. -- Andriy Gapon