From owner-freebsd-net@freebsd.org Wed Apr 21 16:35:08 2021 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8B3625EB387 for ; Wed, 21 Apr 2021 16:35:08 +0000 (UTC) (envelope-from gfoster411@charter.net) Received: from impout003.msg.chrl.nc.charter.net (impout003aa.msg.chrl.nc.charter.net [47.43.20.27]) (using TLSv1.2 with cipher DHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "Client", Issuer "CA" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4FQR3v5Zxwz3mBM for ; Wed, 21 Apr 2021 16:35:06 +0000 (UTC) (envelope-from gfoster411@charter.net) Received: from DESKTOP2VS42MF ([35.132.91.103]) by cmsmtp with ESMTPA id ZFoelMAT9la4bZFoelkT3K; Wed, 21 Apr 2021 16:35:05 +0000 X-Authority-Analysis: v=2.3 cv=ZJpjZkzb c=1 sm=1 tr=0 a=j47tmOq8Jh9JtlcCTmVdTQ==:117 a=j47tmOq8Jh9JtlcCTmVdTQ==:17 a=DAwyPP_o2Byb1YXLmDAA:9 a=rb6oIRTCZ6BDaNaZIB0A:9 a=CjuIK1q_8ugA:10 a=yMhMjlubAAAA:8 a=SSmOFEACAAAA:8 a=sqdEFVgroVDR__41VroA:9 a=EfJZ2cIqxOWw2Wva:21 a=gKO2Hq4RSVkA:10 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 a=pHzHmUro8NiASowvMSCR:22 a=nt3jZW36AmriUCFCBwmW:22 From: To: Subject: Src IP 0.0.0.0 for outgoing off-net ping & SSH packets Date: Wed, 21 Apr 2021 09:35:04 -0700 Message-ID: <067601d736cc$49402d90$dbc088b0$@charter.net> MIME-Version: 1.0 X-Mailer: Microsoft Outlook 16.0 Thread-Index: Adc2xOQE3uOMWh5FRwGnrE8rL7/2gA== Content-Language: en-us X-CMAE-Envelope: MS4wfMAECuYK0hSW4OzND1VHp8YdqjcKs15D9h7+aF/qNCZFRWDZEkwrCx0t8gDNlqRL3NjZGiulgEtDZZ3srt6RV13MWNnvLOC1hwY+A8wHTKcRGc0bpyk2 tcnTAaf8iCxaRBHAafIQh2CUAorlia1jtuQsmGjoPSQmMcV2mYMXryZgpJhr7bdOBkBLbx1oacqDBA== X-Rspamd-Queue-Id: 4FQR3v5Zxwz3mBM X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of gfoster411@charter.net designates 47.43.20.27 as permitted sender) smtp.mailfrom=gfoster411@charter.net X-Spamd-Result: default: False [-3.39 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; RBL_SENDERSCORE_FAIL(0.00)[47.43.20.27:server fail]; FREEMAIL_FROM(0.00)[charter.net]; R_SPF_ALLOW(-0.20)[+ip4:47.43.20.0/24]; TO_DN_NONE(0.00)[]; NEURAL_HAM_SHORT(-1.00)[-0.995]; RCVD_IN_DNSWL_LOW(-0.10)[47.43.20.27:from]; RCVD_TLS_LAST(0.00)[]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[charter.net]; ASN(0.00)[asn:40294, ipnet:47.43.16.0/21, country:US]; MID_RHS_MATCH_FROM(0.00)[]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RBL_DBL_DONT_QUERY_IPS(0.00)[47.43.20.27:from]; ARC_NA(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; NEURAL_HAM_LONG(-1.00)[-1.000]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; DMARC_NA(0.00)[charter.net]; NEURAL_HAM_MEDIUM(-1.00)[-1.000]; RCPT_COUNT_ONE(0.00)[1]; SPAMHAUS_ZRD(0.00)[47.43.20.27:from:127.0.2.255]; FROM_NO_DN(0.00)[]; RCVD_COUNT_TWO(0.00)[2]; MAILMAN_DEST(0.00)[freebsd-net] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.34 X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 21 Apr 2021 16:35:08 -0000 Freebsd-net, We are running FreeBSD 10.4 with multipath routing enabled (RADIX_MPATH) and are using just a single static route (10.18.91.0/255.255.255.0 10.17.118.3) when we infrequently run into the problem described below. The system is running fine with off-net clients (10.18.91.0/255.255.255.0) accessible. Then at some point we can no longer reach the off-net clients with ping and SSH failing. Interestingly, the off-net clients can successfully ping and SSH into our failing node. When the problem occurs we've determined our failing node is sending 0.0.0.0 as it's source IP address, which is why the outgoing pings and SSH fail. We have also found that if we remove the single static route and add it back, the problem is corrected. Is this a known issue that's been fixed in subsequent releases? I've been looking in the function ip_output() and see where it calls rtalloc_mpath_fib() to lookup the route to the destination (e.g., 10.18.91.10). and then later fills in the source IP "if available". There's a comment stating "/* Interface may have no addresses. */" and the code doesn't try to fill in the source IP and continues on without error, which goes along with what we've observed in the failure case. Thus, our problem seems to be in the actual routing code/structures, which I'm digging deeper into every day. Do you have any tips or specific areas of the routing code I should be looking into ? Thanks Greg