From owner-freebsd-transport@freebsd.org Tue May 5 00:22:33 2020 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 622BB2C84CF for ; Tue, 5 May 2020 00:22:33 +0000 (UTC) (envelope-from paul.reynolds@redcom.com) Received: from smtp1.redcom.com (smtp1.redcom.com [192.86.3.143]) by mx1.freebsd.org (Postfix) with ESMTP id 49GL5h4Qdhz42Cp for ; Tue, 5 May 2020 00:22:32 +0000 (UTC) (envelope-from paul.reynolds@redcom.com) Received: from localhost (localhost [127.0.0.1]) by smtp1.redcom.com (Postfix) with ESMTP id 14C4FA02F for ; Mon, 4 May 2020 20:22:26 -0400 (EDT) X-Virus-Scanned: amavisd-new at redcom.com Received: from smtp1.redcom.com ([127.0.0.1]) by localhost (smtp1.redcom.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zvysGfmu8nQR for ; Mon, 4 May 2020 20:22:23 -0400 (EDT) Received: from pie.redcom.com (pie [192.168.33.15]) by smtp1.redcom.com (Postfix) with ESMTP id 97BB3A02A for ; Mon, 4 May 2020 20:22:23 -0400 (EDT) Received: from exch-03.redcom.com (exch-03.redcom.com [192.168.32.32]) by pie.redcom.com (8.11.7p1+Sun/8.10.2) with ESMTP id 0450MNl20914 for ; Mon, 4 May 2020 20:22:23 -0400 (EDT) Received: from exch-03.redcom.com (fd00::8549:68c0:3d5f:ee62) by exch-03.redcom.com (fd00::8549:68c0:3d5f:ee62) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.330.5; Mon, 4 May 2020 20:22:23 -0400 Received: from exch-03.redcom.com ([fe80::a442:ce34:c9c8:268f]) by exch-03.redcom.com ([fe80::a442:ce34:c9c8:268f%3]) with mapi id 15.02.0330.010; Mon, 4 May 2020 20:22:23 -0400 From: "Reynolds, Paul" To: "freebsd-transport@freebsd.org" Subject: SCTP deadlock Thread-Topic: SCTP deadlock Thread-Index: AQHWInEPIgANzT+FZEW5ABoFd20+WA== Date: Tue, 5 May 2020 00:22:23 +0000 Message-ID: <112525e87fce467e97f1d455ef9bf685@redcom.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.224.164] MIME-Version: 1.0 X-Rspamd-Queue-Id: 49GL5h4Qdhz42Cp X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of paul.reynolds@redcom.com designates 192.86.3.143 as permitted sender) smtp.mailfrom=paul.reynolds@redcom.com X-Spamd-Result: default: False [-3.27 / 15.00]; ARC_NA(0.00)[]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:192.86.3.143/32]; TO_MATCH_ENVRCPT_ALL(0.00)[]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; PREVIOUSLY_DELIVERED(0.00)[freebsd-transport@freebsd.org]; DMARC_NA(0.00)[redcom.com]; RCPT_COUNT_ONE(0.00)[1]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; IP_SCORE(-1.07)[ip: (-2.80), ipnet: 192.86.3.0/24(-1.40), asn: 46679(-1.12), country: US(-0.05)]; TO_DN_EQ_ADDR_ALL(0.00)[]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; ASN(0.00)[asn:46679, ipnet:192.86.3.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_SEVEN(0.00)[7] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.29 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2020 00:22:33 -0000 Hi, My apologies if this is not the right mailing list for this question. I am = hoping to get some suggestions on how to debug an sctp issue I have been ha= ving. I have a set of programs that use sctp for communication. For now these pro= cesses are all running on the same machine. They are using SEQPACKET mode t= o send messages back and forth. Each process has multiple threads, but only= one socket. The threads are mutex protected such that the socket cannot be= accessed by more than one thread at a time. Very occasionally one of the s= ockets will become permanently blocked on an sctp send or receive call and = I am trying to figure out why. In the cases where it has become blocked on = a send call, the corresponding receive process is not blocked and can send/= receive data to other destinations. These processes can run fine for months= and then suddenly run into this problem. I have been able to reproduce thi= s once or twice by subjecting the system to unrealistically high levels of = traffic, but it still takes several days or more to reproduce the problem. = I now have a system that is stuck in this mode and am trying to gather as m= uch information as possible. How should I go about debugging this? The output of sockstat and netstat ha= ve not been very helpful up to this point. Thanks for any help you might be able to provide, Paul From owner-freebsd-transport@freebsd.org Tue May 5 06:42:19 2020 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 8AA0D2D9EB3 for ; Tue, 5 May 2020 06:42:19 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from drew.franken.de (mail-n.franken.de [193.175.24.27]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "COMODO RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 49GVWt726sz4Q2F for ; Tue, 5 May 2020 06:42:18 +0000 (UTC) (envelope-from tuexen@freebsd.org) Received: from mbp.fritz.box (ip4d16e760.dynamic.kabel-deutschland.de [77.22.231.96]) (Authenticated sender: macmic) by mail-n.franken.de (Postfix) with ESMTPSA id 632337220CCF5; Tue, 5 May 2020 08:42:15 +0200 (CEST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.80.23.2.2\)) Subject: Re: SCTP deadlock From: Michael Tuexen In-Reply-To: <112525e87fce467e97f1d455ef9bf685@redcom.com> Date: Tue, 5 May 2020 08:42:14 +0200 Cc: "freebsd-transport@freebsd.org" Content-Transfer-Encoding: quoted-printable Message-Id: <7B548240-AF6D-4F40-B506-9B83052EC7CE@freebsd.org> References: <112525e87fce467e97f1d455ef9bf685@redcom.com> To: "Reynolds, Paul" X-Mailer: Apple Mail (2.3608.80.23.2.2) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Queue-Id: 49GVWt726sz4Q2F X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.87 / 15.00]; local_wl_from(0.00)[freebsd.org]; NEURAL_HAM_MEDIUM(-0.90)[-0.900,0]; ASN(0.00)[asn:680, ipnet:193.174.0.0/15, country:DE]; NEURAL_HAM_LONG(-0.97)[-0.972,0] X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2020 06:42:19 -0000 > On 5. May 2020, at 02:22, Reynolds, Paul = wrote: >=20 > Hi, >=20 >=20 > My apologies if this is not the right mailing list for this question. = I am hoping to get some suggestions on how to debug an sctp issue I have = been having. >=20 >=20 > I have a set of programs that use sctp for communication. For now = these processes are all running on the same machine. They are using = SEQPACKET mode to send messages back and forth. Each process has = multiple threads, but only one socket. The threads are mutex protected = such that the socket cannot be accessed by more than one thread at a = time. Very occasionally one of the sockets will become permanently = blocked on an sctp send or receive call and I am trying to figure out = why. In the cases where it has become blocked on a send call, the = corresponding receive process is not blocked and can send/receive data = to other destinations. These processes can run fine for months and then = suddenly run into this problem. I have been able to reproduce this once = or twice by subjecting the system to unrealistically high levels of = traffic, but it still takes several days or more to reproduce the = problem. I now have a system that is stuck in this mode and am trying to = gather as much information > as poss > ible. >=20 >=20 > How should I go about debugging this? The output of sockstat and = netstat have not been very helpful up to this point. Which version of FreeBSD are you using? Best regards Michael >=20 >=20 > Thanks for any help you might be able to provide, >=20 > Paul > _______________________________________________ > freebsd-transport@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-transport > To unsubscribe, send any mail to = "freebsd-transport-unsubscribe@freebsd.org" From owner-freebsd-transport@freebsd.org Tue May 5 11:42:54 2020 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C9D162E30A6 for ; Tue, 5 May 2020 11:42:54 +0000 (UTC) (envelope-from paul.reynolds@redcom.com) Received: from smtp1.redcom.com (smtp1.redcom.com [192.86.3.143]) by mx1.freebsd.org (Postfix) with ESMTP id 49GdBj58TSz3Lrw for ; Tue, 5 May 2020 11:42:53 +0000 (UTC) (envelope-from paul.reynolds@redcom.com) Received: from localhost (localhost [127.0.0.1]) by smtp1.redcom.com (Postfix) with ESMTP id 97C45A02F; Tue, 5 May 2020 07:42:52 -0400 (EDT) X-Virus-Scanned: amavisd-new at redcom.com Received: from smtp1.redcom.com ([127.0.0.1]) by localhost (smtp1.redcom.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id kAxxhsYV0auO; Tue, 5 May 2020 07:42:50 -0400 (EDT) Received: from pie.redcom.com (pie [192.168.33.15]) by smtp1.redcom.com (Postfix) with ESMTP id 53D379F78; Tue, 5 May 2020 07:42:50 -0400 (EDT) Received: from exch-03.redcom.com (exch-03.redcom.com [192.168.32.32]) by pie.redcom.com (8.11.7p1+Sun/8.10.2) with ESMTP id 045Bgnl01555; Tue, 5 May 2020 07:42:50 -0400 (EDT) Received: from exch-03.redcom.com (fd00::8549:68c0:3d5f:ee62) by exch-03.redcom.com (fd00::8549:68c0:3d5f:ee62) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.330.5; Tue, 5 May 2020 07:42:49 -0400 Received: from exch-03.redcom.com ([fe80::a442:ce34:c9c8:268f]) by exch-03.redcom.com ([fe80::a442:ce34:c9c8:268f%3]) with mapi id 15.02.0330.010; Tue, 5 May 2020 07:42:49 -0400 From: "Reynolds, Paul" To: Michael Tuexen CC: "freebsd-transport@freebsd.org" Subject: RE: SCTP deadlock Thread-Topic: SCTP deadlock Thread-Index: AQHWInEPIgANzT+FZEW5ABoFd20+WKiZTQEAgAASHSA= Date: Tue, 5 May 2020 11:42:49 +0000 Message-ID: References: <112525e87fce467e97f1d455ef9bf685@redcom.com> In-Reply-To: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [192.168.84.5] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Rspamd-Queue-Id: 49GdBj58TSz3Lrw X-Spamd-Bar: --- Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=pass (mx1.freebsd.org: domain of paul.reynolds@redcom.com designates 192.86.3.143 as permitted sender) smtp.mailfrom=paul.reynolds@redcom.com X-Spamd-Result: default: False [-3.37 / 15.00]; ARC_NA(0.00)[]; TO_DN_EQ_ADDR_SOME(0.00)[]; HAS_XOIP(0.00)[]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; R_SPF_ALLOW(-0.20)[+ip4:192.86.3.143/32]; NEURAL_HAM_LONG(-1.00)[-1.000,0]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[redcom.com]; NEURAL_HAM_MEDIUM(-1.00)[-1.000,0]; TO_MATCH_ENVRCPT_SOME(0.00)[]; RCPT_COUNT_TWO(0.00)[2]; IP_SCORE(-1.17)[ip: (-3.05), ipnet: 192.86.3.0/24(-1.53), asn: 46679(-1.22), country: US(-0.05)]; RCVD_NO_TLS_LAST(0.10)[]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:46679, ipnet:192.86.3.0/24, country:US]; MID_RHS_MATCH_FROM(0.00)[]; RCVD_COUNT_SEVEN(0.00)[7] X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.30 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 05 May 2020 11:42:54 -0000 > > I have a set of programs that use sctp for communication. For now these= processes are all running on the same machine. They are using SEQPACKET mo= de to send messages back and forth. Each process has multiple threads, but = only one socket. The threads are mutex protected such that the socket canno= t be accessed by more than one thread at a time. Very occasionally one of t= he sockets will become permanently blocked on an sctp send or receive call = and I am trying to figure out why. In the cases where it has become blocked= on a send call, the corresponding receive process is not blocked and can s= end/receive data to other destinations. These processes can run fine for mo= nths and then suddenly run into this problem. I have been able to reproduce= this once or twice by subjecting the system to unrealistically high levels= of traffic, but it still takes several days or more to reproduce the probl= em. I now have a system that is stuck in this mode and am trying to gather = as much information > > as possible. > >=20 > > How should I go about debugging this? The output of sockstat and netsta= t have not been very helpful up to this point. > Which version of FreeBSD are you using? >=20 I am using FreeBSD 11.3p8. Thanks, Paul From owner-freebsd-transport@freebsd.org Sat May 9 15:44:58 2020 Return-Path: Delivered-To: freebsd-transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id DF9052ED9F9 for ; Sat, 9 May 2020 15:44:58 +0000 (UTC) (envelope-from karels@FreeBSD.org) Received: from mailman.nyi.freebsd.org (mailman.nyi.freebsd.org [IPv6:2610:1c1:1:606c::50:13]) by mx1.freebsd.org (Postfix) with ESMTP id 49KBNB5fT9z40sZ for ; Sat, 9 May 2020 15:44:58 +0000 (UTC) (envelope-from karels@FreeBSD.org) Received: by mailman.nyi.freebsd.org (Postfix) id C1D362ED9F8; Sat, 9 May 2020 15:44:58 +0000 (UTC) Delivered-To: transport@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id C19822ED9F7 for ; Sat, 9 May 2020 15:44:58 +0000 (UTC) (envelope-from karels@FreeBSD.org) Received: from mail.karels.net (mail.karels.net [216.160.39.52]) by mx1.freebsd.org (Postfix) with ESMTP id 49KBNB2N6nz40sY for ; Sat, 9 May 2020 15:44:58 +0000 (UTC) (envelope-from karels@FreeBSD.org) Received: from mail.karels.net (localhost [127.0.0.1]) by mail.karels.net (8.15.2/8.15.2) with ESMTP id 049FiqCf018860 for ; Sat, 9 May 2020 10:44:52 -0500 (CDT) (envelope-from karels@FreeBSD.org) Message-Id: <202005091544.049FiqCf018860@mail.karels.net> To: transport@freebsd.org From: Mike Karels Reply-to: karels@FreeBSD.org Subject: review for outgoing TCP port reuse MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-ID: <18858.1589039092.1@mail.karels.net> Date: Sat, 09 May 2020 10:44:52 -0500 X-Rspamd-Queue-Id: 49KBNB2N6nz40sY X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; none X-Spamd-Result: default: False [-1.74 / 15.00]; local_wl_from(0.00)[FreeBSD.org]; NEURAL_HAM_MEDIUM(-0.87)[-0.868,0]; NEURAL_HAM_LONG(-0.88)[-0.876,0]; ASN(0.00)[asn:209, ipnet:216.160.36.0/22, country:US] X-Mailman-Approved-At: Sat, 09 May 2020 16:01:54 +0000 X-BeenThere: freebsd-transport@freebsd.org X-Mailman-Version: 2.1.32 Precedence: list List-Id: Discussions of transport level network protocols in FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sat, 09 May 2020 15:44:58 -0000 I just created a review with the following description: Allow TCP to reuse local port with different destinations Summary: Previously, tcp_connect() would bind a local port before connecting, forcing the local port to be unique across all outgoing TCP connections for the address family. Instead, choose a local port after selecting the destination and the local address, requiring only that the tuple is unique. The review is https://reviews.freebsd.org/D24781. The change originated on the Sidewinder firewall, which is proxy-based as well as packet-filter based, and needed to support more proxy connections than the pool of ephemeral ports as long as the destinations were different. Mike