From owner-freebsd-net@freebsd.org Fri Jul 17 17:27:12 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 34C9D36A4CE for ; Fri, 17 Jul 2020 17:27:12 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from drew.franken.de (drew.ipv6.franken.de [IPv6:2001:638:a02:a001:20e:cff:fe4a:feaa]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "*.franken.de", Issuer "Sectigo RSA Domain Validation Secure Server CA" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B7dNG6TR3z4K6Z for ; Fri, 17 Jul 2020 17:27:10 +0000 (UTC) (envelope-from Michael.Tuexen@lurchi.franken.de) Received: from mb.fritz.box (ip4d15f5fc.dynamic.kabel-deutschland.de [77.21.245.252]) (Authenticated sender: lurchi) by mail-n.franken.de (Postfix) with ESMTPSA id D16E37220CCE1; Fri, 17 Jul 2020 19:27:03 +0200 (CEST) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Mac OS X Mail 13.4 \(3608.120.23.2.1\)) Subject: Re: SCTP problem, how to debug? From: Michael Tuexen In-Reply-To: <20200717160739.GA79604@cicely7.cicely.de> Date: Fri, 17 Jul 2020 19:27:00 +0200 Cc: freebsd-net@freebsd.org, Bernd Walter Content-Transfer-Encoding: quoted-printable Message-Id: References: <20200717160739.GA79604@cicely7.cicely.de> To: ticso@cicely.de X-Mailer: Apple Mail (2.3608.120.23.2.1) X-Spam-Status: No, score=-2.9 required=5.0 tests=ALL_TRUSTED,BAYES_00 autolearn=disabled version=3.4.1 X-Spam-Checker-Version: SpamAssassin 3.4.1 (2015-04-28) on mail-n.franken.de X-Rspamd-Queue-Id: 4B7dNG6TR3z4K6Z X-Spamd-Bar: ++ Authentication-Results: mx1.freebsd.org; dkim=none; dmarc=none; spf=none (mx1.freebsd.org: domain of Michael.Tuexen@lurchi.franken.de has no SPF policy when checking 2001:638:a02:a001:20e:cff:fe4a:feaa) smtp.mailfrom=Michael.Tuexen@lurchi.franken.de X-Spamd-Result: default: False [3.00 / 15.00]; RCVD_VIA_SMTP_AUTH(0.00)[]; ARC_NA(0.00)[]; MID_RHS_MATCH_FROM(0.00)[]; FROM_HAS_DN(0.00)[]; RCPT_COUNT_THREE(0.00)[3]; MV_CASE(0.50)[]; MIME_GOOD(-0.10)[text/plain]; DMARC_NA(0.00)[franken.de]; AUTH_NA(1.00)[]; TO_DN_SOME(0.00)[]; NEURAL_SPAM_MEDIUM(0.57)[0.569]; RECEIVED_SPAMHAUS_PBL(0.00)[77.21.245.252:received]; TO_MATCH_ENVRCPT_SOME(0.00)[]; NEURAL_HAM_SHORT(-0.37)[-0.373]; NEURAL_SPAM_LONG(0.40)[0.401]; RCVD_IN_DNSWL_NONE(0.00)[2001:638:a02:a001:20e:cff:fe4a:feaa:from]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+]; ASN(0.00)[asn:680, ipnet:2001:638::/32, country:DE]; SUBJECT_ENDS_QUESTION(1.00)[]; RCVD_TLS_ALL(0.00)[]; RCVD_COUNT_TWO(0.00)[2] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2020 17:27:12 -0000 > On 17. Jul 2020, at 18:07, Bernd Walter = wrote: >=20 > I'm running an LED matrix with SCTP. > The matrix consists from 24 raspberry pi running NFS-root FreeBSD > 12.0-RELEASE (they have an SD card for u-boot and loader). > A client system is running FreeBSD 12.1-RELEASE. I fixed iterator related bugs and this was MFCed to stable/12 recently. The problem was that the iterator stopped sending. >=20 > The matrix modules have a one to many service socket. > The daemon regularily sends status informations (temperature, etc.) to > each association and has a second thread to receive. >=20 > The client system has two deamons running. >=20 > One daemon is to control power output related to temperature states. > It has one thread reestablishing associations via sctp_connectx() to > each of the 24 modules using a single one to many socket. > Another thread collects all regular received data and updates stored > values. > Yet another thread sends power control data via SCTP_SENDALL, so that = all modules know > the maxed allowed brightness rating. >=20 > The other daemon uses the same threads to reconnect and receive. > It connects to the very same sockets on the modules. > Another thread updates picture data and wanted power rating. > That is sending picture data to the given matrix module and then > SCTP_SENDALL an update trigger to all modules. > That is reduced brightness at night times, ... >=20 > All SCTP_SENDALL are just trigger with 0 length and different ppid = values. Are you really sending messages of length 0? That shouldn't work... >=20 > =46rom time to time (1-5 days) I notice that a module won't get = updates > anymore. > I see that the association got a SCTP_SENDER_DRY_EVENT event. > Therefor my expectation is that there is nothing to send. > I still see the association in the socket list and also receive the = regular > temperature data. > However, obviously sending won't happen. > The other modules still get data. >=20 > When I restart the client daemon, things start to work again. >=20 > Currently I'm clueless on how to debug this problem any fsurther. Can you try stable/12? > Doing a full network log would be too big and I'm not very experienced > to understand the SCTP packets. > I have no idea to see what data is in the send buffer. > netstat with TCP would show send and receive queue, not so with SCTP. > Data is send with a single thread, which sctp_sendmsgx() the data for > all modules sequencially into the same socket. I'm not sure I understood what you are actually doing on which socket and how many associations are involved. Best regards Michael > I havn't checked yet if I get an error with the write to the specific > module IP. >=20 > --=20 > B.Walter http://www.bwct.de > Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm. > _______________________________________________ > freebsd-net@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-net > To unsubscribe, send any mail to "freebsd-net-unsubscribe@freebsd.org"