From owner-freebsd-net@freebsd.org Fri Jul 17 16:08:03 2020 Return-Path: Delivered-To: freebsd-net@mailman.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mailman.nyi.freebsd.org (Postfix) with ESMTP id 17550368A8A for ; Fri, 17 Jul 2020 16:08:03 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from raven.bwct.de (raven.bwct.de [IPv6:2a02:21e0:16e0:fe::101:1]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "raven.bwct.de", Issuer "raven.bwct.de" (not verified)) by mx1.freebsd.org (Postfix) with ESMTPS id 4B7bcx2v4wz4F7L for ; Fri, 17 Jul 2020 16:08:00 +0000 (UTC) (envelope-from ticso@cicely7.cicely.de) Received: from mail.cicely.de ([10.1.1.37]) by raven.bwct.de (8.15.2/8.15.2) with ESMTPS id 06HG7pnL078848 (version=TLSv1 cipher=DHE-RSA-AES256-SHA bits=256 verify=FAIL) for ; Fri, 17 Jul 2020 18:07:51 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=cicely.de; s=default; t=1595002072; bh=vHa+IywwepK46ipqRosgROKyfEpFSmwVPjImaqVQe5I=; h=Date:From:To:Cc:Subject:Reply-To; b=fNXCX4GqChWqxPWS2ysNu+yf5vH9qufU2IxF+V7I14m7UycB0SDcHkXrbpW1njZhh ms/pnpntATh//yU3cTCyKUQn5CjYHrhXTlDG5uBZ0RCM6ZLCY5yXn76fczlI0nO+G7 NnuTe7m2vepaG8y/IT04PcxwmUbIavgLhccxCMTY= Received: from cicely7.cicely.de (cicely7.cicely.de [10.1.1.9]) by mail.cicely.de (8.14.5/8.14.4) with ESMTP id 06HG7fM6062136 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Fri, 17 Jul 2020 18:07:42 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: from cicely7.cicely.de (localhost [127.0.0.1]) by cicely7.cicely.de (8.15.2/8.15.2) with ESMTP id 06HG7eDW080788; Fri, 17 Jul 2020 18:07:40 +0200 (CEST) (envelope-from ticso@cicely7.cicely.de) Received: (from ticso@localhost) by cicely7.cicely.de (8.15.2/8.15.2/Submit) id 06HG7dvv080787; Fri, 17 Jul 2020 18:07:39 +0200 (CEST) (envelope-from ticso) Date: Fri, 17 Jul 2020 18:07:39 +0200 From: Bernd Walter To: freebsd-net@freebsd.org Cc: Bernd Walter Subject: SCTP problem, how to debug? Message-ID: <20200717160739.GA79604@cicely7.cicely.de> Reply-To: ticso@cicely.de Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline X-Operating-System: FreeBSD cicely7.cicely.de 12.0-STABLE amd64 User-Agent: Mutt/1.5.11 X-Spam-Status: No, score=-2.9 required=4.5 tests=ALL_TRUSTED=-1, BAYES_00=-1.9 autolearn=ham version=3.3.0 X-Spam-Checker-Version: SpamAssassin 3.3.0 (2010-01-18) on spamd.cicely.de X-Rspamd-Queue-Id: 4B7bcx2v4wz4F7L X-Spamd-Bar: - Authentication-Results: mx1.freebsd.org; dkim=pass header.d=cicely.de header.s=default header.b=fNXCX4Gq; dmarc=none; spf=none (mx1.freebsd.org: domain of ticso@cicely7.cicely.de has no SPF policy when checking 2a02:21e0:16e0:fe::101:1) smtp.mailfrom=ticso@cicely7.cicely.de X-Spamd-Result: default: False [-1.15 / 15.00]; HAS_REPLYTO(0.00)[ticso@cicely.de]; ARC_NA(0.00)[]; R_DKIM_ALLOW(-0.20)[cicely.de:s=default]; NEURAL_HAM_MEDIUM(-1.03)[-1.029]; FROM_HAS_DN(0.00)[]; TO_DN_SOME(0.00)[]; MV_CASE(0.50)[]; NEURAL_HAM_LONG(-1.01)[-1.008]; MIME_GOOD(-0.10)[text/plain]; REPLYTO_DOM_NEQ_FROM_DOM(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[freebsd-net@freebsd.org]; DMARC_NA(0.00)[cicely.de]; RCVD_COUNT_THREE(0.00)[4]; TO_MATCH_ENVRCPT_SOME(0.00)[]; DKIM_TRACE(0.00)[cicely.de:+]; RCPT_COUNT_TWO(0.00)[2]; NEURAL_HAM_SHORT(-0.31)[-0.311]; R_SPF_NA(0.00)[no SPF record]; FROM_EQ_ENVFROM(0.00)[]; MIME_TRACE(0.00)[0:+]; SUBJECT_ENDS_QUESTION(1.00)[]; ASN(0.00)[asn:21461, ipnet:2a02:21e0::/32, country:DE]; RCVD_TLS_LAST(0.00)[]; MID_RHS_MATCH_FROM(0.00)[] X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.33 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 17 Jul 2020 16:08:03 -0000 I'm running an LED matrix with SCTP. The matrix consists from 24 raspberry pi running NFS-root FreeBSD 12.0-RELEASE (they have an SD card for u-boot and loader). A client system is running FreeBSD 12.1-RELEASE. The matrix modules have a one to many service socket. The daemon regularily sends status informations (temperature, etc.) to each association and has a second thread to receive. The client system has two deamons running. One daemon is to control power output related to temperature states. It has one thread reestablishing associations via sctp_connectx() to each of the 24 modules using a single one to many socket. Another thread collects all regular received data and updates stored values. Yet another thread sends power control data via SCTP_SENDALL, so that all modules know the maxed allowed brightness rating. The other daemon uses the same threads to reconnect and receive. It connects to the very same sockets on the modules. Another thread updates picture data and wanted power rating. That is sending picture data to the given matrix module and then SCTP_SENDALL an update trigger to all modules. That is reduced brightness at night times, ... All SCTP_SENDALL are just trigger with 0 length and different ppid values. >From time to time (1-5 days) I notice that a module won't get updates anymore. I see that the association got a SCTP_SENDER_DRY_EVENT event. Therefor my expectation is that there is nothing to send. I still see the association in the socket list and also receive the regular temperature data. However, obviously sending won't happen. The other modules still get data. When I restart the client daemon, things start to work again. Currently I'm clueless on how to debug this problem any fsurther. Doing a full network log would be too big and I'm not very experienced to understand the SCTP packets. I have no idea to see what data is in the send buffer. netstat with TCP would show send and receive queue, not so with SCTP. Data is send with a single thread, which sctp_sendmsgx() the data for all modules sequencially into the same socket. I havn't checked yet if I get an error with the write to the specific module IP. -- B.Walter http://www.bwct.de Modbus/TCP Ethernet I/O Baugruppen, ARM basierte FreeBSD Rechner uvm.