From nobody Thu Dec 19 16:38:20 2024 X-Original-To: freebsd-net@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4YDbp73JwBz5hR3w for ; Thu, 19 Dec 2024 16:38:23 +0000 (UTC) (envelope-from glebius@freebsd.org) Received: from smtp.freebsd.org (smtp.freebsd.org [96.47.72.83]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (4096 bits) client-digest SHA256) (Client CN "smtp.freebsd.org", Issuer "R10" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4YDbp72m7zz4vpB; Thu, 19 Dec 2024 16:38:23 +0000 (UTC) (envelope-from glebius@freebsd.org) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1734626303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ef2DSbe2uhpLes1cH6103brn1EFdLJPCJkegtwZAtOM=; b=AXoFErShpZ+Rkhmv0ei7ia9i4YI0osbjiO7mpfXK+PdZ/oyX6i4W/HqahsXSLyJaXtLszt bRIfee3LQ9hTaBElnT27Q39CGoVmpu6ik9PJM7ZqY6nHODjtMBVwb7Kfl/fJKsfCoh2ejG 664tsqsMEJ4XNXJ5hbhpvYvNgBbUKEHHPP0DODqXLuMQ1DXTZAVNIJctfPoPnCB5rx9IQO s5erPDPMqRo4eV8EW5rRNB9xep3fTrUSQ6j+175ls/e0p6tmsRtXQc12fwJspcyYpfRllq dNy+5rf/vN9LtMGHVKXCfwk0qyrBqLnXx0mdCHPSvFpW71kil12Zkzk6w1vAEQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=freebsd.org; s=dkim; t=1734626303; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=ef2DSbe2uhpLes1cH6103brn1EFdLJPCJkegtwZAtOM=; b=bU9e6wHeITIfXZK8pIZqPy72g4tRGev1LfE/3pBkcAuWPHh7oue5oH2SnMUQvSwalPz/g0 vx7PCXqBo+JzOlam/I+WgmN3V2kO5+DADc7UVGM9TRQ2RcdrUebRYORmXS5haGgahTs48k 3FYR4Jwy+4shJo/dg9KEYVG1vak9nJtS73aD2R6vO1T813FFG+H1bHLIKGtYoxxfviY3sx GwOlFkrXNGVbfFvoNhu2kOwAvwhOl0I1n0vukj0BkQFNpSpMp7c1xN8ycIY+jzYdY7xd8T zCRq30LSMtj1fUILwec8zLfKbDdtVaOIy7ITFqpal+yV1wcYNudxgDWySiQ90Q== ARC-Authentication-Results: i=1; mx1.freebsd.org; none ARC-Seal: i=1; s=dkim; d=freebsd.org; t=1734626303; a=rsa-sha256; cv=none; b=jZe5dAumRxRYAkLW2eHLjVdwIRjqIZ3mqAVWZxkHhBWzqNboEFkm/d314t3mvSwXOJVzUl zFL783ikB7F5LHJq9yQ61EvtQew+RfDdjgA9yz5LLsqJ+W0CkHxx/35LB4OV3skzd8CrbB w5bmC/RVBKw31qGxQ0Zv2JlSKv2tsE6hwrxvjbBq8ZOOuGX3NPE1jQZaJLo7YVdaJfATiZ RC3yeTzj3hLwzlzCXkz9znKLIFHeACAPVtS0yrTeP3J47EjujJshbh6mUo2ayNZod9XIlb JFSDr7/bSVMn3MsAkcI2EBgdc+FzpC2OuBgWItfKENZcdlRoI84ewIsHPg5pvw== Received: from cell.glebi.us (glebi.us [162.251.186.162]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256) (Client did not present a certificate) (Authenticated sender: glebius) by smtp.freebsd.org (Postfix) with ESMTPSA id 4YDbp673r9zYp1; Thu, 19 Dec 2024 16:38:22 +0000 (UTC) (envelope-from glebius@freebsd.org) Date: Thu, 19 Dec 2024 08:38:20 -0800 From: Gleb Smirnoff To: Sad Clouds Cc: freebsd-net@freebsd.org Subject: Re: TCP socket handling errors Message-ID: References: <20240403131434.c3bfa64c726505d842408c80@gmail.com> List-Id: Networking and TCP/IP with FreeBSD List-Archive: https://lists.freebsd.org/archives/freebsd-net List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-freebsd-net@FreeBSD.org MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20240403131434.c3bfa64c726505d842408c80@gmail.com> On Wed, Apr 03, 2024 at 01:14:34PM +0100, Sad Clouds wrote: S> Hello, I have a client/server networking application that exhibits S> TCP socket handling errors. This only happens on FreeBSD, while NetBSD, S> Linux, Solaris, etc. all seem to work correctly. I was hoping to get S> some advice on what could be the root cause. S> S> I have two processes - client and server, sending and receiving data S> to/from each other on 127.0.0.1 S> S> Client connects to server and calls send(2)/recv(2) in a loop. This is S> a bidirectional data exchange. When all send data is transferred, S> client calls shutdown(sockfd, SHUT_WR) and continues receiving data on S> the same socket until recv(2) returns 0 bytes, which signals end of S> receive data. At this stage client calls close(sockfd) and terminates. S> S> Server has the same data transfer loop as the client. S> S> I frequently get ECONNRESET when calling close(2), sometimes from the S> server and sometimes from the client process. This should not be S> happening, but I'm not sure what could be causing it. S> S> The client logic is as follows: S> S> 1. Set sockfd nonblocking. S> 2. Call send(2)/recv(2) in a loop until N bytes have been transferred in each direction. S> 3. Set sockfd blocking. S> 4. Call send_buf() to send control handshake to server. S> 5. Call shutdown(sockfd, SHUT_WR) to signal end of send data from client. S> 6. Call recv_buf() to receive control handshake from server. S> 7. Call recv_buf() and verify it returned 0 bytes to indicate end of data from server. S> 8. Call close(sockfd) and verify success. S> S> Step 8 sometimes fails and returns ECONNRESET. S> S> Functions send_buf() and recv_buf() are wrappers around send(2) and S> recv(2) which restart those system calls until the specified number of S> buffer bytes have been fully transferred or 0 is returned in the case S> of recv_buf() indicating end of data. They are designed to work with S> blocking file descriptors and avoid short reads/writes. S> S> I don't understand why close(2) sometimes returns ECONNRESET when the S> previous recv(2) call at step 7 returned 0 bytes, indicating the remote S> TCP end sent us a FIN. S> S> I don't set SO_LINGER socket option and when I checked the default on S> FreeBSD it reports l_onoff=0, l_linger=0 so there should be no S> immediate RST on socket close(2). S> S> Does anyone have any suggestions or ideas? Please take a look at https://reviews.freebsd.org/D48148 -- Gleb Smirnoff