From owner-freebsd-net@freebsd.org Thu Sep 29 23:19:46 2016 Return-Path: Delivered-To: freebsd-net@mailman.ysv.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by mailman.ysv.freebsd.org (Postfix) with ESMTP id 5AB86C02CDF for ; Thu, 29 Sep 2016 23:19:46 +0000 (UTC) (envelope-from lew@perftech.com) Received: from smtp-gw.pt.net (smtp-gw.pt.net [206.210.194.15]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (Client CN "smtp-gw.pt.net", Issuer "Let's Encrypt Authority X3" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 20527F8 for ; Thu, 29 Sep 2016 23:19:45 +0000 (UTC) (envelope-from lew@perftech.com) X-ASG-Debug-ID: 1475191154-09411a12c010c76b0001-QdxwpM Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by smtp-gw.pt.net with ESMTP id 2RAvriKBxpPfgeM4 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO) for ; Thu, 29 Sep 2016 18:19:14 -0500 (CDT) X-Barracuda-Envelope-From: lew@perftech.com X-Barracuda-Effective-Source-IP: mail.pt.net[206.210.194.11] X-Barracuda-Apparent-Source-IP: 206.210.194.11 Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id 11E6F8424B1 for ; Thu, 29 Sep 2016 18:19:14 -0500 (CDT) Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10032) with ESMTP id a09MDYPAY_jY for ; Thu, 29 Sep 2016 18:19:13 -0500 (CDT) Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id C59D68425E3 for ; Thu, 29 Sep 2016 18:19:13 -0500 (CDT) X-Virus-Scanned: amavisd-new at pt.net Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10026) with ESMTP id IDlckK3uSzWM for ; Thu, 29 Sep 2016 18:19:13 -0500 (CDT) Received: from dhcp-221-110.perftech.com (dhcp-221-110.perftech.com [206.210.221.110]) (Authenticated sender: lew@pt.net) by mail.pt.net (Postfix) with ESMTPSA id B53688424B1 for ; Thu, 29 Sep 2016 18:19:13 -0500 (CDT) From: Lewis Donzis Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Subject: recv() with MSG_WAITALL appears to be broken (sometimes) Message-Id: X-ASG-Orig-Subj: recv() with MSG_WAITALL appears to be broken (sometimes) Date: Thu, 29 Sep 2016 18:19:13 -0500 To: freebsd-net@freebsd.org Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) X-Mailer: Apple Mail (2.3124) X-Barracuda-Connect: mail.pt.net[206.210.194.11] X-Barracuda-Start-Time: 1475191154 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://smtp-gw.pt.net:443/cgi-mod/mark.cgi X-Barracuda-Scan-Msg-Size: 4288 X-Virus-Scanned: by bsmtpd at pt.net X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.33333 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- X-BeenThere: freebsd-net@freebsd.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: Networking and TCP/IP with FreeBSD List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 29 Sep 2016 23:19:46 -0000 I posted this on bugzilla = (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=3D212716) but thought = I=E2=80=99d see if anyone here can offer some advice. We have a simple test program to illustrate the problem. Run "server" on a machine, and then run "client" on the same machine. Server creates a listening socket and loops accepting connections, = sending a little data, closing the socket, and then back to the top of = the loop. Client creates a socket, connects to the server, does a recv() with = MSG_WAITALL, closes the socket, and loops. This runs anywhere from one to a few dozen times and then hangs. The = server socket is in FIN_WAIT_2, and the client socket is in CLOSE_WAIT. = So the client side is waiting for the application to close the socket, = but it's still stuck in the recv(), never being awakened by the EOF from = the server closing the socket. The same code runs on Linux and QNX without any problem. This seems ridiculously simple and seems like it would affect many = programs, so perhaps we=E2=80=99re just overlooking something. =20 Note that it makes no difference whether it's run in the same machine = (to localhost or to the machine's own IP address) or across the network, = or whether it's IPv4 or IPv6. It=E2=80=99s also the same with a UNIX = domain STREAM socket (but not a SEQPACKET socket), so it appears to be = unrelated to TCP and perhaps more of a general problem in the socket = layer. In all cases, the recv() never unblocks when the peer closes the = socket. Thanks, lew client.c: #include #include #include #include #include #include #include #include #include #include int main (int argc, char *argv[]) { int c; bool verbose =3D false; char *addr =3D "127.0.0.1"; signal(SIGPIPE, SIG_IGN); while ((c =3D getopt(argc, argv, "v")) !=3D -1) { switch (c) { case 'v': ++verbose; break; default: return 1; break; } } argc -=3D optind - 1; argv +=3D optind - 1; if (argc > 1) addr =3D argv[1]; for (int try =3D 1;; try++) { printf("Try %d\n", try); int s; if ((s =3D socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0) err(1, "socket"); =20 struct sockaddr_in sa =3D { .sin_family =3D AF_INET, .sin_port =3D = htons(79), .sin_addr.s_addr =3D inet_addr(addr) }; if (connect(s, (struct sockaddr *)&sa, sizeof sa) < 0) err(1, "connect"); // get the response char rbuf [4096]; int rcvLen; if ((rcvLen =3D recv(s, rbuf, sizeof rbuf, MSG_WAITALL)) < 0) err(1, "recv"); if (verbose) printf("Received: '%.*s'\n", rcvLen, rbuf); close(s); } return 0; } server.c: #include #include #include #include #include #include #include #include #include #include #include int main (int argc, char *argv[]) { int c; bool verbose =3D false; signal(SIGPIPE, SIG_IGN); while ((c =3D getopt(argc, argv, "v")) !=3D -1) { switch (c) { case 'v': ++verbose; break; default: return 1; break; } } argc -=3D optind - 1; argv +=3D optind - 1; int s0; if ((s0 =3D socket(AF_INET, SOCK_STREAM, IPPROTO_TCP)) < 0) err(1, "socket"); int on =3D true; if (setsockopt(s0, SOL_SOCKET, SO_REUSEPORT, &on, sizeof on) < 0) err(1, "setsockopt"); struct sockaddr_in sa =3D { .sin_family =3D AF_INET, .sin_port =3D = htons(79) }; if (bind(s0, (struct sockaddr *)&sa, sizeof sa) < 0) err(1, "bind"); if (listen(s0, 1000) < 0) err(1, "listen"); printf("Listening\n"); for (int try =3D 1;; try++) { socklen_t sl; int s =3D accept(s0, (struct sockaddr *)&sa, &sl); if (s < 0) err(1, "accept"); printf("Try %d\n", try); // send the response static char sbuf [] =3D "Hello from server\n"; if (send(s, sbuf, sizeof sbuf - 1, 0) < 0) err(1, "send"); if (verbose) printf("Sent: %s", sbuf); close(s); } return 0; }