From owner-freebsd-standards@FreeBSD.ORG Sun Aug 28 11:30:15 2011 Return-Path: Delivered-To: freebsd-standards@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id EBEE3106564A for ; Sun, 28 Aug 2011 11:30:15 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id D9E528FC0C for ; Sun, 28 Aug 2011 11:30:15 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p7SBUFMs002620 for ; Sun, 28 Aug 2011 11:30:15 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p7SBUFmt002617; Sun, 28 Aug 2011 11:30:15 GMT (envelope-from gnats) Date: Sun, 28 Aug 2011 11:30:15 GMT Message-Id: <201108281130.p7SBUFmt002617@freefall.freebsd.org> To: freebsd-standards@FreeBSD.org From: =?koi8-r?B?5c3B28XXIO7Jy8/Mwco=?= Cc: Subject: Re: standards/154842: invalid request authenticator in the second and subsequent acct-packets, generated by libradius X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list Reply-To: =?koi8-r?B?5c3B28XXIO7Jy8/Mwco=?= List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 28 Aug 2011 11:30:16 -0000 The following reply was made to PR standards/154842; it has been noted by GNATS. From: =?koi8-r?B?5c3B28XXIO7Jy8/Mwco=?= To: bug-followup@freebsd.org,yv@lifelink.ru Cc: Subject: Re: standards/154842: invalid request authenticator in the second and subsequent acct-packets, generated by libradius Date: Sun, 28 Aug 2011 15:24:46 +0400 FreeBSD 8.2-RELEASE FreeBSD 8.2-RELEASE #1: Sun Aug 7 03:57:35 MSD 2011 root@abills:/usr/obj/usr/src/sys/GENERIC i386 mpd-5.5 freeradius-1.1.8_3 I Have error: Received Accounting-Request packet from client x.x.x.x with invalid signature! (Shared secret is incorrect.) Dropping packet without response. Shared secret is correct. 100% From owner-freebsd-standards@FreeBSD.ORG Mon Aug 29 11:07:18 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 35DE8106564A for ; Mon, 29 Aug 2011 11:07:18 +0000 (UTC) (envelope-from owner-bugmaster@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 1BBF68FC1F for ; Mon, 29 Aug 2011 11:07:18 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.4/8.14.4) with ESMTP id p7TB7Hmr089408 for ; Mon, 29 Aug 2011 11:07:17 GMT (envelope-from owner-bugmaster@FreeBSD.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.4/8.14.4/Submit) id p7TB7H79089406 for freebsd-standards@FreeBSD.org; Mon, 29 Aug 2011 11:07:17 GMT (envelope-from owner-bugmaster@FreeBSD.org) Date: Mon, 29 Aug 2011 11:07:17 GMT Message-Id: <201108291107.p7TB7H79089406@freefall.freebsd.org> X-Authentication-Warning: freefall.freebsd.org: gnats set sender to owner-bugmaster@FreeBSD.org using -f From: FreeBSD bugmaster To: freebsd-standards@FreeBSD.org Cc: Subject: Current problem reports assigned to freebsd-standards@FreeBSD.org X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 29 Aug 2011 11:07:18 -0000 Note: to view an individual PR, use: http://www.freebsd.org/cgi/query-pr.cgi?pr=(number). The following is a listing of current problems submitted by FreeBSD users. These represent problem reports covering all versions including experimental development code and obsolete releases. S Tracker Resp. Description -------------------------------------------------------------------------------- o stand/157050 standards OSS implementation lacks AFMT_FLOAT o stand/154842 standards invalid request authenticator in the second and subseq o stand/150093 standards C++ std::locale support is broken a stand/149980 standards [libc] [patch] negative value integer to nanosleep(2) o stand/147210 standards xmmintrin.h and cstdlib conflicts with each other with o docs/143472 standards gethostname(3) references undefined value: HOST_NAME_M s stand/141705 standards [libc] [request] libc lacks cexp (and friends) o stand/130067 standards Wrong numeric limits in system headers? o stand/124860 standards flockfile(3) doesn't work when the memory has been exh o stand/121921 standards [patch] Add leap second support to at(1), atrun(8) o stand/116477 standards rm(1): rm behaves unexpectedly when using -r and relat o bin/116413 standards incorrect getconf(1) handling of unsigned constants gi o stand/116081 standards make does not work with the directive sinclude p stand/107561 standards [libc] [patch] [request] Missing SUS function tcgetsid o stand/100017 standards [Patch] Add fuser(1) functionality to fstat(1) o stand/96236 standards [patch] [posix] sed(1) incorrectly describes a functio o stand/94729 standards [libc] fcntl() throws undocumented ENOTTY a stand/86484 standards [patch] mkfifo(1) uses wrong permissions o stand/82654 standards C99 long double math functions are missing o stand/81287 standards [patch] fingerd(8) might send a line not ending in CRL a stand/80293 standards sysconf() does not support well-defined unistd values o stand/79056 standards [feature request] [atch] regex(3) regression tests o stand/70813 standards [patch] ls(1) not Posix compliant o stand/66357 standards make POSIX conformance problem ('sh -e' & '+' command- s kern/64875 standards [libc] [patch] [request] add a system call: fdatasync( o stand/56476 standards [patch] cd9660 unicode support simple hack o stand/54410 standards one-true-awk not POSIX compliant (no extended REs) o stand/46119 standards Priority problems for SCHED_OTHER using pthreads o stand/44365 standards [headers] [patch] [request] introduce ulong and unchar a stand/41576 standards ln(1): replacing old dir-symlinks o stand/39256 standards snprintf/vsnprintf aren't POSIX-conformant for strings a docs/26003 standards getgroups(2) lists NGROUPS_MAX but not syslimits.h s stand/24590 standards timezone function not compatible witn Single Unix Spec o stand/21519 standards sys/dir.h should be deprecated some more s bin/14925 standards getsubopt isn't poisonous enough 35 problems total. From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 02:26:10 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 1E146106566B; Fri, 2 Sep 2011 02:26:10 +0000 (UTC) (envelope-from gabor@FreeBSD.org) Received: from server.mypc.hu (server.mypc.hu [87.229.73.95]) by mx1.freebsd.org (Postfix) with ESMTP id A32588FC14; Fri, 2 Sep 2011 02:26:09 +0000 (UTC) Received: from server.mypc.hu (localhost [127.0.0.1]) by server.mypc.hu (Postfix) with ESMTP id 77AE014E5C9E; Fri, 2 Sep 2011 04:08:41 +0200 (CEST) X-Virus-Scanned: amavisd-new at server.mypc.hu Received: from server.mypc.hu ([127.0.0.1]) by server.mypc.hu (server.mypc.hu [127.0.0.1]) (amavisd-new, port 10024) with LMTP id YWxInx5JKUWk; Fri, 2 Sep 2011 04:08:39 +0200 (CEST) Received: from [192.168.1.106] (catv-80-98-232-12.catv.broadband.hu [80.98.232.12]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by server.mypc.hu (Postfix) with ESMTPSA id 0452214DB679; Fri, 2 Sep 2011 04:08:38 +0200 (CEST) Message-ID: <4E603AA3.1040204@FreeBSD.org> Date: Fri, 02 Sep 2011 04:08:35 +0200 From: Gabor Kovesdan User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:9.0a1) Gecko/20110822 Thunderbird/9.0a1 MIME-Version: 1.0 To: freebsd-standards@FreeBSD.org, freebsd-i18n@FreeBSD.org Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: Andrey Chernov Subject: POSIX regex VS. multi-byte characters X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 02:26:10 -0000 Hi Folks, While working on bringing in a new regex code to FreeBSD, I came into an issue. POSIX says here: http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09 "Matching shall be based on the bit pattern used for encoding the character, not on the graphic representation of the character. This means that if a character set contains two or more encodings for a graphic symbol, or if the strings searched contain text encoded in more than one codeset, no attempt is made to search for any other representation of the encoded symbol. If that is required, the user can specify equivalence classes containing all variations of the desired graphic symbol." According to my interpretation of this text, if someone specifies a single bit as pattern that can be a prefix of a multi-byte character that shall match, since match is based on bit pattern not semantical meaning. Besides, in a consistent environment that uses a single encoding and also supposing a user with common sense that would not enter meaningless input, only whole characters should occur in the pattern. However, GNU grep has a test in its regression test suite that contradicts to this and chooses the opposite approach, i.e. it shall not match a fragment of a character. Looking at the standard, I think GNU grep is incorrect and my interpretation is the correct one. Could you please comment on this? Thanks, Gabor Kovesdan From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 06:37:57 2011 Return-Path: Delivered-To: freebsd-standards@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 8E0EA106564A; Fri, 2 Sep 2011 06:37:57 +0000 (UTC) (envelope-from wolfgang@lyxys.ka.sub.org) Received: from saturn.lyxys.ka.sub.org (saturn.lyxys.ka.sub.org [217.29.35.151]) by mx1.freebsd.org (Postfix) with ESMTP id DCDBE8FC12; Fri, 2 Sep 2011 06:37:56 +0000 (UTC) Received: from juno.lyxys.ka.sub.org (juno.lyx [IPv6:fd2a:89ca:7d54:0:20f:feff:fe0e:7312]) by saturn.lyxys.ka.sub.org (8.14.2/8.14.2) with ESMTP id p8263edm006884; Fri, 2 Sep 2011 08:03:40 +0200 (CEST) (envelope-from wolfgang@lyxys.ka.sub.org) Received: from juno.lyxys.ka.sub.org (localhost [127.0.0.1]) by juno.lyxys.ka.sub.org (8.14.5/8.14.5) with ESMTP id p8263d6P008359; Fri, 2 Sep 2011 08:03:39 +0200 (CEST) (envelope-from wolfgang@lyxys.ka.sub.org) Received: (from wolfgang@localhost) by juno.lyxys.ka.sub.org (8.14.5/8.14.5/Submit) id p8263dIl008358; Fri, 2 Sep 2011 08:03:39 +0200 (CEST) (envelope-from wolfgang@lyxys.ka.sub.org) X-Authentication-Warning: juno.lyx: wolfgang set sender to wolfgang@lyxys.ka.sub.org using -f Date: Fri, 2 Sep 2011 08:03:38 +0200 From: Wolfgang Zenker To: Gabor Kovesdan Message-ID: <20110902060338.GA8192@lyxys.ka.sub.org> References: <4E603AA3.1040204@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E603AA3.1040204@FreeBSD.org> User-Agent: Mutt/1.4.2.3i Organization: private site Cc: Andrey Chernov , freebsd-standards@freebsd.org, freebsd-i18n@freebsd.org Subject: Re: POSIX regex VS. multi-byte characters X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 06:37:57 -0000 Hi Gabor, * Gabor Kovesdan [110902 04:08]: > While working on bringing in a new regex code to FreeBSD, I came into an > issue. POSIX says here: > http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09 > "Matching shall be based on the bit pattern used for encoding the > character, not on the graphic representation of the character. This > means that if a character set contains two or more encodings for a > graphic symbol, or if the strings searched contain text encoded in more > than one codeset, no attempt is made to search for any other > representation of the encoded symbol. If that is required, the user can > specify equivalence classes containing all variations of the desired > graphic symbol." > According to my interpretation of this text, if someone specifies a > single bit as pattern that can be a prefix of a multi-byte character > that shall match, since match is based on bit pattern not semantical > meaning. Besides, in a consistent environment that uses a single > encoding and also supposing a user with common sense that would not > enter meaningless input, only whole characters should occur in the > pattern. However, GNU grep has a test in its regression test suite that > contradicts to this and chooses the opposite approach, i.e. it shall not > match a fragment of a character. Looking at the standard, I think GNU > grep is incorrect and my interpretation is the correct one. I think you are misinterpreting the standard here. As I read it, the phrase "bit pattern used for encoding the character" means the complete byte sequence that encodes the character, not just a byte. The paragraph quoted above talks about characters that have several different encodings like e.g. characters that exist as single codepoint but can also be encoded using diacritical marks and a base character. Wolfgang From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 09:16:42 2011 Return-Path: Delivered-To: freebsd-standards@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 5B066106564A; Fri, 2 Sep 2011 09:16:42 +0000 (UTC) (envelope-from ache@vniz.net) Received: from vniz.net (vniz.net [194.87.13.69]) by mx1.freebsd.org (Postfix) with ESMTP id B7C668FC0C; Fri, 2 Sep 2011 09:16:41 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vniz.net (8.14.5/8.14.5) with ESMTP id p828wF01091148; Fri, 2 Sep 2011 12:58:15 +0400 (MSK) (envelope-from ache@vniz.net) Received: (from ache@localhost) by localhost (8.14.5/8.14.5/Submit) id p828wFcQ091147; Fri, 2 Sep 2011 12:58:15 +0400 (MSK) (envelope-from ache) Date: Fri, 2 Sep 2011 12:58:14 +0400 From: Andrey Chernov To: Wolfgang Zenker Message-ID: <20110902085814.GA90871@vniz.net> Mail-Followup-To: Andrey Chernov , Wolfgang Zenker , Gabor Kovesdan , freebsd-standards@freebsd.org, freebsd-i18n@freebsd.org References: <4E603AA3.1040204@FreeBSD.org> <20110902060338.GA8192@lyxys.ka.sub.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20110902060338.GA8192@lyxys.ka.sub.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-standards@freebsd.org, Gabor Kovesdan , freebsd-i18n@freebsd.org Subject: Re: POSIX regex VS. multi-byte characters X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 09:16:42 -0000 On Fri, Sep 02, 2011 at 08:03:38AM +0200, Wolfgang Zenker wrote: > Hi Gabor, > > * Gabor Kovesdan [110902 04:08]: > > While working on bringing in a new regex code to FreeBSD, I came into an > > issue. POSIX says here: > > http://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap09.html#tag_09 > > > "Matching shall be based on the bit pattern used for encoding the > > character, not on the graphic representation of the character. This > > means that if a character set contains two or more encodings for a > > graphic symbol, or if the strings searched contain text encoded in more > > than one codeset, no attempt is made to search for any other > > representation of the encoded symbol. If that is required, the user can > > specify equivalence classes containing all variations of the desired > > graphic symbol." > > > According to my interpretation of this text, if someone specifies a > > single bit as pattern that can be a prefix of a multi-byte character > > that shall match, since match is based on bit pattern not semantical > > meaning. Besides, in a consistent environment that uses a single > > encoding and also supposing a user with common sense that would not > > enter meaningless input, only whole characters should occur in the > > pattern. However, GNU grep has a test in its regression test suite that > > contradicts to this and chooses the opposite approach, i.e. it shall not > > match a fragment of a character. Looking at the standard, I think GNU > > grep is incorrect and my interpretation is the correct one. > > I think you are misinterpreting the standard here. As I read it, the > phrase "bit pattern used for encoding the character" means the complete > byte sequence that encodes the character, not just a byte. The paragraph > quoted above talks about characters that have several different encodings > like e.g. characters that exist as single codepoint but can also be > encoded using diacritical marks and a base character. 1) As I read it, too. "bit pattern" means to be complete, not partial. POSIX don't suppose partial or fragmened charaters match, all characters there are always complete and monolitic. 2) The whole intention says; i.e. graphically same Russsian 'a' should not match graphically same English 'a' inside giving character set like KOI8-R or Unicode. 3) Meaningless input should not match anything with meaning, so only one question remains, should meaningless input match exact the same meaningless input or should exit with error? POSIX grep() says nothing, POSIX regexec() says not more than: "The regcomp( ) and regexec( ) functions are required to accept any null-terminated string as the pattern argument. If the meaning of the string is 'undefined', the behavior of the function is 'unspecified'." Currently GNU grep match meaningless input with exact the same in the file. Fragment of character (not completed) is meaningless input, so I don't see where GNU grep is opposite. -- http://ache.vniz.net/ From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 09:46:18 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 55642106564A; Fri, 2 Sep 2011 09:46:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 6B0908FC12; Fri, 2 Sep 2011 09:46:17 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id MAA16779; Fri, 02 Sep 2011 12:28:25 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QzQ2z-000EZt-6m; Fri, 02 Sep 2011 12:28:25 +0300 Message-ID: <4E60A1B8.7080607@FreeBSD.org> Date: Fri, 02 Sep 2011 12:28:24 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110830 Thunderbird/6.0 MIME-Version: 1.0 To: freebsd-net@FreeBSD.org, freebsd-standards@FreeBSD.org X-Enigmail-Version: undefined Content-Type: text/plain; charset=X-VIET-VPS Content-Transfer-Encoding: 7bit Cc: Subject: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 09:46:18 -0000 I see a problem where FreeBSD kernel (recent head) returns POLLHUP _alone_ (0x10) for a socket that has never been connected - a client socket for which connect(2) failed. There is also a piece of software that doesn't expect that flag and exhibits illogical behavior because of it. This is how POSIX describes POLLHUP: POLLHUP The device has been disconnected. This event and POLLOUT are mutually-exclusive; a stream can never be writable if a hangup has occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND, or POLLPRI are not mutually-exclusive. This flag is only valid in the revents bitmask; it shall be ignored in the events member. For me "disconnected" _implies_ that the device should have been connected first. But this is not explicitly said anywhere. Also, I think it's possible that a socket gets connected and immediately disconnected (before poll(2) is called), then the POLLHUP would be appropriate in any interpretation. So, I am inclined to think that the software should check for POLLHUP. But I would like to ask your opinion since the problem appears to be FreeBSD-specific. -- Andriy Gapon From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 10:40:20 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 255E3106566C; Fri, 2 Sep 2011 10:40:20 +0000 (UTC) (envelope-from jilles@stack.nl) Received: from mx1.stack.nl (relay02.stack.nl [IPv6:2001:610:1108:5010::104]) by mx1.freebsd.org (Postfix) with ESMTP id B63FB8FC1C; Fri, 2 Sep 2011 10:40:19 +0000 (UTC) Received: from turtle.stack.nl (turtle.stack.nl [IPv6:2001:610:1108:5010::132]) by mx1.stack.nl (Postfix) with ESMTP id 193EF3593D6; Fri, 2 Sep 2011 12:40:19 +0200 (CEST) Received: by turtle.stack.nl (Postfix, from userid 1677) id 0CDD7173A8; Fri, 2 Sep 2011 12:40:19 +0200 (CEST) Date: Fri, 2 Sep 2011 12:40:19 +0200 From: Jilles Tjoelker To: Andriy Gapon Message-ID: <20110902104018.GA12845@stack.nl> References: <4E60A1B8.7080607@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E60A1B8.7080607@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@FreeBSD.org, freebsd-standards@FreeBSD.org Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 10:40:20 -0000 On Fri, Sep 02, 2011 at 12:28:24PM +0300, Andriy Gapon wrote: > I see a problem where FreeBSD kernel (recent head) returns POLLHUP > _alone_ (0x10) for a socket that has never been connected - a client > socket for which connect(2) failed. There is also a piece of software > that doesn't expect that flag and exhibits illogical behavior because > of it. That would be a bug in that software. Returning POLLHUP alone is clearly valid. > This is how POSIX describes POLLHUP: > POLLHUP > The device has been disconnected. This event and POLLOUT are > mutually-exclusive; a stream can never be writable if a hangup has > occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND, or > POLLPRI are not mutually-exclusive. This flag is only valid in the > revents bitmask; it shall be ignored in the events member. Together with the description of POLLIN which says that data may be read without blocking, this seems to mean that if POLLHUP is set alone, there is no point in calling read() since it will return 0 or -1 immediately. If POLLHUP and POLLIN are both set, there is still data in the buffer (in FreeBSD, POLLHUP | POLLIN is often returned even if there is no data, to keep buggy applications from breaking). A complicating factor is that there are situations where read() returns 0, but yet it is not a hangup. For example, if a user types the EOF character (such as ^D) in a terminal, this causes read() to return 0 but the terminal remains writable (and writing to it may even block). This is the "zero length message" which is under the obsolescent STREAMS marking, even though it applies to terminals in all implementations. Similarly, if the other side has done the equivalent of shutdown(SHUT_WR), read() will return 0 but POLLHUP must not be returned as it would prevent the application from waiting for space in the outgoing socket buffer; therefore POLLIN must be returned. Similar considerations apply to other shutdown() calls (but note that shutdown(SHUT_RD) is not sent to the other side). The other way around, read() may fail after poll() did not return POLLHUP if the connection had closed in the meantime. > For me "disconnected" _implies_ that the device should have been > connected first. But this is not explicitly said anywhere. Also, I > think it's possible that a socket gets connected and immediately > disconnected (before poll(2) is called), then the POLLHUP would be > appropriate in any interpretation. So, I am inclined to think that > the software should check for POLLHUP. But I would like to ask your > opinion since the problem appears to be FreeBSD-specific. There was a connection attempt in progress that failed at some point, which seems close enough to "disconnected". Alternatively, POLLERR could be set but could activate more bugs in applications. Comparisons with things like "connected and immediately disconnected" and "connection closed after poll() returned something" are useful in reasoning about this. Ports people have complained about poll() behaviour before, are there configure scripts that attempt to check if we ever return POLLHUP alone and only check for POLLIN if not? -- Jilles Tjoelker From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 11:04:41 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id CD9CD106566B; Fri, 2 Sep 2011 11:04:41 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id E7CA28FC12; Fri, 2 Sep 2011 11:04:39 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id OAA18037; Fri, 02 Sep 2011 14:04:37 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QzRY5-000Ee8-4F; Fri, 02 Sep 2011 14:04:37 +0300 Message-ID: <4E60B842.8050506@FreeBSD.org> Date: Fri, 02 Sep 2011 14:04:34 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0) Gecko/20110830 Thunderbird/6.0 MIME-Version: 1.0 To: Jilles Tjoelker References: <4E60A1B8.7080607@FreeBSD.org> <20110902104018.GA12845@stack.nl> In-Reply-To: <20110902104018.GA12845@stack.nl> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-net@FreeBSD.org, freebsd-standards@FreeBSD.org Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 11:04:41 -0000 on 02/09/2011 13:40 Jilles Tjoelker said the following: > Ports people have complained about poll() behaviour before, are there > configure scripts that attempt to check if we ever return POLLHUP alone > and only check for POLLIN if not? Not sure about that other software and how POLLIN is related here. The software in question (mozilla nspr) checks for POLLNVAL, POLLERR, POLLPRI and POLLOUT to determine if anything interesting has happened to a connection supposed to be in progress. They aren't checking for POLLHUP at all and thus they keep thinking that the connection is still in progress when they get it. -- Andriy Gapon From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 11:21:18 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.ORG Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 58147106564A; Fri, 2 Sep 2011 11:21:18 +0000 (UTC) (envelope-from ache@vniz.net) Received: from vniz.net (vniz.net [194.87.13.69]) by mx1.freebsd.org (Postfix) with ESMTP id C95E88FC17; Fri, 2 Sep 2011 11:21:17 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by vniz.net (8.14.5/8.14.5) with ESMTP id p82BLG6D024064; Fri, 2 Sep 2011 15:21:16 +0400 (MSK) (envelope-from ache@vniz.net) Received: (from ache@localhost) by localhost (8.14.5/8.14.5/Submit) id p82BLG1k024063; Fri, 2 Sep 2011 15:21:16 +0400 (MSK) (envelope-from ache) Date: Fri, 2 Sep 2011 15:21:16 +0400 From: Andrey Chernov To: Andriy Gapon Message-ID: <20110902112116.GA23835@vniz.net> Mail-Followup-To: Andrey Chernov , Andriy Gapon , Jilles Tjoelker , freebsd-net@FreeBSD.ORG, freebsd-standards@FreeBSD.ORG References: <4E60A1B8.7080607@FreeBSD.org> <20110902104018.GA12845@stack.nl> <4E60B842.8050506@FreeBSD.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4E60B842.8050506@FreeBSD.org> User-Agent: Mutt/1.5.21 (2010-09-15) Cc: freebsd-net@FreeBSD.ORG, freebsd-standards@FreeBSD.ORG, Jilles Tjoelker Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 11:21:18 -0000 On Fri, Sep 02, 2011 at 02:04:34PM +0300, Andriy Gapon wrote: > on 02/09/2011 13:40 Jilles Tjoelker said the following: > > Ports people have complained about poll() behaviour before, are there > > configure scripts that attempt to check if we ever return POLLHUP alone > > and only check for POLLIN if not? > > Not sure about that other software and how POLLIN is related here. > The software in question (mozilla nspr) checks for POLLNVAL, POLLERR, POLLPRI > and POLLOUT to determine if anything interesting has happened to a connection > supposed to be in progress. They aren't checking for POLLHUP at all and thus > they keep thinking that the connection is still in progress when they get it. It seems for such case it should return POLLERR too. -- http://ache.vniz.net/ From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 12:59:58 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BD779106564A; Fri, 2 Sep 2011 12:59:58 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 9DA208FC19; Fri, 2 Sep 2011 12:59:57 +0000 (UTC) Received: from odyssey.starpoint.kiev.ua (alpha-e.starpoint.kiev.ua [212.40.38.101]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id PAA19362; Fri, 02 Sep 2011 15:59:55 +0300 (EEST) (envelope-from avg@FreeBSD.org) Message-ID: <4E60D34B.6050506@FreeBSD.org> Date: Fri, 02 Sep 2011 15:59:55 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:5.0) Gecko/20110705 Thunderbird/5.0 MIME-Version: 1.0 To: Andrey Chernov , Jilles Tjoelker , freebsd-net@FreeBSD.org, freebsd-standards@FreeBSD.org References: <4E60A1B8.7080607@FreeBSD.org> <20110902104018.GA12845@stack.nl> <4E60B842.8050506@FreeBSD.org> <20110902112116.GA23835@vniz.net> In-Reply-To: <20110902112116.GA23835@vniz.net> X-Enigmail-Version: 1.2pre Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Cc: Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 12:59:58 -0000 on 02/09/2011 14:21 Andrey Chernov said the following: > On Fri, Sep 02, 2011 at 02:04:34PM +0300, Andriy Gapon wrote: >> on 02/09/2011 13:40 Jilles Tjoelker said the following: >>> Ports people have complained about poll() behaviour before, are there >>> configure scripts that attempt to check if we ever return POLLHUP alone >>> and only check for POLLIN if not? >> >> Not sure about that other software and how POLLIN is related here. >> The software in question (mozilla nspr) checks for POLLNVAL, POLLERR, POLLPRI >> and POLLOUT to determine if anything interesting has happened to a connection >> supposed to be in progress. They aren't checking for POLLHUP at all and thus >> they keep thinking that the connection is still in progress when they get it. > > It seems for such case it should return POLLERR too. > I think that I would agree as this is not a graceful disconnect / hang-up but an error in trying to connect. Anyway I am not an expert in this matters and I'd think that POLLHUP should be checked anyway - better safe than sorry. -- Andriy Gapon From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 18:44:18 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id AA893106564A; Fri, 2 Sep 2011 18:44:18 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id A08888FC12; Fri, 2 Sep 2011 18:44:17 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id VAA22449; Fri, 02 Sep 2011 21:44:07 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QzYil-000Etz-1x; Fri, 02 Sep 2011 21:44:07 +0300 Message-ID: <4E6123F4.4010209@FreeBSD.org> Date: Fri, 02 Sep 2011 21:44:04 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.1) Gecko/20110901 Thunderbird/6.0.1 MIME-Version: 1.0 To: Bruce Evans References: <4E60A1B8.7080607@FreeBSD.org> <20110902104018.GA12845@stack.nl> <20110903015445.J957@besplex.bde.org> In-Reply-To: <20110903015445.J957@besplex.bde.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-net@FreeBSD.org, freebsd-standards@FreeBSD.org, Jilles Tjoelker Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 18:44:18 -0000 on 02/09/2011 20:36 Bruce Evans said the following: > It seems likely that there was a transient connection to cause this > problem. Do we know for sure? Not sure if this what you are asking but when I reproduced the problem it was by connecting to a local port where no one listened for sure. -- Andriy Gapon From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 20:08:12 2011 Return-Path: Delivered-To: freebsd-standards@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id BEA95106564A; Fri, 2 Sep 2011 20:08:12 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from fallbackmx08.syd.optusnet.com.au (fallbackmx08.syd.optusnet.com.au [211.29.132.10]) by mx1.freebsd.org (Postfix) with ESMTP id 4243C8FC15; Fri, 2 Sep 2011 20:08:11 +0000 (UTC) Received: from mail02.syd.optusnet.com.au (mail02.syd.optusnet.com.au [211.29.132.183]) by fallbackmx08.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p82HaMib008579; Sat, 3 Sep 2011 03:36:22 +1000 Received: from c122-106-165-191.carlnfd1.nsw.optusnet.com.au (c122-106-165-191.carlnfd1.nsw.optusnet.com.au [122.106.165.191]) by mail02.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p82HaHtX015971 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 3 Sep 2011 03:36:18 +1000 Date: Sat, 3 Sep 2011 03:36:17 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Jilles Tjoelker In-Reply-To: <20110902104018.GA12845@stack.nl> Message-ID: <20110903015445.J957@besplex.bde.org> References: <4E60A1B8.7080607@FreeBSD.org> <20110902104018.GA12845@stack.nl> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@freebsd.org, freebsd-standards@freebsd.org, Andriy Gapon Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 20:08:12 -0000 On Fri, 2 Sep 2011, Jilles Tjoelker wrote: > On Fri, Sep 02, 2011 at 12:28:24PM +0300, Andriy Gapon wrote: >> I see a problem where FreeBSD kernel (recent head) returns POLLHUP >> _alone_ (0x10) for a socket that has never been connected - a client >> socket for which connect(2) failed. There is also a piece of software >> that doesn't expect that flag and exhibits illogical behavior because >> of it. > > That would be a bug in that software. Returning POLLHUP alone is clearly > valid. I might be the author of this (kib committed my fixes). POLLHUP alone is certainly valid, but it is only supposed to happen after a disconnection. My tests (in /usr/src/tools/ regression/poll/pipepoll.c) check for this, but only for fifos. My fixes were mostly to make POLLHUP work at all (actually set it on hangup, so that old readers see it), and to make all the flags properly sticky for incomplete new connections (this includes POLLHUP). From pipepoll.c, about the old reader and a bug in the opposite direction (excessive dependency on POLLHUP in gdb). % /* % * Now we have no writer, but should still have data from the old % * writer. Check that we have both a data-readable condition and a % * hangup condition, and that the data can be read in the usual way. % * Since Linux does this, programs must not quit reading when they % * see POLLHUP; they must see POLLHUP without POLLIN (or another % * input condition) before they decide that there is EOF. gdb-6.1.1 % * is an example of a broken program that quits on POLLHUP only -- % * see its event-loop.c. % */ % ... Then for a new reader, with and without an old reader (this is only tested for fifos since the other supported cases don't have open() and always create a pair of fd's): % if (filetype == FT_FIFO) { % /* % * Check that POLLHUP is sticky for a new reader and for % * the old reader. % */ % fd2 = open(FIFONAME, O_RDONLY | O_NONBLOCK); % if (fd2 < 0) % err(1, "open for read"); % pfd.fd = fd2; % if (poll(&pfd, 1, 0) < 0) % err(1, "poll"); % report(num++, "6b", POLLHUP, pfd.revents); This checks that a new reader on a new fd (while the old reader is still active on the old fd) only sees only POLLHUP. It shouldn't see POLLIN since the test has arranged for there to be no data (otherwise it should see data from the old connection). It shouldn't see POLLOUT sice it is read-only (but see below). And it should see POLLHUP since there _was_ a connection and there isn't a new one yet, especially since the old reader still has it partially open. % pfd.fd = fd; % if (poll(&pfd, 1, 0) < 0) % err(1, "poll"); % report(num++, "6c", POLLHUP, pfd.revents); Check the same for the old reader. % close(fd2); % if (poll(&pfd, 1, 0) < 0) % err(1, "poll"); % report(num++, "6d", POLLHUP, pfd.revents); % } Check that POLLHUP persists even after the old reader goes away. IIRC, this might be different if the old reader goes away before the new reader arrives. It would be reasonable to clear POLLHUP when all readers, all writers and all data go away. But when there is data left, POLLIN must be kept to tell new readers that there is data, and POLLHUP must be kept to tell them that this data is old. The tests are very incomplete for sockets, and cover mainly fifos and normal pipes, with just socketpair() for sockets. The only change that I made to pipepoll.c after it was committed was to check POLLOUT. POLLOUT is normally set (it is set even for a r/o open()!?). But when POLLHUP is set, POLLOUT must be clear (IIRC this is about the one thing that is specified non-fuzzily by POSIX). Thus POLLOUT is expected to be clear when POLLHUP is expected to be set in the above checks, and applications don't need to check for POLLHUP if they only want to do output, and POLLHUP is almost unnecessary (since it almost tracks POLLHUP); POLLHUP is most useful in connection with POLLIN, especially if the fd is r/o and POLLOUT isn't bogusly set for r/o descriptors. There are the following states for POLLHUP | POLLIN: - clear: have a connection but no data - POLLIN: have a connection and data - POLLHUP: had a connection; have no data - POLLHUP | POLLIN: had a connection; have data left by it (or a previous one) Most differences with Linux and bugs are in this area. IIRC, Linux gets this right for some file types but not others. See the test output. >> This is how POSIX describes POLLHUP: >> POLLHUP >> The device has been disconnected. This event and POLLOUT are >> mutually-exclusive; a stream can never be writable if a hangup has >> occurred. However, this event and POLLIN, POLLRDNORM, POLLRDBAND, or >> POLLPRI are not mutually-exclusive. This flag is only valid in the >> revents bitmask; it shall be ignored in the events member. FreeBSD attempts to implement exactly this, with "has been" interpreted as strictly in the past, and the largest possible orthogonality in the setting oF POLLHUP vs the input flags. > Together with the description of POLLIN which says that data may be read > without blocking, this seems to mean that if POLLHUP is set alone, there > is no point in calling read() since it will return 0 or -1 immediately. > If POLLHUP and POLLIN are both set, there is still data in the buffer Indeed. I just remembered that the importance of these flags is more for how they affect the !O_NONBLOCK case that for polling in the O_NONBLOCK case (there is lots of redundancy for the latter). Only POLLIN is critical. If there is data, then a new !O_NONBLOCK open won't block, and read() won't block, so POLLIN must be set and poll() must not block to be consistent. When there is no data but there has been a disconnection, new readers normally block, but if they insist on using O_NONBLOCK for open(), then they will get a hint about the state from POLLHUP. IIRC, POLLHUP causes read() to not block, and the only hint that a reader gets that it should not spin trying the read is that POLLHUP but not POLLIN is set. This spin must be avoided by old readers, and keeping the same state for new readers is very delicate since some choices stop non-blocking opens from being usable to wait for a new writer. The choices made are supposed to be the least worse and the most compatible possible with Linux (see old PRs). Also, select() is more problematic than poll() since it can't return POLLHUP. It esentially has to OR POLLHUP into its input-ready condition, since it must not block() on input if there is a hangup but no data. ISTR that these semantics had to be modified a little so that it doesn't block for an old hangup, though the old hangup can still be seen using poll(). See the select() tests. > (in FreeBSD, POLLHUP | POLLIN is often returned even if there is no > data, to keep buggy applications from breaking). Is it? I think I fixed this in my version. kib didn't want to commit something; maybe it was this. This breaks non-buggy applications that depend on POLLIN being cleared to detect EOF. POLLHUP cannot be used to detect EOF (see the above comment about gdb (gdb loses the final input even on pipes for operations like "echo $command | gdb" when the writer happens to close the pipe first)). Neither poll() nor read() will block if either POLLHUP or POLLIN is set, so POLLIN must be clear to avoid an endless spin trying to read EOF if poll() is trusted. select() can't tell the difference between hangup and no data, so appliations using it must use another method to avoid the spin. Old versions of gdb used select() and then read() returning 0 as giving EOF. This works in most cases, but has races if there are multple readers. > A complicating factor is that there are situations where read() returns > 0, but yet it is not a hangup. For example, if a user types the EOF > character (such as ^D) in a terminal, this causes read() to return 0 but > the terminal remains writable (and writing to it may even block). This > is the "zero length message" which is under the obsolescent STREAMS > marking, even though it applies to terminals in all implementations. Old gdb works on terminals. This is because select() knows the difference between no data on a terminal and the EOF caused by a disconnection, so it blocks if there is no data and no disconnection. When it succeeds, it means that there is data or a disconnection and then read() returning 0 means that it is a disconnection, provided there is only 1 reader and no kernel conditions that clear the data. The "zero length message" stuff is part of the fuzziness in POSIX. The specification is less incomplete for STREAMS, but for other file types it is unclear whether zero length data is data. I think it has to count as data for select() but not for poll(), since select() doesn't have POLLHUP so it has no cause other than null data available to make it return on disconnection, while for poll() counting null data as data makes it impossible for poll() to block waiting for data after a disconnection and thus "fix" the problem with select() not being able to block. EIther choice causes problems and I forget exactly what is done now. > Similarly, if the other side has done the equivalent of > shutdown(SHUT_WR), read() will return 0 but POLLHUP must not be returned > as it would prevent the application from waiting for space in the > outgoing socket buffer; therefore POLLIN must be returned. Similar > considerations apply to other shutdown() calls (but note that > shutdown(SHUT_RD) is not sent to the other side). I'm not very familiar with user-mode socket programming. In the kernel, POLLHUP mostly follows SBS_CANTRCVMORE and SBS_CANTSENDMORE, and these flags work very orthogonally, more so than is possible with POLLHUP. Hmm, don't user-mode sockets (except bound ones?) go away when all the readers and writers go away, unlike for fifos? So the case of new readers with no old readers on an old connection can't happen. > The other way around, read() may fail after poll() did not return > POLLHUP if the connection had closed in the meantime. > >> For me "disconnected" _implies_ that the device should have been >> connected first. But this is not explicitly said anywhere. Also, I >> think it's possible that a socket gets connected and immediately >> disconnected (before poll(2) is called), then the POLLHUP would be >> appropriate in any interpretation. So, I am inclined to think that >> the software should check for POLLHUP. But I would like to ask your >> opinion since the problem appears to be FreeBSD-specific. > > There was a connection attempt in progress that failed at some point, > which seems close enough to "disconnected". Alternatively, POLLERR could > be set but could activate more bugs in applications. It seems likely that there was a transient connection to cause this problem. Do we know for sure? I don't like using POLLERR since it provides almost no information. A transient connection isn't an error. > Comparisons with things like "connected and immediately disconnected" > and "connection closed after poll() returned something" are useful in > reasoning about this. The former isn't really an error. Only failure establishing a connection might be an error. But can that even give a connection? Bruce From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 20:26:22 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 9A3141065672; Fri, 2 Sep 2011 20:26:22 +0000 (UTC) (envelope-from brde@optusnet.com.au) Received: from mail07.syd.optusnet.com.au (mail07.syd.optusnet.com.au [211.29.132.188]) by mx1.freebsd.org (Postfix) with ESMTP id 32DA28FC08; Fri, 2 Sep 2011 20:26:21 +0000 (UTC) Received: from c122-106-165-191.carlnfd1.nsw.optusnet.com.au (c122-106-165-191.carlnfd1.nsw.optusnet.com.au [122.106.165.191]) by mail07.syd.optusnet.com.au (8.13.1/8.13.1) with ESMTP id p82KQJi0032442 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Sat, 3 Sep 2011 06:26:20 +1000 Date: Sat, 3 Sep 2011 06:26:19 +1000 (EST) From: Bruce Evans X-X-Sender: bde@besplex.bde.org To: Andriy Gapon In-Reply-To: <4E6123F4.4010209@FreeBSD.org> Message-ID: <20110903053813.V2093@besplex.bde.org> References: <4E60A1B8.7080607@FreeBSD.org> <20110902104018.GA12845@stack.nl> <20110903015445.J957@besplex.bde.org> <4E6123F4.4010209@FreeBSD.org> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII; format=flowed Cc: freebsd-net@FreeBSD.org, freebsd-standards@FreeBSD.org, Jilles Tjoelker Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 20:26:22 -0000 On Fri, 2 Sep 2011, Andriy Gapon wrote: > on 02/09/2011 20:36 Bruce Evans said the following: >> It seems likely that there was a transient connection to cause this >> problem. Do we know for sure? > > Not sure if this what you are asking but when I reproduced the problem it was by > connecting to a local port where no one listened for sure. Yes, that's what I'm asking. It should be simpler to fix if it is just a bug and doesn't involve reader/writer races. The code for setting POLLHUP for sockets is simple: from uipc_socket.c: % if ((events & POLLINIGNEOF) == 0) { % if (so->so_rcv.sb_state & SBS_CANTRCVMORE) { % revents |= events & (POLLIN | POLLRDNORM); % if (so->so_snd.sb_state & SBS_CANTSENDMORE) % revents |= POLLHUP; % } % } So POLLHUP tracks SBS_CANTRCVMORE && SBS_CANTSENDMORE exactly, and POLLHUP is never set without POLLIN. But in my version: % if ((events & POLLINIGNEOF) == 0) { % if (so->so_rcv.sb_state & SBS_CANTRCVMORE) { % if (so->so_snd.sb_state & SBS_CANTSENDMORE) % revents |= POLLHUP; % else % revents |= events & (POLLIN | POLLRDNORM); % } % } POLLIN is not set by hangup, bit actually means that there is (non-null) input available. jilles referred to this bug in -current. POLLINIGNEOF is an old attempt to fix this. It gives the caller more control. It works poorly, especially since most callers don't know that it exists. Perhaps SBS_CANTRCVMORE && SBS_CANTSENDMORE condition is too raw/low-level even for sockets. Fifos need more to get the behaviour that they want (as similar as possible to Linux). I quote my hackish version since the version in -current doesn't have any comments and my version does even more: >From fifo_open(), for readers: % mtx_lock(&fifo_mtx); % if (ap->a_mode & FREAD) { % fip->fi_readers++; % if (fip->fi_readers == 1) { % SOCKBUF_LOCK(&fip->fi_writesock->so_snd); % fip->fi_writesock->so_snd.sb_state &= ~SBS_CANTSENDMORE; % SOCKBUF_UNLOCK(&fip->fi_writesock->so_snd); % if (fip->fi_writers > 0) { % wakeup(&fip->fi_writers); % sowwakeup(fip->fi_writesock); % } % } % fdp = td->td_proc->p_fd; % FILEDESC_LOCK(fdp); % fp = fget_locked(fdp, ap->a_fdidx); % /* Abuse f_msgcount as a generation count. */ % fp->f_msgcount = fip->fi_wgen - fip->fi_writers; -current has the above line, and fixes the hack using a new field f_seqcount. % /* % * XXX quick fix: keep any POLLHUP condition sticky if we % * now have more than 1 reader, by clobbering f_msgcount % * here so that POLLINIGNEOF will not be set in % * fifo_poll_f(). (The above subtraction of fi_writers % * works similarly but is probably unnecessary since if % * there is a writer then there won't be a POLLHUP % * condition.) % * % * XXX: the generation counts are probably unnecessary now % * that this is broken to POSIX spec. They are to help % * make any POLLHUP condition sticky only for old readers. % * % * XXX: does this hack work right for a new reader that will % * block? % */ % fp->f_msgcount -= fip->fi_readers - 1; -current doesn't have the above line or the big comment. I forget if this is for more compatibility with Linux, or more correctness. I think the committed regression tests assert the behaviour give by this version, so they will certainly fail for -current. I'm surer of this for the spurious POLLIN with POLLHUP. % FILEDESC_UNLOCK(fdp); % } % int % fifo_poll_f(struct file *fp, int events, struct ucred *cred, struct thread *td) % { % struct file filetmp; % struct fifoinfo *fip; % int levents, revents = 0; % % fip = fp->f_vnode->v_fifoinfo; % levents = events & (POLLIN | POLLPRI | POLLRDNORM | POLLRDBAND); % if (levents) { % filetmp.f_data = fip->fi_readsock; % filetmp.f_cred = fp->f_cred; % if (fp->f_msgcount == fip->fi_wgen) % levents |= POLLINIGNEOF; -current has this (with s/msgcount/seqcount/). This is the only place that seqcount is used. I thought I removed all uses of POLLINIGNEOF in the kernel. % revents |= soo_poll(&filetmp, levents, cred, td); % } % levents = events & (POLLOUT | POLLWRNORM | POLLWRBAND); % if (levents) { % filetmp.f_data = fip->fi_writesock; % filetmp.f_cred = fp->f_cred; % revents |= soo_poll(&filetmp, levents, cred, td); Note that fifo polling calls soo_poll() twice, and it only uses POLLINIGNEOF for the second call. Thus the combination always sets POLLHUP if there is a disconnection. But the second call doesn't ask for POLLIN, so in -current there is no spurious POLLIN to echo POLLHUP in the case where POLLINIGNEOF is used, but there is if it is not. -current is subtly different. Where the above masks out POLLINIGNEOF in the user `events' in the first assignment to levents, -current keeps it. But both versions mask it for the second assignment. Thus my version always ignores the user POLLINIGNEOF for polling of fifos, but -current keeps it partially. -current thus sees POLLHUP and fails to completely ignore EOF even if the user passes POLLINIGNEOF, iff the user events also has one of the output flags set. The user would have to call poll() twice, first with only input flags set and then with output flags set get unmixed behaviour, but this won't work since either of the calls may block and the output check will never ignore EOF. % } % return (revents); % } Bruce From owner-freebsd-standards@FreeBSD.ORG Fri Sep 2 21:09:35 2011 Return-Path: Delivered-To: freebsd-standards@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id 4F9F2106564A; Fri, 2 Sep 2011 21:09:35 +0000 (UTC) (envelope-from avg@FreeBSD.org) Received: from citadel.icyb.net.ua (citadel.icyb.net.ua [212.40.38.140]) by mx1.freebsd.org (Postfix) with ESMTP id 61C568FC0C; Fri, 2 Sep 2011 21:09:33 +0000 (UTC) Received: from porto.starpoint.kiev.ua (porto-e.starpoint.kiev.ua [212.40.38.100]) by citadel.icyb.net.ua (8.8.8p3/ICyb-2.3exp) with ESMTP id AAA23623; Sat, 03 Sep 2011 00:09:21 +0300 (EEST) (envelope-from avg@FreeBSD.org) Received: from localhost ([127.0.0.1]) by porto.starpoint.kiev.ua with esmtp (Exim 4.34 (FreeBSD)) id 1QzazI-000Ezc-UR; Sat, 03 Sep 2011 00:09:20 +0300 Message-ID: <4E6145FF.3090803@FreeBSD.org> Date: Sat, 03 Sep 2011 00:09:19 +0300 From: Andriy Gapon User-Agent: Mozilla/5.0 (X11; FreeBSD amd64; rv:6.0.1) Gecko/20110901 Thunderbird/6.0.1 MIME-Version: 1.0 To: Bruce Evans References: <4E60A1B8.7080607@FreeBSD.org> <20110902104018.GA12845@stack.nl> <20110903015445.J957@besplex.bde.org> <4E6123F4.4010209@FreeBSD.org> <20110903053813.V2093@besplex.bde.org> In-Reply-To: <20110903053813.V2093@besplex.bde.org> X-Enigmail-Version: undefined Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Cc: freebsd-net@FreeBSD.org, freebsd-standards@FreeBSD.org, Jilles Tjoelker Subject: Re: POLLHUP on never connected socket X-BeenThere: freebsd-standards@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Standards compliance List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 02 Sep 2011 21:09:35 -0000 on 02/09/2011 23:26 Bruce Evans said the following: > Yes, that's what I'm asking. It should be simpler to fix if it is just > a bug and doesn't involve reader/writer races. The code for setting > POLLHUP for sockets is simple: from uipc_socket.c: > > % if ((events & POLLINIGNEOF) == 0) { > % if (so->so_rcv.sb_state & SBS_CANTRCVMORE) { > % revents |= events & (POLLIN | POLLRDNORM); > % if (so->so_snd.sb_state & SBS_CANTSENDMORE) > % revents |= POLLHUP; > % } > % } > > So POLLHUP tracks SBS_CANTRCVMORE && SBS_CANTSENDMORE exactly, and POLLHUP > is never set without POLLIN. Umm, it seems that when you say that then you are assuming that the events contain POLLIN. In what I observe events == POLLOUT | POLLPRI and thus only POLLHUP is set in the revents. Not that this means too much, just an observation. > But in my version: > > % if ((events & POLLINIGNEOF) == 0) { > % if (so->so_rcv.sb_state & SBS_CANTRCVMORE) { > % if (so->so_snd.sb_state & SBS_CANTSENDMORE) > % revents |= POLLHUP; > % else > % revents |= events & (POLLIN | POLLRDNORM); > % } > % } > > POLLIN is not set by hangup, bit actually means that there is (non-null) > input available. jilles referred to this bug in -current. -- Andriy Gapon