From owner-freebsd-stable@FreeBSD.ORG Mon Jun 28 18:01:32 2010 Return-Path: Delivered-To: freebsd-stable@FreeBSD.org Received: from [127.0.0.1] (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by hub.freebsd.org (Postfix) with ESMTP id 8D7E7106566B; Mon, 28 Jun 2010 18:01:32 +0000 (UTC) (envelope-from jkim@FreeBSD.org) From: Jung-uk Kim To: freebsd-stable@FreeBSD.org Date: Mon, 28 Jun 2010 14:01:16 -0400 User-Agent: KMail/1.6.2 References: <20100623025855.82916.qmail@exxodus.fedaykin.here> <201006251758.56393.jkim@FreeBSD.org> <4C25C3D3.3030109@FreeBSD.org> In-Reply-To: <4C25C3D3.3030109@FreeBSD.org> MIME-Version: 1.0 Content-Disposition: inline Content-Type: Multipart/Mixed; boundary="Boundary-00=_uNOKM6vkj5rWLGa" Message-Id: <201006281401.19017.jkim@FreeBSD.org> Cc: d@delphij.net, python@freebsd.org, Mario Sergio Fujikawa Ferreira Subject: Re: FreeBSD 8.1-PRERELEASE: WARNING ioctl sign-extension ioctl ffffffff8004667e X-BeenThere: freebsd-stable@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Production branch of FreeBSD source code List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 28 Jun 2010 18:01:32 -0000 --Boundary-00=_uNOKM6vkj5rWLGa Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit Content-Disposition: inline On Saturday 26 June 2010 05:09 am, Mario Sergio Fujikawa Ferreira wrote: > On 25/06/2010 18:58, Jung-uk Kim wrote: > > On Friday 25 June 2010 04:54 am, Mario Sergio Fujikawa Ferreira wrote: > >> On Wed, Jun 23, 2010 at 05:08:38PM -0400, Jung-uk Kim wrote: > >>> Date: Wed, 23 Jun 2010 17:08:38 -0400 > >>> From: Jung-uk Kim > >>> To: freebsd-stable@FreeBSD.org > >>> Cc: d@delphij.net, Mario Sergio Fujikawa Ferreira > >>> Subject: Re: FreeBSD 8.1-PRERELEASE: > >>> WARNING ioctl sign-extension ioctl ffffffff8004667e > >>> User-Agent: KMail/1.6.2 > >>> > >>> On Wednesday 23 June 2010 03:38 pm, Jung-uk Kim wrote: > >>>> On Wednesday 23 June 2010 03:10 pm, Jung-uk Kim wrote: > >>>>> On Wednesday 23 June 2010 03:01 pm, Jung-uk Kim wrote: > >>>>>> On Wednesday 23 June 2010 02:41 pm, Xin LI wrote: > >>>>>>> On 2010/06/23 11:37, Jung-uk Kim wrote: > >>>>>>>> On Wednesday 23 June 2010 02:12 pm, Xin LI wrote: > >>>>>>>>> Hi, > >>>>>>>>> > >>>>>>>>> On 2010/06/22 19:58, Mario Sergio Fujikawa Ferreira > > > > wrote: > >>>>>>>>>> Hi, > >>>>>>>>>> > >>>>>>>>>> I am getting more than 4 thousand of the following > >>>>>>>>>> messages a day: > >>>>>>>>>> > >>>>>>>>>> WARNING pid 24509 (python2.6): ioctl sign-extension > >>>>>>>>>> ioctl ffffffff8004667e > >>>>>>>>> > >>>>>>>>> [...] > >>>>>>>>> > >>>>>>>>> I think we may need to check the code and patch it. > >>>>>>>>> Basically this means that python (or some .so modules) > >>>>>>>>> passed an int or unsigned int as parameter 'cmd', we > >>>>>>>>> need to change it to unsigned long. > >>>>>>>>> > >>>>>>>>> The warning itself should be harmless to my best of > >>>>>>>>> knowledge, one can probably remove the printf in > >>>>>>>>> kernel source code as a workaround. > >>>>>>>>> > >>>>>>>>> By the way it seems to be a POSIX violation and we > >>>>>>>>> didn't seem to really use so wide cmd, but I have not > >>>>>>>>> yet verified everything myself. > >>>>>>>> > >>>>>>>> Long time ago, I had a similar problem with termios > >>>>>>>> TIOCGWINSZ and we patched the port like this: > >>>>>>>> > >>>>>>>> http://www.freebsd.org/cgi/cvsweb.cgi/ports/lang/python > >>>>>>>> /fil es /A tt > >>>>>>>> ic/patch-Modules%3A%3Afcntlmodule.c?rev=1.1;content-typ > >>>>>>>> e=te xt %2 Fpl ain > >>>>>>>> > >>>>>>>> I believe it was upstream patched at the time but I > >>>>>>>> won't be surprised if something similar was > >>>>>>>> reintroduced. It happens when a Python internal > >>>>>>>> integer type is converted to a native unsigned long. > >>>>>>> > >>>>>>> Well, only *BSD have cmd a long value so it's likely that > >>>>>>> it would be reintroduced. > >>>>>> > >>>>>> Yes, that's what I mean. > >>>>>> > >>>>>>> I have checked the 4.4BSD archive and understood that our > >>>>>>> ioctl's cmd parameter was made long around 1991 or 1992s > >>>>>>> but didn't see what it actually buy us... > >>>>>> > >>>>>> Like it or not, it is too late to revert. :-( > >>>>> > >>>>> From Python 2.6.5: > >>>>> > >>>>> static PyObject * > >>>>> fcntl_ioctl(PyObject *self, PyObject *args) > >>>>> { > >>>>> #define IOCTL_BUFSZ 1024 > >>>>> int fd; > >>>>> /* In PyArg_ParseTuple below, we use the unsigned > >>>>> non-checked 'I' format for the 'code' parameter because > >>>>> Python turns 0x8000000 into either a large positive number > >>>>> (PyLong or PyInt on 64-bit platforms) or a negative number on > >>>>> others (32-bit PyInt) whereas the system expects it to be a > >>>>> 32bit bit field value regardless of it being passed as an int > >>>>> or unsigned long on various platforms. See the > >>>>> termios.TIOCSWINSZ constant across platforms for an example > >>>>> of thise. > >>>>> > >>>>> If any of the 64bit platforms ever decide to use more > >>>>> than 32bits in their unsigned long ioctl codes this will > >>>>> break and need special casing based on the platform being > >>>>> built on. */ > >>>>> unsigned int code; > >>>>> ... > >>>>> > >>>>> It is still a kludge and it won't be fixed. :-( > >>>> > >>>> Please drop the attached patch in ports/lang/python26/files > >>>> and test. Note I am not a Python guy, so please don't shoot > >>>> me. ;-) > >>> > >>> I found that Python guys added 'k' format since 2.3, which > >>> converts a Python integer to unsigned long. Please ignore the > >>> previous patch and try the attached patch instead. > >> > >> Unfortunately it didn't work. > >> > >> 1) Added patch to lang/python26 > >> 2) Recompiled lang/python26 > >> 3) cd /var/db/ports&& portupgrade -f python* py26* deluge* > >> > >> Restarted deluge afterwards. The message is still there: > >> > >> Jun 25 05:49:38 exxodus kernel: WARNING pid 38635 (python2.6): > >> ioctl sign-extension ioctl ffffffff8004667e > > > > It may be a deeper problen, then. :-( > > > > First of all, I cannot reproduce the problem because deluged > > dumps core on my box and I gave it up. Just staring at the code, > > it seems they rely on a bundled libtorrent for Python binding and > > the libtorrent relies on Boost, in turn. Then, the actual > > non-blocking socket implementation is done with Boost ASIO[1]. > > It is possible that there are more complicated problems between > > these and the Python binding. In fact, I can see a lot of these: > > > > int name() const > > { > > return FIONBIO; > > } > > ... > > if (!ec&& command.name() == static_cast(FIONBIO)) > > ... > > socket_ops::ioctl(impl.socket_, command.name(), ...) > > ... > > > > They are just assuming it is an int type, I guess. > > > > Sigh, what a mess... > > Given that your python patch is a step on the right direction, I > would propose that it be further tested and then committed if > possible. > > > [1] Boost and its Python binding from ports seems to be a newer > > ASIO version than the bundled ASIO headers. Probably it is a > > reason for the crash but I didn't bother too much. > > If you have the time, everything you need to run the latest deluge > 1.3.0 RC1 can be found at > > http://www.freebsd.org/cgi/query-pr.cgi?pr=144337 > > Check the PR, all the necessary ports can be found there. Let me > know what you think. Please drop the attached patch in ports/devel/boost-libs/files, rebuild all dependencies, and try your deluge ports again[1]. Jung-uk Kim [1] Your libtorrent Python slave port and deluge ports don't build/package cleanly. I guess you need to update the PR's. --Boundary-00=_uNOKM6vkj5rWLGa Content-Type: text/plain; charset="iso-8859-1"; name="patch-boost_asio-ioctl" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="patch-boost_asio-ioctl" --- boost/asio/detail/io_control.hpp.orig 2010-01-04 04:36:00.000000000 -0500 +++ boost/asio/detail/io_control.hpp 2010-06-25 18:38:28.000000000 -0400 @@ -46,7 +46,7 @@ public: } // Get the name of the IO control command. - int name() const + ioctl_cmd_type name() const { return FIONBIO; } @@ -96,7 +96,7 @@ public: } // Get the name of the IO control command. - int name() const + ioctl_cmd_type name() const { return FIONREAD; } --- boost/asio/detail/reactive_descriptor_service.hpp.orig 2010-06-25 18:24:52.000000000 -0400 +++ boost/asio/detail/reactive_descriptor_service.hpp 2010-06-25 18:56:32.000000000 -0400 @@ -223,7 +223,7 @@ public: // descriptor is put into the state that has been requested by the user. If // the ioctl syscall was successful then we need to update the flags to // match. - if (!ec && command.name() == static_cast(FIONBIO)) + if (!ec && command.name() == static_cast(FIONBIO)) { if (*static_cast(command.data())) { --- boost/asio/detail/reactive_socket_service.hpp.orig 2010-04-07 04:44:41.000000000 -0400 +++ boost/asio/detail/reactive_socket_service.hpp 2010-06-25 18:55:06.000000000 -0400 @@ -453,7 +453,7 @@ public: // already in the correct state. This ensures that the underlying socket // is put into the state that has been requested by the user. If the ioctl // syscall was successful then we need to update the flags to match. - if (!ec && command.name() == static_cast(FIONBIO)) + if (!ec && command.name() == static_cast(FIONBIO)) { if (*static_cast(command.data())) { --- boost/asio/detail/win_iocp_socket_service.hpp.orig 2010-03-29 21:20:37.000000000 -0400 +++ boost/asio/detail/win_iocp_socket_service.hpp 2010-06-25 18:55:49.000000000 -0400 @@ -564,7 +564,7 @@ public: socket_ops::ioctl(impl.socket_, command.name(), static_cast(command.data()), ec); - if (!ec && command.name() == static_cast(FIONBIO)) + if (!ec && command.name() == static_cast(FIONBIO)) { if (*static_cast(command.data())) impl.flags_ |= implementation_type::user_set_non_blocking; --- boost/asio/detail/descriptor_ops.hpp.orig 2010-01-04 04:36:00.000000000 -0500 +++ boost/asio/detail/descriptor_ops.hpp 2010-06-25 18:37:37.000000000 -0400 @@ -110,7 +110,7 @@ inline int gather_write(int d, const buf return result; } -inline int ioctl(int d, long cmd, ioctl_arg_type* arg, +inline int ioctl(int d, ioctl_cmd_type cmd, ioctl_arg_type* arg, boost::system::error_code& ec) { clear_error(ec); --- boost/asio/detail/socket_ops.hpp.orig 2010-01-04 06:55:09.000000000 -0500 +++ boost/asio/detail/socket_ops.hpp 2010-06-25 18:39:55.000000000 -0400 @@ -596,7 +596,7 @@ inline int getsockname(socket_type s, so return result; } -inline int ioctl(socket_type s, long cmd, ioctl_arg_type* arg, +inline int ioctl(socket_type s, ioctl_cmd_type cmd, ioctl_arg_type* arg, boost::system::error_code& ec) { clear_error(ec); --- boost/asio/detail/socket_types.hpp.orig 2010-03-21 05:39:26.000000000 -0400 +++ boost/asio/detail/socket_types.hpp 2010-06-25 18:48:43.000000000 -0400 @@ -189,6 +189,12 @@ typedef sockaddr_in6 sockaddr_in6_type; typedef sockaddr_storage sockaddr_storage_type; typedef sockaddr_un sockaddr_un_type; typedef addrinfo addrinfo_type; +#if (defined(__MACH__) && defined(__APPLE__)) || defined(__DragonFly__) || \ + define(__FreeBSD__) || defined(__NetBSD__) || defined(__OpenBSD__) +typedef unsigned long ioctl_cmd_type; +#else +typedef int ioctl_cmd_type; +#endif typedef int ioctl_arg_type; typedef uint32_t u_long_type; typedef uint16_t u_short_type; --Boundary-00=_uNOKM6vkj5rWLGa--