From owner-freebsd-ports-bugs@FreeBSD.ORG Sun Sep 2 10:40:07 2012 Return-Path: Delivered-To: freebsd-ports-bugs@hub.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:4f8:fff6::34]) by hub.freebsd.org (Postfix) with ESMTP id C250C106566B for ; Sun, 2 Sep 2012 10:40:07 +0000 (UTC) (envelope-from gnats@FreeBSD.org) Received: from freefall.freebsd.org (freefall.freebsd.org [IPv6:2001:4f8:fff6::28]) by mx1.freebsd.org (Postfix) with ESMTP id 9B1DD8FC1B for ; Sun, 2 Sep 2012 10:40:07 +0000 (UTC) Received: from freefall.freebsd.org (localhost [127.0.0.1]) by freefall.freebsd.org (8.14.5/8.14.5) with ESMTP id q82Ae75i089687 for ; Sun, 2 Sep 2012 10:40:07 GMT (envelope-from gnats@freefall.freebsd.org) Received: (from gnats@localhost) by freefall.freebsd.org (8.14.5/8.14.5/Submit) id q82Ae7ur089686; Sun, 2 Sep 2012 10:40:07 GMT (envelope-from gnats) Resent-Date: Sun, 2 Sep 2012 10:40:07 GMT Resent-Message-Id: <201209021040.q82Ae7ur089686@freefall.freebsd.org> Resent-From: FreeBSD-gnats-submit@FreeBSD.org (GNATS Filer) Resent-To: freebsd-ports-bugs@FreeBSD.org Resent-Reply-To: FreeBSD-gnats-submit@FreeBSD.org, Vitaly Magerya Received: from mx1.freebsd.org (mx1.freebsd.org [69.147.83.52]) by hub.freebsd.org (Postfix) with ESMTP id 17B33106566B for ; Sun, 2 Sep 2012 10:35:59 +0000 (UTC) (envelope-from nobody@FreeBSD.org) Received: from red.freebsd.org (red.freebsd.org [IPv6:2001:4f8:fff6::22]) by mx1.freebsd.org (Postfix) with ESMTP id EB91C8FC08 for ; Sun, 2 Sep 2012 10:35:58 +0000 (UTC) Received: from red.freebsd.org (localhost [127.0.0.1]) by red.freebsd.org (8.14.4/8.14.4) with ESMTP id q82AZwcg017324 for ; Sun, 2 Sep 2012 10:35:58 GMT (envelope-from nobody@red.freebsd.org) Received: (from nobody@localhost) by red.freebsd.org (8.14.4/8.14.4/Submit) id q82AZwdJ017319; Sun, 2 Sep 2012 10:35:58 GMT (envelope-from nobody) Message-Id: <201209021035.q82AZwdJ017319@red.freebsd.org> Date: Sun, 2 Sep 2012 10:35:58 GMT From: Vitaly Magerya To: freebsd-gnats-submit@FreeBSD.org X-Send-Pr-Version: www-3.1 Cc: Subject: ports/171246: [patch] make python curses module work with unicode X-BeenThere: freebsd-ports-bugs@freebsd.org X-Mailman-Version: 2.1.5 Precedence: list List-Id: Ports bug reports List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 02 Sep 2012 10:40:07 -0000 >Number: 171246 >Category: ports >Synopsis: [patch] make python curses module work with unicode >Confidential: no >Severity: non-critical >Priority: low >Responsible: freebsd-ports-bugs >State: open >Quarter: >Keywords: >Date-Required: >Class: sw-bug >Submitter-Id: current-users >Arrival-Date: Sun Sep 02 10:40:07 UTC 2012 >Closed-Date: >Last-Modified: >Originator: Vitaly Magerya >Release: FreeBSD 9.1-RC1 amd64 >Organization: >Environment: >Description: I'm having a problem with Python curses module: it won't work with unicode. Here's a test case: import curses import locale locale.setlocale(locale.LC_ALL, "") def run(win): win.addstr(u"\u03c0r\u00b2".encode("utf-8")) win.getch() curses.wrapper(run) What I expect to see (and what I in fact see on e.g. Linux) is "πrē" (that is, Greek "pi", Latin "r", superscript "2"), what I actually get is "M-O~@rM-BM-2". (Note that this is with LC_ALL set to "en_US.UTF-8" everywhere; I'm seeing the same output in uxterm and putty, with and without tmux inside of those). I think this happens because python curses library is linked with libncurses, not with libncursesw as it is on other platforms: $ ldd /usr/local/lib/python2.7/lib-dynload/_curses.so /usr/local/lib/python2.7/lib-dynload/_curses.so: libncurses.so.8 => /lib/libncurses.so.8 (0x801212000) libthr.so.3 => /lib/libthr.so.3 (0x80145f000) libc.so.7 => /lib/libc.so.7 (0x80084a000) In fact, this appears to be intentional; here's a part of lang/python27/files/patch-setup.py: @@ -642,7 +642,7 @@ # use the same library for the readline and curses modules. if 'curses' in readline_termcap_library: curses_library = readline_termcap_library - elif self.compiler.find_library_file(lib_dirs, 'ncursesw'): + elif self.compiler.find_library_file(lib_dirs, 'xxxncursesw'): curses_library = 'ncursesw' elif self.compiler.find_library_file(lib_dirs, 'ncurses'): curses_library = 'ncurses' @@ -1246,12 +1248,13 @@ # provided by the ncurses library. panel_library = 'panel' if curses_library.startswith('ncurses'): - if curses_library == 'ncursesw': + if curses_library == 'xxxncursesw': # Bug 1464056: If _curses.so links with ncursesw, # _curses_panel.so must link with panelw. panel_library = 'panelw' After some digging through commit logs, I found that this change originated in [1] as a fix for ports/99496 [2]. Note that this patch is a no-op in 2.7 and in 3.2, so it can be safely removed: a few lines above the first chunk of the patch setup.py determines which curses library does readline use (libncurses.so in FreeBSD), and uses that itself (unless readline uses libtinfo, in which case python links with libncursesw). More importantly though, if I remove all that logic and simply set "curses_library = 'ncursesw'", thus making _curses.so link with libncursesw.so, nothing breaks: python builds and runs fine, and my example above works as expected too. Even if I have devel/ncurses installed (as described in ports/99496), _curses.so still links with libncursesw.so from base and everything seems to work. In short, I'm proposing a patch (it's in the attachment and also at [3]) against lang/python{27,32} that removes all that logic and makes _curses.so link with libncursesw from base. As you can see from redports logs at [4], it builds fine on all releases. I also tested patched python27 with devel/ncurses installed, and that works too. Final note: do contact me if there is something wrong with the patch. I am determined to make curses work with Unicode. [1] http://www.freebsd.org/cgi/cvsweb.cgi/ports/lang/python24/files/Attic/patch-setup.py#rev1.8 [2] http://www.freebsd.org/cgi/query-pr.cgi?pr=99496 [3] http://redports.org/changeset/6571 [4] http://redports.org/buildarchive/20120831205350-31320/ >How-To-Repeat: >Fix: Patch attached with submission follows: Index: /magv/lang/python32/files/patch-setup.py =================================================================== --- /magv/lang/python32/files/patch-setup.py (revision 6570) +++ /magv/lang/python32/files/patch-setup.py (revision 6571) @@ -15,5 +15,5 @@ curses_library = readline_termcap_library - elif self.compiler.find_library_file(lib_dirs, 'ncursesw'): -+ elif self.compiler.find_library_file(lib_dirs, 'XXXncursesw'): ++ if self.compiler.find_library_file(lib_dirs, 'ncursesw'): curses_library = 'ncursesw' elif self.compiler.find_library_file(lib_dirs, 'ncurses'): @@ -32,6 +32,5 @@ panel_library = 'panel' if curses_library.startswith('ncurses'): -- if curses_library == 'ncursesw': -+ if curses_library == 'XXXncursesw': + if curses_library == 'ncursesw': # Bug 1464056: If _curses.so links with ncursesw, # _curses_panel.so must link with panelw. Index: /magv/lang/python27/files/patch-setup.py =================================================================== --- /magv/lang/python27/files/patch-setup.py (revision 6570) +++ /magv/lang/python27/files/patch-setup.py (revision 6571) @@ -15,5 +15,5 @@ curses_library = readline_termcap_library - elif self.compiler.find_library_file(lib_dirs, 'ncursesw'): -+ elif self.compiler.find_library_file(lib_dirs, 'xxxncursesw'): ++ if self.compiler.find_library_file(lib_dirs, 'ncursesw'): curses_library = 'ncursesw' elif self.compiler.find_library_file(lib_dirs, 'ncurses'): @@ -50,6 +50,5 @@ panel_library = 'panel' if curses_library.startswith('ncurses'): -- if curses_library == 'ncursesw': -+ if curses_library == 'xxxncursesw': + if curses_library == 'ncursesw': # Bug 1464056: If _curses.so links with ncursesw, # _curses_panel.so must link with panelw. >Release-Note: >Audit-Trail: >Unformatted: