From owner-freebsd-i18n@FreeBSD.ORG Wed Feb 27 21:32:11 2013 Return-Path: Delivered-To: freebsd-i18n@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id 3E442E4D for ; Wed, 27 Feb 2013 21:32:11 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id 23D4B144 for ; Wed, 27 Feb 2013 21:32:07 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r1RLW13O008631 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 27 Feb 2013 13:32:01 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r1RLW1lY008630 for freebsd-i18n@FreeBSD.org; Wed, 27 Feb 2013 13:32:01 -0800 (PST) (envelope-from jmg) Date: Wed, 27 Feb 2013 13:32:00 -0800 From: John-Mark Gurney To: freebsd-i18n@FreeBSD.org Subject: patch for in tree iconv Message-ID: <20130227213200.GB55866@funkthat.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Wed, 27 Feb 2013 13:32:01 -0800 (PST) X-BeenThere: freebsd-i18n@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: FreeBSD Internationalization Effort List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Feb 2013 21:32:11 -0000 I've worked at fixing a few of iconv's locking issues, and I've attached this patch. This should fix most/all of the locking related issues relating to the global lock... Though I ran the tests in tools/test/iconv, and it looks like the checked in reference conversion in ref are wrong... The UTF-32 -> ASCII map (ref/ASCII-rev) has: 0x00A3 = 0x626C 0x00A5 = 0x6E6579 I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII characters.. :) w/ my updates, they are properly(?) mapped to 0x3F... I'm not sure what they were w/o my changes... I already committed a fixed for an issue w/ svn keywords in tablegen/cmp.sh... Hence, why I'm able to run the tests.. Comments? Please CC me, I'm not on the list.. -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-i18n@FreeBSD.ORG Wed Feb 27 21:40:57 2013 Return-Path: Delivered-To: freebsd-i18n@FreeBSD.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id EBA1F169 for ; Wed, 27 Feb 2013 21:40:57 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id B8EDB1DC for ; Wed, 27 Feb 2013 21:40:57 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r1RLevuI008831 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO) for ; Wed, 27 Feb 2013 13:40:57 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r1RLevqX008830 for freebsd-i18n@FreeBSD.org; Wed, 27 Feb 2013 13:40:57 -0800 (PST) (envelope-from jmg) Date: Wed, 27 Feb 2013 13:40:57 -0800 From: John-Mark Gurney To: freebsd-i18n@FreeBSD.org Subject: Re: patch for in tree iconv Message-ID: <20130227214057.GD55866@funkthat.com> References: <20130227213200.GB55866@funkthat.com> Mime-Version: 1.0 Content-Type: multipart/mixed; boundary="4f28nU6agdXSinmL" Content-Disposition: inline In-Reply-To: <20130227213200.GB55866@funkthat.com> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Wed, 27 Feb 2013 13:40:57 -0800 (PST) X-BeenThere: freebsd-i18n@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: FreeBSD Internationalization Effort List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Feb 2013 21:40:58 -0000 --4f28nU6agdXSinmL Content-Type: text/plain; charset=us-ascii Content-Disposition: inline John-Mark Gurney wrote this message on Wed, Feb 27, 2013 at 13:32 -0800: > I've worked at fixing a few of iconv's locking issues, and I've attached > this patch. This should fix most/all of the locking related issues > relating to the global lock... I should of included the patch... Oops... In case the patch gets stripped: http://people.FreeBSD.org/~jmg/iconv.patch -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." --4f28nU6agdXSinmL Content-Type: text/x-diff; charset=us-ascii Content-Disposition: attachment; filename="iconv.patch" Index: lib/libc/iconv/citrus_csmapper.c =================================================================== --- lib/libc/iconv/citrus_csmapper.c (revision 247143) +++ lib/libc/iconv/citrus_csmapper.c (working copy) @@ -312,24 +312,36 @@ get_none(struct _citrus_mapper_area *__restrict ma, struct _citrus_csmapper *__restrict *__restrict rcsm) { + struct _citrus_csmapper *csm_tmp; int ret; - WLOCK; - if (csm_none) { + WLOCK(); + if (csm_none != NULL) { *rcsm = csm_none; + UNLOCK(); ret = 0; goto quit; } - ret = _mapper_open_direct(ma, &csm_none, "mapper_none", ""); + UNLOCK(); + ret = _mapper_open_direct(ma, &csm_tmp, "mapper_none", ""); if (ret) goto quit; + _mapper_set_persistent(csm_none); + WLOCK(); + if (csm_none != NULL) { + UNLOCK(); + _mapper_close(csm_tmp); + } else { + csm_none = csm_tmp; + UNLOCK(); + } + *rcsm = csm_none; ret = 0; quit: - UNLOCK; return (ret); } Index: lib/libc/iconv/citrus_iconv.c =================================================================== --- lib/libc/iconv/citrus_iconv.c (revision 247143) +++ lib/libc/iconv/citrus_iconv.c (working copy) @@ -72,7 +72,7 @@ init_cache(void) { - WLOCK; + WLOCK(); if (!isinit) { _CITRUS_HASH_INIT(&shared_pool, CI_HASH_SIZE); TAILQ_INIT(&shared_unused); @@ -83,7 +83,7 @@ shared_max_reuse = CI_INITIAL_MAX_REUSE; isinit = true; } - UNLOCK; + UNLOCK(); } static __inline void @@ -189,16 +189,16 @@ get_shared(struct _citrus_iconv_shared * __restrict * __restrict rci, const char *src, const char *dst) { - struct _citrus_iconv_shared * ci; + struct _citrus_iconv_shared * ci, *citmp; char convname[PATH_MAX]; int hashval, ret = 0; snprintf(convname, sizeof(convname), "%s/%s", src, dst); - WLOCK; - /* lookup alread existing entry */ hashval = hash_func(convname); + + WLOCK(); _CITRUS_HASH_SEARCH(&shared_pool, ci, ci_hash_entry, match_func, convname, hashval); if (ci != NULL) { @@ -209,21 +209,32 @@ } ci->ci_used_count++; *rci = ci; + UNLOCK(); goto quit; } + UNLOCK(); /* create new entry */ ret = open_shared(&ci, convname, src, dst); if (ret) goto quit; + WLOCK(); + _CITRUS_HASH_SEARCH(&shared_pool, citmp, ci_hash_entry, match_func, + convname, hashval); + if (citmp != NULL) { + UNLOCK(); + close_shared(ci); + ci = citmp; + goto quit; + } + _CITRUS_HASH_INSERT(&shared_pool, ci, ci_hash_entry, hashval); ci->ci_used_count = 1; *rci = ci; + UNLOCK(); quit: - UNLOCK; - return (ret); } @@ -231,7 +242,7 @@ release_shared(struct _citrus_iconv_shared * __restrict ci) { - WLOCK; + WLOCK(); ci->ci_used_count--; if (ci->ci_used_count == 0) { /* put it into unused list */ @@ -243,11 +254,13 @@ TAILQ_REMOVE(&shared_unused, ci, ci_tailq_entry); _CITRUS_HASH_REMOVE(ci, ci_hash_entry); shared_num_unused--; + UNLOCK(); close_shared(ci); + WLOCK(); } } - UNLOCK; + UNLOCK(); } /* @@ -273,6 +286,7 @@ dst = nl_langinfo(CODESET); /* resolve codeset name aliases */ + path[0] = '\x0'; /* XXX - just copy src/dst to realsrc/dst? */ strlcpy(realsrc, _lookup_alias(path, src, buf, (size_t)PATH_MAX, _LOOKUP_CASE_IGNORE), (size_t)PATH_MAX); strlcpy(realdst, _lookup_alias(path, dst, buf, (size_t)PATH_MAX, Index: lib/libc/iconv/citrus_lock.c =================================================================== --- lib/libc/iconv/citrus_lock.c (revision 0) +++ lib/libc/iconv/citrus_lock.c (working copy) @@ -0,0 +1,3 @@ +#include "citrus_lock.h" + +pthread_rwlock_t lock = PTHREAD_RWLOCK_INITIALIZER; Index: lib/libc/iconv/citrus_lock.h =================================================================== --- lib/libc/iconv/citrus_lock.h (revision 247143) +++ lib/libc/iconv/citrus_lock.h (working copy) @@ -27,9 +27,13 @@ #include -static pthread_rwlock_t lock; +extern pthread_rwlock_t lock; -#define WLOCK if (__isthreaded) \ - pthread_rwlock_wrlock(&lock); -#define UNLOCK if (__isthreaded) \ - pthread_rwlock_unlock(&lock); +#define WLOCK() do { \ + if (__isthreaded) \ + pthread_rwlock_wrlock(&lock); \ + } while (0) +#define UNLOCK() do { \ + if (__isthreaded) \ + pthread_rwlock_unlock(&lock); \ + } while (0) Index: lib/libc/iconv/citrus_mapper.c =================================================================== --- lib/libc/iconv/citrus_mapper.c (revision 247143) +++ lib/libc/iconv/citrus_mapper.c (working copy) @@ -75,7 +75,7 @@ char path[PATH_MAX]; int ret; - WLOCK; + WLOCK(); if (*rma != NULL) { ret = 0; @@ -104,7 +104,7 @@ *rma = ma; ret = 0; quit: - UNLOCK; + UNLOCK(); return (ret); } @@ -309,14 +309,14 @@ struct _citrus_mapper * __restrict * __restrict rcm, const char * __restrict mapname) { - struct _citrus_mapper *cm; + struct _citrus_mapper *cm, *cmtmp; char linebuf[PATH_MAX]; const char *module, *variable; int hashval, ret; variable = NULL; - WLOCK; + WLOCK(); /* search in the cache */ hashval = hash_func(mapname); @@ -325,9 +325,11 @@ if (cm) { /* found */ cm->cm_refcount++; + UNLOCK(); +foundwref: *rcm = cm; ret = 0; - goto quit; + goto quitnolock; } /* search mapper entry */ @@ -337,16 +339,28 @@ goto quit; /* open mapper */ - UNLOCK; + UNLOCK(); ret = mapper_open(ma, &cm, module, variable); - WLOCK; if (ret) - goto quit; + goto quitnolock; + + WLOCK(); + _CITRUS_HASH_SEARCH(&ma->ma_cache, cmtmp, cm_entry, match_func, mapname, + hashval); + if (cmtmp) { + cmtmp->cm_refcount++; + UNLOCK(); + _mapper_close(cm); + cm = cmtmp; + goto foundwref; + } + cm->cm_key = strdup(mapname); if (cm->cm_key == NULL) { ret = errno; + UNLOCK(); _mapper_close(cm); - goto quit; + goto quitnolock; } /* insert to the cache */ @@ -356,8 +370,9 @@ *rcm = cm; ret = 0; quit: - UNLOCK; + UNLOCK(); +quitnolock: return (ret); } @@ -369,20 +384,21 @@ _citrus_mapper_close(struct _citrus_mapper *cm) { - if (cm) { - WLOCK; - if (cm->cm_refcount == REFCOUNT_PERSISTENT) - goto quit; - if (cm->cm_refcount > 0) { - if (--cm->cm_refcount > 0) - goto quit; - _CITRUS_HASH_REMOVE(cm, cm_entry); - free(cm->cm_key); - } - mapper_close(cm); -quit: - UNLOCK; + if (cm == NULL) + return; + + WLOCK(); + if (cm->cm_refcount == REFCOUNT_PERSISTENT || --cm->cm_refcount != 0) { + UNLOCK(); + return; } + + _CITRUS_HASH_REMOVE(cm, cm_entry); + free(cm->cm_key); + cm->cm_key = NULL; + + UNLOCK(); + mapper_close(cm); } /* @@ -393,7 +409,7 @@ _citrus_mapper_set_persistent(struct _citrus_mapper * __restrict cm) { - WLOCK; + WLOCK(); cm->cm_refcount = REFCOUNT_PERSISTENT; - UNLOCK; + UNLOCK(); } Index: lib/libc/iconv/citrus_module.c =================================================================== --- lib/libc/iconv/citrus_module.c (revision 247143) +++ lib/libc/iconv/citrus_module.c (working copy) @@ -109,7 +109,6 @@ #include "citrus_namespace.h" #include "citrus_bcs.h" #include "citrus_module.h" -#include "libc_private.h" static int _getdewey(int[], char *); static int _cmpndewey(int[], int, int[], int); @@ -293,11 +292,11 @@ maj = I18NMODULE_MAJOR; min = -1; p = _findshlib(path, &maj, &min); - if (!p) + if (p == NULL) return (EINVAL); - handle = libc_dlopen(p, RTLD_LAZY); - if (!handle) { - printf("%s", dlerror()); + handle = dlopen(p, RTLD_LAZY); + if (handle == NULL) { + printf("%s", dlerror()); /* XXX */ return (EINVAL); } Index: lib/libiconv/Makefile =================================================================== --- lib/libiconv/Makefile (revision 247143) +++ lib/libiconv/Makefile (working copy) @@ -14,11 +14,11 @@ SRCS= citrus_bcs.c citrus_bcs_strtol.c citrus_bcs_strtoul.c \ citrus_csmapper.c citrus_db.c citrus_db_factory.c \ citrus_db_hash.c citrus_esdb.c citrus_hash.c \ - citrus_iconv.c citrus_lookup.c citrus_lookup_factory.c \ + citrus_iconv.c citrus_lock.c citrus_lookup.c citrus_lookup_factory.c \ citrus_mapper.c citrus_memstream.c citrus_mmap.c \ citrus_module.c citrus_none.c citrus_pivot_factory.c \ citrus_prop.c citrus_stdenc.c iconv.c -CFLAGS+= --param max-inline-insns-single=128 -I ${.CURDIR}/../../include -I${.CURDIR}/../libc/include +CFLAGS+= -I ${.CURDIR}/../../include .include --4f28nU6agdXSinmL-- From owner-freebsd-i18n@FreeBSD.ORG Wed Feb 27 21:44:56 2013 Return-Path: Delivered-To: freebsd-i18n@freebsd.org Received: from mx1.freebsd.org (mx1.FreeBSD.org [8.8.178.115]) by hub.freebsd.org (Postfix) with ESMTP id CAD04295; Wed, 27 Feb 2013 21:44:56 +0000 (UTC) (envelope-from delphij@gmail.com) Received: from mail-qa0-f44.google.com (mail-qa0-f44.google.com [209.85.216.44]) by mx1.freebsd.org (Postfix) with ESMTP id 55C9F224; Wed, 27 Feb 2013 21:44:56 +0000 (UTC) Received: by mail-qa0-f44.google.com with SMTP id bv4so3718495qab.3 for ; Wed, 27 Feb 2013 13:44:55 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:x-received:in-reply-to:references:date:message-id :subject:from:to:cc:content-type; bh=O81PTauu1P3VJotMwTrQgn2umNlMkuLrH0vlP6fbHTI=; b=jMEtU9cNb8NorVfGAtaZ3KJ5qQ91fYyoh0giKcAG9Ub3TAMu3p8v3hBlqHNLc0Gz3c FSn0+sFkB0ZxRndQcMzkwLWMBhSohaBjY/6FCvGo7t3mwiYrBFOxHbB8TnIos0iaM0RS 8lo38cf29u8mU7InEvbifgjeJtxGcINWPSItHWlFhdOa1Ady8mAWIADeK0qcdJyM1Cst fnRNAW3N/AkJyMjMCuiwpROhpgG7+vbfp/P133sL+4QdmXQHUwDEwQzxfEjfj/0nufT5 W1FZXnqzlejrKY4cxO5vCHCdHP8K/0tnicgsnDeOmkjC2xiKKF+kTMnRn8+UUyvb794+ 6rPA== MIME-Version: 1.0 X-Received: by 10.224.111.134 with SMTP id s6mr10517764qap.4.1362001126690; Wed, 27 Feb 2013 13:38:46 -0800 (PST) Received: by 10.49.12.162 with HTTP; Wed, 27 Feb 2013 13:38:46 -0800 (PST) In-Reply-To: <20130227213200.GB55866@funkthat.com> References: <20130227213200.GB55866@funkthat.com> Date: Wed, 27 Feb 2013 13:38:46 -0800 Message-ID: Subject: Re: patch for in tree iconv From: Xin LI To: John-Mark Gurney Content-Type: text/plain; charset=UTF-8 X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: Gabor Kovesdan , freebsd-i18n@freebsd.org X-BeenThere: freebsd-i18n@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: FreeBSD Internationalization Effort List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Feb 2013 21:44:56 -0000 I think the list ate your patch, could you please send it inline or (ideally) post it on a web server? Cheers, On Wed, Feb 27, 2013 at 1:32 PM, John-Mark Gurney wrote: > I've worked at fixing a few of iconv's locking issues, and I've attached > this patch. This should fix most/all of the locking related issues > relating to the global lock... > > Though I ran the tests in tools/test/iconv, and it looks like the > checked in reference conversion in ref are wrong... The > UTF-32 -> ASCII map (ref/ASCII-rev) has: > 0x00A3 = 0x626C > 0x00A5 = 0x6E6579 > > I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII characters.. :) > > w/ my updates, they are properly(?) mapped to 0x3F... I'm not sure > what they were w/o my changes... > > I already committed a fixed for an issue w/ svn keywords in > tablegen/cmp.sh... Hence, why I'm able to run the tests.. > > Comments? > > Please CC me, I'm not on the list.. > > -- > John-Mark Gurney Voice: +1 415 225 5579 > > "All that I will do, has been done, All that I have, has not." > _______________________________________________ > freebsd-i18n@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-i18n > To unsubscribe, send any mail to "freebsd-i18n-unsubscribe@freebsd.org" > -- Xin LI https://www.delphij.net/ FreeBSD - The Power to Serve! Live free or die From owner-freebsd-i18n@FreeBSD.ORG Thu Feb 28 07:38:33 2013 Return-Path: Delivered-To: freebsd-i18n@freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id B135EEA2 for ; Thu, 28 Feb 2013 07:38:33 +0000 (UTC) (envelope-from gabor@FreeBSD.org) Received: from server.mypc.hu (server.mypc.hu [87.229.73.95]) by mx1.freebsd.org (Postfix) with ESMTP id 4F416FBF for ; Thu, 28 Feb 2013 07:38:33 +0000 (UTC) Received: from server.mypc.hu (localhost [127.0.0.1]) by server.mypc.hu (Postfix) with ESMTP id 767B314D2512; Thu, 28 Feb 2013 08:38:25 +0100 (CET) X-Virus-Scanned: amavisd-new at !change-mydomain-variable!.example.com Received: from server.mypc.hu ([127.0.0.1]) by server.mypc.hu (server.mypc.hu [127.0.0.1]) (amavisd-new, port 10024) with LMTP id hl9r0NV3hUSj; Thu, 28 Feb 2013 08:38:14 +0100 (CET) Received: from [152.66.168.85] (dhcp-85.q.wlan.net.bme.hu [152.66.168.85]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by server.mypc.hu (Postfix) with ESMTPSA id 9F1B214D242C; Thu, 28 Feb 2013 08:38:14 +0100 (CET) Message-ID: <512F0964.4090506@FreeBSD.org> Date: Thu, 28 Feb 2013 08:38:12 +0100 From: Gabor Kovesdan User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:17.0) Gecko/20130107 Thunderbird/17.0.2 MIME-Version: 1.0 To: Xin LI Subject: Re: patch for in tree iconv References: <20130227213200.GB55866@funkthat.com> In-Reply-To: Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.14 Cc: John-Mark Gurney , freebsd-i18n@freebsd.org X-BeenThere: freebsd-i18n@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: FreeBSD Internationalization Effort List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2013 07:38:33 -0000 Em 27-02-2013 22:38, Xin LI escreveu: > > > On Wed, Feb 27, 2013 at 1:32 PM, John-Mark Gurney > wrote: > > I've worked at fixing a few of iconv's locking issues, and I've > attached > this patch. This should fix most/all of the locking related issues > relating to the global lock... > Thank you, I have had this on my TODO list for a long time just haven't got time for it! > > > Though I ran the tests in tools/test/iconv, and it looks like the > checked in reference conversion in ref are wrong... The > UTF-32 -> ASCII map (ref/ASCII-rev) has: > 0x00A3 = 0x626C > 0x00A5 = 0x6E6579 > > I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII > characters.. :) > > w/ my updates, they are properly(?) mapped to 0x3F... I'm not sure > what they were w/o my changes... > Seems like a good solution. However, the ref tables were generated with GNU libiconv (since the goal is a good compatibility) so we should also track down why does it emit this. > > > I already committed a fixed for an issue w/ svn keywords in > tablegen/cmp.sh... Hence, why I'm able to run the tests.. > Thanks for doing this! Please also keep me CC'd, I'm not on the list but I'm interested in iconv-related things. Gabor From owner-freebsd-i18n@FreeBSD.ORG Thu Feb 28 18:48:59 2013 Return-Path: Delivered-To: freebsd-i18n@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id E2D56574; Thu, 28 Feb 2013 18:48:59 +0000 (UTC) (envelope-from jmg@h2.funkthat.com) Received: from h2.funkthat.com (gate2.funkthat.com [208.87.223.18]) by mx1.freebsd.org (Postfix) with ESMTP id A3718BDE; Thu, 28 Feb 2013 18:48:59 +0000 (UTC) Received: from h2.funkthat.com (localhost [127.0.0.1]) by h2.funkthat.com (8.14.3/8.14.3) with ESMTP id r1SImwJq025125 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NO); Thu, 28 Feb 2013 10:48:58 -0800 (PST) (envelope-from jmg@h2.funkthat.com) Received: (from jmg@localhost) by h2.funkthat.com (8.14.3/8.14.3/Submit) id r1SImwLQ025124; Thu, 28 Feb 2013 10:48:58 -0800 (PST) (envelope-from jmg) Date: Thu, 28 Feb 2013 10:48:58 -0800 From: John-Mark Gurney To: Gabor Kovesdan Subject: Re: patch for in tree iconv Message-ID: <20130228184858.GH55866@funkthat.com> References: <20130227213200.GB55866@funkthat.com> <512F0964.4090506@FreeBSD.org> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <512F0964.4090506@FreeBSD.org> User-Agent: Mutt/1.4.2.3i X-Operating-System: FreeBSD 7.2-RELEASE i386 X-PGP-Fingerprint: 54BA 873B 6515 3F10 9E88 9322 9CB1 8F74 6D3F A396 X-Files: The truth is out there X-URL: http://resnet.uoregon.edu/~gurney_j/ X-Resume: http://resnet.uoregon.edu/~gurney_j/resume.html X-to-the-FBI-CIA-and-NSA: HI! HOW YA DOIN? can i haz chizburger? X-Greylist: Sender passed SPF test, not delayed by milter-greylist-4.2.2 (h2.funkthat.com [127.0.0.1]); Thu, 28 Feb 2013 10:48:59 -0800 (PST) Cc: freebsd-i18n@FreeBSD.org X-BeenThere: freebsd-i18n@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: FreeBSD Internationalization Effort List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2013 18:49:00 -0000 Gabor Kovesdan wrote this message on Thu, Feb 28, 2013 at 08:38 +0100: > Em 27-02-2013 22:38, Xin LI escreveu: > > > > > >On Wed, Feb 27, 2013 at 1:32 PM, John-Mark Gurney >> wrote: > > > > I've worked at fixing a few of iconv's locking issues, and I've > > attached > > this patch. This should fix most/all of the locking related issues > > relating to the global lock... > > > Thank you, I have had this on my TODO list for a long time just haven't > got time for it! > > > > > > Though I ran the tests in tools/test/iconv, and it looks like the > > checked in reference conversion in ref are wrong... The > > UTF-32 -> ASCII map (ref/ASCII-rev) has: > > 0x00A3 = 0x626C > > 0x00A5 = 0x6E6579 > > > > I'm pretty sure 0x626C and 0x6E6579 are not valid ASCII > > characters.. :) > > > > w/ my updates, they are properly(?) mapped to 0x3F... I'm not sure > > what they were w/o my changes... > > > Seems like a good solution. However, the ref tables were generated with > GNU libiconv (since the goal is a good compatibility) so we should also > track down why does it emit this. It's because we generated the ref w/ //TRANSLIT, and our iconv doesn't have an advanced set of characters for transliteration... Heck, it doesn't handle the difference between //TRANSLIT and //IGNORE... -0x00A0 = 0x0020 Non-braking space -> space -0x00A1 = 0x0021 inverted exclamation mark -> exclamation mark -0x00A2 = 0x0063 cent sign -> c -0x00A3 = 0x626C pound sign -> lb I'm "convinced" that the ref is right for proper transliteration... So, do we need transliteration support? -- John-Mark Gurney Voice: +1 415 225 5579 "All that I will do, has been done, All that I have, has not." From owner-freebsd-i18n@FreeBSD.ORG Thu Feb 28 18:54:27 2013 Return-Path: Delivered-To: freebsd-i18n@FreeBSD.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2001:1900:2254:206a::19:1]) by hub.freebsd.org (Postfix) with ESMTP id 094448B3 for ; Thu, 28 Feb 2013 18:54:27 +0000 (UTC) (envelope-from gabor@FreeBSD.org) Received: from server.mypc.hu (server.mypc.hu [87.229.73.95]) by mx1.freebsd.org (Postfix) with ESMTP id AB7C2C14 for ; Thu, 28 Feb 2013 18:54:26 +0000 (UTC) Received: from server.mypc.hu (localhost [127.0.0.1]) by server.mypc.hu (Postfix) with ESMTP id B902D14D2513; Thu, 28 Feb 2013 19:54:24 +0100 (CET) X-Virus-Scanned: amavisd-new at !change-mydomain-variable!.example.com Received: from server.mypc.hu ([127.0.0.1]) by server.mypc.hu (server.mypc.hu [127.0.0.1]) (amavisd-new, port 10024) with LMTP id r5b0I8UD-ZfN; Thu, 28 Feb 2013 19:54:24 +0100 (CET) Received: from [192.168.1.117] (catv-80-99-23-232.catv.broadband.hu [80.99.23.232]) (using TLSv1 with cipher DHE-RSA-CAMELLIA256-SHA (256/256 bits)) (No client certificate requested) by server.mypc.hu (Postfix) with ESMTPSA id 72F2614D242C; Thu, 28 Feb 2013 19:54:22 +0100 (CET) Message-ID: <512FA7DA.9070804@FreeBSD.org> Date: Thu, 28 Feb 2013 19:54:18 +0100 From: Gabor Kovesdan User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:19.0) Gecko/20121226 Thunderbird/19.0a2 MIME-Version: 1.0 To: John-Mark Gurney Subject: Re: patch for in tree iconv References: <20130227213200.GB55866@funkthat.com> <512F0964.4090506@FreeBSD.org> <20130228184858.GH55866@funkthat.com> In-Reply-To: <20130228184858.GH55866@funkthat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Cc: freebsd-i18n@FreeBSD.org X-BeenThere: freebsd-i18n@freebsd.org X-Mailman-Version: 2.1.14 Precedence: list List-Id: FreeBSD Internationalization Effort List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 28 Feb 2013 18:54:27 -0000 On 2013.02.28. 19:48, John-Mark Gurney wrote: > It's because we generated the ref w/ //TRANSLIT, and our iconv doesn't > have an advanced set of characters for transliteration... Heck, it doesn't > handle the difference between //TRANSLIT and //IGNORE... > > -0x00A0 = 0x0020 Non-braking space -> space > -0x00A1 = 0x0021 inverted exclamation mark -> exclamation mark > -0x00A2 = 0x0063 cent sign -> c > -0x00A3 = 0x626C pound sign -> lb > > I'm "convinced" that the ref is right for proper transliteration... > > So, do we need transliteration support? Ok, now I remember about //TRANSLIT. The default choice of BSD iconv was to use transliteration by default so I followed this convention. And once we have it, it should be as complete as possible, imo. Personally, I would prefer making transliteration optional but that requires heavier changes to iconv. Gabor