Date: Mon, 16 Jun 2014 02:25:23 +0400 From: Dmitry Selyutin <ghostman.sd@gmail.com> To: soc-status@FreeBSD.org, Pedro Giffuni <pfg@FreeBSD.org>, David Chisnall <theraven@freebsd.org> Subject: Report #1: Unicode support Message-ID: <539E1D53.6030103@gmail.com>
next in thread | raw e-mail | index | archive | help
This is a report on progress in improving Unicode support in FreeBSD. During the early period, I've been studying Unicode Technical Standard, which describes how to implement Unicode Collation Algorithm. I've tried to use the patch proposed by Konrad Jankowski, but it was a rather unsuccessful attempt, since this patch predates xlocale support, implemented by David Chisnall. The initial plan was to port collation support from Apple's libc library, but we rejected this idea because of poor code quality. Moreover, if we decided to use Apple's libc, we would have broken the entire xlocale support. Having lost a significant amount of time on Apple's libc and Konrad's path, we've decided to implement collation from scratch according to Unicode Normalization Algorithm. One of the requirements for collation is the normalization of the string before performing actual collation. C Standard Library lacks such feature, so I started to implement it. This work is almost finished; the FreeBSD's libc will have __strnorm_l(), __strnorm() and __wcsnorm() functions. They have man pages and can be already used to normalize ASCII, Latin-1 and Hangul strings. The last part is to implement normalization of the other characters, which is usually done using database lookup (usually Unicode data is stored in arrays, where each array denotes single Multilingual Plane). These functions are designed in the way that may allow to include them in POSIX standard later under strnorm(), strnorm_l() and wcsnorm() names. If _LIBC_UNICODE_ADDENDA macro is defined, they will be already available under these names. Unicode Standard is a bit difficult: sometimes Unicode Standard focuses on details, paying little attention to the main part. However, I'm planning to finish normalization algorithm in day or two and then implement a collation algorithm. We lost a significant time on trying to port Konrad's patch and Apple's libc collation algorithm. Now we focus on the Unicode Standard directly; that seems to be a better decision. The first step is to implement collation algorithm in the canonical way, then to focus on its improvements and testing. I'd also like to thank my mentors, Pedro and David, who were (and are) so kind to give me advice throughout my work. It's particularly valuable since our task is not so simple as it may seem to be. :-) -- With best regards, Dmitry Selyutin
Want to link to this message? Use this URL: <https://mail-archive.FreeBSD.org/cgi/mid.cgi?539E1D53.6030103>