2017 © Pedro Peláez
 

library utfnormal

Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present

image

wikimedia/utfnormal

Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present

  • Saturday, July 28, 2018
  • by mediawiki
  • Repository
  • 17 Watchers
  • 5 Stars
  • 224,353 Installations
  • PHP
  • 5 Dependents
  • 0 Suggesters
  • 2 Forks
  • 0 Open issues
  • 7 Versions
  • 5 % Grown

The README.md

Latest Stable Version License, (*1)

utfnormal

utfnormal is a library that contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present., (*2)

The main function to care about is UtfNormal\Validator::cleanUp(). This will strip illegal UTF-8 sequences and characters that are illegal in XML, and if necessary convert to normalization form C., (*3)

If you know the string is already valid UTF-8, you can directly call UtfNormal\Validator::toNFC(), toNFK(), or toNFKC(); this will convert a given UTF-8 string to Normalization Form C, K, or KC if it's not already such. The function assumes that the input string is already valid UTF-8; if there are corrupt characters this may produce erroneous results., (*4)

Performance is kind of stinky in absolute terms, though it should be speedy on pure ASCII text. ;) On text that can be determined quickly to already be in NFC it's not too awful but it can quickly get uncomfortably slow, particularly for Korean text (the hangul decomposition/composition code is extra slow)., (*5)

Bugs should be filed in Wikimedia's Phabricator under the "utfnormal" project., (*6)

Regenerating data tables

UtfNormalData.inc and UtfNormalDataK.inc are generated from the Unicode Character Database by the script "generate.php". Run "composer generate" to rebuild the tables. To fetch updated unicode data from the internet, run "composer generate -- --fetch"., (*7)

Testing

Running "composer test" will run a syntax checker, PHPUnit conformance tests, and run some benchmarks using sample texts from Wikipedia. Take all benchmark numbers with large grains of salt., (*8)

PHP module extension

If the 'intl' PHP extension is present, ICU library functions are used which are MUCH faster than doing this work in pure PHP code., (*9)

It is strongly recommended to enable this module if possible: http://php.net/manual/en/intro.intl.php, (*10)

Older versions of this library supported a one-off custom PHP extension, which has been dropped. If you were using this, please migrate to the intl extension., (*11)

History

This library was first introduced in MediaWiki 1.3 (r4965). It was split out of the MediaWiki codebase and published as an independent library during the MediaWiki 1.25 development cycle., (*12)

The Versions

28/07 2018

dev-master

9999999-dev https://www.mediawiki.org/wiki/utfnormal

Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present

  Sources   Download

GPL-2.0+ GPL-2.0-or-later

The Requires

  • php >=5.6.99

 

The Development Requires

by Brion Vibber

27/02 2018

v2.0.0

2.0.0.0 https://www.mediawiki.org/wiki/utfnormal

Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present

  Sources   Download

GPL-2.0-or-later

The Requires

  • php >=5.5.9

 

The Development Requires

by Brion Vibber

24/05 2016

v1.1.0

1.1.0.0 https://www.mediawiki.org/wiki/utfnormal

Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present

  Sources   Download

GPL-2.0+

The Requires

  • php >=5.3.3

 

The Development Requires

by Brion Vibber

29/08 2015

v1.0.3

1.0.3.0 https://www.mediawiki.org/wiki/utfnormal

Contains Unicode normalization routines, including both pure PHP implementations and automatic use of the 'intl' PHP extension when present

  Sources   Download

GPL-2.0+

The Requires

  • php >=5.3.3

 

The Development Requires

by Brion Vibber

12/03 2015

v1.0.2

1.0.2.0 https://www.mediawiki.org/wiki/utfnormal

  Sources   Download

GPL-2.0+

The Requires

  • php >=5.3.3

 

The Development Requires

by Brion Vibber

07/03 2015

v1.0.1

1.0.1.0 https://www.mediawiki.org/wiki/utfnormal

  Sources   Download

GPL-2.0+

The Requires

  • php >=5.3.3

 

The Development Requires

by Brion Vibber

25/02 2015

v1.0.0

1.0.0.0 https://www.mediawiki.org/wiki/utfnormal

  Sources   Download

GPL-2.0+

The Requires

  • php >=5.3.3

 

The Development Requires

by Brion Vibber