2017 © Pedro Peláez
 

library tokenize

Text tokenizing

image

xp-framework/tokenize

Text tokenizing

  • Saturday, June 3, 2017
  • by thekid
  • Repository
  • 2 Watchers
  • 0 Stars
  • 37,776 Installations
  • PHP
  • 7 Dependents
  • 0 Suggesters
  • 0 Forks
  • 0 Open issues
  • 5 Versions
  • 9 % Grown

The README.md

Tokenize

Build status on GitHub XP Framework Module BSD Licence Requires PHP 7.0+ Supports PHP 8.0+ Latest Stable Version, (*1)

Tokenizing text, (*2)

use text\{StringTokenizer, StreamTokenizer};
use io\File;

// Supports strings and streams
$tokens= new StringTokenizer('He asked: Can you parse this?', ' .?!,;:', true);
$tokens= new StreamTokenizer((new File('parse-me.txt'))->in(), ' .?!,;:', true);

// Can iterate using foreach...
foreach ($tokens as $token) {
  Console::writeLine($token);
}

// ...or with an iterator API
while ($tokens->hasMoreTokens()) {
  Console::writeLine($tokens->nextToken());
}

// Returns: ["He", " ", "asked", ":", " ", "Can", " ", "you", " ", "parse", " ", "this", "?"]

The Versions

03/06 2017

dev-master

9999999-dev http://xp-framework.net/

Text tokenizing

  Sources   Download

BSD-3-Clause

The Requires

 

The Development Requires

module xp

03/06 2017

v8.0.0

8.0.0.0 http://xp-framework.net/

Text tokenizing

  Sources   Download

BSD-3-Clause

The Requires

 

The Development Requires

module xp

28/08 2016

v7.1.0

7.1.0.0 http://xp-framework.net/

Text tokenizing

  Sources   Download

BSD-3-Clause

The Requires

 

The Development Requires

module xp

21/02 2016

v7.0.0

7.0.0.0 http://xp-framework.net/

Text tokenizing

  Sources   Download

BSD-3-Clause

The Requires

 

The Development Requires

module xp

08/12 2015

v6.6.0

6.6.0.0 http://xp-framework.net/

Text tokenizing

  Sources   Download

BSD-3-Clause

The Requires

 

module xp