Package "tokenize"

Title:	A nascent tokenization library
Rating:	Not rated. Create the first rating!
Latest version:	0.1.2
SHA1 sum:	44d5a3d36e13f612474ba19f295638d45f955c60
Author:	Shon Feder <shon.feder@gmail.com>
Maintainer:	Shon Feder <shon.feder@gmail.com>
Packager:	Shon Feder <shon.feder@gmail.com>
Home page:	https://github.com/aBathologist/tokenize
Download URL:	https://github.com/aBathologist/tokenize/release/*.zip

Reviews

No reviews. Create the first review!.

Details by download location

Version	SHA1	#Downloads	URL
0.1.0	7e9cda4d44249249930d18688ae9de2f73a5377f	1	https://github.com/aBathologist/tokenize/archive/v0.1.0.zip
0.1.1	816b05a33fe65a7039c5fb9c545372060626c9ea	1	https://github.com/aBathologist/tokenize/archive/v0.1.1.zip
0.1.2	44d5a3d36e13f612474ba19f295638d45f955c60	47	https://github.com/aBathologist/tokenize/archive/v0.1.2.zip

Synopsis

?- tokenize(`\tExample  Text.`, Tokens).
Tokens = [cntrl('\t'), word(example), spc(' '), spc(' '), word(text), punct('.')]

?- tokenize(`\tExample  Text.`, Tokens, [cntrl(false), pack(true), cased(true)]).
Tokens = [word('Example', 1), spc(' ', 2), word('Text', 1), punct('.', 1)]

?- tokenize(`\tExample  Text.`, Tokens), untokenize(Tokens, Text), format('~s~n', [Text]).
        example  text.
Tokens = [cntrl('\t'), word(example), spc(' '), spc(' '), word(text), punct('.')],
Text = [9, 101, 120, 97, 109, 112, 108, 101, 32|...]

Description

Module tokenize aims to provide a straightforward tool for tokenizing text into a simple format. It is the result of a learning exercise, and it is far from perfect. If there is sufficient interest from myself or anyone else, I'll try to improve it.

Please visit the wiki for more detailed instructions and examples, including a full list of options supported.

Contents of pack "tokenize"

Pack contains 4 files holding a total of 11.0K bytes.