Did you know ... Search Documentation:
Packs (add-ons) for SWI-Prolog

Package "tokenize"

Title:A nascent tokenization library
Rating:Not rated. Create the first rating!
Latest version:0.1.2
SHA1 sum:44d5a3d36e13f612474ba19f295638d45f955c60
Author:Shon Feder <shon.feder@gmail.com>
Maintainer:Shon Feder <shon.feder@gmail.com>
Packager:Shon Feder <shon.feder@gmail.com>
Home page:https://github.com/aBathologist/tokenize
Download URL:https://github.com/aBathologist/tokenize/release/*.zip


No reviews. Create the first review!.

Details by download location



?- tokenize(`\tExample  Text.`, Tokens).
Tokens = [cntrl('\t'), word(example), spc(' '), spc(' '), word(text), punct('.')]

?- tokenize(`\tExample  Text.`, Tokens, [cntrl(false), pack(true), cased(true)]).
Tokens = [word('Example', 1), spc(' ', 2), word('Text', 1), punct('.', 1)]

?- tokenize(`\tExample  Text.`, Tokens), untokenize(Tokens, Text), format('~s~n', [Text]).
        example  text.
Tokens = [cntrl('\t'), word(example), spc(' '), spc(' '), word(text), punct('.')],
Text = [9, 101, 120, 97, 109, 112, 108, 101, 32|...]


Module tokenize aims to provide a straightforward tool for tokenizing text into a simple format. It is the result of a learning exercise, and it is far from perfect. If there is sufficient interest from myself or anyone else, I'll try to improve it.

Please visit the wiki for more detailed instructions and examples, including a full list of options supported.

Contents of pack "tokenize"

Pack contains 4 files holding a total of 11.0K bytes.