Did you know ... | Search Documentation: |
![]() | Packs (add-ons) for SWI-Prolog |
Title: | A nascent tokenization library |
---|---|
Rating: | Not rated. Create the first rating! |
Latest version: | 0.1.2 |
SHA1 sum: | 44d5a3d36e13f612474ba19f295638d45f955c60 |
Author: | Shon Feder <shon.feder@gmail.com> |
Maintainer: | Shon Feder <shon.feder@gmail.com> |
Packager: | Shon Feder <shon.feder@gmail.com> |
Home page: | https://github.com/aBathologist/tokenize |
Download URL: | https://github.com/aBathologist/tokenize/release/*.zip |
No reviews. Create the first review!.
Version | SHA1 | #Downloads | URL |
---|---|---|---|
0.1.0 | 7e9cda4d44249249930d18688ae9de2f73a5377f | 1 | https://github.com/aBathologist/tokenize/archive/v0.1.0.zip |
0.1.1 | 816b05a33fe65a7039c5fb9c545372060626c9ea | 1 | https://github.com/aBathologist/tokenize/archive/v0.1.1.zip |
0.1.2 | 44d5a3d36e13f612474ba19f295638d45f955c60 | 45 | https://github.com/aBathologist/tokenize/archive/v0.1.2.zip |
?- tokenize(`\tExample Text.`, Tokens). Tokens = [cntrl('\t'), word(example), spc(' '), spc(' '), word(text), punct('.')] ?- tokenize(`\tExample Text.`, Tokens, [cntrl(false), pack(true), cased(true)]). Tokens = [word('Example', 1), spc(' ', 2), word('Text', 1), punct('.', 1)] ?- tokenize(`\tExample Text.`, Tokens), untokenize(Tokens, Text), format('~s~n', [Text]). example text. Tokens = [cntrl('\t'), word(example), spc(' '), spc(' '), word(text), punct('.')], Text = [9, 101, 120, 97, 109, 112, 108, 101, 32|...]
Module tokenize
aims to provide a straightforward tool for tokenizing text into a simple format. It is the result of a learning exercise, and it is far from perfect. If there is sufficient interest from myself or anyone else, I'll try to improve it.
Please visit the wiki for more detailed instructions and examples, including a full list of options supported.
Pack contains 4 files holding a total of 11.0K bytes.