|Did you know ...||Search Documentation:|
:- use_module(library(porter_stem)).(can be autoloaded)
Character classification is based on the C-library iswalnum() etc. functions. Recognised numbers are passed to Prolog read/1, supporting unbounded integers.
It is likely that future versions of this library will provide tokenize_atom/3 with additional options to modify space handling as well as the definition of words.