What is the "token"? #5
Description
Hello,this is my question.
In "reader.py" ,the 14line ,tokenize = lambda x: [token.strip().lower() for token in re.split(SPLIT_RE, x) if token.strip()] , I am not search the "token". So what is the "token"?
{{ message }}