tokenize_document() doesn't work if the model does not support fast tokenizer

[Bhuvanesh-Verma]

Model: cl-tohoku/bert-base-japanese
Package Requirements: ipadic, fugashi (pip install ipadic fugashi)

Code to replicate the issue:

from pie_modules.document.processing import tokenize_document
from transformers import AutoTokenizer
from pytorch_ie.documents import TokenBasedDocument, TextBasedDocument

tokenizer = AutoTokenizer.from_pretrained("cl-tohoku/bert-base-japanese")
text_document = TextBasedDocument(text="東北大学で")

tokenized_docs = tokenize_document(
        text_document,
        tokenizer=tokenizer,
        result_document_type= TokenBasedDocument
    )

Error Message:

Traceback (most recent call last):
  ....
    tokenized_docs = tokenize_document(
  File "..../python3.9/site-packages/pie_modules/document/processing/tokenization.py", line 314, in tokenize_document
    for batch_encoding in tokenized_text.encodings:
TypeError: 'NoneType' object is not iterable

Update: Current fix using huggingface/transformers#12381