Skip to content

finetune have bug!!ValueError: char_to_token() is not available when using Python based tokenizers #90

@linhuixiao

Description

@linhuixiao

Start training...
Starting epoch 0

/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/position_encoding.py:41: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats)
/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/position_encoding.py:41: UserWarning: floordiv is deprecated, and its behavior will change in a future version of pytorch. It currently rounds toward 0 (like the 'trunc' function NOT 'floor'). This results in incorrect rounding for negative values. To keep the current behavior, use torch.div(a, b, rounding_mode='trunc'), or for actual floor division, use torch.div(a, b, rounding_mode='floor').
dim_t = self.temperature ** (2 * (dim_t // 2) / self.num_pos_feats)
Traceback (most recent call last):
File "main.py", line 631, in
args = parser.parse_args()
File "main.py", line 533, in main
sampler_train.set_epoch(epoch)
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/engine.py", line 73, in train_one_epoch
loss_dict.update(criterion(outputs, targets, positive_map))
File "/home/mmc_xiaolinhui/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1110, in _call_impl
return forward_call(*input, **kwargs)
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 569, in forward
losses.update(self.get_loss(loss, outputs, targets, positive_map, indices, num_boxes))
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 516, in get_loss
return loss_map[loss](outputs, targets, positive_map, indices, num_boxes, **kwargs)
File "/home/mmc_xiaolinhui/mmc_226_exp_202206/mdetr/models/mdetr.py", line 399, in loss_contrastive_align
beg_pos = tokenized.char_to_token(i, beg)
File "/home/mmc_xiaolinhui/anaconda3/envs/mdetr_env/lib/python3.8/site-packages/transformers/tokenization_utils_base.py", line 547, in char_to_token
raise ValueError("char_to_token() is not available when using Python based tokenizers")
ValueError: char_to_token() is not available when using Python based tokenizers

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions