Token Classification

Functionality for token classification (task 1)

Tokenization

 tokenize_and_align_labels_with_tokenizer (tokenizer, examples,
                                           return_tensors=None)

Tokenize function, to be used as partial with instatiated tokenizer

 tokenize_and_align_labels (tokenizer, return_tensors=None)

Function to tokenize samples and align the labels