======================
Pretrained Enhanced Contextual Language Models
======================

In addition to access to all the pretrained models on Huggingface's model hub, we also provide pretrained "enhanced" contextual models.
These models are trained with some kind of knowledge that is not textual data, such as encyclopedic data from Wikipedia or lexical data from WordNet.
Here is an overview of the models provided by mangoes, and below there are guides on how to instantiate them.
The "Base Model" column is the non-enhanced model in which the enhanced model is based off of or fine-tuned.

.. list-table:: Provided Pretrained Enhanced Models
   :widths: 25 25 25 25
   :header-rows: 1

   * - Model
     - Base Model
     - Non-textual Knowledge
     - Paper
   * - Ernie
     - BERT
     - Wikidata
     - https://arxiv.org/pdf/1905.07129.pdf
   * - Kepler
     - RoBERTa
     - Wikidata
     - https://arxiv.org/pdf/1911.06136.pdf
   * - K-Adapter
     - RoBERTa
     - Wikidata
     - https://arxiv.org/pdf/2002.01808.pdf
   * - LiBERT
     - BERT
     - WordNet
     - https://aclanthology.org/2020.coling-main.118/
   * - Human Conceptual Knowledge Transformer (HCK)
     - BERT
     - Feature norms
     - https://bpb-us-w2.wpmucdn.com/web.sas.upenn.edu/dist/a/511/files/2021/12/Bhatia-Richie-2021-Psych-Rev.pdf


Using a pretrained enhanced model
=================================

The main idea behind using these models is: we instantiate the model and tokenizer class first, then pass these objects to the mangoes class.
When instantiating the model/tokenizer class using the same API as Huggingface's transformer classes (as they are subclassed from transformer model/tokenizer classes), we pass the None keyword in place of the pretrained model/tokenizer name. This will tell mangoes to download the pretrained model or tokenizer from INRIA's server, which will then be used when instantiating the object.

We will demonstrate how to load pretrained enhanced models. We will use sequence classification as our example task, but the same code works for all tasks.
You can find this code in mangoes/notebooks/Enhanced Models.ipynb .

Some models have their own tokenizer, like Ernie:

    >>> model = ErnieForSequenceClassification.from_pretrained(None, label2id=label2id)
    >>> tokenizer = ErnieTokenizer.from_pretrained(None)

Libert:

    >>> model = LibertForSequenceClassification.from_pretrained(None, label2id=label2id)
    >>> tokenizer = LibertTokenizerFast.from_pretrained(None)

or HCK:

    >>> model = HCKForSequenceClassification.from_pretrained(None, label2id=label2id)
    >>> tokenizer = HCKTokenizerFast.from_pretrained(None)

While other models use non-enhanced tokenizer classes, like Kepler:

    >>> model = KeplerForSequenceClassification.from_pretrained(None, label2id=label2id)
    >>> tokenizer = AutoTokenizer.from_pretrained(KEPLER_PRETRAINED_TOKENIZER_NAME)

or K-Adapter:

    >>> model = KAdapterForSequenceClassification.from_pretrained(None, label2id=label2id)
    >>> tokenizer = AutoTokenizer.from_pretrained(KADAPTER_PRETRAINED_TOKENIZER_NAME)

Note that both KEPLER_PRETRAINED_TOKENIZER_NAME and KADAPTER_PRETRAINED_TOKENIZER_NAME variables can be imported from mangoes.modeling

Once the model and tokenizer are instantiated, one can make a mangoes class by simply passing these objects to the init function:

    >>> mangoes_model = TransformerForSequenceClassification(model, tokenizer, label2id=label2id)

From there, the mangoes obeject behaves the same as if it was using a non-enhanced model, so you can extract features or predictions, or fine-tune it for any of the supported tasks.