mangoes.modeling.training_utils module¶

This module provides helper classes for training huggingface models using the interface in mangoes.modeling.finetuning. These classes are mainly called internally from mangoes.modeling.finetuning, however they can be instantiated on their own (or subclassed) and passed to training methods in mangoes.modeling.finetuning for more customization/control.

mangoes.modeling.training_utils.freeze_base_layers(model)¶: Function to freeze the base layer in a fine tuning/pretraining model

class mangoes.modeling.training_utils.DataCollatorPossiblyWithExtraInputs(tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase, padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = True, max_length: Optional[int] = None, pad_to_multiple_of: Optional[int] = None, return_tensors: str = 'pt')¶

Bases: transformers.data.data_collator.DataCollatorWithPadding

Attributes

max_length
pad_to_multiple_of

Methods

__call__(features)

Call self as a function.

max_length: Optional[int] = None¶

pad_to_multiple_of: Optional[int] = None¶

padding: Union[bool, str, transformers.utils.generic.PaddingStrategy] = True¶

return_tensors: str = 'pt'¶

tokenizer: transformers.tokenization_utils_base.PreTrainedTokenizerBase¶

class mangoes.modeling.training_utils.MangoesTextClassificationDataset(texts, labels, tokenizer, text_pairs=None, max_len=None, label2id=None, include_labels=True, text_annotations=None, text_pair_annotations=None)¶

Bases: Generic[torch.utils.data.dataset.T_co]

Subclass of torch.utils.data.Dataset class for sequence or token classification. To be used with transformers.Trainer.

Parameters

texts: List[str] or List[List[str]]: Input text sequences. Note that token classification, inputs must be pre-split to line up with token labels.
labels: List[str or int] if seq classification, else List[List[str or int]] if token classification: Labels can be raw strings, which will us label2id to convert to ids, or the ids themselves.
tokenizer: transformers.Tokenizer
max_len: int: max length of input sequences, if None, will default to tokenizer.model_max_length
label2id: dict of str -> int: if labels are not already converted to output ids, dictionary with mapping to use.

Methods

get_word_ids

get_word_ids(text)¶

mangoes.modeling.training_utils.update_input_spans(old_input_ids, new_input_ids, start_index, end_index)¶

Some enhanced models insert entity ids or other augmented ids into the input sequences. However, if predicting a span (coref, question answering tasks), this can change the span position. This function takes the original and new input id sequences and spans, and updates the spans if the new sequence contains inserted ids.

Parameters

old_input_ids – torch tensor of ids
new_input_ids – torch tensor of possible augmented ids
start_index – torch tensor containing start of span in input sequence
end_index – torch tensor containing end of span in input sequence

Returns

(new_start_index, new_end_index

class mangoes.modeling.training_utils.MangoesQuestionAnsweringDataset(tokenizer, question_texts, context_texts, answer_texts, start_indices, max_seq_length=512, doc_stride=128, max_query_length=64)¶

Bases: Generic[torch.utils.data.dataset.T_co]

Subclass of Torch Dataset for question answering datasets. Currently meant to work with BERT models.

Parameters

tokenizer: transformers.Tokenizer
question_texts: List of str: The texts corresponding to the questions
context_texts: List of str: The texts corresponding to the contexts
answer_texts: List of str: The texts corresponding to the answers
start_indices: List of int: The character positions of the start of the answers
max_seq_length:int: The maximum total input sequence length after tokenization.
doc_stride: int: When splitting up a long document into chunks, how much stride to take between chunks.
max_query_length: int: The maximum number of tokens for the question.

class mangoes.modeling.training_utils.QuestionAnsweringPipelinePossiblyWithEntities(model: Union[PreTrainedModel, TFPreTrainedModel], tokenizer: transformers.tokenization_utils.PreTrainedTokenizer, modelcard: Optional[transformers.modelcard.ModelCard] = None, framework: Optional[str] = None, device: int = - 1, task: str = '', **kwargs)¶

Bases: transformers.pipelines.question_answering.QuestionAnsweringPipeline

Methods

`__call__`(args, *kwargs)	Answer the question(s) given as inputs by using the context(s).
`check_model_type`(supported_models)	Check if the model class is in supported by the pipeline.
`create_sample`(question, context)	QuestionAnsweringPipeline leverages the [SquadExample] internally.
`device_placement`()	Context Manager allowing tensor allocation on the user-specified device in framework agnostic way.
`ensure_tensor_on_device`(**inputs)	Ensure PyTorch tensors are on the specified device.
`postprocess`(model_outputs[, top_k, …])	Postprocess will receive the raw outputs of the _forward method, generally tensors, and reformat them into something more friendly.
`predict`(X)	Scikit / Keras interface to transformers’ pipelines.
`preprocess`(example[, padding, doc_stride, …])	Preprocess will take the input_ of a specific pipeline and return a dictionnary of everything necessary for _forward to run properly.
`save_pretrained`(save_directory)	Save the pipeline’s model and tokenizer.
`span_to_answer`(text, start, end)	When decoding from token probabilities, this method maps token indexes to actual word in the initial context.
`transform`(X)	Scikit / Keras interface to transformers’ pipelines.

forward
get_indices
get_inference_context
get_iterator
iterate
run_multi
run_single

check_model_type(supported_models: Union[List[str], dict])¶

Check if the model class is in supported by the pipeline.

Args:

supported_models (List[str] or dict):: The list of models supported by the pipeline, or a dictionary with model class values.

static create_sample(question: Union[str, List[str]], context: Union[str, List[str]]) → Union[transformers.data.processors.squad.SquadExample, List[transformers.data.processors.squad.SquadExample]]¶

QuestionAnsweringPipeline leverages the [SquadExample] internally. This helper method encapsulate all the logic for converting question(s) and context(s) to [SquadExample].

We currently support extractive question answering.

Arguments:: question (str or List[str]): The question(s) asked. context (str or List[str]): The context(s) in which we will look for the answer.
Returns:: One or a list of [SquadExample]: The corresponding [SquadExample] grouping question and context.

default_input_names = 'question,context'¶

device_placement()¶

Context Manager allowing tensor allocation on the user-specified device in framework agnostic way.

Returns:: Context manager

Examples:

```python # Explicitly ask for tensor allocation on CUDA device :0 pipe = pipeline(…, device=0) with pipe.device_placement():

# Every framework specific tensor allocation will be done on the request device output = pipe(…)

```

ensure_tensor_on_device(**inputs)¶

Ensure PyTorch tensors are on the specified device.

Args:

inputs (keyword arguments that should be torch.Tensor, the rest is ignored):: The tensors to place on self.device.

Recursive on lists only.

Return:

Dict[str, torch.Tensor]: The same as inputs but on the proper device.

forward(model_inputs, **forward_params)¶

get_indices(enc: tokenizers.Encoding, s: int, e: int, sequence_index: int, align_to_words: bool) → Tuple[int, int]¶

get_inference_context()¶

get_iterator(inputs, num_workers: int, batch_size: int, preprocess_params, forward_params, postprocess_params)¶

handle_impossible_answer = False¶

iterate(inputs, preprocess_params, forward_params, postprocess_params)¶

postprocess(model_outputs, top_k=1, handle_impossible_answer=False, max_answer_len=15, align_to_words=True)¶: Postprocess will receive the raw outputs of the _forward method, generally tensors, and reformat them into something more friendly. Generally it will output a list or a dict or results (containing just strings and numbers).

predict(X)¶: Scikit / Keras interface to transformers’ pipelines. This method will forward to __call__().

preprocess(example, padding='do_not_pad', doc_stride=None, max_question_len=64, max_seq_len=None)¶: Preprocess will take the input_ of a specific pipeline and return a dictionnary of everything necessary for _forward to run properly. It should contain at least one tensor, but might have arbitrary other items.

run_multi(inputs, preprocess_params, forward_params, postprocess_params)¶

run_single(inputs, preprocess_params, forward_params, postprocess_params)¶

save_pretrained(save_directory: str)¶

Save the pipeline’s model and tokenizer.

Args:

save_directory (str):: A path to the directory where to saved. It will be created if it doesn’t exist.

span_to_answer(text: str, start: int, end: int) → Dict[str, Union[str, int]]¶

When decoding from token probabilities, this method maps token indexes to actual word in the initial context.

Args:: text (str): The actual context to extract the answer from. start (int): The answer starting token index. end (int): The answer end token index.
Returns:: Dictionary like {‘answer’: str, ‘start’: int, ‘end’: int}

transform(X)¶: Scikit / Keras interface to transformers’ pipelines. This method will forward to __call__().

class mangoes.modeling.training_utils.MangoesCoreferenceDataset(tokenizer, use_metadata, max_segment_len, max_segments, documents, cluster_ids, speaker_ids=None, genres=None, genre_to_id=None)¶

Bases: Generic[torch.utils.data.dataset.T_co]

Subclass of Torch Dataset for co-reference datasets such as Ontonotes. Currently meant to work with BERT models.

Each example is one document. Documents are parsed by first tokenizing each sentence then aggregating sentences into segments, keeping track of label, metadata, and sentence indices.

Parameters

tokenizer: transformers.BertTokenizerFast: tokenizer to use
use_metadata: Boolean: Whether or not to use speaker ids and genres
max_segment_len: int: maximum number of sub-tokens for one segment
max_segments: int: Maximum number of segments to return per __getitem__ (ie per document)
documents: List of Lists of Lists of strings: Text for each document. As cluster ids are labeled by word, a document is a list of sentences. One sentence is a list of words (ie already split on whitespace/punctuation)
cluster_ids: List of Lists of Lists of (ints or Tuple(int, int)): Cluster ids for each word in documents argument. Assumes words that aren’t mentions have either None or -1 as id. In the case where a word belongs to two different spans (with different cluster ids), the cluster id for word should be a tuple of ints corresponding to the different cluster ids.
speaker_ids: List of Lists of Lists of ints: Speaker id for each word in documents. Assumes positive ids (special tokens (such as [CLS] and [SEP] that are added at beginning and end of segments) will be assigned speaker ids of -1)
genres: List of ints or strings: Genre (id) for each document. If strings, genre_to_id parameter needs to not be None
genre_to_id: dict of string->int: Mapping of genres to their id number.

Methods

`get_subtoken_data`(token_data, offset_mapping)	Function to map token data to sub tokens.
`pad_list`(values, target_length[, pad_value])	Function to pad a list of values to a specific length, appending the pad_value to the end of the list.

static pad_list(values, target_length, pad_value=0)¶

Function to pad a list of values to a specific length, appending the pad_value to the end of the list.

Parameters

values: List
target_length: int
pad_value: value pad the list with

Returns

list of values, padded to target length

static get_subtoken_data(token_data, offset_mapping)¶

Function to map token data to sub tokens. For example, if a token is split into two sub-tokens, the cluster id (or speaker id) for the token needs to be associated with both sub-tokens.

Parameters

token_data: cluster ids of tokens
offset_mapping: for each sub-token, a (start index, end index) tuple of indices into it’s original token: As returned by a transformers.tokenizer if return_offsets_mapping=True.

Returns

List containing cluster ids for each token

class mangoes.modeling.training_utils.DataCollatorForMultipleChoice(tokenizer, padding=True, max_length=None, pad_to_multiple_of=None)¶

Bases: object

Data collator that will dynamically pad the inputs for multiple choice received.

Methods

__call__(features)

Call self as a function.

class mangoes.modeling.training_utils.MangoesMultipleChoiceDataset(tokenizer, question_texts, choices_texts, labels)¶

Bases: Generic[torch.utils.data.dataset.T_co]

Subclass of Torch Dataset for multiple choice datasets such as SWAG.

For information on how multiple choice datasets are formatted using this class, see https://github.com/google-research/bert/issues/38

And this link for explanation of Huggingface’s multiple choice models: https://github.com/huggingface/transformers/issues/7701#issuecomment-707149546

Parameters

tokenizer: transformer.tokenizer: tokenizer to use
question_texts: List of str: The texts corresponding to the questions/contexts.
choices_texts: List of str: The texts corresponding to the answer choices
labels: List of int: The indices of the correct answers

class mangoes.modeling.training_utils.MultipleLearnRateFineTuneTrainer(task_learn_rate, task_weight_decay, base_keyword='bert', model=None, args=None, data_collator=None, train_dataset=None, eval_dataset=None, tokenizer=None, model_init=None, compute_metrics=None, callbacks=None, optimizers=(None, None))¶

Bases: transformers.trainer.Trainer

Subclass of Huggingface Trainer to accept different learning rates for base model parameters and task specific parameters, in the context of a fine-tuning task.

task_learn_rate: float: Learning rate to be used for task specific parameters, (base parameters will use the normal, ie already defined in args, learn rate)
task_weight_decay: float: Weight decay to be used for task specific parameters, (base parameters will use the normal, ie already defined in args, weight decay)
base_keyword: str: String to be used to differentiate base model and task specific parameters. All named parameters that have “base_keyword” somewhere in the name will be considered part of the base model, while all parameters that don’t will be considered part of the task specific parameters.
For documentation of the rest of the init parameters, see: https://huggingface.co/transformers/main_classes/trainer.html#id1

Methods

`add_callback`(callback)	Add a callback to the current list of [~transformer.TrainerCallback].
`autocast_smart_context_manager`([cache_enabled])	A helper wrapper that creates an appropriate context manager for autocast while feeding it the desired arguments, depending on the situation.
`compute_loss`(model, inputs[, return_outputs])	How the loss is computed by Trainer.
`compute_loss_context_manager`()	A helper wrapper to group together context managers.
`create_model_card`([language, license, tags, …])	Creates a draft of a model card using the information available to the Trainer.
`create_optimizer`()	Setup the optimizer.
`create_optimizer_and_scheduler`(…)	Setup the optimizer and the learning rate scheduler.
`create_scheduler`(num_training_steps[, optimizer])	Setup the scheduler.
`evaluate`([eval_dataset, ignore_keys, …])	Run evaluation and returns metrics.
`evaluation_loop`(dataloader, description[, …])	Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().
`floating_point_ops`(inputs)	For models that inherit from [PreTrainedModel], uses that method to compute the number of floating point operations for every backward + forward pass.
`get_eval_dataloader`([eval_dataset])	Returns the evaluation [~torch.utils.data.DataLoader].
`get_optimizer_cls_and_kwargs`(args)	Returns the optimizer class and optimizer parameters based on the training arguments.
`get_test_dataloader`(test_dataset)	Returns the test [~torch.utils.data.DataLoader].
`get_train_dataloader`()	Returns the training [~torch.utils.data.DataLoader].
`hyperparameter_search`([hp_space, …])	Launch an hyperparameter search using optuna or Ray Tune or SigOpt.
`init_git_repo`([at_init])	Initializes a git repo in self.args.hub_model_id.
`is_local_process_zero`()	Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process.
`is_world_process_zero`()	Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be True for one process).
`log`(logs)	Log logs on the various objects watching training.
`log_metrics`(split, metrics)	Log metrics in a specially formatted way
`metrics_format`(metrics)	Reformat Trainer metrics values to a human-readable format
`num_examples`(dataloader)	Helper to get number of samples in a [~torch.utils.data.DataLoader] by accessing its dataset.
`pop_callback`(callback)	Remove a callback from the current list of [~transformer.TrainerCallback] and returns it.
`predict`(test_dataset[, ignore_keys, …])	Run prediction and returns predictions and potential metrics.
`prediction_loop`(dataloader, description[, …])	Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().
`prediction_step`(model, inputs, …[, …])	Perform an evaluation step on model using inputs.
`push_to_hub`([commit_message, blocking])	Upload self.model and self.tokenizer to the 🤗 model hub on the repo self.args.hub_model_id.
`remove_callback`(callback)	Remove a callback from the current list of [~transformer.TrainerCallback].
`save_metrics`(split, metrics[, combined])	Save metrics into a json file for that split, e.g.
`save_model`([output_dir, _internal_call])	Will save the model, so you can reload it using from_pretrained().
`save_state`()	Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model
`train`([resume_from_checkpoint, trial, …])	Main training entry point.
`training_step`(model, inputs)	Perform a training step on a batch of inputs.

call_model_init
ipex_optimize_model
store_flos
torch_jit_model_eval

create_optimizer_and_scheduler(num_training_steps)¶

Setup the optimizer and the learning rate scheduler.

This will use AdamW. If you want to use something else (ie, a different optimizer and multiple learn rates), you can subclass and override this method in a subclass.

add_callback(callback)¶

Add a callback to the current list of [~transformer.TrainerCallback].

Args:

callback (type or [~transformer.TrainerCallback]):: A [~transformer.TrainerCallback] class or an instance of a [~transformer.TrainerCallback]. In the first case, will instantiate a member of that class.

autocast_smart_context_manager(cache_enabled: Optional[bool] = True)¶: A helper wrapper that creates an appropriate context manager for autocast while feeding it the desired arguments, depending on the situation.

call_model_init(trial=None)¶

compute_loss(model, inputs, return_outputs=False)¶

How the loss is computed by Trainer. By default, all models return the loss in the first element.

Subclass and override for custom behavior.

compute_loss_context_manager()¶: A helper wrapper to group together context managers.

create_model_card(language: Optional[str] = None, license: Optional[str] = None, tags: Optional[Union[str, List[str]]] = None, model_name: Optional[str] = None, finetuned_from: Optional[str] = None, tasks: Optional[Union[str, List[str]]] = None, dataset_tags: Optional[Union[str, List[str]]] = None, dataset: Optional[Union[str, List[str]]] = None, dataset_args: Optional[Union[str, List[str]]] = None)¶

Creates a draft of a model card using the information available to the Trainer.

Args:

language (str, optional):: The language of the model (if applicable)
license (str, optional):: The license of the model. Will default to the license of the pretrained model used, if the original model given to the Trainer comes from a repo on the Hub.
tags (str or List[str], optional):: Some tags to be included in the metadata of the model card.
model_name (str, optional):: The name of the model.
finetuned_from (str, optional):: The name of the model used to fine-tune this one (if applicable). Will default to the name of the repo of the original model given to the Trainer (if it comes from the Hub).
tasks (str or List[str], optional):: One or several task identifiers, to be included in the metadata of the model card.
dataset_tags (str or List[str], optional):: One or several dataset tags, to be included in the metadata of the model card.
dataset (str or List[str], optional):: One or several dataset identifiers, to be included in the metadata of the model card.
dataset_args (str or List[str], optional):: One or several dataset arguments, to be included in the metadata of the model card.

create_optimizer()¶

Setup the optimizer.

We provide a reasonable default that works well. If you want to use something else, you can pass a tuple in the Trainer’s init through optimizers, or subclass and override this method in a subclass.

create_scheduler(num_training_steps: int, optimizer: Optional[torch.optim.optimizer.Optimizer] = None)¶

Setup the scheduler. The optimizer of the trainer must have been set up either before this method is called or passed as an argument.

Args:: num_training_steps (int): The number of training steps to do.

evaluate(eval_dataset: Optional[torch.utils.data.dataset.Dataset] = None, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'eval') → Dict[str, float]¶

Run evaluation and returns metrics.

The calling script will be responsible for providing a method to compute metrics, as they are task-dependent (pass it to the init compute_metrics argument).

You can also subclass and override this method to inject custom behavior.

Args:

eval_dataset (Dataset, optional):: Pass a dataset if you wish to override self.eval_dataset. If it is a [~datasets.Dataset], columns not accepted by the model.forward() method are automatically removed. It must implement the __len__ method.
ignore_keys (Lst[str], optional):: A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.
metric_key_prefix (str, optional, defaults to “eval”):: An optional prefix to be used as the metrics key prefix. For example the metrics “bleu” will be named “eval_bleu” if the prefix is “eval” (default)

Returns:

A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The dictionary also contains the epoch number which comes from the training state.

evaluation_loop(dataloader: torch.utils.data.dataloader.DataLoader, description: str, prediction_loss_only: Optional[bool] = None, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'eval') → transformers.trainer_utils.EvalLoopOutput¶

Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().

Works both with or without labels.

floating_point_ops(inputs: Dict[str, Union[torch.Tensor, Any]])¶

For models that inherit from [PreTrainedModel], uses that method to compute the number of floating point operations for every backward + forward pass. If using another model, either implement such a method in the model or subclass and override this method.

Args:

inputs (Dict[str, Union[torch.Tensor, Any]]):: The inputs and targets of the model.

Returns:

int: The number of floating-point operations.

get_eval_dataloader(eval_dataset: Optional[torch.utils.data.dataset.Dataset] = None) → torch.utils.data.dataloader.DataLoader¶

Returns the evaluation [~torch.utils.data.DataLoader].

Subclass and override this method if you want to inject some custom behavior.

Args:

eval_dataset (torch.utils.data.Dataset, optional):: If provided, will override self.eval_dataset. If it is a [~datasets.Dataset], columns not accepted by the model.forward() method are automatically removed. It must implement __len__.

static get_optimizer_cls_and_kwargs(args: transformers.training_args.TrainingArguments) → Tuple[Any, Any]¶

Returns the optimizer class and optimizer parameters based on the training arguments.

Args:

args (transformers.training_args.TrainingArguments):: The training arguments for the training session.

get_test_dataloader(test_dataset: torch.utils.data.dataset.Dataset) → torch.utils.data.dataloader.DataLoader¶

Returns the test [~torch.utils.data.DataLoader].

Subclass and override this method if you want to inject some custom behavior.

Args:

test_dataset (torch.utils.data.Dataset, optional):: The test dataset to use. If it is a [~datasets.Dataset], columns not accepted by the model.forward() method are automatically removed. It must implement __len__.

get_train_dataloader() → torch.utils.data.dataloader.DataLoader¶

Returns the training [~torch.utils.data.DataLoader].

Will use no sampler if train_dataset does not implement __len__, a random sampler (adapted to distributed training if necessary) otherwise.

Subclass and override this method if you want to inject some custom behavior.

hyperparameter_search(hp_space: Optional[Callable[[optuna.Trial], Dict[str, float]]] = None, compute_objective: Optional[Callable[[Dict[str, float]], float]] = None, n_trials: int = 20, direction: str = 'minimize', backend: Optional[Union[str, transformers.trainer_utils.HPSearchBackend]] = None, hp_name: Optional[Callable[[optuna.Trial], str]] = None, **kwargs) → transformers.trainer_utils.BestRun¶

Launch an hyperparameter search using optuna or Ray Tune or SigOpt. The optimized quantity is determined by compute_objective, which defaults to a function returning the evaluation loss when no metric is provided, the sum of all metrics otherwise.

To use this method, you need to have provided a model_init when initializing your [Trainer]: we need to reinitialize the model at each new run. This is incompatible with the optimizers argument, so you need to subclass [Trainer] and override the method [~Trainer.create_optimizer_and_scheduler] for custom optimizer/scheduler.

</Tip>

Args:

hp_space (Callable[[“optuna.Trial”], Dict[str, float]], optional):

A function that defines the hyperparameter search space. Will default to [~trainer_utils.default_hp_space_optuna] or [~trainer_utils.default_hp_space_ray] or [~trainer_utils.default_hp_space_sigopt] depending on your backend.

compute_objective (Callable[[Dict[str, float]], float], optional):

A function computing the objective to minimize or maximize from the metrics returned by the evaluate method. Will default to [~trainer_utils.default_compute_objective].

n_trials (int, optional, defaults to 100):

The number of trial runs to test.

direction (str, optional, defaults to “minimize”):

Whether to optimize greater or lower objects. Can be “minimize” or “maximize”, you should pick “minimize” when optimizing the validation loss, “maximize” when optimizing one or several metrics.

backend (str or [~training_utils.HPSearchBackend], optional):

The backend to use for hyperparameter search. Will default to optuna or Ray Tune or SigOpt, depending on which one is installed. If all are installed, will default to optuna.

hp_name (Callable[[“optuna.Trial”], str]], optional):

A function that defines the trial/run name. Will default to None.

kwargs (Dict[str, Any], optional):

Additional keyword arguments passed along to optuna.create_study or ray.tune.run. For more information see:

the documentation of [optuna.create_study](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html)
the documentation of [tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run)
the documentation of [sigopt](https://app.sigopt.com/docs/endpoints/experiments/create)

Returns:

[trainer_utils.BestRun]: All the information about the best run.

init_git_repo(at_init: bool = False)¶

Initializes a git repo in self.args.hub_model_id.

Args:

at_init (bool, optional, defaults to False):: Whether this function is called before any training or not. If self.args.overwrite_output_dir is True and at_init is True, the path to the repo (which is self.args.output_dir) might be wiped out.

ipex_optimize_model(model, training=False, dtype=torch.float32)¶

is_local_process_zero() → bool¶: Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process.

is_world_process_zero() → bool¶: Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be True for one process).

log(logs: Dict[str, float]) → None¶

Log logs on the various objects watching training.

Subclass and override this method to inject custom behavior.

Args:

logs (Dict[str, float]):: The values to log.

log_metrics(split, metrics)¶

Log metrics in a specially formatted way

Under distributed environment this is done only for a process with rank 0.

Args:

split (str):: Mode/split name: one of train, eval, test
metrics (Dict[str, float]):: The metrics returned from train/evaluate/predictmetrics: metrics dict

Notes on memory reports:

In order to get memory usage report you need to install psutil. You can do that with pip install psutil.

Now when this method is run, you will see a report that will include: :

` init_mem_cpu_alloc_delta = 1301MB init_mem_cpu_peaked_delta = 154MB init_mem_gpu_alloc_delta = 230MB init_mem_gpu_peaked_delta = 0MB train_mem_cpu_alloc_delta = 1345MB train_mem_cpu_peaked_delta = 0MB train_mem_gpu_alloc_delta = 693MB train_mem_gpu_peaked_delta = 7MB `

Understanding the reports:

the first segment, e.g., train__, tells you which stage the metrics are for. Reports starting with init_
will be added to the first stage that gets run. So that if only evaluation is run, the memory usage for the __init__ will be reported along with the eval_ metrics.
the third segment, is either cpu or gpu, tells you whether it’s the general RAM or the gpu0 memory
metric.
*_alloc_delta - is the difference in the used/allocated memory counter between the end and the start of the
stage - it can be negative if a function released more memory than it allocated.
*_peaked_delta - is any extra memory that was consumed and then freed - relative to the current allocated
memory counter - it is never negative. When you look at the metrics of any stage you add up alloc_delta + peaked_delta and you know how much memory was needed to complete that stage.

The reporting happens only for process of rank 0 and gpu 0 (if there is a gpu). Typically this is enough since the main process does the bulk of work, but it could be not quite so if model parallel is used and then other GPUs may use a different amount of gpu memory. This is also not the same under DataParallel where gpu0 may require much more memory than the rest since it stores the gradient and optimizer states for all participating GPUS. Perhaps in the future these reports will evolve to measure those too.

The CPU RAM metric measures RSS (Resident Set Size) includes both the memory which is unique to the process and the memory shared with other processes. It is important to note that it does not include swapped out memory, so the reports could be imprecise.

The CPU peak memory is measured using a sampling thread. Due to python’s GIL it may miss some of the peak memory if that thread didn’t get a chance to run when the highest memory was used. Therefore this report can be less than reality. Using tracemalloc would have reported the exact peak memory, but it doesn’t report memory allocations outside of python. So if some C++ CUDA extension allocated its own memory it won’t be reported. And therefore it was dropped in favor of the memory sampling approach, which reads the current process memory usage.

The GPU allocated and peak memory reporting is done with torch.cuda.memory_allocated() and torch.cuda.max_memory_allocated(). This metric reports only “deltas” for pytorch-specific allocations, as torch.cuda memory management system doesn’t track any memory allocated outside of pytorch. For example, the very first cuda call typically loads CUDA kernels, which may take from 0.5 to 2GB of GPU memory.

Note that this tracker doesn’t account for memory allocations outside of [Trainer]’s __init__, train, evaluate and predict calls.

Because evaluation calls may happen during train, we can’t handle nested invocations because torch.cuda.max_memory_allocated is a single counter, so if it gets reset by a nested eval call, train’s tracker will report incorrect info. If this [pytorch issue](https://github.com/pytorch/pytorch/issues/16266) gets resolved it will be possible to change this class to be re-entrant. Until then we will only track the outer level of train, evaluate and predict methods. Which means that if eval is called during train, it’s the latter that will account for its memory usage and that of the former.

This also means that if any other tool that is used along the [Trainer] calls torch.cuda.reset_peak_memory_stats, the gpu peak memory stats could be invalid. And the [Trainer] will disrupt the normal behavior of any such tools that rely on calling torch.cuda.reset_peak_memory_stats themselves.

For best performance you may want to consider turning the memory profiling off for production runs.

metrics_format(metrics: Dict[str, float]) → Dict[str, float]¶

Reformat Trainer metrics values to a human-readable format

Args:

metrics (Dict[str, float]):: The metrics returned from train/evaluate/predict

Returns:

metrics (Dict[str, float]): The reformatted metrics

num_examples(dataloader: torch.utils.data.dataloader.DataLoader) → int¶: Helper to get number of samples in a [~torch.utils.data.DataLoader] by accessing its dataset. When dataloader.dataset does not exist or has no length, estimates as best it can

pop_callback(callback)¶

Remove a callback from the current list of [~transformer.TrainerCallback] and returns it.

If the callback is not found, returns None (and no error is raised).

Args:

callback (type or [~transformer.TrainerCallback]):: A [~transformer.TrainerCallback] class or an instance of a [~transformer.TrainerCallback]. In the first case, will pop the first member of that class found in the list of callbacks.

Returns:

[~transformer.TrainerCallback]: The callback removed, if found.

predict(test_dataset: torch.utils.data.dataset.Dataset, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'test') → transformers.trainer_utils.PredictionOutput¶

Run prediction and returns predictions and potential metrics.

Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in evaluate().

Args:

test_dataset (Dataset):: Dataset to run the predictions on. If it is an datasets.Dataset, columns not accepted by the model.forward() method are automatically removed. Has to implement the method __len__
ignore_keys (Lst[str], optional):: A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.
metric_key_prefix (str, optional, defaults to “test”):: An optional prefix to be used as the metrics key prefix. For example the metrics “bleu” will be named “test_bleu” if the prefix is “test” (default)

<Tip>

If your predictions or labels have different sequence length (for instance because you’re doing dynamic padding in a token classification task) the predictions will be padded (on the right) to allow for concatenation into one array. The padding index is -100.

</Tip>

Returns: NamedTuple A namedtuple with the following keys:

predictions (np.ndarray): The predictions on test_dataset.

label_ids (np.ndarray, optional): The labels (if the dataset contained some).

metrics (Dict[str, float], optional): The potential dictionary of metrics (if the dataset contained labels).

prediction_loop(dataloader: torch.utils.data.dataloader.DataLoader, description: str, prediction_loss_only: Optional[bool] = None, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'eval') → transformers.trainer_utils.PredictionOutput¶

Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().

Works both with or without labels.

prediction_step(model: torch.nn.modules.module.Module, inputs: Dict[str, Union[torch.Tensor, Any]], prediction_loss_only: bool, ignore_keys: Optional[List[str]] = None) → Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]¶

Perform an evaluation step on model using inputs.

Subclass and override to inject custom behavior.

Args:

model (nn.Module):

The model to evaluate.

inputs (Dict[str, Union[torch.Tensor, Any]]):

The inputs and targets of the model.

The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument labels. Check your model’s documentation for all accepted arguments.

prediction_loss_only (bool):

Whether or not to return the loss only.

ignore_keys (Lst[str], optional):

A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.

Return:

Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss, logits and labels (each being optional).

push_to_hub(commit_message: Optional[str] = 'End of training', blocking: bool = True, **kwargs) → str¶

Upload self.model and self.tokenizer to the 🤗 model hub on the repo self.args.hub_model_id.

Parameters:

commit_message (str, optional, defaults to “End of training”):: Message to commit while pushing.
blocking (bool, optional, defaults to True):: Whether the function should return only when the git push has finished.
kwargs:: Additional keyword arguments passed along to [~Trainer.create_model_card].

Returns:

The url of the commit of your model in the given repository if blocking=False, a tuple with the url of the commit and an object to track the progress of the commit if blocking=True

remove_callback(callback)¶

Remove a callback from the current list of [~transformer.TrainerCallback].

Args:

callback (type or [~transformer.TrainerCallback]):: A [~transformer.TrainerCallback] class or an instance of a [~transformer.TrainerCallback]. In the first case, will remove the first member of that class found in the list of callbacks.

save_metrics(split, metrics, combined=True)¶

Save metrics into a json file for that split, e.g. train_results.json.

Under distributed environment this is done only for a process with rank 0.

Args:

split (str):: Mode/split name: one of train, eval, test, all
metrics (Dict[str, float]):: The metrics returned from train/evaluate/predict
combined (bool, optional, defaults to True):: Creates combined metrics by updating all_results.json with metrics of this call

To understand the metrics please read the docstring of [~Trainer.log_metrics]. The only difference is that raw unformatted numbers are saved in the current method.

save_model(output_dir: Optional[str] = None, _internal_call: bool = False)¶

Will save the model, so you can reload it using from_pretrained().

Will only save from the main process.

save_state()¶

Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model

Under distributed environment this is done only for a process with rank 0.

store_flos()¶

torch_jit_model_eval(model, dataloader, training=False)¶

train(resume_from_checkpoint: Optional[Union[bool, str]] = None, trial: Union[optuna.Trial, Dict[str, Any]] = None, ignore_keys_for_eval: Optional[List[str]] = None, **kwargs)¶

Main training entry point.

Args:

resume_from_checkpoint (str or bool, optional):: If a str, local path to a saved checkpoint as saved by a previous instance of [Trainer]. If a bool and equals True, load the last checkpoint in args.output_dir as saved by a previous instance of [Trainer]. If present, training will resume from the model/optimizer/scheduler states loaded here.
trial (optuna.Trial or Dict[str, Any], optional):: The trial run or the hyperparameter dictionary for hyperparameter search.
ignore_keys_for_eval (List[str], optional): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions for evaluation during the training.
kwargs:: Additional keyword arguments used to hide deprecated arguments

training_step(model: torch.nn.modules.module.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) → torch.Tensor¶

Perform a training step on a batch of inputs.

Subclass and override to inject custom behavior.

Args:

model (nn.Module):

The model to train.

inputs (Dict[str, Union[torch.Tensor, Any]]):

The inputs and targets of the model.

The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument labels. Check your model’s documentation for all accepted arguments.

Return:

torch.Tensor: The tensor with training loss on this batch.

class mangoes.modeling.training_utils.CoreferenceFineTuneTrainer(task_learn_rate, task_weight_decay, base_keyword='bert', model=None, args=None, train_dataset=None, eval_dataset=None, tokenizer=None, model_init=None, compute_metrics=None, callbacks=None, optimizers=(None, None))¶

Bases: mangoes.modeling.training_utils.MultipleLearnRateFineTuneTrainer

Subclass of the Mangoes MultipleLearnRateFineTuneTrainer that does not collate examples into batches, for use in the default Coreference Fine Tuning Trainer.

This method introduces a dummy batch collation method, because the batches in the implemented fine tuning method (see paper below) are exactly 1 document each, and are pre-collated in the dataset class. This is based on the independent variant of the coreference resolution method described in https://arxiv.org/pdf/1908.09091.pdf.

For documentation of the init parameters, see the documentation for MangoesMultipleLearnRateFineTuneTrainer

Methods

`add_callback`(callback)	Add a callback to the current list of [~transformer.TrainerCallback].
`autocast_smart_context_manager`([cache_enabled])	A helper wrapper that creates an appropriate context manager for autocast while feeding it the desired arguments, depending on the situation.
`compute_loss`(model, inputs[, return_outputs])	How the loss is computed by Trainer.
`compute_loss_context_manager`()	A helper wrapper to group together context managers.
`create_model_card`([language, license, tags, …])	Creates a draft of a model card using the information available to the Trainer.
`create_optimizer`()	Setup the optimizer.
`create_optimizer_and_scheduler`(…)	Setup the optimizer and the learning rate scheduler.
`create_scheduler`(num_training_steps[, optimizer])	Setup the scheduler.
`evaluate`([eval_dataset, ignore_keys, …])	Run evaluation and returns metrics.
`evaluation_loop`(dataloader, description[, …])	Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().
`floating_point_ops`(inputs)	For models that inherit from [PreTrainedModel], uses that method to compute the number of floating point operations for every backward + forward pass.
`get_eval_dataloader`([eval_dataset])	Returns the evaluation [~torch.utils.data.DataLoader].
`get_optimizer_cls_and_kwargs`(args)	Returns the optimizer class and optimizer parameters based on the training arguments.
`get_test_dataloader`(test_dataset)	Returns the test [~torch.utils.data.DataLoader].
`get_train_dataloader`()	Returns the training [~torch.utils.data.DataLoader].
`hyperparameter_search`([hp_space, …])	Launch an hyperparameter search using optuna or Ray Tune or SigOpt.
`init_git_repo`([at_init])	Initializes a git repo in self.args.hub_model_id.
`is_local_process_zero`()	Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process.
`is_world_process_zero`()	Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be True for one process).
`log`(logs)	Log logs on the various objects watching training.
`log_metrics`(split, metrics)	Log metrics in a specially formatted way
`metrics_format`(metrics)	Reformat Trainer metrics values to a human-readable format
`num_examples`(dataloader)	Helper to get number of samples in a [~torch.utils.data.DataLoader] by accessing its dataset.
`pop_callback`(callback)	Remove a callback from the current list of [~transformer.TrainerCallback] and returns it.
`predict`(test_dataset[, ignore_keys, …])	Run prediction and returns predictions and potential metrics.
`prediction_loop`(dataloader, description[, …])	Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().
`prediction_step`(model, inputs, …[, …])	Perform an evaluation step on model using inputs.
`push_to_hub`([commit_message, blocking])	Upload self.model and self.tokenizer to the 🤗 model hub on the repo self.args.hub_model_id.
`remove_callback`(callback)	Remove a callback from the current list of [~transformer.TrainerCallback].
`save_metrics`(split, metrics[, combined])	Save metrics into a json file for that split, e.g.
`save_model`([output_dir, _internal_call])	Will save the model, so you can reload it using from_pretrained().
`save_state`()	Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model
`train`([resume_from_checkpoint, trial, …])	Main training entry point.
`training_step`(model, inputs)	Perform a training step on a batch of inputs.

call_model_init
ipex_optimize_model
store_flos
torch_jit_model_eval

add_callback(callback)¶

Add a callback to the current list of [~transformer.TrainerCallback].

Args:

callback (type or [~transformer.TrainerCallback]):: A [~transformer.TrainerCallback] class or an instance of a [~transformer.TrainerCallback]. In the first case, will instantiate a member of that class.

autocast_smart_context_manager(cache_enabled: Optional[bool] = True)¶: A helper wrapper that creates an appropriate context manager for autocast while feeding it the desired arguments, depending on the situation.

call_model_init(trial=None)¶

compute_loss(model, inputs, return_outputs=False)¶

How the loss is computed by Trainer. By default, all models return the loss in the first element.

Subclass and override for custom behavior.

compute_loss_context_manager()¶: A helper wrapper to group together context managers.

Creates a draft of a model card using the information available to the Trainer.

Args:

language (str, optional):: The language of the model (if applicable)
license (str, optional):: The license of the model. Will default to the license of the pretrained model used, if the original model given to the Trainer comes from a repo on the Hub.
tags (str or List[str], optional):: Some tags to be included in the metadata of the model card.
model_name (str, optional):: The name of the model.
finetuned_from (str, optional):: The name of the model used to fine-tune this one (if applicable). Will default to the name of the repo of the original model given to the Trainer (if it comes from the Hub).
tasks (str or List[str], optional):: One or several task identifiers, to be included in the metadata of the model card.
dataset_tags (str or List[str], optional):: One or several dataset tags, to be included in the metadata of the model card.
dataset (str or List[str], optional):: One or several dataset identifiers, to be included in the metadata of the model card.
dataset_args (str or List[str], optional):: One or several dataset arguments, to be included in the metadata of the model card.

create_optimizer()¶

Setup the optimizer.

create_optimizer_and_scheduler(num_training_steps)¶

Setup the optimizer and the learning rate scheduler.

This will use AdamW. If you want to use something else (ie, a different optimizer and multiple learn rates), you can subclass and override this method in a subclass.

create_scheduler(num_training_steps: int, optimizer: Optional[torch.optim.optimizer.Optimizer] = None)¶

Setup the scheduler. The optimizer of the trainer must have been set up either before this method is called or passed as an argument.

Args:: num_training_steps (int): The number of training steps to do.

evaluate(eval_dataset: Optional[torch.utils.data.dataset.Dataset] = None, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'eval') → Dict[str, float]¶

Run evaluation and returns metrics.

The calling script will be responsible for providing a method to compute metrics, as they are task-dependent (pass it to the init compute_metrics argument).

You can also subclass and override this method to inject custom behavior.

Args:

eval_dataset (Dataset, optional):: Pass a dataset if you wish to override self.eval_dataset. If it is a [~datasets.Dataset], columns not accepted by the model.forward() method are automatically removed. It must implement the __len__ method.
ignore_keys (Lst[str], optional):: A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.
metric_key_prefix (str, optional, defaults to “eval”):: An optional prefix to be used as the metrics key prefix. For example the metrics “bleu” will be named “eval_bleu” if the prefix is “eval” (default)

Returns:

A dictionary containing the evaluation loss and the potential metrics computed from the predictions. The dictionary also contains the epoch number which comes from the training state.

Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().

Works both with or without labels.

floating_point_ops(inputs: Dict[str, Union[torch.Tensor, Any]])¶

Args:

inputs (Dict[str, Union[torch.Tensor, Any]]):: The inputs and targets of the model.

Returns:

int: The number of floating-point operations.

get_eval_dataloader(eval_dataset: Optional[torch.utils.data.dataset.Dataset] = None) → torch.utils.data.dataloader.DataLoader¶

Returns the evaluation [~torch.utils.data.DataLoader].

Subclass and override this method if you want to inject some custom behavior.

Args:

eval_dataset (torch.utils.data.Dataset, optional):: If provided, will override self.eval_dataset. If it is a [~datasets.Dataset], columns not accepted by the model.forward() method are automatically removed. It must implement __len__.

static get_optimizer_cls_and_kwargs(args: transformers.training_args.TrainingArguments) → Tuple[Any, Any]¶

Returns the optimizer class and optimizer parameters based on the training arguments.

Args:

args (transformers.training_args.TrainingArguments):: The training arguments for the training session.

get_test_dataloader(test_dataset: torch.utils.data.dataset.Dataset) → torch.utils.data.dataloader.DataLoader¶

Returns the test [~torch.utils.data.DataLoader].

Subclass and override this method if you want to inject some custom behavior.

Args:

test_dataset (torch.utils.data.Dataset, optional):: The test dataset to use. If it is a [~datasets.Dataset], columns not accepted by the model.forward() method are automatically removed. It must implement __len__.

get_train_dataloader() → torch.utils.data.dataloader.DataLoader¶

Returns the training [~torch.utils.data.DataLoader].

Will use no sampler if train_dataset does not implement __len__, a random sampler (adapted to distributed training if necessary) otherwise.

Subclass and override this method if you want to inject some custom behavior.

</Tip>

Args:

hp_space (Callable[[“optuna.Trial”], Dict[str, float]], optional):

compute_objective (Callable[[Dict[str, float]], float], optional):

A function computing the objective to minimize or maximize from the metrics returned by the evaluate method. Will default to [~trainer_utils.default_compute_objective].

n_trials (int, optional, defaults to 100):

The number of trial runs to test.

direction (str, optional, defaults to “minimize”):

backend (str or [~training_utils.HPSearchBackend], optional):

The backend to use for hyperparameter search. Will default to optuna or Ray Tune or SigOpt, depending on which one is installed. If all are installed, will default to optuna.

hp_name (Callable[[“optuna.Trial”], str]], optional):

A function that defines the trial/run name. Will default to None.

kwargs (Dict[str, Any], optional):

Additional keyword arguments passed along to optuna.create_study or ray.tune.run. For more information see:

the documentation of [optuna.create_study](https://optuna.readthedocs.io/en/stable/reference/generated/optuna.study.create_study.html)
the documentation of [tune.run](https://docs.ray.io/en/latest/tune/api_docs/execution.html#tune-run)
the documentation of [sigopt](https://app.sigopt.com/docs/endpoints/experiments/create)

Returns:

[trainer_utils.BestRun]: All the information about the best run.

init_git_repo(at_init: bool = False)¶

Initializes a git repo in self.args.hub_model_id.

Args:

at_init (bool, optional, defaults to False):: Whether this function is called before any training or not. If self.args.overwrite_output_dir is True and at_init is True, the path to the repo (which is self.args.output_dir) might be wiped out.

ipex_optimize_model(model, training=False, dtype=torch.float32)¶

is_local_process_zero() → bool¶: Whether or not this process is the local (e.g., on one machine if training in a distributed fashion on several machines) main process.

is_world_process_zero() → bool¶: Whether or not this process is the global main process (when training in a distributed fashion on several machines, this is only going to be True for one process).

log(logs: Dict[str, float]) → None¶

Log logs on the various objects watching training.

Subclass and override this method to inject custom behavior.

Args:

logs (Dict[str, float]):: The values to log.

log_metrics(split, metrics)¶

Log metrics in a specially formatted way

Under distributed environment this is done only for a process with rank 0.

Args:

split (str):: Mode/split name: one of train, eval, test
metrics (Dict[str, float]):: The metrics returned from train/evaluate/predictmetrics: metrics dict

Notes on memory reports:

In order to get memory usage report you need to install psutil. You can do that with pip install psutil.

Now when this method is run, you will see a report that will include: :

Understanding the reports:

the first segment, e.g., train__, tells you which stage the metrics are for. Reports starting with init_
will be added to the first stage that gets run. So that if only evaluation is run, the memory usage for the __init__ will be reported along with the eval_ metrics.
the third segment, is either cpu or gpu, tells you whether it’s the general RAM or the gpu0 memory
metric.
*_alloc_delta - is the difference in the used/allocated memory counter between the end and the start of the
stage - it can be negative if a function released more memory than it allocated.
*_peaked_delta - is any extra memory that was consumed and then freed - relative to the current allocated
memory counter - it is never negative. When you look at the metrics of any stage you add up alloc_delta + peaked_delta and you know how much memory was needed to complete that stage.

Note that this tracker doesn’t account for memory allocations outside of [Trainer]’s __init__, train, evaluate and predict calls.

For best performance you may want to consider turning the memory profiling off for production runs.

metrics_format(metrics: Dict[str, float]) → Dict[str, float]¶

Reformat Trainer metrics values to a human-readable format

Args:

metrics (Dict[str, float]):: The metrics returned from train/evaluate/predict

Returns:

metrics (Dict[str, float]): The reformatted metrics

num_examples(dataloader: torch.utils.data.dataloader.DataLoader) → int¶: Helper to get number of samples in a [~torch.utils.data.DataLoader] by accessing its dataset. When dataloader.dataset does not exist or has no length, estimates as best it can

pop_callback(callback)¶

Remove a callback from the current list of [~transformer.TrainerCallback] and returns it.

If the callback is not found, returns None (and no error is raised).

Args:

callback (type or [~transformer.TrainerCallback]):: A [~transformer.TrainerCallback] class or an instance of a [~transformer.TrainerCallback]. In the first case, will pop the first member of that class found in the list of callbacks.

Returns:

[~transformer.TrainerCallback]: The callback removed, if found.

predict(test_dataset: torch.utils.data.dataset.Dataset, ignore_keys: Optional[List[str]] = None, metric_key_prefix: str = 'test') → transformers.trainer_utils.PredictionOutput¶

Run prediction and returns predictions and potential metrics.

Depending on the dataset and your use case, your test dataset may contain labels. In that case, this method will also return metrics, like in evaluate().

Args:

test_dataset (Dataset):: Dataset to run the predictions on. If it is an datasets.Dataset, columns not accepted by the model.forward() method are automatically removed. Has to implement the method __len__
ignore_keys (Lst[str], optional):: A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.
metric_key_prefix (str, optional, defaults to “test”):: An optional prefix to be used as the metrics key prefix. For example the metrics “bleu” will be named “test_bleu” if the prefix is “test” (default)

<Tip>

</Tip>

Returns: NamedTuple A namedtuple with the following keys:

predictions (np.ndarray): The predictions on test_dataset.

label_ids (np.ndarray, optional): The labels (if the dataset contained some).

metrics (Dict[str, float], optional): The potential dictionary of metrics (if the dataset contained labels).

Prediction/evaluation loop, shared by Trainer.evaluate() and Trainer.predict().

Works both with or without labels.

Perform an evaluation step on model using inputs.

Subclass and override to inject custom behavior.

Args:

model (nn.Module):

The model to evaluate.

inputs (Dict[str, Union[torch.Tensor, Any]]):

The inputs and targets of the model.

The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument labels. Check your model’s documentation for all accepted arguments.

prediction_loss_only (bool):

Whether or not to return the loss only.

ignore_keys (Lst[str], optional):

A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions.

Return:

Tuple[Optional[torch.Tensor], Optional[torch.Tensor], Optional[torch.Tensor]]: A tuple with the loss, logits and labels (each being optional).

push_to_hub(commit_message: Optional[str] = 'End of training', blocking: bool = True, **kwargs) → str¶

Upload self.model and self.tokenizer to the 🤗 model hub on the repo self.args.hub_model_id.

Parameters:

commit_message (str, optional, defaults to “End of training”):: Message to commit while pushing.
blocking (bool, optional, defaults to True):: Whether the function should return only when the git push has finished.
kwargs:: Additional keyword arguments passed along to [~Trainer.create_model_card].

Returns:

The url of the commit of your model in the given repository if blocking=False, a tuple with the url of the commit and an object to track the progress of the commit if blocking=True

remove_callback(callback)¶

Remove a callback from the current list of [~transformer.TrainerCallback].

Args:

callback (type or [~transformer.TrainerCallback]):: A [~transformer.TrainerCallback] class or an instance of a [~transformer.TrainerCallback]. In the first case, will remove the first member of that class found in the list of callbacks.

save_metrics(split, metrics, combined=True)¶

Save metrics into a json file for that split, e.g. train_results.json.

Under distributed environment this is done only for a process with rank 0.

Args:

split (str):: Mode/split name: one of train, eval, test, all
metrics (Dict[str, float]):: The metrics returned from train/evaluate/predict
combined (bool, optional, defaults to True):: Creates combined metrics by updating all_results.json with metrics of this call

To understand the metrics please read the docstring of [~Trainer.log_metrics]. The only difference is that raw unformatted numbers are saved in the current method.

save_model(output_dir: Optional[str] = None, _internal_call: bool = False)¶

Will save the model, so you can reload it using from_pretrained().

Will only save from the main process.

save_state()¶

Saves the Trainer state, since Trainer.save_model saves only the tokenizer with the model

Under distributed environment this is done only for a process with rank 0.

store_flos()¶

torch_jit_model_eval(model, dataloader, training=False)¶

train(resume_from_checkpoint: Optional[Union[bool, str]] = None, trial: Union[optuna.Trial, Dict[str, Any]] = None, ignore_keys_for_eval: Optional[List[str]] = None, **kwargs)¶

Main training entry point.

Args:

resume_from_checkpoint (str or bool, optional):: If a str, local path to a saved checkpoint as saved by a previous instance of [Trainer]. If a bool and equals True, load the last checkpoint in args.output_dir as saved by a previous instance of [Trainer]. If present, training will resume from the model/optimizer/scheduler states loaded here.
trial (optuna.Trial or Dict[str, Any], optional):: The trial run or the hyperparameter dictionary for hyperparameter search.
ignore_keys_for_eval (List[str], optional): A list of keys in the output of your model (if it is a dictionary) that should be ignored when gathering predictions for evaluation during the training.
kwargs:: Additional keyword arguments used to hide deprecated arguments

training_step(model: torch.nn.modules.module.Module, inputs: Dict[str, Union[torch.Tensor, Any]]) → torch.Tensor¶

Perform a training step on a batch of inputs.

Subclass and override to inject custom behavior.

Args:

model (nn.Module):

The model to train.

inputs (Dict[str, Union[torch.Tensor, Any]]):

The inputs and targets of the model.

The dictionary will be unpacked before being fed to the model. Most models expect the targets under the argument labels. Check your model’s documentation for all accepted arguments.

Return:

torch.Tensor: The tensor with training loss on this batch.

mangoes.modeling.training_utils.make_coref_example(tokenizer, document, cluster_ids, speaker_ids, use_metadata, max_segment_len)¶

Function to do co-reference example preprocessing

Parameters

tokenizer: transformers.PreTrainedTokenizerBase: tokenizer to use
use_metadata: Boolean: Whether or not to use speaker ids and genres
max_segment_len: int: maximum number of sub-tokens for one segment
document: Lists of Lists of strings: list of sentences. One sentence is a list of words (ie already split on whitespace/punctuation)
cluster_ids: Lists of Lists of (ints or Tuple(int, int)): Cluster id for each word in document argument. Assumes words that aren’t mentions have either None or -1 as id. In the case where a word belongs to two different spans (with different cluster ids), the cluster id for word should be a tuple of ints corresponding to the different cluster ids. This parameter is optional (can be used for inference)
speaker_ids: Lists of Lists of ints: Speaker id for each word in document. Assumes positive ids (special tokens (such as [CLS] and [SEP] that are added at beginning and end of segments) will be assigned speaker ids of -1)
Returns:
——-
segments_ids: torch tensor: token ids, broken into segments and padded to max_segment_len
segments_attention_mask: torch tensor: attention mask for segment ids
sentence_map: torch tensor: sentence id for each input token in input document
gold_starts: torch tensor: start token indices (in flattened document) of labeled spans. Only returned if cluster information is passed to function.
gold_ends: torch tensor: end token indices (in flattened document) of labeled spans. Only returned if cluster information is passed to function.
cluster_ids: torch tensor: cluster ids of each labeled span. Only returned if cluster information is passed to function.
speaker_ids: torch tensor: speaker ids for each token (only used if self.use_metadata is True)