", num_labels=num_labels
)
# your training arguments here.
training_args = TrainingArguments()
trainer = Trainer(model=model, args=training_args)
# make sure use the checkpoint dir to resume training from the checkpoint
trainer.train(resume_from_checkpoint=checkpoint_dir)
```
### Log and view evaluation samples during training
The [`WandbCallback`](https://huggingface.co/transformers/main_classes/callback.html#transformers.integrations.WandbCallback) in the Transformers library handles logging to W\&B through the Transformers `Trainer`. You can customize this callback to log model predictions, confusion matrices, or other custom data. To do so, subclass `WandbCallback` and add functionality that uses additional methods from the Trainer class.
The following is the general pattern to add this new callback to the Hugging Face Trainer, followed by a code-complete example to log evaluation outputs to a W\&B Table:
```python theme={null}
# Instantiate the Trainer as normal
trainer = Trainer()
# Instantiate the new logging callback, passing it the Trainer object
evals_callback = WandbEvalsCallback(trainer, tokenizer, ...)
# Add the callback to the Trainer
trainer.add_callback(evals_callback)
# Begin Trainer training as normal
trainer.train()
```
#### View evaluation samples during training
The following section shows how to customize the `WandbCallback` to run model predictions and log evaluation samples to a W\&B Table during training. This runs every `eval_steps` using the `on_evaluate` method of the Trainer callback.
The `decode_predictions` function decodes the predictions and labels from the model output using the tokenizer.
Then, the code creates a pandas DataFrame from the predictions and labels and adds an `epoch` column to the DataFrame.
Finally, the code creates a `wandb.Table` from the DataFrame and logs it to W\&B. You can control the frequency of logging by logging the predictions every `freq` epochs.
Unlike the regular `WandbCallback`, this custom callback needs to be added to the trainer **after** the `Trainer` is instantiated, not during initialization of the `Trainer`. This is because the `Trainer` instance is passed to the callback during initialization.
```python theme={null}
from transformers.integrations import WandbCallback
import pandas as pd
def decode_predictions(tokenizer, predictions):
labels = tokenizer.batch_decode(predictions.label_ids)
logits = predictions.predictions.argmax(axis=-1)
prediction_text = tokenizer.batch_decode(logits)
return {"labels": labels, "predictions": prediction_text}
class WandbPredictionProgressCallback(WandbCallback):
"""Custom WandbCallback to log model predictions during training.
This callback logs model predictions and labels to a wandb.Table at each
logging step during training. It allows to visualize the
model predictions as the training progresses.
Attributes:
trainer (Trainer): The Hugging Face Trainer instance.
tokenizer (AutoTokenizer): The tokenizer associated with the model.
sample_dataset (Dataset): A subset of the validation dataset
for generating predictions.
num_samples (int, optional): Number of samples to select from
the validation dataset for generating predictions. Defaults to 100.
freq (int, optional): Frequency of logging. Defaults to 2.
"""
def __init__(self, trainer, tokenizer, val_dataset, num_samples=100, freq=2):
"""Initializes the WandbPredictionProgressCallback instance.
Args:
trainer (Trainer): The Hugging Face Trainer instance.
tokenizer (AutoTokenizer): The tokenizer associated
with the model.
val_dataset (Dataset): The validation dataset.
num_samples (int, optional): Number of samples to select from
the validation dataset for generating predictions.
Defaults to 100.
freq (int, optional): Frequency of logging. Defaults to 2.
"""
super().__init__()
self.trainer = trainer
self.tokenizer = tokenizer
self.sample_dataset = val_dataset.select(range(num_samples))
self.freq = freq
def on_evaluate(self, args, state, control, **kwargs):
super().on_evaluate(args, state, control, **kwargs)
# control the frequency of logging by logging the predictions
# every `freq` epochs
if state.epoch % self.freq == 0:
# generate predictions
predictions = self.trainer.predict(self.sample_dataset)
# decode predictions and labels
predictions = decode_predictions(self.tokenizer, predictions)
# add predictions to a wandb.Table
predictions_df = pd.DataFrame(predictions)
predictions_df["epoch"] = state.epoch
records_table = self._wandb.Table(dataframe=predictions_df)
# log the table to wandb
self._wandb.log({"sample_predictions": records_table})
# First, instantiate the Trainer
trainer = Trainer(
model=model,
args=training_args,
train_dataset=lm_datasets["train"],
eval_dataset=lm_datasets["validation"],
)
# Instantiate the WandbPredictionProgressCallback
progress_callback = WandbPredictionProgressCallback(
trainer=trainer,
tokenizer=tokenizer,
val_dataset=lm_dataset["validation"],
num_samples=10,
freq=2,
)
# Add the callback to the trainer
trainer.add_callback(progress_callback)
```
For a more detailed example, see this [Colab](https://colab.research.google.com/github/wandb/examples/blob/master/colabs/huggingface/Custom_Progress_Callback.ipynb).
### Additional W\&B settings
You can further configure what is logged with `Trainer` by setting environment variables. For a full list of W\&B environment variables, see the [environment variables reference](/platform/hosting/env-vars).
| Environment Variable | Usage |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `WANDB_PROJECT` | Give your project a name (`huggingface` by default) |
| `WANDB_LOG_MODEL` | Log the model checkpoint as a W\&B Artifact (`false` by default)
false (default): No model checkpointing checkpoint: Upload a checkpoint every `args.save_steps` (set in the Trainer's `TrainingArguments`). end: Upload the final model checkpoint at the end of training.
|
| `WANDB_WATCH` | Set whether to log your model's gradients, parameters, or neither.
false (default): No gradient or parameter logging gradients: Log histograms of the gradients all: Log histograms of gradients and parameters
|
| `WANDB_DISABLED` | Set to `true` to turn off logging entirely (`false` by default) |
| `WANDB_QUIET` | Set to `true` to limit statements logged to standard output to critical statements only (`false` by default) |
| `WANDB_SILENT` | Set to `true` to silence the output printed by `wandb` (`false` by default) |
```bash theme={null}
WANDB_WATCH=all
WANDB_SILENT=true
```
```notebook theme={null}
%env WANDB_WATCH=all
%env WANDB_SILENT=true
```
### Customize `wandb.init()`
The `WandbCallback` that `Trainer` uses calls `wandb.init()` under the hood when `Trainer` is initialized. Alternatively, you can set up your runs manually by calling `wandb.init()` before the `Trainer` is initialized. This gives you full control over your W\&B run configuration.
The following is an example of what you might pass to `init`. For `wandb.init()` details, see the [`wandb.init()` reference](/models/ref/python/functions/init).
```python theme={null}
wandb.init(
project="amazon_sentiment_analysis",
name="bert-base-high-lr",
tags=["baseline", "high-lr"],
group="bert",
)
```
## Additional resources
The following are six Transformers and W\&B related articles for further reading.
Hyperparameter Optimization for Hugging Face Transformers
* Compares three strategies for hyperparameter optimization for Hugging Face Transformers: Grid Search, Bayesian Optimization, and Population Based Training.
* Uses a standard uncased BERT model from Hugging Face transformers, fine-tuned on the RTE dataset from the SuperGLUE benchmark.
* Results show that Population Based Training is the most effective approach to hyperparameter optimization of the Hugging Face transformer model.
Read the [Hyperparameter Optimization for Hugging Face Transformers report](https://wandb.ai/amogkam/transformers/reports/Hyperparameter-Optimization-for-Hugging-Face-Transformers--VmlldzoyMTc2ODI).
Hugging Tweets: Train a Model to Generate Tweets
* In the article, the author demonstrates how to fine-tune a pre-trained GPT2 Hugging Face Transformer model on anyone's Tweets in five minutes.
* The model uses the following pipeline: downloading Tweets, optimizing the dataset, initial experiments, comparing losses between users, and fine-tuning the model.
Read the [HuggingTweets report](https://wandb.ai/wandb/huggingtweets/reports/HuggingTweets-Train-a-Model-to-Generate-Tweets--VmlldzoxMTY5MjI).
Sentence Classification With Hugging Face BERT and WB
* This article builds a sentence classifier using the power of recent breakthroughs in Natural Language Processing, focusing on an application of transfer learning to NLP.
* Uses The Corpus of Linguistic Acceptability (CoLA) dataset for single sentence classification, a set of sentences labeled as grammatically correct or incorrect that was first published in May 2018.
* Uses Google's BERT to create high-performance models with minimal effort on a range of NLP tasks.
Read the [Sentence Classification With Hugging Face BERT and W\&B report](https://wandb.ai/cayush/bert-finetuning/reports/Sentence-Classification-With-Huggingface-BERT-and-W-B--Vmlldzo4MDMwNA).
A Step by Step Guide to Tracking Hugging Face Model Performance
* Uses W\&B and Hugging Face transformers to train DistilBERT, a Transformer that's 40% smaller than BERT but retains 97% of BERT's accuracy, on the GLUE benchmark.
* The GLUE benchmark is a collection of nine datasets and tasks for training NLP models.
Read the [Tracking Hugging Face Model Performance report](https://wandb.ai/jxmorris12/huggingface-demo/reports/A-Step-by-Step-Guide-to-Tracking-HuggingFace-Model-Performance--VmlldzoxMDE2MTU).
Examples of Early Stopping in HuggingFace
* You can fine-tune a Hugging Face Transformer using Early Stopping regularization natively in PyTorch or TensorFlow.
* The `tf.keras.callbacks.EarlyStopping` callback makes using EarlyStopping in TensorFlow straightforward.
* PyTorch doesn't provide an off-the-shelf early stopping method, but a working early stopping hook is available on GitHub Gist.
Read the [Early Stopping in Hugging Face report](https://wandb.ai/ayush-thakur/huggingface/reports/Early-Stopping-in-HuggingFace-Examples--Vmlldzo0MzE2MTM).
How to Fine-Tune Hugging Face Transformers on a Custom Dataset
Fine-tunes a DistilBERT transformer for sentiment analysis (binary classification) on a custom IMDB dataset.
Read the [Fine-Tune Hugging Face Transformers on a Custom Dataset report](https://wandb.ai/ayush-thakur/huggingface/reports/How-to-Fine-Tune-HuggingFace-Transformers-on-a-Custom-Dataset--Vmlldzo0MzQ2MDc).
## Get help or request features
For any issues, questions, or feature requests for the Hugging Face W\&B integration, post in [this thread on the Hugging Face forums](https://discuss.huggingface.co/t/logging-experiment-tracking-with-w-b/498) or open an issue on the Hugging Face [Transformers GitHub repo](https://github.com/huggingface/transformers).
### Finish your W\&B run (notebook only)
If your training is encapsulated in a Python script, the W\&B run ends when your script finishes.
If you're using a Jupyter or Google Colab notebook, call `run.finish()` to signal that training is complete.
```python theme={null}
run = wandb.init()
trainer.train() # start training and logging to W&B
# post-training analysis, testing, other logged code
run.finish()
```
### Visualize your results
After you log your training results, you can explore them in the [W\&B Dashboard](/models/track/workspaces/). You can compare runs, zoom in on findings, and explore your data with interactive visualizations.
At this point you have a working integration: your `Trainer` logs metrics to a named project, optionally saves checkpoints to Artifacts, and surfaces evaluation outputs in the W\&B Dashboard.
## Advanced features and FAQs
The following sections cover common follow-up tasks, such as saving the best model, resuming training from a checkpoint, customizing logging callbacks, and configuring W\&B behavior through environment variables.
### Save the best model
If you pass `TrainingArguments` with `load_best_model_at_end=True` to your `Trainer`, W\&B saves the best performing model checkpoint to Artifacts.
If you save your model checkpoints as Artifacts, you can promote them to the [Registry](/models/registry/). In Registry, you can:
* Organize your best model versions by ML task.
* Centralize models and share them with your team.
* Stage models for production or bookmark them for further evaluation.
* Trigger downstream CI/CD processes.
### Load a saved model
If you saved your model to W\&B Artifacts with `WANDB_LOG_MODEL`, you can download your model weights for more training or to run inference. Load them back into the same Hugging Face architecture that you used before.
```python theme={null}
# Create a new run
with wandb.init(project="amazon_sentiment_analysis") as run:
# Pass the name and version of Artifact
my_model_name = "model-bert-base-high-lr:latest"
my_model_artifact = run.use_artifact(my_model_name)
# Download model weights to a folder and return the path
model_dir = my_model_artifact.download()
# Load your Hugging Face model from that folder
# using the same model class
model = AutoModelForSequenceClassification.from_pretrained(
model_dir, num_labels=num_labels
)
# Do additional training, or run inference
```
### Resume training from a checkpoint
If you set `WANDB_LOG_MODEL='checkpoint'`, you can resume training by using the `model_dir` as the `model_name_or_path` argument in your `TrainingArguments` and passing `resume_from_checkpoint=True` to `Trainer`.
```python theme={null}
last_run_id = "xxxxxxxx" # fetch the run_id from your wandb workspace
# resume the wandb run from the run_id
with wandb.init(
project=os.environ["WANDB_PROJECT"],
id=last_run_id,
resume="must",
) as run:
# Connect an Artifact to the run
my_checkpoint_name = f"checkpoint-{last_run_id}:latest"
my_checkpoint_artifact = run.use_artifact(my_checkpoint_name)
# Download checkpoint to a folder and return the path
checkpoint_dir = my_checkpoint_artifact.download()
# reinitialize your model and trainer
model = AutoModelForSequenceClassification.from_pretrained(
"