Triplet loss

Triplet loss is a machine learning loss function widely used in one-shot learning, a setting where models are trained to generalize effectively from limited examples. It was conceived by Google researchers for their prominent FaceNet algorithm for face detection. ^[1]

The triplet loss function minimizes the distance between an anchor and a positive, both of which have the same identity, and maximizes the distance between the anchor and a negative of a different identity.

Triplet loss is designed to support metric learning. Namely, to assist training models to learn an embedding (mapping to a feature space) where similar data points are closer together and dissimilar ones are farther apart, enabling robust discrimination across varied conditions. In the context of face detection, data points correspond to images.

Definition

The loss function is defined using triplets of training points of the form ${\displaystyle (A,P,N)}$. In each triplet, ${\displaystyle A}$ (called an "anchor point") denotes a reference point of a particular identity, ${\displaystyle P}$ (called a "positive point") denotes another point of the same identity in point ${\displaystyle A}$, and ${\displaystyle N}$ (called a "negative point") denotes an point of an identity different from the identity in point ${\displaystyle A}$ and ${\displaystyle P}$.

Let ${\displaystyle x}$ be some point and let ${\displaystyle f(x)}$ be the embedding of ${\displaystyle x}$ in the finite-dimensional Euclidean space. It shall be assumed that the L2-norm of ${\displaystyle f(x)}$ is unity (the L2 norm of a vector ${\displaystyle X}$ in a finite dimensional Euclidean space is denoted by ${\displaystyle \Vert X\Vert }$.) We assemble ${\displaystyle m}$ triplets of points from the training dataset. The goal of training here is to ensure that, after learning, the following condition (called the "triplet constraint") is satisfied by all triplets ${\displaystyle (A^{(i)},P^{(i)},N^{(i)})}$ in the training data set:

${\displaystyle \Vert f(A^{(i)})-f(P^{(i)})\Vert _{2}^{2}+\alpha <\Vert f(A^{(i)})-f(N^{(i)})\Vert _{2}^{2}}$

The variable ${\displaystyle \alpha }$ is a hyperparameter called the margin, and its value must be set manually. In the FaceNet system, its value was set as 0.2.

Thus, the full form of the function to be minimized is the following:

${\displaystyle L=\sum _{i=1}^{m}\max {\Big (}\Vert f(A^{(i)})-f(P^{(i)})\Vert _{2}^{2}-\Vert f(A^{(i)})-f(N^{(i)})\Vert _{2}^{2}+\alpha ,0{\Big )}}$

Intuition

A baseline for understanding the effectiveness of triplet loss is the contrastive loss ^[2] , which operates on pairs of samples (rather than triplets). Training with the contrastive loss pulls embeddings of similar pairs closer together, and pushes dissimilar pairs apart. Its pairwise approach is greedy, as it considers each pair in isolation.

Triplet loss innovates by considering relative distances. Its goal is that the embedding of an anchor (query) point be closer to positive points than to negative points (also accounting for the margin). It does not try to further optimize the distances once this requirement is met. This is approximated by simultaneously considering two pairs (anchor-positive and anchor-negative), rather than each pair in isolation.

Triplet "mining"

Triplet mining - various options for selecting the negative ${\displaystyle N^{(i)}}$ given an anchor ${\displaystyle A^{(i)}}$ and a positive ${\displaystyle P^{(i)}}$. The very-hard and semi-hard negatives violate the triplet requirement, and so we would like the optimizer to push their embeddings farther away from the anchor.

One crucial implementation detail when training with triplet loss is triplet "mining", which focuses on the smart selection of triplets for optimization. This process adds an additional layer of complexity compared to contrastive loss.

A naive approach to preparing training data for the triplet loss involves randomly selecting triplets from the dataset. In general, the set of valid triplets of the form ${\displaystyle (A^{(i)},P^{(i)},N^{(i)})}$ is very large. To speed-up training convergence, it is essential to focus on challenging triplets. In the FaceNet paper, several options were explored, eventually arriving at the following. For each anchor-positive pair, the algorithm considers only semi-hard negatives. These are negatives that violate the triplet requirement (i.e, are "hard"), but lie farther from the anchor than the positive (not too hard). Restated, for each ${\displaystyle A^{(i)}}$ and ${\displaystyle P^{(i)}}$, they seek ${\displaystyle N^{(i)}}$ such that:

${\displaystyle \Vert f(A^{(i)})-f(P^{(i)})\Vert _{2}^{2}<\Vert f(A^{(i)})-f(N^{(i)})\Vert _{2}^{2}<\Vert f(A^{(i)})-f(P^{(i)})\Vert _{2}^{2}+\alpha }$

The rationale for this design choice is heuristic. It may appear puzzling that the mining process neglects "very hard" negatives (i.e., closer to the anchor than the positive). Experiments conducted by the FaceNet designers found that this often leads to a convergence to degenerate local minima.

Triplet mining is performed at each training step, from within the sample points contained in the training batch (this is known as online mining), after embeddings were computed for all points in the batch. While ideally the entire dataset could be used, this is impractical in general. To support a large search space for triplets, the FaceNet authors used very large batches (1800 samples). Batches are constructed by selecting a large number of same-category sample points (40), and randomly selected negatives for them.

Extensions

Triplet loss has been extended to simultaneously maintain a series of distance orders by optimizing a continuous relevance degree with a chain (i.e., ladder) of distance inequalities. This leads to the Ladder Loss, which has been demonstrated to offer performance enhancements of visual-semantic embedding in learning to rank tasks. ^[3]

In Natural Language Processing, triplet loss is one of the loss functions considered for BERT fine-tuning in the SBERT architecture. ^[4]

Other extensions involve specifying multiple negatives (multiple negatives ranking loss).

See also

• Siamese neural network
• t-distributed stochastic neighbor embedding
• Learning to rank
• Similarity learning

References

1. Schroff, Florian; Kalenichenko, Dmitry; Philbin, James (2015). "FaceNet: A Unified Embedding for Face Recognition and Clustering": 815–823.{{cite journal}}: Cite journal requires |journal= (help)
2. Bekuzarov, Maksym (2020-04-19). "Losses explained: Contrastive Loss". Medium. Archived from the original on 2024-06-05. Retrieved 2025-01-06.
3. Zhou, Mo; Niu, Zhenxing; Wang, Le; Gao, Zhanning; Zhang, Qilin; Hua, Gang (2020-04-03). "Ladder Loss for Coherent Visual-Semantic Embedding" (PDF). Proceedings of the AAAI Conference on Artificial Intelligence. 34 (7): 13050–13057. doi: 10.1609/aaai.v34i07.7006 . ISSN   2374-3468. S2CID   208139521.
4. Reimers, Nils; Gurevych, Iryna (2019-08-27). "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks". arXiv: 1908.10084 [cs.CL].