Token Pooling for Efficient Storage
Multi-vector embedding models utilize numerous contextual token vectors per document, allowing various parts of a query to match different sections of a document. However, this approach significantly increases storage costs. Token pooling addresses this issue by grouping similar token embeddings, thereby reducing the overall number of vectors needed.
How Token Pooling Works
HierarchicalTokenPooling employs Ward hierarchical clustering to group similar token vectors, representing each group by its mean. Key pooling controls include:
- max_tokens: Sets a maximum number of vectors per document.
- pool_factor: For instance, a pool_factor of 4 retains approximately one quarter of the original vectors.
- distance_threshold: This is calibrated based on sample embeddings to target an average vector count, allowing for flexibility in vector retention based on document content.
By default, the first token is protected from merging, and pooling is applied only to document embeddings. MaxSim scoring is utilized to compare query vectors with pooled document vectors, ensuring effective retrieval.
Pooling-Aware Training Implementation
Pooling-aware training introduces an explicit retrieval objective for pooled document representations alongside the standard unpooled objective. In the reported example, each training batch contributes two components: retrieval using the original document embeddings and retrieval after compressing the documents to about one quarter of their token count, with both components averaged equally.
During backpropagation, HierarchicalTokenPooling employs a straight-through estimator to pass gradients from pooled vectors back to their constituent tokens. This allows the encoder to learn how to arrange token embeddings that remain effective after clustering.
Training and Evaluation Results
The approach is implemented using components from Sentence Transformers, including MultiVectorEncoder, HierarchicalTokenPooling, and a loss wrapper called MultiVectorTokenPoolingLoss. This wrapper combines a base retrieval loss (CachedMultiVectorMultipleNegativesRankingLoss) with pooling variants. Additionally, kernel acceleration (use_kernels=True) can be enabled to significantly speed up Ward clustering and MaxSim scoring, achieving performance improvements of approximately 30 to 150 times compared to SciPy, depending on the settings.
Evaluation involved training two models based on answerdotai/ModernBERT-base, using the same dataset of 120,000 MS MARCO triplets over one epoch with a seed of 42 and a token cap of 300. The resulting models are available as tomaarsen/mve-ModernBERT-base-token-pooling-baseline and tomaarsen/mve-ModernBERT-base-token-pooling. For a pool_factor of 4, the baseline model retained 94.6% of the unpooled NanoBEIR nDCG@10, while pooling-aware training improved retention to 97.8% within the same storage budget.
Conclusion
The repository provides a training example located at examples/multi_vector_encoder/training/token_pooling/training_cached_contrastive.py for those interested in reproducing the results.
Original source: Hugging Face Blog