Multimodal RAG on EmbeddingGemma 2: The 6x Storage Cut Is a Text-Only Number

Google released EmbeddingGemma 2 on 6 October: an open embedding model that puts text, code, images, video and audio into one 768-dimensional vector space, under Apache 2.0, at 740 million parameters. The announcement pitches it for on-device search and multimodal RAG, and it leans on Matryoshka truncation for up to a 6x cut in vector storage. The model card on Hugging Face carries the table that prices that cut, and the price depends on what you are indexing.

What shipped

The weights are at google/embeddinggemma-2, ungated. The Hub's safetensors metadata counts 744,371,512 parameters in BF16, consistent with the 740M on the card. That total splits into a 270M text model (a 130M transformer plus a 140M embedding table), a 170M vision encoder and a 300M audio encoder. The card shows how to skip the encoders you do not need: text-only loads at 270M, text plus images at 440M.

The context window is 8,192 tokens, shared across modalities. An image costs 280 tokens by default, a video frame 140, a second of audio 25. Native output is 768 dimensions, truncatable to 512, 256 or 128.

On text, the gain over the first EmbeddingGemma is small: 61.36 against 61.15 on multilingual MTEB v2. The headline gain is code, where MTEB code v1 moves from 68.76 to 78.68.

What the truncation table says about multimodal RAG

The blog states the 6x figure without a quality cost. The card prints one, per modality, and it is not uniform:

• 256 dimensions (3x smaller): multilingual MTEB loses 0.95 points, MMEB v2 overall loses 2.77, the MSEB audio retrieval score loses 2.78.
• 128 dimensions (6x smaller): MMEB v2 overall falls from 59.01 to 45.65, a 13.36-point drop. MSEB retrieval falls from 69.54 to 56.71. English MTEB loses 2.78.

The card is candid about this. It says 128 dimensions is best suited to text-only workloads and degrades multimodal quality substantially. So the 6x saving and the multimodal pitch belong to different deployments. For an index that mixes images, video or audio, the card's own numbers put the floor at 256 dimensions and the ceiling on savings at 3x.

Code is the less obvious casualty. At 128 dimensions the MTEB code score is 71.41. That keeps 2.65 of the 9.92 points the release gained over its predecessor, and the predecessor's figure is at full 768 dimensions. A team adopting EmbeddingGemma 2 for coding-agent retrieval and then truncating to 128 to save storage has bought back most of the reason to switch.

This is the inverse of the trade we measured on multi-vector embeddings, where a larger index bought about one NDCG point. Here a 3x smaller index costs about one point on multilingual text, and a 6x smaller one costs more than thirteen on mixed media.

The falsifiable version: on a retrieval set that mixes text queries against image or video documents, 128-dimension EmbeddingGemma 2 vectors will score at least ten points below 768-dimension ones, in line with the MMEB gap. If your own mixed-modality evaluation at 128 dimensions comes in within two points of 768, the card's table does not describe your data and the 6x saving is yours to take.

What breaks on the way in

It does not load on last week's transformers. The checkpoint declares model type embedding_gemma2, and its config was written by 5.18.0.dev0. The released transformers 5.18.0 has no embedding_gemma2 model directory; 5.19.0 does, and PyPI shows it uploaded at 16:38 UTC on 6 October, the day of the launch. The ggml-org GGUF conversion log shows the same gap from the outside: the converter's transformers did not recognise the architecture and fell back to reading config.json directly. The 5.19.0 notes also flag breaking changes elsewhere, including router logits now returned by every MoE model that computes them and a deprecated paged| attention prefix. If your embedding service shares a pinned transformers with a generation model, the upgrade is not free. The sentence-transformers config separately requires 6.1.0 or later, because older versions ignore the order of multimodal inputs and cannot combine several images or texts into one embedding.

Plain encode() skips the prefix. The model was trained with task prefixes such as task: search result | query: for queries and title: none | text: for documents. Its sentence-transformers config sets default_prompt_name to null, and in the 6.1.0 source a null default means encode() with no prompt_name prepends nothing. encode_query() and encode_document() do pick up the query and document prompts the config defines. The card says that leaving the prefix off still works but costs precision, and nothing raises an error. The ggml-org GGUF README gives a single serve command and says nothing about prefixes.

float16 fails quietly. The card warns that activations exceed float16's range and that the model then returns NaN or degraded embeddings rather than an error. Use bfloat16 or float32.

The license has a second document. The card links Apache 2.0 to a Google page carrying the standard Apache License text, and the repository holds no LICENSE file of its own. The same card says deployments must adhere to the Gemma Prohibited Use Policy. Have whoever approves licenses read both.

All three runtime failures are silent: a missing prefix, float16, and truncation without re-normalising produce plausible scores, not exceptions. That is the default failure mode we describe in the gap between a proof of concept and production, and an embedding swap is a good place for it to hide.

The verdict

Adopt it for a text or code index at 768 or 512 dimensions, or a multimodal index at 256, once you can move to transformers 5.19.0. Do not ship 128 dimensions on anything but plain prose without your own evaluation. For a team that wants a multimodal RAG index designed and evaluated against its own documents, that work is what our generative AI consulting covers.

Before you re-embed a corpus:

1. pip install 'transformers>=5.19.0' 'sentence-transformers>=6.1.0', then run your existing generation tests against the new transformers.
2. Load with model_kwargs={'torch_dtype': torch.bfloat16}, or float32 on hardware without bf16, never float16. Assert that no embedding contains NaN.
3. Encode with model.encode_query(q, truncate_dim=256, normalize_embeddings=True) and model.encode_document(d, truncate_dim=256, normalize_embeddings=True), formatting titled documents as title: <title> | text: <body> yourself.
4. Score 768 against 256 on 200 of your own query and document pairs before choosing a dimension. Drop to 128 only if recall@10 stays within two points.