Hey Simon,

Elliott here from Cohere.

We benchmarked against Nomic's models on our consortium of datasets ranging from text-only, image-only, and mixed modalities. Without publishing additional benchmarks, I am confident in saying that our model is more performant.

At Cohere, for our embed models, we have not deprecated any of our embedding models since we started (I know because I've been there that long) and if we were to start doing so, I would take into account the worry of ensuring our users have a way of accessing our models.

One aspect here that isn't factored is also efficiency. Yes there might be strong open weight models but if you're punching at the 7bn+ weight class your serving requirements are vastly different from a throughput efficiency perspective (also your query-inference speed).

All food for thought. That being said, if for your use-case, Nomic Embed Vision 1.5 is better than Embed-v4.0, happy to hop on a call to discuss where the differential may be.

In my experience, a non-finetunable closed source API is a complete nonstarter for a large fraction of possible use cases, especially the higher value ones like law and engineering. Most of these embedding models are trained too much on colloquial use of language on the internet that has little overlap with how terms of art are used, and without the ability to fine tune it to a specific use case, they're only really useful for generic use cases and even then they can be limited.

Hey Luke, Our model does exceptionally well on text and images, and in particular, when text and images are mixed together. An example of where this works well would be in E-commerce where you may have a product title, description, and multiple images of the product. When combining that into a single payload using our inputs parameter we find that our model responds really well to adding more images (i.e. retrieval quality moves up as you add 1,2,3....N images). As you pointed out with Google's multimodal model, most jointly trained multimodal embedding models will suffer in the text modality. Amazon used to have a multimodal embedding model, which also took in a very small text payload. We're thinking about Audio / Video as well but nothing for Q2 at least....

cool project - I like the read-me but it looks like your link is down: https://djwtmt1np1xe4.cloudfront.net/

enterprise GTM has its own set of challenges and needs and warrants someone really focused on it

I had the same thought, although voyage is 32k vs 128k for cohere 4.

Anecdotal evidence points to benchmarks correlating with result quality for data I've dealt with. I haven't spent a lot of time comparing results between models, because we were happy with the results after trying a few and tuning some settings.

Unless my dataset lines up really well with a benchmark's dataset, creating my own benchmark is probably the only way to know which model is "best".

Which benchmark are you referring to?

Voyage-3-large is a text-only and much larger model than Embed-v4. If you want to unlock multimodality with Voyage-3-large, you'd have to either OCR (really bad results usually) or use a VLM to parse your data into textual descriptions (this works alright, but the cost of using a VLM will jack-up your data-pre-processing costs).

Why? How do you pick an embedding model without benchmarks?

Thank you sir! I appreciate you.

Hey Cahaya,

While we benchmarked internally, on BEIR, we opted not to report our model onto MTEB for the following reason:

1) MTEB has been gamed - if you look at this model (https://huggingface.co/voyageai/voyage-3-m-exp) on the MTEB leaderboard, its an intermediate checkpoint of Voyage-3-Large where they finetuned it on datasets that represent MTEB datasets.

2) If you look at the recent datasets in MMTEB, you'll find that it has quite a lot of machine translated or "weird" datasets that are quite noisy

In general, for our Search Models, we benchmark on these public academic datasets but we definitely do not try to hillclimb in this direction as we find it has little correlation with real use-cases

That sounds really cool! Would love to better understand your use-case and make sure it works well for you!

Drop me an email at elliott@cohere.ai

they develop models around a very defined set of used cases, and they are very good at it. Look through their documentation and throughout their API. It’s very opinionated and quite a delight, honestly.