no translation step

I built a search engine over Instagram food posts from Cyprus. The captions are Greek, English, and very often both inside the same caption — a restaurant writes the dish in Greek, the hashtags in English, and the address in whatever came out first.

Two people are looking for dinner. One types seafood in Limassol. The other types θαλασσινά Λεμεσός. Both should get an equally good answer, and neither should have to think about which language the site was built in.

the part I didn't build

The obvious design has a translation step in it. Detect the language of the query. If it's Greek, translate it to English. Search the English side. Possibly translate something back on the way out.

That's four new things: a language detector that is wrong on short strings, a translation API with a per-request cost and a per-request failure, a decision about which language the index is built in, and a second decision about what happens to captions that are already mixed. None of it is hard. All of it is work, forever.

I did not build any of it, because I did not need to.

The model I run is intfloat/multilingual-e5-base. It was trained across languages, so it puts σουβλάκι and souvlaki near each other in the same vector space — not because anything translated one into the other, but because that is what the space is. A Greek query and an English caption come out as two points that happen to be close together. The search is a dot product. It does not know or care which language either side was written in.

The feature works because of a step that isn't there.

the one fiddly bit

The e5 models want prefixes, and they are asymmetric:

# every document, at index time, in batches of 64
docs = ['passage: ' + build_document(p) for p in batch]
vectors = model.encode(docs, normalize_embeddings=True)

# every query, at search time — one string, still a list
vector = model.encode(['query: ' + q], normalize_embeddings=True)[0]

passage: on everything you store, query: on everything you search with, no exceptions. Get it wrong and nothing breaks. There is no error. The results just get quietly worse in a way you will not notice for a week, because the vectors still mean something — just not the thing you compared them against.

It looks like a wart until you know why it's there. The model was trained with those prefixes marking which side of the pair it was reading. Drop them and you are asking it a question in a slightly different language than the one you filed everything under.

where it doesn't hold

There are two ways this search finds a post. One is the embedding. The other is Ctrl-F — a plain SQL icontains over the caption, hashtags, location, restaurant and dish names. The results are fused, so a post found by both ranks above a post found by only one.

Ctrl-F cannot cross languages. Ever.

Post A (English)   "best souvlaki in Limassol"
Post B (Greek)     "το καλύτερο σουβλάκι στη Λεμεσό"

Search souvlaki and Ctrl-F finds A, because those letters are in it. It does not find B. The embedding finds both. A was found twice, B once, so A ranks higher.

Search σουβλάκι and it mirrors: Ctrl-F finds B, the embedding finds both, B ranks higher.

Whichever language you type, posts in that language get found twice and win. Posts in the other language still show up — carried by the embedding alone, one rank lower.

Two consequences, and they are the honest limits of the whole thing.

Half the engine stops working. My captions and hashtags are overwhelmingly Latin script, so a Greek query has almost nothing to Ctrl-F against. Searching souvlaki, 22 results came from the keyword half. Searching σουβλάκι, one did. Greek runs on one engine instead of two.

You get an equally good answer, not the same one. The top ten for souvlaki and σουβλάκι share three posts. seafood and θαλασσινά share none. Each surfaces good posts written in the language you typed. That is a fine answer to "where do I eat" and a poor one to "show me that specific restaurant".

So: no translation needed to get good answers. Translation would still be needed to get identical ones.

The best part of this feature is still the part I didn't write.