I built a vector store over a synthetic product catalog and then went looking at the embedding space it produced. Mainly to find where semantic search holds up and where it quietly does not.
40 markdown product pages across 5 categories, 37,306 characters total. Embedded
with OpenAI text-embedding-3-large at 3072 dimensions, stored in Chroma,
flattened with t-SNE for the plots.
It is a made-up specialty coffee retailer. Every product, price, spec and
description under catalog/ is synthetic. catalog_data.py is the generator:
run python catalog_data.py and the whole catalog gets rewritten from source,
which is the proof that none of it came from a real company.
Query: "the invisible reason a shot tastes sour and bitter". That sentence sits almost verbatim on the Comb Distribution Tool page, so a retriever that is working should pull that page up with room to spare.
| Chunking | Comb rank | Top score | 5th score | Spread |
|---|---|---|---|---|
| 1200 char, 200 overlap | 4th | 0.250 | 0.238 | 0.012 |
| 250 char, 50 overlap | 2nd | 0.464 | 0.334 | 0.130 |
The finding is not "smaller chunks fixed it". At 1200 characters the top five scores sat inside a 0.012 band, so the ranking carried almost no confidence spread. Noise that happens to arrive sorted. Shrinking the chunks opened the spread to 0.130, which is real signal, but the exact-match page still only reached 2nd. It lost to a bag of coffee beans whose page says it is "most likely to taste sour if you brew it carelessly". Vocabulary overlap beat the near-verbatim match.
Separate thing worth keeping: four products were written to belong to two categories at once, and only one of them, the Compass Travel Press, actually landed between clusters on the map. An embedding follows whichever vocabulary dominates a page, not the categories the product belongs to.
Needs an OpenAI key in .env as OPENAI_API_KEY.
python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
jupyter labThen open embedding_space_explorer.ipynb and run the cells top to bottom.