Grounding an LLM so it can't invent a menu item

MealPilot lets you describe what you feel like eating and suggests real dishes from the restaurant's live menu. The obvious risk with any LLM feature here is not latency or cost, it is confidence: a model that invents a dish that isn't on the menu is worse than no assistant at all, because it breaks trust at the exact moment the user is about to act.

The browser sends the message and the current menu to a Next.js server route, which asks Claude to suggest dishes and return their IDs in a structured JSON shape. The route then validates every ID against the real menu before anything reaches the screen. A dish the model invented simply cannot enter the cart, because the ID does not exist.

The interesting part is where the trust boundary sits: I do not trust the model's output, I trust the menu. The model proposes; the data disposes.

For a single restaurant's menu, the whole menu goes straight into the prompt rather than into a vector store. The context window is large enough, the menu is small, and retrieval would add latency and complexity to solve a problem this app does not have. Reaching for embeddings here would have been the right answer to the wrong question.

I wrote a small evaluation script: about a dozen labelled requests, each with a verifiable rule (a price ceiling, veg-only, a cuisine), that checks whether the suggestions satisfy the stated constraint and prints an accuracy score. It gives me a repeatable way to test a prompt change instead of eyeballing a few examples and hoping.

Stream the reply token by token so it feels instant, and grow the eval set as real phrasings surface. And when a menu eventually outgrows the prompt, that is the moment retrieval earns its place, not before.