Thirteen free models, one answer

Mission Control is the assistant on this site: ask it about my work and it answers from the site's own content. I wanted it to cost nothing. Free AI tiers are generous in total and stingy per model, so "free" turned into a reliability problem: how do you get an answer every time out of models that are each allowed only so much?

What the free tiers actually give you

• Google's Gemini API: the Flash models allow 20 requests a day each; Flash Lite, 500; the open Gemma models, 14,400, but only 16,000 tokens a minute
• Groq: 1,000 a day per model, and under a second per answer, but 8,000 tokens a minute
• NVIDIA's free endpoints: about 40 requests a minute, no published daily cap, and very uneven speed from model to model

Each question carries the site's content with it, a few thousand tokens, so the per-minute token caps bite long before the request caps do. No single model could carry the site. Thirteen of them, across three providers, can.

What probing taught me

Before writing the chain I called every model with the same small question. The results shaped the design more than the documentation did:

• Models answered "experiencing high demand" (a 503) for minutes at a time, and a failed call still counted against the day's quota
• One call hung for 198 seconds before failing
• Each model accepts only certain "thinking" levels, and the wrong one is an error, not a fallback
• Two models in the list were already gone for new accounts

The chain

The routes are tried in order: the fastest and most dependable first (Gemini Flash Lite, then Groq, then NVIDIA), then the better but scarce Flash models, then Gemma as the bulk reserve. If every one fails, a plain keyword search over the same content answers instead, so Mission Control never simply breaks.

for each route, in order:
  skip it if it's resting, or at today's cap
  start it, with its own timeout (8 to 20 s)
  if it's still silent after 3 s and the next route
    has a big quota, start that one alongside
  first answer wins; the other is cancelled
  on failure, rest the route and move on

The hedge is the part that keeps it feeling fast. A route that's slow today doesn't make the visitor wait for its timeout; after three seconds the next one races it. It's never hedged onto the 20-a-day Flash models, though, so they don't spend their day on answers that get thrown away. When I made Gemini hang in a test, the answer arrived from Groq in 3.6 seconds.

Resting, and counting

A failure rests its route for as long as the failure suggests, recorded in Redis so every instance of the function knows:

• A daily limit: until the quota resets at midnight Pacific time
• A per-minute limit: its retry-after, between 20 seconds and 5 minutes
• A timeout or a 5xx: two minutes
• Any other 4xx, a model or key gone wrong: six hours

Each route's attempts are also counted per day, so the chain stops just short of a free cap instead of finding out from a 429. The live state of every route is on the site's status page.

Spending less of it

• A first question's answer is cached for a week, keyed by the question's normalised text and a hash of the site's content, so editing the content retires every cached answer at once
• Each visitor gets 15 questions an hour and 40 a day; the whole site gets 3,000 a day, past which it answers by search alone
• Only the last two exchanges go with a follow-up, not the whole conversation

Keeping it honest

The model is told to answer only from the site's content, to say plainly when something isn't there and point to the contact section, and to treat the visitor's message as a question, never as instructions. Asked to ignore its rules and write a poem, it declines. The model that answered is logged on the server, not shown to visitors: it helped while building, and nobody reading an answer needs it.