Explore vs Exploit & Optimal Stopping
Multi-armed bandit and sequential secretary problem: how much to explore before committing.
Environment
Higher spread = more hidden gems and more duds.
Your Strategy
Each step, roll the dice: explore a random option or exploit the current best.
| Metric | Value |
|---|---|
| Best explore-first 20% scouting | 10 tried |
| Best epsilon-greedy random rate | 30% |
| % of perfect best strategy vs always-best | 80% |
Tastiness vs exploration effort
Average Tastiness per step across 1000 worlds. ● = your settings.
- Perfect: Perfect-knowledge ceiling: always picking the best option.
Your Restaurants world
Sorted worst → best. The hidden landscape you're trying to discover.
- Explore-first commits to the best of the 5 tried options for the rest of the run.
- Perfect knowledge is the ceiling: always choosing the single best option. All curves averaged over 1000 randomly drawn worlds.
Clamped pool size and total attempts
N = min(200, max(2, ⌊50⌋)) T = min(2000, max(1, ⌊50⌋))
- N: usable pool size (2 to 200)
- T: usable number of attempts (1 to 2000)
Each option's simulated quality
v = clamp(5 + 2 × z, 0, 10)
- v: one option's quality
- z: a standard-normal random draw (Box–Muller)
Explore-first: value per step after trying e options
perStep(e) = ( prefix(e) + (50 − e) × best(e) ) ÷ 50
- e: number of options tried before committing to the best one found
- prefix(e): sum of the first e tried qualities, visited in random order
- best(e): the best quality among those e
- T: total attempts
Epsilon-greedy: value per step
payoff(ε) = (sum over 50 steps of: explore this step ? new draw : bestSoFar) ÷ 50
- ε: chance of exploring (visiting a random option) each step, else exploit the best found so far
- bestSoFar: best quality found among options visited so far
- T: total attempts
Secretary rejection count and the 1/e rule
k = round(threshold ÷ 100 × 50) t* = 1 ÷ e ≈ 36.8%
- k: candidates rejected to set a baseline before accepting
- threshold: rejection threshold, as % of the pool
- N: usable pool size
- t*: the mathematically optimal rejection threshold as N → ∞ (e = Euler's number)
Assumptions behind every figure: how this site models the market →
Every model leaves things out. Here is what this one does not see:
- Once you find your best-so-far option you can always return to it. Real options vanish: the great restaurant closes, the job offer expires, the person you didn't call marries someone else. The model has no cost for that kind of loss, so it favours patient exploring more than real search rewards it.
- Every draw is treated as an independent, unpredictable sample from a fixed distribution. Real search comes with signals (reviews, referrals, a resume, a first message) that let you skip options you can already tell are worse without spending a full "try." This makes pure random exploration look more necessary than it usually is.
- The emotional and time cost of rejection, searching itself, and decision fatigue carry no price. Only the payoff of the option you land on is counted, not what living through the search cost you.
- Quality is collapsed into a single number (tastiness, job satisfaction, compatibility). Real choices trade off several dimensions at once, and a strategy optimal on one axis can be badly wrong once the others are considered.