x129 · Sundeep

LLM benchmarks, and writing one

Sat, 04 Jul 2026 21:00:00 GMT

I’ve been curious about what it takes to build a benchmark for language models: a fixed set of tasks with known answers that models get scored on. So I’m writing one.

I picked a domain I enjoy: storytelling tropes. The task is for a model to read a passage of fiction and name the tropes in it. New models keep coming, and this gives me my own way of measuring them.

The rough plan is below. Nothing is built yet.

The plan

Step 1: create a vocabulary

The trope names come from TVTropes. The site is a folksonomy, meaning a community built it with no enforced structure, so entries vary a lot in how broad or narrow they are. I’ll curate 100 to 300 tropes by hand into a fixed vocabulary, with clear rules for when each one counts as present. I already have a few favorites in mind. The test of a good rule is that two people reading the same passage agree. Tropes too vague to grade get dropped. Foreshadowing, e.g., is present in almost any story, so a model earns credit by naming it everywhere. The vocabulary could be swapped for a folklore index like Aarne-Thompson-Uther and the benchmark would test the same skill.

Here is what one item could look like. The passage, which I made up: “The old fisherman teaches the girl to read the tides. On the night of the storm he does not come back from the water, and in the morning she takes his boat out alone.” The right label is Mentor Occupational Hazard, the trope where the teacher dies and the student steps up. Revenge is a plausible wrong answer, because the story turns on a death but nothing in the passage seeks payback. The Mentor is a near miss, true but broader than what the passage supports.

Step 2: find the text

The hard part is finding passages the models can’t cheat on. The benchmark fails when a passage and its trope labels appear together in training data, because then a model can recall the answer instead of reasoning. That rules out anything with a TVTropes page, and popular stories in general, because people discuss popular stories and the discussion names the tropes.

The safer sources are synopses I write, obscure works nobody has discussed, and recent posts from writing subreddits like r/WritingPrompts, taken from the middle of the upvote range. Two details on the Reddit option. The prompt often names the trope outright, so the model never sees the prompt, though the annotators can use it as a hint. And the stories belong to their authors, so nothing gets republished without asking. At a few hundred passages, asking is feasible.

Step 3: label the text

Two people label each passage independently, and I measure how often they agree. Cohen’s kappa is the standard number for this. Only labels with decent agreement survive. Where the two disagree, the trope gets cut or its rule gets sharpened. Each passage also gets a few tropes that sound plausible but are absent, to check that wrong guesses get punished.

If I label everything myself, the benchmark is a quiz about my own opinions. I don’t know yet who the second person is. That is the first real problem to solve.

Step 4: grade the answers

The grading is simple on purpose. The model returns a list of trope names from the vocabulary, and the grader checks that list against the labels. No language model judges anything, so there is no judge for a model to persuade.

The score is F1, which balances precision (what share of the guesses were right) against recall (what share of the true tropes were found). Plain accuracy fails here, because most tropes are absent from any passage, so a model that answers nothing looks near perfect. Precision punishes the cheapest trick, which is guessing everything plausible. A near miss scores zero. Answering The Mentor when Mentor Occupational Hazard is correct earns nothing, and the miss goes into a separate analysis instead. Every partial credit rule gives a model a way to raise its score without getting better at the task.

Two checks before trusting a score. First, read about 50 transcripts sorted by score, because some high scores will be garbage the grader missed and some low scores will be right answers it mishandled. Fix the grader or the labels, run again, and expect several rounds. Second, probe each model for contamination. Give it the first half of a passage and see whether it can complete the text or name the source. A model that recognizes a passage can’t be credited with reasoning about it, so the probe results get reported next to the scores.

Step 5: publish

The harness, a small labeled demo set, and results for five to ten current models go in a public repo. The labels for the scored set stay private, because anything published gets trained on within a cycle or two. Public passages still pick up discussion over time, so the plan is to replace them on a schedule. The only passages that stay clean are the ones I write and never publish.

One caveat. A competent annotator can derive the labels from any public passage, so private labels protect against accidental scraping, not against someone labeling the public set on purpose. At this scale it is unlikely anyone would bother, but the writeup should say so.

Later

If the basic version works, the harder question is whether a story plays a trope straight, subverts it, or inverts it. That asks the model to work out what expectation the story sets up and then violates.

The vocabulary is step 1.

Owning my data

Fri, 12 Jun 2026 15:00:00 GMT

I’m trying to keep ownership of the data of my life: my notes, my conversations, my location history. The test I use is whether I can move the data, inspect it, and delete it. If I can’t do all three, I don’t own it.

There seem to be two separate choices here. The first is what gets recorded: how much, in what detail, and when. The second is what gets shown: to whom, and in how much detail. They’re independent. I can record everything and show nothing.

I’m trying this out in Tukur, my location-sharing app. Turning on sharing is one choice. What other people see is a separate choice. The same day of GPS points can show up as a live dot for the few people in my circle, or as a short summary that only I can read.

I don’t think this means self-hosting everything. A third-party service seems fine when I pay it for servers, it doesn’t sell my data, and I can leave with everything. If I want out of Supabase, I can take my Postgres dump and go.

The oldest version of this in my life is a folder of plain text notes that every tool I use reads and writes. Tools come and go, and the files stay.

These days, before I build something new, I ask whether it leaves me owning more of my data or less.

Open web first

Fri, 12 Jun 2026 14:00:00 GMT

So far, everything I’ve shipped is a URL. Send the link, tap it, and you’re in. There’s no install step and no review queue between me and the people I built the thing for.

Half of the reason is arithmetic. I’m one person, and I can’t build and maintain native apps for iOS and Android alone. One web build works on every phone in the family group chat.

The other half is control. Apple and Google decide what goes in their stores and what it costs to be there. On the web, I put the thing up and it’s live.

An installed app can send notifications at any time. A URL waits until I want it. I like that better.

Sometimes the browser can’t do the job. Tukur is at that limit now. iOS won’t let a browser tab read the phone’s location in the background, so Tukur will probably have to become a native app. The plan is to keep the web version as the main one and let the native app cover what the browser can’t. I haven’t tried this yet, so we’ll see.

The cost of web first is discoverability and reach. People look for apps in the store, and a website is easier to miss. So far, with every user in my contacts, that hasn’t mattered.

Home cooking

Fri, 12 Jun 2026 13:00:00 GMT

I’m trying to build software just for myself and the people close to me. Every user is in my contacts. That lets me skip most of the usual product work. There’s nothing to market and nothing to A/B test, and I don’t have to guess whether anyone uses the thing. I can ask them at dinner.

The software still has to be easy. The bar I’m aiming for: someone in my family can figure it out in ten seconds without texting me. I polish things so they’re easy to use, not so they hold attention.

Decisions get smaller too. I hardcode settings that a commercial product would make configurable. I skip the explanations that exist for strangers. I’m building for ten or twenty people, and I know all of them.

The circle stays loose. Someone will share a Tukur link with a person I’ve never met, and that’s fine. I cook for my family, and sometimes a friend brings a friend to dinner. The name comes from Robin Sloan’s essay “An app can be a home-cooked meal”, and I’m more or less following his recipe.

There’s no business model, and I’m not looking for one. Nothing here needs to collect anything from the people using it, so their data stays theirs.

UX principles for engineers

Fri, 24 Apr 2026 20:00:00 GMT

Contents

The three layers
Orthogonalize the axes
Core principles
Reusable patterns
Design process
Vocabulary
Bridges from backend to design
Common traps

The three layers

Most design problems fall into one of three layers. Knowing which layer a problem lives on tells you which tool to reach for.

Layer	What it decides	Fails as	Tool to fix
Information architecture	What concepts exist and how they relate	”I don’t know what this app is about.”	Concept map, rename things
Interaction design	What happens when you do something, and how state moves between screens	”I got stuck / lost / surprised.”	Flow diagrams, prototypes, state machines
Visual design	Type, color, hierarchy, motion (how it looks and feels)	“This feels cheap / cold / generic.”	Mood boards, component libraries, a single aesthetic commitment

Products with a voice (kids’ apps, coaching apps, anything with narrative) have a fourth layer above these: narrative design, how the product speaks. Work the layers in the order a user meets them: voice before pixels, pixels before behavior. A screen that passes one layer and fails another is still failing.

Orthogonalize the axes (find the independent dimensions)

Before designing anything, identify the independent dimensions of your problem. Confused designs are almost always designs where two axes were collapsed into one.

A parent who wants read-aloud mode in Golpo isn’t necessarily targeting a younger child than one who picks read-independently. Reading mode and age are independent: a 6-year-old might want to read alone; a 4-year-old might want a parent along. If you can imagine a user who is high on one dimension and low on the other, they deserve separate treatment. If you can’t, they’re not separate axes.

Common axes worth separating in almost any product:

Concrete attribute vs. abstract stance. Age vs. skill level. Plan tier vs. feature access. Role vs. permission.
Author vs. reader. Creating content vs. consuming it: different postures, different UX needs, often different screens.
Operator vs. end-user. Admin / parent / manager surfaces vs. the person using the product day-to-day.
Session vs. identity vs. profile. “I am here now” vs. “this is me across sessions” vs. “this is who I am managing.” Most products conflate at least two of these.
Private vs. shared. Work done alone vs. work done with or for others. Different stakes, different UI.
Guided vs. free. Step-by-step wizard vs. open canvas. Often toggleable within the same product.
Entry intent. The same user arriving via a share link, a cold homepage visit, or a notification is three different users. Don’t force them through one funnel.

Core principles

On the product as a whole

Jakob’s Law. Users spend most of their time on other products. Novel is not better. Innovate where it creates value and conform where it doesn’t. A grid of cards is a grid of cards. Don’t reinvent it unless you have a reason.

Principle of least astonishment. What happens should match what the user expected. Every surprise is a small failure, even positive ones.

One product, one voice. A feature that feels imported from a different company is a failure. Coherence beats individual screen quality.

On the first session

Time-to-value. The clock starts when the user lands and stops when they feel the product working for them. Every onboarding step is a tax against this clock. Spend the budget wisely.

Deferred registration. Do not ask for an account before the user has felt value. Let them play anonymously first, and claim their work on sign-in.

Avoid the blank page. In creative tools, the empty state is the hardest moment. Pre-fill, suggest, or seed with examples. A thoughtful zero-state is worth ten polish passes elsewhere.

Ask concrete, derive abstract. Users answer “how old are you?” easily; they stumble on “what’s your reading posture?” Ask the concrete proxy, compute the abstract stance, and expose the abstract as an override.

On complexity

Progressive disclosure. Show the minimum; reveal complexity on demand. Power-user settings belong behind a toggle, not on the main surface.

Every element earns its place. If a section feels empty, fix the composition: more whitespace, bigger type, bolder hierarchy.

Forgiving formats. Accept more than you demand. Dates, phone numbers, IDs: strip whitespace, normalize, guess intent. The user should never be punished for a formatting mistake the computer could have fixed.

On feedback and state

Every action has a reaction. If the user does something, the product acknowledges it within ~100ms, even if the real result is async. Silence is the worst feedback.

Undo beats confirm. Prefer a reversible action with an undo over a “are you sure?” modal. Modals interrupt; undo respects the user’s flow.

State visibility. The user should always be able to answer “where am I, what just happened, what can I do next?” from the screen alone. No guessing.

Reusable patterns

Happy path, then edges. Design the most common successful flow end-to-end first. Only then fan out to error states, empty states, edge cases. Designing edges first produces a product that feels defensive instead of purposeful.

Hierarchy of decisions. Every screen has one primary action, zero or one secondary actions, and several tertiary ones. Make the primary action obvious (color, size, position). Demote the rest. If everything is bold, nothing is.

Forgiving first, then strict. The first time a user does something, be forgiving: accept any input, offer examples, don’t punish mistakes. As they become power users, let them opt into stricter, faster, denser surfaces.

Zero state, empty state, error state, loaded state. Every screen has four states. Design all four. The empty state is almost always the weakest in a real product. Invest there.

Pre-commit vs. post-commit. Destructive, expensive, or irreversible actions want a pre-commit pattern: a confirmation, a preview, or a reversible delay. Everything else should be instant.

Convention over invention. A hamburger menu in the top-left. A search icon that looks like a magnifying glass. A primary CTA that’s filled and on the right. These conventions are boring on purpose. Follow them for 95% of your UI; save your novelty budget for the 5% that matters.

Design process

Understand the user. Who are they? What are they trying to do? What do they already know? Everything else compounds from this.
Enumerate flows. List every way a user might move through the product. Draw them. Label entry points, decisions, async waits, terminals.
Orthogonalize. Identify the independent axes. Pin the ones you aren’t addressing now.
Commit to a direction. Decide the aesthetic, the voice, the mode. Do this before pushing pixels. Half the bad designs in the world are designs made without committing.
Design the happy path. End-to-end, one flow at a time.
Fan out to edges. Empty states, errors, re-entry, interruptions.
Show variations. Don’t fall in love with your first idea. Hold it next to two alternatives. Pick by comparison, not by attachment.
Cut. Every element that didn’t earn its place.

Vocabulary

Term	Meaning
Affordance	A visual cue that signals how something can be used (a button that looks pressable).
Information scent	How strongly a link or label hints at what’s behind it. Weak scent = users bail.
Fitts’s Law	Targets that are bigger and closer are faster to hit. Primary actions deserve real estate.
Hick’s Law	More choices = more decision time. Limit options or group them.
Progressive disclosure	Reveal complexity only when needed.
Empty state / zero state	The screen a brand-new user sees before any content exists.
Dead end	A screen with no obvious next action. Always a bug.
Destructive action	Something that can’t be undone or costs real money. Deserves distinct treatment.
Primary / secondary / tertiary	The decision hierarchy on a given screen.
Hero flow / golden path	The single most important flow in the product. The one you polish first.
Entry intent	Why the user landed here (cold visit, share link, notification, etc.). Often deserves separate design.
State machine	The formal set of states a UI can be in and the transitions between them.
Scaffolding	Temporary structure that supports the user until they don’t need it. Training wheels, inline hints, example content.
Deferred registration	Letting users do real work before asking for an account.
Happy path	The successful, normal-case flow. Design this first.
Dogfooding	Using your own product daily. The single best way to find what’s broken.

Bridges from backend to design

You already do most of this thinking, just in a different domain.

Information architecture is schema design. Getting the data model wrong makes every API awkward; getting the IA wrong makes every screen awkward. Both are expensive to migrate.
Interaction design is state machines. Every screen is a state; every button is a transition. Design the graph, not the pages.
Empty states, error states, loading states, loaded states are your four required branches. Handling one and not the others is like catching success but not exceptions.
Undo is transaction rollback. Prefer it over confirmation dialogs wherever the work to implement is reasonable.
Orthogonalization is separation of concerns: the same move as splitting authentication from authorization. The same discipline that makes clean modules makes clean UIs.
Progressive disclosure is the interface analog of good defaults. Everyone gets the reasonable thing; power users opt in.
Forgiving formats is Postel’s Law. “Be conservative in what you send, liberal in what you accept” applies to user input too.

Common traps

Adding filler to fill space. A screen feels empty, so you invent a “stats” section or a decorative illustration. Solve it with composition: more whitespace, bigger type, bolder hierarchy. Let the page breathe.

Designing for the power user first. You know the product, so you surface the advanced options prominently. Design for the first-time user instead. Hide power features behind progressive disclosure. Reveal density only once the user has signaled they want it.

Asking every question your database can store. Onboarding becomes a form. Ask the fewest questions that unlock real personalization. Derive the rest. Defer the optional.

Falling in love with the first idea. Always hold your favorite next to two alternatives. You don’t know if an idea is good until you’ve seen it beat something.

Treating novelty as quality. Spend your novelty budget on the 5% that matters. Conform on the rest. Jakob’s Law applies here too.

Confirming everything. Every action gets a “Are you sure?” dialog. Reversible actions + a brief undo window respect the user’s flow better. Reserve modals for truly destructive or expensive actions.