Cohort windows and the sample you did not mean to take

The Cohort Observatory’s second glance is “entry”: who is in the room. Guests like to answer “new installs this week.” That answer silently drops people who already had the app, people who share a device, and people whose identifier arrived after the first session because a login wall sat in the way.

Sample bias in app analytics is rarely malice. It is a default. Vendor UIs make “new users” a checkbox. Shared family tablets make that checkbox a fiction. A product used in a shop, on a till device, will look like a parade of new users if you key on advertising identifiers that reset.

Write down who is missing

We do not ask for a perfect identity graph. We ask for a sentence: “This cohort excludes anyone who completed a job before creating an account.” That sentence belongs in the weekly desk, next to the curve. Without it, a rise in retention can mean a rise in login walls, not a rise in love.

Late binding — attaching a user id after several sessions — creates a survivorship effect. The people who bounced before login never join the “logged-in new user” cohort, so the remaining line looks cultivated. Finance partners notice this faster than product sometimes admits.

Windows interact with bias

A short window on a biased sample is a double kindness. You only count people who identified quickly, and you only give them a few days to return. Lengthening the window without fixing entry just spreads the same flattery over more days. Fix entry first. Then argue about fourteen versus twenty-eight.

Public-interest apps we have taught in Britain often serve households, not individuals. Their honest unit is sometimes the household job, not the install. That is an uncomfortable sentence in a growth review. It is still the right unit if the product is used that way.

What to do on Monday

List the identifier you currently key on. List one population that identifier cannot see. Add that population to the Observatory sentence even if you cannot yet measure them. Measurement can wait; pretending they are in the chart cannot.

The method is taught in the Lab and summarised on the Observatory page. If your identifier story is the thing that will not leave stand-up, write first.