When completion rates mislead product teams

Completion is a flattering word. It sounds like a learner finished the work. In most learning apps we review, the event named lesson_complete fires when a video player crosses a threshold — often 80 or 90 percent — or when a user taps Continue. Neither of those is evidence that an item was attempted, let alone understood.

We keep seeing the same pattern in Bangkok and remote desks: completion climbs after a player redesign, tutor Slack gets louder, and the product team celebrates a north star. The tutors are watching children skip the check. The dashboard is watching a progress bar.

Split the events or stop using the word

At minimum, fire three different events: video or reading reached end, check presented, check submitted. If you offer hints, log hint separately. If a learner can skip, log skip. Folding these into one “complete” flag is how you get a metric that can only go up when you make the video shorter.

Once the events exist, completion for product reviews should mean check submitted. Video end can live on an engineering dashboard. Mixing them in a board pack is how academic partners learn to ignore you.

What happens to the number

It usually falls. Teams hate this. We still ask them to ship the honest definition before changing pedagogy. A lower, trusted number is usable. A high, theatrical one is not. In one 2025 Cohort Signal Lab seat, check-based completion sat near 54% after the split; video-based completion had been advertised near 80%. The team moved a poorly stemmed item out of lesson 3. Check completion rose later. That rise meant something.

A limitation we state in the lab

Splitting events does not tell you whether the item was fair. Item quality is a different craft. Assessment Telemetry goes further into hint rates and second tries. Completion-as-submit only answers “did they do the work you asked,” not “was the work any good.”

If your current complete flag still includes video end, write that on the slide in plain language. Hiding the definition is worse than a modest chart. More on how we teach the split is in the Cohort Signal Lab syllabus.