---
title: "Finding the unknown unknowns in mobile app performance and availability"
metaTitle: "Finding the unknown unknowns in mobile app performance."
slug: "unknown-unknowns-mobile-observability"
blurb: "Backend observability thinking misses the mobile failures nobody predicted. See how complete data and AI agents using bitdrift find the unknown unknowns."
metaDescription: "Backend observability thinking misses the mobile failures nobody predicted. See how complete data and AI agents using bitdrift find the unknown unknowns."
cover:
url: "/assets/posts/unknown-unknowns-mobile-observability/feature-unknown-unknowns-desktop@1x.webp"
alt: "Illustration for finding unknown unknowns in mobile app performance."
socialThumbnail:
url: "/assets/posts/unknown-unknowns-mobile-observability/feature-unknown-unknowns-desktop@1x.webp"
alt: "Illustration for finding unknown unknowns in mobile app performance."
author:
- "steve"
tags:
- "engineering"
- "mobile"
- "observability"
publishedDate: "2026-10-14T12:00:00Z"
modifiedDate: "2026-10-14T12:00:00Z"
---
Every observability tool ever built asks you the same question before it will help you: *what are you looking for?*
Sometimes that question has an answer. You know the endpoint that matters, the checkout flow that pays the bills, the p99 that wakes people up. You define the SLO, set the threshold, and get paged when the line moves. That's the known-unknown business, and the industry has gotten genuinely good at it.
The problem is that the incidents that actually hurt your mobile app are almost never in that set. They're the ones nobody thought to instrument:
- A specific OEM's aggressive background killer on one carrier's network in one region.
- A cold-start regression that only reproduces on 3-year-old midrange hardware.
- A silent auth failure that shows up as a conversion dip 6 dashboards away from where it started.
Nobody wrote an alert for those, because writing an alert requires knowing the thing exists. That's the unknown-unknown problem.
The industry has two standard answers to it. **Pre-declared instrumentation** means deciding up front which metrics, logs, and events your app will emit, then shipping that decision in a release. The bet is that if you collect broadly enough, the evidence for tomorrow's surprise is already in your data. **Statistical baselines** learn what "normal" looks like from history and flag anything that drifts outside it. The bet is that you don't need to predict a failure, because the system will notice when something stops looking normal.
On mobile, both bets fail for structural reasons.
## Why backend observability thinking breaks on mobile
### Statistical baselines assume a well-behaved population
Backend telemetry is well-behaved in a way that's easy to take for granted. You control the hardware, the OS, and the deploy, and you can roll back in 10 minutes. Traffic shape is roughly stationary. Under those conditions, standard deviation means something, an SLO burn rate means something, and "anomalous versus the last 7 days" means something.
A mobile fleet is tens of thousands of device models and a decade of OS versions, each with its own quirks about background execution, memory pressure, and thermal throttling. Networks range from campus fiber to a train tunnel to a captive portal that returns HTTP 200 with a login page. On top of that come battery states, low-power modes, storage pressure, and permissions your user revoked last Tuesday.
Aggregate all of that and what do you get? Not a bell curve with an interesting tail. You get a pile of overlapping distributions, and the failing cohort you're chasing sits inside a global average that looks healthy. Standard deviation over a fleet like that is a number, not a signal. The aggregate is the disguise.
### Pre-declared instrumentation assumes you can change your mind quickly
When a backend engineer guesses wrong about what to log, the cost is a deploy: add the field, ship it in minutes, done. On mobile, every change to what the app records is an app release. That means code review, app store review, a staged rollout, and an adoption curve where a real slice of your fleet stays on old versions for months. Instrumentation you add today describes next month's incident, on the users who updated. Cost narrows the guess further. Every field is multiplied across millions of devices and paid for at ingest, so teams instrument sparingly and sample what they do collect.
So mobile teams guess, narrowly and in advance. They pick metrics, ship the app, and wait a release cycle to find out whether they picked right. Most of the time they didn't, because the interesting failures are by definition the ones nobody predicted. So they add more instrumentation and wait again. Debugging at the speed of the app store.
## AI is the right tool, fed the wrong data
Unknown unknowns are a search problem over messy, heterogeneous evidence: crashes next to network timings next to lifecycle transitions next to a user's actual path through the app. Large models are good at exactly this. A model can hold all of that evidence at once, form a hypothesis about a cohort nobody defined, and go check it, without anyone knowing the question in advance.
So why hasn't AI already solved mobile observability? Because in most mobile stacks, the data an agent gets is a sliver of what happened. Mobile telemetry is often sampled down to 1–2% to control cost, and the sample is biased toward sessions that survived long enough to upload. Even what makes it in is incomplete:
- It's dropped when the device is offline, backgrounded, throttled, or out of battery.
- It's truncated at ingest.
- It's pre-aggregated into counters that threw away the dimension that mattered.
- It's scoped to whatever fields someone chose to add 3 releases ago.
An agent reasoning over that data is analyzing your sampling policy instead of your app. It will confidently find patterns in the biased 2% and stay blind exactly where unknown unknowns live: rare, cohort-specific, and correlated with failure, which means correlated with not uploading.
Garbage in, garbage out is an old idea. The new part is that the garbage coming back out now sounds extremely confident.
## What changes when the data is complete
bitdrift was built to keep rich telemetry without paying to ingest all of it, years before agents were the obvious consumer. That design turns out to be exactly what an agent needs.
The [Capture SDK](https://bitdrift.io/feature/performance-centric) writes telemetry into a bounded [ring buffer](https://bitdrift.io/feature/ring-buffer) on the device and doesn't ship it by default. Every session on every device stays at full fidelity in a fixed amount of RAM and disk. Nothing is sampled away, because nothing is paid for at ingest until someone asks for it. Metrics are computed on-device, crashes are captured on-device, and both are emitted unsampled: 100% of the fleet, not 2% of the survivors.
A control plane then decides, in real time, what comes back. You can match on any condition (a device model, an OS version, a country, a session where a specific error fired, a metric crossing a line), and the matching buffers upload, including the minutes before the trigger. That last part is the one people underestimate. Rather than looking at the crash, you're looking at everything that led to it, from devices you'd never have thought to instrument.
Two properties follow, and both matter enormously for an agent:
- **The population is complete.** When an agent asks which cohorts regressed on cold start after version 8.4.1, it's querying every device, not a survivorship-biased slice. A hypothesis about a rare cohort can be tested, because the rare cohort is in the data.
- **The instrumentation is live.** The SDK's configuration comes from the control plane, so an agent can change what's captured (new matchers, new workflows, new metrics) and see results in minutes, not in the next release.
## The loop, in practice
Put those 2 properties together and an agent can query reality, then change what reality reports. Through bitdrift's MCP server, any MCP-capable agent can do this with the same access as the engineer driving it.
Say cold start creeps up after version 8.4.1 ships, but the fleet-wide p90 barely moves, so nothing fires. The investigation starts with a question instead of a threshold:
**Prompt**
Compare cold-start p90 on version 8.4.1 against version 8.4.0 by manufacturer and Android API level. Show me any cohort where p90 moved more than 20% while the fleet-wide number moved less than 5%.
The agent finds one OEM's midrange devices on Android 12. They're about 4% of the fleet, and their p90 went from 1.8s to 3.1s. The agent pulls full sessions from those devices and sees a config request in flight during the whole startup gap. Two explanations fit. Either the request now blocks startup, or the devices are under memory pressure and the request is a coincidence. The existing data can't separate them, so the agent changes what the devices report:
**Prompt**
Create a [workflow](https://blog.bitdrift.io/post/new-workflows-ui) for this cohort only. Measure launch to first screen, split by whether the config request finished before first draw. Capture the full session when launch exceeds 2.5s. Expire it after 7 days, and show me the definition before you deploy.
Creating a workflow and deploying it are separate steps, so there's a review point before anything reaches a device. The expiration keeps a temporary investigation from becoming permanent. Minutes after deploy, the answer is in: slow launches line up with the late request, and memory looks the same in both groups. Nobody defined that cohort or wrote that metric in advance. The only thing that waited on the app store was the fix.
The agent needs both halves: complete data to reason over, and a lever to pull when the data isn't enough yet. Without the ring buffer, it's guessing from a biased sample. Without live configuration, its best hypothesis waits for the app store. Neither gap closes with a better model.
## Investigations nobody had to start
That first prompt still told the agent where to look. It named the metric and the dimensions to slice by. The bigger shift is a standing investigation with no question at all: only a definition of "bad" and permission to go find it.
**Prompt** (scheduled, daily)
Study sessions based on poor performance indicators around both user journeys and app and device metrics to find unknown unknowns. Look for cohorts that behave differently from their neighbors, form hypotheses, and pull sessions to test them. Where the data can't decide between explanations, propose a scoped, expiring workflow that would. Keep an investigation log of what you found, what you ruled out, and what's still open. Don't deploy anything without asking me.
From there, the agent builds its own plan. A typical one:
1. **Define bad broadly.** This covers slow and frozen frames, ANRs, slow cold and warm starts, crashes, memory kills, failed or slow requests, and journeys that start but don't finish, like a cart that never reaches payment.
2. **Slice every signal by everything.** That means version, OS and API level, manufacturer and model, network type, locale, foreground state, and any experiment or flag field the app records. For each slice, the agent asks whether the cohort looks like its neighbors.
3. **Rank what diverges.** The agent scores cohorts by how far they sit from comparable ones and how many devices they cover, then drops known issues and anything already alerted on.
4. **Read before concluding.** For the top candidates, it pulls full sessions, lead-up included, and separates what the logs show from what it infers.
5. **Instrument the gap.** When 2 explanations survive, it proposes a workflow that would separate them.
6. **Carry state forward.** Each pass starts from the investigation log, so today's run checks yesterday's workflows before it looks for anything new.
Most passes surface nothing worth a human's time, and that's fine, because a pass is cheap. The workflows get things wrong too. One frozen-frame hunt first counted routine background process kills as user-facing ones, and the corrected matcher went out in minutes instead of a release cycle.
Once a finding holds up, it stops being an unknown unknown. The workflow becomes a monitor, it gets an alert, and the agent moves on. Every pass leaves the instrumentation better targeted at what actually matters. Mobile teams have always done that work by hand, one release cycle per iteration. Now each iteration takes minutes.
## Stop asking engineers to predict the future
Traditional mobile observability asks engineers to know, at instrumentation time, which failures they'll have months later. Then it blames them when the data isn't there during the incident. That's a constraint inherited from pipelines that pay for every byte they ingest.
Keep the data on the device, let an agent decide what to look at, and let it do the looking. Unknown unknowns don't become known because someone got better at predicting what to instrument. They become known because something is finally looking at full-resolution telemetry.
*bitdrift's Capture SDK is free to get started with. If you want to see what's in your fleet that nobody's looking at yet, [try it yourself](https://bitdrift.io/signup) or [contact us for a demo](https://bitdrift.io/contact-us).*
---
## Frequently asked questions
**What are unknown unknowns in mobile observability?**
Unknown unknowns are failures nobody anticipated, so nobody wrote an alert, dashboard, or log line to catch them. On mobile they're often cohort-specific: a crash on one manufacturer's devices, a cold-start regression on older hardware, or an auth failure that only appears as a downstream conversion drop. Traditional tools only surface what was instrumented in advance and survived sampling, so these issues often reach the team through user complaints instead of telemetry.
**What is pre-declared instrumentation?**
Pre-declared instrumentation is deciding ahead of time which metrics, logs, traces, and events an app will emit, and building that decision into the code. On mobile, changing it means shipping a release and waiting for users to update. bitdrift lets teams change what the Capture SDK records in real time through its control plane, without an app release.
**What is a statistical baseline in observability?**
A statistical baseline is a model of normal behavior built from historical telemetry, such as average latency or error rate over the past 7 days, and it's used to flag anomalies. Baselines work well for backend services, where hardware, operating systems, and traffic are fairly uniform. On mobile, telemetry from tens of thousands of device models, OS versions, and network conditions blends into an aggregate that can hide a failing cohort.
**Why does sampled telemetry make AI agents unreliable for mobile debugging?**
Mobile telemetry is usually sampled to a small fraction of sessions, and sessions that crash, go offline, or run out of battery often never upload. An agent working from that data sees a biased slice, and the rare, failure-correlated sessions that explain unknown unknowns are the most likely to be missing. The agent ends up analyzing the sampling policy instead of the app.
**How does bitdrift help AI agents find unknown unknowns?**
The bitdrift Capture SDK records full-fidelity telemetry into an on-device ring buffer and emits metrics and crashes unsampled. Through the control plane and bitdrift's MCP server, an engineer or agent can pull back sessions matching any condition, including the minutes before an event, and change what the SDK captures within minutes. That gives agents complete data to reason over and a way to gather more evidence without waiting for a release.