Turn off what doesn’t fit.
A research agent is meant to explore. An extraction pipeline isn’t. Switch checks off for the features where they don’t apply, rather than living with the noise.
Everything Sam notices.
And what each one needs.
Some checks work from a feature’s first report. Some need its own history. A few need patterns that only show up across many teams. This is the whole list, marked plainly: Today means working in the pilot build. Coming means designed, not built.
See the list
Checks from the job.
The ones only Sam asks for.
Every feature is described before it reports. These checks come from that description, so they catch problems no general threshold would.
-
01
Coming
A job that runs twice
One trigger starting two runs, when you said one. Each run looks healthy on its own.
-
02
Coming
An input that doesn’t fit
An input whose shape doesn’t match what the feature is for, even when it’s within every size limit.
-
03
Coming
A source gone quiet
Reports that keep arriving while nothing in them changes, or reports that stop. Sam says “insufficient visibility”, not “healthy”.
-
04
Coming
A feature nobody described
A new feature that starts reporting before anyone has said what it’s for. Sam asks the owner first.
-
05
Coming
One workload crowding another
On shared hardware, one model’s work slowing another’s. Measured in waiting time and throughput.
No history needed.
Live on a feature’s first report.
These need only the measurements in each report, so they work from the moment a feature is connected.
-
06
Today
Model inventory
Which models each feature calls, and how often.
-
07
Today
Deprecation exposure
A feature still calling a model with an announced retirement date.
-
08
Today
A run that stalled
A call that starts and never finishes: no reply, no error, nothing that would trip an alert.
-
09
Today
Retrying into an error
The same call sent again and again into an error that isn’t going away.
-
10
Today
Growing context
Each turn carrying more context than the last, so every call costs more than it needs to.
-
11
Coming
Activity at odd hours
Runs outside your working hours, when a problem can go on longest before anyone sees it.
One run at a time.
Watched while it happens.
These look at the run in progress: what it has done so far, and what it set out to do.
-
12
Today
A loop
The same, or nearly the same, call repeating.
-
13
Today
Repeated tool calls
The same tool called with the same arguments, over and over.
-
14
Coming
Going in circles
A run that tries one approach, then another, then returns to the first. A simple duplicate check can’t see this.
-
15
Today
Fan-out
One run starting more and more child runs.
-
16
Coming
Not converging
Replies spreading out instead of narrowing towards an answer.
-
17
Coming
Ignoring its instructions
A run drifting away from what its system prompt asks for.
-
18
Coming
Working on the wrong thing
A run drifting away from the request it started with.
Its own baseline.
Sharper as it learns.
These compare a run with earlier runs of the same feature. They stay quiet until Sam has seen enough to know what normal looks like.
-
19
Today
Too many steps
A run taking forty steps where this feature usually takes four.
-
20
Coming
Something new
A call unlike anything this feature has made before.
-
21
Coming
More cost for the same result
The cost of a finished task creeping up while the outcome stays the same.
A longer view.
Within your research window.
These need history. How far back Sam looks depends on your plan’s research window.
-
22
Coming
A slow decline
A feature getting a little worse each week: more steps, more tokens, more retries.
-
23
Coming
The change that made it worse
Which change, on which day, made this feature cost more or fail more.
Patterns no team sees alone.
Learned from anonymised data.
Some problems only show up across many teams. Sam finds them in anonymised measurements and fingerprints. No team can identify another.
-
24
Coming
The model changed, not you
Many teams’ features on the same model getting worse on the same day.
-
25
Coming
Shapes that tend to fail
Prompt shapes that often run into the same problem.
The shape of a good run.
Designed, not yet built.
Sam learns what a feature’s runs usually look like, then notices when one doesn’t. Anything that could stop a run starts as advice only, and stopping still needs your authority.
-
26
Coming
A run unlike the others
This run resembles no earlier run of the feature.
-
27
Coming
Off the usual path
Runs usually take one shape. This one left it at step four.
-
28
Coming
Normal drift, unusual drift
Learning which features change legitimately, so only unusual change is raised.
-
29
Coming
An early warning
A run shaped like runs that ended badly, raised before the cost builds up.
-
30
Coming
A new shape after a prompt change
A change in how runs unfold after the prompt changed, which shows up before cost does.
-
31
Coming
Shapes that tend to fail, across teams
Run shapes that often end badly, learned from anonymised runs across many teams.
A signal is only the start.
Today, every finding comes with its evidence. Bringing it to the owner and checking that the fix worked are coming.
Noticing is step one.
Sam stays for the rest.
See Sam take one signal all the way to a verified fix.
Try the walkthrough