Anomaly Detection for Non-Normal Data
In short
Real data, such as sleep length, is often not bell-shaped, so a normal curve flags the wrong points as unusual. A kernel density estimate fits the shape the data really has. Okaya uses this idea to rank each new check-in result against the same person's own past results.
To find an unusual value in data that does not follow a bell curve, stop assuming the bell curve. A kernel density estimate (KDE) builds the distribution from the data points themselves, so it follows whatever shape the data has: two humps, a long tail, a hard floor. You can then ask how likely a new value is, or what percentile it falls at, without forcing the data into a shape it never had. Okaya uses this approach to compare each person's check-in with that same person's own past results.
The rest of this post shows why the normal curve fails, how a KDE works, and what it means for a readiness check-in.
Why use statistics for anomaly detection at all?
Statistical methods give you a probability for every new point, and they run fast. Machine learning has plenty of other options (nearest neighbors, isolation forests, clustering), but a probability is easy to explain and easy to set a threshold on.
Two practical reasons drive the choice:
- A number you can act on. A density or a percentile tells you how far out of the ordinary a value is. You can draw the line for "unusual" wherever the use case needs it.
- Speed. The math for common distributions is already in standard libraries, so a scoring step stays quick even when it runs many times a day.
Why does the normal curve give wrong answers?
The normal curve assumes the data has one peak and spreads out evenly on both sides. When the real data has a different shape, the curve puts "typical" in the wrong place and calls ordinary values strange.
The usual method fits two numbers to the data: the mean (μ, where the curve is centered) and the standard deviation (σ, how wide it is). For any new value you then read off a density. A low density means the value is rare, so it may be an anomaly.
Here is where it breaks. Take a small, made-up set of sleep lengths in hours, one group on night shifts and one group on a regular day schedule.
| Group (illustrative data) | Sleep lengths, hours |
|---|---|
| Night-shift workers | 5, 5.5, 4.7, 5.6, 6.2, 5.4, 5.7, 4.3, 4.9 |
| Day-schedule group | 8.3, 8.1, 7.9, 8.5, 8.6, 8.4, 8.9, 9 |
Put both groups together and fit a normal curve. On this data the mean comes out at about 6.8 hours and the standard deviation at about 1.7 hours. The curve's peak, the value it treats as most typical, sits at 6.8 hours, where nobody in the data actually sleeps. Meanwhile 5 hours and 8.5 hours, each completely ordinary for one of the groups, land on the slopes and look less likely than the empty middle.
People often lean on the central limit theorem to justify the normal curve. That theorem describes averages over large samples. It says little about the shape of one person's own values, and that is the level a readiness check-in works at.
How does a kernel density estimate work?
A KDE places a small bell curve (a "kernel") on every data point and adds them up. Where points cluster, the bumps stack and the density is high. Where there are no points, the density stays low.
On the sleep data above, with each small bell curve 0.5 hours wide, the result has two humps, one near 5.3 hours and one near 8.5 hours, with a valley between them. A new value of 6.8 hours now gets a low density, less than a sixth of the density at 5 hours. That is the right answer: 6.8 hours is unusual for both groups.
The idea fits in a few lines of Python:
from scipy.stats import norm
night_shift = [5, 5.5, 4.7, 5.6, 6.2, 5.4, 5.7, 4.3, 4.9]
day_schedule = [8.3, 8.1, 7.9, 8.5, 8.6, 8.4, 8.9, 9]
data = night_shift + day_schedule
bandwidth = 0.5 # width of each small bell curve, in hours
def density(x):
# average of one small bell curve centered on each data point
return sum(norm(point, bandwidth).pdf(x) for point in data) / len(data)
print(density(5.0)) # high: ordinary for the night-shift group
print(density(6.8)) # low: unusual for both groups
In practice you would use a library such as SciPy's gaussian_kde, which also picks the bandwidth for you. The bandwidth is the one real choice in a KDE. Too narrow and every point becomes its own island. Too wide and the two humps blur back into one.
Can a KDE handle more than one variable?
Yes. The same method works in two or more dimensions. Each kernel becomes a small bell-shaped hill centered on a point in that space, instead of a bump on a line. That lets the model capture how measures move together, for example sleep length together with the time someone woke up, rather than treating each number as if it had no context.
The same tool can also predict. If you know some features of a new observation, the density built from past observations tells you which values of the remaining feature are most likely. It is a simple, transparent way to get an expected value from data with an awkward shape.
How does Okaya compare today with your earlier check-ins?
Okaya ranks each new check-in result against a distribution built from the same person's earlier check-ins, blended with a distribution from a wider group. A very low or very high percentile means today is unusual for you.
A check-in is short and private: you talk with Sanora for about 3 minutes. It compares how you sound, what you say and how your expression moves today with your own earlier check-ins. Each of those gives numbers, such as the pitch of your voice or how often you blink, and each number gets its own distribution built from your past values. Today's value is then placed in that distribution as a percentile.
This is why the non-normal case matters. One person's sleep, energy or speech can easily have two humps (shift days and days off) or a long tail (the week after a hard call). A normal curve fitted to that history would flag ordinary days as strange and miss real changes. A KDE fitted to the same history follows the pattern the person really has.
The inputs to this comparison are stored numbers, not images. Okaya never identifies who you are from your face. There is no facial recognition.
Key points
- A normal curve assumes one peak. When data has two humps or a long tail, it marks ordinary values as rare and can put its peak where no data exists.
- A kernel density estimate builds the distribution from the data points, so it follows the real shape.
- The bandwidth is the main setting to choose, and libraries such as SciPy pick a sensible default.
- Okaya applies this per person: each check-in result is ranked against that person's own past results.