HMWK 6 — Deriving and Implementing Online Algorithms for Mean and Variance
This article presents concise derivations of the recurrence relationships for the arithmetic mean and variance, followed by minimal, working JavaScript implementations of online algorithms that update these statistics incrementally. Unlike traditional batch methods, online algorithms are efficient in time and memory and offer better numerical stability.
1. Recurrence for the Arithmetic Mean
Let \(x_1, x_2, \dots, x_n\) be observations and let the running mean after \(n\) samples be \(\displaystyle \mu_n=\frac{1}{n}\sum_{i=1}^n x_i\).
Using \(S_n=S_{n-1}+x_n\) and \(S_{n-1}=(n-1)\mu_{n-1}\):
\[ \mu_n=\frac{(n-1)\mu_{n-1}+x_n}{n} = \mu_{n-1} + \frac{x_n-\mu_{n-1}}{n}. \]
This update needs only the previous mean, the new value, and \(n\).
2. Recurrence for the Variance (Welford’s Method)
Define the centered sum of squares \(\displaystyle M2_n=\sum_{i=1}^{n}(x_i-\mu_n)^2\). Let \(\delta=x_n-\mu_{n-1}\) and update the mean \(\mu_n=\mu_{n-1}+\delta/n\). Then:
\[ M2_n = M2_{n-1} + \delta\,(x_n-\mu_n). \]
The variances follow as \(\displaystyle \sigma^2_n=\frac{M2_n}{n}\) (population) and \(\displaystyle s^2_n=\frac{M2_n}{n-1}\) (sample, \(n\ge2\)).
This avoids the unstable identity \( \operatorname{Var}(X)=E[X^2]-E[X]^2 \) that suffers from catastrophic cancellation.
3. Online Mean Calculator
Type a number and press Add Value. The mean is updated via \( \mu\leftarrow \mu + (x-\mu)/n \).
Count (n): 0
Mean (μ): 0.00000
4. Online Variance Calculator
This calculator implements Welford’s algorithm using the \(M2\) accumulator.
Count (n): 0
Mean (μ): 0.00000
Sample Variance (s²): 0.00000
Sample Std Dev (s): 0.00000
5. (Optional) Numerical & Computational Advantages
Online updates minimize catastrophic cancellation and error propagation by avoiding differences of large, close quantities. They need only one pass and \(O(1)\) memory, improving robustness and scalability for streams and large datasets.