PDF

Interactive demonstrations

Interactive demonstrations#

Change a setting, predict its effect, and then examine the result. These small browser experiments expose a mechanism; the chapters develop the assumptions and proofs. Every experiment uses simulated data and a visible random seed. The displays run locally in your browser.

1. Recovering risk from incomplete labels#

Let \(X\mid Y=+1\sim N(\delta/2,1)\) and \(X\mid Y=-1\sim N(-\delta/2,1)\). For the threshold classifier \(g_t(x)=1\{x>t\}\), the true misclassification risk is

\[ R(t)=\pi\Phi(t-\delta/2)+(1-\pi)\{1-\Phi(t+\delta/2)\}. \]

The true curve is available because this is a simulation. A PU learner receives a representative positive sample and an independent sample from the full feature marginal. A UU learner receives two unlabelled mixtures with shared class-conditionals and known priors \(a\) and \(b\).

Try a wrong supplied prior in PU mode. In UU mode, move the two mixture priors together. At \(a=b\), the two-mixture inversion is unidentified. An empirical corrected risk can lie outside \([0,1]\) even though a population misclassification probability cannot.

Black: true risk. Pink: empirical corrected risk. Gold in PU mode: non-negative correction. Each curve evaluates the same fixed threshold grid; choosing its minimum introduces an additional selection step.

The PU display uses \(\widehat\pi\widehat E_+\ell_+ +\widehat E_U\ell_- -\widehat\pi\widehat E_+\ell_-\). Its non-negative version truncates only the estimated negative-class contribution. The UU display uses the mixture inversion from Chapter 5, with the specified target prior. See Chapters 3–4 for PU assumptions and finite-sample qualifications.

2. Exploration with one-arm feedback#

Three stationary Bernoulli arms have means \((0.30,0.45,\mu_3)\). Each round reveals only the reward from the selected arm. Compare round-robin uniform exploration with UCB, which first pulls each arm once and then maximises

\[ \widehat\mu_k(t-1)+\sqrt{2\log(t)/N_k(t-1)}. \]

The plot shows cumulative pseudo-regret \(\sum_{t=1}^T(\mu^\star-\mu_{A_t})\). Only the simulator uses the true means. Both procedures use the same pre-generated reward stream for each arm, indexed by that arm’s pull count, to support a paired comparison.

Black: uniform exploration. Pink: UCB. One trajectory does not establish an expected-regret guarantee. UCB need not win on every finite sample or every short horizon.

This is a reward-bandit demonstration. A duelling-bandit learner observes a comparison outcome instead of an arm reward; it needs an explicit winner criterion and a corresponding performance definition. See Chapter 9 and Chapter 10.

3. Selection changes the reference distribution#

Consider independent \(Z_1,\ldots,Z_p\sim N(0,1)\) and select \(J=\arg\max_j|Z_j|\). Conditional on the remaining coordinates, the selected statistic must satisfy \(|Z_J|\ge m\), where \(m=\max_{k\ne J}|Z_k|\). The selected two-sided p-value is

\[ p_{\mathrm{sel}}=\frac{1-\Phi(|Z_J|)}{1-\Phi(m)}, \qquad p_{\mathrm{naive}}=2\{1-\Phi(|Z_J|)\}. \]

For \(p=1\), take \(m=0\) and the two p-values agree. Under this independent all-null model, selecting the smallest naive p-value produces rejection probability \(1-(1-\alpha)^p\). The repeated simulation reports a Monte Carlo standard error alongside each rejection rate.

This is an elementary Gaussian selection example, not a clustering-significance calculator. Hierarchical clustering and k-means require their own algorithm-specific selection events and conditioning.

The display numerically evaluates the normal tail and repeats the null experiment 3,000 times. Chapter 11 develops the conditional calculation. Chapters 12–13 explain what changes for selected cluster contrasts.