Choosing the knots¶
The VAF on the fitting sample always grows with the number of knots: every extra knot enlarges the spline space of
each variable, so the in-sample criterion cannot choose it. In 0.2.0 the number of knots is chosen by hand, guided
by the transformation plots and the SmallSampleWarning (n < 5 (degree + knots)).
Design decision D12 (status: proposed for 0.3.0, see DESIGN.md) compares four options:
manual choice (today); a score(X) method (VAF of the p components on new data, with new rows clipped to the
training range) for use with GridSearchCV; a QLPCACV class (grid of 1–4 knots at the percentiles, 5-fold
cross-validation, one-standard-error rule, grid limited by n ≥ 5 (degree + knots), results in cv_results_); and an
automatic choice inside QLPCA. The recommendation is score() and QLPCACV in 0.3.0 — neither exists in
0.2.0 — and the automatic choice is rejected (cost, reproducibility, small samples, scikit-learn convention). Both
depend on the option extrapolate="clip" deferred in D11.
Preliminary evidence¶
Out-of-sample VAF (%), p = 2, degree 1, default mode; 5-fold cross-validation repeated 4 times
(RepeatedKFold(5, 4, random_state=0)); test rows clipped to the training range; VAF = share of the test rows'
transformed variance captured by the training loading directions. Mean over the 20 splits, standard error in
brackets. Computed by
comparison/princals/knot_selection.py
(file results/knot_selection.csv).
| data | 0 knots | 1 knots | 2 knots | 3 knots | 4 knots | 6 knots |
|---|---|---|---|---|---|---|
| cylinder_0 (n=51) | 83.4 (1.1) | 94.5 (0.4) | 96.5 (0.3) | 97.3 (0.3) | 97.7 (0.3) | 98.3 (0.2) |
| cylinder_10 (n=51) | 82.0 (0.9) | 92.8 (0.4) | 94.9 (0.3) | 95.7 (0.2) | 96.2 (0.2) | 96.9 (0.2) |
| cylinder_25 (n=51) | 72.8 (0.9) | 87.6 (0.3) | 89.5 (0.3) | 90.6 (0.4) | 91.1 (0.5) | 91.4 (0.5) |
| cylinder_10 (n=500) | 74.1 (2.4) | 93.2 (0.1) | 95.6 (0.1) | 96.7 (0.1) | 97.2 (0.1) | 97.8 (0.0) |
| gas_turbine (n=400) | 79.6 (0.5) | 81.7 (0.4) | 82.0 (0.5) | 82.3 (0.4) | 82.6 (0.4) | 82.3 (0.4) |
Open questions recorded in D12: a criterion on the original scale of the variables; knots per variable (greedy search); the location of the knots, not only their number. The Kelmarsh application compares knots at the percentiles with knots placed at observed control regions.