TRexSelector R package guide

T-Rex+Tikhonov: a general penalty matrix

From the elastic net to a penalty matrix

The elastic net adds the ridge penalty \(\frac{\lambda_2}{2} \|\boldsymbol\beta\|_2^2\) to the lasso. Tikhonov regularization in general form [1], [2] replaces it by \(\frac{\lambda_2}{2} \|\boldsymbol\Gamma \boldsymbol\beta\|_2^2 = \frac{\lambda_2}{2} \boldsymbol\beta^\top \boldsymbol K \boldsymbol\beta\) with \(\boldsymbol K = \boldsymbol\Gamma^\top \boldsymbol\Gamma\). The matrix encodes what the analyst knows about the coefficients. \(\boldsymbol K = \boldsymbol I\) gives the ridge penalty of the elastic net, a first-difference operator \(\boldsymbol\Gamma\) gives the graph Laplacian of a chain and asks neighbouring coefficients to be equal (smoothness), a graph Laplacian in general [3], [4] couples the variables along the edges of a graph, a group-mean matrix reproduces the penalty of the informed elastic net, and a block-diagonal centering matrix asks the members of a group to share a coefficient. The Tikhonov T-Rex selector runs the terminating Tikhonov-net solver TIKNET, a pathwise homotopy algorithm, with such a matrix inside the random experiments.

A smooth coefficient profile

The active variables are a contiguous block with equal coefficients, the case a chain Laplacian is made for.

set.seed(2718)
n <- 120
p <- 60
true_support <- 21:30
beta <- numeric(p)
beta[true_support] <- 1
X <- matrix(rnorm(n * p), n, p)
y <- as.numeric(X %*% beta + rnorm(n))

trex_gamma_to_k() builds \(\boldsymbol K\) from an operator \(\boldsymbol\Gamma\) and returns a sparse matrix of the Matrix package. A dense base matrix is accepted everywhere a penalty matrix is.

gamma <- diff(diag(p))                # first differences, (p - 1) x p
if (requireNamespace("Matrix", quietly = TRUE)) {
  K <- trex_gamma_to_k(gamma)
  class(K)
} else {
  K <- crossprod(gamma)
}
#> [1] "dgCMatrix"
#> attr(,"package")
#> [1] "Matrix"
K[1:4, 1:4]
#> 4 x 4 sparse Matrix of class "dgCMatrix"
#>                 
#> [1,]  1 -1  .  .
#> [2,] -1  2 -1  .
#> [3,]  . -1  2 -1
#> [4,]  .  . -1  2

Running the selector

The Tikhonov control is required, because it carries \(\boldsymbol K\). The penalty weight is fixed here. A base control that is passed has to name the solver of the selector, "TIKNET". Without a base control the selector runs the defaults of trex_control() with that solver.

ctrl <- trex_control(solver = "TIKNET", K = 10)
tik <- TRexTikhonovSelector$new(X, y, tFDR = 0.1, seed = 4, verbose = FALSE,
                                tik_control = trex_tikhonov_control(tikhonov_K = K, lambda_2 = 1),
                                control = ctrl)
tik$select()
tik$selected_indices
#>  [1] 21 22 23 24 25 26 27 28 29 30
trex_fdp(tik$selected_indices, true_support)
#> [1] 0
trex_tpp(tik$selected_indices, true_support)
#> [1] 1
tik$lambda2_used
#> [1] 1

TRexTikhonovSelector inherits the fields of TRexSelector and adds lambda2_used, lambda2_cv_diagnostics, lambda2_pilot and dummy_clusters. A selector runs once, as every selector does.

Choosing lambda_2

lambda_2 < 0 (the default) hands the choice to lambda2_method. Three rules exist.

Without a grid the rules run on trex_default_lambda2_grid(), the grid the C++ core holds for them.

A supplied lambda_2 is used as it is, on the scale of the standardized columns. A grid, pilot_grid or the grid of a power rule, is stated for columns of unit norm. Under scaling_mode = "ZSCORE" the columns have the squared norm \(n - 1\), so the rules run at the grid times \(n - 1\), and one grid means the same shrinkage under every scaling mode. Every rule reports its pick on the scale of a supplied value, so lambda2_used can be passed as lambda_2.

pilot <- TRexTikhonovSelector$new(X, y, tFDR = 0.1, seed = 4, verbose = FALSE,
                                  tik_control = trex_tikhonov_control(
                                    tikhonov_K = K,
                                    pilot_grid = c(0, 1, 10), pilot_K = 4, pilot_T = 2),
                                  control = ctrl)
pilot$select()
pilot$lambda2_used
#> [1] 0
pilot$lambda2_pilot$grid
#> [1]  0  1 10
pilot$lambda2_pilot$counts
#> [1] 13 11 10
profile <- trex_tikhonov_lambda2_power_rule(X, y, tFDR = 0.1, seed = 4,
                                            tik_control = trex_tikhonov_control(tikhonov_K = K),
                                            control = ctrl, grid = c(0, 1, 10))
profile$lambda2
#> [1] 0
profile$counts
#> [1] 10 10 10

Dummies for a structured penalty

The false discovery rate (FDR) guarantee needs dummies that are exchangeable with the null variables under the penalty. Two mechanisms take care of that. cluster_dummies = TRUE (default) draws the dummies as per-cluster coloured layers on a correlation clustering of the variables (prior_groups, or hierarchical clustering with corr_max and hc_linkage), and every dummy joins the \(\boldsymbol K\)-coupling of the variable it mirrors (the folded coupling). cluster_dummies = FALSE exists for validation runs. The field dummy_clusters reports the clusters of a run.

Group centering

penalty_structure = "GROUP_CENTERING" lets the selector build \(\boldsymbol K\) itself as the block-diagonal centering penalty \(\boldsymbol I - \boldsymbol J / p_m\) over the groups of prior_groups, so the members of a group share a coefficient. The structure needs prior_groups (group numbers from 1, one per variable), and the selector refuses it without them. An all-singleton partition is legitimate, the penalty vanishes and the path is the lasso on per-variable cluster dummies. trex_build_group_centering_k() returns this matrix for a group assignment.

groups <- rep(1:6, each = 10)
gc <- TRexTikhonovSelector$new(X, y, tFDR = 0.1, seed = 4, verbose = FALSE,
                               tik_control = trex_tikhonov_control(
                                 penalty_structure = "GROUP_CENTERING", prior_groups = groups,
                                 lambda_2 = 1),
                               control = ctrl)
gc$select()
gc$selected_indices
#>  [1] 21 22 23 24 25 26 27 28 29 30

The solver

The selector runs "TIKNET", the pathwise homotopy algorithm of the Tikhonov net. It is the solver of the base control that applies when control is left out. trex_control() names "TLARS" by default, which the selector refuses as every other solver, so a base control that is passed sets solver = "TIKNET".

Bibliography

[1]
A. N. Tikhonov, “Solution of incorrectly formulated problems and the regularization method,” Soviet Math. Dokl., vol. 4, pp. 1035–1038, 1963.
[2]
Y. Park, L. Reichel, G. Rodriguez, and X. Yu, “Parameter determination for Tikhonov regularization problems in general form,” J. Comput. Appl. Math., vol. 343, pp. 12–25, 2018, doi: 10.1016/j.cam.2018.04.049.
[3]
A. J. Smola and R. Kondor, “Kernels and regularization on graphs,” in Learning theory and kernel machines (COLT/kernel 2003), 2003, vol. 2777, pp. 144–158. doi: 10.1007/978-3-540-45167-9_12.
[4]
F. R. K. Chung, Spectral graph theory. American Mathematical Society, 1997.