Question about the regularization

[leavesandlemmas]

Does the regularization allow for a location other than zero? Usually ridge and lasso regressions are formulated so that when parameter theta=0, it's equivalent to not including the parameter's covariate in the model in the first place. That's not usually the case for BioCro models.

regularization_penalty <- function(
    ind_arg_vals,
    regularization_method,
    regularization_lambda
)
{
    if (toupper(regularization_method) == 'NONE') {
        0.0
    } else if (toupper(regularization_method) == 'LASSO' || toupper(regularization_method) == 'L1') {
        regularization_lambda * sum(abs(ind_arg_vals))
    } else if (toupper(regularization_method) == 'RIDGE' || toupper(regularization_method) == 'L2') {
        regularization_lambda * sum(ind_arg_vals^2)
    } else {
        stop('Unsupported regularization method: ', regularization_method)
    }
}

I think the code could be simplified a bit:

regularization_penalty <- function(
    ind_arg_vals,
    regularization_method,
    regularization_lambda
)
{
    method <- toupper(regularization_method) 
    if (method == 'NONE') {
        0.0
    } else if (method %in% c('LASSO',  'L1')) {
        regularization_lambda * sum(abs(ind_arg_vals))
    } else if(method %in% c('RIDGE',  'L2')) {
        regularization_lambda * sum(ind_arg_vals^2)
    } else {
        stop('Unsupported regularization method: ', regularization_method)
    }
}

What about allowing a custom function to be passed? That way, a user could specify any regularization method. E.g.

regularization_penalty <- function(
    ind_arg_vals,
    regularization_method,
    regularization_lambda
)
{
    if (is.function(regularization_method)) {
        return(regularization_method(ind_arg_vals))
    }
    method <- toupper(regularization_method) 
    if (method == 'NONE') {
        0.0
    } else if (method %in% c('LASSO',  'L1')) {
        regularization_lambda * sum(abs(ind_arg_vals))
    } else if(method %in% c('RIDGE',  'L2')) {
        regularization_lambda * sum(ind_arg_vals^2)
    } else {
        stop('Unsupported regularization method: ', regularization_method)
    }
}

[eloch216]

These are good ideas. I don't have much experience with regularization and haven't actually tried using either of these methods with BioCro. But I like the flexibility of being able to provide a function. I think I would pass both ind_arg_values and regularization_lambda to the regularization_method function though, since I think most types of regularization would still use a hyperparameter.

[leavesandlemmas]

Yeah that's a good idea.

[eloch216]

See PR #10