predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

start time, Cox model only - class numeric of length same as number of patients (n)

output vector: time, or stop time for Cox model, y_ 0 or 1 for binomial (logistic), numeric for gaussian. Must be a vector of length same as number of sample size.

event indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size.

maximum number of steps done in stepwise regression fitting

model family, "cox", "binomial" or "gaussian"

A stepreg() output. If NULL it will be derived.

Indicate whether or not to update progress in the console. Default of 0 suppresses these updates. The option of 1 provides these updates. In fitting clinical data with non full rank design matrix we have found some R-packages to take a very long time or possibly get caught in infinite loops. Therefore we allow the user to track the package and judge whether things are moving forward or if the process should be stopped.

predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

an optional vector of start times in case of a Cox model. Class numeric of length same as number of patients (n)

dependent variable as a vector: time, or stop time for Cox model, Y_ 0 or 1 for binomial (logistic), numeric for gaussian. Must be a vector of length same as number of sample size.

event indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size.

an offset term to be used when fitting the ANN. Not yet implemented in its pure form. Functionally an offset can be included in the first column of the predictor or feature matrix myxs and indicated as such using the lasso option.

model family, "cox", "binomial" or "gaussian" (default)

number of folds for each level of cross validation

number of epochs to run when tuning on number of epochs for fitting final model number of epochs informed by cross validation

for EPoch PRint. print summary info every eppr epochs. 0 will print first and last epochs, 0 for first and last epoch, -1 for minimal and -2 for none.

length of the first hidden layer in the neural network, default 16

length of the second hidden layer in the neural network, default 16

for ACTiVation function. Activation function between layers, 1 for relu, 2 for gelu, 3 for sigmoid.

fraction of weights to randomly zero out. NOT YET implemented.

learning rate for the optimization step in the neural network model fit

a possible weight decay for the model fit, default 0 for not considered

a possible L1 penalty weight for the model fit, default 0 for not considered

1 to indicate the first column of the input matrix is an offset term, often derived from a lasso model, else 0 (default)

Scale used to allow ReLU to exend +/- lscale before capping the inputted linear estimated

Scale used to transform the inital random paramter assingments by dividing by scale

1 as default to re-adjust weights to account for the offset every epoch. This is only used in case lasso is set to 1.

default of 1 for minimizing loss, else maximizing agreement (concordance for Cox and Binomial, R-square for Gaussian), as function of epochs by cross validaition

fit to the end of epochs. Good for plotting and exploration

an optional a numerical/integer vector of length 2, for R and torch random generators, default NULL to generate these. Integers should be positive and not more than 2147483647.

a vector of integers to associate each record to a fold. Should be integers from 1 and fold_n.

predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

an optional vector of start times in case of a Cox model. Class numeric of length same as number of patients (n)

dependent variable as a vector: time, or stop time for Cox model, Y_ 0 or 1 for binomial (logistic), numeric for gaussian. Must be a vector of length same as number of sample size.

event indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size.

an offset term to be ues when fitting the ANN. Not yet implemented.

model family, "cox", "binomial" or "gaussian" (default)

number of folds for each level of cross validation

number of epochs to run when tuning on number of epochs for fitting final model number of epochs informed by cross validation

for EPoch PRint. print summry info every eppr epochs. 0 will print first and last epochs, -1 nothing.

length of the first hidden layer in the neural network, default 16

length of the second hidden layer in the neural network, default 16

for ACTiVation function. Activation function between layers, 1 for relu, 2 for gelu, 3 for sigmoid.

fraction of weights to randomly zero out. NOT YET implemented.

learning rate for the optimization step in teh neural network model fit

weight decay for the model fit.

a possible L1 penalty weight for the model fit, default 0 for not considered

1 to indicate the first column of the input matrix is an offset term, often derived from a lasso model

Scale used to allow ReLU to extend +/- lscale before capping the inputted linear estimated

Scale used to transform the initial random parameter assingments by dividing by scale

1 as default to re-adjust weights to account for the offset every epoch. This is only used in case lasso is set to 1

default of 1 for minimizing loss, else maximizing agreement (concordance for Cox and Binomial, R-square for Gaussian), as function of epochs by cross validation

fit to the end of epochs. Good for plotting and exploration

how many models to run, from which the best fitting model will be selected.

an optional a numerical/integer vector of length 2, for R and torch random generators, default NULL to generate these. Integers should be positive and not more than 2147483647.

a vector of integers to associate each record to a fold. Should be integers from 1 and fold_n.

model summmary

riskset list

the outcome variable in a vector identifying the different potential levels of the outcome

the fraction of the whole sample included in the bootstratp sample

the sigmoid of the link, i.e, the estimated probabilities, i.e. xx = 1/(1+exp(-xb))

the observed data as 0's and 1's

A nested.glmnetr() output object for calibration

Which Beta should be plotted, an integer. This will depend on which machine learning models were run when creating the output object. If unsure the user can run the function without specifying wbeta and a legend will be directed to the console.

The degrees of freedom for the spline function

1 to base the splines on the leave out X*Beta's ($xbetas.cv or $xbetas.boot.oob), or 0 to use the naive X*Beta's ($xbetas). This can be done to see biases associated with the naive approach.

1 (default) to construct calibration plots using the out-of-bag data points, 0 to use in bag (including resampled data points) data points. This option only applies when bootstrap is used instead of k-fold cross validation, and when resample is set to 1. For cross validation evaluations out-of-bag samples (folds) are always used for evaluation. The purpose of oob = 0 is to allow evaluation of the variability of bootstrap calibrations ignoring bias like done in Riley et al., 2023, doi: 10.1186/s12916-023-03212-y and Austin and Steyerberg 2013, doi: 10.1002/sim.5941

1 to calculate bootstrap confidence intervals for calibration curves adjusting for bias, 0 (default) to simply plot the calibration curves based upon the inbag data. This is for exploration only, and only when bootstrap samples were used for model performance evaluation. The applicability of bootstrap confidence intervals for these calibration curves is questionable. If bootci is set to 1 then oob is set to 0.

1 by default to produce plots, 0 to output data for plots only, 2 to plot and output data.

0 by default to not plot the individual fold calibrations, 1 to overlay the k leave out spline calibration fits in a single figure and 2 to produce separate plots for each of the k hold out calibration curves.

plot full data

a power > 1 determining the spacing of the values on the axes, e.g. 2, exp(1), sqrt(10) or 10. The default of 0 plots the X*Beta. This only applies fore "cox" survival data models.

1 (default) to use XBeta used for the spline fit to choose knots in ns() for gaussian and binomial families, 2 to use the XBeta from all re-samples to determine the knots.

the percent of top and bottom of the data to be trimmed away when producing plots. The original data are still used used calcualting the curves for plotting.

Similar to trim but instead of trimming the spline lines, plots vertical refence lines aht the top vref and bottom vref percent of the model X*Betas's

xlim for the plots. This does not effect the curves within the plotted region. Caution, for the "cox" framework the xlim are specified in terms of the X*beta and not the HR, even when HR is described on the axes.

ylim for the plots, which will usually only be specified in a second run of for the same data. This does not effect the curves within the plotted region. Caution, for the "cox" framework the ylim are specified in terms of the X*beta and not the HR, even when HR is described on the axes.

a user specified label for the x axis

a user specified label for the y axis

a number for the line depicting the overall calibration estimates

a number for the line depicting the +/- 2 * standard error lines for the overall calibration estimates

1 to plot a rug for the model x*betas, 0 (default) to not.

an integer seed used to random select the multiple of X*Betas to be used in the rug when using bootstraping for model evaluation as sample elements may be included multiple times as test (Out Of Bag) data.

Deprecated. Use resample option instead.

Deprecated. This term is now ignored.

allowance to pass terms to the invoked plot function

Time variable for a survival analysis, whether or not there is a start time

Event indicator with 1 for event 0 otherwise.

predictor matrix

outcome vector

model family, "cox", "binomial" or "gaussian" (default)

A vector for alpha values considetered when tuning, for example c(0,0.2,0.4,0.6,0.8,1). Default is c(1) to fit the lasso model involving only the L1 penalty. c(0) could be used to ffit the reidge model involving only the L2 penalty.

the gamma vector. Default is c(0,0.25,0.50,0.75,1).

the lambda vector. May be NULL.

a vector of integers to associate each record to a fold. The integers should be between 1 and folds_n.

number of folds for cross validation. Default and generally recommended is 10.

use a finer step in determining lambda. Of little value unless one repeats the cross validation many times to more finely tune the hyperparameters. See the 'glmnet' package documentation.

The path option from cv.glmnet(). 0 for FALSE reducing computation time when the numerics are stable, 1 to avoid cases where the path = 0 option might get very slow. 0 by default.

indicate whether or not to update progress in the console. Default of 0 suppresses these updates. The option of 1 provides these updates. In fitting clinical data with non full rank design matrix we have found some R-packages to take a vary long time or seemingly be caught in infinite loops. Therefore we allow the user to track the program progress and judge whether things are moving forward or if the process should be stopped.

Additional arguments that can be passed to cv.glmnet()

predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

start time, Cox model only - class numeric of length same as number of patients (n)

output vector: time, or stop time for Cox model, Y_ 0 or 1 for binomal (logistic), numeric for gaussian. #' Must be a vector of length same as number of sample size.

event indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size.

model family, "cox", "binomial" or "gaussian"

Maximun number of steps done in stepwise regression fitting. If 0, then takes the value rank(xs_cv).

number of folds for cross validation

method for choosing model in stepwise procedure, "loglik" or "concordance". Other procedures use the "loglik".

a seed for set.seed() to assure one can get the same results twice. If NULL the program will generate a random seed. Whether specified or NULL, the seed is stored in the output object for future reference.

a vector of integers to associate each record to a fold. The integers should be between 1 and folds_n.

folds are to be constructed stratified on an indicator outcome 1 (default) for yes, 0 for no. Pertains to event variable for "cox" and y_ for "binomial" family.

indicate whether or not to update progress in the console. Default of 0 suppresses these updates. The option of 1 provides these updates. In fitting clinical data with non full rank design matrix we have found some R-packages to take a very long time. Therefore we allow the user to track the program progress and judge whether things are moving forward or if the process should be stopped.

-2 Log Likelihoods calculated on the test data

-2 Log Likelihoods for the null models

-2 Log Likelihoods for teh saturated models

sample zize for the indivual foles, or number of events for the Cox model

beginning time for printing elapsed time

last time for calculating split time

start time

stop time

the outcome variable in a vector identifying the different potential levels of the outcome

the numbe of folds to be constructed

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

see help for cv.glmnetr() or nested.glmnetr()

A nested.glmnetr output object.

determines what type of nested cross validation performance measures are compared. Possible values are "devrat" to compare the deviance ratios, i.e. the fractional reduction in deviance relative to the null model deviance, "agree" to compare agreement, "lincal" to compare the linear calibration slope coefficients, "intcal" to compare the linear calibration intercept coefficients, from the nested cross validation.

the power to which the average of correlations is to be raised. Only applies to the "gaussian" model. Default is 2 to yield R-square but can be on to show correlations. pow is ignored for the family of "cox" and "binomial". When pow = 2, calculations are made using correlations and the final estimates and confidence intervals are raised to the power of 2. A negative sign before an R-square estimate or confidence limit indicates the estimate or confidence limit was negative before being raised to the power of 2.

digits for printing of z-scores, p-values, etc. with default of 4

1 to return the deviance ratios in a list, 0 to not return. The deviances are stored in the nested.glmnetr() output object but not the deviance ratios. This function provides a simple mechanism to obtain the cross validated deviance ratios.

A nested.glmnetr output object.

digits for printing of z-scores, p-values, etc. with default of 4

determines what type of nested cross validation performance measures are compared. Possible values are "devrat" to compare the deviance ratios, i.e. the fractional reduction in deviance relative to the null model deviance, "agree" to compare agreement, "lincal" to compare the linear calibration slope coefficients, "intcal" to compare the linear calibration intercept coefficients, from the nested cross validation.

the power to which the average of correlations is to be raised.

Sample size (>=100) for simulated data, default=1000.

Number of columns (>=17) in design matrix, i.e. predictors, default=100.

Vector of length <= ncols for "left most" coefficients. If beta has length < ncols, then the values at length(beta)+1 to ncols are set to 0. Default=NULL, where a beta of length 25 is assigned standard normal values.

either NULL for no interactions or a vector of length 3 to impose a product effect as decribed by intr[1]*xs[,3]*xs[,8] + intr[2]*xs[,4]*xs[,16] + intr[3]*xs[,18]*xs[,19] + intr[4]*xs[,21]*xs[,22]

number of id levels where each level is associated with a random effect, of variance 1 for normal data.

The input seed as a start, NULL, a vector of length 1 or 2, or a list with vectors of length 1 or the number of folds, $seedr for most models and $seedt for the ANN fits

The number of folds in general

The number of folds for the ANN fits

A nested.glmnetr output object.

determines what type of nested cross validation performance measures are compared. Possible values are "devrat" to compare the deviance ratios, i.e. the fractional reduction in deviance relative to the null model deviance, "agree" to compare agreement, "lincal" to compare the linear calibration slope coefficients, "intcal" to compare the linear calibration intercept coefficients, from the nested cross validation.

the power to which the average of correlations is to be raised. Only applies to the "gaussian" model. Default is 2 to yield R-square but can be on to show correlations. pow is ignored for the family of "cox" and "binomial". When pow = 2, calculations are made using correlations and the final estimates and confidence intervals are raised to the power of 2. A negative sign before an R-square estimate or confidence limit indicates the estimate or confidence limit was negative before being raised to the power of 2.

digits for printing of z-scores, p-values, etc. with default of 4

1 to return the deviance ratios in a list, 0 to not return. The deviances are stored in the nested.glmnetr() output object but not the deviance ratios. This function provides a simple mechanism to obtain the cross validated deviance ratios.

A nested.glmnetr output object.

determines what type of nested cross validation performance measures are compared. Possible values are "devrat" to compare the deviance ratios, i.e. the fractional reduction in deviance relative to the null model deviance, "agree" to compare agreement, "lincal" to compare the linear calibration slope coefficients, "intcal" to compare the linear calibration intercept coefficients, from the nested cross validation.

digits for printing of z-scores, p-values, etc. with default of 4

the power to which the average of correlations is to be raised. Only applies to the "gaussian" model. Default is 2 to yield R-square but can be on to show correlations. pow is ignored for the family of "cox" and "binomial".

A nested.glmnetr output object.

digits for printing of z-scores, p-values, etc. with default of 4

determines what type of nested cross validation performance measures are compared. Possible values are "devrat" to compare the deviance ratios, i.e. the fractional reduction in deviance relative to the null model deviance, "agree" to compare agreement, "lincal" to compare the linear calibration slope coefficients, "intcal" to compare the linear calibration intercept coefficients, from the nested cross validation.

the power to which the average of correlations is to be raised. Only applies to the "gaussian" model. Default is 2 to yield R-square but can be on to show correlations. pow is ignored for the family of "cox" and "binomial".

predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in (numeric) matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

optional start times in case of a Cox model. A numeric (vector) of length same as number of patients (n). Optionally start may be specified as a column matrix in which case the colname value is used when outputting summaries. Only the lasso, stepwise, and AIC models allow for (start,stop) time data as input.

dependent variable as a numeric vector: time, or stop time for Cox model, 0 or 1 for binomial (logistic), numeric for gaussian. Must be a vector of length same as number of sample size. Optionally y_ may be specified as a column matrix in which case the colname value is used when outputting summaries.

event indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size. Optionally event may be specified as a column matrix in which case the colname value is used when outputing summaries.

model family, "cox", "binomial" or "gaussian" (default)

1 by default to do the Nested Cross Validation or bootstrap resampling calculations to assess model performance (see bootstrap option), or 0 to only fit the various models without doing resampling. In this case the nested.glmnetr() function will only derive the models based upon the full data set. This may be useful when exploring various models without having to do the timely resampling to assess model performance, for example, when wanting to examine extreme gradient boosting models (GBM) or Artificial Neural Network (ANN) models which can take a long time.

the number of folds for the outer loop of the nested cross validation, and if not overridden by the individual model specifications, also the number of folds for the inner loop of the nested cross validation, i.e. the number of folds used in model derivation.

1 to generate fold IDs stratified on outcome or event indicators for the binomial or Cox model, 0 to generate foldid's without regard to outcome. Default is 1 for nested CV (i.e. bootstrap=0), and 0 for bootstrap>=1.

fit and do cross validation for lasso model, 0 or 1

fit and evaluate a cross validation informed XGBoost (GBM) model. 1 for yes, 0 for no (default). By default the number of folds used when training the GBM model will be the same as the number of folds used in the outer loop of the nested cross validation, and the maximum number of rounds when training the GBM model is set to 1000. To control these values one may specify a list for the doxgb argument. The list can have elements $nfold, $nrounds, and $early_stopping_rounds, each numerical values of length 1, $folds, a list as used by xgb.cv() do identify folds for cross validation, and $eta, $gamma, $max_depth, $min_child_weight, $colsample_bytree, $lambda, $alpha and $subsample, each a numeric of length 2 giving the lower and upper values for the respective tuning parameter. Here we deviate from nomenclature used elsewhere in the package to be able to use terms those used in the 'xgboost' (and mlrMBO) package, in particular as used in xgb.train(), e.g. nfold instead of folds_n and folds instead of foldid. If not provided defaults will be used. Defaults can be seen from the output object$doxgb element, again a list. In case not NULL, the seed and folds option values override the $seed and $folds values.

If to shorten run time the user sets nfold to a value other than folds_n we recommend that nfold = folds_n/2 or folds_n/3. Then the folds will be formed by collapsing the folds_n folds allowing a better comparisons of model performances between the different machine learning models. Typically one would want to keep the full data model but the GBM models can cause the output object to require large amounts of storage space so optionally one can choose to not keep the final model when the goal is basically only to assess model performance for the GBM. In that case the tuning parameters for the final tuned model ae retained facilitating recalculation of the final model, this will also require the original training data.

fit and evaluate a random forest (RF) model. 1 for yes, 0 for no (default). Also, if dorf is specified by a list, then RF models will be fit. The randomForestSRC package is used. This list can have three elements. One is the vector mtryc, and contains values for mtry. The program searches over the different values to find a better fir for the final model. If not specified mtryc is set to round( sqrt(dim(xs)[2]) * c(0.67 , 1, 1.5, 2.25, 3.375) ). The second list element the vector ntreec. The first item (ntreec[1]) specifies the number of trees to fit in evaluating the models specified by the different mtry values. The second item (ntreec[2]) specifies the number of trees to fit in the final model. The default is ntreec = c(25,250). The third element in the list is the numeric variable keep, with the value 1 (default) to store the model fit on all data in the output object, or the value 0 to not store the full data model fit. Typically one would want to keep the full data model but the RF models can cause the output object to require large amounts of storage space so optionally one can choose to not keep the final model when the goal is basically only to assess model performance for the RF. Random forests use the out-of-bag (OOB) data elements for assessing model fit and hyperparameter tuning and so cross validation is not used for tuning. Still, because of the number of trees in the forest random forest can take long to run.

fit and evaluate an Oblique random forest (RF) model. 1 for yes, 0 for no (default). While the nomenclature used by orrsf() is slightly different than that used by rfsrc() nomenclature for this object follows that of dorf.

fit and evaluate a cross validation informed Artificial Neural Network (ANN) model with two hidden levels. 1 for yes, 0 for no (default). By default the number of folds used when training the ANN model will be the same as the number of folds used in the outer loop of the nested cross validation. To override this, for example to shrtn run time, one may specify a list for the doann argument where the element $folds_ann_n gives the number of folds used when training the ANN. To shorten run we recommend folds_ann_n = folds_n/2 or folds_n/3, and at least 3. Then the folds will be formed by collapsing the folds_n folds using in fitting other models allowing a better comparisons of model performances between the different machine learning models. The list can also have elements $epochs, $epochs2, $myler, $myler2, $eppr, $eppr2, $lenz1, $lenz2, $actv, $drpot, $wd, wd2, l1, l12, $lscale, $scale, $minloss and $gotoend. These arguments are then passed to the ann_tab_cv_best() function, with the meanings described in the help for that function, with some exception. When there are two similar values like $epoch and $epoch2 the first applies to the ANN models trained without transfer learning and the second to the models trained with transfer learning from the lasso model. Elements of this list unspecified will take default values. The user may also specify the element $bestof (a positive integer) to fit bestof models with different random starting weights and biases while taking the best performing of the different fits based upon CV as the final model. The default value for bestof is 1.

fit and do a nested cross validation for an RPART model. As rpart() does its own approximation for cross validation there is no new functions for cross validation.

fit and do cross validation for stepwise regression fit, 0 or 1, as discussed in James, Witten, Hastie and Tibshirani, 2nd edition.

fit and do cross validation for AIC fit, 0 or 1. This is provided primarily as a reference.

fit and do cross validation for the full model including all terms (without a penalty), 0 or 1. This is provided primarily as a reference.

This is a vector 8 characters long and specifies a set of ensemble like model to be fit based upon the predicteds form a relaxed lasso model fit, by either inlcuding the predicteds as an additional term (feature) in the machine learning model, or including the predicteds similar to an offset. For XGBoost, the offset is specified in the model with the "base_margin" in the XGBoost call. For the Artificial Neural Network models fit using the ann_tab_cv_best() function, one can initialize model weights (parameters) to account for the predicteds in prediction and either let these weights by modified each epoch or update and maintain these weights during the fitting process. For ensemble[1] = 1 a model is fit ignoring these predicteds, ensemble[2]=1 a model is fit including the predicteds as an additional feature. For ensemble[3]=1 a model is fit using the predicteds as an offset when running the xgboost model, or a model is fit including the predicteds with initial weights corresponding to an offset, but then weights are allowed to be tuned over the epochs. For i >= 4 ensemble[i] only applies to the neural network models. For ensemble[4]=1 a model is fit like for ensemble[3]=1 but the weights are reassigned to correspond to an offset after each epoch. For i in (5,6,7,8) ensemble[i] is similar to ensemble[i-4] except the original predictor (feature) set is replaced by the set of non-zero terms in the relaxed lasso model fit. If ensemble is specified as 0 or NULL, then ensemble is assigned c(1,0,0,0, 0,0,0,0). If ensemble is specified as 1, then ensemble is assigned c(1,0,0,0, 0,1,0,1).

method for choosing model in stepwise procedure, "loglik" or "concordance". Other procedures use the "loglik".

alpha vector for the elastic net models, default is NULL or c(1) for a striclty L1 penalty. Note, in modifications for the elastic net including alpha as a tuning parameter, the current nested.glmnetr() version uses cv.glmnet() for the elastic net calculations.

gamma vector for the relaxed lasso fit, default is c(0,0.25,0.5,0.75,1). The values of 0 and 1 are added to gamma if not described. Note, in modifications for the elastic net including alpha as a tuning parameter, the current nested.glmnetr() version uses cv.glmnet() for the elastic net calculations.

lambda vector for the lasso fit. Note, in modifications for the elastic net including alpha as a tuning parameter, the current nested.glmnetr() version uses cv.glmnet() for the elastic net calculations.

number of steps done in stepwise regression fitting

optional, either NULL, or a numerical/integer vector of length 2, for R and torch random generators, or a list with two vectors, each of length folds_n+1, for generation of random folds for the full data model as well as the the outer cross validation loop, and the remaining folds_n terms for the random generation of the folds or the bootstrap samples for the model fits of the inner loops. This can be used to replicate model fits. Whether specified or NULL, the seed is stored in the output object for future reference. The stored seed is a list with two vectors seedr for the seeds used in generating the random fold splits, and seedt for generating the random initial weights and biases in the torch neural network models. The first element in each of these vectors is for the all data fits and remaining elements for the folds of the inner cross validation. The integers assigned to seed should be positive (>= 1) and not more than 2147483647. Beginning in version 0.5-5 the values from $seedr and $seedt the first last element in each vector was used when fitting each model on the whole data set and the other values were used for the outer cross validation or the bootstrap sample generation. For version 0.4-2 through 0.5-4 the last element from each vector was used when fitting each model on the whole dataset. This change was made so that the set of full models numerical fits do not depend on whether or not resampling is performed (resample is set to 0 or 1 2) or the number of bootstrap resamples. The seeds are generated with the glmnetr_seed() function.

a vector of integers to associate each record to a fold. Should be integers from 1 and folds_n. These will only be used in the outer folds.

Formally to limit the small values for lambda after the initial fit. Note, the current nested.glmnetr() version uses cv.glmnet() for the elastic net calculations and the limit paramter is disregarded.

Formally used for a finer step in determining lambda. Of little value unless one repeats the cross validation many times to more finely tune the hyper paramters. See the 'glmnet' package documentation. Not used for current version of nested.glmnetr which uses cv.glmnetr() for relaxed lasso model fits.

method for handling ties in Cox model formally used for the relaxed model component. In modifications for the elastic net including alpha as a tuning parameter, the current nested.glmnetr() version uses cv.glmnet() for the elastic net calculations which does not allow use of the Efron approach to handling ties, even for the gamma = 0 fits.

0 (default) to delete the input data (xs, start, y_, event) from the output objects from the random forest fit and the glm() fit for the stepwise AIC model, 1 to keep.

1 (default) to retain in the output object a copy of the functional outcome variable, i.e. y_ for "gaussian" and "binomial" data, and the Surv(y_,event) or Surv(start,y_,event) for "cox" data. This allows calibration studies of the models, going beyond the linear calibration information calculated by the function. The xbetas are calculated both for the model derived using all data as well as for the hold out sets (1/k of the data each) for the models derived within the cross validation ((k-1)/k of the data for each fit).

0 (default) to use nested cross validation, a positive integer to perform as many iterations of the bootstrap for model evaluation.

0 to use the bootstrap sample as is as training data, 1 to include the unique sample elements only once. A fractional value between 0.5 and 0.9 will sample without replacement a fraction of this value for training and use the remaining as test data.

optional vector identifying dependent observations. Can be used, for example, when some study subjects have more than one row in the data. No values should be NA. Default is NULL where all rows can be regarded as independent.

1 (default) to track progress by printing to console elapsed and split times, 0 to not track

Deprecated, and replaced by resample

A file name at which to save the intermediate results at the end of each outer loop of the resampling. This may be useful when fitting one of the machine learning models crashes or hangs in one of the iterations of the outer loop (resampling of the nested cross validation or the bootstrap). The value for int_file must be a valid file name for your operating system and installation.

- as in glmnet(). default is 0. glmnet() can have troubles converging for logistic and Cox models in which case one can set path = 1. When not needed path = 1 generally has longer computation times. In modifications for the elastic net including alpha as a tuning paramger, the current nested.glmnetr() version uses cv.glmnet() for the elastic net calculations.

additional arguments that can be passed to glmnet()

predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

an optional vector of start times in case of a Cox model. Class numeric of length same as number of patients (n)

dependent variable as a vector: time, or stop time for Cox model, Y_ 0 or 1 for binomial (logistic), numeric for gaussian. Must be a vector of length same as number of sample size.

event indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size.

model family, "cox", "binomial" or "gaussian" (default)

a vector (numeric) of values to search over for optimization of the Random Forest fit. This if for the mtry input variable of the orsf() program specifying the number of terms to consider in each step of teh Random Forest fit.

a vector (numeric) of 2 values, the first for the number of forests (ntree from orsf()) to use when searhcing for a better bit and the second to use when fitting the final model. More trees should give a better fit but require more computations and storage for the final. model.

This nsplit of orsf(), a non-negative integer for the number of random splits for a predictor.

a seed for set.seed() so one can reproduce the model fit. If NULL the program will generate a random seed. Whether specified or NULL, the seed is stored in the output object for future reference. Note, for the default this randomly generated seed depends on the seed in memory at that time so will depend on any calls of set.seed prior to the call of this function.

a small number, a lower bound to avoid division by 0

1 to output a brief summary of the final selected model, 3 to output a brief summary on each model fit in search of a better model or 0 (default) to not output this information.

a nested.glmnetr() output object.

one of c("lasso", "elastic", "ridge") to plot the deviance curves of the respective model fit. Default is "lasso" for tuned relaxed lasso.

A specific value of alpha for plotting. Used only when type is set to "elastic". Specifies which alpha is to be used for deviance plots. Default is "alpha.min", else must be an element of the alpha vector used in running the elastic net model. This can be reviewed using summary(fit) where fit is a nested.glmnetr() output object. Note, alpha is 1 for the lasso model and alpha is 0 for the ridge model.

a specific level of gamma for plotting. By default gamma.min will be used.

a lower limit of lambda when plotting.

an indicator to plot the upper 95 percent two-sided confidence limits.

a title for the plot.

default of FALSE plots deviances, option of TRUE plots coefficients.

default of TRUE to write to console information on lambda and gamma selected for output. FALSE will suppress this write to console.

line type for the cross validated deviances. Default is 1.

2 to track progress by printing to console, 0 (default) to not track.

Additional arguments passed to the plot function.

A nested.glmnetr output object.

one of c("lasso", "elastic", "ridge") to plot the deviance curves of the respective model fit. Default is "lasso".

A specific level of alpha for plotting. By default alpha.min will be used such that the triplet (alpha.min, gamma.min, lambda.min) minimizes the model deviance.

A specific level of gamma for plotting. By default gamma.min will be used such that the pair (gamma.min, lambda.min) minimzes the model deviance.

A lower limit of lambda for plotting.

A title for the plot

Default of TRUE to write to console information on gamma and lambda selected for output. FALSE will suppress this write to console.

Additional arguments passed to the plot function.

A nested.glmnetr output object

type of plot to be produced from the (nested) cross validation model fits and evaluations. One of c("devrat", "devian", "agree", "intcal", "lincal") to plot estimates of one these performance measures. One of c("lasso", "elastic", "ridge") to plot model coefficients or deviances as function of lambda and to some degree gamma and alpha. Default is "devrat", the fractional reduction in deviance relative to the null model deviance. Use, "devrat" to plot deviance ratios, "devain" to plot devainces, "agree" to plot agreement e.g. R-square or concordance), "lincal" to plot the linear calibration slope coefficients, "intcal" to plot the linear calibration intercept coefficients or "devian" to plot the deviances from the nested cross validation. For each performance measure estimates from the individual (outer) cross validation fold are depicted by thin lines of different colors and styles, while the composite value from all folds is depicted by a thicker black line, and the performance measures naively calculated on the all data using the model derived from all data is depicted in a thicker red line.

A specific value of alpha for plotting. Used only when type is set to "elastic". Specifies which alpha is to be used for deviance plots. Default is "alpha.min", else must be an element of the alpha vector used in running the elastic net model. This can be reviewed using summary(fit) where fit is a nested.glmnetr() output object. Note, alpha is 1 for the lasso model and alpha is 0 for the ridge model.

A specific level of gamma for plotting. By default gamma.min will be used. Applies only for types in c("lasso", "elastic").

A lower limit of lambda when plotting. Applies only for type = "lasso".

A title

Plot upper 95 percent two-sided confidence intervals for the deviance plots. Applies only for type = "lasso".

1 (TRUE) to plot coefficients, else 0 (FALSE) to plot deviances as function of tuning paramters. Only applies for type in c("devrat", "devian", "agree", "intcal", "lincal"). See option 'type'.

Default of TRUE to write to console information on lam and gam selected for output. FALSE will suppress this write to console. Applies only for type = "lasso".

Power to which agreement is to be raised when the "gaussian" model is fit, i.e. 2 for R-square, 1 for correlation. Does not apply to type = "lasso".

y axis limits for model performance plots, i.e. does not apply to type = "lasso". The ridge model may calibrate very poorly obscuring plots for type of "lincal" or "intcal", so one may specify the ylim value. If ylim is set to 1, then the program will derive a reasonable range for ylim. If ylim is set to 0, then the entire range for all models will be displayed. Does not apply to type = "lasso".

By default 1 to produce a plot, 0 to return the data used in the plot in the form of a list.

By default 1 to display model performance estimates form individual folds (or replicaitons for boostrap evaluations) when type of "agree", "intcal", "lincal", "devrat" for "devian". If 0 then the individual fold calculations are not displayed. When there are many replications as sometimes the case when using bootstrap, one may specify the number of randomly selected lines for plotting.

1 (default) to include results for the untuned XGB model, 0 to not include.

2 to track progress by printing to console, 0 (default) to not track.

Additional arguments passed to the plot function.

A nested.glmnetr output object

determines what type of nested cross validation performance measures are plotted. Possible values are "devrat" to plot the deviance ratio, i.e. the fractional reduction in deviance relative to the null model deviance, "devian" to plot deviance, "agree" to plot agreement in terms of concordance, correlation or R-square, "lincal" to plot the linear calibration slope coefficients, "intcal" to plot the linear calibration intercept coefficients, from the (nested) cross validation.

Power to which agreement is to be raised when the "gaussian" model is fit, i.e. 2 for R-square, 1 for correlation. Does not apply to type = "lasso".

y axis limits for model perforamnce plots, i.e. does not apply to type = "lasso". The ridge model may calibrate very poorly obscuring plots for type of "lincal" or "intcal", so one may specify the ylim value. If ylim is set to 1, then the program will derive a reasonable range for ylim. If ylim is set to 0, then the entire range for all models will be displayed. Does not apply to type = "lasso".

By default 1 to display using a spaghetti the performance as calculated from the individual folds, 0 to display using dots only the composite values calculated using all folds.

1 to include results for the untuned XGB model, 0 (default) to not include.

By default 1 to produce a plot, 0 to return the data used in the plot in the form of a list.

2 to track progress by printing to console, 0 (default) to not track.

A nested.glmnetr output object

determines what type of nested cross validation performance measures are plotted. Possible values are "devrat" to plot the deviance ratio, i.e. the fractional reduction in deviance relative to the null model deviance, "agree" to plot agreement in terms of concordance, correlation or R-square, "lincal" to plot the linear calibration slope coefficients, "intcal" to plot the linear calibration intercept coefficients, from the (nested) cross validation.

Power to which agreement is to be raised when the "gaussian" model is fit, i.e. 2 for R-square, 1 for correlation. Does not apply to type = "lasso".

y axis limits for model perforamnce plots, i.e. does not apply to type = "lasso". The ridge model may calibrate very poorly obscuring plots for type of "lincal" or "intcal", so one may specify the ylim value. If ylim is set to 1, then the program will derive a reasonable range for ylim. If ylim is set to 0, then the entire range for all models will be displayed. Does not apply to type = "lasso".

By default 1 to display using a spaghetti the performance as calculated from the individual folds, 0 to display using dots only the composite values calculated using all folds.

1 (default) to include results for the untuned XGB model, 0 to not include.

By default 1 to produce a plot, 0 to return the data used in the plot in the form of a list.

A cv.glmnetr (or nested.glmnetr) output object.

The predictor matrix. If NULL, then betas are provided.

A specific value of alpha for plotting. Used only when type is set to "elastic". Specifies which alpha is to be used for deviance plots. Default is "alpha.min", else must be an element of the alpha vector used in running the elastic net model. This can be reviewed using summary(fit) where fit is a nested.glmnetr() output object. Note, alpha is 1 for the lasso model and alpha is 0 for the ridge model.

The gamma value for choice of beta. If NULL, then gamma.min is used from the cross validated tuned relaxed model. We use the term gam instead of gamma as gamma usually denotes a vector in the package.

The lambda value for choice of beta. If NULL, then lambda.min is used from the cross validated tuned relaxed model. We use the term lam instead of lambda as lambda usually denotes a vector in the package.

type of model on which to base predictds. One of "lasso", "ridge" and "elastic" if elastic net model is fit.

Default of TRUE to write to console information on lam and gam selected for output. FALSE will suppress this write to console.

Additional arguments passed to the predict function.

A cv.glmnetr (or nested.glmnetr) output object.

The predictor matrix. If NULL, then betas are provided.

A specific value of alpha for plotting. Used only when type is set to "elastic". Specifies which alpha is to be used for deviance plots. Default is "alpha.min", else must be an element of the alpha vector used in running the elastic net model. This can be reviewed using summary(fit) where fit is a nested.glmnetr() output object. Note, alpha is 1 for the lasso model and alpha is 0 for the ridge model.

The gamma value for choice of beta. If NULL, then gamma.min is used from the cross validated tuned relaxed model. We use the term gam instead of gamma as gamma usually denotes a vector in the package.

The lambda value for choice of beta. If NULL, then lambda.min is used from the cross validated tuned relaxed model. We use the term lam instead of lambda as lambda usually denotes a vector in the package.

Default of TRUE to write to console information on lam and gam selected for output. FALSE will suppress this write to console.

Additional arguments passed to the predict function.

A cv.glmnetr (or nested.glmnetr) output object.

The predictor matrix. If NULL, then betas are provided.

A specific value of alpha for plotting. Used only when type is set to "elastic". Specifies which alpha is to be used for deviance plots. Default is "alpha.min", else must be an element of the alpha vector used in running the elastic net model. This can be reviewed using summary(fit) where fit is a nested.glmnetr() output object. Note, alpha is 1 for the lasso model and alpha is 0 for the ridge model.

The gamma value for choice of beta. If NULL, then gamma.min is used from the cross validated tuned relaxed model. We use the term gam instead of gamma as gamma usually denotes a vector in the package.

The lambda value for choice of beta. If NULL, then lambda.min is used from the cross validated tuned relaxed model. We use the term lam instead of lambda as lambda usually denotes a vector in the package.

Default of TRUE to write to console information on lam and gam selected for output. FALSE will suppress this write to console.

Additional arguments passed to the predict function.

cv.stepreg() output object

dataset for predictions. Must have the same columns as the input predictor matrix in the call to cv.stepreg().

pass through parameters

A nested.glmnetr output object.

The predictor matrix. If NULL, then betas are provided.

A specific value of alpha for plotting. Used only when type is set to "elastic". Specifies which alpha is to be used for deviance plots. Default is "alpha.min", else must be an element of the alpha vector used in running the elastic net model. This can be reviewed using summary(fit) where fit is a nested.glmnetr() output object. Note, alpha is 1 for the lasso model and alpha is 0 for the ridge model.

The gamma value for choice of beta. If NULL, then gamma.min is used from the cross validation informed relaxed model. We use the term gam instead of gamma as gamma usually denotes a vector in the package.

The lambda value for choice of beta. If NULL, then lambda.min is used from the cross validation informed relaxed model. We use the term lam instead of lambda as lambda usually denotes a vector in the package.

type of model on which to base predictds. One of "lasso", "ridge" and "elastic" if elastic net model is fit.

Default of TRUE to write to console information on lam and gam selected for output. FALSE will suppress this write to console.

Additional arguments passed to the predict function.

a output object from the nested.glmnetr() function

new data of the same form used as input to nested.glmnetr()

ANN model entry an integer from 1 to 5 indicating which "lasso informed" ANN is to be used for calculations. The number corresponds to the position of the ensemble input from the nested.glmnetr() call. The model must already be fit to calculate predicteds: 1 for ensemble[1] = 1, for model based upon raw data ; 2 for ensemble[2] = 1, raw data plus lasso predicteds as a predictor variable (features) ; 4 for ensemble[3] = 1, raw data plus lasso predicteds and initial weights corresponding to offset and allowed to update ; 5 for ensemble[4] = 1, raw data plus lasso predicteds and initial weights corresponding to offset and not allowed to updated ; 6 for ensemble[5] = 1, nonzero relaxed lasso terms ; 7 for ensemble[6] = 1, nonzero relaxed lasso terms plus lasso predicteds as a predictor variable (features) ; 8 for ensemble[7] = 1, nonzero relaxed lasso terms plus lasso predicteds with initial weights corresponding to offset and allowed to update ; 9 for ensemble[8] = 1, nonzero relaxed lasso terms plus lasso predicteds with initial weights corresponding to offset and not allowed to update.

a nested.glmnetr() output object.

additional pass through inputs for the print function.

output from an orf_tune() function

optional pass through parameters to pass to print.orf()

output from an rf_tune() function

optional pass through parameters to pass to print.rfsrc()

A nested.glmnetr() output object

Same xs used as input to ntested.glmnetr() for input object.

Same y_ used as input to ntested.glmnetr() for input object.

Same event used as input to ntested.glmnetr() for input object.

Same type used as input to ntested.glmnetr() for input object.

A nested.glmnetr() output object

Same xs used as input to ntested.glmnetr() for input object.

Same y_ used as input to ntested.glmnetr() for input object.

Same event used as input to ntested.glmnetr() for input object.

Same type used as input to ntested.glmnetr() for input object.

A nested.glmnetr() output object

Same xs used as input to ntested.glmnetr() for input object.

Same y_ used as input to ntested.glmnetr() for input object.

Same event used as input to ntested.glmnetr() for input object.

Same type used as input to ntested.glmnetr() for input object.

1 (default) to derive the tuned model like with xgb.tuned(), 0 to derive the basic models like with xgb.simple().

predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

an optional vector of start times in case of a Cox model. Class numeric of length same as number of patients (n)

dependent variable as a vector: time, or stop time for Cox model, Y_ 0 or 1 for binomial (logistic), numeric for gaussian. Must be a vector of length same as number of sample size.

event indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size.

model family, "cox", "binomial" or "gaussian" (default)

a vector (numeric) of values to search over for optimization of the Random Forest fit. This if for the mtry input variable of the rfsrc() program specifying the number of terms to consider in each step of teh Random Forest fit.

a vector (numeric) of 2 values, the first for the number of forests (ntree from rfsrc()) to use when searhcing for a better bit and the second to use when fitting the final model. More trees should give a better fit but require more computations and storage for the final. model.

This nsplit of rfsrc(), a non-negative integer for the number of random splits for a predictor.

a seed for set.seed() so one can reproduce the model fit. If NULL the program will generate a random seed. Whether specified or NULL, the seed is stored in the output object for future reference. Note, for the default this randomly generated seed depends on the seed in memory at that time so will depend on any calls of set.seed prior to the call of this function.

1 to output a brief summary of the final selected model, 2 to output a brief summary on each model fit in search of a better model or 0 (default) to not output this information.

a summary data frame from summary.nested.glmnetr() obtained using the option table=0

the minimum number of decimals to display the elements of the data frame

1 (default) if the summdf object is a summary for an analysis including nested cross validation, 0 if only the full data models were fit.

predictor input - an n by p matrix, where n (rows) is sample size, and p (columns) the number of predictors. Must be in matrix form for complete data, no NA's, no Inf's, etc., and not a data frame.

start time, Cox model only - class numeric of length same as number of patients (n)

output vector: time, or stop time for Cox model, y_st 0 or 1 for binomal (logistic), numeric for gaussian. Must be a vector of length same as number of sample size.

event_st indicator, 1 for event, 0 for census, Cox model only. Must be a numeric vector of length same as sample size.

number of steps done in stepwise regression fitting

method for choosing model in stepwise procedure, "loglik" or "concordance". Other procedures use the "loglik".

model family, "cox", "binomial" or "gaussian"

1 to output stepwise fit program, 0 (default) to suppress

a nested.glmnetr() output object.

TRUE to also print out the fully penalized lasso beta, else FALSE to suppress.

By default (orderall=FALSE) the order terms enter into the lasso model is given for the number of terms that enter in lasso minimizing loss model. If orderall=TRUE then all terms that are included in any lasso fit are described.

once of c("lasso", "elastic", "ridge") to select for summarizing, with default of "lasso".

Additional arguments passed to the summary function.

a nested.glmnetr() output object.

one of c("lasso", "elastic", "ridge") to select for summarizing, with default of "lasso".

TRUE to also print out the fully penalized lasso beta, else FALSE to suppress.

By default (orderall=FALSE) the order terms enter into the lasso model is given for the number of terms that enter in lasso minimizing loss model. If orderall=TRUE then all terms that are included in any lasso fit are described.

beta values less than betatol are set to 0 when they are approaching the rounding error of the machine architecture. Default is set to 1e-14.

Additional arguments passed to the summary function.

A cv.stepreg() output object

Additional arguments passed to the summary function.

a nested.glmnetr() output object.

default of FALSE to summarize fit of a cross validation informed relaxed lasso model fit, inferred by nested cross validation. Option of TRUE will describe the cross validation informed relaxed lasso model itself.

When cvfit is TRUE, one of c("lasso", "elastic", "ridge") to select for summarizing, with default of "lasso".

the power to which the average of correlations is to be raised. Only applies to the "gaussian" model. Default is 2 to yield R-square but can be on to show correlations. Pow is ignored for the family of "cox" and "binomial".

TRUE to also print out the fully penalized lasso beta, else to suppress. Only applies to cvfit=TRUE.

digits for printing of deviances, linear calibration coefficients and agreement (concordances and R-squares).

1 to print call used in generation of the object, 0 or NULL to not print

0 (default) to not include summary for 1se lasso fits in tables, 1 to include

1 to print table to console, 0 to output the tabled information to a data frame

1 to print tuning parameters, 0 (default) to not print

character width of the text body preceding the performance measures which can be adjusted between 60 and 120.

1 print performance statistics for lasso models calibrated on training data, 2 to print performance statistics for lasso and random forest models calibrated on training data, 0 (default) to not print. Note, despite any intuitive appeal these training data calibrated models may sometimes do rather poorly.

Additional arguments passed to the summary function.

output from an rf_tune() function

optional pass through parameters to pass to summary.orsf()

output from an rf_tune() function

optional pass through parameters to pass to summary.rfsrc()

A stepreg() output object

Additional arguments passed to the summary function.

The data to be used for training the XGBoost model

for now just "gbtree" (default)

one of "survival:cox" (default), "binary:logistic" or "reg:squarederror"

one of "cox-nloglik" (default), "auc", "rmse" or NULL. Default of NULL will select an appropriate value based upon the objective value.

whether the eval_metric is to be minimized or maximized

a seed for set.seed() to assure one can get the same results twice. If NULL the program will generate a random seed. Whether specified or NULL, the seed is stored in the output object for future reference.

an optional list where each element is a vector of indexes for a test fold. Default is NULL. If specified then doxgb$nfold is ignored as in xgb.cv().

a list with parameters for passed to xgb.cv() including $nfold, $nrounds, and $early_stopping_rounds. If not provided defaults will be used. Defaults can be seen form the output object$doxgb element, again a list. In case not NULL, the seed and folds option values override the $seed and $folds values in doxgb.

0 (default) to not track progress, 2 to track progress.

The data to be used for training the XGBoost model

for now just "gbtree" (default)

one of "survival:cox" (default), "binary:logistic" or "reg:squarederror"

one of "cox-nloglik" (default), "auc" or "rmse",

whether the eval_metric is to be minimized or maximized

a seed for set.seed() to assure one can get the same results twice. If NULL the program will generate a random seed. Whether specified or NULL, the seed is stored in the output object for future reference.

an optional list where each element is a vector of indeces for a test fold. Default is NULL. If specified then nfold is ignored a la xgb.cv().

A list specifying how the program is to do the xgb tune and fit. The list can have elements $nfold, $nrounds, and $early_stopping_rounds, each numerical values of length 1, $folds, a list as used by xgb.cv() do identify folds for cross validation, and $eta, $gamma, $max_depth, $min_child_weight, $colsample_bytree, $lambda, $alpha and $subsample, each a numeric of length 2 giving the lower and upper values for the respective tuning parameter. The meaning of these terms is as in 'xgboost' xgb.train(). If not provided defaults will be used. Defaults can be seen from the output object$doxgb element, again a list. In case not NULL, the seed and folds option values override the $seed and $folds values.

0 (default) to not track progress, 2 to track progress.