question about external validation

[fbwg]

Thank you for developing the dynamicLM package. I built a cross-validated landmark model with this package, but I encounter some error in the external validation process:
Firstly, I split my data into training data and validation data.

lss<-filter(all,study == 1)
arcin<-filter(all,study == 0)

then, I used lss data to build model.

# super dataset 
#（1）the outcome columns 
#（2）variable types (fixed vs. time-varying)  
outcome <- list(time = "years", status = "event")  
covars<-list(fixed=c("gender",  "cigsmok", "bmi_g2","bmi_g3","bmi_g4", "edu_g2", 
                     "edu_g3","edu_g4","race_g2","race_g3","race_g4","diagchas","diagchro",
                     "diagcopd","diagemph","famlc"), 
                varying=c("age_dynamic","pkyr","tsle_smok","nod_g5","nod_g7","nod_g8",
                       "maoci","ground","nod_loc",
                       "nod_new","nod_nega","benign_nod","linbajie","wall",
                       "consolidation","nod_growth","reticular"))

w <- 3 			# w-year prediction window
lms <- seq(0, 2, by =0.5)# landmarks
lmdata <- stack_data(lss, outcome, lms,w,
                     covars, format = "long", rtime = "tstart", id = "pid") 

lmdata <- add_interactions(lmdata, func_covars = c("linear", "quadratic"),
                           func_lms= c("linear", "quadratic")) 

# Fit coefficient path
path <- pen_lm(lmdata, alpha = 1)

par(mfrow = c(1, 2))
plot(path, all_causes = TRUE)


# Fit penalized supermodel
# Cross-validated model
cv_model <- cv.pen_lm(lmdata, alpha = 1) 
## Fitting a cross-validaton penalized landmark supermodel
supermodel_pen_cv<- dynamic_lm(cv_model, lambda = "lambda.1se")

### Prediction
p_cv<- predict(supermodel_pen_cv)
# External validation with test data 
lmdata_arcin <- stack_data(arcin, outcome, lms,w,
                covars, format = "long", rtime = "tstart", id = "pid") 

lmdata_arcin<- add_interactions(lmdata_arcin, func_covars = c("linear", "quadratic"),
                           func_lms= c("linear", "quadratic"))
par(mfrow = c(1,1))
outlist_arcin <- calplot(list("Supermodel" = supermodel_pen_cv), 
                   cause = 1, 
                   data = lmdata_arcin$data, 
                   lms="LM",
                   method = "quantile", q = 10, 
                   ylim = c(0, 0.25), xlim = c(0, 0.25))

however, an error was reported for the “outlist_arcin “ as follows:

> outlist_arcin <- calplot(list("Supermodel" = supermodel_pen_cv), 
+                    cause = 1, 
+                    data = lmdata_arcin$data, 
+                    lms="LM",
+                    method = "quantile", q = 10, 
+                    ylim = c(0, 0.25), xlim = c(0, 0.25))
错误于sum(sapply(bet_covars, function(coef_name) {: 
  'type'(list)参数无效

Would you please have a check for the above codes at your convenience?
Thanks.

[fbwg]

when I changed the data=lmdata_arcin$data to data=lmdata_arcin
errors were also reported as follows:

> outlist_arcin <- calplot(list("Supermodel" = supermodel_pen_cv), 
+                    cause = 1, 
+                    data = lmdata_arcin, 
+                    lms="LM",
+                    method = "quantile", q = 10, 
+                    ylim = c(0, 0.25), xlim = c(0, 0.25))
# 错误于sum(sapply(bet_covars, function(coef_name) {: 
#   'type'(list)参数无效
# 此外: 警告信息:
# In (function (object, times, formula, data, lms, id_col = "ID",  :
#   Input is an LMdataframe, argument lms is ignored.

[anyafries]

Is it possible to share a reproducible piece of code? That way I can try to better understand the error.

In addition, could you try the following? Then we can try isolate from where the error stems.

# do the original data frames have the same types?
print(str(lss))
print(str(arcin))

# do lmdata and lmdata_arcin look the same?
print(lmdata)
print(lmdata_arcin)

# does the internal validation work?
outlist <- calplot(list("Supermodel" = p_cv), cause = 1, method = "quantile", q = 10)

# does the prediction on new data work?
p_test <- predict(supermodel_pen_cv, lmdata_arcin)
print(p_test)

# does the external validation work?
outlist_arcin <- calplot(list("Supermodel" = p_test), cause = 1, method = "quantile", q = 10)

[fbwg]

#Is it possible to share a reproducible piece of code? That way I can try to better understand the error.
Can I send the codes to you by email because it was relatively long.

do the original data frames have the same types?

I think the two original data had the same types because they are from one full data, and were selected by covariate “study”.
lss<-filter(nlst_all,study == 1)
arcin<-filter(nlst_all,study == 0)

print(str(lss))
tibble [47,519 × 73] (S3: tbl_df/tbl/data.frame)
…..
print(str(arcin))
tibble [25,564 × 73] (S3: tbl_df/tbl/data.frame)
…..

do lmdata and lmdata_arcin look the same?

Yes, they look the same.

does the internal validation work?

Errors were reported as follows:

par(mfrow = c(2, 2))
cal_iv<-calplot(list("Supermodel" =supermodel_pen_cv), method = "quantile", q = 10,

•             ylim = c(0, 0.25), xlim = c(0, 0.25), 
  

•             split.method = "bootcv", B = 10) 
  

错误于(function (object, times, formula, data, lms, id_col = "ID", :
Prediction object Supermodel does not have data stored. In order to bootstrap, objects of class penLMCSC must be fit with argument x=TRUE.Prediction object Supermodel does not have data stored. In order to bootstrap, objects of class dynamicLM must be fit with argument x=TRUE.

does the prediction on new data work?

No. Errors were reported as follows:

p_test <- predict(supermodel_pen_cv, lmdata_arcin)
错误于sum(sapply(bet_covars, function(coef_name) {:
'type'(list)参数无效

does the external validation work?

No. Errors were reported as follows:

par(mfrow = c(1,1))
outlist_arcin <- calplot(list("Supermodel" = supermodel_pen_cv),

•                cause = 1, 
  

•                data = lmdata_arcin, 
  

•                lms="LM",
  

•                method = "quantile", q = 10, 
  

•                ylim = c(0, 0.25), xlim = c(0, 0.25))
  

错误于sum(sapply(bet_covars, function(coef_name) {:
'type'(list)参数无效
此外: 警告信息:
In (function (object, times, formula, data, lms, id_col = "ID", :
Input is an LMdataframe, argument lms is ignored.

[anyafries]

(I have edited your comments slightly for formatting)

Thanks for sharing.

My email: afries [at] stanford [dot] edu