The regularization parameter optimization is not working on self-measured data

[jianzuo]

Hi,
When computing on an experimental data
drt_check.csv, the regularization optimization methods provided can not be optimized.
I mean, selecting LC method, for example, the exported DRT results show lambda of the input value in the panel.
It seems that the regularization parameter can only be optimized by customized value in the current version.

Thank you very much for checking this issue!
This tool is really useful for my current work, I will cite all related work. Thanks again for making it open-sourced!

[ciuccislab]

Hello,

The regularization parameter selection method can automatically choose the reg. parameter optimally, or you can manually select the reg. parameter as you prefer. It's essential to note that the reg. parameter selection methods adapt its choice based on the data. In certain data situations, the selected lambda values (for each method) might be close to each other, resulting in less noticeable changes in the DRT curve compared to when parameters are selected arbitrarily. For instance, you could choose lambda values like 0.1, 0.01, 0.15, 0.025, etc. On the other hand, the parameter selection methods might choose values like 0.01, 0.011, 0.012, 0.05, and this might not exhibit significant changes in the DRT curve.

Therefore, regularization parameter methods aim to find a balance between over-smoothing and under-smoothing, a consideration that might be overlooked when manually selecting your own reg. parameter (lambda).

[jianzuo]

Thank you very much for the explanation.
I think there is a misunderstanding about this issue.
From my side, I just want to use the regularization parameter optimization function to present the optimized lambda, as you said it is better than manually chosen. But, for my data, I fail to obtain the optimized lambda.

I suspect there is a bug in regularization parameter optimization in the current version.
You may try some trials on the data I provide:

1. For instance, choose LC method, input 0.001 for 'Regularization Method', then in the exported DRT data, you will see the lambda exactly equals to 0.001. I tried several runs with different input values on the same data, the obtained (lambda) values are always exactly the same as the input ones.

I totally agree with what you have explained. But in my experiments, every time the results are exactly the same as the input ones, this seems strange.

[ebruffey]

I am also experiencing this issue where the lambda parameter is not being optimized. For example: I select the GCV method and I do not change the regularization parameter. I perform a simple run and the DRT gives me the optimized regularization parameter of 0.001, which just so happens to be the default value. I then change the regularization parameter to 1E-6 and I rerun the simple GCV. This time the optimized regularization parameter is 1E-6, the value that I unput therefore there is no optimization happening. Any help in fixing this issue would be greatly appreciative as we plan on using this tool for publications. Thank you.

[ciuccislab]

Dear Mr. Edward,

Thank you so much for your interest in pyDRTtools. Based on the latest code I sent to you, we have resolved the issue, as shown in the screenshot below. When you choose "custom" and set your own regularization parameter, "0.001" (which is the default) is returned as the optimal. However, if you select any method (such as GCV), "0.049787" is returned as the optimal, which is different from when the "custom" button is selected.

I would like to confirm whether you used the latest version I sent to you. You did not mention or acknowledge in your message whether you used the latest version with your data. We are very interested in receiving feedback on this.

Could you share your data with us so that we can look into the issue (and understand why your data might be an exception)? We have tested this latest version on many datasets (both synthetic and real) without encountering similar issues.

In the latest version, we will include a functionality where the optimal parameter will be outputted on the GUI, as this might make it easy for the users to locate the optimal reg. parameter value.

We look forward to your response and possibly receiving your data for testing. Thank you again for your message and for getting in touch.

Best regards,

[ciuccislab]

drt_check.csv

Hello Mr. Edward,

Using the latest version of pyDRTtools I shared with you, I just analyzed this data shared by Jianzuo. You can try to reproduce this yourself, while I am still waiting for your data. For L-curve (LC), 0.04853899 is optimally selected. Please check the following screenshots:

[ciuccislab]

For the GCV, 0.04978707 is optimally selected.

[ciuccislab]

Now if I select custom and set the insert your own reg. parameter of "0.001". This is the DRT plots, and your arbitrary value will be returned as the optimal. This can only happen if you select "custom". What custom does is to tell pyDRTtools to allow users to decide on their own reg. parameter (as they like), and the value arbitrarily chosen will be returned as shown in the screenshotted anaconda shell below

[ciuccislab]

To investigate your claim above, I use arbitrary reg. value of 1E-6 and I selected GCV. I got optimal regularization parameter "0.04978707 " (in contrary to your claim that pyDRTtools returned the same value of 1E-6 as the optimal). If pyDRTtool should return 1E-6 as the optimal, there should be under-regularization and the DRT curve should overfit.

[ciuccislab]

This is the case where pyDRTtools use the arbitrary value of "1E-6" inserted when using "custom" (as this can only work by first selecting "custom").

[ciuccislab]

We affirm that the latest code shared with you via mail works properly and has resolved the issue with optimal parameter selection or user-defined (custom) reg. parameter. In case all the illustrations above are not so clear, I will appreciate if you could share your data, like the user (Jianzuo) who shared his data with us here. Thank you so much for your interest in pyDRTtools and for getting in touch.

[jianzuo]

For the GCV, 0.04978707 is optimally selected.

Hi,
Sorry to jump into the discussion. I have downloaded the 'latest' DRTtools and run it on the data that I provided in this issue. From my side two things are strange:

1. The DRT results what I have got is different from the one you are showing (see below), and
2. Maybe you have added some print code to print the optimal lambda, but for me I just export the results to csv file and the lambda value being recorded is still 0.001.
   

[ebruffey]

Hello Mr. Francesco

Thank you for your detailed response. Unfortunately I have not received a new version from you or Maradresa, so I have not been able to test the new version. Maybe it didn't get sent or I didn't receive it for some reason. Please send it again and I will test it as soon as possible. I sent Maradresa an email with one of our data files for you to test on, but I think with the new version of pyDRTtools, the results will be as you have them here. Thank you so much for your time.

[ciuccislab]

Hello Mr. Edward,

Please see the screenshot below so that you can locate the day the latest version was sent your mail. I just tested with your data every works fine, and the reg. parameter is selected optimally. You can locate the latest version which was sent to you via mail on the 29/05/2024. Thank you so much for your interest in pyDRTtools, and for getting in touch.

[ciuccislab]

With your data, I selected L-curve and optimal reg. value of 0.01630123 was obtained.

[ciuccislab]

For the GCV, an optimal value of "0.04978707" was obtained.