From 4b77b68fd89d8801331cc3f2c27d825776316aa1 Mon Sep 17 00:00:00 2001 From: RichardScottOZ Date: Mon, 25 Mar 2024 22:00:06 +1030 Subject: [PATCH 1/2] minor edits --- notebooks/cnn_data_sampler.ipynb | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/notebooks/cnn_data_sampler.ipynb b/notebooks/cnn_data_sampler.ipynb index 5d1b4f71..e27c6058 100644 --- a/notebooks/cnn_data_sampler.ipynb +++ b/notebooks/cnn_data_sampler.ipynb @@ -21,14 +21,14 @@ "- The known occurrences are oversampled and negative samples are selected randomly.\n", " - This ensures that both sample types are represented equally in the created dataset.\n", "- The sampler works well with sparse datasets, where it is relatively rare that the negative samples would contain an unkown occurence\n", - " - Thus there shouldn't be too much misclassifications in the data that would cause problem\n", + " - Thus there shouldn't be too many misclassifications in the data that would cause problems\n", "- User has the option to set buffer size for the known occurences, because one pixel locations are harder to locate for model than .\n", "### TODO\n", - "- Current implementation creates images in order, known occurences first and non-occurences last. This makes suffling necessary at the model script, which complicates things. Suffling should be moved to this script\n", - "- This sampling script should be integrated to the cnn scripts so that leave one out cross validation would be easy to do\n", - " - The leave one out method should be implented in a way that one feature is completely removed from the occurence data before any sampling is done, and then test data is creaed by sampling the left out feature alone\n", + "- Current implementation creates images in order, known occurences first and non-occurences last. This makes shuffling necessary in the model script, which complicates things. Shuffling should be moved to this script\n", + "- This sampling script should be integrated in the cnn scripts so that leave one out cross validation would be easy to do\n", + " - The leave one out method should be implented in a way that one feature is completely removed from the occurence data before any sampling is done, and then test data is created by sampling the left out feature alone\n", " - without this it is hard to assess level of overfitting\n", - " - users should be given a clear warning about htis, if the script is published without these changes. Overfitting is nearly unavoidable with low amounts of data so user should be given tools to asess the level of overfitting" + " - users should be given a clear warning about this, if the script is published without these changes. Overfitting is nearly unavoidable with low amounts of data so user should be given tools to asess the level of overfitting" ] }, { @@ -68,7 +68,7 @@ "outputs": [], "source": [ "def get_features_json(gdf):\n", - " \"\"\"Parses features from GeoDataFrame in such a manner that rasterio wants them\n", + " \"\"\"Parses features from a GeoDataFrame as rasterio requires\n", " \"\"\"\n", " return [json.loads(gdf.to_json())['geometry']]\n", "\n", From bd346a08f31c93fc45223f977ee78f57c5f90e36 Mon Sep 17 00:00:00 2001 From: RichardScottOZ Date: Mon, 25 Mar 2024 22:07:33 +1030 Subject: [PATCH 2/2] minor edits --- notebooks/Mini_Unet_with_uncertainty.ipynb | 26 +++++++++++----------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/notebooks/Mini_Unet_with_uncertainty.ipynb b/notebooks/Mini_Unet_with_uncertainty.ipynb index be302c3f..5479b0d7 100644 --- a/notebooks/Mini_Unet_with_uncertainty.ipynb +++ b/notebooks/Mini_Unet_with_uncertainty.ipynb @@ -7,12 +7,12 @@ "source": [ "# Mini U-net with learned uncertainty estimation\n", "- Semantic segmentation for (small) images, with included learned uncertainty estimation\n", - " - There is separate script for semantic segmentation without uncertainty estimation\n", + " - There is a separate script for semantic segmentation without uncertainty estimation\n", "- The idea is based on\n", " - DeVries, T., and G. W. Taylor. 2018b. “Leveraging Uncertainty Estimates for Predicting Segmentation Quality. https://arxiv.org/abs/1807.00502\n", "- This implementation takes strong inspiration from\n", " - Wu, S., Heitzler, M. & Hurni, L. (2022) ‘Leveraging uncertainty estimation and spatial pyramid pooling for extracting hydrological features from scanned historical topographic maps’, GIScience & Remote Sensing, 59(1), pp. 200–214. Available at: https://doi.org/10.1080/15481603.2021.2023840.\n", - "- Loss based on (Sørensen -)Dice coefficient is utilized due to favorable properties related to unbalanced pixel counts.\n", + "- Loss based on (Sørensen -) Dice coefficient is utilized due to favorable properties related to unbalanced pixel counts.\n", " - Milletari, F., Navab, N. & Ahmadi, S.-A. (2016) ‘V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation’, in 2016 Fourth International Conference on 3D Vision (3DV). 2016 Fourth International Conference on 3D Vision (3DV), pp. 565–571. Available at: https://doi.org/10.1109/3DV.2016.79.\n", " \n", "\n", @@ -20,25 +20,25 @@ "- In addition to segmentation, the model is asked to create certainty estimation\n", "- When overall certainty is high (close to 1) the loss function operates as if no certainty estimate was used \n", "- When certainty values dip below 1, the model is punished less for making mistakes at those pixels.\n", - " - When certainty is close to zero at certain pixel, the model segmentation output does not matter at all for the loss at that pixel\n", - "- To prevent model predicting high uncertainty everywhere, it is punished for being uncertain at large areas\n", - " - This is done by averaging logarithm of the uncertainties at each pixel * -1 together\n", - " - This regularization loss is multiplied with a hyperparameter called uncertainty coefficient\n", - " - The value of the uncertainty coefficient has a large effect on the results, and finding suitable value should be automatised (not yet implemented)\n", - " - Too small values result in poor segmentation results\n", - " - Too large values result in abolute certainty everywhere\n", - " - Suitable value creates resulting values between full range of zero and one for both segmentation prediction and certainty estimate\n", + " - When certainty is close to zero at a certain pixel, the model segmentation output does not matter at all for the loss at that pixel\n", + "- To prevent model predicting high uncertainty everywhere, it is punished for being uncertain in large areas\n", + " - This is done by averaging the logarithm of the uncertainties at each pixel * -1 together\n", + " - This regularization loss is multiplied with a hyperparameter called the uncertainty coefficient\n", + " - The value of the uncertainty coefficient has a large effect on the results, and finding a suitable value should be automatised (not yet implemented)\n", + " - Values that are too small result in poor segmentation results\n", + " - Values that are too large in absolute certainty everywhere\n", + " - Suitable value creates resulting values between the full range of zero and one for both segmentation predictions and certainty estimates\n", " - Thus closer (max cert - min cert) + (max pred - min pred) is to to 2, the better the uncertainty coefficient. This could be used as a basis for optimisation\n", "\n", "### Notes\n", "- Current implementation is experimental and needs further optimization, testing and functionality \n", " - IMPORTANT The possibility to input a large geotiff should be implemented. In that case the trained model should be used to predict the whole area in a sliding window type of setup. Then it should be parsed back together to create output prediction and uncertainty estimate geotiffs for the whole area. This tool should not be included in the toolkit before that can be done.\n", " - hyperparameter tuning should be automatized\n", - " - Dice coefficient and Intersection over Union should be implemented as metric to measure the goodness of model\n", + " - Dice coefficient and Intersection over Union should be implemented as a metric to measure the goodness of model\n", " - custom metrics are needed due to the unusual shape of the output\n", - " - Data suffling should maybe moved to data sampling side\n", + " - Data shuffling should maybe be moved to data sampling side\n", " - Leave one out cross validation should also be implemented and integrated with data sampling, so that user has reliable tools to estimate possible (likely) overfitting\n", - " - current implementation with data sampler does not create reliable test data, because one occurence features multiple times in data with different croppings.\n", + " - The current implementation with data sampler does not create reliable test data, because one occurence features multiple times in data with different croppings.\n", " - The other option is to take out all the times one of the features occurs before suffling and keep data sampling as is\n", "- The current implementation only works with binary classes (0 and 1)\n", "- Image size should be at least 32 x 32 pixels or larger power of 2 (or at least divisible three times with 2, after which size should be at least 3 x 3)\n",