This repository supports the proceeding paper Accelerating Pinned Specimen Digitization: A Deep Learning Pipeline for Collaborative Robots.
It provides resources for training a YOLO model capable of identifying specimens, pins, and labels in cobot digitization workflow. The repository includes detailed steps to:
- Set up the environment
- Prepare the dataset
- Train and evaluate the model
- Visualize the results
Collaboration: This project was developed together with Natural History Museum, leveraging cutting-edge technology to advance museum digitization.
This experiment is designed to run on a platform with an NVIDIA GPU.
Recommended Environment:
- Python 3.10
- PyTorch 2.3.1+cu121
Installation Resources:
Clone this repository and the annotation tool used in this project:
git clone https://github.com/FrankZhangIronly/Pinned_Insect_CV.git
git clone https://github.com/syangeng/ISAT_with_segment_anything.gitDownload the insect image dataset from Zenodo.
If you need to create your own dataset, please use the ISAT tool for annotation and export the YOLO version annotation files.
- ISAT Tool Instructions: ISAT-SAM Documentation
Note: Scripts for splitting data into train and test sets are available in jupyter/create.ipynb.
Start by navigating to the root directory of the repository.
To modify mosaic augmentation from the default value of 4 to 9, follow these steps:
- Open the file located at:
YOUR_PYTHON_PATH/Python3.10/dist-packages/ultralytics/data/augment.py - In the function
__init__, change the parameternfrom 4 to 9.
Use the command below to start training:
# Display available parameter settings
python run.py train --help
# Run training with custom parameters
python run.py train --model_path <path_to_model.pt> --data <path_to_data.yaml> [other parameters]Use the command below to validate and test on datasets:
# Display available parameter settings
python run.py validate --help
# Run validation with custom parameters
python run.py validate --model_path <path_to_model.pt> --data <path_to_data1.yaml> <path_to_data2.yaml> ... --name <val_name1> <val_name2> ...YOLO will generate training visualizations in the project folder. You can view the progress using the plot_training script in jupyter/create.ipynb.
- Use the trained weights located at
best.ptin your project folder. - Test videos are stored in
dataset/videos/.
In jupyter/aug.ipynb:
- Use the code blocks labeled "Video1" for inference on videos.
- The second and third code blocks handle label extraction.
The final version of the Jupyter Notebook file, train_T4_mask.ipynb, has been updated. It contains all the code for the experiment mentioned in the paper.
current detection and segmentation result on test video:
- create utils folders for self-defined tools
- build augmentation and visulization (boxes) tools
Develop CV and ML models to aid in the digitization of over 25 million pinned insect specimens.
The specific scenario we aim to address is assisting robotic arms in identifying specimens within drawers and locating the pins to grasp the target.
- Develop models for pinned insect identification. (Pre-training)
- Develop video datasets for pinned insect detection in drawers.
- Develop models for detection in drawers based on identification models. (Fine-tuning)
- Integrate additional modules for segmentation of insect.
Models are evaluated on COCO val2017 dataset.
| Model | mAPval 50-95 |
params (M) |
FLOPs (B) |
|---|---|---|---|
| YOLOv8n | 37.3 | 3.2 | 8.7 |
| YOLOv8s | 44.9 | 11.2 | 28.6 |
| YOLOv8m | 50.2 | 25.9 | 78.9 |
| YOLOv8l | 52.9 | 43.7 | 165.2 |
| YOLOv8x | 53.9 | 68.2 | 257.8 |
| RT-DETR-R18 | 46.5 | 20 | 60 |
| RT-DETR-R34 | 48.9 | 31 | 92 |
| RT-DETR-R50-m | 51.3 | 36 | 100 |
| RT-DETR-R50 | 53.1 | 42 | 136 |
| RT-DETR-R101 | 54.3 | 76 | 259 |