This project aims to create an easy to use interface for monitoring and controlling the injection process of nanomaterial ink onto a liquid surface (also known as Langmuir deposition) and to turn off the injection once the saturation point is reached. The idea is to observe relative color changes as timeseries data (in HSV space: hue, saturation, brightness/value) to find the point of saturation. The reference frame should be captured before the injection starts or at an early stage.
The interface consists of a webserver that currently performs the following tasks:
- capture a real-time camera feed. Supported are webcams, Raspberry Pi cameras as well as industrial cameras from IDS
- perform edge detection and temporally stable ellipse fitting to find the region of interest: in our case this is a cylindrical container
- capture a reference frame and calculate the (smoothed) relative HSV data for each subsequently captured frame (and their first and second derivatives). Alternatively, plotting the mean differences between frames directly is also supported
The user interface exposes the following functionality:
- Select the camera device to use
- If IDS cameras are selected, expose camera settings for exposure, gamma and white balance corrections
- Change the amount of temporal smoothing applied to the masking ellipse
- Set a masking margin (range 0-1) to discard color changes at the border of the ellipse
- Select which of the captured and calculated timeseries to plot (available as individual channels: raw HSV, smoothed HSV, first and second derivatives)
- Select the history size to plot
- Select threshold profiles from the file
profiles.csv. Threshold conditions in the CSV file can be sparsely populated. If all threshold conditions are met, alert the user visually with a popup. This uses the smoothed values for threshold checking, not the raw values that could fluctuate a lot - Set an alpha value for decay smoothing. This is a destructive operation and only changes for data points collected after the value was updated
- Set a sliding window smoothing range that is applied to the already smoothed data. First derivatives use this factor x2, second derivatives use x4
- Optionally record the currently streamed frames into a video that will be saved to
./recordings - Load a previously recorded video that will replace the current camera stream. After the video finishes, the analysis is paused, and the user can switch back to the camera stream.
Note
This is a work in progress. There will be bugs. Some things will not work as intended. Feel free to poke around in the code and submit a PR if you want to help fix things.
Screenshot of the Gradio-based Web UI.
- Clone this project with
git clone https://github.com/smlpt/deposition-detection - Ensure that a webcam or IDS camera is connected to your computer
- When using IDS cameras: ensure you installed the correct drivers (easily done with the extended setup for your model)
- On Windows: launch
launch_server_windows.batto launch the server. This will also automatically set up a virtual environment with the required packages on first launch. On Linux, make the shell script executable by runningchmod +x run_project.shand then launch./launch_server_linux.sh. - Open
http://localhost:7860/in a browser to access the interface - To stop the server, hit the "close" button in the GUI, or press Ctrl+C in the console. You can then close the browser tab.
- Clone this project with
git clone https://github.com/smlpt/deposition-detection - Ensure that either a webcam or the Raspberry Pi camera is connected to your Raspberry Pi
- Make the shell script executable by running
chmod +x run_project.shand then launch./launch_server_linux.sh. This will also automatically set up a virtual environment with the required packages on first launch. - Open
http://<raspberry-pi-ip>:7860in a browser to access the interface. To find the IP address of the Raspberry Pi, you either have to go looking in your Router, or you connect a display and a keyboard and find the IP viaifconfig. - To stop the server, hit the "close" button in the GUI, or press Ctrl+C in the console.
- Add toggle to switch between using the mean differences to the previous frames directly for analysis vs. using reference frames
- Add toggle to use absolute differences for analysis
- Update offline analysis notebook to include these optional toggles
- Detect IDS cameras when any is connected, otherwise resort to normal webcams
- Expose UI parameters for exposure and gamma correction as well as automatic and custom white balance settings when an IDS camera is selected
- If you already have the repository cloned, update with
git pulland delete the.venvdirectory. The launcher will rebuild it on the next start and include the necessary IDS libraries - added
ids_test.py, a standalone script for quickly testing connected IDS cameras - added
offline_analysis.ipynb, a notebook for offline analysis of existing recordings. If you have large raw videos (AVI), you might want to consider downscaling and re-encoding them to MP4/H.264 (e.g. with Handbrake) for much faster frame analysis
- Allow the user to record videos
- Load existing videos and stream them instead of the camera feed
- Expose more parameters to the UI (decay smoothing, sliding window smoothing, temporal ellipse smoothing factor, ellipse margin)
- Apply sliding window smoothing to the recorded values and to the first and second derivatives
- Timestamp logging and threshold checking now use the smoothed values
- Hid the manual exposure controls for now, as they were only available in Linux and for Logitch C920 webcams
- Adds a profile.csv file to load sparse threshold data for different materials (filled with dummy data right now)
- Plot thresholds with the corresponding timeseries
- Monitor threshold conditions and alert the user when all conditions are met
- Adds an inner ellipse 90% the size of the original ellipse. This inner ellipse is used for calculations to prevent minor pixel shifts/errors around the mask contour
- Adds a log button that outputs the values of the current timestamp for all existing timeseries
- Masks are now temporarily stable by adding the distance to the previous detection to the score function
- Masks transition smoothly between frames
- Calculate first and second derivatives of each H/S/V channel
- Dropdown to select which channels to plot
- Real-time webcam capturing
- Edge detection and basic ellipse fitting
- HSV plotting for relative changes
- Export to CSV
- Manual exposure/WB settings
Make sure your browser allows pop-ups.
Check in the ./recordings folder.
Have you installed the drivers correctly? Double-check you installed them for the correct model. Restart your computer, just in case. If that doesn't help and you get any error messages, open an issue in this repository.