LabGym for Resident-Intruder Assay

[ako-alt]

Hi,

First off, I appreciate your help with the head-twitch response question I asked earlier -- I'm still working on refining it now. I'm currently working on developing a Categorizer for a resident-intruder assay and just wanted to ask for some advice before starting the process.

One problem is that the resident and intruder mouse look very similar to each other! The intruder has a black tail mark which isn't always apparent (it's hard for me to tell sometimes). Here's a sample image, where I boosted the brightness of the image x3 using the preprocessing module.

I want to train a Detector, should I annotate both as a single "mouse" class or split into "intruder" and "resident?" How many images would I need to annotate?

Furthermore, what do you recommend for the next steps. For the behavioral examples, I want to mainly track "aggressive attacks," which are not super common in my dataset (maybe 80 total examples). At the same time, these are very easy examples to see and annotate. I also want to track other behaviors like "dominance," "pursuit," and "social behavior." Also, I only care about interactions where the "resident" does something to the "intruder," not the other way around. How long should the generated behavioral samples be to best capture all these various types of behavior?

Lastly, what should I do when I train the categorizer? Are there any considerations I should make before training and running the analyze videos?

[yujiahu415]

Hi,

You're very welcome!

For this assay, since you only care about the behaviors of one mouse (the resident), you can use the 'interactive basic' mode that always analyzes the two mice as one, but the process is much simpler than the 'interactive advanced' mode that analyzes each individual and distinguish different social roles of different individuals.

In the 'interactive basic' mode, you don't need to worry about identity switching since the two mice are analyzed together all the time. So, the Detector training is very simple: you can select at least 50 frames (100 is recommended) for annotation. These 50 images should ideally from 50 different videos so that the Detector can see as many scenarios of different video recordings as possible. These 50 images should represent different scenarios of mouse locations, poses, whether or not having body contact, etc. Basically, provide the Detector as diverse scenarios as possible to let it be aware of different scenes during analysis. Use EZannot to annotate these images, use single class 'mouse' for both mice. You may not need to include the tail when annotating the mouse's body if the tail cannot be automatically detected with EZannot's AI help. But if the tail is important for identifying some of the behaviors of your interest, you'll need to include it in the annotation in each image when the tail is present. When the two mice have close body contact, you can still annotate them as two 'mouse', but if one mouse is completely (or over half of the body) occluded by the other, just annotate one 'mouse'. Choose all the augmentation methods in EZannot and export a training dataset to train a Detector. If the frame size of your videos is something like 480 X 720, use 720 as the inferencing frame size when training the Detector, and set iteration number to 20,000.

Then generate the behavior examples using the trained Detector and the 'interactive basic' mode. Not sure what the fps of your videos. If the fps is higher than 30, downsampling it to 30 or even 15 using the preprocessing module of LabGym. And use 0.5 second (15 frames if fps is 30; 8 frames if fps is 15) as the duration for each behavior example. Set the interval of two consecutively generated examples as duration of one behavior example (like 15 if the duration of behavior example is set to 15). Don't include background or body parts in behavior examples, set background to white.

When you sort behavior examples, select the most representative ones for each behavior category, and make sure the select ones can cover the majority of diverse scenarios, like mice in different locations, orientations, postures, etc. of each behavior. For each behavior, recommend having at least 200 (400 individual files) pairs of examples, but for rare behaviors that are easy to distinguish, it's fine to have 80 pairs. But the selected examples should not be redundant, meaning that every pair should look different from others.

We can talk more about the training settings after you have the behavior examples generated. Before sorting examples, show me some of your examples so that I can take a look to see if they look good. I don't want your sorting effort to be wasted.

[yujiahu415]

On a second thought, when you annotate the mice, when they have close body contact that is hard for the EZannot's AI to determine the boundaries between the two mouse bodies, just annotate the whole blob as one 'mouse'. Only annotate two 'mouse' when the EZannot's AI can easily determine the boundaries.

[ako-alt]

Hi,

I just finished annotating 100 images with different body poses, videos, etc in EZAnnot and applying all the image augmentations. I had a quick question about your point about the frame size. My videos have different frame sizes, for example one has dimensions of 128 pixels x 352 pixels, while another has dimensions of 142 pixels x 370 pixels. Should I still annotate all of these together? What should I use for the inferencing frame size?

Likewise, I had a question about the annotations (and my video dataset in particular). I have some videos where, on occasion, the camera shifts and you might see a mouse in another box coming into view. The actual original videos have 4 boxes in one video, and I cropped these into four different videos covering each of the boxes. But sometimes, the boxes will be moved slightly in the middle of the recording, so you might see in one box the 2 mice of interest, but another mouse might pop into view briefly (but completely physically separate from the two mice of interest). How should I handle these scenarios?

Let me know if I can provide more clarification and thank you for your help!

[yujiahu415]

Hi,

Yes, you can annotate all of the images together and use 370 as the inferencing frame size.

You can include some non-redundant images of the scenario where a mouse might be in another box coming into view, and annotate three 'mouse' in those images.

[yujiahu415]

Hi,

Happy new year!

Thank you for sending me the annotated dataset. Your annotations are mostly good. One minor thing that can improve the Detector accuracy is that when two mice have close body contact, make sure the border of the annotation for each mouse does not cross over the other. Otherwise, the Detector might have the problem that your previous error message showed.

Since there are too many ID switching when using interactive advanced mode, I suggest you proceed with the interactive basic mode. Generally, the Detector accuracy increases with the resolution of the video. So to minimize ID switching to a manageable extent, you need to make the video resolution sufficient, like 640 X 640, while your current video resolution is way below that. LabGym currently doesn't have an option for users to make different individuals different colors.

For interactive mode, and for mouse behavior, we generally use less than 1 second (30 frames in your case) as the duration of one behavior example. 1.5 second (45 frames) might also work but I can imagine that this might increase the chance that one example includes multiple behaviors, like you said, "some episodes were getting split into 2-3 mini-episodes"? So I suggest you try shorter duration like 30 frames to see if it gives better results.

And you said the aggression could be confused with chasing. One of the easy solutions is, as you mentioned, include a new category "chasing" and sort some chasing examples into it. Make the duration of behavior example shorter may also improve this.

As for the Categorizer complexity, since your behavior is simple, your category system is simple (only two, or three if including chasing), and the number of your training examples is low (like ~100 examples per category), you should make the Categorizer simple. So, level 4 is the highest complexity you can try. Level 5 is too complex. 64 as input size is already sufficient, and you can try lower, like 32, especially for Animation Analyzer. So I suggest your try these settings of the Categorizer and compare their training performance (by looking at the val_loss, the smaller val_loss, the better):

1. Animation Analyzer (AA) LV3 input 32 + Pattern Recognizer (PR) LV3 input 32
2. AA LV4 input 64 + PR LV1 input 8
3. PR only LV4 input 64

Setting 1 is equal weight on AA and PR, 2 has more weight on AA, and 3 only uses PR. This trial aims to determine which path to go for the setting, and then you can fine tune the setting along that path.

Let me know how it goes.

[ako-alt]

Hi,

Happy new year to you too! I followed your suggestions of adding a category for "chasing," using a lower frame duration (of 30 frames), and training the three models that you mentioned.

Of the 3, it seems like setting 2 was the best (the val_loss was 0.4715 compared to 0.6728 for option 1 and 0.6376 for option 2). More importantly, I looked at the analyzed videos to see if there was a difference. I've noticed the following things:

First, I went into the "mouse_probability" file in the "aggression" subfolder to see what the probabilities were. And I noticed that all of the true aggressive events were marked with very high probabilities (often 0.8 or higher), so it seems like the Categorizer can discern between aggression and other behaviors pretty well. However, my counts are not necessarily matching up with what I would expect (even if I remove the intruder attacking the resident counts). I noticed that very short episodes (~0.8 s) have a high associated probability but are not captured. My idea was to reduce the frame duration to 18-20 frames instead, since it's ok if there are false positives as long as all the true positives are captured.

The other thing I noticed is that larger episodes (anywhere from 3-6 seconds) are split into "mini-episodes," so the final count may say there were 3 aggressive attacks in one larger one. Is there a way in LabGym to "combine" these "mini-episodes," for example if only 1-2 low probability frames separated the two events? Or is that something I should do post-processing instead? I also think it's occurring because there may be a frame that looks like an "other" or "chasing" frame in the middle of the longer behavior. I don't think it's too difficult to fix this post-processing, but wondering if there's a way to fix this in LabGym.

Thanks for the help again, I truly appreciate it! I think it's close to being used for processing other videos, just want to fine tune some parameters.

[yujiahu415]

First, "count" is a less reliable parameter than "duration". For example, a behavior lasts for 20 frames and should be counted as once, and if any of the 20 frames is misclassified as another behavior, it will be disrupted and be counted as twice, and the error is big (1/1 proportion). In this case, duration is calculated as 20 frames/fps, and if one frame is misclassified, the duration is 19 frames/fps, and the error is small (1/20 proportion).

Yes, if you can, reduce the duration of a behavior example is highly recommended. In this way, the chance of one behavior example containing different behaviors can be reduced. I don't think a larger episode being split into mini-episodes is a problem, because if the behavior is continuous, like AAAAAAA, it will be counted as once, no matter how long it lasts. Only when it's 'AAABAAAA', it will be counted as twice. And if you use duration to quantify the behavior, this issue can be further mitigated.

The count of 'AAAAAA' and 'AAABAA' cannot be addressed by LabGym itself because misclassification can happen, and I cannot preset the minimum duration of a behavior in LabGym. But I have provided users an option to mitigate this, when you select a Categorizer, you will be asked whether to set a minimum duration/length of behavior to filter out continuous frames of a behavior that are shorter than the criteria. This can help you to filter out the potential misclassification that is brief, like the "B" in "AAABAA". You may also post-process the outputs by yourself.

Lastly, since option 2 gave you the best val_loss, you can fine tune the setting along that direction, like try these settings (or others as you wish):

1. AA LV4 input 32 + PR LV1 input 8
2. AA LV2 input 64 + PR LV1 input 8
3. AA LV4 input 96 + PR LV1 input 8
4. AA LV2 input 96 + PR LV1 input 8

And compare them.

[ako-alt]

Hi,

I wanted to let you know that the Categorizer seems very strong and is matching up very well with the manually validated results. I created a short Python script to do some post-processing (basically sorting examples into resident-initiated, intruder-initiated, and false positive behaviors). I need to add some more examples to the "other" category, experiment more with the behavior duration when running "analyze videos," and also experiment with settings biased towards the animation analyzer. But I am grateful for the help and believe this will cut down a lot of time. I had some questions on future resident-intruder assays that we decide to run.

Like you mentioned, this would be better if one of the mice was more clearly marked (would a tuft of red on their back work?). If I did decide to do an interactive advanced model in the future, would I retrain on those behavioral examples? Or would I be able to use this model, provided that the Detector doesn't switch the identities of the mice, and use the Spotlight feature to make sure only resident attacking intruder interactions are scored?

Also, if I decided to retrain a Categorizer in the future with more behaviors (ex: social behavior, dominance behaviors), what would you recommend the complexities of the AA and PR to be? In general, how do you decide to go to a higher level of complexity or a wider depth (since this would be useful for some of the other models that I am training).

Thank you again for all your support!

[yujiahu415]

Hi,

Thank you for letting me know this!

For your future recordings, a tuft of red on the mice back should work, but I would suggest you use white instead of red to maximize the contrast. And also, if possible, you may show me a picture of how the marker looks like, just for double check.

If you use interactive advanced mode to distinguish individual's behaviors, you will need to re-annotate frames to train a new Detector. You also need to re-generate behavior examples, sort them and train a new Categorizer. Basically, you'll need to re-do everything. But if you use markers to distinguish different individuals, both the annotation and analysis will be more efficient.

As for the AA and PR settings, they are purely based on experience. Different dataset requires different settings. I made LabGym this way to allow it to be fully customizable so that it can suit very different data. But at the same time, this could the tweaking of the settings more difficult, as you may need some trial and error to find the best setting. What I suggested previously was also based on my experience, since I have done similar analysis. That being said, there is some "golden rule" for setting the AA and PR. Generally, you can always start from simple settings, like AA LV2 input 16 + PR LV2 input 32 is a good start point and increase the complexity/input gradually and change one at a time, like AA LV2 input 16 + PR LV3 input 32 and compare the results. You can also do this for the input size, like AA LV2 input 16 + PR LV2 input 64, or AA LV2 input 32 + PR LV3 input 32. You can also compare Categorizers with more weight on AA or more weight on PR. For example, in your case, I previously suggested you compare the Categorizer with different weight, and you found that the one with more weight on AA performed the best.

Hopefully this helps. But you can always reach out to me if you have any questions. And it's always more efficient if you consult me about a setting that you're not sure about before you spend a lot of effort in annotation/sorting with an inappropriate setting.