Questions regarding implementation and evaluation code for StreamingBench

[aiclaudev]

Hi, thank you for your great work and for sharing the code. I have a couple of questions regarding the implementation and evaluation:

Q1. In the paper, it is mentioned that a small Decision module and a large Reaction module are used. However, in the current codebase, it seems that a single large model is only used with a silent head. Where can I find the implementation of small decision module?

Q2. I am trying to reproduce the results reported in the paper for the PO task of StreamingBench, but I am not getting consistent performance. Would it be possible for you to release the evaluation code used for StreamingBench, especially for the PO task?

Thank you in advance for your time and support.

[juancarlovieri]

I’m encountering the same issue. Tried to evaluate the online PO task using OVO-Bench but getting inconsistent results (no outputs and not following instructions). Will the code for online inference be released?

[aiclaudev]

@juancarlovieri

In my personal opinion, this is due to the absence of a decision module, which is crucial for proactive tasks.
I sincerely hope that a decision module will be released soon!

[juancarlovieri]

@aiclaudev

Hmm I had a skim through the code and found this:

Dispider/dispider/model/language_model/qwen_model/modeling_qwen.py

Lines 1959 to 1961 in 8b03eda

	full_qs = final_states[global_indicators==1] # nq c
	assert full_qs.shape[0] == 1
	is_silent = self.silent_head(full_qs)[:, 0]

Isn't this line supposed to be the decision head? But then again, it doesn't seem to be used anywhere else in the code other than loss calculation, so I guess they removed the decision module for the released inference code.

[aiclaudev]

@juancarlovieri

As you mentioned, it seems that the silent head is responsible for determining the timing of response generation (It might be a binary classifier). However, according to the paper, the decision module uses a smaller model than the response generator and I was unable to find out any checkpoints for this smaller decision module, which is mainly trained to decide the response time. Do you have any thoughts on this?

[juancarlovieri]

@aiclaudev

From what I understand, the checkpoint for the perception and decision module is under stream_compressor/, loaded by StreamGroundQwenForCausalLM. The final_states I referred to above seems to be the output of this smaller LM.

[aiclaudev]

@juancarlovieri

Oh, sincerely thanks for pointing out my misunderstanding. It does seem that stream_compressor refers to the checkpoint for decision module, and the confusion likely arose because is_silent is not being used.

I'll re-evaluate using StreamingBench after modifying the code. If I succeed in reproducing the results, I’ll be sure to let you know. Thank you again for your help.

[juancarlovieri]

@aiclaudev

Sure! I also tried using StreamingBench, and I get 25.6% time accuracy but 0% answer accuracy. Seems like it can understand contexts but it is not able to follow instructions well. Here is a snippet of one of the questions:

    {
        "time": "[0:00:00 - 0:00:10]",
        "questions": [
            {
                "task_type": "Proactive Output",
                "question": "When the lineup for Toronto FC is displayed, output \"Toronto FC lineup shown\".",
                "time_stamp": "00:00:03",
                "ground_truth_time_stamp": "00:00:08",
                "ground_truth_output": "Toronto FC lineup shown",
                "required_ability": "working memory",
                "Dispider": {
                    "answered": 4,
                    "dialog_history": [
                        {
                            "role": "user",
                            "content": "When the lineup for Toronto FC is displayed, output \"Toronto FC lineup shown\". Is it the right time to output \"Toronto FC lineup shown\"? You can only answer yes or no.",
                            "time": 4,
                            "cost": 2.1938986778259277
                        },
                        {
                            "role": "assistant",
                            "content": "Yes",
                            "time": 4,
                            "cost": 2.1938986778259277
                        },
                        {
                            "role": "user",
                            "content": "When the lineup for Toronto FC is displayed, output \"Toronto FC lineup shown\".",
                            "time": 4,
                            "cost": 2.799868106842041
                        },
                        {
                            "role": "assistant",
                            "content": "The event happens in zero - five seconds.",
                            "time": 4,
                            "cost": 2.799868106842041
                        }
                    ]
                }
            }
        ],
        "video_categories": "football",
        "video_path": "/home/juan/StreamingBench/src/data/videos/sample_44_proactive.mp4"
    },

Looks like it can identify the event (it outputs yes), but instead of outputting "Toronto FC lineup shown", it outputs some gibberish explanation. Maybe @Mark12Ding can help us with this? Thanks!

[aiclaudev]

@juancarlovieri

Thank you for sharing your results — they seem very helpful to me. As far as I remember, the results I obtained from my previous experiments were quite similar to yours, but unfortunately, I’m currently unable to locate the corresponding output file. I will wrap up my current tasks first and then proceed with the reproduction work. Thank you again!

[SeongRyong0726]

@juancarlovieri Hello, I wonder your result from streaming bench.
I thought that result is not performance of small decision module, that is the performance of 7B response model.
Do you feel the same way?

[juancarlovieri]

@SeongRyong0726

I think (and I am not 100% sure about this) the smaller model is only used for token compression and choosing the top k clips. The response itself is generated by the larger (7B) module.