Error always when evaluating on scan2cap.

[TyroneLi]

Hi, when I try to evaluate on scan2cap the errors as below happen while the other benchmarks work fine, do you have any suggestions? thanks

Traceback (most recent call last): File "/leonardo_scratch/fast/IscrC_EEOL/3D_LLM_Embodiment/3D_Spatial_Mind/codebases/ross3d/ross3d/eval/model_scan2cap.py", line 322, in <module> ret = ray.get(features) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/ross3d_py310/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 22, in auto_init_wrapper return fn(*args, **kwargs) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/ross3d_py310/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 104, in wrapper return func(*args, **kwargs) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/ross3d_py310/lib/python3.10/site-packages/ray/_private/worker.py", line 2849, in get values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout) File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/ross3d_py310/lib/python3.10/site-packages/ray/_private/worker.py", line 937, in get_objects raise value.as_instanceof_cause() ray.exceptions.RayTaskError(AttributeError): ray::eval_model() (pid=499037, ip=10.9.0.254) File "/leonardo_scratch/fast/IscrC_EEOL/3D_LLM_Embodiment/3D_Spatial_Mind/codebases/ross3d/ross3d/eval/model_scan2cap.py", line 242, in eval_model output_ids = model.generate( File "/leonardo_scratch/fast/IscrC_EEOL/softwares/local/anaconda3/envs/ross3d_py310/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/leonardo_scratch/fast/IscrC_EEOL/3D_LLM_Embodiment/3D_Spatial_Mind/codebases/ross3d/ross3d/model/language_model/ross3d_qwen.py", line 386, in generate ) = self.prepare_inputs_labels_for_multimodal( File "/leonardo_scratch/fast/IscrC_EEOL/3D_LLM_Embodiment/3D_Spatial_Mind/codebases/ross3d/ross3d/model/ross3d_arch.py", line 892, in prepare_inputs_labels_for_multimodal cur_input_embeds[query_coord_tokens] += self.get_model().world_position_embedding(box_input.unsqueeze(0).detach())[:, 0] AttributeError: 'NoneType' object has no attribute 'unsqueeze'

[Haochen-Wang409]

Do the evaluations on other benchmarks encounter similar problems?

[TyroneLi]

Do the evaluations on other benchmarks encounter similar problems?

No, only for scan2cap evaluation, so it is very confused. Have you ever met such evaluation issue?

[Haochen-Wang409]

We did not encounter similar issues. Could you evaluate Video-3D-LLM on Scan2Cap?

[TyroneLi]

We did not encounter similar issues. Could you evaluate Video-3D-LLM on Scan2Cap?

I found that your evaluation for scan2cap is here when line['box_input'] is none:
`if line["box_input"] is not None:
# process box input
box_input = line["box_input"][:3]
# box_input = video_processor.discrete_point(
# box_input[:3],
# )
# center_point = [str(i) for i in center_point]
# line['conversations'][0]["value"] = line['conversations'][0]["value"].replace('', ','.join(center_point))

        gt = line.get("annotations", [line["conversations"][1]["value"]])
        qs = line["conversations"][0]["value"]
        cur_prompt = args.extra_prompt + qs
        if model.config.mm_use_im_start_end:
            qs = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN + '\n' + qs
        else:
            qs = DEFAULT_IMAGE_TOKEN + '\n' + qs

        args.conv_mode = "qwen_1_5"

        conv = conv_templates[args.conv_mode].copy()

        input_ids = preprocess_qwen([line["conversations"][0],{'from': 'gpt','value': None}], tokenizer, has_image=True).cuda()
        img_num = list(input_ids.squeeze()).count(IMAGE_TOKEN_INDEX)

        video_dict = video_processor.process_3d_video(
            video_id,
            image_processor,
            force_sample=args.force_sample,
            frames_upbound=args.max_frame_num,
        )
        video_dict["box_input"] = box_input
        video_dict = merge_video_dict([video_dict])
        image_tensors = video_dict.pop('images').half().to(model.device)
        for k in video_dict:
            video_dict[k] = video_dict[k].half().to(model.device)

        stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
        keywords = [stop_str]
        stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)

        with torch.inference_mode():
            start_time = time.time()
            output_ids = model.generate(
                input_ids,
                images=image_tensors,
                modalities="video",
                do_sample=True if args.temperature > 0 else False,
                temperature=args.temperature,
                top_p=args.top_p,
                num_beams=args.num_beams,
                # no_repeat_ngram_size=3,
                max_new_tokens=512,
                use_cache=True,
                video_dict=video_dict,
            )
            inference_time.append(time.time() - start_time)
        
        outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
        outputs = outputs.strip()
        if outputs.endswith(stop_str):
            outputs = outputs[:-len(stop_str)]
        outputs = outputs.strip()

        # print(qs.replace('\n', '\\n'))
        # print(outputs)
    else:
        gt = line.get("annotations", [line["conversations"][1]["value"]])
        qs = line["conversations"][0]["value"]
        cur_prompt = args.extra_prompt + qs
        if model.config.mm_use_im_start_end:
            qs = DEFAULT_IM_START_TOKEN + DEFAULT_IMAGE_TOKEN + DEFAULT_IM_END_TOKEN + '\n' + qs
        else:
            qs = DEFAULT_IMAGE_TOKEN + '\n' + qs

        args.conv_mode = "qwen_1_5"

        conv = conv_templates[args.conv_mode].copy()
        conv.append_message(conv.roles[0], qs)
        conv.append_message(conv.roles[1], None)
        prompt = conv.get_prompt()

        input_ids = preprocess_qwen([line["conversations"][0], {'from': 'gpt', 'value': None}], tokenizer,
                                    has_image=True).cuda()
        img_num = list(input_ids.squeeze()).count(IMAGE_TOKEN_INDEX)

        video_dict = video_processor.process_3d_video(
            video_id,
            image_processor,
            force_sample=args.force_sample,
            frames_upbound=args.max_frame_num,
        )
        video_dict = merge_video_dict([video_dict])
        image_tensors = video_dict.pop('images').half().to(model.device)
        for k in video_dict:
            video_dict[k] = video_dict[k].half().to(model.device)

        stop_str = conv.sep if conv.sep_style != SeparatorStyle.TWO else conv.sep2
        keywords = [stop_str]
        stopping_criteria = KeywordsStoppingCriteria(keywords, tokenizer, input_ids)

        with torch.inference_mode():
            start_time = time.time()
            output_ids = model.generate(
                input_ids,
                images=image_tensors,
                modalities="video",
                do_sample=True if args.temperature > 0 else False,
                temperature=args.temperature,
                top_p=args.top_p,
                num_beams=args.num_beams,
                # no_repeat_ngram_size=3,
                max_new_tokens=512,
                use_cache=True,
                video_dict=video_dict,
            )
            inference_time.append(time.time() - start_time)

        outputs = tokenizer.batch_decode(output_ids, skip_special_tokens=True)[0]
        outputs = outputs.strip()
        if outputs.endswith(stop_str):
            outputs = outputs[:-len(stop_str)]
        outputs = outputs.strip()

    with file_lock:
        new_item = {
                "dataset": dataset_name,
                "video": video_id,
                "sample_id": idx,
                "prompt": cur_prompt,
                "pred_response": outputs,
                "gt_response": gt,
                "model_id": model_name,
                "question_type": question_type,
                "scene": video_id,
        }
        ret.append(new_item)
        ans_file.write(json.dumps(new_item) + "\n")
        ans_file.flush()

`
But at Video-3D-LLM, they simply ignore those without box_input:
outputs = ""

Can you make some clarification about this difference from video-3d-llm?

[Haochen-Wang409]

Does this contribute to the bug?

[TyroneLi]

Does this contribute to the bug?

Yes, from my evaluation side. When I remove this and reset that with the same implementation as Video-3D-LLM, everything works fine.

[Haochen-Wang409]

Does this lead to different final performances?