How do I render a MotionFix npy sample file into an mp4 file?

[JK3WN]

I was able to get some npy sample files out of the trained ckpt file, but I have no idea how to visualize/render these into a video.
It looked like your team was able to make a video out of the sample, so I want to know how your team did this.
Is there an easy way or an easy tool to make a video out of it?
Since I am new to this field, if you could explain the process in an easy way, it would be appreciated.

[atnikos]

Hello, I will release a demo to do this. But here is a standalone script with some instructions. Can you try and tell me?

from src.render.mesh_viz import render_motion
from src.model.utils.tools import pack_to_render
from src.render.video import get_offscreen_renderer
import os 

os.system("Xvfb :12 -screen 1 640x480x24 &")
os.environ['DISPLAY'] = ":12"

# GET A RENDERER
r = get_offscreen_renderer(smpl_p) # this path if you followed my setup should be data/body_models
# Load the npy and get the translation aka trans_can and the rotations aka rots_can
# CREATE A DICT TO GIVE TO RENDERING FUNCTION
smpl_params = pack_to_render(trans=trans_can, rots=rots_can, pose_repr='aa')
# RENDER A MOTION AND SAVE VIDEO!
render_motion(r, smpl_params, pose_repr='aa',
                               filename='/some/path/you/want')

# If you want to render them overlaid motion1 motion2 you can do:
smpl_params1 = pack_to_render(trans=trans_can1, rots=rots_can1, pose_repr='aa')
smpl_params2 = pack_to_render(trans=trans_can2, rots=rots_can2, pose_repr='aa')

# RENDER A MOTION AND SAVE VIDEO!
render_motion(r, [smpl_params1, smpl_params2], pose_repr='aa', colors = [color1, color2]
                                filename='/some/path/you/want', text_for_vid='if you want to put some text')

# to give the correct colors look at src/utils/art_utils.py

[JK3WN]

After typing
python3.10 -m pip install aitviewer

I tried to run the code, but it shows the error above. I'm pretty sure I'm forgetting some arguments.
Can you show the example about how you're supposed to use this code?

[JK3WN]

Update to the situation:
Figured out I was installing aitviewer in a wrong way, so I installed using
pip install aitviewer
in another conda environment.
Still got the same error, so I removed line 25 and 26, which is
render_motion(r, [smpl_params1, smpl_params2], pose_repr='aa', colors = [color1, color2]
filename='/some/path/you/want', text_for_vid='if you want to put some text')
Now it shows this error.

[atnikos]

@JK3WN can you try this?
python visualize_sample.py --path experiments/tmed/3way_steps_300_motionfix_noise_last/ld_txt-2.0_ld_mot-2.0/000136.npy
I just pushed a dirty script. Feel free to adapt it and iterate over multiple ids (000136.npy is one saved sample).

[atnikos]

I hope this fixes the issue and helps you render!
@JK3WN @zhangjingze21 lmk.

[FieldFarmer]

Update to the situation: Figured out I was installing aitviewer in a wrong way, so I installed using pip install aitviewer in another conda environment. Still got the same error, so I removed line 25 and 26, which is render_motion(r, [smpl_params1, smpl_params2], pose_repr='aa', colors = [color1, color2] filename='/some/path/you/want', text_for_vid='if you want to put some text') Now it shows this error.

Hello, I met the same problem as you, may I know how you solved it？

[atnikos]

Hey @FieldFarmer ,
can you check my attached script? There are some environment variable you need to check that are included in the visualize sample script. Can you check that?

Best,
Nikos

[neoory]

Hi, @atnikos ,
When I ran visualize_sample.py, nothing wrong when I ran a single motion. But there are some problems that happened on motion1 and motion2.

My code is as follows:

    # If you want to render them overlaid motion1 motion2 you can do:
    r = get_offscreen_renderer('./data/body_models') # this path if you followed my setup should be data/body_models
    color1 = rgba('olivedrab')
    color2 = rgba('darkred')
    print("Loading second motion for overlay rendering")
    # gen_motion = np.load(path_to_motion, allow_pickle=True).item()['pose']
    gen_motion1 = np.load(path_to_motion.split(".npy")[0] + "_target.npy", allow_pickle=True).item()['pose']
    gen_motion2 = np.load(path_to_motion.split(".npy")[0] + "_source.npy", allow_pickle=True).item()['pose']
    # print(gen_motion.shape, gen_motion1.shape, gen_motion2.shape)
    smpl_params1 = pack_to_render(trans=torch.from_numpy(gen_motion1[..., :3]), rots=torch.from_numpy(gen_motion1[..., 3:]), pose_repr='aa')
    smpl_params2 = pack_to_render(trans=torch.from_numpy(gen_motion2[..., :3]), rots=torch.from_numpy(gen_motion2[..., 3:]), pose_repr='aa')
    # RENDER A MOTION AND SAVE VIDEO!
    render_motion(r, [smpl_params1, smpl_params2], pose_repr='aa', color = [color1, color2],
                  filename='output.mp4', text_for_vid='if you want to put some text')

The error is:

Traceback (most recent call last):
  File "d:\code\motion\editing\motionfix-main\visualize_sample.py", line 58, in <module>
    main(args.path)
  File "d:\code\motion\editing\motionfix-main\visualize_sample.py", line 42, in main
    render_motion(r, [smpl_params1, smpl_params2], pose_repr='aa', color = [color1, color2],
  File "d:\code\motion\editing\motionfix-main\src\render\mesh_viz.py", line 125, in render_motion
    smpl_template = SMPLSequence(body_pose,
  File "C:\Users\10055\.conda\envs\mfix-env\lib\site-packages\aitviewer\renderables\smpl.py", line 146, in __init__ 
    self.vertices, self.joints, self.faces, self.skeleton = self.fk()
  File "C:\Users\10055\.conda\envs\mfix-env\lib\site-packages\aitviewer\renderables\smpl.py", line 400, in fk       
    verts, joints = self.smpl_layer(
  File "C:\Users\10055\.conda\envs\mfix-env\lib\site-packages\torch\nn\modules\module.py", line 1751, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "C:\Users\10055\.conda\envs\mfix-env\lib\site-packages\torch\nn\modules\module.py", line 1762, in _call_impl 
    return forward_call(*args, **kwargs)
  File "C:\Users\10055\.conda\envs\mfix-env\lib\site-packages\aitviewer\models\smpl.py", line 243, in forward       
    return self.fk(*args, **kwargs)
  File "C:\Users\10055\.conda\envs\mfix-env\lib\site-packages\aitviewer\models\smpl.py", line 171, in fk
    assert poses_body.shape[1] == self.bm.NUM_BODY_JOINTS * 3
AssertionError

Do you know the wrong about this?
Really Thanks!

[atnikos]

It seems like there is some shape error.
Can you tell me these shapes?
[smpl_params1, smpl_params2]
They need to match what you are doing for a single sequence (their dimensions).

[neoory]

The results are as follows:

    gen_motion = np.load(path_to_motion, allow_pickle=True).item()['pose']
    gen_motion1 = np.load(path_to_motion.split(".npy")[0] + "_target.npy", allow_pickle=True).item()['pose']
    gen_motion2 = np.load(path_to_motion.split(".npy")[0] + "_source.npy", allow_pickle=True).item()['pose']
    smpl_params1 = pack_to_render(trans=torch.from_numpy(gen_motion1[..., :3]), rots=torch.from_numpy(gen_motion1[..., 3:]), pose_repr='aa')
    smpl_params2 = pack_to_render(trans=torch.from_numpy(gen_motion2[..., :3]), rots=torch.from_numpy(gen_motion2[..., 3:]), pose_repr='aa')
    # RENDER A MOTION AND SAVE VIDEO!
    print(gen_motion.shape, gen_motion1.shape, gen_motion2.shape)  # (120, 135) (120, 135) (120, 135)
    print(smpl_params1["body_transl"].shape)  # torch.Size([120, 3])
    print(smpl_params1["body_orient"].shape)  # torch.Size([120, 3])
    print(smpl_params1["body_pose"].shape)  # torch.Size([120, 129])
    print(smpl_params2["body_transl"].shape)  # torch.Size([120, 3])
    print(smpl_params2["body_orient"].shape)  # torch.Size([120, 3])
    print(smpl_params2["body_pose"].shape)  # torch.Size([120, 129])

[neoory]

Additionally, I tested the gen_motion, gen_motion1, gen_motion2. Each of them can generate the single motion video.

[neoory]

Hi, @atnikos . I found the solution.
I revised the following codes:

smpl_params1 = pack_to_render(trans=torch.from_numpy(gen_motion1[..., :3]), rots=torch.from_numpy(gen_motion1[..., 3:]))
smpl_params2 = pack_to_render(trans=torch.from_numpy(gen_motion2[..., :3]), rots=torch.from_numpy(gen_motion2[..., 3:]))

that is, delete the pose_repr='aa' in pack_to_render.
Now, I can generate the video with two motions.
Thanks for your help. ^_^