Training set up questions

[nviolante25]

Hello,
Thank you for sharing your work!

I'm thinking of preparing a training script based on #27

Regarding data, I have access to DL3DV (with the COLMAP files as well). Here are some questions I have:

• Did you convert DL3DV to WebDataset format?
• Did you use a particular partition of the DL3DV 10K scenes for train/test? I see in section 4.1. you mention DL3DV140 for evaluation
• Did you directly use the COLMAP poses?
• The paper states that the cameras where normalized to [-2,2]. Are the cameras for the complete scene (all frames) normalized,and then the images sampled, or first the images are sampled and the cameras of those images are normalized?
• The paper says "We find it important to jointly sample I with a smaller sub-sampling stride to ensure sufficient temporal granularity and avoid missing critical transitions with a small probability (0.2 is used in practice)" Could you please clarify what this stride means for sampling the images?

Best regards

[jensenz-sai]

Hi @nviolante25,

Thanks for the interest in Seva! Your training script would be extremely helpful! Feel free to make a pull request.

• Yes! We use webdataset library with each .tar containing a couple of preprocessed scenes.
• Yes! We directly use COLMAP poses.
• We first center all cameras using all cameras, and then normalize the scene scale using the source camera.
• With stride being 1, we are sampling without subsampling frames (using consecutive frames from original clip). With stride being max, we conduct random sampling.

Hope that helps!

[nviolante25]

Hi,
Thank you for your response!

Are these the DL3DV140 scenes https://huggingface.co/datasets/DL3DV/DL3DV-Benchmark/tree/main ?

Regarding the last question, if I got it correctly 20% of the time you sample with stride=1 (consecutive frames) and the rest of the time you use random sampling?

Also did you convert the COLMAP poses in OpenCV convention to OpenGL? (this is done in the demo code for the Dust3r poses). And are the cameras used for the Plucker embeddings c2w or w2c?

And finally, is this function used to normalize during training (apart from the [-2,2] scaling and relative pose wrt first frame) ?

Best regards

[jensenz-sai]

Hi @nviolante25,

• Yes.
• Yes.
• The model takes in Plücker coordinates converted from OpenCV c2w and intrinsic. The transforms.json is stored with OpenGL format, and the ReconfusionParser will convert it back to OpenCV format for inference.
• No.

Hope it helps!

[nviolante25]

Hi @jensenz-sai ,

Thank you for the feedback! I have some further questions for you. I could open a PR after fixing some remaining issues.

Cameras and Dataset
At the moment I have code to load the DL3DV data with COLMAP cameras (I have access to the COLMAP files so I can load them directly instead of using the transforms.json and undoing the canera conversion. (Not using WebDataset for now).

• My question about the scaling of the cameras is what function is exactly used. I see these two functions used in the demo, but I don't see how this ensures that all cameras are in $[-2,2]^3$. And also there's no explicit normalization such that the camera poses are relative to the first input frame, instead is done by centering the cameras according to their median.

Configuration files and Conditioning
I integrated the sgm library in the codebase and I've been working on getting the config file right but I'm struggling to match the input_keys expected in the file and the inputs the model expect according to the paper.
According to the paper, the inputs to the diffusion model are:

• concatenation of input frames, binary mask, and plucker coordinates
• concatenation of (noisy) target frames, binary mask, and plucker coordinates
• From figure 4, these two form a single batch that is the input to the model, and it outputs a cleaner version of the target frames, which is used to compute the loss.
• As extra conditioning signal we have the average CLIP embedding of the input frames.

I want to map these operations to the config files in the GeneralConditioner and DiffusionEngine with the proper input_key.

• My question is how to input both the target (noisy latent + plucker + mask) and input (clean latent + plucker + mask) to the network, while using the clean target frames for the loss computation. Also, did you just use the provided encoders for conditioning?
• I'm thinking that I need to define a new encoder to do the plucker embedding after the frames are in latent space, is that correct? Or is there a way to use the default encoders?

At the moment, the output of my DataLoader is (the sampling we discussed in the previous message is handled internally)

output_dict = {
  "cond_frames_without_noise": cond_frames_without_noise,  # Input frames
  "cond_frames": cond_frames,  # Input frames with optional noise
  "cond_frames_mask": cond_frames_mask,  # Boolean mask for input frames
  "c2w": c2ws,  # Camera-to-world matrices
  "K": Ks,  # Intrinsic matrices
  "camera_mask": camera_mask,  # Boolean mask for input cameras (always True?)
}

Thank you again you for time.

Best regards

[jensenz-sai]

Hi @nviolante25,

• The actual logic is now somewhat inconsistent with the writing in the paper - thanks for raising the issue, and we will rectify it in the revised version! The normalization logic is: we first center the camera (cameras will have zero-mean, here); we then normalize all cameras such that the source input camera has unit length being 2 (here, and this does not ensures all cameras falling inside the [-2, 2] cube); Lastly, we convert all cameras into the source input camera space in get_plucker_cordinates functions (here).
• My recommendation is to have clean latent, mask, and plucker from the dataloader. Map clean latent to input_key so it will be added noise in the DiffusionEngine and be taken as gt directly. Convert clean latent and mask into a new condition (say, it is called replace) such that it will operate directly on the noisy input latent and "replace" the input slots with clean latent - something like this in the denoiser.py:

        if "replace" in cond:
            x, mask = cond.pop("replace").split((input.shape[1], 1), dim=1)
            input = input * (1 - mask) + x * mask
        return (
            network(input * c_in, c_noise, cond, **additional_model_inputs) * c_out
            + input * c_skip
        )

• camera_mask is always all true.

Hope it helps.

[XinyaChen21]

I noticed in this line that centering and normalization are performed using all cameras at the scene level. However, in one of your responses, you mentioned that "the normalization happens for each forward, not scene-level," which left me a bit confused. Does this mean clip-level normalization is used during training, while scene-level normalization is applied during testing?

Also, for random sampling: are the frames sampled at equal intervals? And are the sampled frames sorted in temporal order afterward?

[jensenz-sai]

Hi @XinyaChen21,

Ah, thanks for pointing it out! You're right - we first center all cameras using all cameras, then normalize the scene scale using the source camera (this inference trick is to make it consistent with the training logic). I've updated the original comment. What I meant before is that the normalization scale of the scene is not dependent on all cameras.

[jensenz-sai]

Re random sampling - it is as simple as np.random.choice(range(self.num_frames_in_seq), size=self.num_frames, replace=False).

[nviolante25]

Hi @jensenz-sai ,
Thank you for your answers! I have opened a PR #51

I modified the dataloader to output the following:

• clean_latent : all frames (both input and target) tensor of shape [F, C, H, W] where F is the number of frames (set to 21). I also created an autoencoder wrapper that simply reshapes b f c h w -> (b f) c h w and (b f) c h w -> b f c h w before and after encoding/decoding. (Btw, I'm not sure if you meant to encode the frames directly in the datalaoder? Maybe precompute the encoding and load the latents instead of the frames?)
• mask: binary mask, True for indices corresponding to input frames
• plucker : the plucker coordinates of shape [F, 6, H//factor, W//factor] where factor=8 so as to match the downsampling factor of the autoencoder.

Regarding this:

Convert clean latent and mask into a new condition (say, it is called replace) such that it will operate directly on the noisy input latent and "replace" the input slots with clean latent - something like this in the denoiser.py:

As I understand the idea is to concatenate mask as an image to the clean_latent, add the extra lines of code you suggested in the denoiser,py, which would split it back, and apply the mask to the latents. Is this enough to ensure the loss is only computed in for the target frames?

• I also added a wrapper in sgm based on SGMWrapper from this repo.

class SevaWrapper(IdentityWrapper):
    def forward(
        self, x: torch.Tensor, t: torch.Tensor, c: dict, **kwargs
    ) -> torch.Tensor:
        x = torch.cat((x, c.get("concat", torch.Tensor([]).type_as(x))), dim=1)
        return self.diffusion_model(
            x,
            t=t,
            y=c["crossattn"],
            dense_y=c["dense_vector"],
            **kwargs,
        )

I'm moving to the conditioner now, and have a couple of questions:

• Did you use these two conditions? If so, could you clarify what they are (I assume the first one is the original image size, (270, 480) in my case)?

          - is_trainable: False
            ucg_rate: 0.1
            input_key: original_size_as_tuple
            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
            params:
              outdim: 256

          - is_trainable: False
            input_key: crop_coords_top_left
            ucg_rate: 0.1
            target: sgm.modules.encoders.modules.ConcatTimestepEmbedderND
            params:
              outdim: 256

• For the image CLIP embedding, I'll create an encoder module based on this, which will accept the clean_latent and mask and retrieve the input latents as clean_latent[mask], get the CLIP embeddings and take the average.

Best regards

[nviolante25]

Hi, @jensenz-sai
I solved most of the things from my previous message. I added the masking and plucker conditioning and masked the input tensor in the denoiser.py file. However, there are some issues with the forward pass of the main SEVA network to handle a batch of frames.
Using B F C H W as input dim to the model produce shape errors in the TimestepEmbedSequential due to the usage of Conv2D layers, but a reshape to (B * F) C H W also get dimension errors here . Could you clarify what's the input format you use to handle a batch of frames? The demo doesn't include a batch dimension.

Best regards

[XinyaChen21]

Hi @jensenz-sai, may I ask whether you trained the model using fp32, fp16, or bf16?

[jensenz-sai]

Hi @nviolante25,

Since it's a spatial conv f should be flattened into b. You possibly didn't flatten the dense_emb to get the same size.