Finetuning data

[Nebuluss]

How long should be the video for finetuning (1sec?) ? How many does it require ?

[ExponentialML]

How long should be the video for finetuning (1sec?) ? How many does it require ?

Hey! Ideally you would probably want a mix of FPS (randomized), and I'm finding 6 frames at varying FPS to be decent. It's best to cut your clips before the fact.

[sergiobr]

Hi @ExponentialML thank you for this repo. I got it running and commited 2 PR. I think the second one is right although I have not revised the full logic.

I'm using 30-45 second videos to fine tune.

What do you use to get mix FPS?

[ExponentialML]

Hey @sergiobr .

To get a mix of FPS, change this parameter to 30.

@Cubey42 Yes you can!

[sergiobr]

@ExponentialML Thanks. I don't know exactly what this means.

I also discovered that if the videos does not have the same number of frames you'll get an error.

Your last config is set to 4 frames. I believe it's for a high frame rate training. I'll keep it and check the results.

[ExponentialML]

@sergiobr

Here's the code.

sample_index = list(range(idx, len(vr), self.sample_frame_rate))[:self.n_sample_frames]

Essentially, you create a list of frame indices starting at idx, at max length len(vr), at sample rate self.sample_frame_rate which essentially acts as frame skip, mimicking high motion. Then we just slice the list to the range of frame numbers we're sampling.

Also, could you report that error so we could resolve the issue? Thanks!

I'm also finding that 2 frames is sufficient when testing with a single video. It's very interesting, but it's possible that you may just be able to fine tune on images alone.

trainable_modules:
- "attn1"
- "attn2"
- "temp_convs"

I'll keep reporting back with anything I find.

[sergiobr]

@sergiobr

Here's the code.

sample_index = list(range(idx, len(vr), self.sample_frame_rate))[:self.n_sample_frames]

Essentially, you create a list of frame indices starting at idx, at max length len(vr), at sample rate self.sample_frame_rate which essentially acts as frame skip, mimicking high motion. Then we just slice the list to the range of frame numbers we're sampling.

Also, could you report that error so we could resolve the issue? Thanks!

I'm also finding that 2 frames is sufficient when testing with a single video. It's very interesting, but it's possible that you may just be able to fine tune on images alone.

trainable_modules:
- "attn1"
- "attn2"
- "temp_convs"

I'll keep reporting back with anything I find.

That's very interesting. I was looking for such a thing (train single images on this model).

Yes of course, I'll reproduce the error and post on Issues.

[patrickjonesdotca]

I have trained a model and converted it to a ckpt. Now where do I place it in the text2video webui directory?