Variable frame rates in video recordings

[anna]

Hi Jeremy,

We’re running an audio/video recording experiment that consists of 36 trials. Each trial has two images projected simultaneously on the screen and a soundfile that begins playing at the beginning of the trial (at the same time the pictures show up). Each trial is recorded through the MediaRecorder function. The recording should begin at the start of each trial and end 4 seconds after the audio file finishes playing. This is all on PennController 1.8 using the old PCIbex farm (expt.pcibex.net).

These recordings are uploaded to our Dreamhost server after each trial, to make sure we get partial data if a participant quits the experiment early. Our usual protocol is to download the recordings, convert them from .webm files into .mp4s, and combine all of the individual trial recordings into one video that covers the whole experiment (so from calibration trials to the last experimental trial).

We noticed that these individual trial recordings have varying frames per second rates, even within the same participant. For example, the recording for trial 4 has 24 frames per second, but the trial 5 recording has 30 frames per second. The participant is using the same computer and browser for both trials and hasn’t reported any noticeable lag while completing the experiment. The different fps rates show up in over half of the participants (5/9) we’ve looked at so far. We can’t proceed with our original plan of combining all trial recordings into one long video clip, since the frame rates aren’t consistent across the experiment.

What could be causing this difference in frame per second rate? Is it related to the MediaRecorder element or the way we’re uploading the data to the server? It seems unlikely that it’s purely due to participants’ internet speed or browsers, since we’re seeing the difference within the same participant.

[Jeremy]

Hi,

Thank you for bringing this to my attention. It seems that the MediaRecorder API produces video streams with inconsistent frame rates, so it’s going to be a challenge to fix this for the next release of PennController.

As for the video clips of different frame rates that you have collected, you can try to normalize them before merging them. For example, ffmpeg has an option for that: ffmpeg -i input.mp4 -r 30 -y output.mp4 (output.mp4 will have the same duration as input.mp4, but in 30fps)

Jeremy

[anna]

Hi Jeremy,

Thanks for the response! It’s good to know what’s causing this, even if it can’t be fixed easily.

Mostly the individual trial recordings have had consistent frame rates within each one, so that if we expect to see 24 fps for trial_1, we’re seeing that in all of trial_1. But for some of these recordings, we also see variable frame rates within the same one trial video, so that about half the frames correspond to one duration while the other half correspond to slightly different duration. Is this caused by the same issue or is that a separate problem?

Thanks!

Anna

[Jeremy]

Hi Anna,

I don’t know enough about this to answer confidently, but from my read of the github issue I shared above, I suspect this is caused by the same issue indeed. Some video codecs such as VP8 or VP9 support a variable video frame rate, and your webm videos were probably encoded using those.

Here is a reddit thread that gives pointers on how to normalize variable-framerate .webm files. You may need to look up some ffmpeg options if you are not familiar with them, as not all are explained in the thread. Also, feel free to use the specific keywords vfr (variable frame rate) and cfr (constant frame rate) in your searches if you look for more information about this process

Jeremy

[Author]

Posts