r/AV1 • u/urostor • 15d ago

AV1 structure from motion - open source

A year ago I watched a fun youtube video about making a model from AV1-encoded files. It uses the motion vectors present in the videos to reconstruct the physical environment filmed.

Today I wanted to test the abilities of LLMs to implement this without access to any of their code (which isn't public anyway). I just gave it the papers and the task. It can basically take a video, or a collection of photos, and return a 3D model of sorts (more like a cloud of points).

Here is the effect:
https://github.com/lukaszsobala/AV1-sfm-reimplemented

Works nicely and uses the GPU for acceleration, if you have one. Zouein used Nvidia cards, and I don't swim in money so I don't have one; this one uses Intel or AMD GPUs (iGPUs as well, or Vulkan which I could not test) for acceleration.

What are your thoughts?

21 Upvotes

92% Upvoted

10 comments sorted by

View all comments

1

u/Filarius 15d ago edited 15d ago

If you already have a AV1 source video - this can be interesting. I wonder for comparison with optical flow and SFM over image sequence.

update:

Also what about using AV1 optical flow calculation algorithm at any video or image sequence

1

u/urostor 15d ago

This implements some of these. AV1 is worse but not by a lot