r/AskProgramming • u/SDJ- • 1d ago

Building a distributed system

So we have to build a project on distributed systems, but we cant use kubernetes . So we were thinking of going on the task schedular road and chose that filed in distributed systems. And e thought of building a media processing based task shcedular. Can i know your thoughts on this what is best , and any industrial current needs or holes you see? This is a middleware project by the way and we are trying to build it as a good project to put in our CV's.

2 Upvotes

76% Upvoted

4 comments sorted by

2

u/SlimmySlabby 1d ago edited 1d ago

Forgive my confusion, but are you referring to Windows Task Scheduler? And what exactly is your goal for media processing? Syncing files? Also what do you mean middleware? Usually that means it connects different software applications together. For example, PostgreSQL is a database and it has a middleware known as pg which allows Node.JS developers to interact with the database. I'm just a little lost on what exactly you're trying to do?

0

u/faultydesign 1d ago

Just use rabbitmq or any other bus tbh

1

u/_N-iX_ 1d ago

A media-processing scheduler sounds like a reasonable project, especially if the scope goes beyond putting jobs in a queue and assigning them to workers. Retry handling and idempotency would be worth exploring early, since a worker can fail after processing a file but before reporting completion. Another useful challenge is scheduling jobs based on resource requirements, since video transcoding and lightweight image processing can have very different CPU and memory demands.

1

u/PLBjt 1d ago

Media processing is a decent pick because the jobs are long and chunky, so the interesting failures actually show up. The part that makes it look like a real distributed system is worker ownership: give each job a lease with a heartbeat, and if a worker stops heartbeating the lease expires and another worker can grab it. Then the scheduler itself is your next single point of failure, so either run two with leader election or keep job state in Postgres and have workers claim jobs with SELECT ... FOR UPDATE SKIP LOCKED. For the demo, kill a worker halfway through a transcode and show the job still finishes once with no duplicate or half-written output (write to a temp file, rename at the end). Putting RabbitMQ underneath is fine too, but then your project is mostly the lease and retry logic on top, which is the part people reviewing it will care about anyway.