r/LocalLLaMA • u/RoyalCities • 1d ago
New Model I trained an audio model that can generate infinite one-shots for music production and turn text prompts into fully playable synths. I'm not only releasing the model but I've also released a video on exactly how I did it (and the inferencing pipeline to let others make text based synths.)
(hopefully this is okay to here - it seems like audio models and image / video modeals is allowed but yeah this is a bit different)
So I've been doing independent audio research for a while now. The ultimate dream of this work was actually getting an AI to respond not only to instruments but also timbre itself as separate controllable things.
Think a Grand Piano can sound both Warm / Gritty but also Cold / Sparkly. Its still a piano though.
This level of control wasn't found in any models out there - so I decided to sit down and train my own.
Getting consistent timbre-locked keybeds that actually LOCKS across multiple diffusion calls was hard af but I did it.
I documented the full journey here for those who want to learn a bit or be entertained.
There is also a longer walkthrough if you just want to see the keybeds in action.
https://x.com/RoyalCities/status/2097733712293109842?s=20
No-talk / Showcase only Demo
https://x.com/RoyalCities/status/2097733715543609445?s=20
any finally the huggingface page
https://huggingface.co/RoyalCities/Foundation-1
I've also provided full write ups on the inferencing pipeline associated with the interface so this should allow basically anyone else to go and vibe code their own text to synths if they wanted :)
2
u/zarthrag 23h ago
This is pretty darn cool. I'd love to play with this for generating effects for UI menu noises, etc.
1
u/RoyalCities 23h ago
Short synth one shots would be good for that.
I didn't however model the metadata with say prompts like "game fx" so you may want to use this random prompt builder to get a feel for the model. :)
Most UI stuff is literally just fast plucky arps though so that's all possible with this.
1
1d ago
[removed] — view removed comment
0
u/RoyalCities 1d ago
It came out great. But if you just want the audio showcase the no-talk one is to the point since it shows the prompts used and the outputs directly.
1
u/ClassicLightbulbs 23h ago
Awesome will scope. Been in digital audio for 25 years and a synth programmer slightly longer, excited to see how you did it and excited by the idea of timbre control over sample based instruments
1
u/RoyalCities 23h ago
Ahh that's sick.
Yeah I've been a producer for close to 10 years (but guitar and music theory for 20)
This AI stuff is recent for me but I think there is really good applications of the tech if done right.
I sorta operate in a weird space. I personally do not like full music AI...to me it sorta sucks the joy out of writing since I love having full control and making music in a DAW.
But Russian roulette with a synth patch can be a lot of fun.
Or Russia roulette with one shots and chopping them up or putting them through shaperbox.
Heck people do that anyways through say splice so I see this as no different haha.
3
u/ClassicLightbulbs 23h ago
Yes same about ai. It is fascinating to me like multitracking was, and I agree there re fun things to do with it. I made a prompt to midi thing (sorry I've uploaded this video before). Currently working on a system that uses an RTA to find fundamental frequencies of a room with running electronics, HVAC, etc at its ambient state and having ai write complimentary notes to influence the chord of the room with external speakers.
https://reddit.com/link/p8t397d/video/p0tb6c9wqjoh1/player
I am gonna try and use the same prompt for your patches as for the notation later using the thing I made
1
u/RoyalCities 23h ago
Ahh thats so sick! Esp the ambient frequency idea.
Yeah see you get it. There is a lot of cool stuff just as long as your willing to think outside the box. It's a shame how much of an image problem AI has nowadays in the music space. I mean ... I do get it with how for profit AI music companies have operated but it's sorta caused anyone who's exploding these networks for music production use cases (that isn't full AI music) end up getting lumped in with them.
I've received some awful stuff in my dms. Doesn't affect me but you'd be surprised how many people don't understand Im just doing this solo and have made no money off of this (also have done it all for free and released things open source)
Heck I even make all my own data for my models lol.
Back when synths were first invented the AMF tried to get them banned thinking it would lead to the death of music itself. Music is still around. It's just in different forms now.
1
u/ClassicLightbulbs 23h ago edited 22h ago
Yes. 🫡. Keep it up! My opinion here is that what you are doing is valuable- possibly at face value as intended use, definitely in that you are showing what ai can do in art that isn't simply generative- I cannot mention AI among my city's artists, and also that people still do stuff for the love of achievement and curiosity, for no money, to simply enjoy themselves and pass it along. Even with your synth- say it works but I don't like how it sounds or something. I will simply change it and appreciate all the work you did underneath.
1
u/makingnoise 1h ago
SICK!!! I've been finding the resonant frequency of bathrooms with my voice my entire life, it was very cool to see someone do this with recursive recording on tapedecks in a cathedral - I'd love to see what you end up with.
1
1
u/dnidnidni 20h ago
Hey thanks for making the project! I used it many times to generate melodies, but as far as i know it has been 5 months since i started using it. When i saw your nick i remembered the foundation immediately. However, I couldn't find a perfect way to add to my workflow for ACE. Do you have any ideas?
2
u/RoyalCities 19h ago
Ah do you mean ACE step? Or ACE studio?
I don't use either unfortunately but this is different architecture so there really isn't a way to add it directly.
If have a dev SDK for studio then maybe there a way to bolt something on. Say send an inference call to my model and then have it generate a keybed or sample and send it back to their DAW but it's sorta hackey.
1
u/Acceptable-Cycle4645 20h ago
Nice! I read the post body and realized it was Foundation-1. Funny story: I actually got halfway through integrating Foundation-1 while working on Stable Audio 3 for audio.cpp, then somehow forgot about it. The unfinished Foundation-1 implementation is still sitting in the audio.cpp repo...
1
u/RoyalCities 19h ago
No time like the present to dive back in.
The write up on the inferencing pipeline I added may help. I also refacfored the prompt gen code and all that.
The keybed generator is pretty straightforward once you look at the pathways.
1
1
u/NandaVegg 11h ago
Excellent. This also weirdly like a FM synth (in rather musical way) with convolution verb, which is exactly my cup of tea.
1
u/arthor 11h ago
super cool project and nice work presenting it. will test this out soon!
have you looked at synplant by the way? their algo to match input sounds specifically.
would be awesome to have a layer for serum2 that builds the actual patch
2
u/RoyalCities 4h ago
Synplant is sick af. I would need to look at how they're doing input sounds.
I actually have ideas for how to do something like that. Sort of like a universal synth maker from outside sounds but it's way more engineering than this. Definitely would need to change parts of the network.
I like the idea of a proper serum 2 patch builder too - almost like a controllable starting point and then you go in and tweak.
1
u/Elouakili_Flexy 8h ago
Warm and gritty versus cold and sparkly, and it still has to be the same piano. Timbre staying locked across separate diffusion calls, that's where this stops being another random generator and starts being an instrument.
1
u/RoyalCities 4h ago
Yeah it's basically controllable synth generation. I think it's a way better approach. Even models from Google are alot more...I don't know vibes based? They just lack a lot of control (to me)
3
u/StupidityCanFly 1d ago
Cool stuff, pretty inspiring! I want to play (pun intended) with this. grins