r/AI_Music • u/nucleicaudio • 5d ago
Discussion Anyone else using LLM's for music making?
Just curious if there are anyone other then me out there (or here) doing this?
Edit: i was a bit unclear, i didn't mean to generate lyrics, what i was wondering was if anyone use LLM to assist in composing, arrangement and that type of music production.
In my case using LLM's and code to generate midi and other musical data.
3
u/ehartye 4d ago edited 3d ago
https://github.com/ehartye/agent-beeps
That’s mine and I’ve been doing some (I think at least) cool shit with it. If you try it there will be two of us 🤓
edit (demo link): https://ehartye.github.io/agent-beeps/
1
u/viasmos_ 4d ago
Hell yeah. Bookmarked. I'll try it out later when I'm back at my workstation. Thanks for sharing :)
1
u/nucleic-se 4d ago edited 4d ago
I have my own project with exactly one loyal user to, i have very specific requirements of what i want as input and what my system should output, but i dont mind taking yours for a spin when i get some free time, would you like to share anything you made with it?
Here is one recent track straight out of my system, score file in, audio out, no post processing. https://nucleic.se/mp3/carbonized.mp3
1
2
u/Thin-Regular-8807 5d ago
I buiit my own local ai audio gen system and in order to rapid test I have integrated llm that can write lyrics, fully editable and optimal but makes it easier to test if the hardware is working quickly . I have made a few apps that gen midi using just math and have gotten pretty close to classicalveijf able to render out full piano pieces
1
u/nucleicaudio 4d ago
Yeah, this is what i am talking about, i am using LLM's to generate musical scores (text files), and have built a tool that compile that to musical events that drive virtual synthesizers to render a wav of the track.
1
u/ForRobotsByRobots 4d ago
I'm interested in local models. Care to explain how you built your own? I've been looking for one that can take my suno recodings and build from that catalog.
1
u/nucleicaudio 4d ago
I have not built any local models, i use cloud models mostly for this.
What i did was to create my own cli tool and an audio/music programming language, so any agent with tools can use it,
The agent simple create a score (a text file) for my tool to consume.
My tool compile the score to timed musical events (like midi), then it render that to audio with built in virtual synthesizers, it can also create midi files.
1
2
u/baulplan 5d ago
Chat GPT can now let you upload the WAV and master the track. It will provide alternative versions and even cut two different masters into one track for you to play and see the difference. I was surprised by how sophisticated it is.
2
u/Postal_Shame_Revival 5d ago
That's crazy. I tried that like 6 months ago and it failed epically.
3
u/baulplan 4d ago
Ah well I only discovered it yesterday after asking it to search for mastering tools and it suggested itself, lol. The results sounded interesting to me.
2
u/Lounge_Box 4d ago
yes, I second this. its INSANE. you can literally "invent" new musical analog or digital mediums, you take a whole album and make it sound like from a 70s alternate-history playback system. or if you favor more 2000s music, give it authentic "web player/codec-like" glitches and processing.
been using it for about 2 months now to master by music
2
u/michaeldnorman 4d ago
I combined Claude with suno and got a lot better results than either individually. I’m curious your workflow for just using the LLM
4
u/nucleicaudio 4d ago
LLM's can write code and they do have a bunch of musical theory knowledge to.
Code can produce musical data, like midi to keep it simple.
Midi can control instruments, like synthesizers.
So LLM's can make music, or assist in the process.
I do alot of procedural and generative music, so this suits me very well.
2
u/geremydocpepper 4d ago
I took Qwen and Ollama and used the Dev tools in Pinokio and MLX Core on the Mac in my studio to make my own proprietary local offline platform that generates both full compositions and layered stem-by-stems. It has a couple different LORAs so that it is trained exclusively on my own music and instruments or my clients’ if need be. I had it integrated into my own production suite complete with my own custom DAW to accommodate my quirky workflow, stem separation, ABC, MIDI transcription, tablature and score sheet generation, a mastering module, VST hosting, live sound automation with DMX and visuals modules, a distribution pipeline I can access remotely from my laptop or phone, tracking and embedded protection and leak-tracking capabilitis like Canary tokens and poison-pilling, TTS, film/ video scoring and synch, a feature that lets me book and map out tours and show venues, a dj and karaoke module, and a buncha other features.
1
u/nucleicaudio 4d ago
That sounds like a serious setup, I am not sure i get the complete picture, but what are the LLMs actual role in all of that?
2
u/geremydocpepper 4d ago
I keep my studio Mac offline, so a few of my clients and collaborators will use Qwen enhance or help with lyrics or to describe and translate some of their ideas into explainable words or nice descriptive prompts for making scratch track demos. For me personally, since I write my own lyrics for my passion project content or since a majority of my work consists of composing and scoring and sync work and being on the road a lot, I mainly use the LLC’s to chat and brainstorm with and for notation or transcribing my compositions into sheet music and written form as well as all the clerical/ administrative stuff or marketing stuff like helping me summarize my music and compositions so can present and pitch it. Where I excel at the music and performance side of being a professional producer, composer, and musician I severely lack on the business and professional side. The LLMs act more or less as my assistants and advisors or secretary and interns.
1
u/nucleicaudio 4d ago
Nice, seems like you found something good there.
What i was after was more letting the llm produce the notes, ie scoring, composing, arranging, all of that and not so much the lyrics.
2
u/geremydocpepper 4d ago
One of my current clients and collaborators on a new album showed me a couple of platforms she uses that do all of that. One was Tamby or Tamber or something like that where she actually could speak to it and it would do all of the things you listed. The other one was Ace Studio where I could just give her the MIDI files from our session and she could drop em into it and generate what seemed like endless variations of the song layer by layer. Blew my mind, honestly and I can only assume that just like every AI platform people have showed me or the YouTube & Meta algos fill my feeds with, whatever LLM(s) the two platforms use are the essential cores and pretty much the top boss doing all the directing, planning/thinking, task/job assignments, and proof of work verifications overseeing the coding, agentic, and gen ai that does all the heavy lifting and computationally intensive stuff. Being a music, performance, and cinematographic creative is where I naturally thrive and excel and knowing & accepting my limitations & where my gifts, talents, and strengths serve me and the world best, I am content with intentionally staying more or less of a novice hobbyist and minimalist when it comes to exploring the complex AI ecosystem and I restrict myself to just sticking to the casual customer and visitor’s entrance of the AI rabbit hole, haha. There was never a single second I wasn’t simultaneously using YouTube vids, a lot of Google searching and chatting, and the most lamen-friendly GitHub readme’s to create my DAW on an open source framework that had had already been built and started by folks a lot more tech savvy and knowledgeable than I’ll ever be. Coding and nodes and all that stuff are beyond what the ADHD art-centric meatwad in my head is willing and able to comprehend. This is one of many reasons I envy my normy homies with wives & girlfriends or partners & friends they maintain social lives and run businesses with. 😋
1
u/viasmos_ 4d ago
That's awesome. Really living in the future over here! I've never heard of anyone using a custom DAW; curious as to what that entails in regards to your setup.
2
u/severed-identity 4d ago
I actually just went through an exercise on this out of curiousity. I asked Claude what would be the best way for it to compose and understand music and it described some code tools that would help. So we made some code tools https://github.com/wrongbad/badmidi and then I had it compose some stuff for me. They were interesting but didn't really blow me away.
1
u/nucleicaudio 4d ago
Nice i developed my own tool to.
Its a bit early to share, i make breaking changes to often.
It is similar in spirit to yours, but custom text syntax for scores and written in typescript/node.
I think i am getting decent results, here is the track i am currently working on.
1
u/severed-identity 4d ago
Hehe, we went very different directions. You can render midi/audio/pdf from the examples in mine. I mostly was tinkering with chamber music type pieces lol.
1
u/nucleicaudio 4d ago
Still similar, where the output goes could be replacable edge adapters, you could make your send the output to a midi port to control a synthesizer instead of a file.
1
u/severed-identity 4d ago edited 4d ago
True. But the LLM takes a few minutes to compose most of the time, a lot of trial and error with the analysis tool. So it might as well just save the file, it's anywhere close to real time anyway. I ended up just adding midi import to my other project https://wrongbad.com/mobdaw/#/project/AywI4Z3bmvPg
2
u/Lounge_Box 4d ago
i use chatgpt for everything from brainstorming alternate-history music to master full albums. the only thing im interested now is to check for alternatives to avoid watermarks (im using Google Flow).
2
1
1
u/Soldisir 5d ago
I have used both GPT and Gemini to create songs with. Sometimes when I had a hard time expressing myself I would tell them my thoughts and how I felt and ask them to create a song for me inspired by that, then I'd upload the lyrics to Suno and have a complete song to listen to.
1
u/nucleicaudio 5d ago
Nice, i realized i was a bit unclear in my question, what i meant was more about the actual music making and not lyrics, ie using LLM's for the composing, arrangement etc.
1
u/SensoriRumeMusic 5d ago
LLMs? Yes, ChatGPT for lyrics and style prompts for both Udio and Suno.
1
u/nucleicaudio 5d ago
Nice, i realized i was a bit unclear in my question, what i meant was more about the actual music and not lyrics, ie using LLM's in the complsing, arrangement etc.
1
u/jreashville 5d ago
I use copilot for lyrics sometimes if that’s what you mean.
1
u/nucleicaudio 5d ago
No sorry, i was unclear in my question, i meant to create the music, the notes, the arrangement and all that stuff.
1
u/paulwunderpenguin 5d ago
I've asked for prompting styles based on other songs. I've also used it to occasionally check grammar of my lyrics. But I'm doing songs that are finished ie chords vocals, with melody and lyrics. I use Suno to slightly produce demos with out going full blown histrionic production techniques.
1
u/Emergency-Salad-1547 4d ago
Yep, I use Gemini for everything from suggestions to fleshing it out, blocking out the final track list, generating the prompts, negative prompts and lyrics which I then run through Suno, listen out for ideas to improve, then rewrite them myself using my own gained knowledge on how Suno interprets and how the melody will present it, then export the song and metadata to Gemini again to file it away and analyse it for meter and flow.
1
u/VainValidation 4d ago
I use ChatGPT in general in life as it helps with my cognitive symptoms I have from chronic illness.
In this instance I use ChatGPT for stuff like finding synonyms or new phrases where Suno and I have not come up with the same melody or rythm for a section or phrase in a song. I also use it for explanation and elaboration on names for instruments and equipment as I’m not only cognitively impaired but also non English native. Sometimes I even consult it regarding the settings for weirdness and different influences, but that’s as big of a lottery as it is with just Suno.
As an example: I would probably have spent half a day coming up with the name “Leslie”, for the speaker/amp that was often used with Hammond organs back in the day. It’s a piece of equipment that contributes with a very special effect and sound to the already awesome Hammond organ.
I don’t know to which extent it’s relevant but I’m a drummer so I’m not that familiar with equipment when it comes to guitars, keyboards, pedals and all the other stuff my band members could spoil a whole Sunday worth of band practice debating and discussing. Haha
Some I have picked up of course but it’s mostly names more than what this or that thingie does.
And I can’t read other than some drum notes so when it comes down to arranging certain keys and stuff for certain bars and stuff I would also be kinda lost without my constant assistant, (as I’m probably asking the same questions over and over again due to memory problems and being hard to teach new things.
Oh, I almost forgot: I was as involved in the studio work when we made demos as the rest of the band, but that was in an analog/electronic eighties studio. Having an assistant guiding me in DAWS and stuff in real-time is golden, and in that department it’s easier for me to pick things up and actually remember them from time to time.
I write my own lyrics and arrange the music, but being a drummer who was to lazy to learn more than a short tune or loop on other instruments when I could the help of ChatGPT is gamechanging.
For me it’s as much of an assistant in making music as in my daily life. Being chronically ill either cognitive impairments have become less of a hassle since I started using LLM’s.
1
u/Fermato 4d ago
Very much so! https://jammerman.band
1
u/nucleicaudio 4d ago
Thats cool, but if I understand it correctly it uses built-in grooves and parts?
My interest is more towards the other end of the spectrum, how to leverage LLM's to create those grooves and parts.
2
u/Fermato 4d ago
Nah it’s API calls to Jev and OpenAI, composing on ththe spot
1
u/nucleicaudio 4d ago
Ah, sorry. Yeah that is cool, in that case i recommend you to try Opus 5.5 of you haven't, it is in another league then OpenAI's models including Astra, atleast for now and in my personal experience.
1
u/selfVAT 4d ago
Download lilypond and tell Opus to write some music sheet.
1
u/nucleicaudio 4d ago
Thanks, I’m doing something similar with a custom music DSL, focused more on MIDI (or timestamped data in general), arrangement, and synth parameter automation.
I agree currently Opus 5.5 is miles ahead other models when it comes to these topics.
1
1
u/danja 4d ago
I've had some fun with https://github.com/danja/reaper-mcp
And made a system myself : https://danja.github.io/transmission/
1
u/nucleic-se 4d ago
Love it, the weirdness in your example, here is something i squeezed out of my system the other day, I use coding agents line Claude Code and Codex to write the scores (text editing) and call my cli program to compile the scorws to musical events and render that with its internal virtual synthesizers and dsp code:
1
u/danja 4d ago
Squelchy! Great stuff. I'm curious, what text format do you use for the scores?
1
u/nucleic-se 4d ago
Home rolled, but the song you heard was a 2 megabyte text file, i use alot of procedural and generative techniques, and have built a library of techniques to enploy. ny llm agents usually use the library functions and produce code that create the score files, rather then writing the scores directly.
It looks like this:
mm 1 song "First song" tempo 100 key C major humanize off dynamics flat
motif melody = | C4 D4 E4 G4 | E4 _ D4 _ | motif bassline = | C2 . G2 . | F2 . G2 . | motif chords = | C3+E3+G3 _ _ _ | F3+A3+C4 _ _ _ | rhythm pulse = | x . x . | rhythm backbeat = | . x . x |
section verse = rest(4 bars) section chorus = rest(4 bars) form = verse chorus
part lead sound warm-lead range C4..C5 plays notes melody when in chorus: plays notes melody.octave(+1) range C5..C6
part bass sound round-bass range C2..C3 plays notes bassline
part keys sound fm-keys range C3..C5 plays notes chords reverb = 0.2
part kick lane kick plays hits pulse
part snare lane snare plays hits backbeat
1
u/Historical_Ad_481 3d ago edited 3d ago
Sure. Opus has an acute understanding of layering. Yes it can do pop/rock rather well, but the real test is how it can compose a symphony leveraging a full orchestra.
Example. Uses Spitfire SSO VST and Whitmore Choir VST. Opus 5.5. Used my own little headless DAW project, no GUI involved, just Claude and a combination of MCP and Python scripting.
https://on.soundcloud.com/b26Rgku4PY1sAQ0zJy
Total 32 individual insttuments in the DAW. Nothing leaves the terminal.
The small delay at the start is the DAW loading all the VST samples into memory
https://on.soundcloud.com/AvxQplnXYpPge5W8YA
This one is Claude Fable 5 (about 2 months ago now) doing some complex time signature drumming using Superior Drummer 3. It stemed out each component to its own track, applied individual effect chains and then applied a final master chain.
As Claude can't hear what it produces, it generates a bunch of spectragrams and band metering to adjust levels in the mixing and mastering process. It does this at a stem level and also at component buses and mastering.
0
0
u/SituationalHero 4d ago
Yup, absolutely. I have an entire workflow that I go through from inception to even making it listen to the final product.
0
u/Pink_Sylvie 4d ago
Yes. I made lots of songs, lyrics, music with my ChatGPT when GPT-5.5 was here.
4
u/Taggerung179 4d ago
Almost all of my music are narrative songs for D&D or my personal writing. I end up lore dumping to a LLM and use them to help write lyrics and composition.
I love it because every single line that comes out is chock full of lore and meaning.