This music video doesn't have the virality potential of a Lil' Vader - I know that well; but the future of AI media is realism. There's a long way to go to get there, but most of what delivers the realism is in the process (prompt+craft+edit). The tools themselves are capable of it, and without budget constraints you can absolutely create flawless realistic videos.
Music videos are a great challenge because of the necessary lipsyncing and because any idiosyncrasies become painfully obvious. There are many in this video - but process can smooth them out a lot.
A lot of people have asked me for process; i'll describe it quickly below:
1 - Idea - that comes from you, the human
2 - Work with Grok (or favorite AI) to hone in on the lyrics. Lyrics are very hard for AI to get right, it nearly never does it properly; it still requires a human ear to not be so cringe. But using a tool like Grok you can produce starting lyrics that you can adjust to a final state. The final adjusting of the lyrics works best while listening to the song, and Suno is instrumental in this.
3 - Suno - you can prompt the specific style of the song you are looking for and add the lyrics; at first the output is generic, you have to wrestle it into producing the song that best matches what you want. Suno is outstanding for rapid iteration; listening to song after song with the same lyrics in different styles let's you better identify the weakest spots in a song so you can adjust the lyrics. It becomes an ear thing - you have to listen and train your ear for it.
4 - Grok Imagine - once you have a song you are proud of, you need to generate the reference frames. For this you can use Grok Imagine; the 4 attached frames were generated between Grok and Gemini; i prefer the anamorphic format, so i specifically prompt it and include keywords like cinematic, indie, film, depth of field, and other cinema industry terms. I then format the images in Photoshop, apply color correction (if needed) and a grain filter (that's just my preference). This gives the footage the best starting point for realism.
5 - Runway - video generation is what trips up most people; the trick is actually to view the video track as multiple layers (or better yet, multiple camera angles); you need to treat a video prompting session as if you are doing a live shoot; you will run the prompt multiple times (at least 2x) and ask for multiple angles, creative framing, medium/tight/wide shots and most critical: B-ROLL. B-roll is the footage that allows editing to work, because lipsyncing is never perfect (eventually it will be and it will be much easier, but for now, B-roll helps). That's the first prompt to Runway; for lipsyncing, their Hailuo model is best; it generates 15sec clips, but it can take your audio as reference and chop it up into 15sec clips which you can then import into Premiere to edit as final video. For realism, Seedance 2.5 and Kling 4 are best, but they do not lipsync properly - so you should use them for B-roll or contextual shots. Grok Imagine is very good for realistic footage as well, but not for lipsyncing, so use it for contextual or b-roll. The last piece of the puzzle is prompting Runway for the same take but from unique angles that are silhouetted, over the shoulder, above/below/aerial, that show movement to the beat but not lipsyncing. Between these two takes you can get enough footage for a full music video.
6 - Editing - I personally use Premiere, but any video editing tool will work. I've built custom plugins to make the process smoother, like audio syncing the clips in one click to snap them to the sound track; then it's a matter of watching the clips and trimming/cutting as needed - it's light editing work (can be done in ~1hr is footage is good). From there you add captions and you're done.
That's the process; from my experience, I think AI tools still have room for improvement, but it is more on the process side than anything - they have to understand how filmmakers and editors use footage/transitions/foreground/background elements in order to streamline the process. I think we could automate 90%+ of the edit process today if a few features were added to these tools.
Quote
Mark Soares
@marksoares
Music videos are hard; realistic music videos are even harder. This tune is an original of mine from the POV of Merry and Pippin from Lord of the Rings. It was a quick cut - the process of creating it is collaborative with Grok, and then prescriptive to Suno+Runway
"Back to the
The media could not be played.
