With all the noise around AI video right now, how do you filter the reality from the hype?
For me, the answer was to make something. And so I created this piece of work, which uses a mix of techniques and tools (Claude, Higgsfield, Midjourney, Runway, ElevenLabs, Premiere Pro) to tell a true story.
The project has helped crystallise my view on the current state of play:
– the results can be very good, and the savings in time and cost vs traditional production are real
– but the tools are clunky, technically limited and thirsty for tokens
– and the process can be imprecise, requring you to sometimes settle for what you can get
Perhaps the biggest revelation? There is no magic button – you still need real creative and production expertise to maximise the benefits.
If you want an honest, hype-free appraisal of how AI video could work for you, I’d love to chat.
Or, If you want to geek out on the specifics for this project, check out the scene-by-scene breakdown below.
SCENE-BY-SCENE:
1. The boy in the cinema
The big win for me with this scene was to get the look right, as well as the character. Having lived the experience, I had a very specific aesthetic in mind, which, working with Claude and Higgsfield, I was able to get very close to, and define as a style that made further generations stylistically consistent. Continuity was challenging though. My experience was that we could get the character face right, but when we tried to generate shots for coverage, found it tricky to maintain continuity between shots – wardrobe, environment, background extras. That was an issue within and across tools – Higgsfield, Runway, Midjourney – as shown in some of the cultish failures of the reverse shot of the boy watching the TV screens. The tech really struggled to get this even close, let alone right. But I pushed through, eventually having enough front-on generations to be able to cut a sequence together that works. And persistence paid off for the reverse tracking shot, which came out right at about the exact point I was going to give up. Not so easy, and not so cheap.
A FEW FAILS:
2. Camera evolution
Narratively, it was tricky to work out what equipment to cover – there have been so many technical leaps in filmmaking over the years – to cover everything would make it a ten-minute promo. So I decided on a simplistic journey through the evolution of cameras to illustrate the idea. In a technical sense, this was the easiest sequence as AI was able to render the 3D models and animations pretty easily. Pieced together with traditional edit techniques, because you can’t be too exact in the timing of the shots you generate and each camera needed to generate as a separate shot. One basic failing was that the AI generations didn’t handle simple, dark/neutral gradients very well – there was heavy colour banding in the backgrounds that needed traditional editing to resolve.
3. “The Only One” content ecosystem and Internet animation
Went a little down the rabbit hole for this sequence, as I had some real ambition for the internet animation that the tech didn’t quite meet. My hope was that we could flow through from the movie scene on the smartphone, to an exploded view of the whole content ecosystem around a film, which we could use to populate the screens for the Internet scene, and provide a narrative link to the end scene. It works OK, but is probably a good illustration of the need to “settle” in some instances with AI video, which still has limits. It was fun to develop a whole fictional film, including the synopsis and main character, and extend that to a content ecosystem. I got some lovely shots and pieces of content, with a very consistent character, but the tools failed to populate them with 100% success into the smartphone at the start of the sequence, and the Internet screens at the end. In this instance, a human motion designer would have done a better job, but would have cost more money and taken more time, neither of which I had a lot of. So I settled for what I had.
SOME CONTENT ELEMENTS:
4. The gallery of hands
Moving onto the representations of AI past, present, and maybe future, I started with a gallery of hands, which I hoped would be a quick “get” for the audience – dodgy hands being a bit of an AI video trope and all. As with most scenes, this started with generating stills to establish the elements and aesthetics, which was a bit of a battle. Depending on the tool, I could get the elements right, but not the environment, and vice-versa. Eventually, I settled on the look we have, which wasn’t as photo-realistic and cinematic as I wanted, but I couldn’t elevate the look without getting some super-weird results for the hands. The camera move was another story, and had to end on the exact hand in the same position, with the same lighting as for the next scene, which was very tricky and largely failed badly. But I got close enough to make it work with a bit of traditional fudgery.
A FEW FAILS:
5. The influencer
There’s a lot going on here. Started with defining the “influencer” character, in this case a fashion influencer / brand ambassador. I chose that character as a generic representation of the kind of editorial/commercial content that might be executed at scale with AI. Establishing the initial character was easy enough, and gave me something to interpret into video with basic prompts for consistency, but with subtle variations for wardrobe and product shape. Getting those variations right took a lot of generations and so it was was the most expensive sequence in the promo. AI couldn’t work out the composite shot – once again, I got some great executions in terms of style, but none of the models I tried could hold the content together, in sync, so traditional post was used to create that shot. Continuity through to the “bad” influencer was also tricky, and I got some weird variations, but the one I landed on was really great and close to what I briefed Claude to help me prompt. I hope this is a funny sequence to watch, because it was fun to put it together.
A FEW FAILS:
6. The girl in the lounge-room
The opening and closing scenes were the most “directed” in the promo, in that I defined the aesthetic with as much detail and thought as if I was working with a real crew. The limitations of AI in that sense are that it isn’t really a collaboration, rather I’m telling the machine what I want and getting there through trial and error. Human beings offer a lot more creatively and collaboratively, and I think ultimately deliver a better outcome through that process. But that all said, defining the character/s, environment, lighting, wardrobe, action and coverage can still be a creatively rewarding process, which I found with the scene of the girl in the lounge-room watching the film. I experienced similar issues around continuity, character consistency and building a complex tracking shot as with the opening sequence, which speaks to the limitations of the technology. But again, I pushed through to get enough shots to build a sequence from, if not exactly the sequence I had in mind.
7. “The Only One” film scenes
The film the girl is watching is “The Only One”, the made-up movie I developed for the earlier Internet sequence. The actor, “Humanity Baud” was made up too, and was able to be executed consistently in a lot of different shots and contexts. This was one of the more cinematic sequences we generated. The scope to offer large-scale production value, with improving consistency, for a greatly reduced budget, is one of the more exciting aspects of AI video. I would caveat that with the reality I experienced that it is not easy to work with sync dialogue and generate genuine emotion from the performances. Most people would follow that up with “yet”, but my vibe is that hybrid production will be the method used to solve those issues – the more intimate-scale shots of real people doing real performances composited into AI scenes, intercut with high-value shots that don’t rely as much on performance to add spectacle.