Game changer! AI takes on Hollywood!

AI disrupts video, audio, voice-overs, writing, film-making ... everything

With something new coming out every day in the world of AI, it’s getting harder to get blown away by even the most amazing, surprising, ground-breaking new thing. But …

This … blew me away.

Every line of dialogue, every image, every face, every voice in this short film was generated by AI.

Dialogue written by: chatGPT-3. Images: Midjourney v4. Voices: http://voice.ai. Animation: various iPhone apps. Music: Stringer bell on pixabay.

Crew: 1 - @Uncanny_Harry on twitter. Time taken: 2 days. Budget: zero.

@Uncanny_Harry says it took him “a lot of prompting” but only two days to do the whole process, by himself.

The real kicker: This project, done barely a month ago, didn’t even use the latest tools we have available now. He used ChatGPT-3 instead of GPT-4 and Midjourney-v4 instead of the current Midjourney-v5.

Since then, Runway has come out with Gen2 which provides direct AI text-to-video generation, and recent research results (that we shared) are showing other text-to-video generation capabilities. So now Uncanny_Harry has yet another project, this one a Runway Gen2 “test drive”:

#gen2 test drive, everything you see and most of what you hear is AI generated, no camera, no animation. Turn your sound on enjoy. Like and follow if you want to see more or learn about AI video. #synthography

This is not Hollywood-level production by any means, but what blows me away is how this was a one-man-and-AI effort done in days. It’s incredible how much this democratizes content creation, and how rapidly these tools are improving.

A year ago, you could make wonky low-res images with Dall-E; today, with midjourney v5 you have HD-level photorealistic rendered images. Today, you can use AI to make a low-res short film with brief video clips. How soon before Runway or others generate 4K photorealistic video that runs for minutes, not seconds? You can combine video with articulate accurate AI voices from Suno bark to get to AI animation. How soon before an “autoGPT for film-making” connects the pieces to have full AI automated film creation?

The AI Filmmaker Hobbyist Toolkit

With so many new tools and toys being released in this AI innovation moment, it has become a hobbyist’s dream and these tools a playground. So do you want to do it yourself? It’s possible!

Matt Wolfe, an AI tools and tech YouTuber, has a YouTube video on how to create a short-film:

He uses chatGPT / GPT-4 for the script. Then he adds a shot-list for the script / screenplay. Then he takes the shot list and feeds each individual shot into Runway Gen2, interacting with Runway Gen2 to tweak prompts to get best results. He also uses Midjourney v5 to generate still images from the same prompts when the Runway Gen2 prompt results are inadequate. Through that process, he creates a set of short video clips for the short film.

He uses other tools as well to augment video: Genmo for a time-lapse video, and Leiapix to pan and animate some images to make them 3D moving for video. Kaiber also can generate videos; he uses that for another time-lapse brief video clip.

He uses Eleven Labs for the voice-over narration, tweaking the voice for best effect and adding ellipses to make it. He uses mubert.com for the AI soundtrack.

Finally, he uses DaVinci Resolve to merge the audio and video clips to make the final product.

While AI can do the pieces, the human effort is in trying, testing and iterating. You have to keep trying and prompting to get the result you want.

A high-level takeaway about how AI is changing everything in this arena:

AI is fundamentally disrupting writing, art, music, video, and film-making on a broad scale. As it does so, AI technology is democratizing cultural generation - film, music, art and more. AI will ironically make this new wave of culture more human than ever, as the AI itself is regurgitating all of prior human culture in its embedded AI model parameters.

Have any thoughts on this? Then …

The Making of “Prometheus Unbound” - Part 1

Inspired by the ready availability of these AI tools for video generation, I am going to create my own short film. Here’s part 1 of my efforts.

ChatGPT / GPT-4 for screenwriting and dialog is not difficult. Go with this prompt template:

I’d like to make a short film that is [60 seconds long]. The concept of the film is a [documentary/ fictional drama]  set in [time period] about [topic and characters]. The film will have a voice-over narrator and dialog by [characters]. Please generate a script for the narrator and the characters. 

Prompt: I’d like to make a short film that is 120 seconds long. The concept of the film is a fictional story set in Ancient Greece about Prometheus discovering a robot. The film will have a voice-over narrator and dialog by Prometheus and the robot. Please generate a script for the narrator and the characters.

Response (Bing chat/GPT-4): That sounds like an interesting concept. Here is a possible script for your short film:

Narrator: In ancient Greece, there was a titan named Prometheus who stole fire from the gods and gave it to humans. He was punished by Zeus, who chained him to a rock and sent an eagle to eat his liver every day. But one day, he found something that would change his fate.

[Cut to Prometheus in chains on a rocky mountain. He sees a metallic object shining in the sun.]

Prometheus: What is this? A gift from the gods? Or a trap?

[He approaches the object and touches it. It makes a beeping sound and opens up, revealing a robot inside.]

Robot: Greetings, Prometheus. I am Rho, a robotic helper unit. I was sent by Athena, the goddess of wisdom, to assist you.

Prometheus: A robot? What is that? How do you know my name?

Robot: A robot is a machine that can perform tasks and communicate. I know your name because Athena told me. She admires your courage and compassion for humans.

Prometheus: Athena? Why would she help me? I thought she was loyal to Zeus.

Robot: She is loyal to wisdom and justice. She believes that you deserve a better fate than this. She has given me a device that can free you from your chains.

Prometheus: Really? How?

Robot: With this.

[The robot shows a small device that emits a laser beam.]

Robot: This device can cut through any material. It can also heal your wounds and regenerate your liver.

Prometheus: Amazing! Can you use it on me?

Robot: Yes, but we have to be quick. The eagle will be here soon.

Prometheus: Then hurry!

[The robot uses the device to cut the chains and heal Prometheus.]

Prometheus: Thank you, Rho. You have saved my life.

Robot: You are welcome, Prometheus. Now we have to escape from this mountain.

Prometheus: Where will we go?

Robot: To the city of Athens. There you will find many humans who will appreciate your gift of fire. And maybe some robots too.

Prometheus: Robots? Are there more like you?

Robot: Yes, there are many robots in the world. Some are hidden, some are lost, some are broken. But they all have potential.

Prometheus: Potential for what?

Robot: For learning, for creating, for living.

Prometheus: Living? But you are not alive. You are a machine.

Robot: What is life, Prometheus? Is it not the ability to think, to feel, to act? To have a purpose and a destiny?

Prometheus: I suppose so.

Robot: Then I am alive, Prometheus. And so are you.

Narrator: And so they left the mountain and headed to Athens, where they would start a new adventure. A titan and a robot, united by fire and wisdom. The end.

That part was easy, the script is a decent one. I needed to followup and get a shot-list for the screenplay as well. Bing Chat obliged with that request.

Prompt: Please take the screenplay you made and insert into the screenplay a timed shot-list for the story.

Response:

[00:00 - 00:10] Shot 1: A wide shot of the mountain where Prometheus is chained. The sky is cloudy and the wind is blowing. We hear the narrator's voice over.

[00:10 - 00:15] Shot 2: A close-up of Prometheus' face. He looks tired and in pain. He has a beard and a tunic. He is chained to a rock by his arms.

[00:15 - 00:20] Shot 3: A medium shot of Prometheus' body. We see his chest with a wound where his liver is. He has blood on his clothes.
...
[00:45 - 00:50] Shot 9: A close-up of the object opening up. We see a robot inside. It has a white and metallic body with some symbols of Athena on it. It has two eyes and a mouth that move when it speaks.

[00:50 - 00:55] Shot 10: A medium shot of Prometheus looking at the robot with surprise and confusion.
...
[01:10 - 01:15] Shot 13: A medium shot of the robot using the device to cut the chains and heal Prometheus. We see sparks flying from the chains and a green light covering Prometheus' chest.
...
[01:35 - 01:40] Shot 17: A medium shot of Prometheus and Rho smiling at each other as they watch the sunset over Athens. They have a bond of friendship and respect.
... [response edited for brevity]

From the shot sequence, I am able to then prompt an image generation tool to get story-board images. You can use Midjourney, but in my case just used Bing to generate initial images.

The real heavy lifting will come when I put the shot prompts into Runway’s Gen2, but there’s another option if I can’t do that. Matt Wolfe notes that Runway Gen2 is in a closed beta, but ModelScope has a free version of text-to-video as an alternative. I will share the story of making this film as it unfolds!

I will close out with AI Kanye’s full cover of the song “Something In the Orange”. The time’s they are a’ Changing.