About Posts

Experiments in AI

A few weeks ago, I decided to finally play with GPT-3 seriously. I've been thinking of applications for over a year now, but hadn't started concrete work yet. This page documents some experiments.

Background

GPT-3 is a giant Transformer neural network. It was essentially trained on Common Crawl, a large swath of the internet. The very basic idea is that given a few thousand words, it predicts the next word.

I was making n-gram Markov chain models back in 2013 and earlier. Those are just very simple statistical models that take the last few words and predict the next word. You often get hilarious results, such as the fake Hackernews post titles "PHP Sucks But I Didn't" and "No, really, pi is wrong: The NSA slide you haven’t seen".

GPT-3 is much more advanced than this. It takes in thousands of words and actually can extract useful patterns from them.

For example, you can give it the prompt "blah, bleh, blih, glah, gleh," and it will correctly predict "glih", even though none of these are real words and this sentence was almost certainly not in its training data. It has learned a pattern at runtime.

You can trick it into having conversations. For example, you can give it the prompt

The following is a transcript of a phone conversation between a salon receptionist and a woman named Madison Baker trying to get an appointment scheduled Thursday after 6pm. Earlier is better. Salon: Hello, you've reached Susy's Salon, how may I help you? Madison:

and GPT-3 spits out "Hi, I'm looking to schedule an appointment for Thursday after 6pm."

Imagine you now hook this into a text-to-speech program and feed that to your phone call, and then speech-to-text back into GPT-3:

The following is a transcript of a phone conversation between a salon receptionist and a woman named Madison Baker trying to get an appointment scheduled Thursday after 6pm. Earlier is better. Salon: Hello, you've reached Susy's Salon, how may I help you? Madison: Hi, I'm looking to schedule an appointment for Thursday after 6pm. Salon: Let me see.... I have a 6:30 and a 7:30 slot left, would either of those work? Madison:

We give this prompt to GPT-3 and it says "6:30 would be great." Repeating this process, we get the full conversation:

Salon: Hello, you've reached Susy's Salon, how may I help you? Madison: Hi, I'm looking to schedule an appointment for Thursday after 6pm. Salon: Let me see.... I have a 6:30 and a 7:30 slot left, would either of those work? Madison: 6:30 would be great. Salon: Perfect, we'll see you at 6:30 on Thursday then! Madison: Thank you. Salon: Buh-bye! Madison: Bye!

GPT-3 generated all of the "Madison" lines. The "Salon" lines were generated by me, pretending to be a salon receptionist. I predict by 2024, all of this functionality will be integrated into Siri and friends, and we'll enter the era of Actually Useful AI Assistants.

Story Generation

Inspired by DALL-E 2 and Stable Diffusion, I started to appreciate that the most practical use case for the recent spate of large-language models is in creative rather than technical work.

The image generation models have made huge inroads in that modality, such that complete pieces can be generated. Text, however, is lagging behind. I figured I'd take a stab at generating stories.

Given GPT-3's limit of a few thousand words, it can only really generate a page or so of text that is coherent. Imagine you're an author without long term memory retention. You can only remember the last couple of pages you wrote. How could you write a whole book that way?

I feel like you'd need to take copious notes, and use a very algorithmic approach. My initial thoughts was a prompt like the following. Mentally replace the <descriptions> with actual content.

The following is a summary of a story, and an excerpt of a few paragraphs. The author is an award-winning, critically acclaimed writer. Summary: <a paragraph summary of the plot> Characters: <a list of characters and their traits> Summary of the current chapter: <a paragraph summary of the current chapter> Excerpt: <the last 5 paragraphs written> Next paragraph:

Then, you'd use this prompt to generate the next paragraph. You'd use a different prompt to extract new character details that the new text might introduce, and merge that into the <character summaries> field on the next prompt. You'd do the same thing to the story summary and chapter summary, using a different prompt to extract information and update the fields. Then you'd recursively feed this back in to generate the next paragraph, etc.

When you finished a chapter, you'd use a prompt to generate a plot outline of the next chapter, then recursively feed that in, etc. In this way, you could maybe work through a whole short story.

Let's try it out.

Generating plot ideas

GPT-3 works well when you give it a few examples to learn from. I've found 3-5 examples is usually sufficient. I used this prompt to generate story ideas:

A diverse group of 5 critically acclaimed film script writers wrote their top story ideas: 1. A sci-fi / mystery story in which weird things start happening, e.g. power outages, unidentified radio traffic, etc. The protagonist is an AI researcher who spends his spare time trying to create AGI. Eventually he realizes that the mysterious happenings were caused by his own AI that escaped his experiments and got loose on the internet. The story ends when the AI takes over the control system to a rocket launch and fires a satellite into deep space. Presumably, the AI has loaded a copy of itself onto the satellite and has now escaped Earth. It's unclear if the AI is still present on Earth or not. 2. A high-fantasy story in which a girl is an outsider in her village. The village shuns her because they believe she has evil powers of foresight. She has dreams that seem to tell the future. If she dreams of rain, it rains. If she dreams of a fire, there is a fire. One day, a traveling monk comes to town and hears of the girl with the gift of foresight. He explains to her that her dreams don't show the future, but actually create the future. She must have a god-spirit as a distant ancestor whose power has been passed down to her. She decides to leave with the monk and trains at a temple to master her mind. This involves years of trials and tribulations, growth, happiness, and sadness. One day, she hears her village is in trouble -- a drought. She travels home and uses her mental powers to save the village. Everyone loves her now, which makes her feel like less of an outsider. They welcome her back in, but she realizes she has grown beyond this village and can't be happy here. She leaves and becomes a traveling monk. Years later, she enters a village with an outsider boy who has powers. The story ends with her completing the cycle, taking guardianship of the boy and beginning to train him. 3. A pulpy / noir detective story, but set in Edo period Japan. A wandering ronin samurai plays the "loose cannon cop" character who is recruited to solve a mystery. The basic plot is that the daughter of the local warlord has been kidnapped by yakuza gangsters and sold into prostitution. The unnamed ronin, who is just called "the ronin" for the whole story, investigates the kidnapping using his underworld connections and prodigious swordsmanship. After cutting down many yakuza, he finds the daughter, only to realize she wasn't kidnapped at all. She ran away with her lover, a yakuza leader. The yakuza leader is impressed by the ronin's skill as a samurai and asks if he wants to join the gang. The ronin is too independent to do it and leaves town to go find other adventures. 4. A techno-utopian alternate history in which an ancient Maya priest has a hallucinogenic trip on an indigenous plant and has a flash of insight into how to go from their hunter/gatherer society to an industrialized society powered entirely by green energy. For example, he invents the water mill, and a wind powered sawmill. He teaches his people they should only use resources they know how to renew, such as wood, and only use metals that they can recycle. He teaches this because during his trippy vision, he realizes that doing things quickly is no different than doing things slowly. Things will get done in one lifetime or the next. The most important thing is sustainability. So they slowly develop a green industrial base with water and wind power, using low-impact technology like wood and a few metals (sparingly). At some point, the Spanish conquistadors show up according to history, but this time the Maya easily defeat them with warriors mounted on wind-surf boards, trebuchets, rapid-fire crossbows, and other fantastical low-tech devices. 5.

These are all just plot ideas I came up with. I am not a good writer, I wouldn't be tricking the AI to write for me if I were!

Some outputs the AI gives for this prompt are:

A heart-wrenching drama set in the not too distant future where a mother is forced to watch her child suffer from a painful, incurable disease. The child is in so much pain that the mother starts to believe that mercy killings are the only way to end her child's suffering. She begins to look into ways to end her child's life painlessly and with dignity. She meets with resistance from her husband, who is a devout Christian, and from the rest of her family. She also faces legal hurdles, as mercy killings are not yet legal. The story follows the mother's struggle to come to terms with her decision and to find a way to carry it out.
A story set in a future dystopia in which the rich have retreats in space, while the poor are left to fend for themselves on a ravaged Earth. The protagonist is a young woman who is born into the wealthy elite, but she rebels against her privileged life and joins a group of freedom fighters who are working to take down the power structure. She falls in love with a poor man who is fighting on the side of the oppressed. Together, they find a way to expose the truth about the ruling class and inspire a revolution.
A story set in the future where an augmented reality game has taken over the world. The game, called "Existence," is a massively multiplayer online game that is so realistic and addictive that people have stopped living in the real world. The story follows a group of people who are trying to break free from the game and rediscover the real world. They have to battle the game's artificial intelligence, which is trying to keep them trapped in the game. They also have to battle other players who are addicted to the game and don't want to leave. eventually, they succeed in breaking free, but they are changed people. They have to find new meaning in their lives outside of the game.
A coming-of-age story in which a spaceship crash-lands on a hostile planet. The surviving crew must band together to find a way back home. Along the way, they face deadly creatures, treacherous terrain, and their own personal demons. The story follows the crew as they struggle to survive and ultimately find a way home. Along the way, they discover that the planet is not as hostile as they thought, and that there may be more to this planet than meets the eye.
A heartwarming story of a woman who returns home to care for her ailing father. She finds that her childhood home is in disrepair, but she is able to fix it up with the help of her father and the community. She also finds love along the way.

Improving plots

The generated plot ideas are nothing special. Can we use a prompt to improve them?

The following are excerpts from a story plot editing session: Story: A post-apocalyptic story in which the world has been ruined by a nuclear war. A small group of survivors are living in a bunker. They are able to eke out a living, but it's a dark and dreary existence. One day, one of the survivors, a woman, finds a baby on the surface. She brings it back to the bunker and the group takes care of it, teaching it their ways. The baby is the ray of hope in their dark world. As it grows up, it becomes evident that the baby has special powers. It can make things happen just by thinking about it. The baby grows up and becomes the leader of the group. One day, they find another group of survivors and there is a confrontation. The baby uses its powers to stop the fighting and create a utopia. Critique: Post-apocalyptic stories are fine, but a little overplayed. What makes this one interesting? Also, the summary should include how the baby becomes the leader of the group, as this is an important plot detail. What kind of man (or woman) does the baby become? Also, when they confront the other group of survivors, the baby is already grown up, so it shouldn't be a baby anymore. The plot summary should describe how the utopia is created, and what kind of utopia it is. Story: <insert plot summary> Critique:

Once we get the critique, we can prompt the AI to rewrite the summary with the critique in mind with this prompt:

The following are excerpts from a short story plot editing session. An original story idea is presented, then a critic offers suggestions for improvement. The author then writes a new story idea that combines the original idea with the suggestions. Original Story Summary: {pitch} Critique: {critique} Revised Story Summary:

Running this algorithm, we get the following sequence:

Original story idea:

A Coen Brothers-esque black comedy set in the modern day film industry. The story follows a struggling screenwriter as he tries to navigate the cutthroat world of Hollywood. He constantly faces rejection, but keeps getting back up and trying again. He's finally able to catch a break when he's hired to write a low-budget horror movie. He pours his heart and soul into the script, but the studio executives are only interested in making changes that will make the movie more "commercial." He eventually has a nervous breakdown and quits. The movie is released and becomes a huge hit, but he never gets any credit. He's left wondering what could have been.

Critique:

The plot of this story is quite thin. What else happens to the protagonist? Does he ever find success? Perhaps the story could focus on his relationship with his family and friends, and how they deal with his failures. There could also be a love interest who motivates him to keep going. As it stands, the story feels unfinished.

Revised:

A struggling screenwriter tries to navigate the cutthroat world of Hollywood. He constantly faces rejection, but keeps getting back up and trying again. He's finally able to catch a break when he's hired to write a low-budget horror movie. He pours his heart and soul into the script, but the studio executives are only interested in making changes that will make the movie more "commercial." He eventually has a nervous breakdown and quits. The movie is released and becomes a huge hit, but he never gets any credit. He's left wondering what could have been. In the meantime, his personal life falls apart. His wife leaves him and he becomes estranged from his children. He eventually bottom out and is living on the streets. He's able to get his life together and makes a comeback with a new movie that's a critical and commercial success.

You can run this revision process this any number of loops. Let's send one through 5 loops.

Original story idea:

A space opera in which the protagonist is a cyborg who was born on a mining colony and has always dreamed of going to Earth. One day, a ship arrives at the colony, and the captain of the ship offers to take the protagonist to Earth. The protagonist is hesitant at first, but decides to go. On the journey, the ship is attacked by pirates, and the captain is killed. The protagonist takes control of the ship and leads the crew to safety. When they finally arrive at Earth, the protagonist realizes that it is not the utopia they were led to believe. The Earth is a harsh, cold place, and the people are xenophobic and suspicious of cyborgs. The protagonist must find a way to fit in and survive in this new environment.

After 5 critique loops, in which the critic asks for more details and more plot development on Earth:

Sarah, a cyborg, is born in a space colony. She has always dreamed of going to Earth, and one day, the captain of a ship that arrives at the colony offers to take her there. Sarah is hesitant at first, but decides to go. On the journey, the ship is attacked by pirates, and the captain is killed. Sarah takes control of the ship and leads the crew to safety. When they finally arrive at Earth, Sarah realizes that it is not the utopia they were led to believe. The Earth is a harsh, cold place, and the people are xenophobic and suspicious of cyborgs. Sarah must find a way to fit in and survive in this new environment. Despite the challenges, Sarah perseveres and finds a place in a small community of outcasts and misfits. Sarah is able to use her knowledge and skills to help the community face the challenges they encounter. They eventually come to understand and appreciate the people of Earth. This is a story of Sarah's journey, from outsider to leader, and the importance of determination and hope.

Generating outlines

The next step is to generate some basic chapter outlines. I am kind of thinking about this in terms of describing a series of movie scenes, like each chapter is a sort of scene.

I take one of my plot ideas from above and break it into concrete scenes to use as an example prompt:

An award-winning movie screenwriter takes plot premises and imagines a series of detailed scenes that comprise a movie based on the premise. Scenes include a setting which describes where the scene takes place, and when the scene happens relative to the last scene, and a sequence of events that define the action of the scene. Plot premise: A high-fantasy story in which a girl is an outsider in her village. The village shuns her because they believe she has evil powers of foresight. She has dreams that seem to tell the future. If she dreams of rain, it rains. If she dreams of a fire, there is a fire. One day, a traveling monk comes to town and hears of the girl with the gift of foresight. He explains to her that her dreams don't show the future, but actually create the future. She must have a god-spirit as a distant ancestor whose power has been passed down to her. She decides to leave with the monk and trains at a temple to master her mind. This involves years of trials and tribulations, growth, happiness, and sadness. One day, she hears her village is in trouble -- a drought. She travels home and uses her mental powers to save the village. Everyone loves her now, which makes her feel like less of an outsider. They welcome her back in, but she realizes she has grown beyond this village and can't be happy here. She leaves and becomes a traveling monk. Years later, she enters a village with an outsider boy who has powers. The story ends with her completing the cycle, taking guardianship of the boy and beginning to train him. Scenes: 1. Setting: A village in the jungle. The village is primitive with straw huts. It's mid-day. Events: The protagonist, a girl, is being bullied by other kids. They tease her like "what are you going to dream tonight, freak?". She runs away crying. 2. Setting: Later that night, in a a hut in the village, the hut has a primitive bed. Events: The protagonist has a dream of the mean kids burning in a fire. Her dream seems so vivid, she can hear their screams and smell the smoke. She wakes up in a panic, but everything is calm. She goes back to sleep. 3. Setting: The next morning, in the forest surrounding the village. Events: The protagonist is out picking berries and she smells smoke. She runs back to the village and finds the storehouse is on fire. She hears screams inside. People are rushing to put out the fire, but it's too strong. They find the charred bones of the children inside. The girl feels horrible and guilty that she didn't tell anyone about her dream, but also secretly good that the mean kids were punished. 4. Setting: A year later, in the village. Events: Life has returned to normal. The protagonist is shown interacting with the villagers, as if she has tried to integrate into normal life as she has grown up, but without much success. A stranger comes into the village. He is dressed like a monk. He sees her and looks as if he recognizes her. He walks towards her and tells her to come with him. They walk to a nearby stream and he explains that her dreams actually change reality. She is descended from powerful spirits. He asks her if she wants to come train with him at his temple. Without hesitation, she says yes, and they leave without saying goodbye. 5. Setting: Five years later, at the temple. Events: This scene is more of a montage. The protagonist is now shown to be a young woman. She goes about her daily training routine at the temple. It's a secluded stone building on a mountaintop. Training is tough physically and mentally, but she is now a master of her powers. She can close her eyes and warp reality to be whatever she wants, but it does take effort and can exhaust her. This scene includes an example of her powers, e.g. closing her eyes and making a seed grow into a flower in a few seconds. 6. Setting: A few days later, at the temple, in the early morning. Events: The protagonist wakes up and the temple is quiet. Too quiet. She finds the old monk has died. She finds a scroll he was writing and reads it. It says "The leaf does not blow the wind. The wake does not drive the ship. Your past does not create your future.". She resolves to go back to her village and use her powers for good. 7. Setting: The village, but now we see it has been plagued by drought, crops are dying and people are famished. Events: The protagonist arrives. She does not recognize the gaunt faces at first. And they do not recognize her, because she was only a girl when she left, but now she is grown. The villagers try to turn her away, because they cannot support her, but she smiles warmly and sits in the middle of the village, closes her eyes, and meditates. A few minutes later, a drop of water falls on her face. Then more drops. Before long, the rain is pouring down. The villagers are amazed and grateful, they try to convince her to stay and lead them, even though she is still a stranger to them. But she knows she can't stay. She has grown beyond this simple life. She wishes them well and departs without revealing her identity. 8. Setting: Many years of later, another village, the protagonist is shown to be visibly aged, like she has experienced the world. Events: The protagonist enters the village, and sees a boy being bullied by kids. The kids say "stop having stupid dreams, you loser!". The boy cries. She walks towards him and holds out her hand. The story ends like it began, now she is the master monk. Plot premise: {pitch} Scenes:

So, iterating on our pipeline we've built, we feed it the 5-loops-revised plot premise:

The stranger is a post-apocalyptic western tale set in the southwest United States. The land has been devastated by a nuclear war, and the few remaining survivors are constantly under attack by raiders from the east who are armed with high-tech weapons. The town is able to hold off the raiders, but they are always on the verge of being overwhelmed. One day, a stranger rides into town from the west. He is a drifter with no home, no family, and no past. He is a master of the bow and arrow, and he offers to help the town defend itself against the raiders. The townspeople are initially suspicious of him, but he quickly proves himself to be a skillful fighter. They eventually come to trust him. He becomes their secret weapon in the fight against the raiders. The stranger's goals are unclear, but he seems to be helping the town for his own mysterious reasons. Ultimately, the stranger leads the townspeople to victory against the raiders. However, in the final battle, he is mortally wounded. Before he dies, he tells the townspeople that he was once one of the raiders, but he left because he couldn't stand the thought of destroying innocent lives. He came to the town to make up for his past sins. The townspeople are moved by his story, and they mourn his death as a hero.

And get the following scenes:

1. Setting: A small town in the southwest United States. The town is made up of a few dozen ramshackle buildings. There is a makeshift wall made of debris and scrap metal around the town. It's mid-day. Events: The town is under attack by raiders. The raiders are throwing rocks and firing arrows over the wall. The townspeople are firing back with whatever they have -- guns, arrows, rocks. It's a desperate battle. The town is about to be overrun. 2. Setting: Suddenly, a stranger appears on a hilltop overlooking the town. He's a drifter with a backpack and a bow. Events: The stranger watches the battle for a moment, then nocks an arrow and starts firing at the raiders. His arrows are incredibly accurate, and he takes out several raiders with a few well-placed shots. The townspeople are amazed. The tide of the battle starts to turn. 3. Setting: After the battle, in the town square. Events: The townspeople are gathered around the stranger. They are questioning him, asking him where he came from and why he's helping them. He doesn't say much, but he offers to stay and help them defend the town. They are initially suspicious of him, but he quickly proves himself to be a skillful fighter. They eventually come to trust him. 4. Setting: The next few weeks, montage. Events: The stranger is shown helping the townspeople to fortify the walls and train for the next attack. He is a master of the bow and arrow, and he teaches the townspeople how to fight. They are starting to see him as their secret weapon in the fight against the raiders. 5. Setting: One night, around the campfire. Events: The townspeople are gathered around the campfire, singing songs and telling stories. The stranger is sitting off to the side, alone. The mayor's daughter, Liza, comes over and sits with him. They start talking. He tells her about his past, how he was once one of the raiders, but he left because he couldn't stand the thought of destroying innocent lives. He came to the town to make up for his past sins. Liza is moved by his story, and she starts to see him in a new light. 6. Setting: The next day, the town is attacked again. Events: The raiders are more prepared this time, and they have better weapons. The townspeople are fighting bravely, but they are outnumbered and outmatched. The stranger is leading the defense, but he is wounded in the battle. He kills several raiders before he finally falls. 7. Setting: The town square, a few days later. Events: The townspeople are gathered around the stranger's body. They are mourning his death. He was a hero. He saved the town, but at the cost of his own life. Liza is crying. The townspeople are vowing to never forget his sacrifice.

This is a pretty simple but decent and coherent plot! The next step is to turn these scenes into actual chapters.

Stay tuned, I will update this page as I get more time to experiment.