Future of Elder Scrolls & RPG - Skyrim NPCs & Inworld AI (like GPT-4 for gaming)

Real-time convincing-sounding TTS is extremely close - I’d be surprised if we don’t get an elevenlabs equivalent to gpt-turbo before the end of summer. I spent an hour or three over the weekend messing around in Skyrim VR with a mod that does player speech recognition, pipes it out to GPT with identifier tags to give scene context, sends GPT output to elevenlabs (optional), and then the mod integrates it into Skyrim mouth rigging etc. Yes, there’s an extremely obvious lag before you get a response, but it’s on the order of seconds even though this is an early Skyrim mod with a ton of moving parts interacting. And the result is…astounding. As someone said in a comment on the below linked video: this is the biggest leap for video games since Half Life 1. [VIDEO]

I saw a demo of a Skyrim mod on Youtube [1], where they wired the dialog screen with NPCs in the town of Windhelm up with the OpenAI API. Each character had all their Bethesda-written dialog fed to the LLM as a starting prompt, so each one communicated with the player within the bounds of their authored identity, and knew what that character was written to know. The human player used freeform in-game text-to-LLM as input to talk to the people in the town square. The latency was large, but not more than a few seconds. He was able to walk up to someone and say things like "Where do you live? Who is your father?" .... latency pause ... "Oh me? I live in the Cloud District. It's where all of us Battleborn clan live, of course!" The real kicker of this demo was that they passed the LLM-generated response dialog in to a deep-fake model trained on each character's original voice actors (since the corpus in Skyrim is vast, this ended up quite accurate). All of this was a major generational shift for gaming as a whole. I had just witnessed the first time a video game player could actually TALK to an NPC, in a fully unstructured and unscripted fashion, while each character remained as in-character as the original game's authors intended. So maybe this purchase by OpenAI is about something like that. Building NPCs for Biomes.gg that you can actually talk to. [1] [VIDEO]

Now I'm just imagining a video game with characters each having their own fine tune applied on top for their dialog. I'm guessing you could use some relatively small models. In each case you would be feeding all the context to the model (player name, current relevant quests, summary of previous interactions, etc). Though maybe fine tuning/training isn't even needed and a good enough prompt will work (Not sure what all they used for this [0]). I'm excited for the first AAA game that tries this. Anyone that has played a RPG-style game knows that after a few times going into a city (or a couple play-throughs) the dialog feels repetitive. I love the idea of Skyrim but with better dialog. You could either run the models on the user's computer or maybe just run it on the backend so you can block certain generations (wrong/misleading/"unsafe") and just ship updated dialog lists to the client occasionally. [0] [VIDEO]