Creating a dynamic conversation with an NPC: ChatGPT in Unity (2023)

ChatGPT + WhisperAPI+Unity+ReallusionCC4+iClone8+ElevenLabs+SALSALipSync

I wanted to create a non-player character to be able to talk to in VR and have them respond to our questions through speech and lip-syncing accordingly. In this demo, I told the NPC to respond as they were Dolores from the Westworld TV series.

Step 1: Speech to text -> OpenAI WhisperAPI

Step 2: Text(question) to text (answer) -> OpenAI ChatGPT3.5

Step3: Text to speech -> ElevenLabs API

Step4: Speech to LipSync -> SALSALipSync

Character created with ReallusionCC4 and iClone8