A lore-aware game character

Summary

A self-hosted conversational NPC for an MMORPG.

An in-game character had to answer within the game world while respecting player progress and changing lore.

A fine-tuned Llama 2 character with retrieval over game knowledge, progress-based filtering and infrastructure for serving the model.

Tech Challenge

• Scale: MMORPGs, by nature, cater to many players simultaneously. The Oracle NPC, being a unique feature, was expected to be frequently consulted by thousands of players. This meant the system had to handle a massive volume of queries in real-time, ensuring each player received accurate and timely responses without any noticeable lag.

• Latency was a design constraint: responses needed to arrive promptly enough for an in-game conversation.

• Personalized Knowledge: The Oracle's knowledge had to be adaptive, revealing information based on a player's progress. This meant the system couldn't simply provide generic answers. It had to recognize each player's achievements, unlocked content, and current game status, tailoring its responses accordingly.

• Privacy: While cloud-based solutions are prevalent, the client wanted to host their LLMs to ensure better control, customization, and data security. This introduced challenges related to infrastructure, maintenance, and seamless integration with the game's existing systems.

• Patches: MMORPGs are dynamic, with new content, quests, and lore frequently added. The Oracle NPC had to be designed to quickly assimilate this further information, ensuring its knowledge base remained up-to-date and relevant without requiring extensive manual intervention.

Solution

• Rust was used for the serving layer, with attention to concurrent requests and memory safety.
• GGML was used for inference and quantisation to address response latency and serving-resource requirements.

• To make Oracle's knowledge adaptive and tailored to each player's progress and to have up-to-date info on new patches, we've built several data pipelines to transform and feed the data into Milvus and FAISS. These technologies facilitated efficient knowledge storage and retrieval, allowing the system to quickly access relevant information based on a player's achievements and game status. Also, additional metadata-based tweaks were done to efficiently answer SQL-like questions (e.g., counts, time period-based info, comparisons, etc.).

• The decision to host their LLMs introduced infrastructure challenges. However, we've achieved dynamic load balancing and scaling by utilizing Kubernetes (EKS) with AWS. This ensured the system could handle varying loads, from quiet times to peak player activity, without any hitches.

• Fine-tuning LLaMa 2 with Python & PyTorch: The Oracle's unique voice and integration of game lore were achieved by fine-tuning LLaMa 2 using the HuggingFace toolset powered by PyTorch. The instruct dataset gave the Oracle a specific style, seamlessly weaving the game's lore into the LLaMa 2 weights. Additionally, SetFit was trained to optimize the embedding space, ensuring Oracle's knowledge remained relevant and contextually accurate.

Impact

A conversational NPC architecture combining character behaviour, game context and private model serving.