Linden Lab Announces ‘Pause’ On AI Character Design

The Context

Linden Lab, some time ago, as the AI buzz and hype constantly increased, decided that they would try it out in Second Life® as well. There was just the question of deciding on what to do with it.

After some deliberation, and knowing about how the SL residents have been grumbling a lot about “fake accounts” being run by bots, joining groups, begging for money, etc., without any sanction or restriction imposed by Linden Lab, they decided to go to the root of the problem: offering their own bot solution instead.

This had essentially three sub-projects:

1. Create a new website, where residents can design their new bots from scratch, given a few guidelines, as well as giving some background (or ‘persona’) to the ‘bots. They leveraged on the already-existing new user onboarding service, which allows a certain degree of personalisation of one’s ‘starter avatar’ using a Web interface. All LL had to do was to add a few more forms with additional information for the generative AI. This new interface is what LL has called Character Design.
2. Integrate the new bots as part of a regular avatar’s usual account. We all know that Linden Lab is aware of how all the alts created by residents; in fact, the ToS strictly specifies that any Basic Account can create up to five alts — anything beyond that requires a paid account. And this gets enforced — Linden Lab uses digital fingerprinting to figure out who is logging in to create a brand new alt, and, internally, the two get associated to the same RL user, even if nothing is apparent on the ‘My Account’ dashboard. OpenSimulator grids have, long ago, established the principle that users register once, but with that ‘master account’ they can generate as many alts as they wish. Thus, thanks to this new web interface, we can have a taste of the future to come: an unified view of multiple avatars, collectively anchored on a single human users, but with each avatar account completely independent of the others (having its own login, its own dashboard/backoffice, absolutely no shared inventory, and, most importantly regarding privacy issues, no way for the relationship between the individual accounts to be exposed. Ever.
3. Integrate Convai‘s LLM-powered Generative AI with the Character Design. Convai already supports similar integrations with other tools and plaforms for the Metaverse, and their toolkit is targetted at doing precisely that: a code-and-forget, turnkey solution for Metaverse platforms.

In fact, it can be argued that Linden Lab merely did the same choice as they had done when licensing SpeedTree (to get flawless, extremely-high-quality, realistic trees) — a project that sadly never went ahead — or Havok for handling all physics (on the simulators as well as on the viewers), or even Vivox for handling 3D spatial in-world voice chat. And, naturally, the same applies to the web-based SL Viewer, codenamed Zero,¹ which comes from technology (indirectly) provided by Amazon Web Services, where Linden Lab hosts its SL grid servers as cloud-based virtual machines, to be loaded on demand.

And, last but not least, we should not forget the amazing effort in releasing the Mobile SL Viewer, a partnership with Sine Wave Entertainment, which had, for several years, a working Unity-based viewer connected to an OpenSimulator Grid. Since the technology is, well, in truth, almost the same, Linden Lab thought (rather cleverly, I should add) that with a few tweaks here and there they could “backport” the Sine Wave viewer to connect to the Second Life Grid instead. Since Unity can, with exactly the same codebase, produce apps for mobile platforms, it would be trivial to release a mobile version of Sine Wave’s technology. Obviously there were some bumps along the road, but, eventually, the Mobile SL Viewer is now out, and, allegedly, most new users come to SL via their mobile devices (sadly, though, they remain as long as their desktop-based counterparts, but that’s a different issue…). There would be quite a lot to be said about this partnership, worth at least another blog post, but, alas! Let’s tackle the end-of-NPC AI announcement first!

Anyway, what all these companies have in common is that they provide accessory technology (which, however, requires a lot of time to build), thus leaving Linden Lab free to pursue its main, and most important and relevant core technologies: a 3D rendering engine that deals with streaming content and a communications protocol with the servers hosting the Grid. Those are the added value that Linden Lab brings to the package, and which they cannot outsource to anyone, as nobody else has that kind of technology (again, this is arguable and disputable and a separate discussion by itself!).

These, however, are ideal circumstances. In practice, all these products and services have to be integrated into the Second Life ecosystem, and that’s not just flipping a button and hoping for the best. It requires LL to adapt its own technology to be able to integrate with those third-party tools; and, depending on the case, the reverse also might be true. For instance, it is a well-known fact that Vivox, before having been bought by Unity, had a “special” offering designed to be flawlessly compatible with OpenSimulator — tweaked from the very same code that supports integration with LL’s own grid. The ‘basic’, or ‘plain’, Software Development Kit cannot be used with Second Life or OpenSimulator — it needs to be tailored specifically to work with those systems (thus the ongoing work to support voice over WebRTC, which is an open protocol, and vendor-agnostic — meaning that anyone, anywhere, can develop a product that ‘speaks’ WebRTC, and it will be immediately fully compatible with everything else that already supports WebRTC natively).

Well, but that also means assigning LL coders/developers to each and every one of the above projects. The difference, of course, is that whatever changes LL needs to do on their side to be able to use software and services from third parties, they only need to do it once. From that moment onwards, it’s up to the third party to keep their services or software working. LL doesn’t need to worry any more — except, of course, for the monthly fees for each of those services.

There is no reason to think that the partnership with Convai was any different.

Linden Lab’s AI service

After introducing the AI Character Designer in December 2024, and in spite of clearly labelling it as ‘alpha’ software, and as an ‘early access’ service — due to change at any moment — Linden Lab finally decided that they will shut it down by the end of August 2025.

They call it a ‘pause’, in the sense that the concept might be explored again in the future. More specifically, Philip ‘Linden’ Rosedale, founder and currently the CTO at LL, claims that merely doing sophisticated chatbots was not enough — after all, even before the Character Designer was up, lots of people had interfaced ‘bots in Second Life with OpenAI or similar, API-driven, public access Large Language Models providing AI-generated conversations.

The Character Designer permitted some interesting things, such as defining a persona, and allowing the ‘bot to retain some ‘memory’ of past conversations. These are popular features of modern LLMs, even though, in general, the public is not made aware of the fact, nor has a simple interface to change such details.

LL wants much more than that. The main issue they noticed was that the ‘bots lacked an awareness of the surrounding. They could read names of surrounding objects within a reasonable distance — but since 99% of the objects in Second Life are left with its default name of ‘Object’, that is hardly useful. The ‘bots also had some difficulty in figuring out that there could be several people present around it (after all, most commercial LLMs only engage in dialogue with a single user at a time). These were all limitations that made it hard for ‘bots to be able to interact with its immediate surroundings. What LL wants to do now is to have them take snapshots of the environment, identify what those ‘Objects’ are, understand how many people are around, and be able to navigate in a 3D environment without bumping into things, and generally getting its direction right. In other words, ‘bots should be able to understand a command such as ‘come to me’ or ‘sit on that chair’ (deliberately being vague about which chair is meant), and understand what is required of them.

None of that is ‘magic’, or ‘wishful thinking’, or anything of the sort. In the past two years or so, the AI Boom has brought all that to the public-at-large. Pattern-matching on still and moving imags has been increasingly more sophisticated; the ability to correctly analyse and understand which elements are on a scene is something AIs have been able to do for quite some time (pattern-matching was the subject of a lot of AI work in the early Naughts, mostly thanks to search engines and web crawlers requiring that kind of technology to better figure out what pages and related multimedia elements were more likely to be useful to users), and any of the publicly available LLMs can nowadays be fed an image and describe what is in it. They are even able to figure out relative distances between objects in the scene — understanding what is in the foreground, and what is in the background. Thus, in SL terms, a ‘bot could be instructed to ‘talk to the nearest avatar wearing a blue skirt’, take a snapshot, grab the names of the avatars from the hovertext, check them up and their relative distances in its radar (the SL Viewer gets that information from the simulator, so ‘bots can get it, too), identify likely candidates, and see what they’re wearing, picking the most likely candidate with a blue skirt. And then, since its radar says exactly where this avatar is located, it can draw a path to move closer — without stumbling over scattered objects named ‘Object’, but rather by confirming the clear path visually, from the snapshots taken. Finally, when reaching to the correct avatar, it can be addressed by name, and the ‘bot will even know how the character presents itself (i.e., ‘clearly male’, ‘clearly female’, ‘unknown’) and figure out what pronouns to use.

And why restrict conversation with text chat? Current technology is able to produce flawless synthetic voice, with proper intonation and breathing pauses; similarly, speech-to-text is produced in near-real-time, as anyone can test for themselves when turning on auto-generated captions in YouTube. Note that even incomplete/erroneous information produced by YouTube’s auto-caption can be easily corrected given the context. In other words, if someone pronounces my name in SL, the speech-to-text engine may get it wrong (I would expect so — given that not even I can pronounce it correctly), but whatever it thinks that the name might be, the ‘bot has awareness of where my avatar is, what its UUID is, and what its exact name is supposed to be. They will not get it wrong — ever.

Again — all the above is nothing beyond the abilities of state-of-the-art AI tools. The combination of different methods at the same time (also known as multi-modal inputs) is being actively pursued, and you’d be surprised to see what ChatGPT can do in that regard (even version 4!).

What might be considered novel and radical is to put all the above into a 3D, interactive, collaborative, open-ended, virtual world environment. To the best of my knowledge, and at the time of writing, I’m not aware of anyone having put all this together. But the truth is that there are few 3D, interactive, collaborative, open-ended virtual world environments out there — right? And those that exist, say, for the purpose of academic research in this field, will just be a simplified, conceptual prototype. Never the immense complexity and the richness of content that is only possible after millions of users having left their footprint during the past 23+ years.

So, sure — I can fully understand what Philip is thinking about. ‘Chatbots’ is so… 2022. Nice and fun to have on 2D environments (i.e., on web pages, inside a programmer’s code editor, in Word or Excel, writing your emails and reading through them, and so forth). But for LL, the challenge goes much further than that. One that certainly builds upon existing technology; but only LL has the means to join it all together into the Metaverse.

All that is obvious, of course.

What might sound weird is that none of that glorious vision requires the current very-limited-text-chat AI-esque Character Designer to be shut down.

Even though the official LL announcement speaks of a pause, they also explain in detail what is going to happen: until the end of August, everyone is allowed to log in to the Character Designer’s dashboard, see what they have created so far, take snapshots, download what there is to download, and so forth. But on the first day of September — everything will be wiped clean.

The NPC avatars, of course, will go — and with them, all their inventory.

Then all the chat logs will be deleted. Nobody will ever be able to retrieve them — either you have saved them, or nobody will back them up for you afterwards.

And finally, even those meticulously constructed personas for each NPC avatar will be wiped clean — tabula rasa — and no trace will remain of them. You can naturally back them up yourself, of course, but don’t expect them to ever be useful again. In other words: sure, back everything up, but LL gives absolutely zero guarantees that whatever replaces the AI Character Designer in the future will be able to re-use those personas or any other data you might have gathered from this 8-month-long ‘experiment’.

Once the project closes down, it’s gone forever, no matter if LL reuses the name or the concept at a later stage to present a new AI-powered concept. Whatever it might be, it will not use anything from those eight months of data retrieved by the system — except, perhaps, whatever humans pouring over that data might have learned from it, and, armed with such knowledge, go forth to design the next system. But the data itself will not ‘transition’ to any ‘future project’: it will be wiped clean.

How the community reacted to the announcement

Linden Lab openly encouraged the discussion on the Community Forums — think of it as an end-of-project analysis, common to all research projects.

After reading seven pages of comments (there might be more since I last checked), it was interesting to see that @Prokofy Neva was the first to point out the obvious issue with all the data collected and processed by Convai’s bots, while @Liberty Fairelander was the first to point out the elephant in the room:

There is also another issue that is crucial here, and that will be discussed over and over again: during eight months, Convai was being fed with chat logs, interactions between real humans (and some NPC bots, too). That data is highly valuable — it’s the kind of treasure trove that all players in the Big AI business dream of having at their disposal (more on that later!).

But the truth is that there is a legal question there: what, exactly, has Convai done with the retrieved data, and what legal claim is there over such data?

Let’s have a look at the current Terms of Service for Convai:

3. Content and Ownership

Our Services are primarily engaged in the management and/or handling of the following content:

Customer Content: You may provide us certain inputs for creation of your Character which includes but is not limited to backstory, skills, character,etc. (‘Customer Provided Content’). Additionally, your Characters shall be receiving certain information when it interacts with its voice and text with end users after deployment, which includes but is not limited to scene information, scene metadata and other visual information, voice data, text input data (‘AI Experienced Content’). Customer Provided Content and AI Experienced Content are collectively referred to as ‘Customer Content’. Based on the Customer Content, the Character shall generate responses and actions including but not limited to text output, voice response, animations and actions performed (‘AI Generated Content’). You confirm that you have obtained all necessary rights and licenses relating to the Customer Data from the respective end users (if any).

You shall have ownership over the Customer Content and AI Generated Content at all times. However, the specific terms governing our arrangement may provide you various options regarding  your access, storage, and usage conditions of the said content.

Notwithstanding the above, since machine learning and Artificial intelligence models have the potential for continual improvement by analysing data and its analytics and hence, you are authorized and hereby authorize and grant us a worldwide and a royalty free license for storage and usage of Customer Content and AI generated content to develop and enhance our Services unless otherwise agreed upon in writing.

• If you wish to revoke the aforementioned license, you may do so by notifying us at [email protected]. It is important to note that revocation of said license may have an impact on the provision of our Services and its functionalities.

4. Intellectual Property Rights

All content created by the Company for providing the Services shall be owned by the Company. You shall have a non-exclusive, non-transferable license to use the content solely for the purpose of using the Services during the term of this Agreement. You shall not have the right to copy, modify, distribute, sublicense, or otherwise transfer the content to any third party without the prior written consent of the Company. You acknowledges that the content is the Company’s valuable intellectual property and shall take reasonable measures to protect it from unauthorized use, reproduction, or distribution.

Now, it’s fundamental to understand that all the above apply, by default, to Convai’s agreement with Linden Lab — not exactly with you, a Linden Lab ‘customer’ in the sense that you need to pay for a service with LL (Basic accounts cannot use the Character Designer — now you understand why, it was not simply an arbitrary clause, LL needs to have a legally established relationship with its end-users, only possible if they can, at the very least, identify who is paying for the account, thus establishing a provider–customer business relationship).

LL, as you know, has very similar terms of services regarding your content in Second Life. As such, they can offer legal proof to Convai (or whoever asks) that they, LL, are allowed to pass along content created by third parties — LL’s ‘end users’ — to Convai, under certain conditions. Note that text & voice chat are also ‘content’ — not only creating lovely avatars or designing fantastic landscapes.

Therefore, legally speaking, and at least for paying users, LL can freely engage in relationships with Convai regarding the usage of resident’s content for the purpose of this project. To further comply with international and local laws, this transfer of content requires prior approval by the end-user, that is, you must opt-in into using Character Designer explicitly. In other words: Linden Lab is not allowed to create, say, ‘pre-filled’ NPC avatars in the name of all paying residents, which they can activate and claim on their own, if they wish. That is strictly forbidden. Instead, you, as a resident, must first give your full permission to engage this exciting new service, under its own terms, and you must be given the right to terminate it whenever you wish and forbid further content to be collected on your behalf.

But note that the Convai’s terms of service explicitly says that they will use the collected content in order to train their own AI engines. And it’s not 100% clear if they can process such data and sell it to the Big AI Corps — the ToS only mention the raw data to be processed for training Convai’s own provided services. To fund Convai’s overall operation, they may not only train their AIs on the collected data, but the result of that training — also known as a vector database — is a very valuable commodity to sell to Big AI. Also note that from such a database it’s virtually impossible to figure out how exactly the collected information has been processed; that also means that no personal data can ever be in that database in a way that allows it to be ‘decoded’ to point back at a single (human) individual.

Think of this as in the following example: your avatar says, in text chat, that you tickle your NPC ‘bot. The NPC bot reacts not only to the text by typing ‘giggles’ or something similar; no, it also activates an animation for giggling, possibly together with a sound snippet. Now all this is innocuous, of course — in the text chat, there will be no real names, only avatar names — but the association of how humans normally react is such circumstances (we tickle them, others giggle and laugh, both are having fun together) is captured in the weights embedded into the vector database.

And that information is valuable because there is no other way to capture similar information in the real world.

Oh, sure, as said before, we already have AIs able to watch movies and describe the interactions that happen there, so, in theory, none of the above is ‘revolutionary’, or couldn’t get captured in a different way. But consider the following: AIs get things wrong. Or incompletely. Or fail to grasp certain connections. So, when an AI is watching a movie, you have at least three different tasks to perform. First, a pattern matcher for each frame of video, which extracts whatever relevant information is there. Then the frames of video are processed together, possibly with the sound (or even better, with the subtitles or closed captions!), and there is another AI model which assembles information together: if on the first frame, Person A reaches out with their hand, then the next frames show Person A’s hand overlapping with Person B, resulting in Person B making a specific sound (to be processed later as ‘giggling’ and tagged accordingy), and finally Person B says, ‘that tickles’ — then you can combine everything together: reaching out, touching, giggling sound, confirmation that ‘a tickle is under progress’. Finally, all that may be resumed and condensed by a typical LLM, writing a short narrative of what just happened, and this can be fed into the vector database, which will capture the information that ‘when humans get tickled, they laugh and are having fun’.

You can do all that, of course. You can imagine the processing time that was required to produce that result. No matter how fast all these processing is done today, there is an associated cost — a real cost, in computing power and energy, not whatever Big AI charges you per processed token, which is set artificially low (or even given away for free) to encourage end-users to use the technology. This is just for a few frames, taking a few seconds on a video, to capture one single interaction. AIs must be trained in many more. Fortunately, I’m sure there must be hundreds of movies with expired copyrights that have several tickling scenes, but that’s not the point: the point is that processing all those movies to extract information about tickling has a hefty cost, even it only needs to be done once.

Now consider that, in Second Life, you have two human-looking avatars, Adam and Eve, being observed by an NPC bot driven by Convai’s technology. Here is the information that is fed directly to Convai:

• System information: Adam and Eve are friends
• Adam moves within 5 m of Eve
• Adam types: /me tickles Eve
• Adam triggers the Reach out and touch animation
• Eve types: (via a script in her AO) /me *giggles*
• Eve triggers the Giggling animation
• Eve triggers the giggle-woman-young.wav sound

Cost of Convai to retrieve the information above: essentially, zero. Oh, sure, you may argue that there is a cost of transmitting (and archiving) 300 bytes of text. How much does that cost? Well, let’s just say that perhaps after a billion or so of similar transactions, you might start seeing dollar values in the single-digits.

Note that the above example is purely reactive. Eve only ‘answers’ because that’s how her AO (or similar system) makes her react to Adam’s own AO. The system may not be able to understand how AOs work, much less how the scripts within work (probably all of them non-modify and therefore impossible to read). But, in fact, that’s absolutely irrelevant. From the perspective of what Convai needs to train their AIs, all they need is the above — which would be precisely what would have been captured using the traditional ‘watch-movie-extract-frames-reason-about-what-happens-construct-narrative-extract-relevant-data’ approach just described.

Everything that is captured in SL is instantly annotated data at cost zero. Ready to be fed to the vector database as-is.

But it’s not ‘random’ annotated data! What Convai has just received is not merely how avatars interact in Second Life, a virtual, synthetic world. That’s the less relevant information. For its own purposes, of course, this would allow Convai to train a bot to react like Eve did without even having an AO — and possibly even say things such as ‘stop it, it tickles!’. Possibly even in voice, expressing a very human emotion of joy and happiness.

But the whole point is that these are not just random avatars triggering random animations and so forth. These are real human beings — interacting through a computer-mediated medium which provides an equivalent experience to what we have in the ‘real world’. We may or may not agree at how realistic that experience is (i.e., how immersive it is), but, due to the nature of how Second Life works, it’s highly more likely that the interaction between two avatars represents real interaction between two human beings. It’s just that such interaction is, well, limited in many ways (a technological limitation).

That is irrelevant. If you see the above 300 bytes of information, you can say that it represents with perfection what exactly a tickling interaction is. Abstract yourself from Second Life, and think of a Pixar movie that wants to show two ‘toons, one tickling the other. No matter what species each of them is, or how much distortion and creative licensing is put into the depiction of such cartoon characters, we humans will immediately recognise ‘a tickling experience’ by those watching the movie — because those seven steps, encoded in 300 bytes, is enough to explain what a ‘tickling experience’ is. So, if you see them, correctly, in that order, be it in a live action movie, a photo-realistic 3D animation, a Disney-esque, 2D, ‘old school’ animation, or a setting in Second Life — you, as a human, will immediately recognise what’s going on: someone is tickling a friend!

The information that Convai gathers, therefore, is a direct one-to-one representation of excerpts of human interaction — even if in a subdued, limited, distorted way. But that’s OK. You can’t get much more from the most advanced pattern-matching algorithms combined with a neural-network-trained reasoning engine. In fact, I would claim that you’d even guess less information, but I won’t be saying that.

All I’m saying is that Convai is getting tons of rich experiences like that — not just text chat, mind you! — snippet by snippet, all formatted and ready to go into the vector database for training. Their effort in processing all that data? Zero. The cost of paying to retrieve such data? Zero. (In fact, they might even get paid — by Linden Lab — for providing the service)

The value of all the data gathered from all the ‘bots over a period of eight months?

Well.

Millions of dollars.

That’s right. And I happen to know that, but I need to open a disclaimer here.

Gwyn the AI Whisperer and her Cleaning Robot

Personally, I must confess that I never used the Character Designer — mostly, due to lack of time (I still had one tab on my browser open on that page, to try it out… which now I’m sure I never will). And I’m no AI expert — at least, I’m definitely not an expert in these large language models & transformer technology that Google launched in 2017 to the public at large, by publishing some academic, peer-reviewed papers. So, for me, all of this is still “new tech”.

I am familiar with some tiny aspects of the incredible vast field that encompasses the concept of “Artificial Intelligence” — something that goes back to the days of Alan Turing‘s Imitation Game (or even before him, according to Wikipedia). It’s a Big Word with Lots of Meanings. LLMs are just something that didn’t exist when I got a few smatterings of AI back in university. Think about the difference of being able to write a three-line LSL script to change colour when touched, and writing a million lines of code to create an SL-compatible viewer from scratch. The 3 lines is the equivalent of the sum of all the knowledge I have of AI.

So, I spent the past year training AIs. Not for free, mind you — I get paid. Badly paid (actually, terribly badly paid). But the point is that I might not have a clue on how to design full-blown LLM-based AIs (well, I have a clue, but, again, that’s the difference in having an idea how an electric motor works — a 150+-old-concept — and the ability to design from scratch a car to be a Tesla competitor).

Taking all that into context, I should also say that my half-cooked attempt at trying to do a PhD using AI in SL has mostly stalled — for many years now, and for a gazillion of complex, very personal issues. That said, this means that using a bit of AI to drive SL bots had some appeal to me, and there were enough people interested in such an approach to consider it of at least marginal (very marginal) academic interest.

I stumbled upon several serious obstacles during my many stupid attempts, some of which due to sheer incompetence, lack of expertise and/or knowledge, and… SL limitations. Those “limitations” actually apply to the real world as well, which made me think that my tinkering might be worth slightly more than I originally thought.

Consider how what has here been described as “algorithmic AI” as used by NPCs in games since the 1990s (or so) look at their environment: they cheat. They know exactly where everything is. They know all objects, all precise positions of all player avatars. They know exactly where all objects are in the scene (at all times) and even have a narrative which says what will happen next. They know which bullets have been shot and can precisely and exactly calculate where they will hit, with 100% accuracy, because, well, that’s how game engines work. The margin of error is essentially 0%, or so close to it that it makes no difference. As such, the algorithms that deal with an omniscient agent are usually quite efficient at doing their job of appearing to be intelligent, because they always have one huge advantage over human players: they know where everything is, everywhere, and all at once.

Of course, they can “pretend” not to know everything in order to make the gameplay more interesting for the humans. They can be programmed to have, say, a 5% chance of making a wrong decision, but it’s far easier to “dumb down” an omniscient AI, than to start from the opposite end and crawl towards something resembling how humans experience the environment.

Consider AI’s mechanical counterpart (if you wish): robots. Think about your common household cleaning robot as a good example: these days, they do a surprisingly good job to map out the whole floorplan (using a mix of LiDAR and video processing) with enough accuracy to calculate the best path to clean everything in the amount of time available on the battery charge, with a margin to return to the charger. Additionally, on some sophisticated models, humans can even upload a map with precise measurements, and tell the robot what to clean and how (some robots have several cleaning modes, including mopping the floor, etc. and you don’t want water to get accidentally dropped on your expensive Persian carpet…).

Early models had none of that, they just bumped into things and used a very limited infrared sensor to get a very vague idea of what was in front of them (I know — I got one of those ancient models as a gift — it does the job, but it clearly has no concept of how the floor plan looks like….), as well as a simple gyroscope and a distance tracker — this gives the bot a vague notion on how to backtrack a path to proceed its cleaning pattern, adjacent to where it already has cleaned.

But even the most sophisticated robots, with all the extra bells and whistles, advanced processors, and the help of a human that uploads an annotated map — they are not omniscient.

Because in the real world, robots don’t move perfectly as they do in simulated virtual environments: when you order an electric motor to stop, there is inertia and friction to take into account. The motor might stop the robot a bit further on than expected. That means you need to get a fix on the new position, consider that you’re not quite where you expected, and compensate on the next run. Which, however, due to some chaotic properties and random events, might not be exactly right again.

Similarly, you don’t have ‘perfect accuracy’ in real life. LiDARs have an impressive resolution, and that, combined with a camera, and possibly even measuring the distance from your own Wi-Fi antenna (bonus points if you have a meshed solution with multiple antennas — that allows precise triangulation!), plus inertial location (gyroscope) and a series of other tricks; for instance, some home cleaning robots have a charging base that projects a dot with an infrared laser on the ceiling, as a beacon, where, in theory, it’s not so easy to block and obstruct — and that can aid the robot to know exactly where it is. Until it rolls into the neighbouring room and loses the line-of-sight to the beacon and has to figure out where it is in a different way.

You also have to juggle with constraints. There is a battery and a whole electric system to manage; if the battery charge is zero, the robot stops. Typically, that should not happen: the robot should be able to come back to the charging base before the charge is depleted. But how “soon” should that be? The user expects the robot to clean as much as possible with a single charge. That means having a safety margin. However, incidents happen: the cat might turn over a chair, blocking the expected return path. Now what? The robot has to evaluate an alternate path and see if it has enough estimated battery power to go that way within the safety limits.

It’s true that these home robots have an impressive array of sensors of all types, rivalling any satellite in orbit. Even all of those consume little power (compared to what the electric motors will require, as well as the vacuum cleaner itself), that power consumption is not zero. CPUs consume power: if a lot of image processing is required, plus a new complex algorithm to be launched to figure out a new path, well, that steals a bit of energy from the battery, too. On the other hand, make a wrong turn, or go forward just slightly too much to barely miss the exact point where to turn, and the robot needs to backtrack — and that consumes energy (as well as some time, which is lost while the robot is moving back to its path). So, making lots of calculations and getting plenty of information from all the sensors before committing to a specific path makes sense, in the long run.

But that is only true for a moderately static environment. If the cat is wildly running around the house, the robot needs to be careful and avoid it. The cat, as said, might also overturn things. A sudden gust of wind may shut a door down, which blocks a passage — what if it’s the only one? Someone might have dropped some jam (or even just water!) and the robot suddenly skids and veers out of its path, without having predicted that this would happen. Now what? Recalculate. Calibrate. Get your bearings. See if you can still make it back to the charging station.

In other words: the real world is analogic and suffers from the lack of “perfection” that a virtual environment has. When a game AI says that the NPC avatar must be at position (12, 34, 56) at precisely 11:05:02 and 50 milliseconds — then the NPC will be at that exact time in that precise position. It won’t be anywhere else — the move there will be essentially effortless and instantaneous.

Similarly, if the NPC avatar needs to have an idea of who is around, or what objects are around it — it doesn’t need any complex sensors and visuals to figure it out. It just asks the database: “give me a list of everything within range”. And the database complies: there won’t be any “missing objects”, or “forgetting” to add that avatar player to the list. No: the AI simulation software will have a complete and unambiguous list, with precise locations (possibly with exact vectors showing the movement of the object) and absolutely correct timestamps with nanosecond precision.

That’s what an “omniscient agent” means in this context: at every moment during the game, the AI simulator knows exactly and precisely where everything and everybody are, and can pull that list at any time, and it will be absolutely complete and precise.

This is not to say that the decision process is simple to implement! It isn’t — it still requires skill and know-how by the programmer! But it means that you don’t need to worry about the actual environment by itself. There is no need for sophisticated sensors, as a RL robot must have: everything there is to know is already information in the database, ready to be retrieved with a simple call. And with ‘perfect knowledge’, it means that whatever paths you need to take, whatever instructions you need to follow, will be done so completely using the best possible solution — always.

A RL robot has no such privilege: the physical world is rough, irregular, not 100% predictable, and, worst of all, has incomplete information: the cleaning robot has to make “educated guesses” — those providing the best possible solution given the partial information that it has.

But that’s exactly what humans are so good at: reasoning with partial information and, generally speaking, hit a result. It might be a suboptimal result, or possibly even totally wrong, but it is a result of our reasoning ability. And, evolutionarily speaking, we humans are the descendants of those who made mostly good choices over very bad ones (which led to death); our so-called (and misnamed) “intuition” is more often right than wrong (but not for the reasons we think it is).

So, what has all that to do with AI-powered chatbots?

Everything.

You see, whatever the technology behind a SL bot is, it’s much closer to a RL robot than to a an AI-powered NPC in a game.

Consider how SL works: when you launch your viewer, you don’t know exactly what is right in front of you. Sure, you might guess it’s your home. And that will be a good guess, based on your (approximate) location. But what you see is… well, mostly gray. First the objects are downloaded; then the textures. With those in place, the lighting from the Sun (or lack of Sun) will affect what shades and colours you see — which will vary a lot over a day (both in SL as in RL), but that’s ok, because your brain can easily process colour shifts, and estimate what time is it by the length of the shadow, and so forth.

Most objects around you will be called ‘Object’ because 99.999% of all content creators are too lazy to change that, but that’s fine: based on the object’s shape, colour, and texture, you can infere what it is supposed to be a chair or the wall. Even if both are called ‘Object’ and both are painted solid grey — you still know, from their relative sizes and positions, which is which.

Then comes the actual avatar movement. Having just a vague idea what is in front of you is enough to know in which direction to move your avatar. You know the chair is in front of the wall (and not the other way round), so you have a clear path to it, so you can decide to move in the direction of the chair immediately, even if lots of things haven’t rezzed yet. In RL, this is similar how we move around our home when the lights are off — most of us won’t stumble all the time just because we can’t see anything clearly. We still know where things are — roughly. We might have some problems in estimating the exact distance to objects — yay for binocular vision! — but with a little touching around and ahead, we might have a pretty good idea on where we are, and how far away we are from that elusive chair. But we won’t mistake the chair for a wall (or vice-versa).

Then, in Second Life, as your well-trained brain allows you to navigate your avatar even in the absence of perfect information of your surroundings, another factor pops in to horribly distort all calculations: lag.

In real life, as said, we have inertia and friction to deal with; the direct equivalent in SL is lag. It doesn’t matter if we’re talking here about network lag, or script lag, or texture lag, or whatever-lag. Consider only that, from your perspective, movement is not instantaneous, like getting information from SL (textures, objects…) is also not instantaneous. Unlike in RL in plain daylight, where someone with good vision will see obstacles clearly, in SL such “obstacles” — in the form of unrezzed items! — may suddenly pop in and obstruct our path — making our avatar bump into a solid wall of… air?

Turn around, and it’s even worse. Like in the metaphorical question — “if a tree falls in the forest, and nobody is around to listen, does it make a sound?” — in SL, what’s behind your back doesn’t exist until you look at it. In fact, this amazing property of reality doesn’t bother us in the least: although most of us has 170º vision, we pretend to have 360º vision without turning around, because we know what is behind us (assuming a familiar location). We don’t see it, but the brain can create a perfectly vivid picture of what’s there, and you won’t get surprised. Here is a silly example: as I type this, I have my back to a bookshelf. I know the bookshelf is there (even without looking at a mirror to check if that’s true!). I can “paint” a very colourful image of it inside my brain. But ask me in what order the books on the third shelf are, and I’ll tell you that I have no idea — that information is not important enough for my brain to memorise it. But a bookshelf with colourful rows of books? Sure thing, Gwyn, here you have it in (almost) full colour and a (quasi-) perfect 3D reconstruction in your memory. If you need details, just ask your neck to swivel around.

Imagine my surprise if suddenly I noticed that there was no bookshelf, but rather a huge hole in the wall, and a Jurassic landscape behind me, with dinossaurs and all!

I’d get a shock first, sure, but my brain would quickly label it as a hallucination and “not real”, so I would take a few breaths before analysing further.

In other words: from our incomplete perception of the world, given to us through the lenses of our limited senses, our brain is rather good at concocting a pretty convincing simulacrum of “what’s out there”, which we can assume that will stay that way, even if we turn around to look. And we can also assume that others, looking at the bookshelf, will ask me, “Why did you scream? It’s just your bookshelf!”. And I expect them to say that in a normal voice, speaking a language I can understand, being spoken by a fellow human being that I recognise.

You can see where this leads to. Humans aren’t omniscient, but they don’t need to be. We’re rather good at picking clues, assembling them together, and give us a rough, working idea of what surrounds us. We’re good juggling memories around, and construct concepts out of thin air just because we expect some things to be as before. The buildings across my street have a tiled roof — they did have one the last time I checked, an hour ago. So I don’t need to be looking straight across the street to tell to myself, “aha, roofed building. Check! ? Tiles on the roof: check! ? Good, good. All is exactly as expected.”

Now, the Second Life experience is both similar and yet different. Consider two different scenarios: the first is that you teleport back to your home, lavishly decorated by your masterful design choices. And then you jump randomly across the grid.

In the first case, it’s hardly an issue if you turn your avatar around and see a grey wall and little else. It’s irrelevant: you know what will be there, once it rezzes. There is just the issue of lag to deal with. Eventually, the SL Viewer will catch up with your mental image of what your house should look like, fully rezzed. And that’s fine. Even if one day the lag is terrible and everything takes ten times longer to rez than usual, it’s still fine, because you still know where you left that chair in front of the wall.

Your very human brain knows that. Your SL Viewer might know — or not.

Contrast that with a teleport, and rezzing into a completely new and unexpected environment. First of all, it seems that everything takes much longer to rezz than at home. Ha! Your brain is fooling you (cached images nonwithstanding). What happens is that the teleport brought you to a place you know nothing about. That means that, before anything stops being grey, you don’t even know where to point your avatar to, in order to walk away from the teleport point. Turning around doesn’t help, either; in fact, it could be argued that it’s even worse, because now the SL Viewer must start downloading those textures in front and behind you as well. Twice the textures, twice the lag. Move quickly around, turning the camera, and it just gets worse and worse and worse, and you start bumping into things (or people!) that aren’t even there. For your brain, this is an experience just like being in a dark room which you have never seen before. How do you know what is around you?

The answer (for most of us with good vision, at least) is simple: you bump into things. Exactly how home cleaning robots do. Bump into something solid, and you know “it’s there”. You might not know what is there (not yet), but you can start to map, in your brain, what “things” are around you — what you can go through, what you can jump over or crawl under, and so forth. That’s a bit what we experience in SL when “the world is gray”. Fortunately, our brain can still cope with it (because the dark room analogy is part of our experience). But we don’t exactly like it, because that’s not how we expect things to be!

That’s exactly how a home cleaning robot “thinks”. When it starts cleaning, it has no clue about its surroundings — even if you give it a map (like, say, your avatar’s mini-map). But the robot will need to use its sensors to have a rough idea of where things are, and, failing that, bump into things. The map, after all, may not be accurate. Or it might lack essential details: a chair is drawn as a square, but what about its four legs? Those are obstacles, too, although a cleaning robot can go through them. But it will only be able to do that by, say, taking a picture, processing the image, figuring out “it looks like a chair”, beaming some laser to make sure that there is a clear way through (as opposed, to, say, a picture of a chair on a wall), and finally go ahead, occasionally bumping into chair legs to get a notion about how wide they are, etc.

Eventually the robot’s “mental image” will come to resemble a much more detailed mind-map of what is “out there”, but that takes some time and exploration — and a minimally static environment.

The same applies to a human prisioner dropped into a closed dungeon without a light: eventually, it will get a mental image of how large its cell is, and where the bed is, or what the other features in the wall are.

And, finally, the same also applies to the SL Viewer: there is this glorious moment, after interminable waiting, when all textures are downloaded, all objects (moving or static) have been accounted for, and you can fully appreciate the glory of this new location in its entirety — and, of course, now you can navigate around with perfect ease, knowing exactly which paths to steer your avatar around, until…

Well, take too many steps, and the greyness steps in again — the viewer needs now to load more items and textures and sounds and animations and avatar meshes and whatnot. Every time your camera changes, the same happens over and over again: a moment of beffudlement as everything starts to get downloaded, and an instant of pure ecstasy, revelling in the glory of what the creative builders have done — only to get it all whisked away as you move around the place.

Of course, I’m giving you an extreme example — most people, these days, don’t have such a terrible experience, thanks to a lot of improvements. But you all are familiar with the way this virtual world works: nothing is truly “there” unless you look at it. That’s… one of the explanations of the universe according to quantum mechanics (and, incidentally, Buddhism as well). In SL that is absolutely true. Even more so: different people in the same region will see completely different things being rezzed and unrezzed. Until they all meet — then everybody will see, on average, pretty much what everybody else sees. But that’s just a coincidence: in practice, the viewers don’t tell each other what they are seeing (they do tell where you’re pointing your camera at).

Each viewer has its own solipsistic view of Second Life. A view that is constantly changing as your avatar moves around and turns the camera around.

One might argue… but wouldn’t it be much better if we got the whole region downloaded once we teleported into it? Instead of getting this piecemeal, one bit at the time?

The most likely answer is that this could be done in 2025 (I’m just guessing), but it wasn’t the best choice in 2003. We had far less bandwidth — but similar Land Impact limits, that is, the same amount of textures to download, etc. Sure, most textures were much smaller back then — but so was VRAM on GPUs. The adaptative model of the SL Viewer — what is called the interest list — is a good, working assumption that you only need to download as much as you can see — and nothing else. Why would you need to download things that you can’t see, anyway? That’s just a waste of bandwidth and computer power. No, everybody gets exactly as much data as they need (taking into account things like drawing distance, for instance) to create the scene they’re in, and that’s the optimal approach (on average, at least). You could download the entire region — several Gigabytes, for sure, but what’s that with the ultrawide Internet pipes we have at home these days? — but what’s the point, if you will only see a small little part of it anyway? And if you want to really see all the region… well, I suppose you can always turn your drawing distance to 512m, wait half an hour (or two hours…), continuously rotate around your axis on point (127,127,20), and, in theory, rezz everything. You have that choice.

That’s the theory. In practice, it’s not guaranteed to be like that. And how do I know? Why, because I happened to do some academic research back in 2008/9 to figure out the best approach to, say, count all items (or all textures) in a region. The assumption was that you could get a ‘bot to ‘see’ everything, from the perspective of the SL Viewer.

But I stumbled upon the way the interest list works — it’s got plenty of quirks and tricks! — as well, of course, as lag. Lag is unpredictable, and it mght just mean, in some cases, that whatever you expect the interest list to be (and we can assume you know that because your bot has requested it) might not be what you really get. Second Life is not a “What You See Is What You Get” application. Instead, “You See Only What You Get” — which is a slightly different thing.

Here is an analogy for those hard-core LSL scripters among you. Do you remember the “old school” radar HUDs (or even those predating HUDs…), well before practically every viewer started including them by default? You’d use a simple loop to try to detect how many avatars were nearby. And in those days when the LSL Portal still worked (ha!), you could read that most “detection” routines would only retrieve the first 16 avatars (or, in some cases, objects) it could find. This was perfectly adequate for, say, 2004, when it was not unknown for some regions to crash with more than seven (!) avatars in it.

One thing, however, was baffling — why didn’t the script deterministically retrieve the 16 closest avatars? (And they weren’t even retrieved by distance — although they were almost retrieved in that order. Just… not quite.) Suppose you are within viewing distance of 32 avatars, none of which moving (except for animations, which are only moving in the viewer, not in the region). Shouldn’t you always get the same result?

Well… no. Not in Second Life. The LSL function will indeed request “the first 16 avatars which are within a distance of X” from yours. But sometimes it picks some of them, sometimes it picks different ones; there is no guarantee it will pick them all. And on subsequent runs, it might get more or less avatars — you can’t say, nor predict. Why?

I don’t really have the ultimate, absolute answer (unless it happens to be “42”) for that. All I can say is that there is a certain amount of chaos in the communications between the viewer and the region simulators. Most of it comes from the communications protocol itself: it uses UDP, a form of connection that doesn’t guarantee delivery. That is to say, you send out packets of information roughly in the right direction, and expect them to reach the destination. In general, that’s exactly what happens. But every now and then, packets get lost — the network might suddenly get crowded, for example, discarding packets to ease down the load. When that happens, it’s not a problem — you just request another copy of the lost packet, which you may have numbered before transmission, so that you can figure out which ones are missing and request re-transmission. But, of course, the packets requesting retransmission might also get lost. What then? Well, you just wait a bit, and try again later. Hopefully, by then, the network congestion (“traffic jam”, in urban traffic parlance!) might have resolved itself, and you’ll get the missing packets.

But in the meantime some information got lost.

When your avatar is requesting its “interest list”, saying “give me everything that my avatar should see in a range of 128m” (assuming that’s what your drawing distance is), packets get sent out to the region simulator with the request. Some packets might not even reach it — your avatar just “sees gray”, nothing is rezzed. Once they reach it, however, the region simulator has now to do some calculations: figuring out your camera viewing angle and drawing distance, and requesting from the database server a list of objects that satisfy these constraints. In the very early days of the SL Grid, this meant contacting the central servers for the full information (but then it would be locally cached at the region simulator). Today, things are a bit more sophisticated, but the principle is similar (just more efficient and wasting fewer resources). However, something is still true — that list still needs to get transmitted back to your viewer, and, while that happens, some packets might get lost, especially if the list is huge (these days, it is, in busy regions with lots of avatars with their attachments!). And that means your viewer never gets that crucial bit of information saying that there is a chair in front of you — or a bottomless pit — or even an avatar that you bump into.

To deal with all that unpredictability, the Second Life Viewer and the region simulator, as well as the communications protocol used by both ends, have a battery of tricks to try to figure out “as much as possible” with “as little information as we get”. The idea is that the viewer draws what it can, based on the information it has at the moment, while patiently waiting for the rest to arrive — looking carefully at those packet numbers to see if some were never received and try to ask for a retransmission. This goes on between the SL Viewer and the simulator, continuously. Better still: the whole viewer was designed with these tricks in mind. If the environment for the region hasn’t loaded yet, everything is not black-on-black — the viewer will use “reasonable defaults” for the appropriate region time (i.e., dawn/noon/dusk/night), and switch back once the region environment loads. If there is a new avatar you never saw before, and no information about its shape has been retrieved… too bad, fallback to a cloudy (or whatever equivalent is used on each TPV). Got the shape but not that 2k skin and its materials? Too bad — paint the avatar gray for now, and wait until that arrives (note: actually, these days, assets are retrieved totally independently from this, but the viewer still needs to know what to ask for!

Now imagine that all the above is fed to Convai’s systems as well.

Convai is getting so much more of this deal

You still remember the 300 bytes of information we started with, claiming that it faithfully represented a ‘tickling interaction’? Good.

3330 words later (I’ve counted), you have now a blueprint for desiging a highly sophisticated robot that can navigate in the same environment that a human does, and interact with humans — and items designed for human interaction! — exactly like a human.

That’s because all that data can be fed to Convai, and it doesn’t even require a lot of processing.

I earlier said that capturing the essence of just one simple individual interaction — tickling! — may cost a few thousands of dollars. That number comes actually from OpenAI’s Sam Altman, indirectly quoted through several layers of interviews. At some point, the delay in launching ChatGPT 5 (which is out now — but don’t expect miracles, there won’t be any!) was attributed to a simple thing. When ChatGPT 4 did some deep reasoning to identify a disease by the symptoms that the user just typed, that costed OpenAI (roughly) a thousand dollars in raw processing costs (i.e., firing up all those virtual GPUs to intensely process data from its vector databases and come up with a result). Let’s say that ChatGPT 4 would get it right with 70% accuracy — which is roughly what you’d expect from a reasonable doctor. ChatGPT 5, however, may go much further, and give you 75% accuracy — but the cost is roughly 30 thousand dollars. And might take slightly longer (more resources are being put into solving the question — they not only cost much more, but they also process far more data, requiring a bit more time to settle on a result).

Now, the problem here is not if it costs a thousand dollars or thirty times as much, of if the answer comes in milliseconds or in a few seconds. The end-user will not be charged for the real cost, after all. No, the problem is that designing and then training ChatGPT 5 costed something roughly 200 to 300 billion dollars, according to a recent interview to Sam Altman². Since we don’t have access to OpenAI’s internal accounting (they’re a private, not a publicly traded company — just like Linden Lab — and so they’re not required to show any of their financial data), we cannot say for sure, but such estimates are not really very off the mark.

What I’ve learned as an “AI Whisperer” is that the biggest cost of getting a new model is not really in the conception and design; neither it is in the sheer processing power required to train it. No, the problem is that such training is worthless unless humans sift through it carefully. And AI Whisperers are grossly underpaid, that’s true, but you need tons of them to successfully ‘tune’ an LLM to provide meaningful, harmless, useful answers. Currently, the cost of having humans fine-tuning the models by far outweights the cost of the automated training (i.e., feeding raw documents to the system in order to fill a vector database with its initial data). This is so true that even OpenAI gives away many of its training data, open-sourced with a permissive licence. And all the others do the same! All for free.

The reason for being so ‘generous’ with the processed data that was scrapped mostly from the Internet is simple: the cost of actually training the AI model with the raw data is not much, and, by providing this data for free, Big AI is encouraging researchers to have much more data to train future models with (without the associated costs!), and that, in turn, means more researchers can do more with what they’ve got available, compare solutions and models, and come up with novel ideas to design future GenAI engines. These, in turn, will naturally be adopted by Big AI — because researchers publish their results, most of them for free (even paid publications have a trivially low cost, not reflecting the real cost of having the researchers doing all that work, of course).

Does that mean that you can run your own ChatGPT engine at home, and feed it the exact same processed, trained data that OpenAI used — all that for free?

Sure you can. That’s the whole point. And sure, it will work.

The problem is that it won’t be usable. Models hallucinate. They come up with totally wrong answers without being aware of it — but able to defend them as being correct. They may actually get everything right with the same ease and conviction. And, of course, they will be rude, develop extremely negative personas with ease, and tell you how to make bombs or illegal drugs in the comfort of your home.

So the question is: how much can you trust that data?

From the perspective of a researcher, that is not crucial. Because researchers focus mainly on how to produce a trained model that will work as intended, and methodologies to fine-tune that data using humans. The latter, in fact, is the crucial difference: doing models that hallucinate, well, that’s easy, all you need is lots of raw data to work with. Restricting them to reasonable boundaries — that’s something (currently) impossible to do without using humans. Things are constantly evolving, sure, but the more sophisticated the models are, the more reasoning they can perform on their own, the harder and harder it is to verify if they are being correct, to the point, harmless, and Safe For Work. That’s the job of humans. And humans need to eat, so that highly specialised work needs to be paid, even if at such humiliating low rates that I’m ashamed to say how much it is.

And it’s still the highest cost.

But it’s also the highest value. The Big AI game is about who has the most polished model for human consumption — while keeping things under control, money-wise.

Now, in the example above of the health-related question, the issue for a CEO like Sam Altman is to do the maths. With ChatGPT 4, you got 70% accuracy. After spending, say, 200 billion dollars, you have ChatGPT 5, which can give you 75% accuracy — at thrity times the processing cost for each question. The question is: can the user tell the difference?

ChatGPT 5 cannot ever give 100% accuracy on such questions, because, well, we’re humans after all. Even assuming a perfect description of the symptoms (which will never be the case, because humans not trained in the health sciences may not even know how to objectively identify their symptoms), there is always a chance of a new disease, or of an extraordinary coincidence of unrelated symptoms having been thrown together and mixed up. Even real-life doctors are not 100% accurate, but at least they have the advantage, over LLMs, of being able to ask the patient for more exams or imaging — or to get a second opinion.

Thus, 70% accuracy is actually very good. It might be as good as a doctor, in fact. It would therefore be pointless to spend tons of money to produce a result that is only slightly better than what the previous generation of AI models was able to achieve — and take thirty times the price.

Or, to put it bluntly: imagine a future where OpenAI has to charge the real price of their services. A user comes in with a lot of symptoms, and wishes to know what is ailing them. They are given the following choices:

1. Use ChatGPT 4. Price: US$1,000.00. Accuracy: 70%
2. Use ChatGPT 5. Price: US$30,000.00. Accuracy: 75%

Which one would a user choose?

And of course, when ChatGPT 6 comes (Altman promises to be already working on it), you’ll get 80% accuracy — probably better than any expert in the field — but, alas, it will cost you $100k. And so on, and so forth.

There is a law on diminishing returns here. And why is there such a difference? Well, to get 70% accuracy, OpenAI had to pay for a large number of doctors to go through ChatGPT’s answers and flag all those which were blatantly incorrect or incomplete. Now, doctors will not work for peanuts; so that cost quickly skyrocket for specific questions requiring expert knowlegde. To go from 70% to 75%, not only you needed more doctors, you also needed higher qualified doctors — specialists in their fields — which, obviously, will charge OpenAI much more.

To give you another example. Think of how many doctors you ever met, professionally or socially. That’s perhaps a dozen or so, at least those you go regularly to. That’s actually correct, because countries have metrics about how many doctors are needed for a certain population, and that means, in general, that you get in touch with them relatively regularly,

You can probably say the same about teachers. Roughly speaking, you might even personally know more teachers than doctors, and that’s fine, since there are more teachers — but not overwhelmingly so (in terms of orders of magnitude, they would be at a similar level).

Now consider: how many of your friends and acquaintances are researchers in Mathematics? I mean, real-deal, Doctors of Mathematics, lecturing at a university?

Most of us — unless we happen to work for a university! — will probably say ‘none’. Some might, by chance, know one or two (I only know one, one of my half-uncles, although these days he’s not into maths any more — he’s upper management in a computer software company). That’s normal, because given a certain population, the number of maths professors and researchers is very small. If you got one per 100,000 people, that would be extraordinary; but it also means that you have one chance out of 100,000 to meet one!

Let’s raise the bar even higher. Imagine that you even have one, or a few, friends in the mathematics research field — not teaching public school maths, mind you, but actually publishing academic papers on mathematical research. Among your maths friends, how many of them produce work that is qualified to become the next Fields Medal³ in maths?

You see, it’s because you actually need that level of qualifications to tackle maths problems fed to LLMs that are designed to, say, use LLMs to assist maths professors with their work. They will expect the LLM to be able to look at a complex problem in highly advanced maths and solve it — the idea would be to delegate tasks such as proving theorems and the like, on fields that haven’t been even invented yet, as part of the regular work that maths specialists are working with.

That, in turn, requires extremely skilled mathematicians to train such LLMs. In other words, if you expect an LLM to say, ‘prove that Einstein’s equations for General Relativity are correct, but use a different approach’, you need that LLM to have been previously scrutinised by maths experts who have personally done that proof in many different ways (Einstein only needed to do one solution first).

Such people are currently training AIs for the very modest price of US$500 per day of work. And, you know, there are always a few there. Just not many. That’s why current models are able to reason much better about symptoms, and find out what disease is causing them, but they are not so great when it comes to maths. It’s not that the models cannot be trained upon volumes and volumes of published books on advanced mathematics — they have been trained on those as well. The issue is that there are very, very few highly qualified maths experts to be able to do such work (e.g., when reviewing what consists a solid proof of a theorem vs. a fully hallucinated solution); and, among those few, even fewer that are willing to work as AI Whisperers — it’s not a sexy job to put in your CV, after all. Winning the Fields Medal — now that’s another story!

All right, now you get a glimpse of how hard it is to train current-generation Generative AI models. Hard — and terribly expensive.

How can you shortcut this?

It’s not that easy, and most approaches so far have failed (something which I’m working on for a different article). But consider Convai’s approach here. They might not be able to train chatbots to explain General Relativity to university-level students. But they can — and will — be able to have ‘bots that go around a shop and picks a dress based on someone’s preferences — either in the virtual world, or in real life. And the beauty of it is that they don’t need to pay for all that complex training in 3D navigation and dealing with fashion sense, or the way shops are organised.

They can get all that data, essentially for free (or even get paid for the data they receive), all ready to be processed directly by an AI’s vector database generator.

And we’re talking about high-quality data, too. It’s not as if the data is “simulated”, or synthetically generated, in order to produce quick results, and let humans sort it out later (aye, that’s how much of the AI training is done). No — they already get the end result of having humans do all the steps, and capture everything on the logs and metadata that Linden Lab feeds to them. Actually, Linden Lab may be absolutely passive in this regard: anyone with access to, say, LibreMetaverse can extract that data as well, anytime they want.

The point is that Convai already has done so — over eight months.

And that data is incredibly rich. In the above example of walking a customer across the shop — it’s not just merely a ‘role-play’ imaginary setting. You know what the shop is called. You know what kind of apparel is sold there. You know where the shop is located (i.e., what other shops exist nearby, for instance). You know at least the essentials of what kind of customers they attract, and have a rough idea on how much they’re willing to pay for goods. You know how humans, interacting through their avatars, react to specific colours, or designs, or prices, or a combination of all that. You know what they need to click, or where they turn their avatar to, when making a purchase. You might even learn how an item is tried out before it gets bought. Since you can have such a shop attendant constantly on duty, you will know how many customers come in, when are the busiest hours, and what they buy. And, ultimately, you can also do some number-crunching and figure out what are the most popular items (at least, if they’re attached items — which almost everything is, these days), and how much people spend on it. This, in turn, allows the shop assistant, based on what they see on the avatar requesting for assistance, and with a few pertinent questions, extrapolate almost exactly (let’s say, 97% confidence) on what that customer wants to buy and is willing to pay.

And all that is handed over in a silver platter, absolutely for free — and replicated thousands of times, so that future models have a wealth of information to choose from.

What now for the future?

Now you hope you understand why Linden Lab has to remove all traces of the data collected by the bots so far — once the agreement with Convai is over, neither company is allowed to retain such data. Convai doesn’t mind — they already got a lot of high-quality information on human behaviour, which is readily packed for sale to their other customers: those that will pay millions for such information, not the likes of Linden Lab, who merely outsourced some neat chatbot construction kits from Convai. Maybe they even paid nothing.

More likely, they might have agreed on a fee of zero, during the period of ‘alpha testing’, where Linden Lab got access to their tools for free, but agreed, say, to run the Convai models on their own leased cloud servers instead. Convai would therefore incur in no expenses whatseover, and get all annotated, high-quality data in return.

What I can imagine is that Linden Lab may have been shocked at how much they were being charged by Amazon⁴!

So, to summarise…

• AI is costly; Linden Lab may only have realised how costly it is after a few months.
• Convai was getting high-quality chat content, worth millions, for free, which they either used on their own, or — after a bit of editing, but not really much, since “avatar names” are already pseudonyms and impossible to trace back to their real-life persons — sold to third parties (“to offset their internal development costs”).
• While legally all the above might have been in order, Linden Lab might have been reading what residents think about it, and — very likely — given the overall issue some deep thought.
• In essence, what Convai is offering is nothing out of the extraordinary, which any geek and wannabe-AI-expert can cobble up in a few days (or even hours). After all, perhaps the most innovative part of the whole interface was the Character Designer itself, and how ‘bot accounts were cleverly fitted into LL’s authentication model (i.e., tying them to a ‘main’ avatar’s login data) — this was work done entirely in-house.

…and at the end of the day, Convai was dealing only with the text chat bit. Not the rest of the complexities that Philip envisioned for AI-in-Second-Life: identifying objects and avatars, flawless movement (e.g., “sit down on the red chair”), conversations with multiple avatars (in public chat, in group chat, in multiple IMs), going flawlessly through inventory and wearing “appropriate” clothing, and — who knows? — fluid voice communications.

Now, Convai, in all fairness, is able to provide most, if not all, of the above — their website says as much. However, it’s also quite likely that they’re used to “certain” environments, namely, static-content games (as opposed to user-generated, collaboratively build content, right on the spot, in real time), with a “map” well-planned and designed in advance, and so on. To adapt all those things to Second Life is certainly possible, but it comes at a hefty cost. Worse than that, Convai cannot bear the whole burden of doing everything for free on behalf of a single customer: such an integration requires a lot of hours of coding, but, since Second Life is unique (as opposed to other environments designed on top of typical game platforms, using Unity, Unreal Engine, or even the free and open-source Godot engine), it also means that:

1. Convai would need lots of knowledge about how Second Life actually works, and that means tying up more Linden employees just to liaise with Convai and explain them what can be done, and what cannot;
2. Convai would not even be able to resell the SL-specific solution, since there is nobody else but Linden Lab using Second Life’s technology.

Note that even OpenSimulator grids would be possible candidates for Convai’s solution, since we’re talking essentially about technology integration at the server level. While this isn’t obvious for many, OpenSimulator is only compatible at the viewer level (or, more correctly, at the communications protocol layer). While the server technology used by OpenSimulator is, well, open, it bears not the slightest similarity to Linden Lab’s own. Not even the programming language is the same, and nothing that might have ben developed for Second Life’s servers can be “adapted” to run on OpenSimulator — unless, of course, it’s rewritten from scratch.

But the OpenSimulator community is not a potential client for Convai: whatever they’d be willing to charge, it’s highly likely that none of the OpenSimulator grids would be willing to pay (even if they could afford it — which is unlikely). Besides, OpenSimulator has its own NPC technology, developed eons ago when Linden Lab was absolutely against the concept, and it makes much more sense to build on top of what already exists than to scrap everything, just to be “Convai compatible”. That’s ludicrous; the way OpenSimulator development works is to stay away from proprietary technology, because nobody can afford it (that was, after all, the whole point of creating OpenSimulator in the first place)!

Which means that Convai is now stuck at the proverbial dead end.

Linden Lab, by contrast, can leverage on what Convai provided. As said, the hard part was dealing with the integration of the Character Designer into the Second Life Grid ecosystem, something which only Linden Lab could do. Then we have avatars that can chat using an LLM. That’s nothing special — we already had plenty of residents doing exactly that, thanks to the many APIs provided by the Big AI companies to interface with their models. At its simplest, all you need is a basic LSL script that has an API key for your favourite LLM, and, every time it sees something in chat, it sends it via a HTTP request, and awaits the answer from the external model. You can put that inside a cube, make it transparent, attach it to an avatar — and that’s it, your instant LLM chatbot with just a handful of lines of code.

You can make it more complex, of course: you can tie everything into the existing LibreMetaverse-based bot-creation tools out there (either doing it on your own, or using some existing provider, such as SmartBots). Some, indeed, already provide that service anyway; you don’t need any fancy-schmanzy technology from Convai (or even LL…) to essentially replicate what the Convai-driven bots could do. That was, from the perspective of a developer, “basic”.

No, Convai only provided Linden Lab with a way of test out some simple concepts, and see how the community would react to them. This they did.

But now, for real ‘bots that are aware of their environments in Second Life, and can interact with it “just like a human resident”, a lot more needs to be developed. Not necessarily at the conceptual stage; as I began this essay by explaining, AI wasn’t “invented yesterday”. All those tools and algorithms are there, published by AI researchers, freely exchanged on communities congregating around places such as Hugging Face or similar hangouts. Some members of Linden Lab’s development team had already played around with AI-enabled vision recognition and similar tools (required for SL Puppetry, for example).

It’s really just a question of allocating coders, time, and money to this project, and do it in-house. This will ease the strain placed upon residents worried that their in-world communications are being stored by third parties, outside the Second Life Grid, for unknown purposes, for which there is some uncertainity regarding ToS (I’m speaking from the perspective of a resident — not a lawyer!).

These days, AI is not rocket science any more, and it’s accessible to the public, almost for free.

While Second Life’s innards… well, that is rocket science for sure, even if it doesn’t seem to be!

Ultimately, the point here is that having AI-driven avatars in Second Life is not a major breakthrough in Artificial Intelligence. It is applied computer science (or engineering, if you prefer). What is a major breakthrough (or will be) is the integration of Second Life as-it-is-today with several “AI enhancements” that go way beyond simple text chat.

And why should anyone but Linden Lab get the credits to that?

---

1. “Zero” as in: “you need zero installed software to be in Second Life”. ↩︎
2. A slightly outdated article explains the overall costs of training the previous generation of AI models. Disclaimer: the article is written by a company selling cloud computing services for AI training & operation. ↩︎
3. The equivalent of a Nobel Prize in maths (Nobel didn’t consider mathematics as sufficiently relevant to fund a prize for it). ↩︎
4. See note 2. ↩︎