Wow, this is really a “bomb” that LL has planted among the community… it’s going haywire right now, but the cool thing is, we have a very long fuse, and lots of time to disarm it. Thanks to Cory for warning us in advance!

I have very mixed feelings about the current proposal, and that’s why I don’t want to discuss the ideas yet on the “official place” - Cory’s own blog. Unlike many people (and I’m proud of saying that in public), I like to analyse things from several perspectives, both emotionally and logically, and have changed my mind often about “things written on stone tablets”. So I’m quite prepared to revert my position in about 6 months or so

First, I think that we should not view Cory’s proposal as Linden Lab’s “final word” on it. It’s just that - a proposal. A way to let people think about it, discuss it, argue it over, and come up with good ideas that can actually be implemented. That is what I feel right now.

Both the forums, blogs, and several informal meetings that happened inworld since the “announcement” reached the same preliminary conclusions: creators are not happy with weak copyrights, period. Consumers can’t care less. They currently make their choices based on what they want and what they’re willing to pay. There is a large variety of choices here. I can buy fantastic clothes from Torrid and I know the quality I’ll get - but they won’t be unique items. I can order custom-made dresses from Von, and pay premium for that, because I don’t want anybody else to have the same “unique” dress (and I establish a trust relationship with those merchants “informally” - I know that Torrid won’t sell me a below-quality product, or that Von will charge me premium for one “unique” item but sell 100 copies afterwards…). In some cases, for new merchants that I haven’t worked with, I can appeal to resident-created trade assotiations to protect my rights as a consumer - like RATE, for instance. And finally, for everyday use, well, as everybody else, I have hundreds of freebie articles (sometimes from Torrid or Mistress Midnigt!) and change them to fit me. All this mirrors somewhat what goes on in the real world. All this happens with the current permission system very well.

“Reverse engineering” is a fact of life (1st and 2nd) - we usually think about it on the industrial scale, or on the software/computer industry, but it’s not so. I grab pictures from web sites and ask talented designers to build me a copy for it inworld. I look at a cool device inworld and see if I can replicate its scripting. I see someone inworld with a “limited monopoly” on some sort of item, and I fry my brains overnight trying to think “how did he do it?”. If I can’t figure it out, I can always ask an expert to take a look at it and replicate it for me. Again, this happens, despite our current permission systems.

And finally scamming. Of course, I don’t really expect the scammers to actively participate in forums, blogs or meetings. But just because they don’t discuss it, we can’t ignore them. Inworld, my only action against them (with the current permission system or otherwise) is to warn them that they’re doing something immoral, and scare off their customers. This normally means I’m a target for negative ratings *shrugs* but morality is more important to me than the ratings anyway. Most unsuspecting customers usually thank me. Still, some had a problem in understanding why I should pay for a skin when it was so easy to copy it. Once more, scamming *already* exists in SL, despite current permissions.

What I’m not sure of is, economically speaking, what will happen to SL’s rather stable economy when the new system is deployed (and remember, nobody knows what the “new system” is and how it will be implemented - Cory just presented a few key ideas on what it could be). More than fearing that a batch of creators will go away forever and take their items with them, I fear unstability due to the need to adapt to the “new rules”. Creativity can go down the drain and no new objects appear; or new objects will be much higher priced since the creators will know that their creations have a very limited time of existence before being copied and resold by scammers. This would mean a stagnation - currently items inworld are given for free, or sold in the L$20-200 range (very special items are much higher than that, of course, but they are also a tiny fraction of the total offerings, even if they represent a large fraction of the economy’s turnaround). Actually, pricing new content higher will very likely give the scammers an incentive to make more and more copies! If Windows was available for free, what would be the incentive of making “illegal” copies of it?

So. I guess most of you can agree with the above paragraphs. Now to concrete suggestions to solve those problems - and taking into account that we (Lindens and residents) don’t know how things will work out - here it goes:

1. Traceability of Intellectual Property

   Anything which is created should be flagged with the creator name, and a timestamp. That’s all you need to trace it. If you can somehow use clever scripts to duplicate an object perfectly (my, people are ingenious these days!), if you can make sure that the creator name is never lost, at the very least you will able to trace your IP back to you, and the timestamp will prove to anyone that you have created it first. So the copied version won’t have either your creator name or timestamp - and thus can be identified as a “forgery” (even if it just looks the same and works the same way).

   Co-creation should also be allowed (see my question on the Town Hall transcript). These days, many builders will work together with scripters, and the final item/device is usually a “joint effort”.

   Now this point will not prevent scamming - ie. scripts can be copied & pasted, or you can reassemble prims by just copying the data from the tabs and replicate them, etc. This is just to establish your IP!

   RL example: Creative Common’s own registry.

2. Scam detection

   It came to a surprise to me that Cory didn’t mention the so-very-simple “tracking mechanism” - when the wrapper is broken, you get an IM (or similar information). Actually, when Tiger Crossing and I suggested that, Cory explained that they had already discussed this with James Grimmelmann of LawMeme. So either you’ll get an event to detect for the wrapper being broken, or in some “magic” way you’ll be informed that this happens.

   Why is this so important? Well, again, for tracking purposes. If suddenly everybody breaks the wrappers on your objects, you’ll know where the potential scammers are going to be. Since after breaking the wrapper you can’t “show” the object to anyone else, it means that any scam much come from the person that broke it (even if you copy & paste a script and give it to your friend, the Master Scammer, to have him replicate it… you’re nevertheless accessory to the offense!).

   And since you’ll have established creationship with a timestamp on the ORIGINAL item, it will be easy to “prove” that you are the original copyright owner. I only wished that in RL it would be so easy to immediatly get a list of your “prime suspects”…

   So, we have a way to register IP, and to track copies of it. Next point!

3. Scam reporting

   What happens with scamming nowadays? Uh, nothing. Nobody controls scam, there is nobody to report to, and there is nothing you really can do about it except fret (or waste your precious time, which would be much better employed in creating stuff, by going around and see who is taking advantage of your IP right now…). In SL, we have no police and no courts to appeal - just the Lindens. And since is hard to know when there is “abuse” (if I buy a chair from you, and resell it to a friend for a higher price, is that really a “scam”? Or am I just an “unlicensed retailer”? Or just doing a friend the favour of sparing her the time to search for the item herself - and in this case I’m a “value-added retailer”?), you can’t really report it.

   This is something which has to be changed now - and it means changing the TOS. Since due to the previous two points you’ll have a way to establish IP uniquevocally, and a way of tracing it, you can “legally” act upon that. Since there is no other way, this means presenting the Lindens with “evidence”, and forcing them to abide by a revised TOS - expelling that user (and its alts) from SL and all its copied items from the world.
   Eventually the person buying illegal copies could be subject to the same threats (more on that below).

4. Scam prevention

   Ok, but it’s a shame to do things “afterwards” - in the mean time, customers get angry, since they have payed for an item they wanted, and they didn’t know it was a scam.

   Again, let’s see how the RL deals with this. It’s simple - trade associations and “seals of approval”. We’ve got RATE, but we could have a few more. And again this is simple to implement. Allow any SL item to have a special branded “tag”. You can have as many tags as you want (and create your own tags if you wish), similar to the concept of SSL certificates for web servers, or DigiMark for digital images. But here is the trick: you can only sell/transfer items with a tag you own (or issued by a group that you belong to).

   This allows for self-regulation. So you can develop your own tag, and only you will be able to sell your own objects. If someone breaks the wrapper, the tag is still there - meaning that if the scammer copies your items and replicates them, they will not get your tag, but his own. Established creators will be able to inform the public that they should only buy items with their own tags. More than that: if you have several shops, or your own retailers, give them your tag - so they become “licensed, official retailers”. You may revoke tags on them if suddenly you’ve lost confidence on them.

   What about small, unknown merchants? That’s easy, that’s why you get “trade associations”. Use RATE (or a similar association) to give “credibility” to your own name. This is the same thing as having VeriSign signing off your SSL certificate, or DigiMark’s “stamp” on digital images. So, by using a RATE tag, your objects for sale will show the customer that you are legitimate - since RATE could revoke your right to use their tag at any time if you don’t abide by their rules.

   Vouching for third parties is how trust relationships are done in the real, digital world. The good thing is, we don’t really need a mechanism like There has (company-only approved sales…), but we can work it out for ourselves inworld, assuming we have simple extra tools!

   (Allegedly this could currently be done just by setting an appropriate group on the object, but also see the next point.)

   (A problem with this system is how to implement GIFTS! One alternative would be allowing you to give stuff away that you’ve bought, but the recipient would get a Dialog Box telling “XXX is giving you an unlicensed object YYY. This may be a scam. Do you wish to accept it?”, and the creator would also get an IM/similar thing telling her that an “unlicensed transfer was in progress”. If it happens just once in a while, that’s ok. If it’s systematic, you caught a scammer!)

5. Information/education

   Last but not least, I find that the current information levels provided to new users are appallingly low. If “newbies” never read the TOS, the forums, or never go to Mentor events, or don’t talk to anyone inworld - how can they know about stuff? They need more and better information. I’m currently on the process of writing something like a “newbie guide” - not from the perspective of the marketing department of LL, but from the resident’s point of view. It’s no use to have a clever ownership tracking system and tags to certify your products if newbies never heard about those stuff. If I make an illegal copy of Microsoft’s product CDs inside prettily printed packages, and give them away, people will only know it’s a scam if they know that all Microsoft products have a seal with a hologram on it. But if I haven’t heard about that, how can I know I’m not being offered a genuine item? So information is one of the key issues to prevent scamming.

   This means warning newbies that items without a proper, conforming tag are probably scams, and by buying them off scammers, both the scammer and the person buying the object are liable to be reported for abuse and expelled. Now that will make people think twice about buying stuff from “unlicensed” retailers!

As you can see, I’m not really “against” either the current permission system, or against any other system that will replace it. Emotionally, I both understand LL’s idea of a “weakly copyrighted world” to promote more creativity, as well as creator’s concerns about the scamming market growing exponentially and without control. However, there are ways to deal with both issues at the same time. I think that we should concentrate on them instead of just “throwing off” the proposed new permission system. Linden Lab has never hidden the fact that they want to do several social experiments with Second Life. That’s quite ok, I think, provided we have mechanisms to mantain our society and economy stable - and this means a constant influx of creativity and somewhat stable prices for products, as well as a genereal willingness from customers to pay a “fair price” for creative content.

It’s still fresh in my mind, so I guess it’s time to write some thoughts on it, and then I’m going to see if Lordfly Digaridoo’s SL Economy Analysis Blog” already has some insight on the Town Hall meeting yesterday with Cory Linden. I like his economic insights very much, and as one of the leading “economical geniuses” of SL (the other one being Jacqueline Richelieu, of course), I wonder what he’ll say on this subject…

So Linden Lab™ now wants to set up an example on digital copyrights. Actually I currently think that their vision is so overwhelming, I’m almost afraid that this will miss completely the point with the totality of the residents - past, present and definitely future. People are really “not ready” for it yet

In layman’s terms, the proposed change on the permissions’ system (read all about it on Cory’s blog) is technically quite simple. The current system protects the creator, but not the owner. This means that the owner can’t “disassemble” something he just bought to understand how it works.

LL thinks this is actually a “bad thing”. In real life, you can dismantle a toaster you just bought and take a look on what’s inside. Of course, you can also take some parts out and build a better toaster. Eventually, this will lead to a spread of knowledge, enabling people to build better toasters (and why not, make a profit from it).

The argument is slightly skewed because in the real world you mostly have “strong copyrights” and patents. So you can break an item apart, but you can’t create your own, using the same bits, and sell it as your own, at least in theory.

But LL wants to implement this “breaking apart” idea. You should read Cory’s blog for more input, but, very briefly, in the summer of 2005 or so, objects will have a “wrapper” or a “seal”. If you break that seal, you can see the contents, but you won’t be able to transfer the object to someone else - thus enabling you to understand how it works, but not to resell it “as your own”. LL thinks that people will be able to acquire more knowledge faster, and improve on existing designs, thus creating better and better content.

“Weak copyrights” are in place when you allow people to use your work by forfeiting several of your “rights”. To get a much more complete view on this fascinating subject, take a look at Creative Commons. According to what I’ve read, there have been many talks between LL and Creative Commons, a non-profit organization which is trying to bring new licensing models to the authoring world. You can see them as the Free Software Foundation for non-software authored content - music, images, video, text/hypertext. So things like the GPL or BSD licensing (which are widespread among the software community) are “converted” with appropriate legal wording to other “copyrighted” material.
LL seems to really believe that this is “the way to go” and is starting to pave the road for it to become a reality inside Second Life®.

A simple example. I write these silly articles on my blog. I’m interested that people read them for free and even copy my text and use it on their own blogs (that has actually happened a few times ). They can even put the text into commercial magazines and sell it if they want. But I want to make sure that at least I’m credited for what I write. Now current copyright laws are simply to restrictive - I either place the whole site in the public domain, or reserve all rights to myself, and people will have always to ask permission to copy things and ideas from it.

In our fast-paced digital economy this may not make much sense. After all, my “audience” are just 5000 or so page views per month, and I don’t really “benefit” from “reserving all rights” on my work. I would benefit much more if people simply copy & pasted text here, not needing to ask me anything about it (that means, I’m actively encouraging people to do it!). I just want the credit. Even if tomorrow’s New York Times opens its headlines on “Visions on the Metaverse” and someone wins the Pulitzer Prize with that text … well, Gwyn will have some credit That’s what matters!

The same applies to all content created inside Second Life®. As you know, for several months now LL has allowed digital content creators to retain their own copyrights on objects/items created inworld. However, enforcement on those copyrights have been a very hard task. The current system is not sufficiently flexible to allow for “partial” or “weak copyrights”, and that’s why we have the proposed changes.

Are they so important at all? Well, LL’s directors and visionaries seem to answer that with a strong “yes”. They really believe that “weak copyrights” is what the digital world needs to exchange information quickly and bring better products to consumers. This applies not only to your fabulous 15-prim vehicle or 40-prim house you just built in SL - but LL advocates this view for all digital content, as a matter of principle.

So LL is entering the bandwagon of GPL/BSD licensing advocates, and getting closer to the open source community. Yes, Cory talked again about this. There is no question on releasing SL’s source code to the community in the future - just the timeline is not fixed yet, and LL has to see what their priorities are right now. They understand thar open sourcing it will allow them to get much more features/bug fixes in a short time, but may also scare off some residents - or RL merchants setting shops in SL - because of eventual chaos on “non-LL-managed” sims. This will all have to be approached very carefully.

Still, the message was clear - LL wants to have a “model society” on SL, where innovation, knowledge and information are exchanged freely, to the benefit of everybody. This reflects a little what goes on in the Internet - where people started to download “free” MP3 from Kazaa and then Steve Jobs made that idea into a solid business venture. But, like the Internet (which was coined as “an anarchits’s paradise” not so many years ago), Second Life® is NOT Utopia, and thanks goodness for that, or it wouldn’t be so challenging and addictive!

Both inworld and out there, I have been busy spreading Philip’s vision. The preliminary results are quite fascinating, and I do hope that Linden Lab™’s own marketing department has had much better experiences.

Since the “metaverse building” is something radical for the overall population - they will have to “grow into it” slowly - I tried to concentrate on the groups who I’d expect more willingness to adopt these views.

The first group would be the SL residents, of course. Here, unlike my expectations, I found out a very strong opposition. Basically, the more time people “play” the game, the less they “believe” in LL’s capacities to bring the platform beyond its current state. All seem unanimous to say that for a two-year development, the platform should be much more advanced. And all also think that we have too few residents. Even 4000-5000 new users per month is just a “small growth” compared to other “Internet killer applications”. They may be right, but there are limits to LL’s marketing department - they don’t weave miracles and conjure big campaigns world-wide from scratch.

Also, SL’s growth has “word of mouth” as its most successfull marketing tool. It is not enough, but at least it guarantees a steady supply of “satisfied” new residents. If you like something, you’ll naturally recommend it to 5 more. This is an “almost-universal” old marketing rule!

In this way, I was able to interest a few “real world” business ventures - mostly SME’s - to take a look at what SL could mean to them. Strangely enough, I had much more success with companies than with individuals. Here is a breakdown from the reactions from so-called “enlightened individuals” (people connected to the science fiction medium, or the computer industry):

It’s too expensive

Actually this is usually the first argument used ever. People just go to SL’s site, understand that they have to pay for the game, and they go away, complaining about the capitalistic view of LL which exploits us poor residents in order to make shareholders filthy rich.

Well, I usually have several counter-arguments for that. The first one, of course, is pointing out that you can try SL for free, and that you’ll get lifetime access for the price of two meals at MacDonalds’. This usually makes people reconsider their initial argument and they go back to the site and at least download the SL client.

Another argument is pointing out that, through GOM, Premium Accounts get a stipend which can be converted into more money than the monthly fee. LL gets happy due to increased income, and residents are able to make money from the game. They won’t be rich from living on stipends, but at the very least they will “feel” less “capitalistic oppression” by the nice guys at LL.

And finally, I usually point out that the problem with the 3 or 4 open source metaverse projects is the lack of enthusiasm on coders. Replicating SL, from a pure technological point of view, is not very hard. The real problem is managing schedules, antecipating user expectations, building a community, mantaining the servers, give excellent technical support both inworld and offworld. And, For that, you have to charge people some money. Again, no one in the western world, able to pay for a broadband connection, is going to be “exploited to the death” by contributing just 10 bucks for LL to give us all that

Second Life® only appeals to people with serious mental disorders, like schizofrenia and split personality

This is a very interesting phenomena, and apparently it has been well documented for other types of chat systems (I’m trying to get a good collection of pointers to them). It is a much stronger argument since it is hard to undervaluate the potential of SL’s appeal to those types of mental disorders.

Actually, SL is currently being used by some therapists to help people with social disabilities to function in a “normal way”. People recovering from crippling diseases also use SL to “have a normal life”, ie. interacting with other people and having fun. Since the old truism “on the Internet, nobody knows you’re a dog” certainly applies to SL, too, this means that people won’t have prejudices against you. You are defined by your actions and statements in SL, and not by your appearance in RL.

What this means is that both extreme cases do exist in SL - as they exist on chat systems. However, the argument is as flawed as the old question if “violent games” incite violence in youth. Several studies could only conclude that certain people - a tiny minority - have a “predisposal” towards violence which could be triggered by watching violent games or movies. Still, the majority of us is unaffected, and for several of us, it’s even an escape for our eventual violent behaviour (in those cases, violent games are “therapy”).

Certainly the vastest majority of SL residents don’t have “split personalities”. Picking on a selected few who may (or not) have mental disorders, and extrapolating for the whole group, is actually far-fetching.

This will never be an alternative to WWW/chat systems

This is a typical Luddite argument: current technology works, so why replace it for something as yet unproven?

I almost always counter-argue telling people how the WWW began it’s history. The arguments are spread over other articles in this blog, so I won’t repeat them. Human beings have short memories. The vast majority (90%) of Internet users nowadays joined after 1999 - meaning we had already established the WWW as “the” medium for communication at the century’s end. Oldtimers remember the Internet without WWW. It was just 10 years ago, and those types of arguments still can see on historical forums like “Déjà News”: I have been reading my own posts from 1994 and they are hilarious! Paragraphs usually start like this: “if you haven’t installed Mosaic, please try it out, the WWW is the next big thing on the Internet”. Replies are almost verbatim on what people tell nowadays about Second Life!

So the best argument for that type of opinion is simple: go to Google. Read the very old posts about WWW in 1994. See the kind of “flame wars” that “radical new technology” aroused on everybody. Lookup the posts for Tim Berners-Lee or Marc Andreesen, and see how the Luddites where all over those guys. See the first post of Linus Torvalds telling about what would become Linux, and cry with laughter at the replies like “that will never work”.

Aah, it’s so easy to be a visionary when you know the future! But poor contemporary visionaries. Quoting someone from SL, “I don’t have sufficient data to tell me if SL will be the metaverse of the future, but my guess is, it won’t be”.

The technology is too primitive for this to become a serious platform for the metaverse

This is another truism. Sure, SL has bugs and lots of limitations. At an age where even mediocre 3D games sport 30 fps and photo-realistic rendering, SL shows cartoons with stuttering framerates, lots of lag, and a broken interface. And that’s after 2 years of hard programming.

However, once more people forget their history lessons. Microsoft was a rich company already when they launched Windows 2.0 - and there wasn’t anything more ridden with bugs than that. Mosaic, the first graphical browser of the WWW, just had a gray background that you could not change. This is how things start in the Internet - they aren’t served as “perfect applications” since Day One. We are the pioneers of this brave new world, and we suffer from that.
The next generation will have it much more easier. Around 2008 or so, SL will have probably a few hundred thousand users. Graphical cards and CPU power will have quadrupled by then, and you’ll be able to have a 10 Mbps connection for a few dozen
dollars per month, so average throughput to SL will be probably 1 Mbps and not 100 Kbps. That means at least ten times more textures, ten times more avatars per sim computer, ten times higher prim limit, and much more realistic landscapes. That will be the time SL will “explode” exponentially. And 4 or 5 years later, we will have 1 Gbps links for a few dollars, supercomputer-class CPUs on our laptops, and VR googles for display.

So the technology is there! Philip and his team could run SL on a supercomputer today and show you the metaverse of tomorrow. But who would be able to afford it? Remember how Internet videoconferencing started, with Philip’s technology back in 1995, using modems to transfer data. It was awful, but look how far we’ve come today, when things like videoconferencing are a “common feature” on chat clients…

By learning our history of the past, and seeing exactly the same arguments over and over again, it’s very easy to counter them very successfully. The Luddites will never leave us. We just need to accept that.

The approach is all wrong - a company trying to use proprietary technology will never succeeed

I have mixed feelings about this argument, since I also believe that nowadays the best projects are lead by companies willing to engage in open platforms (not necessarily open source platforms). The good news is, Philip is aware of it, and from what he said at the last Town Hall meeting, we can expect the “opening up” somewhere next year.

Why so late? Well, my guess is, Philip is trying to follow Netscape’s steps. Start with a “proof of concept”. In the near future, there will be lots of alternatives, and he’ll open up the code to make people use SL as the preferred alternative. So instead of “stamping out the competition”, he will probably encourage people to develop “open source clients” to connect to SL’s main grid, and “open source servers” for people to set up their own grid nodes. How exactly this will be accomplished is still unknown, but it will happen.

The “stable economy”, which is the driving factor behind SL, will always be “in the hand” of LL. But “islands in the net” will be added slowly to SL’s major grid - and probably even grow at a larger pace than SL’s own. But they will be the “badlands” - places where there won’t be Liaisons to help you out, where the L$ won’t be accepted as currency, or your objects won’t rez properly. This will be a first step of development as soon as we can get our hands on the server technology

I also think that’s the reason why most of the code used for SL is either open source (Linux, Apache, Squid…) or developed by LL itself - with the major exception of Havok. This will enable them to easily “open it up”. Still, there is a lot of work to do before this becomes a reality.

The biggest advantage of having LL behind SL is essentially “driving force”. By presenting the technology as “something cool for a entertaining game”, LL is able to get enough income to pay for their resident staff of programmers and sysadmins, which improve the game continuously. Ok, we all know there should be a much faster turnaround time. But things can only improve!

So, what will defined SL’s success? In my humble opinion, getting a “critical mass” of people believing in Philip’s vision. From what I gathered, people inworld are mostly skeptics. They prophesize LL’s bankrupcy “any time real soon now” and point out that this almost happened several times in the past. So, inworld residents are “players”, but they aren’t believers or visionaries, so they won’t “spread the good news”. Actually, some of them are even detractors - they want to be left alone in peace, and the more people join every day, the worse there will be to the current resident base (ie. “oh no, newbies flying around”).

But critical mass is the key. Both in “believing” and spreading the word, as well as in joining SL, of course.

So do us all a favour, tell 5 friends about SL today

After reading several white papers on online economy, browsing around the forums, and having discussions with people inworld, as well as checking up on what Philip Rosedale (CEO of Linden Lab™) says about it, I finally did it: I started to pull in the real world economy into Second Life®.

So far, what we have seen is, residents are pushing the virtual frontier beyond the game. The best example is GOM, of course, but every time a blog goes up talking about SL, we are actually crossing into RL, and tell people “out there” how important this “game” is to us. GOM is perhaps the “bridge” we needed - a connection between economies. And economy is what SL is all about.

Statistics are harsh. We are just 15,000, and only 0.5% are really interested in what goes on beyond our cozy place in the virtual frontier (wow, I’m really getting into buzzwords again!). The rest is just having too much fun right now to give it a second thought. What they want is entertainment, and that’s what SL provides for them (yes, scripting and doing “hard work” in SL is entertainment, too!!).

I’m already 10 years in the future, and this is why I think I sound so strange to most of my fellow residents. You see, I was a skeptic when Tim Berners-Lee “invented” the World-Wide Web. We would gather around our USENET newsgroups (the “forums” at that time) and discuss if we really needed a new protocol, there were already so many around. And we made fun about the tag. That’s what people in late 1992 were discussing.

Some “visionaries” just said, “hey, but the cool idea about WWW is that you can interconnect sites using hypertext”. So what? We had dozens of alternatives. There was no centralized searching, for instance (Gopher had it). So who would care about it?

A group of people developed a new type of browser which displayed images and not only content (i.e. text). This was called Mosaic (much later the code was “absorbed” by Microsoft into IE) and it was a revolution. Other technologies were “pure text”. Now suddenly you could get designers to play around with pages. And if you have designers, you are able to do fancy sites, reflecting corporate images. And suddenly this was not a “cool technology” any more. Web sites popped up around the world pretty quickly. What later became the W3G used to have a list of “new sites today” - that’s how I was able to “prove” for a long while that I did with a friend the first site in my country back in 1993 (yes, the list has disappeared from the Web ages ago). After a few months, you simply couldn’t click on all “new sites” any more, and simple search tools were added to find the page you wanted. That’s how it all started.

By late 1994, silly companies - small, understaffed, with a handful of techies and with luck a crazed-out marketeer - sstarted convincing other companies that “the Web would be the major platform for business in the world”. You can’t even imagine how people looked at us that time. If they laughed, that meant that at least they listened. Most couldn’t care less for that “major business platform”. It was academic stuff, or stuff for kiddies. Later on, the excuse was that the Web just had too much sex, and no “decent company” would really want to associate with that.

I don’t know when suddenly critical mass started to leverage things out, but I would guess it was at the time Microsoft launched Windows 95 and said that the Internet thingie was silly, and they would give the world a new and amazing technology, called Microsoft Network. Just by despising the Internet, Microsoft shook and rattled the world - people started to talk about it, even if only to spite us poor Internet “oldies”, who were placing our bets on the wrong horse.

Well, we all know what happened. Microsoft ate its hat, and “fully supported” the Internet after a few months. As we all know, Bill Gates is amazingly clever. He made a mistake, admitted it publicly, and went back to business as usual, launching IE and killing Netscape. Still, the world was out. Microsoft believes in the Internet (after a hiccup and a false start). So everybody started connecting like crazy and put web sites up at a much faster pace!

About that time, a young visionary thought that “the Internet would be the communication network of the future”, probably reading articles on how people wouldn’t be needing to pay their phone companies for calls over the Internet. And he did some amazing videoconferencing stuff at a time where people were marvelling at the brand new 28800 bps modems. When videoconferencing started to become “commonplace”, this guy just about “invented” audio and video streaming. Now let’s appeal to the media - newspapers were covered with the graphical web browser, but what about TV and radio stations? You know all about this guy. He made SL.

Being able to predict correctly what will happen in 10 years is a very difficult task for a “visionary”. Steve Jobs had it, and he was kicked out of Apple for a while, until they brought him back. Larry at Oracle thought it would be cool if his database server worked over all operating systems, instead of just a selected few. And the guys at Sun really believe that the computer is the network. They were the only ones at that time, of course.

The technology behind Second Life™ is the “next generation web”. It’s the metaverse. It’s version 0.01 of the Matrix - still crude, still with many bugs and glitches. Just like that first version of Mosaic who gave the world the wonders of WWW, but was still in its infancy and gave the “visionaries” room to dream about the future. And IE or Netscape or Mozilla or Safari are not so different from Mosaic, after all. All the basic stuff - navigation across pages, bookmarks, the way pages are rendered - was invented back then in 1993. We just made it better, we improved the technology, we expanded the feature list, we interconnected more stuff. But it’s basically the same technology.

So that’s what will happen in 10 years with LL’s “technology”. It will have a few hundred million users. They will have faster computers, and perhaps millions of prims per sim. Users at home will buy cheap VR goggles (they are already very cheap today) and do on the “next generation of SL” what we currently do on the Internet - business and entertainment. Companies will set up virtual franchises on SL and sell virtual items there, and ship real items to the avatar owners. We’ll browse the Web inworld, and send IMs or inworld email to colleagues and customers. We will do technical support on our products, like Liaisons and Live Help do support on SL right now. We will set up virtual meetings with our colleagues and customers in SL, and use XyText or similar stuff instead of blackboards and video projectors. We will do e-learning in SL - currently, classes by the several universities in SL already provide much better education and training than any other commercially available tool for e-learning.

You’re laughing right now? Yes, I thought as much. I suggested yesterday to one of my colleagues at work that we should do the meeting in SL instead. He laughed, too.

But people also laughed at Amazon when they started. They said that a “Web shop” selling books online would never catch, people need to get in touch with bookshop owners and get recommendations. They were probably right, but Amazon is still here. The same, people said, would apply to clothes - you need to feel the textures, see how they fit you, nobody would sell clothes over the Internet. And how about supermarket shopping? How would you know if the products were fresh by seeing pictures on a browser?

We all laughed at those “visionaries”, but the truth is, you do it all right now - even after the dot.com burnout. Now we’re used to the technology.

SL is just another technology - far better. It’s not because of the nice computer graphics. It’s because, unlike web pages, you get a human touch. So you can put Amazon inside SL right now, and have the friendly bookshop owner there to recommend books. You can have clothes shops with their online catalogues on SL - but you’ll be able to see how the clothes fit your avatar, and talk to the shop owners and ask for recommendations. And you can even have a virtual supermarket where suppliers also pay more for shelf space in a virtual world. Suddenly, everything which applies to the “real world” can be done in SL, and it’s far better than using the Web.

Because we humans adapt easily, we thwarted our minds to deal with the unhuman Web, and in ten years we pushed it to the limits. But we’re still much more familiar with “natural” and “humane” things. If I want to buy a new Mercedes, I would much prefer talk to William Beckett in SL and see his latest recreation of the “new Mercedes”, drive it around for a while, instead of just seeing pictures online.

All of this - getting the RL economy to take an interest into SL - is what makes me shiver. Philip is right. The Web is dead, we can’t go much farther with that in the next years. We need the metaverse.

So I did it. I started to send emails to RL companies to take an interest in SL and set up “virtual shops” there. I’ll be glad to help them out It just feels like 1994 back again. The good thing about it is that I’ve learned quite a lot since then, and while I had serious problems to convince people that the Web was here to stay and that it was the best thing invented after sliced bread (because I wasn’t so sure of it at that time), now I know all the skeptics’ arguments by heart

This time, I will be laughing

This is taken directly from my post in the forums.

---

After a discussion on the rating system with Moon Adamant, we have figured out an interesting “alternative” rating system. It has a slight problem of “computational resources” required.

We both feel there is a need to “extra-rate” someone very special to you. Since ratings are over-abused, how can we mark in some way that “special friend” we love in SL?

Some forums/online dating systems have a “fan system” beyond simple “rating”. These ideas are loosely based on that.

First, “friends” are public. You expect people to know who you make friends with. So they will be listed in your profile somewhere. Like in RL, choosing the “wrong” friends means getting a “bad” reputation. Making friends with the “popular” people mean you are seen at with awe! To make things interesting, there should be an upper-scale limit - let’s assume 25. This makes profile still manageable and captures the spirit correctly.

So, “partners” are two-way commitments where both agree with the commitment. “Friends” are one-way commitments without need for the other party to agree (more on that later), but you assume the “friend” will reciprocate (or else you’re not really a “friend”, you’re a “groupie”!). The rest of the residents are “acquaintances” (and you keep their calling cards).

To keep the “friend” status, you need to spend with him, say, one hour per week. If you don’t, you get a warning - “Are you really sure X is your friend?? This week you haven’t even said hi to him/her” - and the “friend” just reverts to “acquaintance” status (ie. a normal contact on a calling card). Both will receive a message informing that the status was “lowered” to acquaintance (so your friend may become angry for being neglected…).

What is “spending an hour per week”? Well, in RL you go out with friends and phone/email them to keep in touch. Since IM can be easily abused - you can write a script to “spam” your “friend” and this would count toward “friend status” - I’d limit it to “physical presence”: being at the same place, at the same time, for at least an hour, every week. Ideally there should be a “limiting distance factor” to avoid abuse (ie. friends should be talking together, or going out to the same spot, etc.) but this would mean lots of additional processing. You can do it much more simply by listing the time both entered the same sim. It’s crude, but allows for offline processing of logs in a straightforward way. So at the very least, you’ll have to IM your friend with a “hey, it’s time we keep our status, let’s meet at X for an hour, we don’t really need to talk, just be there”. Of course, since you have to do this to up to 25 friends, time management will become an issue!

Also remember that “friendship” is not two-way. This means that you can list up to 25 people that you consider “friends”. But you can have several hundreds or thousands of residents considering themselves your friends (ie. “groupies”). They’ll try to get to same spot where you are to retain “friend status” - so a popular guy, hosting events, will probably get much more than 25 people listing him/her as friend. They’ll be at all events/places where he/she goes. This can be irritating, as “groupies” are irritating in RL, too!

Of course you can subvert the system. You can pay people to have them list you as a friend, and pay them every week to stay with you. However, this easily gets out of hand, and probably only the very wealthy, with lots of free time, will be able to do that. And will it “pay off” in the end? (you may need to pay people more than you’ll receive from “friend status”…). Remember that you can only have up to 25 friends - so how many of those will you keep as “payed friends”?

Sooner or later, you’ll have to make choices: you already have reached the limit, and have just met this fantastic guy you want to list as a friend. Which one will you drop? This is a similar choice that happens with the group limit, but here we are talking about social interaction. People will tend to keep friends they really, really like, and that are willing to spend some time with. So, like in RL, you won’t have time to spend with ALL your friends, even if this would mean earning lots of L$ for having many friends. Better to concentrate on a few key friends that you usually hang around with, instead of having a whole lot of 25 friends and never get time enough to spend with all of them.

This encourages grouping, or ganging together. Instead of having 5 separate friends, it’s much easier having your own group/gang. You 5 do everything together anyway. So it’s much easier to go out together, and spend 1 hour of online time with all five at the same time! This will encourage you to seek out a larger community, but not spread it too thinly.

So how would Lindens “encourage” friendships? Instead of a rating-based bonus, you would get a “friendship bonus” at PayDay, based on the number of hours you spend with each friend. This should be a balanced formula, so that spending 25 hours with just 1 friend (”my, he is really, really your friend!”) gives you a very good score, but spending 1 hour with each of 25 friends makes you more popular, so you should get much more. There are tradeoffs, however. A “recurring friend” (ie. someone that you keep week after week with “friend” status) should give extra bonuses. This means that a long-term relation pays off, instead of getting “a new friend every day”. This is also relatively easy to implement.

When you look at someone’s profile, you’ll see a list of friends he/she has, and you can order them by “popularity” or by the profile owner’s own preferences. “Popularity” is just a number of people that refer you as “friend”, multiplied by the number or hours that person actually spends with their “friends” online. But the profile owner may list “more special” people instead of “more popular” ones. After all, you may be an Eggy or Torley groupie (who isn’t, these days…? :-D), but still list your sweetheart first.

First, a disclaimer…

I do not have any type of connection to Linden Lab (wish I had, though ). Information collected about how the system works has three main sources: looking at the logs (SecondLife.log), reading the forums, and talking to older residents (who sometimes have exchanged ideas with Jeff, Ben, or even Philip himself).

Philip and Cory themselves have written a technical paper on the technology used. This is excellent reference material!

Most things are “obvious” for someone with some experience managing Internet server farms. This is not “magic” or “reverse-engineering” or “hacking into machines” to see how they work from the inside. Like a doctor can tell you why you’re ill by just feeling your stomach and making some questions, a system administrator can make a pretty good guess at how things were implemented by just seeing how they work (or better even, how they fail to work). The more experienced you are, the more different solutions to the same problem you know, and the more easier it is to see which one was used. It’s never easy to do something radically new from scratch when you know about “best ways” to deal with some issue. Why reinvent the wheel when someone already has done the crude work?

Since I have no way to confirm most of my assumptions, they may be wrong. Don’t take them literally. Yes, I have been wrong several times in my life. I usually just don’t admit it in public (RL or otherwise) because I like to be considered a stubborn, know-it-all loudmouth.

And finally, my natural curiosity impels me to try to understand how things work. I hate things I don’t understand, and am stubborn enough (I prefer the word “persistent”) to waste my time thinking how things could have been done.

Many, many thanks to Eggy Lippmann who reviewed this article and told me where I was completely off-track. This “version” already reflects some of his competent technical review.

Of course, I also try to update this article ever so often, as new things are “discovered”, but may rest assured that I’ll probably never get it absolutely right!

Now a warning:

This will be filled with geekspeak (yeah, Gwyn talks the lingo, too). So this probably be BORING to most of you.

Why is this article here, then?

99.5% of the residents thing LL is a crappy company which never cares for the players’ needs. Everybody knows about the bugs in SL, and some of them are here since beta (18 months ago or so) and have never be fixed. New releases just introduce new features with more new bugs. Actually, people still keep playing the game and pay a handful of dollars every month to LL because they have so much fun - and not because the game is any good. The interface stinks. There is always lag. Sims crash a lot. People keep losing objects in inventory. And there are thousands of posts every day about feature requests and bug reports, and the Lindens don’t care about it.

Pretty picture, isn’t? But this is what we hear every day…

The remaining 0.5% are too fascinated by the game (as a sociological, economical or technological experiment) to gripe about it. These things are “annoying”, but they are a price to pay when you become a pioneer exploring the New Frontier. And, of course, we know why things are so hard to fix.

The first thing you need to understand is that Linden Lab is NOT Microsoft with its 5000 software coders (or are they more?). They have, at most, 15 programmers, and perhaps 3 or 4 system administrators. Even taking into account that “core functionality” was developed by Philip a loooong time before beta was ever launched, that’s not much. And these guys are developing a “game” on 3 different platforms with a new software release every week or so, fixing about a dozen bugs, launching about a dozen new features, and testing them all before they release it to the public. You may not appreciate that much, since Microsoft does about the same. Yeah. Right. But they have 5000 guys working just on that. Oh, so you’re going to tell that Linux does even better, and there is not even a company behind it? But Linux has, like, ONE MILLION programmers working on it!

So let’s keep thing into perspective, shall we? Linden Lab is a small company, running 1000+ servers with a game with 100.000 angry players who complain every day with thousands of requests which are never fulfilled, with an understaffed tech team, and employees not only working overtime, but gladly replying in the forums, being online off-duty (even on weekends and holidays) helping people out, and spending as much time as they can to get things running smoothly. Microsoft hasn’t 1% of that attitude and is the biggest software company of the world

Core functionality

You read about the many good things about Second Life in their Press Releases, but their marketing is targeted to a broad audience. If they would do a press release for a geeky audience, here is what they would write about it:

LL is a hosting provider (not a content provider; content is designed by residents, not employees). Their core business is setting up a technology which enables creative people to provide content and entertainment. So they are not really competitors to other MMORPGs, but rather to hosting providers. Think about Second Life as a 3D combination of WWW with chat rooms with entertainment/games.

Hosting providers sell “Web space” by megabytes of consumed hard disk (or by bandwidth). LL sells “SL space” as “land” with “prim allotments”. You pay monthly fees to have some megabytes of Web space, and in SL you pay for tier. Same concept, different words. Hosting providers can give you “virtual servers” or even whole machines, where you get some tools to manage it, but you’re pretty well on your own. LL gives you the opportunity to rent a whole sim. Actually, prices are very, very similar. Finally, some companies also sell software to manage machines (eg. operating system and some remote administration tools). In the future, LL will give you a license to run your own sim, or even open source the code.

So this means that the “grid” (how we name the Matrix in Second Life) has to be segmented in “managable bits” so that LL may rent it (that’s where their income comes from, and NOT from players joining the game or paying for the Premium account…). As we will see, this will seriously condition the whole system architecture, and you’ll also understand better why LL used an approach which is radically different from their competitors (I’m not saying it was a good choice. It’s different, because LL is not a “gaming company”, but a “3D content hosting provider”. It’s not the same!).

The “game client” - the piece of software you install on you PC, Mac or in Linux early 2006 - is just a 3D browser and nothing more. It will work with every sim it connects to (that doesn’t amaze you? Think about your web browser has problems connecting to different sites…). When you get the client application, “almost nothing is stored there”. Even your avatar is not stored on your computer. This is precisely the same approach used by Web browsers (ie. when you download a Web browser, there are no “pages” or “default images” or whatever which come with it… you connect to web sites, you download pages, and they get stored in a cache for later usage).

Actually, according to Eggy Lippmann, a few things are stored in your client: your agent’s inventory (just pointers to UUIDs), the “default” ground textures, default “system” animations and sounds, etc. They simply pale in comparision to what is available online (around 60 MBytes, compared to several TB online).

Why is this so? Why, unlike other games, do we have to download everything? Wouldn’t it be more effective to have already “major” textures/objects ready on our client application instead of getting it from the Internet?

The answer is: no, it’s impossible. With almost 2000 sims (one “sim” [short for simulator] is just an area, and it used to run on a single server; nowadays, the newest servers run 4 sims each), and 100.000 residents, content is several terabytes of data. Not 100 or 200 MBytes like on “conventional” MMORPGs. And the worse part is, what are the “major textures”? Every day the fashionable places (or clothes!!) change, and there is a different set of “major textures” - just like on the Web there are trends which define the “most hot” sites at the moment. Now think what would happen when SL becomes a game with a few million users and some petabytes of data! (1 petatype = 1024 terabytes = 1024 * 1024 gigabytes)

So the reverse approach - big, fat application clients with lots of content built-in - are great for closed and controlled games, where the company has set up an environment for the players to explore, but it simply doesn’t work for the kind of company and “game” SL is.

Streaming technology fundamentals

The core of SL is streaming. It’s no wonder, since Philip Rosedale (the persona behind Philip Linden) is the guy who “wrote the book” on streaming media We associate streaming with either video or audio, but LL was clever - they did it with textures and other stuff.

Basically they super-compress textures and deliver it with a technology not unlike RealPlayer to the application client. Just like Real (or QuickTime, or MS, or MP3 streaming…), the trick here is being able to have a single server streaming the same stuff to several “browsers” at the same time, and deal with things like packet loss, textures arriving out of order, and similar stuff. What they do is almost uncanny - you can only appreciate it with an awful connection (like I have) which usually drops packets like crazy anyway, and has fluctuating ping times. I think I actually have the worse possible connection to SecondLife - it has lag, it has noise, it has jitter, and, worse, it has a dynamic allocation of packet priority for heavy usage (this means my ISP simply drops all my packets first and lets the kiddies download their pirated DVDs just because they don’t do it as often as I play SL ). Still, everything works smoothly in SL, most of the time!

Now, for those of you who know a little bit about streaming media, you know you usually have an “emitter” (where the media stream is encoded) and a “broadcaster” (where the stream is broadcasted to the users’ application client). Most also allow for “repeaters” or “reflectors” - they are a type of broadcasters who allow for additional streaming (they connect to a broadcaster “like a client” but also allow clients to connect to them and get a “streaming feed” from them). This is, for instance, the way Akamai does it - they try to put “repeaters/reflectors” all around the world to provide the fastest streaming content to end users.

In SL, the broadcasters are the sims we connect to. So, besides holding all the terrain data (and lots of more stuff that we will see shortly), the sims are also responsible for feeding us with streams of textures.

Where is the “emitter” then? Actually, the sims emit and broadcast at the same time. When a sim boots up, it talks to the asset server, looks what it needs to load (i.e., which objects/textures are rezzed in at that particular sim), and broadcasts it to all users on that sim (and also on the neighbouring 4 sims, of course).

There are no “repeaters” or “reflectors” in SL at the moment. These could be a way to provide better service on the East Coast, Europe or Australia/Asia - just set up a few repeaters there.

It takes two to tango, but…

… things in SL get confusing very fast. Where are objects stored? Actually on several different places - according to the white papers, each object is initially stored at the place it was created, and moved (or better: copied and stored at local caches) across sims when needed. The asset server seems to have “pointers” to every object, texture, animation, sound, etc. Overall composition and current “physical location” - ie. the vertices, how prims attach to each other, which texture go into each face, pointers to owner, group, and, finally, where it currently is stored, are actually on MySQL databases, stored inside each and every sim. Inventory has a link to the object when not rezzed in. And sims also have the position of the object when it is rezzed in (and its textures, too). Universal IDs (UUID) are keys for all those components. Think about the “asset server” as being similar to the DNS servers on the Internet — they have just pointers to things, not the things itself.

To do this efficiently, every sim has it’s own “local copy” of the asset server (apparently, according to Eggy, they have squid acting as cache server, so a request goes for an UUID to the asset server, and the reply is cached locally). Of course, when you upload things, or move it across inventories, you have to make sure the asset server is properly updated.

This is also the reason why new releases usually are a bit slow at first - Eggy says that the cache is cleaned on each grid server for a new release. This means that Squid will need some time to fill up the caches!

(A question I still have is about intra-communication between
grid servers and the “auxiliary” servers. According to the white papers, UDP streaming is only used for broadcasting objects to the users - ie., prims, textures, etc. The sims run Apache and Squid, so I’d guess it’s plain XML over HTTP in queries between them. This tends to strengthen a paper where the people at Linden Lab thought they could do everything over an HTTP connection but gave it up because of the overhead needed to handle thousands of TCP/IP connections).

So the most hard part of SL is when you buy an object from a vendor (the vendor has a pointer to the object in inventory… then the object has to be rezzed in that sim… but it goes into the user’s directory instead and is not really displayed), attach it (suddenly the object is rezzed in), you move to a neighbouring sim (no problem here, since every avatar is listening to streaming from the 4 neighbouring sims anyway and can get the necessary data directly from them) and then use the teleporter! Oops… now you have to tell the asset server that you’re bringing all that stuff with you, inform the sim (and the neighbouring sims) that these objects are suddenly not here any more, contact the new region, ask it to get the objects’ location and textures from the original region, update inventory and asset server with the new locations, display your avatar… and hope that you got all the objects correctly positioned since you left the region you were in!

No wonder people keep losing stuff from inventory when teleporting! (fortunately, this has progressively improived over time)

Drawing avatars

And don’t forget that your avatar “comes with you”, too. It’s just a 10.000-polygon object (or thereabouts… just to have an idea, a prim cube has 6 faces, so 6 polygons if SL uses squares, or 12 polygons if it uses triangles. Older graphical engines only used triangles), highly difficult to draw correctly, and as you know, SL deals much better with avatars than with high-prim objects!

The trick, according to Eggy Lippmann, is that we actually are all “Children of Ruth”. What does this mean?

Well, according to SL Myth and Legend, Ruth supposedly “really existed” and was one of the first alpha testers. At that time, avatars were built with prims, as everything else, and later avatars were converted to meshes (which we use nowadays). How exactly the “Ruth” avatar became the “first” avatar, is not clear to our myth recorders But in a sense she’s our “mother” - that is, ALL avatars start being Ruth, and then settings are applied on top of Ruth’s AV, so that they look like us! Weird idea, huh? Actually, this is what makes SL render avatars superquickly - the application client is something like a “superfast Ruth renderer”. It just gets a small string of settings (200 settings which you set using the Appearance interface - just a few bytes!) from the asset server, and - ta-da! Instant avatars! You just need a few textures for the clothes and skins, they are “baked” (i.e. all clothes layers and the skin are sort of “fused together” in a single texture, and sent to you), and the avatar is done (thanks for Eggy for that piece of info, I really hadn’t figured that out for myself, and I didn’t understand how SL could render so many prims so fast - after all, one sim can only hold 15.000 prims or so, and we know how long it takes to render all of those…)!

The Sim Lottery

When a sim boots up, it’s “empty”, and it loads all the data it needs from a central server. This is a very intelligent approach, because it enables the Lindens to concentrate on a failing machine and have a replacement very fast online (the central server is also an excellent place to make backups from). If a sim fails, you just tell this central server that a new, empty machine needs the sim data, and afterwards you can do maintenance on that sim at your leisure (probably means rebooting and just formatting the machine from scratch with a default configuration). In the mean time, people will already be connecting to the “new sim”. There will be just a “glitch in the grid” for the avatars who were on the failing sim when it was “switched” to another machine - they will gracefully time out and try again by connecting to the new sim. With luck, your application client won’t even crash.

There must be hundreds of different tricks beyond this. A sim, on the other hand, very very rarely looses its objects - the worse that happens is having “lost objects” (ie. objects stored there with no entries on the asset server). Terrain data is a 512×512 “raw” image data (with special channels for indicating land, water, parcels, etc. - the information is available elsewhere, you can start with the Bad Geometry Wiki for a reference), so you can easily backup that on a few bytes. And 15.000 prims is not really much. If we assume they need to have 1 item for each face - and taking into account positioning data, which is done with floating numbers, and trimming on the prims - worse case scenario seems to be a cut torus/ring with 9 faces. So, say, 20 to 30 items per face, 100.000 faces per server, each item won’t take much more than 8 bytes (probably just 4), so 20-30 MBytes to save the whole sim’s information. It’s really not much. Probably they update it dynamically, if they’re using MySQL on their Linux boxes, you can do remote syncronization - takes about 5 minutes to setup, and you get “almost instantaneous” sync’ing. Actually I would do it in a different way - have 5 MySQL databases for each sim, 1 storing objects on the current region, the remaining 4 sync’ing with the neighbouring sims. This means you’d have 4-way backup on most sims (except for islands), which is more than enough for redundancy purposes. In practice, LL does not do it their way — they just keep open connections from each neighbouring sim to your avatar (or better said, to your agent session). But they still do a remarkably good job in mantaining those databases consistent. My theory is that on the mainland, the chances of dropping an object and “losing it” are virtually nil. The worse that can happen is that you can’t update it on the asset server…

The big, big problem is backuping all those textures… if they were distributed evenly among all servers, it would mean that each has around 250 MBytes of texture data. You understand now why Linden Lab is so reluctant on “restoring backup data” on a sim, just to get your precious, lost object back.

This random allocation of sims is called by oldtimers “the sim lottery”, because you never know when it will be your turn to get an “older” machine. Newer machines use hyperthreading CPUs, and they run 4 sims at the same time (using mostly virtual machines) but there are still a few old machines around (according to the Bad Geometry Wiki, they’re 2.6 and 2.8 GHz Pentium Xeons with 512 MB RAM). Since sims reboot once in a while, there is a fair usage of all machines, both old and new. So you won’t always be on an “old” machine even if you are in the oldest parts of the mainland!

The user server

Another key server is the user server. What exactly it holds I’m still not sure, but certainly unified login is handled on that machine. You can test it, it uses some XML-based authentication scheme (probably OpenLDAP…), and it’s used for the web servers, too. Information about everything related to yourself and your avatar is here - name, profile data about your avatar, probably the groups you’ve joined, the land you own, etc.

This is the server that you connect to when you start the game, of course, but it also does lot of work. It has to be checked to know that you’re online, for instance. Or when you fly over land, it has to be checked to see if you belong to the proper group.

When this server crashes, something interesting happens. Of course, you can’t login any more, but people inworld will still be able to do some things. If you have an active IM session, or chat, you’ll be able to continue it - but your name will be “blank” (since everything in SL uses UUID as keys, stuff will still work using them, even if names aren’t available). This has been happening rarely and rarely with the latest updates. I never tried to “break permissions” when this happens, but it should be possible. However, the user server is pretty stable, because, unlike the grid sims and the asset server, it doesn’t provide much information, it’s mostly read only. Grid sims and the asset server are constantly exchanging information back and forth, and that’s why they are much more susceptible to failure.

More work for poor sims…

Besides streaming, sims really do lots of stuff… they track all object’s positions, they run Havok to detect collisions, they do the physics, run all scripts, give hints about lighting, and even do the weather on the spare time. That’s why the Lindens have to put some limits on all the stuff - maximum number of avatars per sim, maximum number of prims, maximum memory a script may use (16k), etc.

There is some stuff done client-side, too - handling some types of rotations and all the particle stuff. This means that a rotating object will show differently when seen by different people!

In any case… this means that the application client does not do much stuff except rendering. Since content is so dynamic - it literally can change while you turn your head! - LL seemed to prefer a “quick & dirty” approach: feed all data on objects and textures to the client and let it sort it all by himself how things should be rendered (and keep stuff in the cache in order not to ask for objects again). This explains why the rendering is quite crazy, with things appearing randomly at the client. There is some logic in it - apparently, scripted objects are drawn first, then physical objects (ie. no ghosts, no transparencies, no alphas, etc) next, and pretty much all the rest is rendered as soon as it arrives. Since broadcasting textures over UDP is done randomly, they can arrive out of order, and textures are stored in differently compressed versions, so that you may render them with increased quality when they arrive, but start with low-res textures if you have the overall information (according to the Lindens, JPEG2000 is an excellent format for this kind of behaviour). Things are fuzzy at the start but will increasy quality very quickly (or not… if the sim is too slow).

Chat and IM

I’m still not sure about how these work. IM is pretty easy, it’s just a two-way channel (or several-way channel) between a group of people, and it doesn’t have any restrictions at all (ie. no need to consult the sim, or any other thing, besides the user server to see if the users are all online or not). I would think that the user server handles IM as well. According to the white papers, there is no direct client-to-client communication.

On the other hand, chat is much more difficult to do. First, it is subject to “listening distance”, so it has to be tied in into the sim - the sim needs to know how far your words go, figure out who is listening on that range, and display words accordingly. That is why we get chat lag and not IM lag - a busy sim will not be able to process chat as fast as desired, and that’s why you get messages out of order (or even lose some). There is no easy way to do this: everything which needs to track sim position needs to be processed by a sim…

Bottlenecks and improving performance

If you have read up so far, this structure of the SL grid pretty much resembles how the Internet works. You have tons of individual and private Web servers, and DNS which is the “glue” which enables addresses to be converted to IPs. There are “accessory” servers like databases, email, chat, etc. but they don’t interfere with Web sites.

A similar construct is used by LL, since they want people to have their “private” sims - meaning that one sim has to handle everything pretty independently. However, inter-sim communication is easily done under this model.

Common login/user information is handled on a separate server, which makes sense - you use the same login information on the whole grid.

The problem is mainly with the asset server. You see, if one sim fails, you affect at most the 25-35 people that are on the sim at that time. If the asset server fails (or is slow), it affects everybody. Actually, the asset server is the major bottleneck on SL! Recently, they have added several asset servers running in tandem, but often they’re simply too slow.

I think that LL thought that they could minimize impact on the asset server because the streaming would be done by the sims, who would not need to do a lot of lookups on the asset server - since sims only need part of the whole asset database (theory: you have 1000 sims, so each sim just has at most 1/1000th of the database, on average). This would be probably reasonable if there weren’t many updates on the asset server! But there are attachments, inventories to update, permissions that change, scripts that are edited, and all kinds of stuff that are updated at the asset server.

I think that to improve redundancy, every time something gets updated, it’s written on the the asset server, but cached almost immediatly on the sim’s local copy. This is why you don’t lose objects so easily. If you delete something by accident, it’s on the asset server marked under Trash. If the asset server dies, you’ll have it either on the sim (and next time the asset server goes up again, the sim will try to sync the data with it) or at lost & found (basically, the sim has an object from you which isn’t rezzed in, which shouldn’t happen - it belongs somewehere to your inventory, but only the asset server knows where).

So the grid is pretty stable in tems of content persistence, even if glitches/lag occur very frequently. Again, this mirrors the way the WWW works - a stateless protocol, where you can connect “partially” and still get some content (say, text but no images, or some images but not all of them, etc.).

However, the bottleneck created by the asset server is almost unbearable - there are more and more objects and textures around, and it takes lots of time just to update it. When I started the game, dropping things from my inventory into an object was instantaneously (we had 10,000 residents then). Saving a script took a few seconds. Nowadays, with around 100,000 residents, the same operation takes between 8-30 seconds on average, and sometimes even several minutes - or times out. The problem is that there is just one centralized asset server cluster for 1000 sims! If it weren’t like that, you couldn’t bring your objects from one sim into the next…

How can you deal with this and improve overall performance?

Some older players think that “Lindens is doing everything wrong”, there should be a redundant array of grid computers, dynamically allocating CPU timeslots as needed. Under the current model, some sims are almost empty doing nothing, while others are overcrowded. Under a redundant grid of dynamic allocation (hint: read about the Beowulf project), also called a cluster, or, if you will, a loosely connected supercomputer, things could potentially run much more smoothly. You basically just have one big computer running everything - one system to manage, one system to do backup, one system to look at to see if something is going wrong. No bottlenecks. If one of the 1000 computers goes down, well that’s too bad, it just means you loose 0.1% of the performance! That’s perfectly acceptable! And there are not “bottlenecks” since applications are uniformly distributed among all systems.

Also, you could have perfectly all online users at the same region, if you wished, or one zillion objects just for one building in one region (assuming, of course, that your graphics card could render so many polygons fast enough!). There would be absolutely no problem with that model - that’s how supercomputers are built. Better still, teleporting would be as simple as just changing the avatar’s position. The whole concept of “sims” or “one region per sim” would not make any sense, you could design a world as large as you wished - 40.000 km x 40.000 km, if you wanted. To increase performance if things are getting laggy, you just need to add a few more computers and more bandwidth. Better: instead of the “sim lottery”, you could use all kinds of crazy hardware to hook up on the cluster. Things like Beowulf clusters handle load balancing application CPU & memory consumption - so you could have a few 486 running there, the CPU power would NOT be wasted.

Of course, this also means rewriting SL from scratch for such a type of system. The current software base simply won’t do it. The best they can do is running 4 sims at a time on a single hardware server — that way, CPUs are shared, and unless all four sims are very busy simultaneously, you get better resource allocation among them.

But under the current model, if one sim fails, you drop 25-50 people out. If the asset server cluster fails (and yes, even with the redundancy added, this happens), no one can play the game. If the user server fails, no one can login (still happens quite often). And so on. Even if you have “empty” sims, you cannot use their CPU cycles to help “heavilly saturated” sims. It seems to be a much worse model than the alternative!

Actually, the “alternative” is used by some of the competitors (they usually go the hard way about it, writing their own cluster software, instead of using open source solutions, but the principle is the same). They have much less content to worry about, so the easiest way is to replicate all data among identical machines, and split users among several machines. That’s why they usually get much better performance than we do in SL. World of Warcraft claims to have 4 million users, 1 million of them simultaneously online, and just a hundred servers to handle them all. There is some problem in getting in touch with your friends logged in on a different machine, but there are clever tricks to deal with that.

As good as this model sounds, it simply cannot be used by Second Life. Remember where we started our discussion - Linden Lab is a hosting provider, not a content provider. To be able to “host” things easily, they have to have clearly defined “borders”. One region per sim sounds a very good compromise. As to the asset server cluster… it has to be universal, and it has to be read/write, and deal with objects that cross regions. And finally, SL simply has too much dynamic content. You can’t pack it all on the same machine and expect it to work.

Still, you could have “virtual regions” on the game… ie. “define” that this parcel of land is used by company X wishing to rent it. Why give them a “physical” machine?

The question is, LL is once more looking towards the future… and in the future, there will be several servers around the world - probably isolated islands, or clusters of islands - hosting completely private sims, licensing the software from LL, and doing maintenance on their own sims. For this to become reality, LL needs to be able to have this region/sim concept ready. Still, I’m not sure how they’ll deal with the asset server (IM is probably easy to do and easy to replicate; user server can be cached, since user data is much more read-only than write; just the asset server is a real problem here!). The only “easy way out” is having several asset servers, and contact them using a simple hash on the keys. This means that you have to apply a simple algorithm to know which asset server you should contact - say, all keys starting from “0″ to “7″ go into asset server #1, the ones from “8″ to “f” go into asset server #2. This is a pretty simple implementation and could be easily done with a simple patch. When logging in, the application client would just need to get the hash function and a list of asset servers available [Gwyn's note: rumour says that this is eaxctly the approach they took when moving from a single server to a cluster of servers, but there is no easy way to check this from the logs]. This is vaguely similar to the concept of DNS servers outside the US (there is just one DNS system, but each country has its own server to host its own “national” data, and you know which servers to look up if you need to know other countries’ data).

Still, as there is a “mainland” (as opposed to several disconnected islands), LL could do a mixed approach. Buy an IBM mainframe designed to host 10,000 virtual Linux servers (for the cost of about 100, and the size of a mini-fridge). Since each virtual server gets its own IP address, from the perspective of the players, there would be absolutely no difference! And on this type of hardware, the mainframes’ CPU distribute the load much better - so if you have, say, 32 CPU’s, you could have 20 or 30 to handle a big event with 300+ avatars on the same region, while the remaining CPUs would run the “empty sims”. No waste of resources there - you could even have the asset server and the remaining servers running on the IBM mainframe, too. But independent private islands would still have their own, individual machines. And sims on other parts of the world - run and mantained independently from LL - would still be able to run their own software and be connected to the grid. Actually, if someone gets into the business earnestly, they could also buy their own IBM mainframe and host a “rival continent” with the same features of CPU load balancing as LL’s mainframes. Finally, I haven’t read much about it, but since you can have virtual Linux servers running on top of Linux itself, theoretically you could run those virtual servers on a few 16-CPU machines for a fraction of the cost of an IBM mainframe.

There are some technical solutions around. It’s up to LL to look into them and see what they can come up with!

Some of us have been privileged to listen “live” (or through the many repeaters) what Philip Linden told SL’s population at large about his visions and the questions he answered for the residents, at the last Town Hall meeting on October, 1st. You can read the official transcript to give you an idea.

Personally (and this blog of mine surely has shown a few mutations since my original plan for it) I believe that Philip Linden should really host two different types of events. One to explain about his visions about SL. The other to answer insignificant and irritating technical questions. The proof that he’s got a golden heart is that he answers both types with the same ease and depth, and treats them with the same respect.

Opinions on his speech have differed, and I have been talking to lots of residents afterwards. It’s so clear to me now how different and varied our own experiences are. Anyone with a business experience (ie. having owned or managed a company, having set goals and objectives for a group of employees, having managed projects inside companies or even the whole company itself) was drooling at the corner of their virtual mouths - be they youngsters in their early teens or old vets with 40 years of business experience. Unfortunately, the number of people with that kind of experience is very diminute compared to the whole of the population. Over 95% of the residents are either students, academics, employees or self-employed as freelancers - no experience in understanding how important a “vision” is for running a successfull company. It’s not ignorance. You can read about it and never get it, you just have to feel it in your bones to be able to understand it.

This means that the vast majority of residents just shrugged their virtual shoulders, shook their heads, and went on saying “yeah, yeah, yabba-dabba-da, it’s all the same ‘we will look into that bug in the future’ but never a real commitment to a delivery date, we will never have that bug fixed, and we have been experiencing it since beta”. This is not being silly, stupid, or plainly egocentric, it’s just ignorance on how things are done in the real world.

I’m not able to convince anyone who hasn’t had a previous working experience to understand my thoughts, so I can only give you a rough idea on what was important of Philip Linden’s “speech”. Not what was important to me (yes, I also was glad to hear about the upcoming Linux version, or the inventory getting a revamp), but what was important to the SL community, the Internet community, and the World At Large.

Big words? No. First you have to read who Philip Rosedale (the persona behind Philip Linden) really is. You can check his CV at Linden Lab management page, but to give you an idea, Philip is something like the Steve Jobs of streaming media. When people were playing around with WWW in 1995 and hearing downloaded MIDI files from silly web sites, Philip was doing videoconferencing on the Internet. When people were thinking that the Internet would be a cool way to download videos and WAVs with music, Philip was at RealNetworks developing the RealPlayer, which did streaming using low bandwidth requirements, by putting a plug-in on about every browser that existed at the time, and “invented” the concept of Internet media broadcasting (oh, it had existed for a while, it simply didn’t ever “take off” from the academic structure and entered the private sector). Nowadays RealPlayer is a minor player compared to giants like Apple’s iTunes/Quicktime and Microsoft’s Windows Media Player - or MP3 streaming for the open source community - but RealPlayer was THE solution back in 1996. Philip did it.