http://www.secondlifeherald.com/slh/2006/01/interview_with_.html

The focus I have here is describing the asset server as a “server cloud”; there may be a couple hundred machines that contribute to storing your inventory completely seperate from the grid…

Lex, last time a rumour was spread about the way the asset server was done in the past, it was a very, very easy system, which didn’t even require a “database”. But basically it was simply a collection of pointers to assets stored in a sim. Of course, the whole asset server got a major overhaul, as did the inventory servers, and it’s not easy to understand how things might have changed.

Still, one thing is certain. Upload a texture to a sim, it’ll load way faster there (even a very busy one). After uploading it, rez it anywhere else — even an empty sim! — and see the difference. With enough time and patience, and enough uploads, this seems quite consistent — textures (and sounds, and animations…) load for the first time much faster on the sim they were uploaded.

Now this can be due to two things:
1) The texture is immediately cached locally and then goes to a central server (possible)
2) The texture stays locally, and just a reference is made to it on the central asset server (more likely)

Of course, after the cacheing kicks in, in theory at least, you’ll be able to rez things fast, no matter where you currently are. In the past, I tended to upload everything on the then-new Sandbox Island (as opposed to my residence at the time, which is on a painfully slow early sim), and it was quite a difference…

In essence, it’s more reasonable to have the assets distributed among 5000 servers, each with its own database, and just have the pointers to the assets on the asset server, instead of storing everything there. With “hundreds of millions” of objects (I’m wildly quoting Philip from his last or before-last TH) I find it hard to imagine that LL has put all the assets on the same server – specially taking into account textures, which will take vast amounts of bandwidth to connect with all other servers…

Many people correctly conclude that there are better ways to implement the various parts of the SL project. What they often fail to understand is just how hard it is to write perfect software in a timely manner. It really is easier to conceptualize the idea than it is to build it. Even unlimited resources (time and manpower) cannot completely collapse the time it takes to build new stuff.

So how does a company with limited resources (money and manpower) bootstrap something like SL into existence? The answer is to punt the hardest parts for later, make the required parts work good enough, and complete the minimum set of features to attract enough early adopters to prove the concept to the unbelievers. As the whole project aquires more people it becomes more interesting, and the company is able to convince not only new people to use the product, but also convince investors to buy into the effort, so that the employees can be paid as they complete or fix the missing or broken pieces.

I think the obvious shortcomings of the project are a consequence of the bootstrap system. The good news is that most of the shortcomings really are solvable. I think you’ll be seeing lots of improvements to SL over the next year.

Well, currently, assets are stored on the simulator server they were uploaded first, but the information on that storage is kept on the asset server. Every time you need an asset on a different sim, all it takes is a request to the asset server to ask where the asset is, contact the sim where it was uploaded, and download it to you, while keeping a local cache at the sim you are.

I’m almost sure that this isn’t the way things are currently done. I’m basing this off the same thing you are, namely my assembly of the various tidbits of architecture information that Lindens have dropped in the forums and such. I’m pretty sure that all actual asset data is stored on the asset server cluster. Inventory is stored on a separate server as, essentially a list of “this person owns something named Foo which has asset UUID ___”. Any time something needs to be downloaded by a client, it asks the server it’s currently in, which asks the asset server, grabs the data, and forwards it on.

A sim also stores a local copy in its own cache, which is a really awesome opportunity for optimization. Given LL’s copy-on-write storage paradigm, every asset UUID is read-only, so caching can be done with 100% confidence that the cache will never go stale. Since people are always going through the sim they’re in for assets, that means that a sim’s cache fills mostly with the data referenced within the builds in the sim itself.

Also, about the whole UUID thing… I’m pretty sure Kelly Linden once referenced RFC 4122 (or one very similar) as the standard that LL based their UUIDs on. One nice thing hinted at by a commenter above was this little piece of data:

node = 6hexOctet

The RFC suggests using the MAC address, but LL may well have used another kind of node ID because sims might share the same MAC address. Anyway, every UUID already has SOME kind of unique node ID, so it’s likely that it’s very easy to partition the existing set of asset UUIDs by their sim of origin.

But why?

I’m not sure I see a marked advantage to having the existing sim servers take on the distributed burden of serving assets. As far as I understand things, a given Region can show up on any Sim at any time. It gets its “simstate” from the asset server, loads it up, and connects up to the grid and registers the Region, and people can connect. The only performance loss is that the cache has to be recreated.

If assets had to be stored on each sim, you’d have to somehow cart around all of those assets every time a sim moved. Think about a hard disk failure on any given sim (I’m sure it happens a bunch). You’d have to recover all of those assets somehow, or store them elsewhere… say on a centralized asset server…

Also, clients would have to be opening hundreds or possibly thousands of connections to various sims. Sims would have hundreds or thousands of clients connecting to them. Kernel network tables could fill.

Instead, if the asset system is a cluster, you know where all of the data is and can make it redundant with all of the standard redundancy techniques like RAID and such. You can partition the UUID space up and assign various chunks of assets to various nodes (I bet they do this in some form or another). You can insert an arbitrary number of layers of caching in between requests from sims and the actual asset nodes, as needed. You can add more asset nodes as needed. All in all, I think the existing system is a good solution.

LL could in theory extend that for inventories of avatars who make that sim their ‘home’ point (i.e. if I still had land on Boardman a localized cache copy of inventory could reside there).

Gwyn – I’ve said for a very long time now that LL should consider licensing the code base out to companies that would like to be part of SL but not part of the centrally-managed grid. I still think this has merit since companies that take a bite at this would have the impetus (and apparently time) to update that code to either fix/complete features or make maintenance easier. I’d like to see some comment by more people on this, but in your case it may be easier to just start a new post.

–TSK

The article you linked to on Wikipedia on UUID’s took a bit of wading through before I found that link to RFC4122 which is probably where you were trying to send me to in the first place.

There are a number of different places that LL can “squeeze” in extra information such as the sim that generated the UUID and whether they are dealing with an “old” UUID or a “new” one. Many of the different UUID formulas incorporate the MAC address identifying the computer that created it (and it’s possible that the UUID’s in the system already have a “machine assignment” then, albiet one that doesn’t move with the sim). I have a number of virtual VPN adapters and as they need to have MAC addresses of their own they came up with “00:FF” followed by what may be a random number (or opaque baton, who knows)

My reference to MS is that they seem to be doing their own thing (as usual) and have come up with a completely different (and probably undocumented) way of building UUID’s (described in section 4.1.1 of that RFC). I think that one point that I was trying to make is that things like UUID’s should only be understood by the machine that created them, everyone else should just look at them as if they were these long strings of digits that have no inherent meaning other than to identify an object.

I do agree that if LL does change the mechanism that UUID’s are generated with, that they are going to have a major legacy issue, which will grow the longer they wait before doing something like this.

However, i think that SL’s major problem is not so much technical as user-psychological: LL have failed to make the transistion from delivering cool toys to to a bunch of techie pathfinders, to the situation today where a lot of users are here because it’s “just another cool game”; and a significant minority are in SL because they want to be part of a serious, money-earning future.

Perhaps LL was more than a tad unwise to go for the rapid expansion phase at the time they did, but that’s water under the bridge, and they have to learn to live with their self-imposed new order!

In this current, quasi-mature phase of operations, *every* change is going to annoy someone. New bugs will annoy huge numbers — as they do! Every proposed change to the grid needs to be examined very carefully, from a customer-management point of view as well as from a technical standpoint. I suspect there is a good case for switching to a “no new features at all” mode, and just fix the bugs, in order of seriousness. This has the attraction that more of the team could be released to work on a major re-vamp.

If it was up to me (and if I wasn’t up to my neck in rolling out cool fibre broadband to a reluctant British public, I’d be applying for the job , I’d make available a rapidly changing (but nominally releasable) public alpha grid of SL2 as soon as possible, and pay people L$ on the main grid for spending time in the alpha grid (testing the new version would certainly beat camping for money on the main grid!)

Let me make a prediction; I suspect that the upgrade on Wednesday will be met with howls of anguish and dissatisfaction; that complains will go UP, not down, even if the bugs fixed do improve the “technical” situation. In my opinion, LL would be much better advised to stick to addressing bugs they can fix safely with rolling upgrades, until they have pushed that approach as far as they can. When the grid is a lot more stable, users will be a little more tolerant to ups and downs from Wednesday upgrades.

It’s all about the customer, LL!

(1) The “butterfly effect” is alive and well in SL, and

(2) We should just be happy the damn thing works in the first place!

Niko