Living on the bleeding edge
For the past two years I have been always patiently explaining to everybody who complaind about LL and SL and threatened to move over to OpenSim that OpenSim is not really “production ready” yet. What this mostly means is that you’ll be accessing a SL-compatible grid that is in its pre-alpha stages — pretty much at the stage where LL started to do their own first closed betas, in 2002 or so. That’s not for the faint of heart.
Most people have contact with OpenSim by logging in to one of many OpenSim-based grids. Although many of those residents do have a technical background and understand what’s going on, most have just “abandoned” SL in disgust and are looking for an alternative that looks as similar to SL as possible and is familiar enough to them.
They get quickly disappointed.
First, they have read extraordinary claims about OpenSim, the first being that “everything is free” (including land, of course), and that attracts them quickly. What is often not mentioned is that everything has a cost, and that you pay for what you get. Running a virtual world takes computer resources, but most importantly, it takes a team of system and network administrators to tweak the servers to make them run smoothly. It also requires a lot of bandwidth. The first OpenSim grids just ran from everybody’s home — enough for you to build and have some fun and invite a handful of friends, but not more. And, of course, when your Windows PC crashed, you’d dump all your friends out of your sim, and probably lose all content that way, with corrupted files on your disk.
Secondly, people confuse “many OpenSim grids” with particular grids. “I’ve moved to OpenSim”, they say, but what they have done is a registration with one OpenSim grid, probably one recommended by a friend or on a blog they read. And their judgement of OpenSim will often come from the experience they had with one grid — you can get a very thorough (and critical!) review of OpenLife done by Prokofy Neva (he actually additionally tested several different OpenSim-based grids; you can get the many reviews from his blog).
The issue here is that it’s “not obvious” for a new resident of an OpenSim grid to understand what’s all about. All they know is that they can connect to “a grid not run by LL” by adding a parameter on their SL client, and that’s all. They understand they’ll “start with a new avatar” — that’s all right, they’re used to start with alts from scratch, so this is nothing new. What they don’t understand is that LL’s infrastructure and operations team are really good at what they do and that SL’s software is production-grade, not “a very early alpha”.
They’ll be very happy to learn that whole sims with 45,000 prims will cost them as little as, say, US$50/month on a OpenSim-based grid. But what they don’t understand is that LL doesn’t charge what they do because LL is big, mean, and greedy — but because providing high-end servers and running them in a very complex environment is very expensive. I try patiently to explain that LL’s prices are not that high (although the setup fees are — more on that in a bit) for the service they actually provide.
Let’s do some maths. People usually shop around on the net for server hosting services and get quite cheap prices there. My company’s own mini-grid of eight sims runs from two low-end servers, one of them hosted by aplus.net, which just charges US$/70 or so for a low-end, single-core server. That’s cheap. If I take into account that I run 4 sims on that server, plus all UGAIM servers, and even a few monitoring tools, well that means that each sim costs about US$16 or so, if I’d be willing to sell land on an OpenSim-based grid.
But that’s obviously not the whole story! aplus.net is actually pretty reliable, but if you go to the top tier providers, like Rackspace or Verio, you’ll pay for the same service perhaps US$300/month. Why the huge price difference?
Well, reliability mostly. Rackspace or Verio give you guarantees on their performance. They have multiple redundant high-speed links to the Internet. They have the best engineers, system admins and network admins in the world looking over your servers. They almost give you 100% uptime, or your money back. These are the kinds of providers you use if you want to be absolutely sure that your server never goes down. Aplus.net is reliable enough for a small company with a specific project and purpose in mind — but I certainly wouldn’t be running “hundreds of avatars” and selling land services on OpenSim from a server that only costs US$70/month!
Aplus.net obviously also has far better hardware for higher prices. Something close to what LL uses costs above US$ 300/month, and that’s the kind of hardware that you’d be using for hosting just four sims, if you do the same as LL (ie. on sim per CPU). It’s cheap — if you don’t wish to provide rock-hard reliability to your customers. The same server at Verio or Rackspace, of course, will cost you four times as much — or pretty much what LL actually charges!
But there is a difference. LL is a solid company with ten years of existence and 300 employees. The ops team has been running the SL grid for years and years and deals with 16 million registered users every day. Granted, they’re not perfect, but they are used to provide 3D content hosting.
John Doe who runs his own OpenSim grid from his home, with a handful of friends, and low-end hosting providers, and perhaps a few months of experience in tweaking OpenSim… is a completely different story. So, to recap: you get what you pay for. For US$70/month for an OpenSim, well, you get a service that is far below what LL can provide.
(As a side note, setup charges by LL are terribly inflated, since they used to charge the time it took for a member of the ops team to install the software on a server and give it to you, which took a few days — or several weeks. These days, the whole process is totally automatic, so there is no point in charging any setup fees. I believe that in 2009 LL will probably drop those. By contrast, most OpenSim grid operators still do that work manually. It’s not hard but it’s not exactly “instantaneous” either, specially if the OpenSim grid operator has to lease more servers in a hurry!)
Also remember that most — not all — of those OpenSim grids out there are ran by tiny start-ups or even individuals with little or no business experience. There are obviously a few exceptions (notably, 3Di), and more and more will appear over the next few years, but you have to take into account that most of them just saw this as an opportunity to get some extra income and have “dream jobs” elsewhere (you know, the type where you work from 9-to-5 with an adequate salary and don’t have pretty much anything to do during the whole time 🙂 – giving you plenty of opportunity to run your own 3D content hosting business from your workplace that way). The good news is that many of those independent 3D content hosting providers have had previous experience running their own 2D (i.e. Web) hosting provider businesses in the past, so at least they’re definitely not clueless. But I imagine that most will never present their credentials if asked.
Things will certainly change in the next few years, but the important thing to remember is: OpenSim performance depends a lot on who’s managing the servers. While it’s pretty easy to download the software and run it on your own computer for some fun with your friends, providing a 3D content hosting business professionally is something entirely different.
No matter how good your hardware and networking infrastructure is, there is the other side of the coin: OpenSim software is at the early alpha stage, even though these days it has more features than LL’s own server software. What this means is that development is oriented more to adding new features — matching the feature set as closely as possible to LL’s own — and less on “fixing bugs”. That’s the only way they have managed to reach almost the stage where they’re quite close to LL’s own product.
For instance, groups don’t work (yet), although you can parcel out plots and set them for sale (buying requires a special viewer, since SL’s viewer will always try to retrieve your SL account’s data from LL’s user servers). So this means that group ownership will not work, which leaves out creating a rental business that looks like the ones in SL. Similarly, you cannot set an object to group, allowing people in the same group to collaborate in building it with you. You either give it full permissions or no permissions at all. This is good enough for selling items on OpenSim-based grids (interestingly enough, there is an “economy server plug-in”, where you can track down people’s L$ using an external server to log transactions, give people access to their balance from a website, etc. just like in SL), but it’s not easy to, say, buy textures from someone and use them on your own content to be sold later.
This is the sort of thing that will annoy residents most: finding out that most of the things actually work, but not all. A relatively recent version of OpenSim did not allow directly creating clothing items on Appearance mode, or creating new scripts inside a prim, or edit notecards inside a prim. You had to create the clothing item (or body part) or the script/notecard in your Inventory first, and drag it to your avatar (or a prim) next. Then it worked! (even more recent OpenSim versions allegedly allow the same behaviour as SL). Attaching items to your avatar (as well as HUDs), even scripted ones, works quite well. But until recently, to save the position, you had to adjust the item and not forget to detach it, and re-attach it again to double-check. If you just logged out, OpenSim would not save the last position. Also, high-prim attachments tended to be very hard to correctly position (again, this has improved a lot in the latest versions).
Depending on the server and the bandwidth used, you’ll find terraforming and building an immensely pleasing experience (ie. terraforming might work tens of times faster than in SL, because it uses a different way to store the terraforming data, which apparently is far more efficient than LL’s own) or an absolute nightmare (if your server is underpowered and has little bandwidth, the reverse would be true: laggy terraforming, which is almost impossible to do correctly, and you’d have to rely to an upload of a RAW file with the terrain). With very large buildings that are totally unlinked, you’ll very likely see some prims missing now and then — not unlike building in SL in 2004/5, when the same problem existed. OpenSim’s latest version has a more efficient way to push data about a linked object to your viewer, so the trick is to properly link everything.
Since you can use your own physics engine (out of a choice of several) and different mesh calculations for checking volume boundaries (important for the client to know if your avatar is “bumping” into walls or other objects), depending on the combination, you might get almost the same “movement feeling” as on LL’s grid (namely, if you go with ODE and full meshes). But on a slow server you might use a different combination (simple physics, no mesh calculations but just boundary cubes), and things will feel quite different — and you might not even be able to pass under arches or through doors made as holes in a prim. Again, this will depend on what performance your server has, and what options you’ve selected.
Vehicles don’t work yet — Cory Linden had implemented “vehicle engines” on Second Life Grid which are quite complex — but they will work, sooner or later. The rest pretty much works as advertised, although you can expect some scripting limitations, or some things to be slightly differently implemented and not give the exact results, specially on complex scripts. The good thing is that you can move with scripted objects across sims, just like in SL, and with a small setting, they’ll even cross sim boundaries and keep running. Since people can compile scripts to do malicious things (specially when using the “os” function set that can ‘talk’ directly to the sim and even to the server beneath it…), OpenSim admins can disable that feature for security reasons (you’ll actually see a lot of weird options to deal with security — thus dispelling the myth that OpenSim developers never worry much about security. It’s not true. Some OpenSim grids were already victims of griefer attacks, and have now far more powerful tools to deal with that than LL has on the Estate Tools).
The bad news is that sim crossing is way harder than LL’s (see again the start of the video above… running around sim corners with high-prim attachments is definitely a bad idea), and, again, feels like SL in 2003 or 2004, when LL had far slower servers and little bandwidth.
There is an incredible exception, where OpenSim outshines SL in performance. If you’re in SL and let everything painfully rez, have downloaded all your items in the inventory, and stand still, doing nothing, in an area with a high density of textures, you’ll see from the Statistics Bar that you’re always consuming some bandwidth. Depending on SL’s version and the time of the day tis can go as low as 10-12 Kbps, but is usually at 30-40 Kbps, sometimes a bit more. This is intriguing because technically “SL is doing nothing”.
By contrast, OpenSim takes up so little bandwidth that it feels strange. Yes, if you’ve loaded everything, the bandwidth will go to zero, as expected. Even when turning the camera around. If you move on a fully loaded sim, you’ll see 1 or 2 Kbps of bandwidth (because some positioning data about your avatar has to be sent and acknowledged), but that’s all.
So why does SL always consume a bit of bandwidth even when doing nothing?
There is some speculation about it. First, remember that the Second Life Protocol was reverse engineered (even if with some help from LL’s employees). Some messages were not understood, and dropped from the implementation. So it seems that LL’s simulator software are a bit more “chatty” with messages that we don’t really know what they’re for. Tateru Nino and others believe that every second or so your SL client is checking all those calling cards on your inventory to see, for each one of them, if your friends are online, looping over and over again. This, of course, is an insane amount of resources being consumed (OpenSim definitely does not work that way!) — both CPU (to do constant polling) and bandwidth (to constantly get presence data from your friends). Knowing LL’s obsession about doing Instant Messaging “their own way” and reinventing the wheel, this somehow rings true, even if it’s one of the worst possible ways of implementing things… even if it tends to guarantee that you have always an updated list of online friends.
I also think that the other reason has to do with metrics. LL’s viewer is constantly sending out metrics and information about the status of your SL client. That’s how LL can look at the statistics and claim that, overall, the SL client is improving FPS for all computers and crashing far less. Remember that the reports by angry people complaining about the lack of performance are just a vocal minority — granted, a large minority, and a very vocal one, but a minority nevertheless. LL needs to have access to the real number of people with problems, or without them, which will never comment on how good (or bad) SL runs for them.
Or perhaps there are still legacy communications going on, that are implemented differently, and these have been dropped by the OpenSim team. The end result is that OpenSim consumes orders of magnitude less bandwidth than SL. And, as a side bonus, not having to decode all those messages will free up your CPU to push more data to the graphics card and worry about the viewer and the UI, and less about the communication. In my case, with the same settings, I get twice the FPS, on sims with an equivalent amount of prims and textures. That, of course, for servers that are 16 times slower than LL’s own (my guesstimate), which means that often there is not enough CPU/bandwidth to keep those pipes flowing, and my OpenSim regions need to retransmit a lot of data (which always causes some lag; I can spend up to a minute “floating in the air” when jumping to a new region, since my poor overloaded servers need to push all that data back to my SL client).
The experience will always be subjective, and since there are so many differences in hardware, bandwidth, and geographical location, I’m quite sure that everybody will feel OpenSim to be different than LL’s grid. For our internal testing, we used two servers with 4 regions each. One is physically co-located between LL’s Dallas and San Francisco grids, it is low-performance with just 512 MB of RAM, but unlimited bandwidth. The other server is in Portugal (which means ultra-low latency for me, although ping times will always surprise Americans connecting to it), has 2 GB of RAM, but quite limited bandwidth (just 1o Mbps). There are a lot of differences between them! OpenSim, as a software, will eat up as much RAM as it can, but will not be a dramatic hit in terms of CPU. However, bandwidth is required in an infinite supply, much more than I actually expected. I tend to tell people to forget to run OpenSim from your homes, since the best types of connections will give you 1 or at most 2 Mbps upstreaming bandwidth. That’s far less than you need — you can forget anything less than 10 Mbps to have at least a few avatars happily around. Unlimited bandwidth is, however, the way to go. If you’re operating your servers on a budget, aim for more cores or processors, as much RAM you can afford to buy, get unlimited bandwidth, and worry less about high clock rates for the CPUs.
