OpenSimulator 0.6.3 – The Stability Era?

Granted, I know I’m biased 🙂 I like bleeding-edge technology, specially when it works; and I’m not trying out OpenSim for “political purposes” of any sort. As I keep repeating to friends, colleagues, and clients — OpenSim is cool, it’s even useful, but forget about “a cheap alternative to Second Life”. All what you save on not paying tier to Linden Lab goes into improving the hardware and bandwidth of your own servers, and, mostly, in spending long hours in tweaking the configuration and making sure your grid is running at all.

No, I’ll avoid the age-long discussion about “open source software reducing dramatically the costs of everything”. You always trade-off costs: in this case, the cost of maintenance and hosting, for the cost of outsourcing it to Linden Lab to do it on their own grid. For some people, this looks like it’s free, since your own time to play with it as a hobby is unpaid. But for serious, professional use, someone has to get paid to maintain your OpenSim grid, and believe me, it costs way more than simply renting a private island from Linden Lab 🙂

The pitiful centuries-old Pentium IV with merely 512 MB of RAM that used to host Beta Technologies‘s OpenSim grid had been allocated to a different project for about two months. That meant two months of patiently delaying all my current work on OpenSim (mostly related to my mastership degree and a few customer’s experiments). By Friday, at last, I was allowed to push all the thingies from that other project away, clean up the mess on the old faithful server, and take a look at OpenSim again.

I’ve missed most of OpenSim’s recent fun. Version 0.6 was finally launched, but that was months ago (for your information, version numbers have some logic: 0.6 means that 60% of all LSL functionality is implemented); when 0.6 was launched, we had a real project for a real client using OpenSim, and thus it was not appropriate to do a grid upgrade during that time (how wonderful it would be if you could do the same on LL’s grid!…). But now it would certainly be a good excuse to bring OpenSim back to the latest version!

I’ve tried to dig around the newest features on OpenSim. My usual source is Justin’s blog, but as you can see, every week there is something new in OpenSim — bugs fixed, features added, every week after week after week after week. I saw that OS Grid got a working economy (or at least I think so; why is it ever so hard to google for information about OpenSim??) and missed all the excitement about that. But I wasn’t aware of any earth-shattering announcements. People on the OpenSim source forge have been hammering away at the code… in silence.

Well, I thought I’d go for some dramatic changes. Upgrading an OpenSim Grid, even a two-server one, is a very painful experience. Since the developers have pretty much thrown every kind of file on a single directory (./bin) it means that newer versions will override older ones. Although things are getting better (a lot of config files have become obsolete), there is still the master config file to deal with: if you copy the old one over the new one, you’ll lose sight of the cool new features and the ability to activate them… so for me that means opening the old config file and the new one, and looking at each setting, one by one, to see which one has changed.

The next issue that almost everybody will think about it… what about my database? Will it be still compatible? OpenSim, like many modern software that deals with constant tweaking, tries to patiently upgrade older databases to the most recent scheme — automatically. So far, I have been lucky — not a single prim was lost! They warn impatient OpenSim users that they should do regular upgrades, since upgrading from way old installations might fail. Alas, if you’ve got all the time in the world, don’t have a job, and do nothing else but install OpenSim and play with it, you can afford to do that. The rest of the world has to cross their fingers and hope for the best — upgrading, say, 3-4 times a year, if you can spare the several hours for upgrading.

This time, however, I was bolder 🙂 The totally underpowered server I’ve been using also ran the five central servers (User, Grid, Assets, Inventory, and Messaging) and all used a local install of MySQL. This is always a trade-off that you will, at some point, have to deal with: what is best, even for a small grid? Putting it all on the same server, where connections will all be in memory and ultra-fast, or spread the load around different servers, where communications become a bottleneck? (What if the server is so slow that it times out on the connections with the database server?)

Since I had thoroughly tested it with the “everything on the same server” approach, I tried this time the reverse approach. I’m a customer of Dreamhost (where my blog is hosted), and they have a separate cluster of MySQL database servers — for thousands of their clients, many of which are always abusing the system with badly-formed queries. So this would be the real challenge: would moving the database out of my underpowered server into a overcrowded, dedicated MySQL server on a different data centre impact performance in any way? And would it be “better” or “worse”?

Sadly, system administration is more an art than a science. I reasonably assumed that getting rid of MySQL out of the underpowered server would have two advantages: less memory consumption (and when you only have 512 MB of RAM, that is a serious consideration!) and less disk accesses (remember, if you exhaust your RAM, your poor system will use the hard disk instead — which is a thousand times slower). It would add two new problems: Internet connectivity (who knows how bad it can get) and a shared environment with thousands of unruly, abusive users.

I was bold enough to give it a try, though, since there are two worst-case scenarios: 1) Dreamhost’s MySQL is way too bad; 2) Dreamhost’s MySQL is good enough, but they block my account due to very intensive use (they do that sometimes). In both cases it will mean migrating back to the old, underpowered server.

My the first impression was not a very good one. Lots and lots of “sleeping” threads on MySQL. Oops! I hoped that Dreamhost’s system administrators were looking the other way, they hate when that happens. Apparently, OpenSim tries to keep as many threads open on MySQL as possible, using them “on demand” and letting them sleep while they’re unused. It’s a good compromise — wasting some memory for “always available” connections — but it naturally leads to possible abuse. Still, it doesn’t seem to open more than 30 or so threads, and MySQL will close them down after 15 minutes or so anyway, with little impact — so I guess I’d be safe.

And now, it’s time to experiment! I teleported over to the 8000-prim nightmare of one of my previous projects (you can take a look here on how long that used to take to rezz). And… whoops, everything was pre-rezzed already?! I was flabbergasted, but then I remembered that the current versions of the Second Life client are way better at managing a local cache. So probably I still had all those objects from my last login… two months ago.

Imprudence to the rescue! Cleaning up the cache, making sure it’s not on the same directory as the main SL viewer on my Mac, and I was ready to try it out, this time, making sure that I was really downloading things from my OpenSim grid.

Hmpf. As you can see on the video below, there is no difference at all 🙁

So what’s going on here??

Enter the new, improved OpenSim. The hugest difference from what I’ve seen in previous ones is a new caching model (which you can fine-tune from the configuration file). Now OpenSim tries to think and behave a little bit more like Second Life’s own simulators. It looks at for how long a linkset has been around without being tweaked: if several minutes have passed without a change, it becomes “persistent”, and it’s moved to a special, local cache (no need to fetch it again). If it’s been days since you have last changed a linkset, it even goes to a “more persistent” cache. Objects on those don’t require much validation — e.g. checking on the database if they’re moving around, or their settings being changed, and so on — so they’re “always ready” to be served. The result is, of course, astonishingly good — and blindingly fast!

I’ve walked a bit around the same scenario that you saw on the previous movie. The difference here is that in just a few minutes, 8000 prims — with textures — had been downloaded and stored on a cache-cleaned SL client (you can get a stopwatch and watch the movie to see how quick that was). To give you an idea, sometimes I had to wait an hour on previous OpenSim versions to get all those 8000 prims, and I had to do several “dirty tricks” like forcing a refresh (server-side). Sometimes, a few prims would never appear (the builder had often redone the very same walls, and the flicker you see is where she basically put one wall exactly on top of another, simply because she couldn’t see if that wall had been rezzed or not). Granted, once everything was safely stored on the client’s cache, things would work well.

I tp’ed over a colleague of mine, half an hour or so after this video was shot. He rezzed on the neighbouring sim, and I gave him some inventory objects, and we chatted a bit. Then I told him that the only thing worth seeing was at the other side of a mountain we have there. As he walked the mountain up and down again — everything was perfectly rezzed. A chat, an inventory offer, walking up and down a mountain. That’s enough to download 8,000 prims and textures!

So, yes, that was a huge improvement. Congratulations to the OpenSim developers.

But there is more!…

Now, you can change scripts on-the-fly inside objects. Previous versions were a bit tricky: you needed to create them in your avatar’s inventory first and drag them to your object. A minor inconvenience, but it was the OS-do, or The Way of OpenSim 🙂 This version allows it to work just like in SL — and OpenSim correctly detects when a script was compiled under one engine and warns you if you’re running it on a different one, recompiling it on the run. And it’s also way faster, even with debug mode on, which they warn it’ll be “a thousand times slower”. (Oh, btw, you can also create clothes from Appearance Mode. It’s not yet exactly how SL works — the item appears immediately on your avatar’s inventory, but you don’t wear it automatically — but… it’s getting there).

And sim border crossings are now almost as flawless as on Second Life. This actually quite impressed me, since it’s one of the most hardest things to do. Waiting a minute to walk across a sim border was something not unusual, and, taking into account that a contiguous landscape across simulators is the hallmark of Second Life, replicating it on OpenSim was something I was not hoping to see to be done so well and so soon. Indeed, most people will now be able to walk/fly across region borders in OpenSim and not notice any difference from the same experience in Second Life. It’s almost the same imperceptible delay!

I’m pretty sure that there are a lot of under-the-hood developments. I wish I had more time to explore, for instance, how everything ties into the economy server. I was amazed at looking at the “graffiti” board: an OpenSim-only implementation that allows text to be displayed on a texture’s prim, which is way better than using hovertext. It’s not a new feature of OpenSim, it’s been part of it for a while, but, alas, something I haven’t been able to play with it. Oh, and there are a lot of things, like the ability to ask the server for a lot of information from a web page, without using LSL or a bot. In fact, I believe that pretty much every so-called “ethical” use of ‘bots in SL can be done in OpenSim in a different and much easier way: getting land data, parcel info, number of prims, metrics and statistics, and so on — all these have special requests to ask for data which can easily be retrieved via simple HTML calls.

Hypergrid support has also improved. Now, if you know about a Hypergrid-enabled OpenSim grid, you can just log in to your grid, and follow a simple secondlife:// link — and you should be on your way. It couldn’t get easier than that. It’s amazing how simple Hypergrid has become, when you compare it to the insane amount of time spent on the standardisation effort at MMOX.

Oh, and as always, you’ll be drooling at how little bandwidth OpenSim continues to use. I was expecting that, as the SL viewer complexifies, and as OpenSim gets more and more features, your bandwidth consumption would increase, since more messages between the client and the server are transmitted (for the extra features). Not so: when everything has been downloaded, and is safely stored in your cache, and you’re not moving at all, but just looking around, your bandwidth consumption will be… zero. As it should be!

There are no groups yet, and you still cannot assign parcels to groups, of course. A lot has to be done on that side to deal with groups properly. But permissions work! Your content created on OpenSim remains yours, unless you allow it to be otherwise (ie. turn your content into a freebie). In fact, since you cannot properly share items across groups, it’s even harder to accidentally “share” an object. Of course, you have to trust the grid operators with your content…

Anyway… there is a whole universe to explore — as always, with every new OpenSim release. 🙂 In the mean time, I leave you with the “major features”, knowing full well that you will find here and there a few more…

CC BY 4.0 OpenSimulator 0.6.3 – The Stability Era? by Gwyneth Llewelyn is licensed under a Creative Commons Attribution 4.0 International License.

About Gwyneth Llewelyn

I'm just a virtual girl in a virtual world...

  • I should add, for the sake of completeness, that if you’re serious about doing some real work on OpenSim, you should not use the kind of specifications I use 🙂 I’d definitely recommend emulating Linden Lab’s own Class 5 servers at least: quad-core CPUs with at least 2 GB of RAM can get inexpensively leased for a few hundreds of dollars per month, and you should definitely get your central servers and the database on separate servers — inside the same provider to keep bandwidth bottlenecks down to an absolute minimum. I can imagine that if I could afford it, I’d have at least 6 servers, all interconnected on a local Gigabit Ethernet switch: a MySQL cluster of two machines; one for the UGAIM servers; two quad-core latest-generation servers with a minimum of 2 GB (but I’d go for 8-16 if I had more users) to run, say, 20-30 sims; and a separate server for tests, maintenance, a support website (where you could do things like remote administration — which is supported by OpenSim, btw), the economy server, an IRC server for people who can’t log in at the moment, and gathering metrics and statistics. That will set you back a thousand dollars or so per month — plus, of course, the cost of your own work to keep the minigrid up.

  • Gwyn, now this a new way to avoid multiple-page post: being the first to comment on your own post!

  • Hey Gwyn. Nice to know that my weekly development roundup is useful to you 🙂 It can be a bit of a labour of love – OpenSim is already so diverse that it’s really quite difficult even for a developer to keep a handle on everything that’s going on in the core project, let alone in the forge.

    I’m also glad to see that the OpenSim experience has become a lot better for you. As you’ve seen, better region object caching (by core developer idb) has played a good part in that. However, it’s also been improved by a host of other factors, such as a better implementation of the server side of the Second Life protocol and some excellent work by Teravus Ousley to implement progressive texture sending (only just relative recently we were always sending full texture data for every quality request which was a big unnecessary load on the system).

    Moreover, our region crossing stuff has been completely overhauled by Cristina Lopes, a very talented core developer (as well as an academic!), who, as you know, is also the brains behind the Hypergrid architecture. And as usual Melanie Milland has been making big improvements to the script engine, she has been the real talent behind that component for a long time now.

    Of course, lots of other fantastic improvements have been made by other core developers too. Many thanks for the congratulations.

    All that’s not to say that OpenSim is anywhere near perfect yet – right now, for instance, we seem to have a bad memory leak and a superb ability to provoke Mono to crash (though that may be more due to mono bugs than our bugs).

    But I did find the stability title of your piece interesting. There are a few glaring absences in terms of Second Life environment features for OpenSim (groups, as you can see, and little things such as the ability to take a collection of unlinked objects as a single item into inventory). But enough has been done now that I would speculate that we’ll see further OpenSim growth start to spread out into at least five other areas.

    1) Support for other protocols and environments. For instance, Bruce Joy announced that Vastpark have started to experiment with plugging the VastPark environment into OpenSim.

    http://www.vastpark.com/comments/vastpark_opensim_integration

    Realxtend are also continuing a project to formulate their environment as modules which can sit on top of OpenSim.

    2) Core stability. I think this is actually a huge challenge. To me, throwing up features in a big system such as OpenSim is relatively easy compared with the work of fixing memory leaks and other stability bugs. Moreover, as we mature I think that it becomes clearer that some of our core mechanisms need an overhaul in order to provide a stable platform for growth.

    3) External interface stability. We have a region module mechanism, but at the moment I would say that it is something of a black art to write any truly complex module. The methods required to manipulate OpenSim are not very well documented and in places confusing. To a large extent this is the result of simply exposing internal methods to module writers. This will need to be improved if we are to truly offer a decent modular platform. The good thing is that work is already underway to improve the situation (by people such as Homer Horowitz)

    4) Performance. Up till now there hasn’t been a major focus on performance, so there are lots of areas where we can improve. Again, the good news is that people are starting to look at this area. In particular, Dan Lake from Intel has already contributed some very valuable analysis and patches for the juicy low hanging performance fruit.

    5) Experimentation. As an open platform, OpenSim provides a wonderful opportunity for experimentation. As you know, it allows projects such as the Hypergrid to develop very quickly. John Hurliman from Intel (and of libomv/libsecondlife fame) has also contributed the ‘Cable Beach’ asset/inventory server. This aims to facilitate future architectural experiments where a viewer can fetch data directly rather than streaming everything through the region server.

  • Great review. My experience is very similar. After the Openspace fiasco, I tried Open Life during some months and it was a nightmare — I’ve seen OSGrid underperforming severely too, so I decided to start my own small grid, cloning my SL islands. I’m running 8 islands in a Quad-code Q6600 at 2.40 GHz with 3 GB RAM and Windows XP SP3; logging in with the cache cleared into the main island, which has 7500 prims, I get fully rezzed in 25 seconds; prims from adjacent islands take a little more to rezz, but at 50 seconds all is there. In small grids, Opensim is wonderfully stable now. Hypergrid is fantastic, and with the new Grider proxy Cristina Lopes is writing it will also be soon absolutely safe. And if you add to that Justin’s Parallel Selves Message Bridge, you practically can’t tell whether you are in SL or in Opensim! 🙂 Indeed, apart from some occasional problems in the rendering of other avies (i.e., they appear as clouds, or never fully rezz), there’s no difference between SL and Opensim for most uses right now. The only two things I’m really lacking is a better script engine (it’s impossible to work with many timers without overflowing the event queue, for example), and better quality anims. In any case the Opensim team are doing a fantastic job 🙂

  • @Zonja – just out of interest, are you using the XEngine scripting engine rather than DotNetEngine (DNE)? I know we used to see the event issue on DNE but I’m not sure if it’s so prevalent on XEngine (which is a newer implementation).

  • I’m on DNE, will try XEngine asap and report back, many thanks for the tip! 🙂

  • @Justin: Finally found some time to test XEngine, and realized that I had already tried it and switched back to DNE, because as soon as I start using XEngine, all my attachment scripts (i.e., texture anims for jewellery, AOs, etc) stop working 🙁 Back to DNE for the moment…

  • @Zonja – Oh how odd. It might be worth filing a bug report though to see if that can be fixed.

  • “But for serious, professional use, someone has to get paid to maintain your OpenSim grid, and believe me, it costs way more than simply renting a private island from Linden Lab :)”

    No, for small developers and 3d artists who are long on time and short on cash, things like Opensim are the right sort of “price”.

    Actually I am running it on my local network, complete with UGAIM also on my local network, and its pretty fab. Yes there are bugs but I am happy to test and contribute to it. Is there an alternative for the small time content creator who just needs an offline “SL” for SL work with true privacy for the price? No, there’s not. Most don’t make enough on SL sales to buy a whole additional sim in SL for development work. It has allowed me to tier down to just the amount of land I need in SL and no more, and remain in a community area where it is far more pleasurable to be in SL. It lets me really work on items without worrying about upload fees, and the fact that I control my UGAIM servers gives me the security I need about my content in development phases.

    If I need an SL sim for public access at some point, I don’t have to buy it until I am ready.

    The fact that a developer’s content is truly local on the hard drive as a backup is a major plus.

  • @Justin: Just for the record, the XEngine problem is gone in 0.6.4.9561! 🙂 Now I can begin to try all these color changing scripts again 🙂