The Battle for the Desktop Begins!

kirstens-viewerIn late 2007, Linden Lab had published their viewer roadmap for 2008, a way for us residents to take a look at what they were planning to release in future versions of the official Second Life® client. It all looks very promising, when you suddenly realise that this page has not been updated since August 2008; other Wiki pages linked from there are even older.

So, what happened? The drive to “increase stability” has been mostly understood internally to mean “more quality assurance testing” and also “collecting more statistics”, or so at least it seems. Put it bluntly, this means that every time a single line of code is added to the official SL viewer, it routinely goes through a series of exhaustive testing to make sure that line of code doesn’t “interfere” with the rest and pop up new hitherto unknown bugs into existing. LL has devised a complex and very lengthy procedure to run a series of tests for that.

The result? From the moment a bug is found and a patch is submitted by a developer to the Public JIRA can take days — sometimes even just a few hours, which happens when a volunteer developer (non-LL employee) suddenly gets inspired and patches the code. But it then takes months until that patch is approved. Typically it takes around 6-14 months until a bug gets fixed, although exceptions exist at both extremes — security issues can be patched much quicker (sometimes in hours!), and non-essential bugs or some new features can take 18 months… or, in some cases, like introducing Havok 4, it took four years.

M Linden is still keen in providing residents in 2009 with a new, light version of Second Life that will enhance the new user experience. The question is, is Linden Lab really up to it?


It’s tough to be a SL client developer at Linden Lab. I feel their frustration.

Until somewhen in 2005, developing the SL client was pretty much “hack and slash”. Someone would have a great idea, it would be immediately coded and developed into a new feature, and released to the public. Having just a few thousands of residents, these would try it out, complain, LL would get it fixed, then the residents would complain again, a new fix would be produced, and after two weeks of nightmares, LL would eventually get it right, and they’d start on the next set of features. At some point they started to do the tests on the Preview grid (then called “Beta grid” for testing out beta versions of the SL client).

In those days, both the SL client and the simulation software on the servers had to be in sync, so if something went wrong with the client (not enough testing!), the servers had to be rolled back as well. This became incredibly more messy as new residents — now millions and not thousands — started to use Second Life. The vast majority of them have no clue on how to configure their computers properly, they have widely differing hardware, and also different expectations of how SL should work. So at some stage LL’s quick feature development had to stop.

There was a huge gap in 2006 or so when no further features were introduced, and the focus was on stability. I believe that it was back then when LL started to introduce the concept of quality assurance testing. In the very early days, this consisted of a sequence of “test sheets” that were on their Wiki, and volunteer residents would go with the Lindens on the Preview grid and check each item off and report back. It was slow and painful work. The problem is, due to the nature of LL’s codebase, you never know if something as harmless as correcting a spelling mistake on a dialogue box won’t suddenly introduce a bug in texture caching. So this means that for every single line of code you had to do all the tests — again.

In 2007, a bit of innovation entered Second Life’s development cycle. The source code was released as open software; libSL was promptly created, and the first user-contributed patches started to appear on JIRA; later on, the first “alternate SL viewers” started to make an appearance. OnRez, from the Electric Sheep Company, was probably the first that had some real impact, thanks to its use on the CSI:NY virtual presence in SL. LL had also acquired the company that developed WindLight and this meant, for a time, that residents got a whole new engine to be delighted — SL images and machinimas never looked the same again.

The complexity of dealing with all that change prompted LL to do a lot of things at the same time. First, the server and client code were detached from each other as much as possible, and the Heterogeneous Grid was born — a grid that can run different versions of the simulation software, but also different versions of the SL client. As time has gone by, we’re able to use a wide variety of SL clients to connect to LL’s grid, even some that are hopelessly outdated (but that might somehow work better on a specific resident’s computer for some reason). Linden Lab now has two major ways of releasing new viewers: on the Preview Grid (which has little use) and through the series of Release Candidates (which are used by far more people, but still just “thousands”, not “millions”).

2008 was a return back to “stability” and little innovation — so this goes in cycles. This year, 2009, ought to be “innovation” again, and there were quite a lot of projects accumulating to be deployed. SpeedTree®, to get rid of the Linden Trees and get 21st-century tree rendering into the SL client. Shadows in SL. Flexisculpties. Full meshes. “Puppeteering” or “physical avatars”, even probably with different skeletons/meshes. The group tools, version 2.0, which would also include different sets of permissions (allowing, say, Creative Commons licenses to be tagged to an object). Alternative compilers to Mono (ie. making LSL just one of the supported scripting languages). Client-side plugins. A new lighting system to get good avatar lighting (and get rid of those Facelights, the bane of every machinimist and SL photographer). The list of “ongoing projects” is endless, and some would definitely be introduced in the next “innovation cycle”. We know, however, that LL will skip 2009’s innovation cycle and still continue to focus on “stability” for at least another year.

But… what does “stability” really mean? From a resident’s perspective, it only means longer development cycles, as more and more quality assurance tests are made each time a bug gets fixed. But on the reverse side of the coin, this means more time is spent per bug, and that less bugs are actually fixed. This in fact applies not only to the SL client, but to the simulation software as well: the last release of the sim software (1.24.10) was deployed in December 2007. The current version under deployment, 1.25, has been tested on the main grid in some form since October, but bugs kept it from being the current version and LL promptly rolls back to the previous version. We’ll see if the “final” rollout this week will make it, or if LL will roll it back again, and work on more bugs.

In the mean time, this naturally means that the bugs introduced in December 2007 haven’t been fixed yet.

On the client side, things have been better, but not by much. The new, aggressive statistics capture has been a bane on all LL clients — main client, release candidates, OpenGrid client. For some reason, LL thought that by sending a vast quantity of stats every frame or so will help them to improve their knowledge of what’s going wrong on the resident’s side. Sadly, this also means that the CPU is spending almost as much time processing statistics and sending them to LL than it’s rendering scenes. How is that possible? But it’s a claim very easily sustained: just download one of the alternate clients (almost all disable statistics) and you’ll see twice the performance on the same sim, for no particular reason. Some of the alternate clients don’t even add much besides some patches and disabling statistics. The performance loss we get from the statistics is just incredible.

But if it actually allows LL to develop better viewers… that’s a good thing, right?

About Gwyneth Llewelyn

I'm just a virtual girl in a virtual world...

  • Hi Gwyneth^^

    Actually libsecondlife existed before the Second-Life-Viewer was open-sourced. At least the early versions of it were completely reverse engineered, using their SLproxy.

  • I’m thoroughly unconvinced that Linden Lab’s lengthy QA process is working. Or if it is working (i.e. it’s reducing the number of bugs introduced), it’s a poor trade-off: the effect is disproportionately small to the amount of time they seem to put into it. There are still many bugs being introduced all the time, and even very obvious and visible ones often make it past QA and into the releases.

    As one example of many, consider local ruler mode being oriented incorrectly, a bug which had plagued builders for a long time. I supplied a patch to fix it, but LL decided (after a long delay) to fix it their own way. Their fix, however, broke building in an even worse way: objects would spin wildly out of control when trying to rotate them! Somehow, this extreme and obvious new bug introduced by LL’s fix was not detected by QA.

    LL’s response to the new bug was to comment out the fix for the old bug, which meant that things were now broken again in the old way. This re-introduced bug was also not detected by QA. Or, if it was, LL didn’t bother to try to fix it again — even though I had long ago supplied them with a patch that I had tested thoroughly and which has no apparent side-effects.

    So, I am left to wonder: how can these two very obvious bugs — not to mention all the other bugs thare are constantly introduced — pass through the QA system, when LL supposedly goes to great effort to screen out such bugs? The only conclusion I can come to is that their process is ineffective, and that they are just slowing down progress with no real improvement in quality to show for it.

    This is why I’m sure community-based viewers will always outpace LL’s own viewer. A programmer on a community viewer might introduce just as many bugs as an LL programmer does, but the testing on these viewers (which often simply involves people using the software) detects the bugs just as well or better than LL’s QA process, and the bugs get fixed in a hundredth or thousandth of the time it takes LL.

    ——

    P.S. A correction regarding Imprudence: we are not planning a completely new UI, as you say we are. However, we are planning a plugin engine, which would allow for other applications (plugins) to interface with the viewer. That will enable people to make detached UI widgets, but that’s just one of many things that will be possible.

  • Liberty Tesla

    Gwyneth, there is another possible reason that third-party patches have not been applied: ownership of the code.

    Many open source projects require that any code added to their official codebase be owned by the company or foundation that manages the software (e.g., the Free Software Foundation, MySQL AB, TrollTech). It allows them to update the licensing (which has been a problem for the Apache Foundation, which has no such policy); it also makes liability issues simpler to deal with.

    If the developers of the third party patches can’t or won’t assign ownership to LL (and I’m not saying that they must or should do so), then LL isn’t going to use the code.

  • Ranma Tardis

    Working these viewers is easier said than done. I have been able to log in using hippo viewer but still have not figured out what the address is for second life. This makes me feel like a real DOLT! In open sims have not figured out how to make a skin, someone gave me a shape which I modified to suit. I want to open my own region but need a good server, my pc is loaded down and would not function correctly. I wish for my server to be hosted correctly, etc. Am I stupid? have seemed to have forgotten even the bacis…I am far from being the average user and everything seems so hard!

  • It’s ok to have a comment.

  • Kirstens has left. NWN reports that KirstenLee has quit with releasing clients because of repeated harrassment over GPL issues despite her efforts to try and satisfy those claims.

    http://nwn.blogs.com/nwn/2009/01/kirsten-quits-citing-gpl-enforcementrelated-stress-groundbreaking-open-source-developer-quits-viewer.html