Breath-taking improvements and how to implement them

Linden Lab, since at least mid-2005, has been caught into a tough dilemma. Second Life is a notoriously hard to configure piece of software — too demanding from the computer it runs on, and almost always requiring detailed configuration and fine-tuning of your own computer to be able to enjoy it properly.

When I tell people that it’s unusual for me to crash more than once per month (logging in every day for an average of 4 or more hours), many are very surprised. My computer is old, almost 4 years now, and uses a graphics card that Linden Lab supports only partially (apparently, the original firmware for the ATI Mobility Radeon 9600 does not support some features properly, and LL prefers not to allow to enable them). Still, I have a rather solid application — Second Life is as stable for me as, say, Firefox 2.0.X (which also crashes once per month or so).


But if one reads the Official Linden Lab blog, the horror stories abound. What makes my poor old computer so “special” then? It is slow. Just to open my Mail application, I have often to wait a whole minute. I have not much disk space left, in spite of being continuously deleting old folders (Mail consumes almost all my disk space). The motherboard once melted down due to overheating — caused by running Second Life so often 🙂 It’s definitely not a “top-of-the-line” model, neither a particularly rough model. Like many of the recent-generation Macs (at least the ones built after 2000), it uses medium-to-low quality components. And while Windows users have all sorts of maintenance tools, Mac users, by default, don’t get any, since it is “assumed” that an Unix-based machine doesn’t “require” maintenance at all (which is not exactly true, but it’s a good selling point).

So why do people crash so often? Why do they report so many problems? It’s very hard to say. In many cases, however, the answer is simple: people don’t have the required knowledge to properly tweak and maintain their own computers. It is an art which is only available to a few. How many know which Windows services they should safely shut down to improve performance, or go into RegEdit to change some settings? How many Mac users know what a “sysctl” call is and how it can dramatically improve bandwidth usage? It’s a very, very tiny minority — most people assume that applications and the operating system auto-configure themselves for the “best” performance.

Second Life tries, indeed, to set some reasonable defaults — which work in some cases, but not so well on others. In many cases, tweaking the Edit > Performance settings bring better results, but not always. Second Life has lots of ways to get configured: from Edit > Performance, from the Client/Server menus, from the configuration files on the disk. Most people don’t know how to play with those — fearing, very correctly, that Second Life stops to work at all! — and will let SL assume what it thinks is “best” for them. But things don’t stop there — then there is the whole operating system to configure. Bandwidth settings should be set to “broadband” — all computers (Windows, Macs, Linux) usually come with the defaults for modem usage, mostly due to historical reasons. Tweaking those is not for the faint of heart — and also, these are operating system parameters that don’t have “defaults”: you need a good understanding on how operating systems work, no fear of experimenting (it might render your system useless), a thorough level of understanding of all hardware devices you have (how much memory they use, etc.) and often a calculator 🙂 Nothing that one should require from “normal” users. And these are often 99% (if not more) of the population.

So, users have to live with “reasonable defaults”. Sometimes, these are not reasonable at all; installing a new device driver for your graphics cards is often the way to go to get rid of those strange and nasty “bugs”, but there is a lot that you can do to improve your performance in SL (as well as its stability), if only you knew how!

Users then clamour for Linden Lab to “stop all innovation” and focus on “getting rid of the bugs” instead. This is a recurring nightmare for Linden Lab. In most of the cases, the problem is dealing with the “reasonable defaults”; Linden Lab is getting better at “guessing” those, but there are so many new systems, cards, motherboards, and drivers in the market every month, that “guessing” is a hard work. And then, of course, there are true bugs which need to be fixed. Focusing the development team on these things means less time free for innovation, and Second Life, as a platform, is left behind in terms of being the state-of-the-art of virtual, 3D environments. But… the users (at least the most vocal ones!) seem to prefer stability to features, so that’s what Linden Lab has to give them.

Then Linden Lab has to focus on the scability and stability of their own servers. This is something which is rather tough for users. Often they see, with every patch, just a handful of client bugs that have been fixed. However, the work “under the hood”, invisible to all of us, is often dramatic. We don’t appreciate it until there are 30,000 or so users online and suddenly we notice that Second Life doesn’t lag anymore as it used to do last week! This “under-the-hood” development, which we’ll never see, only indirectly, is crucial to Linden Lab. They now have the grid in two different locations — California and Texas — and both have to run smoothly without anyone noticing a difference when their avatar leaves a sim in California and enters one in Texas. In the near future, Linden Lab will have lots of co-location facilities around the world, not unlike, say, Akamai or even Google. All of them have to work together seamlessly. And they will need to give good performance to, say, 100,000 or more simultaneous users in a year or so, at the current rate of growth. All this needs a lot of work on the server side, which has been one of their major focus.

What this all means is that there is no time left to develop and deploy innovative technology on new features. New features always mean new bugs, new “default settings” to guess at, and more client-side instability that not everybody knows how to fix. So, adding one or two minor features — and hoping that a few residents test them on the Beta grid (which they often don’t do; only a few dozens or perhaps hundreds ever do some testing there) is enough. More often, it isn’t — the Beta grid is not as complex as the main grid, and although client-side issues can sometimes be found out, server-side ones are difficult to test without having 30,000 simultaneous users and 5,000 or more sims.

All this leads to technology stagnation — Linden Lab wants to innovate, but the users clamour for less bugs and more stability, which needs more resources. That is ultimately the doom of a company in the bleeding edge of technology — when its own users don’t allow it to innovate (think about Microsoft Vista and the time it has needed to be launched in the market!).

So, what do software houses do to address this issue? Well, in the Unix and open source world — of which Linden Lab is now a part, with their release of the client source code under the GPL — there has always been an elegant solution: branching the code into a “stable” version and an “unstable” version. This is what Linden Lab now calls the “First Look” viewer.

The first results are impressive. First we got a complete overhaul of the graphics rendering pipeline — a much needed one. It now uses native OpenGL Vertex Buffer Objects. This allows reducing some CPU overhead when sending data to the GPU, among other nifty tricks. Secondly, the way textures are loaded is quite different from the “normal” SL client. As the CPU is more free to do other things, Linden Lab now loads the textures according to the objects that are in sight of your camera (previously, it seemed to work rather randomly, as CPU cycles to decode the textures became available). The net result, on a fast computer, is that as you turn the camera, SL will get only the textures you see in front of you, and load them pretty quickly, apparently from the nearest to the ones further in the horizon. After a few minutes (not hours!) you have all the textures you need for the whole sim you’re on.

To improve this further, SL now has a far bigger cache — up to 10 GBytes are possible, and you can even use a different path location for your cache. This is more important as a new improvement was introduced: threaded texture loading. What this means is that while your SL client is mostly trying at all costs to render the scenes as fast as it possibly can, textures get loaded in the background, without interfering (in terms of costly CPU cycles!) with the rendering.

But even slow CPUs benefit from this. They will get low FPS when suddenly rezzing in a new scene (when, say, teleporting to a new sim where you haven’t been before) while the CPU gathers all the textures it needs to render the whole scene in front of you (and not, as on the regular viewer, a “best effort” method where it basically renders the objects for which the SL client “happens” to have some textures…). When finally the whole textures for that scene are loaded — on my very ancient PowerBook G4, it takes from 5 to 30 seconds, depending on how clever the content was designed (not a fault of Linden Lab here, but of the content creators who don’t know how to optimise their builds) which I might suffer from very poor performance with sub-3 FPS rendering, but then — as if by magic, the “fog” clears, and the whole scene appears, often with 15-17 FPS, something I haven’t seen on my poor Mac since the days of SL 1.4!

The improvements don’t stop here. Now that a scene is rendered, the CPU can start downloading all the remaining textures for the rest of the sim — in the background, not affecting the rendering (unless you suddenly turn the camera around). Due to the way people use Second Life, very likely you’ll teleport to a place and say to your friends: “still rezzing…” and while you exchange a few minutes of similar comments, Second Life will struggle to load all textures in the background 🙂 (A fast CPU, however, will probably not even notice the delay…)

Once this was tested with a few releases of the First Life viewer, Linden Lab became more bolder. Now we can have something way cool and definitely wonderful to have in a world that does not have ray-tracing: mirrors, once the domain of the unthinkable:

https://www.youtube.com/watch?v=gLsWxyLxVZc

They are not perfect — lower resolution, and the angles are not optically correct — but who cares? This is purely amazing rendering technology using all the tricks of the trade. And something the “better engines” do all the time, of course — with static content. Linden Lab has to do it with dynamic content. And… it works! Even on the ultra-slow machines, if you happen to be on a “reasonably” intensive environment, and not in the middle of a building with 5,000 prims, each with 6 faces with 1024×1024 textures. 🙂

The mirroring effect also extends to the rippling water, with fascinating results, since water reflections do have complex optic properties and most people are not so familiar with those to understand what exactly is “wrong”. All in all, fascinating effects overall! And… this is a faster SL client!
So, once having “broken free” of the tyranny of bug fixing, and allowing to apply all the creative power to developing new improvements (while the bugs are fixed on the regular viewer), what will come next? Almost certainly, dynamic shadows, a la SketchUp. With the First Look series of viewers, Linden Lab is showing off its prowess to introduce more steps on the rendering pipeline while dealing much better with occlusion (objects behind a solid prim don’t render at all) and LOD (objects further away from the camera render with less detail). So they have more “rendering cycles” available for more dramatic effects. Simple shadow projections, like SketchUp does, should be easy to do with a few tricks. After all, if you have noticed, the Linden Trees do already have pretty convincing shadows on the ground — now the same algorithm can be applied to all objects and avatars. The result? Second Life will come closer to rendering engines from applications like AutoCAD or Archicad, which do raytracing, but which don’t have all the features of the best rendering engines in the planet. We won’t get Pixar-quality, real-time, photo-realistic rendering for a few years, but a rather good approximation which will make us more happy 🙂

There is more. We all know how hard Linden Lab has worked on the HTML-on-a-prim integration, and we are still waiting for that to become reality and not just myth. Well, several difficulties come from it — mostly, integrating interactive Flash on pages, and making sure you can click on hyperlinks on a prim, which the current engine does not allow to do easily. But now Linden Lab has a brand new branch where they can test these things out. On the First Look series, nobody is expecting “stability”, so they can go wild. It will work for some, and break everything for others, but you can always fall back to the old, boring, uninteresting — but stable and with less bugs — regular client.

As an innovative company, this approach is crucial to them. They can show off what they have in mind with Second Life. They can finally address the two kinds of users: the huge and vast majority who only want bug fixes, and don’t care about innovation (they aren’t using any of the competition’s platforms; they use Second Life, and want it running smoothly), and the ones that are able to trade off some crashing now and then and favour the “latest and greatest” — a tiny minority, but one that truly needs to show off what Linden Lab is able to do.

Add to that the efforts like Open Second Life, where a community of developers are deeply engaged in extending the open source SL client further, as well as playing around with a reverse-engineered server, and things look rosy for Linden Lab in 2007. They have to play “catch up” with the other platforms. They have to show that a platform with fully dynamic content and amateur (user-created) content can approach the quality and the smooth usage that is expected from platforms with static content developed by the top designers with all the tricks of the trade to make each scene render impressively fast. This is a huge task for Linden Lab, but with this model — a separate branch for innovation — they will successfully be able to do it.

Oh, of course, this does not mean that we will have Havok 3 soon, or Mono, or any of those nifty features that require server-side changes. Still, things like the new physical avatars might get previewed on the First Look client.

On some things, however, we can’t expect miracles so soon. I can imagine that Linden Lab could introduce things like mesh-based objects, and do some tests on how easy these would be able to be changed with the in-world tools. But… the whole economy of Second Life depends on a prim-based model! It would have to change to a polygon-based economy instead, and this would dramatically change everything (think about how many polygons are needed for rendering “tortured” torii! — but they just count as “one prim”). So this will be way further the timeline. The same, of course, applies to higher-polygon avatars for added realism (bye-bye, hundred million clothes!). All this is now technically feasible, and could even be previewed on the First Look series, but — it would completely break the whole model and assumptions upon which Second Life currently relies.

Thus, it’s likely that we will continue to see “gradual” improvements over time, and not “dramatic” ones. Still, a “gradual” improvement — like super-fast rendering, mirrors, and eventually shadows — are now possible in a very short timeframe, and we will all benefit from them!

All? Well, no — just the ones willing to download the First Look viewer, of course 🙂

It will also be interesting to see if content designers will now use both viewers to target their content specifically to address the new features, while still making them “compatible” with the “stable” version. This sounds like content development in SL will resemble HTML/CSS/Javascript development, where most professional developers need to address the differences between Internet Explorer and Firefox…

CC BY 4.0 Breath-taking improvements and how to implement them by Gwyneth Llewelyn is licensed under a Creative Commons Attribution 4.0 International License.

About Gwyneth Llewelyn

I'm just a virtual girl in a virtual world...

2 Pingbacks/Trackbacks

  • BTW, in case you’re wondering, to enable shiny reflections all you need to do is to go to the Client menu, select Rendering, and then Dynamical Reflections.

    If you don’t have a “Client” menu, you can enable it with the Ctrl-Alt-D key combination.

  • I think we’re raising the bar on configuration dilemmas and issues here, Gwyn, inadvertidly blurring the fact that LL never really cared to follow platform’s Guidelines. I’ll talk for Mac OS X for which just until a few months ago the client was a bare, skinny unoptimized CC’s recompile.

    SL violates about everything, which is not necessarily wrong, possibly typical of a revolutionary undertaking. Then sometimes LL violates English as in the FIRST LOOK viewer, where Toggle Dynamical Reflections ON and OFF does not produce the expected results (i.e. there’s no OFF, there is a FREEZE LAST, if you want OFF relaunch the client:)

    That’s not silly, it’s not their fault, Lindens just have no idea how to build an industry-standard UI.

    I am concerned when you blame it on the user. User should not, ever, have to run a {top} or a {renice} ever. A new attitude at seeing the broad masses out there versus supporting everything (Microsoft obsession) is very much required. Do not expect Joe Avatar to ramp up over an even steeper learning curve. Retention numbers say a lot about this.

    Life on the Mac is easier after all. Newbies should be happier a bit. On Windows XP ppl are nailing down the stop-lag problem in First Look to a corrupted mouse driver. On Vista Microsoft is doing all they can to discourage OpenGL and depict those libraries as “slow” as compared to their own DirectX. When you deal with tech shame such as Microsoft plus Microsft Mafia I dunno what else to tell you.

    Now, make sure you have a mouse driver and that is not corrupted 🙂

  • Ah, but the issue here is having two different branches, Starcomber 🙂 One is for the clueless newbie — install it, and it will work for you. If the performance is too bad, well, buy a faster computer — the usual answer for anything that works “out of the box”.

    The second branch, however, is different. That one is supposed to be the unstable and “not for casual users” branch: the ones willing to trade-off a ‘flawless’ installation by one which is feature-rich and requires a lot of tweaking and knowledge to have it run properly.

    I also think that Linden Lab designs awful interfaces, but at least they make pretty reasonable renderers. They’re also getting better and better on the way they handle “reasonable defaults”. After I wrote this entry, a good friend of mine, an IT researcher, tried to use the First Look viewer, and found out that lots of things — like rippling water — weren’t even checked. After some checking we managed to find out that he hadn’t the latest and greatest graphics drivers for his card. We could only assume he was using the Windows defaults or an outdated driver.

    Now… the amazing is that SL worked at all, in spite of LL’s comments to the contrary (“SL does not work except with the latest drivers for your card”). Although the “latest drivers” will naturally allow people to have access to many more features and increased performance, SL apparently is even able to handle outdated/wrong drivers as well, and still do something about it. I find this outstanding work, and “the way things should be”.

    As for the overall interface… well… that deserves one way longer analysis on what is wrong with LL’s design 🙂 People can really appreciate things like Windows 98 after looking at the quirky, unprofessional, clumsy, and badly implemented interface of LL’s. The way they handle the focus right now — differently from before, and differently from the “beta” Focus Viewer; a mix of the worse of both! — is really less than adequate. I totally agree with you on that, and it certainly the interface requires a lot of improvement, which we’ll very likely never see from LL but perhaps from the open source community in a year or two…

  • Pingback: There really are Breath taking Improvements available for Second Life » VTOR - Virtual TO Reality()

  • Extropia DaSilva

    That was an interesting essay. It is odd that Xbox Live offers a wonderfully stable online experience and yet it is the ‘unreliable’ Sl that I would rather come to. Personally I feel that the slightly less than perfect stability of SL is a fair price to pay to be among such wonderfully creative people.

    But we can all dream of the a day when the freedom to collaboratively create content happens within an online environment as user-friendly and stable as Xbox Live. Mmmmmm…But in the meantime it would serve us well to know where the Linden’s responsibility towards SL ends and ours begins, your essay was a decent start in that direction.

    But I can think of an improvement LL could bring about. If the grid is offline, I should know about it even before I click ‘connect’. But no, all seems like business as usual with 20,000 logged in and a message saying ‘THE GRID IS ONLINE’ in encouraging green…

    And when I log on it gets worse. You get a message saying something to the effect of ‘YOU CANNOT LOG ON BECAUSE YOU BROKE SOMETHING. GO AWAY AND FIX WHATEVER IT WAS, STUPID’. Ok that is an exaggeration but it is not THAT far from the truth. It certainly does not say ‘You cannot connect to SL because the grid is down’.

    Of course, a trip to the site (might) reveal the real reason why loggins fail but really, I SHOULD KNOW THE GRID IS DOWN EVEN BEFORE I TRY TO CONNECT THE FIRST TIME!!! It is not too much to ask, surely?

    Oh yeah..your essay reminded me of an old post on the forums. Somebody showed that, even if your PC was as powerful as IBM’s Blue Gene/L, you would still not have enough grunt to run SL with everything maxxed out! Woah..SL is seriously future-proofed huh? 🙂

  • BTW, there are now some changes on the way this feature gets enabled. You need to go to the Debug Settings (from the Client menu) and search for a property called RenderDynamicReflections and turn it from “FALSE” to “TRUE”. LL said that they didn’t wish that such an “unstable” option was so easy to add from the Client menu… although, depending on how you define “unstable”, it works rather well, even if it consumes quite a lot of CPU on my poor old PowerBook G4 🙂

  • Pingback: Gwyn’s Home » Blog Archive » Kirstens Viewer S18 Strikes Back!()