Breath-taking improvements and how to implement them

Linden Lab, since at least mid-2005, has been caught into a tough dilemma. Second Life is a notoriously hard to configure piece of software — too demanding from the computer it runs on, and almost always requiring detailed configuration and fine-tuning of your own computer to be able to enjoy it properly.

When I tell people that it’s unusual for me to crash more than once per month (logging in every day for an average of 4 or more hours), many are very surprised. My computer is old, almost 4 years now, and uses a graphics card that Linden Lab supports only partially (apparently, the original firmware for the ATI Mobility Radeon 9600 does not support some features properly, and LL prefers not to allow to enable them). Still, I have a rather solid application — Second Life is as stable for me as, say, Firefox 2.0.X (which also crashes once per month or so).


But if one reads the Official Linden Lab blog, the horror stories abound. What makes my poor old computer so “special” then? It is slow. Just to open my Mail application, I have often to wait a whole minute. I have not much disk space left, in spite of being continuously deleting old folders (Mail consumes almost all my disk space). The motherboard once melted down due to overheating — caused by running Second Life so often 🙂 It’s definitely not a “top-of-the-line” model, neither a particularly rough model. Like many of the recent-generation Macs (at least the ones built after 2000), it uses medium-to-low quality components. And while Windows users have all sorts of maintenance tools, Mac users, by default, don’t get any, since it is “assumed” that an Unix-based machine doesn’t “require” maintenance at all (which is not exactly true, but it’s a good selling point).

So why do people crash so often? Why do they report so many problems? It’s very hard to say. In many cases, however, the answer is simple: people don’t have the required knowledge to properly tweak and maintain their own computers. It is an art which is only available to a few. How many know which Windows services they should safely shut down to improve performance, or go into RegEdit to change some settings? How many Mac users know what a “sysctl” call is and how it can dramatically improve bandwidth usage? It’s a very, very tiny minority — most people assume that applications and the operating system auto-configure themselves for the “best” performance.

Second Life tries, indeed, to set some reasonable defaults — which work in some cases, but not so well on others. In many cases, tweaking the Edit > Performance settings bring better results, but not always. Second Life has lots of ways to get configured: from Edit > Performance, from the Client/Server menus, from the configuration files on the disk. Most people don’t know how to play with those — fearing, very correctly, that Second Life stops to work at all! — and will let SL assume what it thinks is “best” for them. But things don’t stop there — then there is the whole operating system to configure. Bandwidth settings should be set to “broadband” — all computers (Windows, Macs, Linux) usually come with the defaults for modem usage, mostly due to historical reasons. Tweaking those is not for the faint of heart — and also, these are operating system parameters that don’t have “defaults”: you need a good understanding on how operating systems work, no fear of experimenting (it might render your system useless), a thorough level of understanding of all hardware devices you have (how much memory they use, etc.) and often a calculator 🙂 Nothing that one should require from “normal” users. And these are often 99% (if not more) of the population.

So, users have to live with “reasonable defaults”. Sometimes, these are not reasonable at all; installing a new device driver for your graphics cards is often the way to go to get rid of those strange and nasty “bugs”, but there is a lot that you can do to improve your performance in SL (as well as its stability), if only you knew how!

Users then clamour for Linden Lab to “stop all innovation” and focus on “getting rid of the bugs” instead. This is a recurring nightmare for Linden Lab. In most of the cases, the problem is dealing with the “reasonable defaults”; Linden Lab is getting better at “guessing” those, but there are so many new systems, cards, motherboards, and drivers in the market every month, that “guessing” is a hard work. And then, of course, there are true bugs which need to be fixed. Focusing the development team on these things means less time free for innovation, and Second Life, as a platform, is left behind in terms of being the state-of-the-art of virtual, 3D environments. But… the users (at least the most vocal ones!) seem to prefer stability to features, so that’s what Linden Lab has to give them.

Then Linden Lab has to focus on the scability and stability of their own servers. This is something which is rather tough for users. Often they see, with every patch, just a handful of client bugs that have been fixed. However, the work “under the hood”, invisible to all of us, is often dramatic. We don’t appreciate it until there are 30,000 or so users online and suddenly we notice that Second Life doesn’t lag anymore as it used to do last week! This “under-the-hood” development, which we’ll never see, only indirectly, is crucial to Linden Lab. They now have the grid in two different locations — California and Texas — and both have to run smoothly without anyone noticing a difference when their avatar leaves a sim in California and enters one in Texas. In the near future, Linden Lab will have lots of co-location facilities around the world, not unlike, say, Akamai or even Google. All of them have to work together seamlessly. And they will need to give good performance to, say, 100,000 or more simultaneous users in a year or so, at the current rate of growth. All this needs a lot of work on the server side, which has been one of their major focus.

What this all means is that there is no time left to develop and deploy innovative technology on new features. New features always mean new bugs, new “default settings” to guess at, and more client-side instability that not everybody knows how to fix. So, adding one or two minor features — and hoping that a few residents test them on the Beta grid (which they often don’t do; only a few dozens or perhaps hundreds ever do some testing there) is enough. More often, it isn’t — the Beta grid is not as complex as the main grid, and although client-side issues can sometimes be found out, server-side ones are difficult to test without having 30,000 simultaneous users and 5,000 or more sims.

All this leads to technology stagnation — Linden Lab wants to innovate, but the users clamour for less bugs and more stability, which needs more resources. That is ultimately the doom of a company in the bleeding edge of technology — when its own users don’t allow it to innovate (think about Microsoft Vista and the time it has needed to be launched in the market!).

So, what do software houses do to address this issue? Well, in the Unix and open source world — of which Linden Lab is now a part, with their release of the client source code under the GPL — there has always been an elegant solution: branching the code into a “stable” version and an “unstable” version. This is what Linden Lab now calls the “First Look” viewer.

The first results are impressive. First we got a complete overhaul of the graphics rendering pipeline — a much needed one. It now uses native OpenGL Vertex Buffer Objects. This allows reducing some CPU overhead when sending data to the GPU, among other nifty tricks. Secondly, the way textures are loaded is quite different from the “normal” SL client. As the CPU is more free to do other things, Linden Lab now loads the textures according to the objects that are in sight of your camera (previously, it seemed to work rather randomly, as CPU cycles to decode the textures became available). The net result, on a fast computer, is that as you turn the camera, SL will get only the textures you see in front of you, and load them pretty quickly, apparently from the nearest to the ones further in the horizon. After a few minutes (not hours!) you have all the textures you need for the whole sim you’re on.

To improve this further, SL now has a far bigger cache — up to 10 GBytes are possible, and you can even use a different path location for your cache. This is more important as a new improvement was introduced: threaded texture loading. What this means is that while your SL client is mostly trying at all costs to render the scenes as fast as it possibly can, textures get loaded in the background, without interfering (in terms of costly CPU cycles!) with the rendering.

But even slow CPUs benefit from this. They will get low FPS when suddenly rezzing in a new scene (when, say, teleporting to a new sim where you haven’t been before) while the CPU gathers all the textures it needs to render the whole scene in front of you (and not, as on the regular viewer, a “best effort” method where it basically renders the objects for which the SL client “happens” to have some textures…). When finally the whole textures for that scene are loaded — on my very ancient PowerBook G4, it takes from 5 to 30 seconds, depending on how clever the content was designed (not a fault of Linden Lab here, but of the content creators who don’t know how to optimise their builds) which I might suffer from very poor performance with sub-3 FPS rendering, but then — as if by magic, the “fog” clears, and the whole scene appears, often with 15-17 FPS, something I haven’t seen on my poor Mac since the days of SL 1.4!

The improvements don’t stop here. Now that a scene is rendered, the CPU can start downloading all the remaining textures for the rest of the sim — in the background, not affecting the rendering (unless you suddenly turn the camera around). Due to the way people use Second Life, very likely you’ll teleport to a place and say to your friends: “still rezzing…” and while you exchange a few minutes of similar comments, Second Life will struggle to load all textures in the background 🙂 (A fast CPU, however, will probably not even notice the delay…)

Once this was tested with a few releases of the First Life viewer, Linden Lab became more bolder. Now we can have something way cool and definitely wonderful to have in a world that does not have ray-tracing: mirrors, once the domain of the unthinkable:

They are not perfect — lower resolution, and the angles are not optically correct — but who cares? This is purely amazing rendering technology using all the tricks of the trade. And something the “better engines” do all the time, of course — with static content. Linden Lab has to do it with dynamic content. And… it works! Even on the ultra-slow machines, if you happen to be on a “reasonably” intensive environment, and not in the middle of a building with 5,000 prims, each with 6 faces with 1024×1024 textures. 🙂

The mirroring effect also extends to the rippling water, with fascinating results, since water reflections do have complex optic properties and most people are not so familiar with those to understand what exactly is “wrong”. All in all, fascinating effects overall! And… this is a faster SL client!
So, once having “broken free” of the tyranny of bug fixing, and allowing to apply all the creative power to developing new improvements (while the bugs are fixed on the regular viewer), what will come next? Almost certainly, dynamic shadows, a la SketchUp. With the First Look series of viewers, Linden Lab is showing off its prowess to introduce more steps on the rendering pipeline while dealing much better with occlusion (objects behind a solid prim don’t render at all) and LOD (objects further away from the camera render with less detail). So they have more “rendering cycles” available for more dramatic effects. Simple shadow projections, like SketchUp does, should be easy to do with a few tricks. After all, if you have noticed, the Linden Trees do already have pretty convincing shadows on the ground — now the same algorithm can be applied to all objects and avatars. The result? Second Life will come closer to rendering engines from applications like AutoCAD or Archicad, which do raytracing, but which don’t have all the features of the best rendering engines in the planet. We won’t get Pixar-quality, real-time, photo-realistic rendering for a few years, but a rather good approximation which will make us more happy 🙂

There is more. We all know how hard Linden Lab has worked on the HTML-on-a-prim integration, and we are still waiting for that to become reality and not just myth. Well, several difficulties come from it — mostly, integrating interactive Flash on pages, and making sure you can click on hyperlinks on a prim, which the current engine does not allow to do easily. But now Linden Lab has a brand new branch where they can test these things out. On the First Look series, nobody is expecting “stability”, so they can go wild. It will work for some, and break everything for others, but you can always fall back to the old, boring, uninteresting — but stable and with less bugs — regular client.

As an innovative company, this approach is crucial to them. They can show off what they have in mind with Second Life. They can finally address the two kinds of users: the huge and vast majority who only want bug fixes, and don’t care about innovation (they aren’t using any of the competition’s platforms; they use Second Life, and want it running smoothly), and the ones that are able to trade off some crashing now and then and favour the “latest and greatest” — a tiny minority, but one that truly needs to show off what Linden Lab is able to do.

Add to that the efforts like Open Second Life, where a community of developers are deeply engaged in extending the open source SL client further, as well as playing around with a reverse-engineered server, and things look rosy for Linden Lab in 2007. They have to play “catch up” with the other platforms. They have to show that a platform with fully dynamic content and amateur (user-created) content can approach the quality and the smooth usage that is expected from platforms with static content developed by the top designers with all the tricks of the trade to make each scene render impressively fast. This is a huge task for Linden Lab, but with this model — a separate branch for innovation — they will successfully be able to do it.

Oh, of course, this does not mean that we will have Havok 3 soon, or Mono, or any of those nifty features that require server-side changes. Still, things like the new physical avatars might get previewed on the First Look client.

On some things, however, we can’t expect miracles so soon. I can imagine that Linden Lab could introduce things like mesh-based objects, and do some tests on how easy these would be able to be changed with the in-world tools. But… the whole economy of Second Life depends on a prim-based model! It would have to change to a polygon-based economy instead, and this would dramatically change everything (think about how many polygons are needed for rendering “tortured” torii! — but they just count as “one prim”). So this will be way further the timeline. The same, of course, applies to higher-polygon avatars for added realism (bye-bye, hundred million clothes!). All this is now technically feasible, and could even be previewed on the First Look series, but — it would completely break the whole model and assumptions upon which Second Life currently relies.

Thus, it’s likely that we will continue to see “gradual” improvements over time, and not “dramatic” ones. Still, a “gradual” improvement — like super-fast rendering, mirrors, and eventually shadows — are now possible in a very short timeframe, and we will all benefit from them!

All? Well, no — just the ones willing to download the First Look viewer, of course 🙂

It will also be interesting to see if content designers will now use both viewers to target their content specifically to address the new features, while still making them “compatible” with the “stable” version. This sounds like content development in SL will resemble HTML/CSS/Javascript development, where most professional developers need to address the differences between Internet Explorer and Firefox…

Print Friendly, PDF & Email