Lag Myths Dispelled

Gwyn Posing — Myth Busted!

Fashion shows have always been popular in Second Life, and with the current generation of highly talented fashion designers bringing a unique — and very realistic! — touch to high-quality clothes in SL, it’s interesting to see how, in a little less than a week, people have been talking about “rules of conduct” on a fashion show.

In fact, they’re much less “rules of conduct” but guidelines of what you’re allowed to wear or not when attending those shows. So far, so good (fashion shows in Maoist China weren’t a pleasure to attend…), but strangely enough, those “guidelines” are based on an incorrect perception of the hugest problem on those shows: lag.

While it’s undeniable that all highly attended events are laggy — it’s a limitation of the technology — I was surprised to see that almost all “rules” are based on very old limitations of the SL technology, which plagued us in 2003-2005, but that have since then be “fixed” by Linden Lab, as both the client and the server software have dramatically improved.

Improved, yes, but the lag is still with us. And, in a desperate attempt to fight down lag, people are coming up with ancient “recipes” for fixing lag — unaware that they’re not really helping out, but just repeating old myths, that simply don’t reflect the state-of-the art of LL’s technology these days. Lag will remain with us for many more years, but not for the same reasons we had it in 2003-2005.

Read Brace Coral‘s, Hamlet Au‘s or Ana Lutetia‘s blog posts on fashion show lag… and then let’s take a look at those myths, why they are still popular enough to be part of the “rules of conduct”, and, well, what we can really do about fighting lag using the technology of 2008.

And let’s enjoy those fashion shows!

Myth #1: Scripted attachments create client-side lag.

In fact, this happened to a degree back up to late 2005. LL has introduced a new scheduler somewhere in 2006, and since all scripts run on the server, what they did was to “slow them down” until they don’t bother the server’s multiple activities — namely, tracking avatars. During 2002-2005, it made sense to remove scripted attachments, since the server gave them a lot of priority in its scheduling, thus having little time left for the other, higher-priority tasks. Nowadays, scripts just… run slower. But — and this is the crucial bit! — more scripts don’t mean that everybody suffers due to server-side lag! In effect, what happens is that the scheduler lowers their priority, handles the “important stuff” (avatar tracking, texture loading, public chat) first, and only when there is a spare cycle or two will run the scripts. So, if you’re wearing a scripted attachment, it won’t affect the server much. You will just notice it running slower and slower. There is no need to detach it any more, and for almost two years this has been the case.

Myth #2: Particles also generate server-side lag.

Particles might have generated server-side lag at some point in LL’s history, but since I’ve been in SL (mid-2004), they are 100% client-side. This basically means that a fast GPU + CPU will have no trouble at all rendering all the 4096 particles by default. If everybody’s got bling on and are generating 10,000 particles, you won’t see them anyway — but just 4096 at a time. If your graphics card or CPU can’t handle 4096 particles at the same time, turn it down: you have your own configuration set too high anyway, and you should tweak it down to what your comnputer is able to handle! If you’re still skeptic that particles are a local issue, well, turn them off temporarily with Ctrl-Alt-Shift-= (see http://wiki.secondlife.com/wiki/Help:Keyboard_shortcut_keys for a complete list). In any case, your particles will not affect the server (who has no concept or use for particles) and won’t slow it down. People who are pushing their computers way over their limits should learn how to tweak them to get the best performance anyway — and yes, my 5-year-old laptop can render everything at 20 FPS or so if I turn everything down!

Of course, each of us would like to have ultrafast performance on low-end, ancient computers — but that’s not the point. If your computer can’t handle so many particles, don’t push it and complain that others are affecting your experience! SL is still beautiful with 2048 or even 1024 particles, and that will immensely help your graphics card to render everything properly on a busy area.

On a side-note… flexiprims are also handled client-side, not server-side. The server has no clue if the prim is a flexie (or a sculptie), so it takes exactly the same amount of server time to track down a flexie/sculptie than a plywood cube. So, the same rule applies: if your client is lagging because of flexies, you have the settings set too high for your graphics card (flexies are rendered on the GPU hardware) — you can start by lowering the “flexie mesh” (one of the options on Preferences) or simply turn them off completely (there is an option for that too). Or, well, just focus the camera on the model’s flexiprims instead — flexies that you can’t see are flexies that your graphics card is not going to render, so they won’t affect performance at all.

Myth #3: Primmed hair generates huge lag (client-side and server side).

Again, this was the case back in late 2005, when primmed hair started to become popular. In effect, the major source of client-side lag are avatars, with their 7500 or so polygons. Naturally enough, if you have way-complex hairdos, these will have thousands of polygons, and render slower… right?

Welcome to SL in 2008. Linden Lab has developed very aggressive culling and occlusion strategies, meaning mostly that attachments viewed from afar will have way less polygons for your SL client to render. With “way less” I mean that a hairdo with 100 prims, which might at some time have as many polygons as a whole avatar, will, from a distance (say, 20 m), have perhaps just a handful to render. Now obviously 100 avatars with 100 polygons, viewed from a 20 m distance, will have more polygons to render than a single avatar very close up! But… that’s not how one works on a fashion show… you’re not looking at the audience, which is far away when you zoom onto the gorgeous models on the catwalk. When you zoom in to a model, you’ll only get her avatar and attachments rendered with the whole lot of polygons — while the audience fades into the background and is aggressively culled down to a handful of polygons, pretty easy for your hardware to render. If you wish to do a group shot, showing the audience as well as the models, it means zooming out to fit them all on the picture — and, again, by zooming out, the polygon count gets automatically reduced!

So what should you do?…

On the other hand, if you don’t want to turn any setting down, and enjoy your graphics card overheating (believe me, I’ve blown up a few myself!), there is a very simple trick: don’t look at the audience. Focus your SL client on the catwalk only. Since mid-2006 or so, the SL client will thankfully ignore anything it can’t see (another change which has come with the times; long are gone the days where a particle emitter behind you would affect client-side lag!). Alt-Zoom on what is the focus of the show and ignore the rest. In any case, the golden rule is that others are not going to affect your computer’s performance — client-side lag is only your problem!

So, to recap: up to 2005, SL did indeed render all those millions of polygons in full, since they didn’t use any culling strategy, and detaching your hair would make a difference. A huge difference, in fact! But this is 2008, and the SL client works in a completely different way; in fact, it tends nowadays to render almost a constant number of polygons all the time, culling the meshes down as you zoom out, adding more polygons as you zoom in (but the background gets occluded anyway and won’t interfere with the rendering if you’re zooming in very closely!). Granted, none of this is “perfect” yet, but the difference is HUGE!

Still, urban legends and myths about SL are still with us, even if the technology advanced quite a lot. The difference is that back in 2004, most people around in SL were computer nerds who patiently explained what happened, and we picked up those “old tips” and got them perpetuated over the years. As SL grew in number of residents, the number of computer graphics experts remained the same, and they never explained to a large audience how things work in 2008… so we’re stuck with the old myths, and keep repeating them over and over again, even if they’re just plain wrong!

So why are we still lagging, if everything’s so cool and perfect? You might have noticed that on some of the fashion shows there is an “attachment nazi” yelling at everybody (and sometimes even banning people) to remove all attachments. As an increasing number of people comply — you can see jewelry, hair, sometimes even shoes, promptly getting removed — lag does not decrease. Frustrated, the “attachment nazi” continues to yell, over and over again, to have people remove their invisible attachments — AOs, multitools, facelights, and so on. There is no way to know if they’re being removed or not, so the session usually ends with the “attachment nazi” insulting the audience that is still “hiding some scripted attachment somewhere and causing the whole sim to lag”. But, in fact, most people — strong believers in the old myths — might have, indeed, removed all their attachments.

But the sim will still lag. Why?

Lag comes in all sort of varieties. Place the catwalk facing a sim border with nothing but the ocean behind it, and create lots of opaque prims behind the stage — and the lag will suddenly drop immensely. Why? There is “nothing” behind the stage to render, so the SL client will be superfast. Turn the stage towards an open beach, complete with marina, boats, a lovely strip of land full with houses and delightful gardens — and everything will lag like crazy. Sure, it’s beautiful — but unfortunately it means that the SL client will now have to worry about all those thousands of prims, all those swaying Linden trees, all those reflections on the water — instead of rendering the models on the catwalk. So, positioning the stage correctly will improve things dramatically.

Then, although the SL client is much better at its rendering strategy, and in spite of the huge performance improvements from the new script scheduler (Mono will improve that further), the simple and plain fact is that the sim server will have a lot to do on a sim full of avatars. First and foremost, skins. The old rubber-Linden-skins have just a handful of KBytes to render; any new-generation skin (from 2005 onwards) is megabytes in size (due to the increased detail), and we have three textures to load: head, torso, legs. 3 MBytes per avatar at a minimum; 100 avatars on the audience, that’s 300 MBytes to load. And where do all the textures come from? You guessed it — from the sim server. Which has to send those 300 MBytes to 100 avatars, or 30 GBytes total. So, on any busy event, we have the poor sim server struggling to push 30 GBytes downstream to all the unhappy and laggy avatars around. There is no way to avoid this step!

… surely people are not going to suggest that we also drop our skins and the detailed clothes, and run around naked into white Linden undies? 🙂 Of course not. So, we suffer huge texture lag — and blame it on attachments 😉

Well, attachments have problems, too! The more detailed they are, the higher it’s likely that the creator has just photosourced something and repainted it on Photoshop, for a beautiful 1024×1024 texture that looks gorgeous when zoomed in — even if it’s just a tiny detail on a earring. Now, although the number of polygons get dramatically culled down when viewed from a distance, the texture still has to be downloaded to your computer — and that’s additionally to all those lovely skins! And, on average, each prim has seven textures to load. 15,000 prims, that’s 100,000 textures, or 100 GBytes (on average), assuming that you’re loading them all (which most people aren’t). PLUS all the textures on the attachments!

“Ah-hah!” I hear you say, “So, after all, there is a good reason for people to drop their attachments, right?” Well, yes and no. It all depends on the content creator. For instance, I have 70-or-so-prim hair… but… it only has one texture for the whole hair. That’s as bad as Linden hair, which also needs to load one additional texture anyway! So, assuming you’re using one-texture-hair (which most of the best hair designers know how to employ successfully), you’re actually not putting a huge strain on the infernal texture loading, even if you have 100 prims on your hair! (note that Linden hair also has a huge amount of polygons — as many, in fact, as 100-prim-hair! — so for the purpose of the polygon count, which is what really creates client-side lag, it’s about the same)

Of course, someone coming in with a 2000-prim-hair, where each hair prim has its own 1024×1024 texture… well… those people should be shot on sight 🙂

Note also that most jewelry work the the same way. You might have 100 prims on your bracelet, but a professional jewelry designer (and fortunately we have thousands of those in SL) might just use one or two textures — and for jewelry, they can easily get away with 64×64 textures, since things are so tiny (which means that even at a short distance, the SL client will drop polygons very easily). The good ones are all done like that. Similarly, even shoes are done with few textures, and sculptie shoes are even better — although they have one dramatic issue, which is the “invisiprim” script (which lags both the server and the client). Although we seldom hear the “attachment nazis” complain about shoes, but focus only on the poor AOs, shoes are actually quite worse for lag — even if they are low-prim (like sculptie shoes) and have no scripts for bling, walks, or heel sounds. But… they’re so lovely to wear 🙂 (Who uses Linden shoes these days?)

In conclusion:

  • Client-side lag is caused (mostly) by insanely-high-polygon count; avatars and Linden hair/skirts are one of the major sources of high polygon count.
  • Server-side lag is caused (mostly) by textures that need to be uploaded; avatar skins are the major source of texture upload.
  • Badly done attachments (with too many useless prims with insanely huge textures on each face of each prim) and badly built catwalks and stages are a major source of server-side lag.
  • Invisible attachments create zero client-side lag. Shoes create more lag than AOs or multitools. Particles don’t lag the server at all; if they’re lagging your client, you have far too high settings on your Preferences.

But ultimately, removing scripted attachments, bling, or hair won’t help the lag a bit — if you’re allowed to keep your skin and prim shoes! That’s why the “attachment nazis” can ban as many people they want, “suspecting” hidden attachments somewhere, but the lag will not improve… it’ll only get the audience furious.

Finally, even if everybody used rubber-Linden-skins with just white undies, things would still lag on the client. And this is for a simple reason: there is a limit to how many textures (even small ones!) that your graphics card can handle. If you have more textures to render than your graphics card has memory, you’ll always lag, as SL swaps out textures to computer memory (and in extreme cases to disk). This happens all the time. The reason why some games and virtual worlds never lag as bad as SL is because professional graphic designers and 3D modellers know exactly, for a specific scene, how many polygons are going to be viewed, and how many textures need to be in the graphic card’s memory for that scene. Also, it helps a lot that these games have static content that is fetched locally from your hard disk (so you don’t need to have a server pushing 30 GB of textures during an event!). We have no luck in SL: there are always textures to load, putting a strain on the poor server…

So why is it important to dress good on a fashion show? We go there to look at the models on the catwalk, right?

Fashion shows are social events, like going to a live music show, or a discussion, or dancing at a club. On social events, we dress for the occasion — and take pictures (or even make movies) — since that was always, in the past 250 years or so, the whole point of attending those events in fancy dresses. Yes, of course we want to see the models (as we also want to listen to live music or to the DJ…). But if we’re going to be there, we’re also there to be seen. This is natural, and also part of our social behaviour on such types of events. We interact socially by gossiping in IM (or in public even!) while we watch the show. If the “attachment nazis” force us to get Ruthed (the only avatar that does not require a Shape upload, since it’s a default stored on our disks), and the audience has to be a crowd of Ruthed avatars… what’s the point then? It’s so much better to watch the show on a video stream instead! Or, well, visit the fashionista blogs… where there is no lag (but also no social interaction besides a few comments on the blog posts).

Fortunately for us, Linden Lab has dealt with the issues of the “three myths” enumerated above. They’re no technical excuse any more for the “attachment nazi crowd” to impose us a “dress code” (or rather, an “undress code”). Of course the myths are very hard to dispel; they will persist for ages and ages, long after people have forgotten why we had them in the first place. So, quoting Brace Coral, “Have a Great Time” — and make the show entertaining and pleasing for everybody without “forcing” them to role-play Ruth.

Thanks to Ana Lutetia for the lovely pose 🙂 She’s great at doing cute anims!

CC BY 4.0 Lag Myths Dispelled by Gwyneth Llewelyn is licensed under a Creative Commons Attribution 4.0 International License.

About Gwyneth Llewelyn

I'm just a virtual girl in a virtual world...

5 Pingbacks/Trackbacks

  • Pingback: Lag: myths and solutions()

  • cyn vandeverre

    You’ve done us all a great service with this post. Thank you.

  • Gwyn, you’re forgetting that for a lot of people SL is a very emotional experience, and a lot of lag is psychological, and that’s important to address.

    It’s like this lady who has discovered that people have “e-mail apnea” and stop breathing when writing email.

    Rotating prims may not lag the sim and may only render client-side, but they create the *perception* and *feel* of lag, tackyness, irritation — and so it’s best to kill them.

    It’s hugely just plain RUDE to show up at an office meeting showering prims all over hell and wearing a huge complex avatar or dress with all kinds of flexi or moving parts batting people or just getting in their view.

    Again, I dont’ *care* if it only renders client side. That’s bad enough! My set-up freezes every 10 minutes, rendering client side doesn’t mean the problem is OVER just because it’s shoved from one side of the ledger to the other.

    And it ruins the *view* and again, creates the *perception* of lag, which is 3/4 of lag often.

    Once again, I want to point out that this much ballyhooed open-ID does not work here. It’s like you have some sort of your own OpenID or something. My OpenID when typed in goes nowhere. It’s stupid.

    Prokofy Neva

  • These tips are frankly heretical. The only source for lag reduction tips is the Book of BigJade.

    “And BigJade decreed: ‘Big view what cause lag, club have many walls stop lag’. And as it were so, it was good, and Club Elite flourished.”

    Amen.

  • Very nice article! I hope that this will mean that I won’t get odd stares when bringing my Luskwood avatars (Old furry avatars made with lag in mind) to a fashion event.

  • Actually you’re wrong about skins I’m afraid.

    We know that the three clothes “items” that’s head, upper body and lower body, are composited and streamed as 512 X 512 pixels. If you waste your time uploading a clothing texture as 2048 X 2048 any “improved quality” is a figment of your over active imagination.

    Each texture has no alpha channel, so it rolls in at 3MB. Each avatar wearing just clothes and skin is 9MB, regardless of the detailing or simplicity of the clothes and skin. Stripping off to a Linden skin won’t help at all, you’re just changing the textures that cause the load (that might not be entirely true, if you all change to the same skin, once it’s loaded once it should load for all I guess).

    Attachments still add lag, because, as you’ve identified, they have extra polygons to draw and moderately often have stupidly high texture details too.

    Sadly, removing AOs *can* have a benefit. You have people who have “shifting sit” AOs: that gives you cycling animations to load if you can see them, and that contributes to lag badly. Don’t believe me? Go to a class with 40 seated students and watch the fps etc. Take the same students to a dance and watch them – even in the same location the fact your client has to draw, and stream, all those moving polygons will slow things down (and you can see a pause in dance anims as the animation is streamed to you if you’re not dancing the same as everyone else).

    But, a good, thought-provoking article, thanks Gwyn.

  • Pingback: MichaMi Style Blog » 68 Prims()

  • @Prokofy, you’re very right on the perception of lag — but this “perception” comes, indeed, from myths, legends, rumours, and misinformation. That’s the reason why prim shoes rarely get “attacked” by the “attachment nazis” (so long as they don’t have bling…) — they came into existence later in the history of Second Life. The same, of course, applies to skins — nobody ever asked me to remove a skin when entering a laggy sim. So the “perception” of what causes lag and what doesn’t widely varies, and only in some cases it is related to actual technical issues.

    As for OpenID authentication, my apologies — I was using an older version of one of several OpenID plugins for WordPress. I’m trying a new one that has almost “official” status (it’s being supported by the WordPress team directly) and hopefully it works better. Except for Yahoo OpenID, it seems to work fine for most OpenID servers, although it doesn’t display the Gravatar properly (I’m working on that, too!).

    @Eloise, the skin calculations I’ve got are from two sources I’ve got: Linden skins and (old) Second Skins (the PSD for version 1.0 Second Skins were available as a FTP download for L$6000 back in 2005). Both types are, of course, 512×512 pixels. The TGA files, on my disk, occupy 23 KBytes (Linden skin) and 1 MB (Second Skin) respectively. Why the huge difference? Linden skins have far less detail, so they can effectively be more compressed during transmission. How bad is the difference in compression when downloading it from the grid? I don’t know, but certainly there is a huge reason why two files with the same pixel dimensions have such a huge difference when locally stored!

    Obviously an avatar with absolutely no attachments (and no Linden hair and no Linden skirts) will create less lag than someone wearing, say, a ring. Still, the difference is not perceptible to the end-user — aggressive culling will get rid of almost all polygons on that ring, even from a short distance (and, again, when zooming in on the ring, you’ll basically be shutting everything else down, so that doesn’t change the polygon count dramatically). Granted, if the ring has 500 prims, each with 3-4 faces, and each face with one 1024×1024 texture, it would be a “monster lag generator” just because of the textures that had to be downloaded — but, fortunately, most jewelry designers don’t work like that.

    As for anything that rotates animations, yes, granted, that will cause some lag, for two reasons. Each new animation requires about 160 KBytes of downloads (a few might be larger than that). But setting a new animation also demands a “full update” of both the avatar and the chair it’s sitting on. Full updates are a big, bad, no-no thing — sadly, for some things, LL was too lazy and simply forced them. Most things in SL don’t require full updates (you can check this out from the Client menu, it has an option to show you what objects are sending full updates and what are happy with partial updates). So, yes, granted, a hundred avatars, all with AOs, all furiously forcing full updates, will lag the sim (and lag less the viewer, since once all the anims are loaded, they’re stored locally — and unlike textures, they don’t “compete” for precious graphics card memory).

    Also, granted, vehicles lag the sim more than attachments (due to interactions with the physical engine), and some objects are scripted as vehicles (a few brands of animation overriders do that in order to capture keys in a different way), or sometimes people just appear on motorcycles, wheelchairs, or similar vehicles. These will obviously lag the sim more than if you walk to the event.

    And obviously if we turned off all animations whatsoever, the sim would lag even less 🙂 It would also be interesting to see what lags the sim less: all avatars sitting down or all standing up. I tend to believe that sitting down will lag the sim more, even if the only animation you’re using is the “typing-in-the-air” animation…

  • Ta a lot for such a detailed discussion. Old myths die hard, it seems, and most of us do not know enough of the technical background to even start to contest their validity :).

  • Good article. Unfortunately, we live in a world where people just want everything to magically work – and ‘client side’ or ‘server side’ doesn’t matter to people too awful much. It is just plain lag to them, and the urban myths will prevails.

    Also – scripted attachments do still use server resources that could otherwise be free, as is the case with prims. The rendering and all the other ‘fixes’ you mention certainly help but it really just masks the fact that the algorithms and hardware have a lot of catching up to do to meet inflated expectations of people.

    I recently explained to a client that WoW and other synthetic worlds have better graphics because they are all installed when they run the CD. Not so with SL. And texture lag is a strain on everything – but it is generally worth the price (IMO).

    In the end, applying common sense is a risky business. And a ‘social event’ that has more people than one can talk to – honestly, I question the value of those, but then my personality is not one that permits me to be pleasured by avatar fashion as much as avatar ‘content’. 🙂

  • While I’m happy with anything that dispels myths about lag, I don’t think that attachments should be discounted so easily. You can’t spend the entire time at a show with your camera pointed at the stage — at the very least, you have to find a place to park yourself first. It’s a bad start to the event if you have to swim through terrible lag during that time period. I don’t think it’s unreasonable to ask that guests keep their prim counts moderate, and save the hundred torus hair and thirty flexi skirts for another time.

    If I were running a fashion show, I’d also be reluctant to rely solely on people adjusting their own graphics settings and pointing their cameras properly. Some people are… unskilled with that kind of thing (more the flexi preference detail adjustment than the particles, which never seemed all that laggy to me, or the basic camera control, which isn’t that difficult). Plenty of them will just get frustrated instead of mitigating their lag problem on their own.

  • Good article darlin!

    I’m in the camp where client or server side – anything you can do to make things run smoother for yourself – is a good thing.

    If you want to actually be able to see whats on the runway, check yaself before ya wreck yaself I always say.

    Might just make a better experience for everyone overall.

    The main thing that concerns me really, is not the lag, but what seems to me to be a swinging back to 2004/2005 where people aren’t attending fashion shows to see whats new and to then go SHOP.

    Its like entertainment. See and be seen and then that’s it. Oh well. I suppose things will swing back at some point, but I know its been a discouraging trend for many designers.

    -BC

  • @Miriel, in fact, there are three issues at stake here.

    The first one is technical — “hard facts”, so to speak. It’s undeniable that, for instance, Linden hair has usually less polyons than prim hair with twisted torii. I’ve done a rough count of polygons, and apparently Linden hair has about 500. A single twisted torus in my 100-prim-hair has about 100 polygons — so, if the SL client renderer would treat them in the same way, we would be talking about (at least) 20 times as many polygons!

    But as I tried to patiently explain, that’s not how the SL renderer works these days. Culling is very aggressive; from about a distance of 6 m (or the camera distance!) the polygon count for such small detail decreases at least about a factor of ten for the small torii, and only the visible ones are even calculated (which is about half of the total). So my insanely-laggy 100-prim-hair, viewed at “third person camera”, has about as many polygons to render as Linden hair.

    Naturally enough, when zooming in on prim hair, SL has to display all those polygons (well, at least the half that is visible to the camera). So, yes, prim hair viewed on high levels of zoom will, without a shadow of doubt, have at least 10 times the number of polygons to render!

    But there’s a catch. When zooming in at that level of detail, the insanely-laggy-prim-hair will also occlude everything “behind” it. So in fact it’ll be preventing all those polygons from all the detail behind it (other avatars and their attachments, ground, buildings, etc.) to be rendered. So zooming in effectively lowers the polygon count for other agents in the scene.

    Obviously this doesn’t mean that rendering 10,000 polygons close up is “better” than 500 polygons close up. Not at all! However, rest assured, your graphics card, even if it’s an old one, will easily render a few million polygons every 1/30th of a second, so it doesn’t matter that much.

    Now this is the technical reason why prim hair is not that bad.

    The second reason is educating the resident to learn how to work with the SL preferences. I also agree that we can’t expect everybody to be an “expert” in knowing how to tweak the application (in RL, people pay me to tweak servers to give better performance, lol — so, yes, knowing about expert configurations is a skill to be learned, and quite a valuable one!). SL is simply too complex to deal with “reasonable defaults”. Even the experts can get stumped! I humbly admit that I don’t know (yet) how to tweak WindLight properly; after several hours at giving it a try I managed to get almost all settings at their maximum level with just a 33% hit on performance (from an average of 30 FPS down to 20 FPS). When I install WindLight from scratch, it insists to go to the lowest possible settings — at a 25% hit on performance compared with the regular viewer. So it’s not worth staying at such low settings — but getting them right (with reasonable performance and as high as possible) took a few hours!

    Yes, I fully understand that most people don’t care or don’t even know how to start doing that kind of patient trial-and-error configuration (even if it pays off hugely). The same applies to the simple tricks like focusing on the show and not on the audience… some people have no willingness to learn how to do that, and there is little one can do about that.

    So, thirdly, we have the social issue. When you go iRL to a fancy show on a fancy place, there is a dress code. If the place is very fancy, the dress code might even be enforced in the sense that they don’t let you in. Naturally, it’s fine to apply the same set of social norms (or a different set, like, “wear a Ruth avatar”) to Second Life as well. Prokofy is right on his comment: the perception of lag is important, too.

    I have no problem with “social norms”; in fact, I enjoy them very much, since those are the ones that bind us together as a community, in RL but in SL as well. Still, in SL, most of these norms are usually based on an incorrect perception of the technical issues, and the technical issues are given as a pretext for imposing a dress code.

    But that’s not necessary! When I visit Caledon, I adapt to their dress code naturally — I have enough Victorian outfits ready for that. I’m not forced to go to Caledon — but once I’m there, everybody assumes I know about the dress code there, and it would be rude and impolite to ignore it (and, ultimately, Desmond might ban me 🙂 ). If I disagree with their dress code, I simply don’t go there. But nobody at Caledon argues that there are tecnical reasons for their dress code. It’s just a dress code. It’s adopted because everybody wants it. Nothing else; no “excuses” are needed.

    Thus, it would be fine for me to jump to a fashion show and see a large sign saying: “Dress code for today: Ruth avatar. We love Ruth! Make sure you wear one of those, or else, the bouncer won’t get you in.” And it would be my choice to accept the dress code or leave. But, well, I don’t go for the paternalising and condescending comment of: “We know you don’t know how to tweak your settings properly, and that you’re clueless about lag issues, so, well, wear your Ruth avatar and enjoy the show.” It’s insulting to everybody’s intelligence in the extreme, and, marginally, to LL’s programmers as well.

    Using “technical excuses” to impose social norms is something that I really disagree with, specially, of course, if they’re ungrounded on facts, but only on “perceptions”, superstition, myths, and legends. As said, play it straight and honest: impose a social norm because you can do it if you wish, without using a pathetical “technical” excuse…

  • I think you may be somewhat overstating the benefits of culling. I actually have several instances of hundred torus hair in my inventory, and I tested this using the ktris drawn numbers. At 6 meters out, the difference between the linden hair and the prim hair is relatively small (between 5-10, depending on the hair), but it rapidly goes up when you get closer. At the default camera distance from your own avatar, the difference can be huge (138 vs. 40, in one case). Even the small difference at 6 meters will add up when you’ve got many avatars in one place. Windlight’s avatar imposters help with things, and avatars in close proximity will result in them occluding some polygons, but still. I make hair, and when I line up a dozen copies for texturing, things lag — even on my good system, and even at a distance. My experience has been that little things add up.

    However, I don’t have a perfect grasp on the statistics bar, so feel free to tell me to slink off into a corner if I’m yammering on about the wrong numbers. 🙂

    “We know you don’t know how to tweak your settings properly, and that you’re clueless about lag issues, so, well, wear your Ruth avatar and enjoy the show.”

    Except a number of people really don’t know these things, and they come to a fashion show to look at clothing, not wade through a sea of flexiprims or learn how to optimize their settings. I don’t think it’s condescending to recognize this and try to keep the experience from being frustrating for anyone who hasn’t delved into their settings. Even people that do know how to do that shouldn’t have to turn their settings way down just because their host was unwilling to state some basic ground rules.

    I’m not saying that everyone should go to these things as Ruth — a few prims here and there aren’t a big deal — but requesting that people avoid their primmier attachments for the sake of the other guests seems like a perfectly reasonable compromise.

  • Thanks Gwyn, you and your visitors/commenters made me finally clear on what/how to approach bloging on SL things. Also it was thought & emotion provoking to extent – Prok (business as usual) has evoked me to recall and read on Copernican system vs Catholic church. And this in turn made me clear why I feel often weird coming to CDS sims – I do not see church building as library and science/culture thing (just can not imagine church building as place of knowledge/science dissemination), and medieval/Roman things are more associated in my head with all that oppression from Catholic church and Catholic population at large.

  • No, Miriel, actually you are right. And I think you’ve summarised it quite well:

    My experience has been that little things add up.

    That’s certainly true!

  • Banana Stein

    Thank you Gwyneth for taking on this technical subject and explaining it so nicely. Much of the SL experience for residents can be improved just by builders using common sense and remembering the old 3 second rule for web-sites; “If it does not load, people will not stay.”

    The top tips I give to any new builder in second life is #1 – prims are precious, and #2 – standardize your textures.

  • When I have seen a server lagged to death by a script — which I don’t see very often — it’s usually a “temp rezzer” script. There have been some servers running at a time dilation of <0.5 that sped right back up after a handful of temp rezzers were removed. Those are scrips to avoid if you want to avoid server lag.

  • Sylvia Sonoda

    The way I read it, it says that ALL server side LAG is caused by texture
    loading.
    So in theory, when you put 60 avatars in a room eventually everybody has the
    textures in and the serverside lag should drop to zero?
    In other words the Dilation should go back to 0.99 or 1.0?
    And what about collisions?
    Sorry, I am not convinced.

  • Actually, Sylvia, quoting myself on the conclusion:

    Server-side lag is caused (mostly) by textures that need to be uploaded;

    The keyword here is mostly. Obviously that the more avatars there are on a place, the more things the server has to track down — positioning; collisions (as you mentioned); chat; and all sort of client< ->server communications. Back in 2005 or so, Philip & Cory had written some things about the technology explaining that each SL client opens up 13 connections to the grid simultaneously, for several reasons, but mostly to track position information for all objects and avatars on a region and the surrounding regions. This is exponential, of course, and everything that scales exponentially is a Bad Thing — namely, for each of the hundred avatars, a connection needs to be open to tell the server where that particular avatar is (or where it is moving); but then the server has to tell the other 99 avatars in the sim all about that avatar’s move. This means 10,000 notifications of movement, which have to happen “several times per second” — ideally, 45 times per second (servers endeavour to work to display 45 frames per second), for all avatars to get informed. Movement prediction algorithms can effectively limit the number of updates, of course; interpolation techniques (which are used at the client) allow the client to need fewer “checkpoints” to correctly display an avatar move between two points; and obviously, if nothing changed (the avatar is sitting down and doing nothing), there is no point in sending any notification about it. Still, it’s quite obvious that, the more avatars the system has to track down, the less CPU time it’ll have to track other things down — like, well, chat or sending assets to the clients.

    Of course, if the sim server also needs to be constantly uploading textures, well there is little time (and bandwidth!) left to send them positioning information. The sim time dilation steps in — updates are getting delayed and slowed down as the server tries to catch up with everything.

    We know that tecnologically it’s possible to have a server track down about 20-25,000 avatars simultaneously with little or zero lag. All virtual worlds and MMOGs are quite able to do that pretty neatly. If you look them up, you’ll see that most “shards” on MMOGs claim to be able to track down that amount of users — it’s a number I found rather consistent among the many MMOGs (and it also dispels another “myth”, which is that some MMOGs have as many active users as registered users…). Actually that number is also pretty accurate for the number of simultaneous users of a web site server (I have seen claims of being able to support more users per server, but, well, I don’t know how accurate those claims are). So, we know that at least in theory, and assuming that Linden Lab programmers are neither better nor worse than the average programmer, they could have sims just with tracking information (and sending no assets) and allow quite more avatars per sim! And the difference is about two orders of magnitude!

    Granted, on SL we wouldn’t have 20,000 users on a single sim, even if all assets were downloaded from elsewhere, because, unlike MMOGs and other virtual worlds, content is created on demand. This means that you can’t simply say: “this scene is static, these are the agents (avatars, vehicles, moving bits) that I have to track down, and I only need to send information for the moving bits”. In SL, everything is ultimately a “moving bit”, and not only that, everything can be on a state of constant update (as people build objects interactively, rez them and tweak them, or, well, these objects have moving parts, auto-rez new objects as part of their details, and so on). SL is incredibly dynamic — and this means tracking down a lot of data. And a lot of data several times per second!

    So, yes, a lot of server-side lag comes from just that — tracking down a fully dynamic environment, 45 times a second, and sending all the information about the environment to the client.

    While, at the same time, uploading 30 GBytes of textures.

    One might ask why LL doesn’t simply offload the textures on a separate set of servers, and let the sim servers just deal with the data tracking. There are two reasons, one being mostly philosophical: in theory, a “simulator” is a self-contained entity, that can “survive” all by itself — it contains all assets, a small (replica) of an asset server, a physics engine, and tools to track down data (this is, in fact, what allows OpenSimulator servers be runnable at your home PC — you just need the sim server software to basically run everything you need. To integrate your own sim with another person’s sim in order to exchange information between both, you’ll need an external asset server). If this is a good strategy or not, well, it remains to be seen.

    The other issue is pragmatical. The Lindens publicly claim that they store about half a Exabyte of asset data. That’s about half a billion Gigabytes. Simply put, there is no technology yet — except, well, a distributed grid — to give you that amount of storage from a single array of managed disks. Currently, these are measured in the Petabyte range (a million GBytes), but LL would need a thousand or so of these. It is way more practical to store textures and other assets on the distributed grid and let each sim server deal with a fraction of the data. Also, it means that if a sim server breaks down completely (without backup), only a tiny part of the assets are lost.

    If LL actually has so many assets as they claim, I have no idea (my own previous calculations estimated a much lower number, around 2 or 3 Petabytes only). One thing is for sure — content (in the form of assets) outnumbers tracking down avatars by far. Even taking into account that avatars (and objects) in SL will need way more complex information to be properly tracked down, compared to regular MMOGs (like, way, WoW…), still, “content is king” and His Majesty The User-Generated Content demands a lot of attention of the poor stressed-out simulator servers…

  • I think that the whole thing really revolves around one thing (funny enough): If people invest enough of themselves into the technical aspects, as with all things technical, they will be able to tweak things. And let’s face it: No one says – ‘Hey, join Second Life to tweak everything’.

    At the end of the day, people just want stuff to work. What they get is technology. The true magic isn’t explaining technology – it is making it transparent. Remembering that would make a lot of other things much more simple.

  • Aaah Nobody, so well put! Yes, that’s precisely it. There is no need to “tweak settings” on a fixed phone: just dial the number, and it works. All the time! Magic!

    Yes, we’re light-years and eons away from that in Second Life.

    On the other hand, there is obscurantism: because of an improper knowledge of the technology, myths are born, which have little or no correspondence to reality. If they are harmless, they’re ok (like the silent prayer to get your car started in a cold morning; so long as you don’t really believe you’re placating any demons inside the combustion engine, that’s fine).

    My only issue is when these dictate social norm.

  • Hypatia Callisto

    a few things:

    the majority of damage is already done when someone has actually teleported into the sim with the 100 tori hair. The more attachments you wear, the harder a hit the sim takes when you rez into it. It is the actual process of rezzing into the sim that causes the sim to temporarily lag.

    You then compound the hit by… taking it off. You take it off, the sim fps will dip, again … as you derez it! Watch that statistics bar when you remove and attach your hair, your sim fps does dip, the heavier an attachment is in prims, the heavier the sim will dip.

    Yes, you will lag more when your camera gets closer to the 100 tori hair, but you can also adjust your objects slider in prefs so that they are rendered less fully.

    So yes, the hair and attachment police are actually contributing to sim lag by making people derez the objects they came in with.

    Image time is the number one reason sims have to be rebooted – this is what happens when a sim gets overloaded sending textures to too many systems over an extended period of time – the image times go up and the sim gets slower and slower. The image times get higher and higher, and that sim fps gets lower and lower – you can actually see it happening on the statistics bar in your client.

    The one issue after that are scripts that constantly rerez things – those lovely temprez vendors and other assorted temprez scripts to save on prims. Too much temprez will cause performance issues – I’ve seen it with my own eyes when managing sim lag issues in Caledon. Its the same issue pretty much, every time you rez a prim the sim takes a hit, the more complex and numerous those prims are, the bigger a hit the sim takes.

    and lastly particles – particles can contribute to sim lag if they use large textures and have to display many large textures to various people. It’s not particles per se that do this – its the process of loading textures that does the job. So a sensibly scripted particle system does not lag sims, no – but particle systems designed to grief and lag sure can.

    So bottom line – textures lag sims, the more textures you have to load at once, the more the sim lags – and rezzing and derezzing prims lag sims, and the more ktris a prim has, the more it lags the sim as it rezzes, the more prims you rez at once, the more the sim lags.

    I think it might make sense to someone… maybe 😀

  • Hypatia Callisto

    oh and… I’ve ignored the Caledon “dress code” off and on for my entire time living there (and I’m not the only one)… nobody will get banned for not dressing Victorian 😛

  • Pingback: Prim Lag — Myth or Reality « Around the Grid with Harper()

  • Vikarti Anatra

    Making centralized ‘texture server’ is of course impossible. But there is another solution – why sim should be handled by only one CPU after all?Why LL don’t make sim process consisting of 2 parts:actual sim(which is doing all regular things) and texture management server for that sim(that will store texture and send texture data over network). Both parts should communicate via internal high-speed network (or,if they use different CPUs of one machine, via shared memory). This way simulator will offload texture sending to another process(may be running under OS instance specifically configured for high network bandwith).
    Scripts could also be offloaded(this is of course more difficult becouse this will require more tight interaction with main sim process).
    Only problem with this solution is the fact that more hardware needed for this.

    As for client-side lag, afaik, client currently just has 3 base ‘level’s of graphics settings(after hw detection each video card is fall into one of 3 groups and default settings are configured according to that). May be client should pay more detail to which hardware user actually has and reconfigure particle count/flexiprims mesh resolution accordingly?

  • Shirley Marquez

    First, the math on the downloads of baked skin textures in comment #6 is wrong. Avatar textures are baked at 512×512 pixels, which is 0.75 megabytes of uncompressed data, not 3 megabytes (that would be a 1024×1024 texture). There are three per avatar, for a total of 2.25 megabytes. (And that is the uncompressed size; textures in SL are actually stored with JPEG2000 compression, so the files will be much smaller.) If your skin has textures larger than 512×512 they will be downloaded to YOUR client and used for the texture baking, but will not be sent to anybody else.

    Re comment #8: if the Second Skin textures really are 1MB, they’re probably uncompressed TGA files. SL often has trouble uploading compressed TGA, so people tend to just save uncompressed ones so that the upload will work properly. The viewer applies JPEG2000 compression before uploading (1.19 viewers have an option that allows lossless uploading of small textures, but that won’t apply to skins), so the size of the texture on the SL asset server is the same whether the uploaded TGA was compressed or not.

    On comment #20: the baked textures USED to be handled by the asset server. LL changed things about a year ago to store them on the sim you are visiting instead, and that was, overall, an improvement; it lessened the load on the horribly overburdened asset server, at the expense of putting some additional load on each sim server. Textures other than the baked avatar textures still come from the asset server cluster, and delays in downloading them are one of the major sources of lag in Second Life as we know it today.

  • And… what about models? When we are in the catwalk, what we see is the audience.

  • For another opposing view, see Laynie’s article on reducing lag.

    Thanks to Ana Lutetia for the link!

  • Pingback: [review] Gumi’s Bad Box ~ Take Five | Ana Lutetia()

  • Pingback: [review] Fashionity Fantasy ~ Victoria, Elisa and Sofia | Ana Lutetia()

  • Shimere Felisimo

    Hi Gwyn, great article! Sorry I’m commenting so late – here via an article on Hair Fair.

    I wonder if you know the answer to this question…a few months back, I would have an issue in a texture heavy (not necessarily avatar-heavy – think Nicky Ree or Bare Rose Tokyo) sim where after a while both my movement and camera would slow and then eventually lock up. (I know this was client-side since the problem vanished when I got a new, rawkin’ graphics card – I can even move at Hair Fair now!) I found I could solve the problem by tp’ing home, camming around a bit, then going back. But this doesn’t make sense if the lag was caused by LOADING the textures initially. It seems like getting rid of the textures allowed me to move freely and having them loaded was what caused the lockup. Do you know why?

  • Jenny Thielt

    @Vikarti Anatra

    Actually now its 2 SIMS per CPU and you can see that info on the SL wiki land page

    http://wiki.secondlife.com/wiki/Land

    “Full Region

    There are two full regions per server host CPU. A region can hold up to 100 avatars and 15,000 prims. ”

    That means they kept getting faster servers and at the same time, increasing the number of regions per server, and I think that’s why they still didn’t release a new class of SIM since ’06

  • Thanks, Jenny, for keeping this up to date! Wow, things certainly change after a year and a month…

  • Dahlia Trimble

    I suspect a lot of lag may be due to comunication bottlenecks and mesage management on the server. If I teleport to a region near a populated event and the people in that area are each wearing a few hundred prims which are textured with many high rez textures, then the server has to send me all of those prims and textures that I should be able to see in my viewer. It needs to compute which I should be expected to see and which I should not see, and LL calls this “interest list” management. These messages about prims and textures add to the burdon the sim must process, and it could slow the messages about a new model changing clothes and another model turning on the catwalk or displaying a new particle effect at some important time. Likewise if people in the audience are chatting, the sim has to compute who needs to receive each chat message and which listener scripts in which attachments need to process each chat message. Having more scripts, especially with listening events, means even more work the sim must do to handle each chat message, all at the same time that others are teleporting in (or trying to teleport in) and when each audience menber is downloading the recently baked avatar textures as the models change clothes offstage. Lets add some new animations… I have a new ao and custom sit animations, if I sit in the audience and I decide that I want to use a new sit animation that nobody has seen before, then everyone within range has to download it. If my AO were turned off, they wouldnt.

    I think in general LL has made a lot of progress on the lag forefront, and while I usually agree with Gwen’s opinions, I think this article does present some interesting factors about lag but in this case may be incorrect or too simplistic.

  • Jenny Thielt

    And the “attachment nazis” are still around.

  • Dahlia, yes, indeed, avatars are a major source of lag, and always will be 😉 That’s one thing we won’t ever be able to change…

    Good point about in-world chat and old-style attachments with open listeners: these will definitely place a burden on the sim. So a silent audience is a less laggy one 🙂 The difference might not be huge, but yes, there is still a difference.

    You’re also right about the animations, of course. In a 100-person sim, one person sitting down will force everybody else to download about 160 KB (I think that’s the worst-case scenario) for that anim. Add a few more poses and it gets only worse. However, here is the good news: if people are allowed to sit instead of standing, they will just force one download to everybody; while if they’re standing, they will have at least usually 5 poses 😉 Also, turning off an AO after teleporting in will do little difference (except for avatars arriving after you): if someone wishes to follow the “attachment nazis” rules, they ought to turn their AO offs before teleporting… or it won’t make any difference whatsoever.

    In the year and so since this post, Mono has been introduced, which will run much faster (and lighter on the sim server), even more aggressive culling techniques have been employed, and a lot of improvement has been made in the way prims are fetched from the server, so the “myths” are even more “mythological” than before.

    Then again, the point in my article was mostly to explain that there is server-side lag and client-side lag, and that what makes us to suffer most is client-side lag, specially if we cannot do much to reduce it.

    Still, these remain true:

    – avatars themselves are the major source of lag, even without attachments (both client-side and server-side)
    – bad design (ie. too many and too big textures) on the environment are a secondary source of lag, but nevertheless quite an important one (mostly client-side though; the server will only start to get very very laggy if a lot of the environment is constantly moving or is set to physical)
    – using badly scripted devices (like, for instance, non-Mono scripts; devices with open listeners; devices with short “polling” cycles, of which the major culprit is sadly the animation overrider, but if you’re sitting down, this shouldn’t make a huge difference; devices with an abuse of sensors like avatar radars; etc.) will cause some overload on the sim servers, but it will bear little influence on client-side lag: turning them off will not decrease client-side lag at all, although it might have a small influence on server-side lag.

  • Thanks for the explanation. Now, only to think of the way to implant this into minds of attachment nazis.

    BTW,

    And where do all the textures come from? You guessed it — from the sim server. Which has to send those 300 MBytes to 100 avatars, or 30 GBytes total. So, on any busy event, we have the poor sim server struggling to push 30 GBytes downstream to all the unhappy and laggy avatars around. There is no way to avoid this step!

    Well, there is a way. It would require reworking of the server structure from the scratch but if textures are shared P2P between avatars on the sim, server load would be much lighter. Before IP guardians start screaming at me on the mere metioning that textures are stored locally, they are stored locally already. One can grab them anyway if want to. So it won’t hurt IP rights more then they are in risk now. But it would make the world running much faster.

  • @Dandellion wellllll yes and no 🙂 It really mostly depends on how large the residents’ own upstream pipe is. A typical example: throughout Europe, ADSL 24 Mbps usually has not more than 1 Mbps upstream. 100 avatars would at most provide 100 Mbps for P2P sharing. That’s about what LL’s sim servers ought to have anyway (possibly more, we don’t know). In practice things are not so easy to calculate, as each resident’s computer would have to open up to a hundred connections to each of the other residents in the same sim, to start texture sharing… and split their upstreaming pipe for all those uploads.

    However, it’s clear that this model would lower the burden on Linden Lab’s own servers and thus obviously reduce server-side lag to a degree. That would mostly mean that each resident logged to the sim would probably move around without any lag, while waiting for all the textures to rez properly (which would not depend on LL’s servers any more). I guess that would be an advantage, of course. You’d still need to download individually all those gigabytes of textures — just not from a single server.

  • true, there is a lot of factors in that equation. One more to add…. traffic between two users on the same (Earth) continent would be much faster than Europe-America for example.