Wrong Answer, Mr M Linden

For some time now we’ve been sort of expecting with anticipation what M Linden’s style of management would be. One thing was sure: the old Tao of Linden (where developers would be pretty much allowed to do what they wished) would slowly be replaced by a goal-oriented strategy (in this case: “more stability” and “a better ‘first hour’ experience for new residents”), and the focus would switch from “making residents happy” to “make companies happy”. We’ve seen both happening in the past few months. It’ll be the work of several years, though, not just a few months.

But how would M Linden react to crisis? We’ve finally figured it out. And, sadly for all of us, the answer to a crisis was nothing we expected.

The Case of the Overused Openspace Sims

Becoming detectivesque in my approach, I picked up my Sherlock Holmes magnifying glass and took a critical eye to the whole openspace sim fiasco. Openspace sims, once called void sims, was an experiment in the early days of SL.

As you can see in the map to the right, these were “filler” sims to allow people to sail across the “inner sea” of the old mainland continents. Linden Lab had just upgraded their infrastructure to a newer generation of servers (I don’t remember if they had just upgraded to Class 3 servers) but they had a lot of old servers lying around.

Why didn’t they simply throw the old, obsolete servers away? Well, very likely — but I’m just speculating — these servers were bought under a long-term agreement. Probably a 24-month or 36-month lease, computers and bandwidth included, and disconnecting the servers would not get LL a refund from their co-location providers. So it would be better to continue to use them, instead of disconnecting them and keep paying the monthly fees. A few of those migrated to the Beta Preview Grid (Aditi), some to the many internal grids, a few others became webservers for tests, but still… a few could be used for low-traffic sims.

At that time, sailing in SL was a huge boom (it is still fashionable; probably there are more people these days who are fans of sailing, but SL is so huge with so much variety that it “feels” the fad has gone. It has not), but you could only manage to sail by the coast, which — thanks to terraforming and huge over-the-ocean platforms and houses — became more and more complicated. “Void sims”, where there are no prims and just a handful of Linden trees, was an interesting solution. It could be used by a handful of sailing enthusiasts at the time (regattas seldom had a huge participation, and when spread across a large area of the “inner sea”, it would mean less than 20-30 ships in each of the “ocean sims”).

They had little practical use. Some very few allowed you to build on them (with an auto-return of several hours), very likely by mistake — I used to go there as an alternative to the Linden Sandboxes. Once I met with a group which was quite amusing. They landed on boats, sported uniforms and weapons, and their leader said something like: “Ok, now we split in two teams, Red Team remains here at this beach, the Blue Team will go to the other side. We’ll work towards the middle and see who captures it first.” For about an hour, I amused myself watching the tiny battle unfold, as the two teams had some fun shooting at each other, until finally a group won. It took less than an hour; they peacefully retreated to their boats and went away. I just smiled. It was one of those little experiences in SL that happen all the time and will be part of my memories.

Later, void sims were also used for Linden forests (not that there are so many), and were pretty much unused, except for romantic walks under the moonlight.

For Linden Lab this just meant a way to put those long-term leases of obsolete equipment to work.

Then at some stage the big landowners (the ones with their mini-continents) started to attract the sailing fans, and they came to LL to ask if they could use void sims too. Why not? After all, it couldn’t hurt anybody. Linden Lab cleverly demanded that they buy 4 sims at a time (thus using a full server) and that you could only get void sims if you already owned a “regular sim”. This worked well for LL to keep using those obsolete machines, and now earn a small income from them — they would in fact earn the same from those 4 sims, using old hardware, than from a full, regular sim on new hardware. It was good business.

Technology improved, and we moved into Class 4 sims, then Class 5 ones, and soon we’ll migrate to Class 6. Now I don’t have my tables around me, but at some stage, LL started to buy quad-core servers (I can’t remember when Class 4 servers were introduced). What this mostly means is that, at some point, the new servers could handle four regions at the same time, while the very old ones (Class 1 and 2) could only use one or two (Class 3).

This is very important to understand. LL managed to increase the density of their servers: this means that they could support the whole grid with less servers. Granted, bandwidth costs very likely outweight the cost of individual servers, but the difference will still be a cost reduction to maintain the infrastructure (if the Class 6 servers, as I suspect, have 16 cores, LL will even cut more costs that way, as a single Class 6 server would be able to run 16 regular regions or 256 openspace sims!).

So what happened next is that the “old” single-core or dual-core servers suddenly became twice more obsolete: not only would they be old hardware, but they would only run one or two sims at the time — or, obviously, four or eight void sims. But LL would still have to keep them until their lease (or long-term agreement) expired.

What to do?

Clearly, even for someone with little business sense, the option was to start selling those old servers aggressively, but making sure that these are “low quality” products, not fit for regular use. The “openspace sims” were born. And the strategy was such a huge success that in March 2008 allowed them to be bought in isolation, increased the number of prims to 3750 (what a regular quarter-sim is supposed to have), and allowed parcels to be sold and announced on the available parcel list (i.e. on the map).

The conditions for a stratospheric growth were set.

About Gwyneth Llewelyn

I'm just a virtual girl in a virtual world...

  • Ananda Sandgrain

    Oh, I don’t know Gwyneth… I think you did find a reason while you were writing this. Namely, that Linden Lab is more interested in development than in being the biggest hosting provider in the world.

    As you mentioned, after introducing the OS product they basically saw the number of sims (if not servers) they needed to manage double in less than a year. That’s a crazy amount of growth to try and keep up with. I’m betting they’d rather just try to improve (and figure out how to save money on) a more sedately growing world. If the world grows at a slower more manageable pace it buys more time for things like developing that licenseable server product, making grid interoperability happen, consolidate onto more efficient server setups, etc. One way to reign in crazy growth is to raise prices. Just my speculation.

  • Prokofy Neva

    There is a widespread belief that all throughout the mainland, the Linden used, and are using, void sims. They aren’t. They are using full-prim sims. Go and look. They have always used them, with lots of prims left over. In fact, “void sims” i.e. actually in voids and not for sale are all over, and not the low-prim kind, but the full-prim kind. Perhaps they have or will start swapping those out, but I see many still full-prim, as they always have been.

    You’re not watching what the land barons are actually doing. Anshe Chung is frantically hustling 4-packs of OS sims back into islands at the grandfather tier of $195 and selling them, and people are buying them thinking they have that lower price for a year. She is selling them outright, so that people get that tier of $195, not pay her on top of that tier.

    Others are giving or selling their OS sims to those who can pay. And if you brush aside all this agida, you see that a bargain is to be had: instead of a $1000 sim, you get a $375 sim, and you pay $125, instead of $295. 3750 prims, to be sure, doesn’t seem like much, but for a house and furniture and a boat and dock, it’s plenty. Put in some sculpty rocks and a waterfall, you’re done. Far from losing a million, they are just making sure this product goes into the hands of those who can really pay it, and not default on 100 of them at once because they are clueless newb landlords unable to cope.

    Philip long dreamed of flat tier, tier by the square meter, without the jumps. That would kill off rentals quick, and maybe that’s the plan, it sure looks like it, though M’s letter seems to speak of rentals still in benign tones.

    Your theory of the aging out of servers as the single greatest driving factor of the land market is probably true, but you don’t explain why they suddenly could “pass on a big price break” all in one swoop, and then suddenly “have to pass on big usage costs” in another big swoop. Well, which is it?

  • Damanios

    On the subject of offering a service in the ‘sweet spot’ 10-30US range, people tend to forget that LL has been offering services in this range for years. It’s the staggered tier model, which was the basis for land sales in the first place.

    All LL decisions of the last year (except for the openspaces), basically point to reaffirming this sales model again. (Cleaning up adfarmers, hiring a public works team, creating themed environments, excessive marketing of areas like Nautilus etc.)

    The open spaces are basically ruining the tier sales model though. (Why go through the hassle of mainland/tier if you get a full configurable area for 75US.) So the only way to make the model viable again is to get rid of these. Which is exactly what LL did. The fact that areas in Nautilus sold on auction with a huge profit is probably one of the main reasons for LL to take this strategic decision.

    From a strategic business perspective it seems like a reasonable logical step. Consolidate to one sales model, fix the issues, and then roll out enough server space within this new sales model.

    Sadly, for LL, you can’t put back the genie in the bottle… I’m afraid it won’t turn out the way they expect it to turn out…

  • I am not sure you have the core issue right. From my discussions with various techie types there is no specific coding that would allow for a direct mapping of a region to a core. A CPU in fact does not get arithmetically more powerful as there are more cores–ie a quad core is not 2x as powerful as a dual core. So it is unlikely that a quad-core, dual cpu machine would effectively handle 8 regions under the same code that is running 2 regions on a dual-core, dual-CPU class 5.

    In fact it would seem odd to map a Region to a core when in fact the usage on Regions is not flat, but more bursty depending on load. For example–why leave a core processing unit idle when there is no contention for processing. So if everyone is at a party on part of the CPU, then why reserve processing for empty “habitations”.

    In fact Linden has envisioned a more dynamic virtualization for Regions, but as usual started it only and not carried it along. http://wiki.secondlife.com/wiki/AWG_Virtualization.

    I am pretty sure (but actually don’t know 100%) that the Region code hits the CPU and the core is an internal CPU tool that increases parallelism of processing and pushes more work through the CPU. The whole reason that Intel started the core technology was that it could not increase CPU speed cost-effectively and parallelism was next best thing.

    Also on the LL website they say that a Class 5 is a dual-core, dual-CPU and they run two regions = 1 region per CPU.

    I would also doubt that even LL is dumb enough to lease a server for more than 12 months at a time. I think running 10,000 and more servers already puts them in a good enough negotiating position. I would see it more likley that they are rotating the older machines into grid services rather than hosting even voids.

  • “So, in essence, the Premium account, as a money-making service, is not a success, and no wonder that LL is planning to get rid of it altogether.”

    Based on?

    -ls/cm

  • Gwyn: I really appreciate your detailed analysis here. Much to think about. And I think I do now understand why LL did what they did: They had been undermining their other products too dramatically. Simple as that. This had NOTHING to due with resource management and increased usage of the opensims at all. ( …and how could it, as the prims and usage numbers are a zero sum game?)

    On the mainland, openspace sims stopped production of new mainland sims cold. This happened due to the profit model of renting mainland at the same price became far less attractive than opensim rentals. (Mainland land prices as a consequence also dropped as low as 2.4 L per m2. A BIG reflection of no demand…)

    On the estates, openspace sims undermined the previous market model of renting space completely.

    So,Linden Lab’s decision here I think was NOT for the short run, but intended as a means for rescuing the long term model of their future sim sales.

    Let’s take Mainland for example here. A single buyer could buy 16 K of land and 3750 prims at $75 per month tier OR get 4 times the space and the same prims with water, no crappy neighbors and estate priveleges. Which one are you going to choose? No brainer here. Hence the end of Mainland new sim sales. (What we do see here by the way is the intense demand at that $75 price point, especially with an attractive product.)

    Now let’s look at Estate Land. Same prims per price and 4 times the area? Of course this totally undermines estate rental as well. People don’t feel crowded and it is a MUCH more desirable product. I’m sure LL took a hit on its increase in Estate sales.

    This is why they knew something had to be done. They didn’t like the future: the future in which SL would become mostly all opensims! The sheer speed of adoptions showed them this was coming.

    So, once the decision is made, LL’s problem now becomes:how do they “sell” this news to the thousands of people who are about to get screwed? That is what all the confusing nonsensical gobbledy gook about how they had been bad and were getting what they deserved. Total rationalization and excuses for what they KNEW would be a firestorm of protest. Less said the better they are thinking, weather the storm and all will blow over eventually. That is their strategy.

    It is simple to understand where they are coming from here when looked at this way I am thinking.

    The new ‘Homestead’ pricing at $125 makes people have to choose between “prims per dollar value” of the mainland, and the advantages of water and space of the Homesteads. OR They must now choose between the lower prims per dollar and more space of the ‘Homesteads’ versus the more prims per dollar and smaller space of the Estates.

    All in all I think LL is banking on the fact that most opensim SLers are not going to leave SL because the alternatives are not yet up to snuff.

    As an aside, I already have two friends who have bit the bullet and upgraded to full estate sims from openspace sims. I am sure they are not alone.

    Have a good day all!
    Scarp Godenot

  • The whole Openspace mess is a catastrophic and overwhelmingly obvious planning error that could easily have been avoided, but is now likely to wreak immense and permanent damage on the entire economic infrastructure of SecondLife.

    Linden Lab have, in effect, drastically undercut themselves with what was supposed to have been a low-grade product used to fill in with a little greenery, but with what residents soon discovered was nearly as good as an ordinary sim. Months passed as the number of Openspace sims, almost all of which were used as substitutes for full-prim sims (and blindingly obviously so) burgeoned in unprecedented numbers. Vast numbers of residents spent huge amounts of time and effort setting themselves up to work with the new Opensim economy that Linden Lab had created.

    Eventually realising, well over half a year after the fact, that Openspace sims were not just being used for forests and sailing and wilderness, and that their margins had been drastically compromised by, in effect, selling access to their basic resources at a substantially lower margin than they had previously been doing, they decided to reverse the policy. The consequences of their original decision to unbundle the voids was overwhelmingly obvious at the time, and would have been even more so as the numbers of their sales increased beyond all expectations in the first month or two. If they had not unbundled them in the first place, or if they had realised the position within the first month or two after the unbundling, then they could quite readily have introduced a prospective price increase, whilst grandfathering the existing (relatively few, by October 2008 standards) Openspaces. Some people would have been unhappy, but massive upheaval would have been avoided.

    The position now is different. The margins might already have been too severely compromised to allow grandfathering. Profitability might well be at stake (there is no use in having increased revenue if the costs increase more than the revenue: in that case, it is more profitable to have less revenue and much less cost), even the financial viability of Linden Lab, especially since credit is now expensive to obtain. So, Linden Lab drastically change their pricing structures, and, in so doing, cause disruption on a scale never before known in SecondLife.

    Untold thousands of people had put immense amounts of time and effort over the course of many months into projects which depended for their viability on the Openspace pricing structure. Linden Lab’s abject failure to comprehend the blindingly obvious until far too late will destroy a large proportion of that work. A large number of people will leave never to return because of that destruction, even if they would have been perfectly happy to stay if they had never been lead to believe that their project was viable. People are entitled to expect stability, and expect that a project that appeared viable in the conditions in which it was started to remain viable for the long-term. People who are given to believe that a project is viable, and who, on that basis, invest immeasurable time and effort into it will very likely never trust again the institution that deliberately made the project impossible to complete after it was started, having originally organised itself such as to allow for such projects.

    As a result of that, a very considerable number of people are likely to leave SecondLife, many of whom would have remained in SecondLife quite happily if there had never been such a thing as an Openspace sim in the first place. As Gwyneth correctly identifies, it is inconceivable that the claim on the Linden Lab ‘blog that the issue is either unexpected or the fault of residents could have been made in good faith. Linden Lab have always been manifestly incompetent: they are now fundamentally dishonest. A great many people who have trusted them hitherto will, quite rightly, never do so again. Many people who have not yet had any contact with Linden Lab will never trust them to begin with.

    Ultimately, such disruptive and dishonest behaviour is self-destructive. A virtual world can only expect to be economically viable – for any purpose – if it is populated with large numbers of people willing to engage in whatever counts as economic activity in that world. To focus on corporates at the expense of individuals is suicidal: the corporates are only likely to have any long-term interest in SecondLife as a platform so long as there are a large number of people using it as individuals. If they want private meeting spaces, away from the main grid, then they will have no long-term need to have any involvement with Linden Lab: OpenSim will be viable (and far cheaper) for such an application long before it will be viable to rival SecondLife as an integrated, generalised, social virtual world. If Linden Lab’s business model is in the future going to be selling connexions to the main grid, then it must continue to ensure that the main grid is worth connecting to. Making drastic and sudden changes that have the effect of destroying the hard work of a large proportion of the economically most active population is not consistent with that aim.

  • T C

    Just a couple of tidbits to chew on, don’t know if they help or hurt understanding what LL is up to.

    LL buys servers outright, it doesn’t lease. http://blog.secondlife.com/2006/11/09/concierge-townhall-transcript-2/ – search that page for Martin Mounier.

    There are still full island regions on class 4, though mainland was all bumped up to 5 in the last year or so. Voids have worked pretty much like full islands, with the older ones on 4 and the later on 5. Some older hardware does get repurposed, but not really on Agni.

    Old servers may seem almost “free” on the surface, but they have lower rack density, higher power consumption and potentially higher maintenance needs, no it’s not necessarily a bargain to hold onto them in numbers.

  • Mark C

    *** I am pretty sure (but actually don’t know 100%) that the Region code hits the CPU and the core is an internal CPU tool that increases parallelism of processing and pushes more work through the CPU. ***

    You’re right — you don’t know. Try as I might, I can’t even make sense of your first sentence, in fact.

    Cores are CPU’s. They’re just CPU’s that are on the same chip. They may share part of their cache, but other than that there is no difference between multi-core and multi-cpu.

    *** From my discussions with various techie types there is no specific coding that would allow for a direct mapping of a region to a core. ***

    Then you aren’t talking to the right “tech types.”