New model for software development for Second Life?

Showing off
Using Firestorm

When Linden Lab released SL Viewer 3.0.3, I was dreading the worst. During the 3.0.0 “final release” cycle, I spent uncountable hours sending debugging reports for the crash-on-login bug which plagued certain hardware configurations. Linden Lab correctly tracked down the bug and fixed it; one developer mumbled something about “fast timers” and promised it would be released on the next development cycle.

But after seven years of dealing with Linden Lab, and in spite of o many changes at the development team, I was not completely at ease. While the development version I was using definitely fixed whatever bug was causing the crash-on-login, there was no guarantee that this “fix” would actually be part of whatever version LL would release next.

And, indeed, it wasn’t. SL 3.0.3 has precisely the same bug as 3.0.0 had, and added a new one: the inventory does not get sorted by “most recent date”. This is an intermittent bug: Linden Lab swears that the bug does not exist, that it’s actually caused by a racing condition happening when the SL Viewer starts with a clear cache and no objects are in the inventory cache, and thus sorting by “most recent date” becomes impossible — waiting for the inventory to fully download, according to LL, is supposed to fix the problem for the handful of users who ever complained about that. But, alas, that’s just theory —plausible theory, true, but one which does not hold in practice.

Similarly, the bug on the “fast timers” which caused the crash-on-login error pop up now and then. No matter how much LL promises to “fix” it, the truth is, 3.0.3 is unusable for me right now. And the worse thing is that the development version that did fix the bug and did have a working inventory was trashed and replaced by newer versions on all versions of LL’s viewers. Not a single one fixes the bug that had been fixed just after 3.0.0 was released. Linden Lab thinks the bug is fixed — they have internal reports saying so! — but it is not.

Now, you might never have experienced any of these two bugs. Be content! In fact, the major problem in software development, when hunting bugs, is to reproduce them. There are uncountable different hardware platforms running different software, and when all these interfere with each other, unless a developer happens to be very lucky, they might never be able to reproduce a bug, and thus, from their perspective of reality, the bug does not exist. Without having a clue why a bug seems to be fixed in one version, but pops up again in the next, even though all internal reports show otherwise, it’s truly very hard for the LL developers to actually know if they have fixed something for everybody or not.

So far, this is pretty much what you’d expect a Linden Lab employee to answer you if you complain about “bugs from the past” coming back to plague you after they have been somehow fixed. And if you have a background in programming, IT consultancy, or at least worked with software development companies before, you might be able to nod and agree that LL is pretty much telling the truth. And of course you’d be willing to help them finding out about the bug that plagues you and your friends (at the extreme, by eventually shipping them your hardware! Where can I buy stamps to ship something so big I wonder…). You’d be patiently helping them out, finding more ways to reproduce the bug, turning things on randomly, launching the viewer with a combination of other applications to see if a pattern emerges, and so forth. Everything to be able to figure out what it’s wrong.

But there is a catch here.

When none of the current LL Viewers — released, in Beta, in development, in Preview, or whatever LL calls their “channels” these days — have the crash-on-login bug fixed for me (I realise millions have no problems), until by chance some LL developer takes a moment to look at the logs and go “oops, this was fixed shortly after 3.0.0, why didn’t it make it to 3.0.3 even though the logs say otherwise?” This can take weeks, but usually takes years, until eventually the feature which has the bug is completely designed from scratch and thus the bug itself becomes obsolete…

So I had no choice but to try out a different viewer, and I’ve opted for the third-party viewer Firestorm, which had just released their first mesh beta. No, this is not how Firestorm is the best thing since sliced bread and how I’ve hopelessly fallen in love with it and want to have babies from the head developer 🙂 You can read that kind of story elsewhere 🙂 I still have my misgivings about the team behind Phoenix/Firestorm — people’s memories are fickle and fade quickly, but I still remember what former members of the development team have managed to do with previous incarnations of their viewer code. We’ve been given guarantees that the current team is honest (they even include some ex-Linden developers!), I know a few of them personally (or was it just one who left…? See, my memory is fickle and fades too!), and, in general, no drama has been vented by them. Considering that the computer where I use Firestorm is pretty much empty of any valuable document, and that Macs are harder to infect with Trojans anyway (or at least a bit easier to detect), I gave it a try.

In spite of being “based upon Linden Lab’s own code”, Firestorm, like most third-party viewers, has absolutely no weird Linden Lab bugs. Of course it has its own share of bugs; of course it crashes very occasionally; but, in general, that’s one thing in common with most third-party viewers: they’re rock-solid and always work as expected (yes, inventory sorts correctly by “most recent date”, too). Sometimes they’re faster than LL’s own code; sometimes they’re slower; but they’re always more solid and predictable.

I might wonder why. After all, 99% of the code or thereabouts is pretty much the same (the Imprudence team is the only one worried about complex licensing terms and thus have rewritten most, if not all, libraries with conflicting licenses; that’s why Imprudence, in general, shows less features and less advancements than other viewers, at the exchange of having cleanly licensed code; other TPV teams don’t care about that at all). Sure, there are a lot of fixes that all TPV developers apply to the code. It’s like getting a list of “here are all the bugs that Linden Lab refuses, for some obscure reason, to apply to their own viewers; let’s patch the code first and work from there”. Note that this list is ancient — some bits of it date to the first TPV developed by Nicholaz Beresford. Since those patches are not “secret” but fully open and published, one might wonder why LL never bothered to patch their own viewers; if they’re worried about licensing (they should!), they can get Imprudence’s versions of those patches, which are guaranteed to be “license-clean”. But no, LL never bothered to apply these patches contributed by the community, with very small exceptions.

So the irony, for me, is how this development process works. LL launches a viewer. It has bugs. The bugs are quickly fixed by a TPV developer, and on the next cycle for all TPVs, all get fixed. LL stubbornly refuses to admit the bug exists. After several weeks, months, or years, LL finally admits there might be a bug after all, and releases its own fix. It works for a while. Sometimes, LL’s fix is actually very good code — and all TPVs implement LL’s fix instead, if it proves out to be better. And then LL launches a new version, where miraculously the fix disappears and the bug is evident again. Why? We have been asking this question since at least 2006, but I’m sure this has been part of LL’s hallucinating development policies for way longer.

In some cases, the bug is merely an “annoyance”, and people like me who are very faithful to LL will patiently endure the long period of time until a release comes out again which fixes what was wrong (inventory sorting, for instance, gets broken every odd version, so residents relying upon LL’s viewers just need to patiently wait for an even version again. It’s not critical — just very, very annoying). But when LL’s viewers completely fail to allow us to even log back in to the grid, well, then we have no choice. Or rather, we’re very lucky to have a lot of choices among TPVs, all of which work fine.

Well, bugs are one side of the whole story, of course; the other are features. LL will never add a feature from a TPV back in their main code — they are still trying to convince good, independent developers to contribute code back to Snowstorm by signing up a development agreement under LL’s own terms. Snowstorm is like the little research lab where LL’s developers and some crazy volunteers are able to test a few new clever ideas with the Official LL Stamp of Approval. Snowstorm is developed in a very different way than anything else, either at the Lab or among the TPV community of developers, and tends to pretty much ignore the JIRA’s bug reports and feature requests, but the team has their own method for prioritising work. I’m not saying it’s “better” or “worse”, it’s just… different. Unfortunately, from my perspective, Snowstorm is a bit too conservative for my tastes — it includes none of the radical features seen on most TPVs.

Let’s take mesh as an example. It took eons to implement, but most of the work was done by Karl Stiefvater, formerly known as Qarl Linden, and Pastrami Linden (with the integration work into SL Viewer 2.X lead mostly by Esbee Linden. All left LL; Esbee came back again to lead the Snowstorm team; but Qarl remains enthusiastic about SL. In fact, his feverish dreams about SL and its potential never died; he really conveys us the idea that he very reluctantly left LL — not by his choice! — because he’s amazed about what SL can actually do, and had very good ideas on what it could become.

When mesh was finally released, we were all drooling… for a bit. Then all builders suddenly realised that the algorithm calculating Prim Equivalents would mean that almost all meshed models would invariably take much more prim space on parcels than alternatives built with regular prims and sculpties. Sure, as a demo of the kind of thing possible to be done now in SL, meshes are fun. But as a replacement for a prim-based grid, it’s economically unsound: people want lower tier, and more items per parcel, not less. This, of course, was never a problem on the Beta grid, which had volunteers testing mesh for almost a whole year — the economics were simply never put into practice. Now that they are, well, it means that the dream of having a glorified, meshed SL was just that — a dream. LL snuffed the candle of hope for a meshed future, and were quite adamant in saying that they wouldn’t even dream of reconsidering changing the algorithm for Prim Equivalence — if at all, they would start applying it to sculpties and tortured prims in the future as well. I dread the day they do this.

So where did mesh developers go with all their content? To OpenSim, of course. OpenSim supported meshes 24 hours after LL started their open tests on the Beta grid. Although the mesh format has changed a lot of times, as well as the way to upload them, OpenSim 0.7.2 now fully supports the latest implementation; and, of course, OpenSim grids can safely ignore prim equivalence constraints or even prim limits. It’s sad to see so much development put by LL into meshes just to see how SL users will never truly benefit from it — while seeing LL hand out the marvels of mesh to the OpenSim grid. Somehow this feels completely wrong!

Well, I originally thought that meshed buildings wouldn’t be the biggest issue anyway. The largest market in the SL economy is, after all, fashion — avatar attire and attachments — and not really buildings and furniture (even though these have a fair share of the market as well). So I might feel sad for the mesh-less builders, but glad that at least the fashion designers had some way to make beautiful, lovely avatars.

Then I had to face the reality: Linden Lab broke the mesh implementation regarding attachment points. So those lovely meshed dresses, who actually “bend” well when you sit down (no more through-the-chair skirts!), actually don’t follow the avatar’s body very correctly, due to some stupid bugs. The workaround is to wear a transparent skin which will selectively “delete” parts of your body so that you can only see the mesh. Obviously this works, but it also means that your avatar shape, which you might have carefully crafted, or dearly paid for, will be useless — as well as layered clothes. They will all disappear in order to allow the meshed clothes to be properly visible. The result is that meshed clothes are a novelty, but not something to be seriously used for now. Meshed avatars — animals, robots, and so forth — are the only ones that might be workable with the current implementation.

Most people naturally expected LL to fix what was wrong. After all, it’s been a huge development for over two years or so, and one would expect that LL would like to see some return from that investment — namely, a new fashion revolution as designers started to replace “old” clothes by the new generation of meshed clothes, thus pushing consumers to start buying a lot more content to replace their “old” one (like we all dumped our prim-based shoes with sculptied ones, not so long ago) — a boost to the economy, more L$ sales, more income to LL from those L$ sales, and so forth. So LL had a strong incentive — money talks! — to fix what they did wrongly.

Our old friend Qarl took a good look at the code, and quickly figured out what was needed. It was not overly hard to implement.

What did LL do?

They announced that the “Mesh Project” was finished and closed, and no further development would be made. Well, to be honest, they did say that they would “someday” think about ways to add further tools to get meshes properly working on individual avatars. But for now… they’re all happily developing their 2D Flash-based game for Facebook, Android, and the iPhone. Oh, I’m just speculating, nobody knows what they’re really doing; we only know that Rod Humble did announce “a new product” outside the 3D virtual world of Second Life. It’s not hard to predict that they will follow the Electronic Sheep Company and 3Di lead, both prominent metaverse developers, which are just doing Flash-based 2D “worlds” embedded in web pages, mobile phones, and Facebook. Linden Lab is very likely to start doing the same; after al, they have a bigger brand name than either ESC or 3Di and a bigger development team. So long as everybody is pulled out of any “secondary projects” — like fixing mesh! — and start implementing Rod Humble’s new toy.

Now so far that’s not very interesting (just an expected development). What was puzzling for me was to see, in a relatively short tim frame, a new trend emerging. Frustrated by LL’s reaction to an “unfinished” and “broken” feature, the community turned to Qarl to help them. But we have to understand that Qarl is a professional developer; he doesn’t sit in a cave paid by his parents or a government stipend to spew out lines of code for free. He’s a top programmer with an impressive background of former clients. So while he loves SL in spite of everything, he’s quite aware that he cannot stop all his other paid projects to get some free time to develop for free — specially knowing that LL will never implement the code he develops on LL’s own viewer (you do remember his “prim alignment” builder tool that he released some months back? It’s only implemented on TPVs).

So what did the community do? They agreed to raise funds to pay Qarl for his work. If you believe this is a good idea (I certainly do!), you can join the fundraising on IndieGoGo.

But not many days after this incredibly new development was announced, another surprising story emerged. KirstenLee Cinquetti, who has been developing Kirstens Viewer for a long time — the first TPV running the SL 2.0 code — and who has always been looked as “the artist’s TPV” because KirstenLee’s work has been mostly on getting the renderer working better, and showing even more fantastic things than LL’s own renderer somehow doesn’t… because, as KirstenLee found out, a lot of code in the viewer comes from abandoned projects, things started but never finished. So, well, KirstenLee just went ahead and finished all those semi-functional projects, added a few of her own, designed a new interface, and so forth. It’s probably not the fastest viewer out there, but among the artistic community (including photographers and machinimists), it was considered to be the one creating the most beautiful images out of SL. For a long time, for example, it was the only viewer that got me reasonably good-looking shadows (SL 3.X considers my card too old for shadows and doesn’t implement any code for it).

The person behind KirstenLee Cinquetti sadly had to take care of her wife, who had to stop working due to illness, and that meant shutting down the project to find time to get a new job. The community’s reaction was unprecedented, and after a few days, KirstenLee wondered if there was a way to turn the viewer into the main income source of the couple… by raising funds on CrowdFunder. It’s a far more expensive project than Qarl’s, but it’s a long-term commitment (and if the limit is not reach, CrowdFunder will return your money, so it’s a safe funding experience).

Now this is definitely an odd turn of events. In the past, we had to beg LL to fix bugs and add features, and this was hard to accomplish — impossible before JIRA, but pretty much ignored with JIRA (there are simply too many bugs, too many feature requests, and too few votes on each category). Then LL released the viewer code as an open source project, and third parties started to fix the code and add the features on their own — but rarely, if ever, used the contributed code themselves. This mostly meant that some marginal changes — and sometimes important bugs! — got fixed, yes, but only on a non-LL viewer. For some, this is actually an advantage. But the problem is that LL has a whole team of professional developers, paid to create code, while TPVs are small groups of volunteers contributing their time. Some actually have other means of surviving in order to be able to be somewhat free to develop code for the community; but, in general, the better the developer, the more likely they are to get jobs elsewhere and leave them with little time left for open source projects. This is normal in the whole open source community, and that’s why, after so many years, we still haven’t got a fully separate implementation of a viewer that doesn’t share a single line of code with LL’s own viewer (although Radegast is probably getting there) and implements all functionality — the task is simply overwhelming.

So what’s the alternative? If LL is not willing to hire more people to fix the code or add features to it, so, well, the community seems to be willing to do just that. The irony is that Qarl did work for LL, and I heard unconfirmed rumours that KirstenLee was also hired by the ‘Lab at some point (but which I won’t confirm). Now they’re doing — or will be doing, if enough funds are raised — pretty much the same, but their employer will be the community, not LL.

If the trend catches on, this might mean a rather interesting change for Second Life: one where instead of “begging” or “pleading” LL to do what is needed, we — the community — simply hire people to do the work we want them to do. No silly JIRAs or similar mechanisms that will only dilute people’s opinions and never have any real leverage with the Lab. Instead, we put the money where our mouth is — we want this feature, we want this bug fixed, if LL doesn’t do it, we’ll pay someone who does.

Now I have no idea if this is something that ever happened before: users paying developers to fix a product from a company for them. Sure, there have been all sorts of companies offering “fixes” and “improvements” for Windows or Mac OS (before Mac OS X at least), and those “improvements” would come at a small fee. But that was different: that was just an opportunity for business — creating tools improving a product, and finding people willing to pay for those tools. This is slightly different: it’s the community claiming for a certain feature/improvement/bug fix and hiring their own developers to implement the changes — and releasing the code as open source so that everybody — even Linden Lab, if they ever wished to do so — can benefit from the code.

You can think of it as “money-driven open source development”. And as some of you might imagine, I’m all for it!

On a completely different environment, this kind of development is not unheard of. For instance, many developers create amazing plugins for WordPress, which are funded by companies or organisations which need “just that” change. Later the plugin gets released as open source, because, well, its development had already been funded — many plugins, which wouldn’t otherwise ever be developed, saw the light of the day that way. Still, this is not quite the same thing: usually there is one company or one organisation willing to pay for the plugin development for their site, and then just don’t bother if the plugin gets universally available for free or not. This is rather common.

Even in the Second Life/OpenSim environment, that kind of approach already existed: viewer developers were routinely hired to create certain extra features on behalf of a company which needed a specially-tailored version of the SL viewer for their own purposes. Because of LL’s licensing policies — GPL — this meant that the code had to be released publicly afterwards. A few OpenSim grids are only accessible through a specially modified version of the SL Viewer, but if there are some extra nifty features hidden on that version, the whole community can just grab the bits and use them for their own purposes instead. But, again, this is different — there is a company hiring the developers to do the original work.

This new trend is far more interesting. There is no “company” behind the fundraising for Qarl’s or KirstenLee’s work. There are just unhappy SL residents — unhappy because LL keeps ignoring them. But these residents are used to pay to get high-quality goods and services — that’s what they get inside SL every day. Paying US$10 to get a feature you need is more than acceptable for someone used to spend that in one or two new pairs of virtual boots. So there is already a community willing to spend money in getting quality service. This is something that should embarrass Linden Lab and make them feel uncomfortable: we’re so loyal, so faithful to Second Life, that we’re willing to spend some more money to fix things that Linden Lab doesn’t want to fix on their own — and give the fix back to LL as well (even though they won’t implement it, of course).

What’s so wrong with this picture? We, as residents, pay LL to provide us a service (I’m not complaining that SL is “too expensive” — it might be for some, it’s dirt cheap for others, there is no way to please everybody). LL hires developers to work (most of them full time) to provide services for us. But the services we get are not making us happier — instead, there seemed always to be a cleft between what LL thinks that we need and what we think we need. Of course often LL picks the right option, but, more often than not, they simply don’t — they have a “Steve Jobs” kind of stubbornness which makes them totally ignore what their customers want or what the competition is doing. Steve Jobs managed to pull that off and turn Apple into a giant; I’m still doubting that anyone at LL can do the same.

So, in the mean time, we’re paying for developers out of our own pockets to get us the features and bug fixes we need. But how wrong can that be? What a waste of money and resources! If we already pay LL to provide a service, why should we pay extra for external developers to fix what LL refuses to fix?

On the other hand, we might just have given Rod Humble a new business model. Instead of JIRAs, feature requests, “stories”, or whatever they might invent to gather customer support… let people pay for features. Just set up a site with things like: “Fixing chat lag — 6 months, US$50k” and let people contribute towards the total 🙂 That way, LL not only gets the faetures we want, but they get paid for it as well — no more silly wasting money and resources on things that, once deployed, might get little support.

Granted, it’s only fair that they slash the costs of Second Life under that model. After all, LL might still require co-location facilities, infrastructure engineers, and technical support — but they wouldn’t need to pay the development costs under this model. I’m sure that would mean lower tier costs overall 🙂

The only problem with that model is that it’s unfair: some residents might be willing to pay for development, but most wouldn’t, and would just be willing to wait for others to pay for the features that all benefit. Nevertheless, this didn’t prevent both Qarl and KirstenLee to do exactly that.

Who knows where this trend will ultimately lead?

Single-page article on demand. If you wish to have multiple pages again, please leave comments saying so 🙂 The majority wins!