Half the Grid is Gone!

Here’s something I haven’t seen in many, many years. Yesterday, with Hair Fair hitting the grid as one of the most desired places to visit, and hundreds of exhibitors working hard to get their builds finished for SL6B, the grid suddenly started to misbehave, in a quite serious way (note: probably there is no correlation): logins failing, unusual high sim time dilation, and textures utterly failing to rez, to say the least.

Regions started to go down on their own and not get up automatically. This forced Linden Lab to get their teams working hard to restore around 8000 failed regions, with low rates of success. 6000 were brought up again, until only 2000 remained to get restored, and then another 6000 failed once more… and so on.

The result? Well, it was as bad as jumping to a brand new sim, with only a hundred prims in it, completely empty except for myself, and still watch time dilation floating around 0.4-0.7, and sculpty textures totally unable to rezz even after half an hour of patient waiting.

You can imagine how things looked back at Hair Fair, then — which was totally crammed full with avatars hopelessly trying to see anything. The usual Lag & ARC Nazis blamed everyone of bringing in their primmy hair and contributing to lag, while completely failing to understand that things yesterday were simply extraordinarily bad. Things were so bad, in fact, that I believe we had the lowest simultaneous number of avatars on the grid in the past year — since people were not only unable to log in, but they also quickly dropped out when it was clear that nothing would rezz, teleports would fail, and Group IM Chat was way more erratic than usually.

Linden Lab thus decided to bring in their emergency response team to deal with the issue. In an unusual feat of boldness, a decision was made to shut half the grid down at 11 PM yesterday. Now oldbies might remember the days when the grid, for “maintenance work”, was shut down several hours in a row — sometimes even taking days to get opened again. These days have been gone since Linden Lab introduced the Het-Grid — pretty much two years ago (the content of the original post is missing) — we now get “rolling upgrades” instead of a full grid shutdown. So an announcement that Linden Lab would shut down half the grid was quite unexpected!

And it’s definitely more serious that even they thought. After several hours have passed, at the time of writing this, a lot of regions are still down. It’s not been a nice weekend for most of us.

A big hug to the Linden Emergency Response Team as they delve deep in the code and probably in the dark corners of one of their main co-location facilities and work hard to get everything working again as it should!