1 Day Free

[Sword_of_Doom]

Wow, Blizzard trotted out the big shot CEO, Mike Morhaime for damage control.

" Due to the increase in downtime caused by today’s hardware maintenance, we will be giving all players with characters on the realms listed below a one-day time extension. In addition, players on the realms below will also receive a rested experience bonus, whether or not they had left their characters logged out at an Inn.

We realize that this extended downtime is very inconvenient, but it is a necessary step for us to take in our efforts to increase performance and stability on these realms. For the last few weeks, these realms have experienced poor stability during peak times, which resulted in NPCs not spawning, lag, and frequent disconnects. We have determined that a large percentage of realms on the same backend infrastructure were operating at maximum capacity. This overpopulation caused problems with the service that ultimately led to the issues mentioned above.

Our first step in trying to alleviate the situation was to offer character transfers to reduce the amount of players on these high-population realms. The character transfers did reduce the total amount of players on the shared infrastructure, and we noticed an immediate performance increase. However, this increase only lasted a few days, as populations continued to rise.

Once it was shown that the character transfers did not sufficiently disperse the population, we began to modify the hardware backend for the affected realms. The most recent and substantial step of that process involved transferring the realm data to a new hardware setup. The goal of this procedure is to decrease the overall load on the infrastructure that the affected realms share.

The new hardware infrastructure was completed yesterday, and we are currently copying the character data from eight of these realms onto the new hardware. This transfer will significantly decrease the load on the current infrastructure, which should result in improved performance on all of the realms listed below. The data transfer is taking us longer than expected, but it is vital for providing stability to these realms.

We are totally committed to providing our players with the best play experience possible, and we will continue to streamline our code and add new hardware to our infrastructure until we accomplish that goal. We once again thank you for your patience during this time, and wish you continued good fortunes on your adventures in Azeroth!

Please note that this time extension discussed above is in addition to the two-day time extension announced yesterday that is being provided to players on all realms.

Sincerely,
Mike Morhaime
President, Blizzard Entertainment"

I love you Blizzard, you are always thinking of us your faithful, devoted customers.

[Skandranon]

You know, reading that section about new hardware, I think I feel a twinge of hope that the nightly crashes on Stormrage will end.

Or maybe that was heartburn. It was a twinge of something.

On an unrelated(?) note, I think Morhaime's credibility is going to start taking hits soon.

[Sword_of_Doom]

You know, reading that section about new hardware, I think I feel a twinge of hope that the nightly crashes on Stormrage will end.

Or maybe that was heartburn.  It was a twinge of something.

On an unrelated(?) note, I think Morhaime's credibility is going to start taking hits soon.
[right][snapback]72394[/snapback][/right]

You have hit the nail right on the head! Thats the exact same thing i was thinking. They have repeatedly promised after hardware changes (this has happened before, not to our realm as far as i know) that things would stabilize. It has not happened yet and your right, Mike Morhaime is putting his integrity and name on the line, i actually have some sympathy for him because i think his credibility will be shot in a couple of weeks, tops.

[Malakar]

I'm just glad my server was listed, and there's a chance this will make some improvement.

[Quark]

You have hit the nail right on the head!  Thats the exact same thing i was thinking.  They have repeatedly promised after hardware changes (this has happened before, not to our realm as far as i know) that things would stabilize.  It has not happened yet and your right, Mike Morhaime is putting his integrity and name on the line, i actually have some sympathy for him because i think his credibility will be shot in a couple of weeks, tops.
[right][snapback]72395[/snapback][/right]

Back in late January/early February they mentioned hardware updates to a few servers, Stormrage being one of them. Right after that was when Stormrage was stable as a rock. Then they put the game back on shelves, and everything went to the hounds again. On Feb 24, they promised hardware updates again for the "terrible 20" or so. Them adding hardware to these servers is only living up to what they said they would do a month ago ...

[Darian]

Back in late January/early February they mentioned hardware updates to a few servers, Stormrage being one of them.  Right after that was when Stormrage was stable as a rock.  Then they put the game back on shelves, and everything went to the hounds again.  On Feb 24, they promised hardware updates again for the "terrible 20" or so.  Them adding hardware to these servers is only living up to what they said they would do a month ago ...
[right][snapback]72426[/snapback][/right]

They don't need "hardware updates." Throwing hardware at the problem is almost never the right solution unless the only problem is scaling, and it's clear by the nature of the problems and the times at which they occur that's not the case. (Let's face it, when the server starts throwing up at 6am on Friday morning, you know it's not load. Some of my most nightmarish lag issues have happened at the bottom of their usage curve.)

I'm thinking they need to seriously re-architect the server/database layout, and I'd bet good money that the server code's got severe memory leaks as well (which would explain the off-hours problems).

Of course... they can't even keep their website in operation, so I'm starting to have doubts about their ability to fix the game itself. From what I've been able to gather, they refused to even accept applications for operations folks outside of Orange County, when pretty much everyone who has experience operating mass-usage 24/7 systems is in Virginia and San Jose. I mean, when's the last time you heard of AOL having problems like this? They solved this issue almost 10 years ago, but in that time not one MMO (and keep in mind we're talking about services with 1/10 as many users as AOL is handling) has managed to properly architect their systems so they're appropriately scalable? Madness.

/endrant

[Mavfin]

One comment I have on this, for perspective: Yesterday was the first unexpected downtime I've had the whole time I've played WoW, and still got to play a couple hours in my normal playing time, so, not all on WoW have had the experiences you who are so bitter have had. YMMV.

[Bob the Beholder]

I very much wanted to play on Gorefiend yesterday. But it was down until about 1 AM. It's not on the list, though. Funtimes.

[MongoJerry]

Incidentally, this is after already giving everyone two days credit for the other instabilities over the past week.

[Occhidiangela]

Incidentally, this is after already giving everyone two days credit for the other instabilities over the past week.
[right][snapback]72489[/snapback][/right]

Three days credit? For me, that is a 1.39 cent refund, though really it is for my son's play, I am . . . not hooked yet.

While I am pleased to see the gesture, something more than words . . . did you notice me not swooning? :D

Occhi

[savaughn]

I mean, when's the last time you heard of AOL having problems like this?  They solved this issue almost 10 years ago, but in that time not one MMO (and keep in mind we're talking about services with 1/10 as many users as AOL is handling) has managed to properly architect their systems so they're appropriately scalable?[right][snapback]72428[/snapback][/right]

AOL's virtually exclusively read based cacheable systems are trivial compared to what is done in an MMO. If you find someone who's solved the performance issues inherent in steady state databases (where business logic variables are required to be DB posted before completing) please let me know. Most DB environments are like raid 5. Easy to read from, easy to cripple on fast simultaneous writes.

[Darian]

AOL's virtually exclusively read based cacheable systems are trivial compared to what is done in an MMO.  If you find someone who's solved the performance issues inherent in steady state databases (where business logic variables are required to be DB posted before completing) please let me know.  Most DB environments are like raid 5.  Easy to read from, easy to cripple on fast simultaneous writes.
[right][snapback]72514[/snapback][/right]

This... isn't very accurate (although in some respects it used to be), and unfortunately I'm not allowed to go into too much detail as to why it's inaccurate. Memory caching with periodic writes whenever something changes which must be stored in case something else tries to read it is doable, and being done, at much higher transaction rates than 600,000/(however many realms there are now) users can generate. You have to gauge your level of acceptable loss; if the server crashes and I've killed 8 standard mobs since the last write for my character, well... waaah, I'm thinking.

In fact, we've seen part of the problem already in Blizzard's explanations: they're running more than one server per DB host, when there's absolutely no interaction between Stormrage and Alleria or whatever. If one DB host can't handle the load 23 servers are throwing at it... then why were 23 servers pointed at one DB host? That's just crappy architecture, period.

[LochnarITB]

I think I'm going to have to dedicate a night to creating alts on servers. We have 50 characters per account, right? OK, if I take 30 and spread them out over 30 servers, I stand a good chance of having an alt on any realm that gets a usage credit. It's kind of like playing roulette. Put them on the right ones and get a credit if the outage ball lands on one of my servers. The way things seem to be going, I think I might actually be playing for free. :(