View Full Version : EVE-Online crippled by uppgrade - cite DB issues
Looks like it has been out of whack for two days now. What is interesting is they also disable their forum during the down time. Is this typical for them?
Deployment continues for the Trinity 1.1: Boost Patch
As reported earlier, database issues were discovered during an early point in the process. Those issues required us to go into our backups for data and this restoration and data integrity check have taken longer than originally anticipated. We regret any inconvenience these delays may cause. Our most recent assessment indicates that service may be restored at 02:00 GMT, and we will provide additional updates as often as possible.
--------------------------------------------------------------------------------
12/03/2008
Tranquility is currently undergoing an extended downtime to deploy the Trinity 1.1: Boost Patch
This downtime was originally scheduled to end at 16:00 GMT. While monitoring the progress of the patch deployment, a database issue came to light that we felt was important to address as quickly as possible. In order to ensure the best quality patch possible for our players, the estimated time for the downtime has been extended as we correct that issue. Please watch the MOTD and our website for more information as it becomes available. Patch notes are available on the Trinity webpage.
mystery
03-12-2008, 06:05 PM
Looks like it has been out of whack for two days now. What is interesting is they also disable their forum during the down time. Is this typical for them?
Aren't forum accounts tied to in-game accounts? If they're having database issues, I'd expect them both to be flummoxed.
fenrrris
03-12-2008, 06:06 PM
They've dropped the forum during the last few big updates, far as I can recall.
CastOutDevil
03-12-2008, 06:06 PM
Yep, forums go down when the servers go down.
SirBruce
03-12-2008, 06:16 PM
Wow, they don't have a hot backup of their database? If they're doing an actual restore from nearline or offline storage that could take a LOOOONGCAT time.
Bruce
mlatin
03-12-2008, 06:25 PM
they'd mentioned in several dev posts, several days beforehand, that they were going to be doing some extended fiddling around with the db.. not surprised they're taking so long to get back up.
Skipper
03-12-2008, 07:11 PM
Eve is single instanced, so the DB is the DB for one and all. I remember a pic for one of the presentations they had where they physically carried one DB server ... THE server ... to a new location once. Which makes me think no, they don't have hot backups, they don't have anything but tape. A dangerous, dangerous game if that's what they are playing.
Alan Dunkin
03-12-2008, 07:30 PM
Yeah it all depends on how they store it; if there's just one DB server (I find it hard to believe but people have done crazier things), downtime could be very long. Hell they'll need to have downtime just to do anything other than normal usage I guess. If they have to backup from tape, jesus. See you next week.
Having forum access use game auth isn't too unusual, though of course it could also be a preventative measure (if the game is down, people will flood the forums and will cause them to collapse anyway, or at least create so much activity that it's not moderatable).
--- Alan
AlanT
03-12-2008, 07:53 PM
Looks like it has been out of whack for two days now. What is interesting is they also disable their forum during the down time. Is this typical for them?
Not sure where you're getting two days from. Downtime started Wednesday at 02:00 GMT, and they're sorta hoping they might be back in business 02:00 Thursday. Which looks like one day to me.
I'm actually willing to believe that they have somewhat of a clue about their database and how to run it, peanut gallery be damned. As far as I know, they haven't lost any of my data to database problems (but I will of course bitch horribly should it ever happen).
They were down most of the day before as well.
AlanT
03-12-2008, 08:05 PM
I was able to log in just fine, and didn't read anything about anybody else having problems. They were originally planning to roll out the patch on the Tuesday rather than the Wednesday and then delayed it 24 hours. The game was online during that time, but that might be the source of the confusion.
mlatin
03-12-2008, 08:10 PM
well, they're allowing the patch to be downloaded now, so i'm guessing the server is back up, or soon to be back up.
EDIT: yup, it's up.
AlanT
03-12-2008, 08:21 PM
Oooh, tempting to log in to look at the new overview stuff. But kinda late for me.
Skipper
03-13-2008, 06:54 AM
I'm actually willing to believe that they have somewhat of a clue about their database and how to run it, peanut gallery be damned. As far as I know, they haven't lost any of my data to database problems (but I will of course bitch horribly should it ever happen).
So I'm not misconstrued here, they do have a clue about what they are doing, hell they are one of the few to have a single DB working at that scale of concurrent users in MMO's today.
The "moving the database server" pic was very serious though, even talked about during the presentation. I would hope they've moved to a more robust environment than a single server by now, their userbase has grown by leaps and bounds.
Probably the worst thing that can happen to an online game is loss of user data. Even a day or two would be a major problem. Eve has daily downtime each morning, I'm assuming one of the things that happens then is some sort of backup or snapshot.
SirBruce
03-13-2008, 07:26 AM
You shouldn't need to take your servers down to take a snapshot, and you should be making your backups from the snapshot.
I will say that in general I've heard great things about EVE Online's architecture. That's why this amount of downtime was so surprising. I suspect they were doing more than a restore but fixing the software issue that caused the corruption in the first place.
Bruce
Nick Walter
03-13-2008, 07:50 AM
I suspect from the story and my own experience with fucked up DB outages that they violated rule number one of db backups; "you have no valid backup until you've tested and validated your restore plan"
It's a surprisingly common goof, someone just installs a DB snapshot or backup tool and starts dumping data to backup media every day. Sure it's being monitored in that the tools logs are consulted for success/failure every day. Then the DB goes tits up and it's emergency restore time and people find out that they weren't backing up everything they ought to have been, or don't actually know how to use the restore tool, or some other issue that would have been blindingly obvious if they'd actually tested a restore.
Rward
03-13-2008, 08:41 AM
Back in Feb 2006 they had this (http://myeve.eve-online.com/devblog.asp?a=blog&bid=303)
Then in September 2006 they had this (http://news.softpedia.com/news/EVE-Online-Readies-the-Largest-Supercomputer-in-the-Gaming-Industry-35225.shtml):
The upgraded server cluster features dual-processor 64-bit AMD Opteron-based IBM BladeCenter LS20 blade servers, along with several enhancements to the cluster internet connectivity. Apparently, the EVE Online supercomputer can manage over 150 million database entries each day, thanks to the 64-bit hardware architecture from IBM. Being recognized as the fastest storage devices up to date, Solid State Disks (SSD) replace regular hard disks, performing over 400,000 random I/Os per second. Eve Online is steadily setting brand new records in terms of concurrent users on a single shard, the latest being acknowledged just one week ago with over 30,000 players online at the same time.
Heres the "official post" on moooore hardware (http://myeve.eve-online.com/devblog.asp?a=blog&bid=392)
Looks like the same stuff again (http://myeve.eve-online.com/devblog.asp?a=blog&bid=387)
I'm sure that if they're running solid state disks then they kinda know what they're doing and not backing up to tape...
Nick Walter
03-13-2008, 09:04 AM
I'm sure that if they're running solid state disks then they kinda know what they're doing and not backing up to tape...
Bad assumption. The presence of fancy hardware may indicate savvy tech types, or it may indicate management that believes whatever the hardware salesman says and buys fancy shit for no reason. Sales reps love managers like that and will gleefully sell them way more kit than they need.
SirBruce
03-13-2008, 09:19 AM
Again, I've read those articles, but they offer little guidance as to their backup and recovery schemes. I doubt they actually restored from tape in this instance, but clearly they had a database issue they had to deal with before they could go live.
Bruce
Cougar
03-13-2008, 01:33 PM
The game was down for less than 24 (but quite near 24hr) which is Eve's standard operating procedure for any patch of significant size or upgrades.
There is really nothing news worthy to see here :)
~C~
Did anyone else join the channel the devs made today to hear player issues? They sent out a server wide notice that it was opening. The constant spam inside was both totally predictable and a little sad.
schurem
03-13-2008, 06:55 PM
luckily the chat chann was only a carrier to the voice chann wich had only the devs in. what was said? i didnt attend.
SpoofyChop
03-13-2008, 07:11 PM
It's almost miraculous what they've accomplished really. They've managed to take pure boredom and put it into a database! I didn't think it was possible but 20 minutes of the demo proved it.
Jon_Danger
03-13-2008, 07:15 PM
I just started a 21 day free trial through steam too, and I can't play... BOO!
Dravalen
03-13-2008, 08:01 PM
I will say that in general I've heard great things about EVE Online's architecture. That's why this amount of downtime was so surprising. I suspect they were doing more than a restore but fixing the software issue that caused the corruption in the first place.
Bruce
I've heard the opposite, I don't have a source but I remember hearing they are running Python on the games backend. I wouldn't want to let any interpretive language near my server's backend(client, sure) but that's just me.
It's almost miraculous what they've accomplished really. They've managed to take pure boredom and put it into a database! I didn't think it was possible but 20 minutes of the demo proved it.
You played 20 mins of the demo and you classify the entire game as boring? Or am I not understanding correctly?
Marcus Walser
03-13-2008, 11:47 PM
You played 20 mins of the demo and you classify the entire game as boring? Or am I not understanding correctly?
To be fair, the game is pretty boring most of the time. :-P
Rward
03-14-2008, 06:46 AM
I've heard the opposite, I don't have a source but I remember hearing they are running Python on the games backend. I wouldn't want to let any interpretive language near my server's backend(client, sure) but that's just me.
I think its a semi custom stackless python build.
You can download it somewhere...
It's almost miraculous what they've accomplished really. They've managed to take pure boredom and put it into a database! I didn't think it was possible but 20 minutes of the demo proved it.
The game is entirely what you make of it - if you want boredom you'll find it,
you want action, it'll give it to you in spades, a fine mix in between - sure.
It all depends on the what you want to do and how you go about doing it..
vBulletin® v3.8.4, Copyright ©2000-2009, Jelsoft Enterprises Ltd.