News
28 Apr Mobile Vikings, powered down by Belgacom
Last weekend, Mobile Vikings experienced an outage of 24 hours. The cause was a power outage at the Scarlet data centre.
The events started on Saturday evening around 8:30 PM. Customers who wanted to surf on the internet received a message “Cannot connect”. We checked our network monitoring in 4 distinct locations in Belgium and confirmed that all locations lost their internet connectivity. Throughout the outage, voice and sms connectivity was luckily not affected. Around 9:30 PM we got initial feedback from Base, stating that they could confirm there was a problem and they started their investigation. At 1 PM on Sunday engineers arrived on location to try solve the problem. Connectivity to the internet was restored around 9:50 PM on Sunday evening.
Post-mortem investigation showed that the root cause of all this misery was a major power failure in the Belgacom-owned data centre of Scarlet. We didn’t realize that Mobile Vikings are literally “powered by Belgacom”, but in this case it was the lack of power that got to us. Most data centres test their backup power supplies at least monthly, but at least in this case the backup power of the data centre didn’t take over properly. This of course caused a shut down of the Base ICT systems located in that data centre, once UPS batteries ran out. In the end a manual intervention was needed to restart all systems in the correct sequence.
Events like these are very unlikely and unpleasant at the same time. I want to thank all of our users for not killing us in public, even though we failed and there is no excuse for an outage like this. I also want to thank all the people at Base and their suppliers who have been working with us throughout the weekend to fix this problem. If anyone knows the engineers involved in the on-site intervention, I would love to put their picture up here as a sincere “thank you” for getting our Mobile Vikings back online.
On the bright side, as an apology for the downtime I am entitled to offer our users 10 VIP tickets to the Base-sponsored Suikerrock event. The first 10 comments on this post that include a valid Mobile Vikings user name with an active SIM card will be contacted to join us on Suikerrock!
Comments
« previous Page 3 of 4 next »
-
Greg
Tue 28 Apr 2009 22:03Think I'm going to be a bit late for the tickets... But doesn't matter.
It's me, a client, who wants to thank all of you for making mobile internet a reality that anyone (even students) can afford!
Thanks a lot! Keep it going!
-
Greg
Tue 28 Apr 2009 22:05By the way, username : Greg
(of course)
-
Greg
Tue 28 Apr 2009 22:17I don't know where to post that, but if you meet a french-speaking client, you can send him here : http://forum.astel.be/showthread.php?t=144309
That's a thread I did opened 5 months ago, by the time I heard about your services.
We'll take care of your clients and convinced the others! ;)
-
Alpaca
Tue 28 Apr 2009 23:21Pff,dit vind ik flauw! Iedereen in belgie heeft problemen gehad afgelopen weekend (ik vandaag opnieuw!!). En dan zijn er maar een paar die een ticket krijgen, in plaats van iedereen een dag langer internet te geven of iets dergelijk zodat iedereen er kan van profiteren!!
Hopelijk worden hier lessen uitgetrokken!
Grtz
AlpacaP.s.: ik vind het ook belachelijk om de schuld op iemand anders te gaan steken en belgacom zwart te maken!
-
Greg
Wed 29 Apr 2009 07:11He s got a point there
-
Koen Delvaux
Wed 29 Apr 2009 08:58Point taken.
When I wrote the article yesterday, the "powered down by Belgacom" sounded funny to me.
Of course these things can even happen to the best. We are taking lessons from this as we speak, such as how to make our escalation procedures quicker and get more accurate and faster diagnostics.
So no doubt about it, it was us that failed to serve our users in the end. -
Thomas B
Wed 29 Apr 2009 17:49Ik had gisterenavond ook problemen om te sms'en (lees: het ging niet). Lag dat aan mij of had dat hier ook iets mee te maken?
-
Stef Bontinckx
Wed 29 Apr 2009 18:17username: stefbontinckx
hopelijk nog binnen de tien? Maar ik vrees ervoor
-
tijs
Wed 29 Apr 2009 18:48bij mij ging smsen gisteren avond ook héél moeizaam... op een gegeven moment zelfs niet... ik denk dus niet dat het aan jou lag
-
pragmarc
Wed 29 Apr 2009 22:02Wat mij ontzettend verbaasd, is de wereldvreemde, volstrekt on-professionele communicatie van Scarlet:
""De switch van het netwerk naar de reservestroomgeneratoren heeft het laten afweten", aldus sales- en marketingdirecteur Marnix Kuypers van Scarlet. "We hebben onze excuses aangeboden aan onze grote klanten."
Zo, blijkbaar zijn enkel de 'grote' klanten van tel bij dat voortreffelijke telecom-bedrijf... Goe bezig die Marnix Kuypers!
Een officiële mededeling over de onderbreking van de dienstverlening werd niet verstuurd. "Maar we hebben wel meteen een boodschap op onze website geplaatst, en onze grote klanten werden allemaal de avond zelf nog gecontacteerd en begeleid", aldus nog Kuypers. "De impact op de residentiële klanten zal sowieso klein geweest zijn. De meeste ADSL-klanten hebben gewoon even zonder internet gezeten. That's it.
THAT'S IT???
Jeetje, beseft die man wel wat hij uitkraamt?? Echt wel de juiste man in de juiste functie (kuch)...
« previous Page 3 of 4 next »
Loading page...
