Post-mortem on the BlackBerry outage

The linked article from RCR discusses RIM's failure to notify users about the BlackBerry outage last Tuesday evening. Overall, I agree with the critics that they interviewed. RIM should have, at least, acknowledged the outage ASAP. Instead, a pre-recorded message on their tech support line was their de facto PR contact for almost 12 hours. RIM should have posted something to their website ASAP regarding the outage. Again, no argument. Most importantly, most critics believe that they should have notified their users of the outage. I can see some good reasons why this would have been good, and bad.

It would have been good to notify the IT people who support the BlackBerry Enterprise Servers. A simple e-mail to these people could have saved them hours of time chasing apparent ghosts in their machines when it was a RIM issue from the start. It also would have given them the much needed defense of "It's not our fault" when all of their end-users called them to complain that their BlackBerries didn't work. This was probably a bigger blow to RIM than the reaction from actual BlackBerry users. Many of the IT staffs that are supporting BES are the same people who already support Microsoft's Exchange Server. When an Exchange Server goes down it is within their control to fix it, and they know it. RIM's failure to come to the aid of their own caretakers in the field will certainly be remembered as new options are presented.

It would have been bad to notify the media immediately without a well constructed explanation of the situation. If they had issued a press release stating they have a problem, but don't know what it is, or how to fix it, they would have compounded the PR hit they knew they were going to take anyway. While I think it should have come out sooner, I think RIM was smart to make sure they had all the facts in hand before they went to the media.

Should RIM have notified the actual BlackBerry users right away. Ideally, yes, if they could have, but sending an e-mail to the user would have been foolish. The vast majority of BlackBerry users are not going to have e-mail up on their computer after 5pm in the evening. That is what the BlackBerry is for. An e-mail to end-users announcing the problem would have been insulting at 8am the next day. On the other hand, an e-mail sent before 8am saying that they are aware of the problem and are working to fix it would have been a welcome message on their computer Wednesday morning.

I think the missed opportunity lies in the laps(e) of the carriers. While they were not at fault for the failure, the impact still hit their customers. The carriers probably didn't loose any good will by sitting silent, but they missed the opportunity to provide an extra level of customer service that BlackBerry users would have appreciated and valued. Even with the RIM system down, there were 2 means of communication open to the impacted users, voice and text messaging. I don't believe that RIM has access to either of those contact points for its customers, so it was up to the carriers to recognize the opportunity and capitalize on it.

I manage BlackBerry servers for 3 different companies. As soon as I knew the outage was a RIM issue, I sent text messages to all of my BlackBerry users letting them know that the issue was not a problem with their device or their servers. Tuesday evening I had received several text messages thanking me for the heads-up. Wednesday morning I had only one call complaining about the outage.

The carriers have the tools to send blanket text messages to their users. They also have the ability to put a canned message in your voice-mail box. Would it have taken some time to discern the BlackBerry users from their database? Sure, but I bet the loyalty returned for the effort would have been payed back in hard, measurable dollars.

I'm still a fan of RIM and their solution. I just hope that they do as thorough a post-mortem on their public response as I know they will on the outage.