Everybody Freeze!
In the Payment Systems World, we’ve now entered “The Freeze.” This is the industry-wide term associated with the period running from approximately the Thursday before Thanksgiving (which is the last Thursday of November in the US) through to about the second full week in January. During that period, we don’t do any production releases, unless it’s a fix of a critical nature. We advocate the same practice for our clients. We also recommend that they not undertake material changes to hardware configurations, databases, scripts, or any other piece of supporting or underlying technology.
Traditionally, it’s been a good time for us to focus on big projects that have a Q1 delivery date. We can stay under the covers and make some serious progress on those bigger initiatives.
We also send out a customer letter re-emphasizing how to get a hold of us. The letter stresses the importance of vigilance and watchfulness over key production systems. It reminds our clients that we’re Always Available. Our firm was founded on the back of 24x7x365 support of mission-critical production systems. We get paid to make sure our clients can – to the fullest extent possible – enjoy the holidays with their family knowing that we’ve got their back on support.
Why all the heightened concern? It’s the nature of payment systems: there’s tremendous upsurge in volume in the freeze period. If you’ve got a latent bottleneck laying dormant and ready to strike, the unfortunate reality is that it’s going to nail you right between the eyes on a killer day like Black Friday, Cyber Monday or Christmas Eve. We service some Stored Value authorization endpoints that get massive 20x surges in volumes on December 24th. So, you’ve got to be ready.
We work the other ten-and-a-half months of the year to make this month-and-a-half as uneventful as possible.
On legacy payment switch platforms you had to do your peak planning very diligently because of capital budgeting implications. A new CPU board alone could run a couple of hundred thousand dollars. If you did your forecasting incorrectly and didn’t mention an upgrade requirement, then got squeezed by a capacity shortage running up to Dec 25th…well, there was hell to pay. That’s ended careers.
In our new generation switch environment, the dynamics are different. Since our clients run on commodity hardware solutions (with the same levels of up-time), more capacity can be attained in short order by scaling vertically (replacing old servers with new – thanks to Moore’s Law, we can get double the horsepower every 18 months or so) or horizontally (like Google’s build-out model). [Note: There are important considerations on the horizontal model for payment systems. You can’t just slap it in. That’s a subject for another post.]
Great post Andy! This morning I had a very interesting call with a potential OLS.Switch customer. They are in a mild panic state because of factors you mention.
Randy San Nicolas
http://www.prepaidenterprise.com
Posted by: Randy San Nicolas | Tuesday, November 25, 2008 at 18:05