Andy Orrock | Payment Systems

February 14th - Procrastinators of the world unite!

There was some question in my mind as to whether February 13th (the day before Valentine's Day) or the holiday itself would be the peak processing day for our flagship OLS.Switch payment acquirer implementation.  Then last night my wife noted that when she was lined up to buy bananas at our local supermarket, she was "stuck behind every loser boyfriend in the world buying picked over chocolates and dead flowers."  That's called field analysis.   It became apparent to me at that moment that Valentine's Day itself would be the winning ticket.  The statistics bear that out - I had noted we processed 1,075,441 transaction requests on February 13th.   Yesterday clocked in at 1,168,700.  Procrastinators of the world unite! 

I did one of my volume, distribution and performance breakdowns of the day.  You can see the pop to the left (PDF is here).  You can see that most of the transactions are still processed sub-second.  Only Debit exceeds 1000 ms...not unexpected, given that a typical Debit transaction has to undergo three separate PIN translations along the way (at our location as the acquirer, at FDR North as gateway, at the regional debit network) and then a PIN validation at the card issuer.  We typically see a 300+ ms differential between online (PIN-ed) Debit vs. Visa/MC credit + offline (signature) Debit when routed through the same gateway provider.

Another note of interest is that most of those transactions get routed externally for authorization using jPOS' excellent QMUX facility.  For the record, the breakdown in this implementation goes like this:

• Credit Card brands Discover (DS), MasterCard (MC) and Visa (VI) get routed to the FDR North gateway as 0100 authorizations.  [Note that offline Debit transactions are treated like VI/MC credit and get routed similarly.]
• EBT (Food and Cash) and Debit also get routed to FDR North as 0200 Purchases.
• American Express (AX) brand gets routed to AMEX as 1100 authorizations (AMEX uses the 1993 ISO 8583 standard, so the auth is a 1100 instead of a 0100).
• There's also a link for credit card auths to JC Penney for approval of their card (a special arrangement with our client).  This is a custom message format.
• We have four separate Stored Value-class auth links set up to providers Incomm (here as card types 'CP' and 'IN'), Comdata's Stored Value Systems aka 'SVS' (card type 'GF'), Verizon (card type 'PC') and Green Dot (card type 'SV').  Each of those is a variety of the 1987 ISO 8583 standard, except Verizon (a custom fixed format which we solved via jPOS' FSD facility)

Most of those interfaces have redundant links in place to avoid a single point of failure.  In total, we have seven separate QMUX implementations inside this implementation managing a total of 12 channels.  Additionally, we have redundant application nodes, meaning that our status table and the User Interface that displays it now gives a birds-eye view to 24 separate authorization links at this customer site.

February 13th

In the world of retail payment switches, we know to be alert for sharp peaks on December 24th.  For our flagship OLS.Switch client, another peak day - though not quite as frenetic - is February 13th...the day before Valentine's Day.  Looking at our stats from yesterday, I see we processed 1,075,441 transaction requests at 4,118 retail locations (> 20,000 register lanes...I'll get an exact number for a subsequent post) across four US time zones. 

We had only nine remote timeouts.  This is a testament to our gateway and payment authorization partners, especially FDR North through which we route all Visa, MasterCard, Discover, Debit and EBT requests. 

The previous day we reached a really nice milestone:  for the first time ever, we recorded zero remote timeouts (on 846,023 transaction requests).  It takes a bit of luck, actually, to get that to happen.  Every card issuer in the country (hell, in the world, really) has to respond within 25 seconds.  And, on our side, it takes a really nice multiplexer implementation to facilitate these volume surges.  Thankfully, we did the needful and used jPOS as our payment engine underpinning.  Alejandro's QMUX facility is a scalable, reliable bulwark of the jPOS project.

Check out Dave's blog

My OLS cohort, Dave Bergert, has initiated a blog worthy of inclusion in your RSS Reader.  Dave's working on our PABP certification, so his first couple of posts are straight to the point on that ongoing effort: one references a great piece clarifying the not-too-obvious differences beween PABP and PCI compliance; and the other discusses the changes Dave is incorporating into the jPOS-EE UI (a.k.a., eeweb3) in order to meet Section 3.1 of the PABP requirements (that's the section that deals with things like user names and password complexity). 

As background, Dave's obtained his CISSP, CISA, and CompTIA Security+ designations.  He's also a former Visa-certified QSA (Qualified Security Assessor) himself.  He's one of the few guys out there who is equally adept in both worlds:  He can build payment systems from scratch; and audit them as well.  That made him very dangerous as an auditor...if you were like this guy and had something to hide.

In short, the man for the job...and a blog well worth reading, especially given that PABP compliance has forced its way centerstage in our payment system world.