Andy Orrock | Payment Systems

Making the list

Visa just released its October 15, 2008 update to the list of validated payment applications that have been assessed for compliance with the Payment Application Best Practices (“PABP”).  OLS.Switch is on it.  See more about our efforts and my colleague Dave Bergert’s role in it here.

Our thanks to the K3DES team – Jim Richardson, Wayne Varga and Davi Ottenheimer.  As our QSA, they were instrumental both in the audit itself and in making sure the submission stayed on-track at Visa.  [As you can imagine, Visa has been swamped with assessors’ reports as vendors the world over look to meet this year’s deadlines.]

Over half-a-billion served

As recently noted as a pending occurrence, the odometer just flipped past the ‘500,000,000 Transactions Processed’ mark at our flagship OLS.Switch acquirer-side payment switch.  Dave Bergert captured the moment via Camtasia; then, he and I played it over and provided some commentary.  Dave did the mix in his studio and came up with this nice result.

The gist of my commentary:  when people talk about what they want in a payment switch, they often speak in terms of raw power:  “It’s got to do 800 TPS,” or something outlandish (for the majority of us) like that.  But running a production switch and doing it day after day after day is quite a different set of disciplines – it takes staying power, diligence, and a commitment to continuous improvement.  It’s a classic ‘tortoise and hare’ story.  We can do the ‘hare’ bit, but the tortoise’s commitment eventually wins the day.

If you’re not proactive about all the various things I’ve talked about on these pages – reversals, duplicates, database management, extracts, on-boarding, creating repeatable processes, monitoring, an operational focus – what will happen is that, inexorably, you will lose a little, tiny bit off your performance edge with each passing day.  And, lamentably for you, the way it’ll manifest itself is that one day you’ll wake up and the whole thing will have simply fallen off a cliff. 

It’s akin to politics and the difference between campaigning (here, the ability to ‘wow’ people and initially win them over with high TPS throughput ratings) and governing (here, a focus on daily operations and continuous improvement).

On-boarding a New Auth Interface in jPOS, Part 6

Part of a continuing series…

We’ve now on-boarded a wide range of authorization endpoints onto our jPOS payment transaction frameworks.  In my On-Boarding Guide, I give relatively short shrift to the thinking you ought to go though in terms of the implications of supporting a wide range of endpoints via your Transaction Manager(s) (hereafter, I’ll refer to this as a ‘TM’).  In the section of the guide entitled “Add new participants to TransactionManager,” I go through the ‘vanilla’ exercise of assuming that the new endpoint gets added into your main TM.

Now, let’s go beyond vanilla.  In a situation like we have at one of our acquirer-side customers, it could prove problematic to have all endpoints piled into one TM.  First there’s just the sheer maintenance of the thing:  even with some separation (see discussion belong), our main TM there has extended to 42kb.  There enters at that level the simple baseline concern of over-complexity and the resulting potential for error during new service on-boarding. 

Moreover, there’s my previously stated concern that smaller, less proven endpoints could prove unstable.  Excessive timeouts or simply elongated response times could easily suck up all your tasks leaving incoming transaction requests piling up in the server queue.  Not to cast aspersions at stored value card providers, but that’s where I had most of my concern because that class of authorizer is subject to massive run-ups in holiday volume (20x vs. a ‘typical’ day is not unusual).  Your good performance becomes totally predicated on some external party doing their peak planning correctly.  Which, frankly, ought to scare the crap out of you.

This situation is especially acute in situations like the one I’m describing here where we have transactions coming from 4,900+ locations across the four US continental all funneling into one listening point.  Here, we have a server that look like this:

<xx-server class="org.jpos.xx.Server" logger="Q2">
<property name="port"    value="30000" />
<property name="space"   value="tspace:default" />
<property name="queue"   value="NNNNN.TXN" />
<property name="timeout" value="60000" />
</xx-server>

Now, we forward to our main TM.  That entity starts off looking like this:

<xx-txnmgr class="org.jpos.transaction.TransactionManager" logger="Q2">
<property name="sessions" value="120" />
<property name="space" value="tspace:default" />
<property name="queue" value="NNNNN.TXN" />
<property name="debug" value="FALSE" />

<participant class="org.jpos.xx.PrepareContext">
   <!-- need to know the Space and Queue in order to get the
        number of outstanding transactions at the start of
        this transaction -->
   <property name="space" value="tspace:default" />
   <property name="queue" value="NNNNN.TXN" />
</participant>

<participant class="org.jpos.transaction.Open" logger="Q2" realm="open">
  <property name="checkpoint" value="open" />
</participant>

<participant class="org.jpos.xx.ParseRequest"
    logger="Q2" realm="parse-request">
  <property name="checkpoint" value="parse-request" />
</participant>

Now, the all important switch step – you can see that the thing here is that we process Debit, EBT and Credit in the main TM while using jPOS’ powerful Continuations facility to push the stored value stuff out to other TMs:

<participant class="org.jpos.xx.Switch" logger="Q2" realm="switch"> 
<property name="X.41" value="EmployeeVerification LogAndReply" />
<property name="X.42" value="RewardsProgram LogAndReply" />
<property name="X.70" value="CheckPurchase LogAndReply" />
<property name="X.93" value="CreateDebit Decrypt DebitSale" />
<property name="X.96" value="CreateDebit FindOrigPurchaseSoft CopyFromOriginal DebitSaleReversal LogAndReply" />
<property name="X.60" value="CreateDebit Decrypt EBT_FS_Sale" />
<property name="X.61" value="CreateDebit FindOrigPurchaseSoft CopyFromOriginal EBT_FS_Void LogAndReply" />
<property name="X.62" value="CreateDebit Decrypt EBT_FS_Return" />
<property name="X.63" value="CreateDebit Decrypt EBT_FS_BalInq LogAndReply" />
<property name="X.68" value="CreateDebit FindOrigReturnSoft CopyFromOriginal EBT_FS_Return_Rev LogAndReply" />
<property name="X.69" value="CreateDebit FindOrigPurchaseSoft CopyFromOriginal EBT_FS_Sale_Rev LogAndReply" />
<property name="X.65" value="CreateDebit Decrypt EBT_Cash_Sale" />
<property name="X.66" value="CreateDebit FindOrigPurchaseSoft CopyFromOriginal EBT_Cash_Void LogAndReply" />
<property name="X.67" value="CreateDebit FindOrigPurchaseSoft CopyFromOriginal EBT_Cash_Sale_Rev LogAndReply" />
<property name="X.54" value="CreateCredit Decrypt CreditSale" />
<property name="X.55" value="CreateCredit FindOrigPurchaseSoft CopyFromOriginal CreditSaleReversal LogAndReply" />
<property name="X.08" value="CreateCredit FindOrigPurchaseSoft CopyFromOriginal CreditSaleReversal LogAndReply" />
<property name="X.06" value="CreateCredit Decrypt CreditReturn" />
<property name="X.07" value="CreateCredit FindOrigReturnSoft CopyFromOriginal CreditReturnReversal LogAndReply" />

<!—svc provider 1 Transactions -->
<property name="S.43" value="ForwardToProvider1TxnMgr" />
<property name="S.44" value="ForwardToProvider1TxnMgr" />
<property name="S.45" value="ForwardToProvider1TxnMgr" />

<!—svc provider 2 Transactions -->
<property name="T.43" value="ForwardToProvider2TxnMgr" />
<property name="T.44" value="ForwardToProvider2TxnMgr" />
<property name="T.45" value="ForwardToProvider2TxnMgr" />   

<!—svc provider 3 Transactions -->
<property name="V.43" value="ForwardToProvider3TxnMgr" />
<property name="V.44" value="ForwardToProvider3TxnMgr" />
<property name="V.45" value="ForwardToProvider3TxnMgr" /> 

<!-- Unhandled transactions -->
<property name="Unhandled" value="Unhandled" /> 
</participant>

The problem here:  we’ve inoculated ourselves against unexpectedly poor performance by the SVC class endpoints.  But – if the Debit/Credit/EBT endpoints were to suffer – then SVC-bound transactions would sit in the server queue and not be able to traverse the first steps of the main TM.  [Side note:  Your friend here is the threshold timer mechanism.  Don’t leave home without it.]  In short, the continuations don’t help us at all in that situation. 

That’s exactly what happened to us in May of this year.  Despite some vaunted geographical separation in data centers by the gateway provider, Debit/EBT transactions were found to have a single point of failure (since resolved) somewhere in their solution framework.  The lesson learned for us is that you’ve got to provide endpoint separation where ever and whenever possible.  We had assumed “hey, let’s protect the strong from the weak.”  In practice, SVC providers like Stored Value Systems have been superstars, while other presumed stalwarts have been a source of some unwanted surprises.

For that reason, we’re in the process of re-architecting the main TM – it’ll do those preliminary steps, handle those first transactions (EmployeeVerification, RewardsProgram, CheckPurchase) internally, then take the Continuations road on all transactions that are externally authorized.  We end up like so (we don’t separate Debit, Credit and EBT in this instance because the majority of the time they end up getting routed through the same gateway provider):

<participant class="org.jpos.xx.Switch" logger="Q2" realm="switch">

<!-- These first three we do right inside the main TM here –>
<property name="X.41" value="EmployeeVerification LogAndReply" />
<property name="X.42" value="RewardsProgram LogAndReply" />
<property name="X.70" value="CheckPurchase LogAndReply" />

<!-- Everything else, we forward in order to protect one set of
     transaction classes from bad performance from another  -->

<!-- Debit Transactions -->
<property name="X.93" value="ForwardToDebitCreditTxnMgr" />
<property name="X.96" value="ForwardToDebitCreditTxnMgr" />

<!-- EBT Transactions -->
<property name="X.60" value="ForwardToDebitCreditTxnMgr" />
<property name="X.61" value="ForwardToDebitCreditTxnMgr" />
<property name="X.62" value="ForwardToDebitCreditTxnMgr" />
<property name="X.63" value="ForwardToDebitCreditTxnMgr" />
<property name="X.68" value="ForwardToDebitCreditTxnMgr" />
<property name="X.69" value="ForwardToDebitCreditTxnMgr" />
<property name="X.65" value="ForwardToDebitCreditTxnMgr" />
<property name="X.66" value="ForwardToDebitCreditTxnMgr" />
<property name="X.67" value="ForwardToDebitCreditTxnMgr" />

<!-- Credit Transactions -->
<property name="X.54" value="ForwardToDebitCreditTxnMgr" />
<property name="X.55" value="ForwardToDebitCreditTxnMgr" />
<property name="X.08" value="ForwardToDebitCreditTxnMgr" />
<property name="X.06" value="ForwardToDebitCreditTxnMgr" />
<property name="X.07" value="ForwardToDebitCreditTxnMgr" />

<!—svc provider 1 Transactions -->
<property name="S.43" value="ForwardToProvider1TxnMgr" />
<property name="S.44" value="ForwardToProvider1TxnMgr" />
<property name="S.45" value="ForwardToProvider1TxnMgr" />

<!—svc provider 2 Transactions -->
<property name="T.43" value="ForwardToProvider2TxnMgr" />
<property name="T.44" value="ForwardToProvider2TxnMgr" />
<property name="T.45" value="ForwardToProvider2TxnMgr" />   

<!—svc provider 3 Transactions -->
<property name="V.43" value="ForwardToProvider3TxnMgr" />
<property name="V.44" value="ForwardToProvider3TxnMgr" />
<property name="V.45" value="ForwardToProvider3TxnMgr" /> 

<!-- Unhandled transactions -->
<property name="Unhandled" value="Unhandled" /> 
</participant>

The PABP Compliance Threshold: We’re so over it

The Word arrived yesterday from Visa – via our Qualified Security Auditor (‘QSA’) – that our acquirer-side payment switch solution will be added to the List of Validated Payment Applications that have been assessed for compliance with the Payment Application Best Practices (‘PABP’).  We’d completed the effort a number of months ago, but Visa has been under a virtual deluge of submissions.  We’ll be on the list dated October 15, 2008. 

This isn’t a one-time effort though.  We’ll need to be reassessed and revalidated each year.  Like any effort of this nature, the first time is always the one that takes the most effort.  We are most fortunate to have my colleague Dave Bergert spearheading our compliance efforts.  What was eye-opening to me was that we (‘we’ = Dave) spent a minority of time addressing code shortfalls vs. PABP mandates, and a majority of the time compiling our PABP Compliance Guide (it’s important that we tell our users how to implement to assure compliance), creating and providing other supporting documentation, and shoring up our processes and procedures.  Our focus on process has always been strong. In the PABP era, it’s even better.

 The ‘Compliance Mandate’ at left gives you a good summary as to why this is so important to solution providers in the payment systems industry.  The original PABP announcement was very clear that Visa has determined that “[v]ulnerable payment applications have proved to be the leading cause of compromise incidents, particularly among small merchants.”  Anyone out there currently considering a payment switch is going to have ‘PABP Compliant App’ as a tick-box.  It you don’t have it, you aren’t playing.  There’s no finessing the point with a prospective buyer.

Here’s a passage from our press release on this news:

"We take the security of cardholder data seriously," said David Bergert (CISSP, CISA) of OLS.  "Obtaining our PABP certification ensures that when our customers implement OLS.Switch in accordance with its PABP Implementation Guide and within their PCI DSS-compliant environment, it will protect stored data and will not retain sensitive authentication data.  Our PABP validation also provides assurance that as a software vendor, we follow secure development processes and secure troubleshooting procedures.”

What my gateway processor needs to do

I’ve experienced some frustration with one of my client’s gateway processor over the past couple of weeks.  That’s spurred me to collect some thoughts about how to select a processor – beyond financial terms like $ per transaction, what should you request and expect from a gateway?

First, a quick definition – a ‘gateway processor’ is a critical selection for transaction processing Acquirers.  By establishing one connection, it provides our acquirer-side customers access to regional and international debit card networks, card associations and their issuers, EBT processors, prepaid processors, and FSA/HSA authorizers.  Through a series of institution-to-institution connections and decision-making based on BIN routing tables, we can get to any card issuer in the world by via the single step of forwarding the transaction request to the gateway.

A typical configuration is one large Acquirer customer of ours where we send Visa, MasterCard, Discover, all Debit, all EBT, FSA/HSA and TV Converter Coupons to the gateway, but break out AMEX to go direct.  Needless to say, picking a gateway isn’t a trivial decision.  In fact, it’s one of the most important decisions you’ll face. 

Bottom-line:  You shouldn’t make the decision on cost alone.  There are other important factors, namely:

1. Bandwidth – Can the gateway handle your projected TPS, especially during peak season?  Depending on transaction mix, some gateways can experience crazy volume run-ups in the days leading to Christmas.  December 24th is a hell of a time to find out that your gateway can’t cut it.  Everyone in our industry has December 24th horror stories.
2. Uptime – Does the gateway have a connection model that maximizes uptime?  I want to connect to two or more geographically-separated data centers, one of which is up and running at all times.  You want one data center upgraded at a time.  Ask them to explain their upgrade procedure to you.  Also, they need to prove to you that the transaction paths of items sent to destination A and B are 100% separable for all transaction types.  We had a shock recently when one of our gateways discovered – to their surprise and ours – that all Debit transactions were reliant on a single piece of hardware in a single data center.  The upshot: nothing with a PIN got authorized for over an hour during a Friday afternoon.
3. Test Facilities – There needs to be a fully-functioning test facility.  It must accurately reflect production capabilities.  It must keep pace with all new mandates.  It should be available 24x7 without manual intervention.
4. Well-defined Certification – There should be well-defined, well-working certification scripts and processes.  Analyst feedback should provide insight above and beyond what’s available in the specifications.  In many cases, analyst feedback has been a lifesaver for me.
5. Simplicity – We’ve spent countless hours dealing with unnecessary complexities in gateway implementations, especially on the extract side.  This isn’t a trivial point.  You’ll pay a price in terms of time to market and maintenance.  There’s also a drag on the sheer processing time required to format and pump out the extract.  We have one gateway where a single retail transaction can result in the formatting of up to five (!) separate settlement records.  Extracts run on a single CPU and simply take time to process.  This one extract is simply a beast.  For a good contrast, take a look at the simplicity and clarity of the Discover and AMEX extracts.
6. Proactive Nature – TELL ME about problems as they’re happening.  Don’t make me ask you to research your problems.  [More on this below.]  Another manifestation of this item:  you should not be a position where you’re asking your partner about where they stand on important new association mandates.  I know one client who made a gateway decision between providers A and B.  He chose A.  He got overruled and the organization went with B.  In a brilliant bit of selling and fortitude, A keeps in touch with him and pings him on upcoming mandates and their implications.  He sends those missives directly to B (zing!) asking where they stand on it.
7. Financial Accuracy – I’m not on the frontlines of this one, but I know business owners at my clients get very frustrated with gateway relationships where the daily settlement numbers never seem to foot.

There are hidden costs that will reach out and bite you if any of those factors are found lacking in your gateway relationship.

Here’s some recent frustration – note how many Host-based reversals we got at one client over a week’s period (see highlighted numbers at left).  You can see how out of range those are.  I did some research and found out that they were all Debit transactions, they came in waves (100- 200 over a minute’s period), and in each case we eventually got a late response indicating that the gateway couldn’t connect to the sub-network. 

Going back to my point above about how you want a gateway with a proactive nature, we never got notified about this as it was happening.  [You can see it got very intense for two to three days.]  By contrast, I’ll note that I once had NPC as my gateway processor when I was running the Tosco operation.  What they gave us was something I covet: proactive visibility.  Had NPC been my gateway in this situation, I would have received an SMS saying something like “Approval rate for the XXXX network has dipped below YY%” – so, I know right away that (a) my gateway is aware of the problem and (b) they know the root cause of culprit (they are a lot of actors at work behind the scenes).  I would get a second SMS when the problem was resolved.

With increasingly degrees of incredulity on my part, here’s what happened when I launched my investigation:

• They ‘held off’ launching an investigation so we could see if it was something on our side.
• They stated that ‘none of the other merchants are complaining.’  [Complaining?  I thought this was customer service you were providing?]  Other merchants’ inattention is not my problem.
• Next:  “This might be an Interlink problem.  Are those Interlink cards?”  You’re a gateway processor.  You tell me.  I have zero visibility to that as an acquirer.  You protect us from that complexity.  It’s the essence of your job. 
• And finally, the capper:  I provided shared reference numbers (Retrieval Reference Number – RRN - and System Trace Audit Number – STAN) for all transactions involved.  I’m going to let my colleague, Dave Bergert, blog about the story from here because it gets into his wheelhouse.

[By the way, as I write this post, I’m eight days out from that issue.  We’ve received no insight.  I should go back and add number 8 to my list:  Don’t get a gateway that just hopes problems go away and are forgotten.]

Vendors we like

It was a treat to have a chance to talk to John O’Brien and Mark Kenney of MagTek at the just-concluded ATM Debit and Prepaid Forum 08 in Chandler, Arizona.  Paul Deignan of MagTek Express has been a great help to us on a couple of projects.  He had put me in touch with Kiran Gandhi of MagTek.  Kiran’s been proselytizing MagnePrint to me and many others for a number of years – taking it from concept to realization.  When I mentioned to John and Mark that I was aware of MagnePrint, they immediately said “Ah, Kiran must have talked to you.”

Here’s how MagTek describes the technology:

What is MagnePrint?

MagnePrint® is a card authentication technology which allows any magnetic stripe card to be recognized as a unique and non-reproducible security token. MagnePrint is able to detect cards that have been illegally reproduced (“skimmed”) as well as cards that have had their data re-encoded or magnetically altered. The term itself is derived from the following expressions: “Magne” as in magnetics, and “Print” as in fingerprint.  Just as fingerprints can identify human beings, Magnetic Fingerprints (MagnePrint) can identify magstripe cards. This is possible because of the stripe composition.

John and Mark also provided Dave Bergert and me with MagneSafe™ Secure Readers for evaluation (that’s me using it in the photo above).  MagneSafe is designed to attack the vulnerabilities that Dave talks about here.  MagTek Business Development Manager Larry Meyers described MagneSafe’s features to Dave as follows:

[It] combines the following security features into a single card reader:

1. Data Encryption  (Encrypts TK1 and TK2 card data via TDEA DUKPT)
2. Device Authentication  (Allows a challenge/response authentication sequence between a host and reader to authenticate both nodes)
3. Card Authentication  (MagnePrint card authentication technology detects the use of fraudulently copied (skimmed) cards)

Each of the above features can be used independently, or in tandem, to provide layered protection of cardholder data based upon the risk/regulatory requirements for a given system.

OLS Event at the ATM Debit & Prepaid Forum 08

Here are some photos from Monday at SourceMedia Conference’s ATM Debit & Prepaid Forum 08.  True to our Texas roots, we sponsored a Texas Hold ’Em Tournament on Monday evening.  I’ve captured some of the action at the final table.  As people who were there will tell you, it was a fun and exciting time.

Trade Show Set-up

Here we are doing Day One set-up at the ATM Debit & Prepaid Forum 08 in Chandler, Arizona:

Booth by Hugh.  Systems by Dave.  Backbone and yeoman effort by Chuck and Randy.   

“The fundamentals of the [OLS trade show] economy are strong.”

Headed to the Debit, ATM & Prepaid Forum 08

Our team is winging it towards Chandler, Arizona this coming weekend for the ATM, Debit Prepaid Forum 08, ably presented by Source Media Conferences & Events.  We’re a gold-level sponsor. 

Our appearance has been preceded by a lot of hard prep work by my colleagues Hugh Bursi and Dave Bergert.  Hugh’s laid the groundwork for our appearance, from booth design to collateral to special event sponsorship (featuring our own Texas Hold ’Em poker tournament) to shaping message content.  We’ll show complete working, production-ready instances of our acquirer and issuer payment switch applications. 

Dave’s done the spadework on the system- and application-levels, from hardware and software acquisition to application set-up and testing to development of some really great user- and admin-facing tools and facilities.  Read Dave’s post here.  Dave has surpassed his own high standards here.  In a visual, intuitive way, he shows you the tools he’s created for the show, how they enable transaction processing in the demo environment, how they could also be used for production, and how everything fits together for end-to-end processing.

Last night, Hugh, Dave and I came up with this short list of ‘talking points’ about what we’re showing (I’ve masked a couple of things):

1. We are running live, production-ready transaction switches here at the show. [Refer to diagrams referenced in Dave’s post.]
2. Even on this small server (<$1,000), we would be capable of supporting a mid-sized enterprise.
3. The Acquirer side demo features two ways to originate transactions. [Refer to diagrams referenced in Dave’s post.]
4. We can accept all four major card brands in the demo: Our Acquirer solution emulates the sending to a remote authorizer (MC, Visa, Discover --> FDR North simulator; AMEX --> AMEX Simulator).
5. We accept one gift card type (YYY – cards starting with ‘7’) and send that to a Stored Value Systems (‘SVS’) simulator. [We’re showing this for functionality only – don’t imply we’re limited to YYY or that YYY uses SVS.]
6. We’re sending production-certified transaction requests over the FDR North, AMEX and SVS interfaces using their respective message formats.
7. The Issuer-side demo is set up to support a series of selected card BINs – AAA; BBB; CCC; DDD.
8. The Acquirer-side and Issuer-side demos work together to demonstrate the end-to-end processing of your card brand ‘in the wild.’