Andy Orrock | Payment Systems

Solving Visibility Concerns with the jPOS UI, Part 2

Here are some more thoughts about visibility concerns in a payment processing solution...

You put all this time and effort into engineering your jPOS solution. It gets great throughput.  You can on-board new services and endpoints with a repeatable process.   It's reliable.  You get management and user buy-in.  But, in legacy systems, we had this frustration: how do you make things visible?  Payment systems aren't like iPods:  glitzy demos and overall Jobs-like product sexiness is tough to spotlight because all the goodness tends to happen under the covers in payment systems.  I mean, the Gladiators get 900 TPS, a major achievement, but sitting there watching the thing real time, you'd never know without a UI.  Back in the day when I was a Stratus-centric payment processing guy, visibility was always a major concern.  It was usually phone calls from stores that alerted you to a problem.  You don't want to be in that position.  My operations philosophy is to stay on the offense.  Strive to never be reactive...never be wrong-footed.  Alejandro and I share a love of the continuous processing discipline.  Call it Kaizen or whatever.  But the idea to iteratively improve your approach and your results every day.

Playing a big part in all this is the jPOS UI, part of the jPOS-EE Professional Edition Program.   In Part 1 of this series, I tried to give a flavor for the UI in practice by showing some screenshots.  Since then, Alejandro and I have been talking about how to advance those thoughts and give potential jPOS users a better idea of the UI possibilities.  Over the past couple of months, he, Roque and I have been enchanted by Techsmith's Jing Project.  In addition to being a great tool, it's a beautiful marketing strategy by Techsmith because it encouraged me to step up to their top-of-the-line Camtasia Studio offering.

Yesterday, I gave it a first shot.  Despite lots of rookie mistakes in here and a notable word-finding problem by your narrator (me) working scriptless, I was able to piece together something that I think does a better job than the screenshots.  You can view it here (approx. 16:17 length):

Can jPOS handle it?

It's a common sequence of events for people to discover the jPOS project, get intrigued by it, then excited by it, then wonder, hmmmm...is it going to handle my volume?  This concern is usually expressed on two levels:  the technology team is crossing their fingers that jPOS will be the winning ticket, because they always love working with it (as one client put it to me, "we hope management selects jPOS, because that's when the fun starts"); and the management decision-making team is wondering whether he or she can select jPOS without it being a career-killer.  Let's face it, a bad OLTP Payment Systems solution has the potential to be a career killer.  It's mission-critical, high-profile stuff, without question.

Here are some statistics to give you the confidence to march down the jPOS road.  At one client site (a major retailer transaction acquirer), we just finished up a working week that looked like this (number of transactions per day):

• Monday, Sept. 24: 609,365
• Tuesday, Sept. 25: 620,291
• Wednesday, Sept. 26: 610,324
• Thursday, Sept. 27: 612,664
• Friday, Sept. 28: 647,116

This jPOS implementation services 3,650 stores across the US.  Each location has up to 12 register-based points of origin.

Transactions are a mixture of major credit cards (AMEX, MC, Visa, Discover), Debit, EBT, four Stored Value-class requests and some internal applications which account for about 8% of volume. 

Average end-to-end response time for Credit (including external auth) is 0.75 seconds.  Average for Debit (including external auth) is 1.2 seconds (reflects impact of HSM-based PIN processing required by us as acquirer, FDR as gateway, and the issuer).

There are two replicated application server nodes.  There are two database servers configured as a virtual cluster.  [See some recommendations we typically here.]  Each application node runs at about 10 - 12% CPU usage at peak.  The DB virtual cluster runs at abut 4 - 5% CPU usage at peak.  We define peak as the highest observed TPS that is sustained over 30  minutes.  Here, it's about 25 TPS.

On a different level of magnitude, with additional crack engineering, thoughtful design and more hardware firepower, the Gladiators are getting 900 TPS in sustained overnight tests (that's me on the right in both of Alejandro's photos).

How many of these interfaces do you have written?

A co-worker relates a story that he was regaling a legacy client about the advantages of the jPOS-based approach to payment systems.  The customer posed the question:  "We've got 11 different outgoing ISO8583-based interfaces. How many of those do you have written?" 

It's a very interesting question because it reveals the way things used to be done.  Each interface took 30 - 45 days to write - the result being these big-ass (technical term) 'switch control programs.'  These beasts often stretched to thousands of lines.  Then, it was another month to tweak and work out your bugs in ad-hoc testing.  Then, when another interface requirement came along, you'd have to repeat the exercise - or (maybe even worse), you'd start with the just-completed programs as your starting  base, then modify the hell out of that thing.  Then, someone would pick that modified model as their starting point for a third interface.  By the time you get to interface #11, you got a watered-down piece of dreck that would crumble the first time it received an ISO field for which it had not been explicitly coded.  Which, of course, completely invalidates the whole ISO8583 approach.  I've lived the nightmare of supporting those 'products.'   Not fun.

This isn't just an issue with legacy payment products.  Anyone who decides - either through lack of research or some show of brute programming strength - to solve the ISO8583 problem from scratch is susceptible to this same outcome.  We just encountered something like that the other day - a CTO called to say that they had to scrap their ISO8583 efforts, which his team had attempted to solve from scratch.  Oops.  Only subsequently did they happen upon jPOS, which is hiding in plain sight via our friends Larry and Sergey.  This Reinventing the Wheel is the exact problem that Alejandro set out to solve - and did solve - with jPOS.

Now, there are various configuration steps you need to go through in any exercise to bring a new interface on-board.  In our jPOS-based environment, we've got a nice repeatable process defined.  But in this post, I want to focus on the question of the amount of pure 'new' code required to put a nice interface into place in the jPOS world.  Bottom line:  not much at all (specific examples to follow below).  That's mainly because of a couple of things:

1. Despite these big-ass (again, that technical term) specs that these authorizers plop in your lap, there is - at the core - really not a lot of variance between the various implementations out there.  [Okay, there's the occasional oddball interface.]
2. Alejandro has moved a lot of the [what used to be] grunt work to the packager.  As Borat says, "It's nice, I like!"
3. Once you get a base program in place that covers the elements that are the same from interface to interface, you can focus squarely on an extension that tweaks the model to meet the unique qualities of a specific interface.

So, we did this...

We've got a base program in place to set the most common ISO8583 fields - 2, 3, 4, 7, 11, 12, 13, 14, 18, 22, 24, 25, 35, 37, 41, 42, 48, 49 and 54  (see my post on Implementing the jPOS Packager for some insight on those fields).

First, we implemented the FDR North interface, which for us is certified to act as a gateway and do Debit, EBT and Credit (MasterCard, Visa and Discover).  Unique actions taken here were:

• Do endpoint-specific 'getEntryMode' logic to populate Field 22.
• Populate Field 63 (always an endpoint-specific thing if it's in play).
• If it's a credit transaction, put the ZIP Code into Field 59 and the POS (Device) Capability into Field 60.

Total Program Length: 68 lines

Then, we implemented American Express.  Unique actions taken here were:

• Make sure Field 2 (the PAN) is always set (the base program only sets the PAN if the transaction was key entered).  AMEX wants the PAN (as well as the Track II in Field 35) even when the transaction was swiped.
• Go to 'yyMMddHHmmss' model on Field 12.
• Do endpoint-specific 'getEntryMode' logic to populate Field 22.

Total Program Length: 56 lines

Then, we implemented Stored Value Systems (SVS).  Unique actions taken here were:

• Like AMEX, always set the PAN in field 2, regardless of the entry method.
• Put the local time into Field 12 (the base program uses GMT).
• An SVS oddity: they don't put a TID into Field 41, but instead re-purpose it as their 'invoice number.'  So, we stick our Retrieval Reference Number ('RRN') - which typically goes into field 37 - there.
• Then, we unset Field 37 (related to the last point).

Total Program Length: 78 lines

Then, we implemented GreenDot.  Unique actions taken here were:

• Do endpoint-specific 'getEntryMode' logic to populate Field 22.  
• Put the store name, city and state into field 43.
• Put a GreenDot-specific 'function code' into Field 48.
• Put the ZIP Code in Field 59.
• Do endpoint-specific population of Fields 41 (TID) and Field 42 (MID).  The TID is simply the merchant's store number + terminal number.  GreenDot essentially no-ops the MID and asks that we fill it with all nines.

Total Program Length: 59 lines

You get the idea.  There's another couple of interfaces that follow these, but the point is to show you that using the jPOS way, you're very "code-light."  In terms of how you determine the appropriate coding extension for each new interface you on-board, take a look at my On-Boarding Game Plan notes - you'll find them here.