Andy Orrock | Payment Systems

ISO8583 Secondary Bit Map

There was some good dialogue over the past 24 hours on the jPOS Users mailing list about how to implement the ISO8583 secondary bit map.  I find that most ISO specs gloss over this concept or don't do a good enough job explaining the implementation.  It's a fairly tricky concept.  I bet I've had to explain it in-depth about half a dozen times over the past year.

The best and most comprehensive description of the secondary bit map is something I ran across in Total Systems' ('TSYS') ISO8583 specs.  It says:

BIT MAP, SECONDARY

The first bit of the Bit Map, Primary indicates the presence or absence of a second map called the Bit Map, Secondary. Like the Primary map, the secondary map is a control field consisting of 64 bits (8 bytes). It can be considered an extension of the primary map, since it is associated with fields 66 through 128. Data field 65 does not exist. This position is used to indicate the present of another bit map. However, as no third bit map is currently defined, the first bit of the secondary bit map must always 0 (zero). The Bit Map, Secondary is included only when the message contains information in any field from 66 through 128. When present, the secondary map immediately follows the primary one and precedes the data fields.

Perfect.  Hey, they even describe the possibility of a tertiary bit map.  You can't get any more comprehensive than that.

So, you implement the secondary (and placeholder for the tertiary!) bit map(s) as follows:

<isopackager>
  <isofield
      id="0"
      length="4"
      name="MESSAGE TYPE INDICATOR"
      pad="true"
      class="org.jpos.iso.IFA_NUMERIC"/>
  <isofield
      id="1"
      length="16"
      name="BIT MAP"
      class="org.jpos.iso.IFB_BITMAP"/>
  <isofield
      id="2"
      length="19"
      name="PAN - PRIMARY ACCOUNT NUMBER"
      pad="false"
      class="org.jpos.iso.IFA_LLNUM"/> 

[snip]
 
  <isofield
      id="65"
      length="1"
      name="BITMAP, EXTENDED"
      class="org.jpos.iso.IFB_BINARY"/>

NOTE:  That's a packager that implements an ASCII, character-based implementation with a BCD bit map.  While your choice of classes will vary based upon your specific implementation needs, the roles of Fields 0, 1, 2 (showing 2 here simply as a boundary for the Bit Map) and 65 should look exactly like this.  I've got half a dozen different ISO8583 implementations running in jPOS, and each one looks like this in terms of the names and roles.

As Alejandro noted in his reply to the e-mail that started this discussion, jPOS takes care of the secondary bit map behind the scenes.  It doesn't need to be explicitly defined.  In other words, you see the bit map here is 16 in length.  If Field 1 is present, then the bit map you receive or build is 32 in length...but the underlying jPOS packager takes care of constructing (or deconstructing) that for you.  That's the beauty of what Alejandro has done:  everyone around the world was dealing over and over again with these quirky ISO8583 complexities, but he solved them for everyone once and for all.  As I've noted before:  if you're about to tackle an ISO8583-based project, why reinvent the wheel?

ISO8583 Starting Points

A lot of readers come to my blog as a result of doing searches like "ISO8583 download" or "ISO8583 XML Schema."  That's understandable since ISO8583 is the 'lingua franca' of payment systems.  And, as I've mentioned many times, facilitating the implementation of ISO8583 is the original challenge that sent Alejandro along the path to his jPOS vision and its increasing success and market acceptance. 

Yesterday, a had a nice chat online with a work friend, Vipul.  The two of us just concluded a successful payment system project (my client's system is now 'talking' to his company, which provides stored value authorization services).  Now, Vipul is interested in doing a 'deeper dive' into ISO8583.  He asked for some recommendations.  I'll recap them here:

• You can buy a copy of the spec from ISO here, but you can see that it's a pretty steep price.  [This document is a good luxury to have in your possession.  But, it's not essential to your ability to learn the standard.]
• Alejandro's key document - the jPOS Programmers' Guide - has an outstanding nine-page 'primer' on ISO8583.  Even if you're not a programmer, if you want to know ISO8583, you've got to read his write-up because - better than any official doc - he gets to the essence of the thing and how to construct and deconstruct messages.  If you're serious about learning, then the Guide is something you need to have for your reference.  So, please - Do the Needful and make this excellent investment.  And, even if you're not a Java programmer, its still a good reference.
• Some of you seek to tackle ISO8583 in a way in which you separate yourself from a deep-dive into the standard.  This is the reason for the "ISO8583 XML Schema" searches.  Your best path to success there is to consider the jPOS 'ISOBridge' offering.
• Those of you who like what they see in jPOS but are a C shop, not a Java shop, ought to consider Alejandro's ISO8583 C Lightweight Library.
• I think this is my best post on ISO8583 and Getting Things Done in jPOS - its called Implementing the jPOS Packager.   I go step-by-step on how you take someone's ISO8583-based auth interface spec (AMEX in my example) and translate that into an implementable model.

My key point is always that the best single thing you can do is to study how other institutions have implemented their ISO8583-based auth interfaces - in the right-hand column here in my blog, you'll see references as to how to obtain the AMEX and FDR specs.  In both cases, you have to go through a short registration process, but it's free and there's no commitment.  These are much better references for you than the standards doc.  It's a tough leap from that fairly antiseptic document into a working model.  By contrast, with the AMEX and FDR specs you can see how others have implemented the standard, pick up best practices that you can leverage, note the similarities and divergences in the approaches, etc.  I'll make special note of the FDR North platform choice - we've a working implementation of that interface.  Since it encompasses Debit, EBT and Credit, you get a real flavor for a full range of working techniques as put into practice.  And, with Debit/EBT, the bar is raised really high since you've got to tackle real-time reversals, PIN translation and dynamic key exchange.

ISO8583:2003 and jPOS

A jPOS Users list member asked how jPOS could be adapted to work with an implementation of the ISO8583 2003 standard.  Murtuza answered that "it should be a matter of creating a new packager defining the field formatter and using it, i.e., define it in XML and use the generic packager to load it for use."

Right.  The Generic Packager gets the "most bang for the buck" by adhering to the 1987 and 1993 standards, which are still the most prevalent ones in field use today.  [The three standards are 1987, 1993 and 2003.  Amongst large authorizers, American Express stands out by being one of the only ones to make the leap even from 1987 to 1993.]  jPOS' GenericPackager has 'out-of-the-box' XML files for the 87 and 93 versions.

Here's some info I put together to further flesh out Murtuza's answer:

In ISO8583:2003, for example, the STAN goes from six bytes to 12 bytes in length.

So, the Generic Packager contains this definition to begin with…

  <isofield
      id="11"
      length="6"
      name="SYSTEM TRACE AUDIT NUMBER"
      pad="true"
      class="org.jpos.iso.IFE_NUMERIC"/>

…and you’d change it to look like this:

  <isofield
      id="11"
      length="12"
      name="SYSTEM TRACE AUDIT NUMBER"
      pad="true"
      class="org.jpos.iso.IFE_NUMERIC"/>

Similarly, you’d make a sweep through the entire packager for all 128 fields (or at least the sub-set “in play” for you).

I’d recommend a ‘two-step sweep’ exercise:

1. Go through the generic packager XML file provided with the jPOS download and adjust those items like the STAN as indicated.  [You can find a list of these changes in “Annex F” of the ISO8583:2003 spec provided by the International Standards Organization.] 
2. Secondly, you’ll want to go through the spec provided by the authorizing organization (or gateway) to you (that’s assuming, of course, that your exercise is to put in place an ISO implementation created by someone else).  When you get into “private use” fields like 62 and 63, the Standards spec gives you only rough guidelines.  You’ll want to see what the authorizer/gateway has done in its implementation.  [Unfortunately, for most of those private use fields, the packager definition will only get you so far.  These private use fields typically have esoteric, proprietary table structures inside of an ‘IFE_LLLCHAR’ (or similar) construction.  In those cases, you’ll need some in-line code to complete the packaging and unpackaging tasks.]

Alejandro posted this follow-up (which provides more clarification on my previous sentence about the need for in-line code in some places): 

There's a twist with composite elements...some of them use standard TLV format and others (71, for example) use a bit-map format.  I believe that it may require the creation of a customized ISOFieldPackager in order to handle these.  I had to do something like that while implementing ANS X9.15 which uses five 32-bits maps.

[NOTE: 'TLV' here refers a 'Table, Length, Value' construction of field contents.]

AMEX Merchant/Acquirer Specs Are Online

American Express is another card-issuing and/or gateway organization that puts its ISO8583 spec online.  I suggest you grab a copy. Look for it here:  http://www.americanexpress.com/merchantspecs. The AMEX spec is a good reference in terms of being able to review and study a well-tested, widely-used spec which is inherently multi-national.   There are a lot of good practices in there that you can adapt for your own usage.  It's also a good idea to start assembling a repository of different specs for your reference.  Speaking from experience, I know I can address a lot of the questions that appear on the jPOS.org mailing lists** by being able to reference our spec repository to see how major players like AMEX have addressed a particular issue.

My direct experience working with American Express is that the people you deal with in certifications and testing are top-notch.  It's a pleasure to work with people who really seem to understand the task at hand, as opposed to being just professional project managers who really don't understand the underlying task.  The Amex crew in Phoenix has always impressed me as having the best team in the payment industry in that regard.

Amex has entitled its auth spec “Global Credit Authorization Guide (v3.1).” The v3.x-level designation is notable because it is the version level that lays out a series of small enhancements related to Amex’s Card Acceptance Processing Network (‘CAPN’) initiative, which are:

Additional point of sale security – See Amex spec ISO Field 53 regarding the option of passing the key-entered, four-digit ‘CID’ (from face of card) if card number itself is manually entered. [Implementing this change would require store system changes and special certification from Amex.]

Support for expanded amounts – See comments in ISO Field 4.

Most importantly, the introduction of a ‘transaction lifecycle identifier’ – This identifier goes into Field 31 (Acquirer Reference Data). The ID is 15 bytes (field is LLVAR).  The lifecycle identifier works much like Visa’s PS/2000 Transaction ID in that the value received in the authorization response must be placed into the corresponding settlement record.  The ‘transaction lifecycle identifier’ is the linchpin of CAPN.

The Amex Spec discusses how this field ties into subsequent data capture effort:

“The value in this field must be retained by the merchant’s system and returned to American Express in the (transaction detail) financial settlement records that correspond to this authorization response.”

** The jPOS mailing lists are:

jpos-dev@yahoogroups.com (the developers' mailing list)

jpos-users@googlegroups.com (the users' mailing list)

FDR's ISO8583 Specs Are Online

One of the best ways to familiarize yourself with ISO8583 as it is used in practice is to review a 'real world' specification as implemented by one of the payment systems industry's major players.  First Data (a.k.a. FDR or FDMS) posts their specs online.  You can sign up here:  http://www.fdms.com/specs - there's no checking or vetting that goes on.  First Data welcomes your interest.

These specs are good resources for a couple of reasons:

1. They're well-written.
2. They've been proven to work in production across a wide range of industries.
3. FDR does a good job keeping them up-to-date and relevant.

FDR being an acquisitive organization, there are specs for a number of different 'platforms' found there.  The list currently includes Atlanta, Nashville, North, Omaha, PayPoint and South.  Nashville, for example, is probably the old 'Concord/EFS' interface. 

Where to start?  We've implemented the host-to-host (a.k.a. 'leased-line') version of the 'North' interface.  It's a good one.  To find it, select North in the "Platform" drop-down and keep all other selections as "- All -".  In the match list that is compiled for you, you want document No. 4 - ISO 8583 Format Authorization Network Processing Specifications.  [NOTE:  Remember, this is a spec from First Data Merchant Services (FDMS), so the spec is all from an acquirer's viewpoint.]

Note that some times to get the complete picture, you'll need to augment the main spec with what FDR calls its 'QRGs' or Quick Reference Guides (see right-hand column of the match list).   Typically, these are industry-specific guides like Airline and Rental Car requirements that would otherwise clutter up the main document for the masses.   In this case though, there's one really important QRG you want to download: the one entitled North Authorization Guide ISO8583 Debit.  With more and more retailers pushing BIN lists to the POS with the goal of prompting or forcing debit, transaction mixes are moving inexorably away from credit and towards debit.  So, that 11-page QRG is quite a significant addendum.

For those of you using jPOS, please note that it's your responsibility as a developer to read these specs (or one like it) in their entirety on a field-by-field basis and then build the appropriate ISO Packager according to the guidelines laid down in the jPOS Programmers' Guide.