The jPOS Deploy Directory

Recently I wrote a document which described the significance and operational workings of each of the items contained in the ‘deploy’ sub-directory at one of our client sites running a jPOS-based Acquirer implementation. These XML-based system components represent the key ‘touchpoints’ of the jPOS application as implemented. They are the critical control points, so the more you know of their workings, the more capable you will be in terms of support and tuning of a jPOS application.

I thought it might be helpful to list those components here, so everyone can get a flavor for the type of functionality you can define in a jPOS application. Here’s the list:

1. 00_logger.xml

Summary: Governs how system information (e.g., system start, stop; channel connect, disconnect; protected transaction breakout) is placed into files in the log sub-directory.

Implementation Notes: 

• Logger is specified as ‘q2’ (so current log is ‘q2.log’ and previous generations are q2.log.1, q2.log.2, etc.).
• ProtectedLogListener ensures sensitive fields in logs are protected or wiped.
• RotateLogListener defines how many copies to keep, as well as maximum log size in seconds (see ‘window’) or bytes. On cycle, q2.log becomes q2.log.1, the previous q2.log.1 becomes q2.log.2, the previous q2.log.89 becomes q2.log.90, and the previous q2.log.90 is eliminated. This facility is enabled via the application’s use of the ‘log4j’ logging service.

[For further info, see http://logging.apache.org/log4j/docs/index.html.]

• Removal of 00_logger from directory will disable logging but not impact transaction processing. However, the q2.log is instrumental to any attempt at troubleshooting and is a key component of continuous improvement efforts.

2. 01_capture_date.xml

Summary: Defines end of business day. On any given day, transactions received prior to ‘cutover’ time (as measured by server time) are logged to the tranLog table with captureDate = today’s calendar date. Transactions received after cutover time are logged with captureDate = tomorrow’s calendar date. captureDate is also placed into <capture-date> value on VG2 reply.

Implementation Notes: Note that the extract – which works on a captureDate basis – should run (in its current one extract per 24-hour cycle incarnation) after cutover time else transactions will be skipped from the extract. [For more information, see ‘90_extract.xml’ description.] For ‘real’ production, the model can be aligned to a calendar date approach (if desired) by using a value like “23:59:59.”

3. 01_start.xml

Summary: Generates an entry in the SysLog table each time an application server (‘node’) starts, and then another entry each time the node has a clean stop (not a crash); the stop row will include a reference to the total time that the application node was up and running (since the previous start).

Implementation Notes: We set the ‘value’ in 01_start to ‘APP01’ and ‘APP02’ on each respective server so that the start/stop entries on ‘syslog’ table are discernable by node.

4. 01_status_heartbeat.xml

Summary: The status subsystem works in a passive way. Each status entry is 'touched' from the system component that controls it. Each component has an associated timeout. If a status entry has not been touched within the given timeout period, then somebody needs to move that entry from its current ‘OK’ status to the destination error condition (e.g., WARN, CRITICAL, OFF, etc.). When an OLS.Switch UI user hits the status page, we run that check on all entries; but if for some reason the users are not looking at that page, we need a way to check all statuses and move timed-out entries to the specified error condition. 01_status_heartbeat is in charge of that.

Implementation Notes: ‘heartbeat1’ and ‘heartbeat2’ are the two heartbeat IDs defined (one per node). In reality, we just need one heartbeat system wide (because there’s only one database table to scan), but it doesn't hurt to have more than one – the ‘1’ and ‘2’ setup gives the online nodes good parallelism in the UI status dashboard. Different versions of 01_status_heartbeat are placed into the respective deploy directories in the APP01 and APP02 nodes so that unique heartbeat entries appear in the status table.

5. 01_fdr_channel.xml

Summary: Defines the IP address and port of the FDR connection (1 of 2). Also defines the packager used to package and unpackage ISO8583 requests to FDR and responses from them.

Implementation Notes: There are two FDR connections, one to Hagerstown (MD), the other to Melville (NY). 01_fdr_channel maintains the Hagerstown connection. Note that each node (APP01, APP02) makes and maintains a separate connection to this address. FDR must enable each port to accept multiple listeners, else the second connection attempt will fail.

The FDR packager can be found in cfg/fdr.xml.

6. 01_fdr_channel_1.xml

Summary: Defines the IP address and port of the FDR connection (2 of 2). Also defines the packager used to package and unpackage ISO8583 requests to FDR and responses from them.

Implementation Notes: There are two FDR connections, one to Hagerstown (MD), the other to Melville (NY). 01_fdr_channel_1 maintains the Melville connection. Note that each node (APP01, APP02) makes and maintains a separate connection to this address. FDR must enable each port to accept multiple listeners, else the second connection attempt will fail.

The FDR packager can be found in cfg/fdr.xml.

7. 01_amex_channel.xml

Summary: Defines the IP address and port of the AMEX connection (1 of 2). Also defines the packager used to package and unpackage ISO8583 requests to AMEX and responses from them.

Implementation Notes: There are two AMEX connections, one to IPC (AZ), the other to NROC (NC). 01_amex_channel maintains the IPC connection. Note that each node (APP01, APP02) makes and maintains a separate connection to this address. AMEX must enable each port to accept multiple listeners, else the second connection attempt will fail.

The AMEX packager can be found in cfg/amex.xml.

8. 01_amex_channel_1.xml

Summary: Defines the IP address and port of the AMEX connection (2 of 2). Also defines the packager used to package and unpackage ISO8583 requests to AMEX and responses from them.

Implementation Notes: There are two AMEX connections, one to IPC (AZ), the other to NROC (NC). 01_amex_channel maintains the NROC connection. Note that each node (APP01, APP02) makes and maintains a separate connection to this address. AMEX must enable each port to accept multiple listeners, else the second connection attempt will fail.

The AMEX packager can be found in cfg/amex.xml.

9. 10_thales.xml

Summary: Defines the connections to two Thales Hardware Security Modules – ID in status table, Host address of Thales box on the network, ‘well-known port’ defined in the Thales configuration (see Thales manuals for details).

Implementation Notes: For throughput purposes, each of the nodes makes multiple connections to each of the two HSMs. Different versions of 10_thales are placed into the respective deploy directories in the APP01 and APP02 nodes so that unique entries appear in the status table for the Thales connections from each server.

10. 20_fdr_mux.xml

Summary: Defines the in, out, ready and unhandled pieces of the ‘fdr-mux’ QMUX class. [Note that the ‘mux’ here in the naming convention is short for ‘multiplexer,’ as in the multiplexing of messages.] QMUX can multiplex several channels for redundancy purposes.

Implementation Notes: 20_fdr_mux multiplexes messages to two distinct FDR channels (per node). [See FDR channel definitions in 01_fdr_channel (Hagerstown) ) and 01_fdr_channel_1 (Melville).] If one channel connection is down, the multiplexer makes sure all transactions are routed to the surviving channel.

11. 20_amex_mux.xml

Summary: Defines the in, out, ready and unhandled pieces of the ‘amex-mux’ QMUX class. [Note that the ‘mux’ here in the naming convention is short for ‘multiplexer,’ as in the multiplexing of messages.] QMUX can multiplex several channels for redundancy purposes.

Implementation Notes: 20_amex_mux multiplexes messages to two distinct AMEX channels (per node). [See AMEX channel definitions in 01_amex_channel (IPC) and 01_amex_channel_1 (NROC).] If one channel connection is down, the multiplexer makes sure all transactions are routed to the surviving channel.

12. 20_keystore.xml

Summary: Defines where OLS.Switch (that’s the name of our jPOS implementation) can find the Base Derivation Key (‘BDK’) required for encrypting and decrypting cardholder information (part of our PCI/CISP compliance approach).

Implementation Notes: The BDK must be locked down in a secure directory location.

13. 20_saf.xml

Summary: Defines how the store and forward (‘SAF’) features associated with the fdr-mux function. [All reversals are handled via the SAF feature.]

Implementation Notes: Note that ‘initial-delay,’ ‘inter-message-delay,’ and ‘wait-for-response’ values are in milliseconds, while ‘expire-after’ is in seconds. The following response code configuration…

<property name='valid-response-codes' value='*' />
<property name='retry-response-codes' value='91,NL,NM,N8,CE' />

…can be interpreted as follows:

• If ISO Field 39 on the FDR response is 91, NL, NM, N8, or CE, then some type of transient error has occurred on the FDR side, so maintain the item at the top of the queue for retransmission.
• For all other values received in the ISO Field 39 reply to SAF-initiated requests, accept the response as valid and remove the item from the SAF queue.

14. 20_sm.xml

Summary: Defines where OLS.Switch (that’s the name of our jPOS implementation) can find the Local Master Key (‘LMK’) required for encrypting and decrypting cardholder information (part of our PCI/CISP compliance approach).

Implementation Notes: The LMK must be locked down in a secure directory location.

15. 30_ev_txnmgr.xml

Summary: This is the most intensive and largest member of the deploy directory. It is the “transaction manager” for Debit, EBT and Credit transaction types. [NOTE: We chose to forward Stored Value card transactions into separate transaction managers so that slower seasonal traffic from SV authorizers does not impact core Debit/EBT/Credit processing.] The transaction manager defines a series of ‘participants’ which control the effective processing path of a transaction as it navigates from the point entry into the node and back out again. It controls such elements as parsing the transaction request, populating the transaction log, building an outbound request to the external authorizer (e.g., FDR), calling the Thales adapter (for Debit and EBT), handling the response from the external authorizer (and reversing through SAF if required) and building the VG2 reply back to the store system.

Implementation Notes: Note that 30_ev_txnmgr references a series of include books:

<!ENTITY translate_pin SYSTEM "translate_pin.inc">
<!ENTITY store_and_forward SYSTEM "store_and_forward.inc">
<!ENTITY validate_terminal SYSTEM "validate_terminal.inc">
<!ENTITY query_host  SYSTEM "query_host.inc">
<!ENTITY query_host_or_reverse SYSTEM "query_host_or_reverse.inc">
<!ENTITY debit_response SYSTEM "debit_response.inc">
<!ENTITY ebt_response SYSTEM "ebt_response.inc">

These items can also be found in the deploy sub-dir. There are some important touchpoints in those include books. For example, query_host and query_host_or_reverse contain the time allotted to the external authorizer to come back with a response. Currently, the value is set to 25 seconds. The terminals at this location have a 30-second timeout; 25 seconds allows for transport time from and back to the device. These items also include the important ‘threshold’ parameter value setting. The ‘threshold’ setting implements a ‘source-based timeout’ mechanism – before committing to send a transaction out for external authorization, the total time elapsed on the transaction so far is compared to the threshold. If elapsed time exceeds the threshold value (indicative of internal problems slowdowns), then the transaction will be internally rejected prior to committing to external routing.

16. 30_fdr_logon_mgr.xml

Summary: Governs the application logon and various application parameters involving the connection to FDR, including: how often to send the 0800 logon message; how often to send the 0800 echo message; where to find the ZMK (and other applicable keys); the pseudo MID and TID to use on the dynamic key exchange, etc.

Implementation Notes: The ZMK is used to request and receive a new ZPK (the ‘working key’). Typically these are stored in memory. But because we run two replicated application nodes and FDR only ‘sees’ its customer as one acquirer entity, the working key must be placed into the SQL database for cross-node usage.

17. 30_amex_logon_mgr.xml

Summary: Governs the application logon and various application parameters involving the connection to AMEX, including: how often to send the 0800 logon message and how often to send the 0800 echo message.

18. 50_ev_server.xml

Summary: This is the transaction server. Despite the ‘ev’ moniker, this is actually the server for all transaction types. The ‘listening port’ and transaction queue are defined here, as is the ‘space’ which is the application’s general purpose coordination component.

Implementation Notes: The underlying jPOS transaction engine provides several space implementations. 50_ev_server uses ‘TransientSpace’ (a.k.a., ‘tspace’), a local (i.e., accessible only by that one node), in-memory space implementation.

19. 50_fdr_unhandled.xml

Summary: Defines the queue for ‘unhandled’ responses from FDR. [Typically, these are the late responses to transactions already timed out and reversed (via SAF) by OLS.Switch.]

Implementation Notes: All late, ‘unhandled’ replies result in an entry being placed into the syslog table with the ‘source’ value of ‘FDRDELAYED’.

20. 50_amex_unhandled.xml

Summary: Defines the queue for ‘unhandled’ responses from AMEX. [Typically, these are the late responses to transactions already timed out (AMEX transactions are not reversed upon Host timeout).]

Implementation Notes: All late, ‘unhandled’ replies result in an entry being placed into the syslog table with the ‘source’ value of ‘AMEXDELAYED’.

21. 90_ev_import.xml

Summary: The presence of this file in the deploy directory means that this application instance will kick off ‘import’ via jPOS’ ‘DirPollAdaptor’ facility whenever an cardholder file is dropped into the OLS\spool\request directory. DirPollAdaptor moves the file to the spool\run sub-directory, uses that input file to update the card and cardholder tables, and moves the processed input file to OLS\spool\archive (with an appended date on the file name) upon completion.

22. 90_extract.xml

Summary: Defines the time the daily extract will start.

Implementation Notes: 90_extract works very closely (concept-wise) with 01_capture_date (see above). Unless you plan to run multiple extracts per day (additional extracts can be run ad-hoc via the UI), the capture date MUST be pushed forward prior to extract execution lest you will miss transactions. The automated execution uses the 01_capture_date value as its pivot point. To explain, consider these two scenarios:

·     Value in 01_capture_date set to 13:45:00; value in 90_extract set to 14:00:00. When 90_extract runs, it examines 01_captureDate value and compares to current time. It sees that current time is greater, so it assumes extract is to be run on transactions containing today’s captureDate.

Using the values in this example, an extract run at 14:00:00 on July 16^th will seek to extract all transactions where the value of the captureDate column (for all transactions where reconID = 0) is July 16^th (note that since MS SQL Server doesn’t have a ‘date’ variable type, captureDate is implemented as dateTime, meaning that a 00:00:00.000 time is appended to all captureDate column values).

·     Value in 01_capture_date set to 23:59:59; value in 90_extract set to 00:05:00. When 90_extract runs, it examines 01_captureDate value and compares to current time. It sees that current time is less, so it assumes extract is to be run on all transactions containing yesterday’s captureDate.

Using the values in this example, an extract run at 00:05:00 on July 17^th will seek to extract all transactions where the value of the captureDate column (for all transactions where reconID = 0) is July 16^th.

23. 90_jetty.xml

Summary: The presence of 90_jetty enables an HTTP server, thus allowing the UI to run in any application instance referencing this deploy directory. According to Wikipedia, “Jetty is a pure Java based HTTP Server and Servlet Container. Jetty is released as an open source project under the Apache 2.0 License. Due to Jetty's small size, Jetty is suitable for providing web services in an embedded Java application.” [For further information, see http://en.wikipedia.org/wiki/Jetty_%28web_server%29.]

Implementation Notes: The file acts as little more than a marker. Jetty configuration information can be found in cfg\jetty.xml.

24. 90_thales_ping.xml

Summary: This component pings the Thales with its diagnostic command (‘NC’) to ensure connectivity. The Thales – if properly communicating – will respond with an ‘ND’. This command exchange is especially useful during periods of low activity.

Implementation Notes: Currently the diagnostic command is set to send every five minutes. This value ensures that the Thales will not time out/reset any of the connections.

25. 95_fdr_monitor.xml

Summary: Establishes an entry in the status table (and on the UI) for one of the two FDR connections.

Implementation Notes: This is the Hagerstown (MD) connection. Different versions of 95_fdr_monitor are placed into the respective deploy directories in the APP01 and APP02 nodes so that unique entries appear in the status table for the Hagerstown connection from each server.

26. 95_fdr1_monitor.xml

Summary: Establishes an entry in the status table (and on the UI) for the other of the two FDR connections.

Implementation Notes: This is the Melville (NY) connection. Different versions of 95_fdr1_monitor are placed into the respective deploy directories in the APP01 and APP02 nodes so that unique entries appear in the status table for the Melville connection from each server.

27. 95_amex_monitor.xml

Summary: Establishes an entry in the status table (and on the UI) for one of the two AMEX connections.

Implementation Notes: This is the IPC (AZ) connection. Different versions of 95_amex_monitor are placed into the respective deploy directories in the APP01 and APP02 nodes so that unique entries appear in the status table for the IPC connection from each server.

28. 95_amex1_monitor.xml

Summary: Establishes an entry in the status table (and on the UI) for the other of the two AMEX connections.

Implementation Notes: This is the NROC (NC) connection. Different versions of 95_amex1_monitor are placed into the respective deploy directories in the APP01 and APP02 nodes so that unique entries appear in the status table for the NROC connection from each server.

29. 95_saf_monitor.xml

Summary: Establishes an entry in the status table (and on the UI) for the store and forward queue affiliated with the FDR connection. Regardless of the number of connections/channels to an endpoint, there is only one SAF queue per endpoint per node.

Implementation Notes: Different versions of 95_saf_monitor are placed into the respective OLS\deploy directories in the APP01 and APP02 nodes so that unique entries appear in the status table for the SAF queue on each server.

30. 99_sysmon.xml

Summary: Presence of this item in the deploy directory results in the executing application log placing diagnostic (see XML tag ‘<SystemMonitor>’) information into q2.log at intervals specified by the ‘sleeptime’ parameter. Information includes memory usage, thread summary and a review of the name registrar. These entries are important debugging and troubleshooting devices.

Comments

[Andy Orrock]

A correction on this post:

It's actually 02_status_heartbeat.xml, not 01_status_heartbeat.xml.