tcVISION is used for the real-time replication and Apache Hadoop is used as a cost-efficient system for storing the data. This can cover additional use cases, such as real-time event handling & stream processing, analytics based on real-time data as well as the possibility to report and analyse structured and unstructured data with excellent performance. The system can be inexpensively operated on Commodity Hardware and has no scalability limitations. Considering the savings, the costs of replication (CPU consumption) with tcVISION are very low.

tcVISION performs a very economical replication based on log files. Numerous application cases based on the replicated data, which would be too expensive on the mainframe, can be implemented now.

The distribution of the subscription data is performed by tcVISION.

The subscribed data is captured in real-time from the active log of a shared Db2 system on a z/OS system. The capture mechanism of tcVISION uses the Instrumentation Facility Interface of Db2. The changed data is transferred to an Intel based Linux system via TCP/IP, where the distribution of the data records to the customers is performed based on data record attributes.

Technically this was realised using a bit string as an integer attribute for the relevant tables. A comparison is performed for every replicated record by using a logical bit compare with different comparison values to check whether the record should be distributed to the customer. If the bit comparison is not successful, the record is rejected.

In addition to the bit string logic, a comparison based on a Lookup SQL and character string comparisons are performed. To serve each individual customer based on his requirements, a subscription system has been developed that only transfers the stock exchange values to which the customer has subscribed.

All processing definitions are stored in a central tcVISION repository.

The tcVISION replication is based on a central repository which can be stored in any relational database, like Db2 or Oracle. The metadata, transformation and replication rules required by the processing are part of that repository.

tcVISION uses the Db2 Instrumentation Facility to capture the changed data in real-time from the active Db2 log. A combination of archived and active logs is used for the batch processing.

The tcVISION Managers on z/OS are connected to other tcVISION Managers (z/OS and z/LINUX) via TCP/IP. The changed data is applied to the target systems either via Db2 or the Oracle Call Interface (OCI).

The tcVISION Control Board (GUI) is used to monitor all systems and to define and implement the replication scenarios.

The replication is fail-proof and can be restarted at any time.

The production implementation of tcVISION supports more than 370 Db2 tables. The operational changes to mirrored Db2 tables are replicated on the Windows replication server. The replication processes are part of the daily IT operations. The online systems and batch applications are responsible for the changes applied to Db2. All tables that are supposed to be replicated have the attribute ‘DATA CAPTURE CHANGES’ and all the changes are kept in the Db2 log. The log files are sent to the replication servers using FTP processes as soon as the online logs are archived by Db2. Once received, tcVISION processes the log files on the replication server. To process the log files and capture the changes, tcVISION has to understand the structures of the Db2 tables. This metadata is stored in a central repository which is also stored in the Db2. tcVISION provides import functions for creating the metadata from the Db2 source database. This metadata is used for the initial load of the Db2 tables to the server as well as for the processing of the Db2 log files.

The initial load of the Db2 tables on the replication server is performed by the BULK TRANSFER function of tcVISION. The input is a Db2 imagecopy that is sent to the replication server via FTP. The copy is processed and loaded to Db2 with the help of a LOAD statement generated by tcVISION. Then, the tables are available on the replication server and changes can be applied via the log file processing.

A special challenge was presented by the Db2 Compression Dictionaries and the fact that the structures of the tables can be changed during the daily operations. The information about the compression algorithm in use is scanned by tcVISION while importing the metadata and stored in the central repository. Each table is stored in a table space of its own. The different dictionary versions and corresponding compression algorithm are kept in the repository, so tcVISION can always use the correct algorithms for the decompression when processing the log files or imagecopies. The metadata reflects the structure of the data at the time of the import. tcVISION historicizes this structural information during a new import after a structure change. This way, tcVISION recognises changes of the data structure and is able to apply the changes to the target tables in accordance with the structure in place at the time the change was performed. That is why all Db2 structural information is refreshed every day by an automated process. This automated process guarantees the correct metadata and compression information to be used. tcVISION provides tools and utilities to delete all the historical repository entries that are no longer required.

The daily processing of all the Db2 changes is nearly completely automated.

The replication is bidirectional. Both systems are of equal priority and changes applied to both systems must be replicated. The near real-time replication method provided by tcVISION is used. tcVISION extracts all changes found in the Db2 Active Log for tables with the 'Change Data Capture' attribute and replicates them to the partnering system. Since the replication is bidirectional, it is important to assure that changes already applied to one system are not replicated back. tcVISION provides functions to check this in the replication scripts, so a possible loopback effect can be avoided. The replication between the Db2/zOS and the Db2/LUW uses a dedicated LINUX machine.

The synchronization scenario is based on changed data capturing on the mainframe in real-time. The Adabas extension of tcVISION captures the changes and these changes are then propagated to a LINUX system using a mainframe collector and data pool storage. The LINUX system hosts an Oracle database that acts as a mirror of the Adabas databases. tcVISION directly propagates the mainframe changes into the Oracle database.

Both platforms act as equals (master/master). The tcVISION DBMS extension for Adabas captures changes on the mainframe in real-time and replicates those changes into an Oracle mirror database, from which the changes are applied to the Oracle production database.

The replication path from Oracle to Adabas on the mainframe is maintained through Oracle and the JDBC component of tcACCESS.

A tcACCESS Stored Procedure developed in PL/I obtains the Adabas Internal Sequence Number (ISN) for new inserted records for the Oracle environment.

The bidirectional replication guarantees that changes already applied on one of the platforms are not replicated back via the capturing mechanisms (tcVISION Loopback processing).

Log files are created to document the correct replication scenario.

Through tcVISION, a method to recognise bidirectional changes and securely exclude multiple changes was developed.

tcVISION is mainly used to offload VSAM files on a time-controlled base and bulk load them into SQL tables with similar structures. Approximately 30 million data records per week are transferred with this method.