DTS/NV: Features and benefits

Audit trailing

DTS/NV includes a comprehensive mechanism for querying transactions over an elapsed period of time, as well as filtering and collating the results in accordance to user-defined criteria. In our experience, most audit trail viewers allow the user to filter transactions based on a single criterion, where the record is included in the results if it matches the criteria, or omitted otherwise. This is illustrated in the figure below.

The audit trailer in Night Vision is not dyadic in that respect, but operates around the premise of collations. While this mechanism can be used to filter records in the traditional way, i.e. show or hide, the notion of collations allows the audit trailer to display all records, but to treat each record differently, depending on which collation it is a part of.

Every record that shows up on the audit trail is treated as an object and can be inspected using the Night Vision object inspector. This component presents the object visually, showing all of its public attributes and allows the user to "drill" into the transaction object to see what persistent objects it affects and the nature of the change. Through hyperlinks, the user can navigate to and inspect the affected objects, as well as inspect the reverse chain of transactions that the original transaction depends on. Together, with the combination of the audit trailer and the object inspector, the user can gain a complete insight on the state of the system. All this achieved through visual tools, such as VisualObjects (see following sections), and without writing a single query.

Statistics and trends

The notion of collations in the Night Vision audit trailer can be used to build statistical trend data over the collated records. For example, if a company predominantly sold two widgets A and B, Night Vision can filter the records to only include widget sales over the last three months, and then collate those sales by each widget, with a separate collation for widget A, widget B, and an optional "all others" collation to include the minority product lines. Collated sales data can then be aggregated on a per-week basis, and displayed as a stacked area chart, while the overall "spread" of data aggregated over the entire period can be displayed as a pie chart.

A fairly involved data analysis report such as this one takes literally seconds to produce in Night Vision. Data produced as a result of statistical analysis can be displayed in a variety of formats and exported for use elsewhere in the organisation. Collations are not simply an evolution, but a revolution of the area of transaction processing that is concerned with transaction auditing and analysis. Previously, it has never been possible to trace transactions while simultaneously looking for several predicates without conducting a number of independent traces with one predicate in each, and then combining the data in an application with charting and statistical analysis capabilities, such as Minitab or even a spreadsheet. With Night Vision collations, you do the trace once and let the built-in aggregation and charting capabilities do the work. It's faster and more streamlined - there are no third party tools to tinker with.

Entity lifecycle management

DTS stores business entities as persistent objects. These undergo a life cycle, commencing from their creation, and spanning all the way to their deletion, as depicted in the figure below.

Night Vision provides customisable means for graphically presenting those entities on the management console. These include wizards, editors and validation mechanisms to create, locate and modify entities. Building a wizard for creating a new user account, catalogue item, stock code or whatever entity your business deals with is a trivial undertaking with Night Vision. Similarly, it is straightforward to create GUIs for searching and then editing business entities, as per the call centre scenario. The Night Vision distribution provides numerous documented examples on creating fairly advanced entity management components - all of which can be tailored to your model in no time. All of the heavyweight work of providing and managing the GUI components is handled by the framework, while the application developers concentrate on fleshing out the business-specific rules.

VisualObjects ^(TM)

DTS differs from other database management products in that the latter choose to do all of their data querying, mining and analysis though either some flavour of SQL (Structured Query Language) if it is a relational product, or OQL (Object Query Language) or a proprietary query language in the case of an object-oriented system. In most cases, a single query language is the only means of exploring the state of the system, ranging from complex data mining to simple tasks like "get me user X". Being consistent as it may, it poses a serious challenge to users who are, firstly, not familiar with SQL and, secondly, not willing to use the proverbial "sledge hammer" to kill a fly.

While still offering a "sledge hammer" (see Remote Object Query) for the more advanced users, particularly data mining and analysis experts, DTS presents the user with a world-first, "light-weight" query language. VisualObjects is not strictly speaking a language, per se; it is a user interface that presents objects from a graphical perspective, depicting their fields and referenced objects as graphical components on the screen. VisualObjects is, effectively, a query builder where simple to moderately complex queries can be constructed visually within seconds with no prior experience in query languages. Most importantly, VisualObjects can not only be applied to persistent objects, but also to transactions and can be used to filter and collate records for statistical analysis. VisualObjects has been designed from the outset to be extremely intuitive and easy to use. In fact, our initial trials have shown that users with no prior exposure to VisualObjects can issue queries and locate the correct objects without any direction from our observers.

Given our own experience of working with query languages, and that of the people who have been involved in the trialling of VisualObjects, we have come to a realisation that this technology is the holy grail of object query languages. As such, we have made it a core part of DTS/S1 and Night Vision, and have provided mechanisms by which it can be extended. VisualObjects does not displace ROQ, our mainstream query language, but provides a lighter, simpler, albeit less powerful alternative for use by people who's skill set includes day-to-day business administration, not a degree in computer science.

Remote Object Query (ROQ)

Several man-years of experience in working with, designing and building database query languages have culminated in a technology we have called Remote Object Query (ROQ). Considering ROQ (pronounced 'rock') from the perspective of functionality, ease of use and extensibility, it is a superset of all query languages that have stood before it, including SQL, OQL and proprietary languages that have been build by the authors of ROQ. ROQ is an XML-based, template and pattern-oriented query language that is a hybridisation of both declarative and procedural programming paradigms. The development of ROQ has been inspired by functional languages, such as Prolog, and the template-oriented XSL transform.

Like SQL and OQL, ROQ can be driven purely declaratively, by telling it what needs to be done, rather than how. Unfortunately, declarative programming suffers from poor scalability - simple things are easy to write, because the system already knows the "how" part. Less trivial queries can become very difficult to write because the "how" tends to be declared by the user, using "what" semantics. People with experience with SQL and OQL will immediately identify with this statement. Writing complex expressions using declarative semantics in most cases decreases code readability - writing what could be a few lines in a procedural language often evolves in a completely incomprehensible declarative expression. The collateral in this approach is not only readability, but also performance.

More so, declarative SQL and OQL expressions are inherently limited in functionality to the build-in functions that have been scoped by the language designers. Consider, for example, a query that computes the standard deviation of sales figures for customers in a particular region. Had we asked for a sum or an average, we could have used the corresponding built-in functions in SQL. However, we have asked for more than what SQL/OQL can offer, but we have no way of extending their internal capabilities - defining a new function, perhaps. By contrast, ROQ can employ declarative programming to define an overall query, but then break the query into parts, some of which can be carried out procedurally. The procedural elements of ROQ have features equivalent of functions, classes, looping and conditional constructs. Best of all, the use of these constructs if fully optional: where a query does not require extended functionality, it can be depicted completely using declarative semantics.

One fundamental construct in ROQ is a pattern. A pattern's behaviour can be compared to a declarative query, or a function in a procedural language. The difference is that unlike functions and subqueries, patterns aren't invoked directly, but are chosen by the ROQ runtime automatically when required. The pattern-oriented nature of ROQ enables complex queries to be broken up and matched across several simpler patterns, rather than one monolithic query. Both SQL and OQL force developers into employing nested queries and the level of nesting grows with the complexity of the query. In our experience, we have encountered SQL queries boasting 7 levels of nesting, and we are confident that it gets much worse than this. Needless to say, the merits of such queries will be overshadowed by the software maintenance overhead they require. By contrast, ROQ allows the user to modularise any query, however complex, without requiring a single level of nesting.

ROQ is a standalone part of the DTS product line that has been integrated into Night Vision. ROQ is distributed with a level 4 JDBC driver so that it can be driven programmatically and integrated into other products. ROQ supports query pre-compilation, so that queries tunnelled through the JDBC driver can be compiled on the client and re-used in their compiled form, thereby positively affecting performance. Like all aspects of Night Vision, the ROQ JDBC drivers are completely secure, employing the AES cipher with 256-bit key strength, RSA key exchange and X.509 certificates.

With the combination of VisualObjects and ROQ at our disposal, we wanted to completely depart ourselves from the traditional "Swiss army knife" approach of carving out queries with a single, morally outdated and frankly cumbersome instrument. Having experienced the alternatives, you too will struggle to comprehend how languages such as SQL's and OQL's, given their complete lack of scalability and extensibility, can still be entrusted to any serious data mining work.

Internationalisation

Night Vision has localisation built into the core of the framework. There is a simple mechanism for translating all text labels, errors, prompts, date and currency formats, even multimedia content into the locale where the management console is run. Best of all, the entire mechanism is completely transparent and self-configuring. And if a translation for a particular locale has not been defined, the framework will revert to a default language. This is truly commensurate with Java's write once run anywhere motto, and readily facilitates business expansion and global operations.

Extensibility

Night Vision has been developed using documented, reusable and extensible GUI classes built on top of the Swing toolkit. Swing is easily the simplest, best known and most widely used Java GUI toolkit. Additionally, Swing conforms to the general Model View Controller (MVC) architecture, separating data from presentation. Perhaps the biggest benefit of using Swing is that it employs "lightweight" widgets that don't map to operating system specific "peer" components. Stated otherwise, Swing components can be made to look on Linux precisely like they look on Mac OS or Windows XP. Swing's support for pluggable look and feels allow for any extra degrees of customisation should the need arise.

Night Vision comes with a set of components that allow for the rapid development of management screens. These include wizards, filters, validators, editors, Swing data models, renderers and layout managers, to name a few. These follow specific user interface design guidelines that ensure a consistent look and feel across all management components.

Branding

One area of extensibility that has been built into Night Vision from the outset is organisation-specific branding. All generic graphic portions of DTS/NV, such as logos, backgrounds, even the default icon set can be replaced with media branded with logos of the adopting organisation. Because Night Vision is built on top of Swing, even the look and feel can be substituted for the one supplied by your company.

Security and Authentication

Night Vision communicates with the DTS/S1 host via a standard Java RMI (Remote Method Invocation) interface, tunnelled over an SSL socket employing the AES cipher suite with 256-bit key strength, RSA key exchange and X.509 certificates used for end-point authentication. This is bullet-proof by all current security standards and is anticipated to last decades into the future.

At the operator (admin user) level, Night Vision caters for a set of pluggable rules that define operator accounts, authorities and credentials - demarcating precisely which users have access to which functions of the system. As an added level of security, Night Vision enforces access policies on the server, so that no amount of tampering with the management console can lead to a breach of security.

Because the framework is just a tier of DTS, every change that an operator makes to the system, even merely signing on or off is fully auditable, while the audits themselves are read-only and are stored perpetually in a WORM (write once read many) fashion, for the life of the system. These levels of security are crucial in financial systems, and only through this degree of diligence can most financial systems get certified for use by their respective governing bodies. Even if your system does not require explicit certification and field approval, such levels of security guarantee you full accountability should your company encounter fraud. Best of all, these measures are by no means intrusive on the end-users. They exist silently in the background, recording everything that happens, and can be called upon should the need arise. The facets of Night Vision's authentication and security framework are illustrated in the figure below.

A real gem in the security mechanisms in Night Vision is allowing an operator to temporarily sign off from the console, perhaps to go on a lunch break, and then sign on later and continue working from whether they had left off, providing that the console application had not been closed. The console recalls the states of all GUI components across short-term departures so that the operator is not tempted in compromising on security in favour of productivity.

Remote access

One typical constraint on back office applications is deciding where they should reside and how they should interface with the storage back end. Ordinarily, a back office deployment would consist of a number of identical machines located in secure premises within the organisation, in close proximity to the database and potentially on the same network segment. The back office subnet is typically an Ethernet network that is isolated from the outside world either physically or, in more relaxed environments, through firewalls. This level of security is necessary to ensure that only the right machines make contact with the database, as well as to protect the back office traffic from unauthorised viewing. Making a connection externally is usually either not possible, or restricted through use of a VPN (Virtual Private Network) or similar Extranet technologies. This topology is represented in the figure below.

Night Vision communicates with the DTS/S1 cluster using an encrypted TCP connection that employs a secure key exchange and certificate-based authentication to ensure that it connects to only specific DTS/S1 nodes and keeps its traffic secure. At the same time, the RMI-based Remote Object Model Broker (ROMB) interface that the DTS/S1 cluster exposes employs similar measures as well as username/password authentication to limit access to specific users with well-defined access privileges. The added bonus is mobility: because the security model is completely contained within DTS, back end clients are not limited to isolated networks in close proximity to the database. They can be across the wall, or across the equator. As depicted in the figure below, there is no reliance on third party secure tunnelling technologies to provide mobility for any DTS component.

Accelerated development

Given a database and a fully-functioning front-end, an organisation would have to commit to potentially several man-years of effort in order to develop a secure, easy to use back office interface that would address some of the business requirements outlined in the previous sections. The core benefit of using DTS/NV is that the overwhelming majority of the software engineering work has already been carried out. The missing parts are rules that define how business entities are constructed, searched for and modified. By comparison with the workload involved in, for example, building a call centre application, the work involved in defining the business rules is orders of magnitude less. With Night Vision, you can literally develop the user interfaces to locate and maintain customer records within a few days. To help accelerate development, Night Vision is shipped with documented examples that are, effectively "templates", and can be readily customised to suit the business workflow.

Night Vision is wholly geared to one purpose: provide as much of the back office foundation as possible, and let application developers concentrate on fleshing out the rules specific to their organisation.

Compatibility

Night Vision does not rely on any specific hardware or software platform to function, with the exception of a Java runtime environment. This means you can equip your back office personnel with inexpensive, low-end desktop machines with a network connection to the DTS/S1 cluster. DTS/NV is designed to operate stand-alone and can be run in full screen mode, thereby replacing the existing desktop. Because DTS/NV fully embraces the Java write once read anywhere paradigm, it can be run on any operating system, including the entire Windows line-up, Linux, Solaris and Mac OS. This is, in fact, a feature that extends over the entire DTS product line. Should you capitalise on the cross-platform compatibility of DTS, you can further reduce start-up costs by running any or all of its components on Linux, BSD or Open Solaris.

Cost

DTS/NV Night Vision is free for all DTS/S1 licensees. Not only are we offering something that others aren't, we are offering it for free: as a token of appreciation and to prove once again that there is no better platform for developing OLTP applications.

Proceed to the gallery to see screenshots of DTS/NV in action.