Testing Information for Vendors
The Initiative performs compliance and interoperability testing, jointly referred to as conformance testing. Compliance testing checks the software submitted for test to see if data output is done so in compliance with the requirements of the Specification and that the software can accept data that has been formatted as required by the Specification. Interoperability testing checks two different pieces of software to ensure that data and service calls are exchanged between the pair of software products in a manner that is supported by the MultiSpeak Specification. The Initiative does not perform testing to ensure that software performs as advertised or meets any specific load or volume performance requirements.
Once an application meets all of the requirements for testing, NRECA will post the testing report.
Software testing can be performed at three levels. Successive levels of testing provide increasing assurance that two applications will interoperate seamlessly.
- Compliance Testing – Compliance testing is performed to assess whether the software meets the requirements of a standard or specification. Note that the more “flexible” or general the standard, the more likely that two products, tested as “compliant” to the standard may still require customization in order to interoperable smoothly. This type of testing was the first type done by the NRECA-approved MultiSpeak testing laboratory on products submitted from a single vendor for compliance with MultiSpeak Versions 1.1 and 2.2.
- Interoperability Testing – Interoperability tests are typically performed between pairs of software applications to ensure that they work as nearly seamlessly as possible together. The MultiSpeak Initiative has performed interoperability testing since 2005 on software meeting the requirements of Versions 3.0 and 4.x.
- Site Conformance Testing – Site conformance tests are performed at an individual utility site to ensure that pairs of products interoperate seamlessly given the utility’s specific data structures and application data. Even if two software applications have passed both compliance and interoperability testing, it is still possible that choices of data structure made by a specific utility or poor quality data stored in one of the applications may result in interface errors. Such errors may require work on the part of the software vendors or data scrubbing on the part of the utility. The MultiSpeak Initiative does not perform nor sponsor conformance testing.
For a number of reasons, neither compliance nor interoperability testing can provide complete assurance that two software applications will work together seamlessly at your specific installation.
For example, if a specific utility has chosen to use non-standard formatting for customer identifiers or has chosen to store those identifiers in a non-standard data field, the link between customer records in a customer information system and an automated meter reading system could be jeopardized. The MultiSpeak interface might work properly and pass interoperability testing provided the customer identifier is stored as anticipated by the two vendors. However, the fact that the customary data field is missing at a specific utility could cause the link to fail only at that utility. Such a failure would not be a reflection on the generic suitability of the MultiSpeak interface. Rather, the utility’s choice to use non-standard data is the root cause of the problem. In this case, the standard MultiSpeak interface could still be customized slightly to work for this utility or the utility could decide to change their data structures to match the anticipated condition.
There is another situation where two products that have passed interoperability testing might work properly at one utility and fail to work together as expected at a different utility: poor data quality. For example, if customer telephone numbers are not accurate in the customer information system, an outage management system or an interactive voice response system that relies on those telephone numbers might give unacceptable results, even though the telephone numbers were sent accurately from the CIS to the other system.
If it is necessary (i) to document the level of integration functionality, (ii) to determine the root cause of failure of two software applications to work together as expected, or (iii) to determine what customization is required to achieve the expected result in non-standard situations at a specific utility, site conformance testing is required. Utilities may request that an independent party perform site conformance testing on their behalf. In addition, utilities may find it desirable to require site conformance testing as an acceptance criterion when purchasing new software products.
Understanding how the tests on products have been conducted will help clarify what assurances can be derived from knowing that a product has been termed “MultiSpeak compliant” or “Interoperable”.
Products submitted for testing to MultiSpeak Version 3 or Version 4 are tested for interoperability. In interoperability testing, two products (typically from different vendors) are submitted for a joint test. The vendors develop an assertions document that states the joint MultiSpeak functionality that they wish to claim and how those capabilities may be used to support utility business processes. An independent testing agency then verifies that the two products integrate in the manners described and that the integration is achieved using MultiSpeak. The verified assertions document is then posted on this web site for download by interested parties. The verified assertions documents are also available either from the vendors participating in the test or directly from the MultiSpeak Technical Coordinator. These documents are used as a way to see, in detail, exactly what MultiSpeak-compatible integration is included in vendor product interfaces. Please read The Assertions Document section on this page.
Interoperability tests are usually between applications from two different vendors. However, a single vendor may test against a test “harness” in lieu of a second vendor’s offering. The testing harness acts as a universal client or server, providing the necessary partner for the software under test. This particularly used when the second application is not readily available for testing against.
The purpose of the assertions document is to provide information to utility staff members about what level of MultiSpeak integration could be expected between two specific products that have completed Version 3.0 or 4.x interoperability testing. Assertions documents are written with the utility audience in mind. They answer the following questions:
- What can I do with the software products that would not be possible without the MultiSpeak integration?
- Why is this shared functionality important to my business processes?
- How have the vendors used MultiSpeak to accomplish the integration described?
- How much of the MultiSpeak defined interface capability does this interface support?
Note that assertions documents do not necessarily describe all of the capabilities of an interface between two products, only those that are accomplished using MultiSpeak. It is certainly possible for two vendors to support other integration features that are not included in MultiSpeak; for a description of such features, contact the affected vendors directly.
Utilities can use assertions documents in two ways:
- You have MultiSpeak-compatible software and you want to know what it does, and does not, do.
- You are considering the purchase of software and want to compare the MultiSpeak integration supported by alternative vendor offerings.
This type of testing is performed by the NRECA-approved testing agent when pairs of vendors specifically request and pay for such testing. It should be noted that, in some cases, products may be interoperable, but the vendors have not chosen to pay for testing to document this fact. For some vendors, whose products are expected to integrate with a large number of software products offered by different vendors, the cost of testing all possible combinations of software can be burdensome. Alternatively, a utility may choose to require interoperability testing for a pair of products that have not yet been jointly tested as a software quality control step during software procurement. In such a case, the cost of the test may be borne by the utility or by the vendor(s) of the software in question.
It is very important to note that this does not mean that all of the products features have been tested to the MultiSpeak specification, only those interfaces indicated in the assertions document and in most cases, only for specific functionality. Specific details can be found in links from the products page. The products are listed by product type (for example, SCADA software), then alphabetically by company and product name. Note that no preference or ranking of software is implied by the order on the listing for a given type of software.
In the period 2000-2006, software was tested for compliance (rather than interoperability) with the MultiSpeak Specification Versions 1.1 and 2.2. Software which tested as “compliant” to these earlier versions of the MultiSpeak specification is listed in Appendix C.
MultiSpeak Specification Version 1 Compliance Tests
(Performed during 2000 – 2004)
MultiSpeak Version 1.1 defined a model for batch file data transfers. Vendors were tested using a file of sample test data which was imported into their software. Once the data were deemed to be successfully imported into the vendor software, the data was manipulated per the tester’s instructions, and then exported back out. The data file was checked for proper XML validation and examined to ensure that the required data manipulations were handled correctly.
The thoroughness of Version 1.1 application integration depended in large extent on how completely the vendors sharing an interface implemented the specification. The Version 1.1 specification defined many data items, only a few of which were considered to be mandatory for testing purposes. If any of the vendors chose not to implement non-mandatory items required to support a real business process, the utility sometimes found the interface to be lacking.
MultiSpeak Specification Version 2 Compliance Tests
(Performed during 2003-2006)
The MultiSpeak Version 2 standard supported both file-based (batch) and real-time data transfers. Compliance testing was conducted on both batch interfaces and real-time interfaces. A vendor’s real time software interface was tested against a real-time connected server (via the internet) which accepted and replied to MultiSpeak requests, and generated “real-world” requests to the vendor’s software. It also logged all transactions and reported inconsistencies.
The Version 2 test suite was been developed to reflect actual data requirements to implement utility business practices, thus it addressed the shortcomings sometimes identified in the Version 1.1-compliant interfaces.
In many cases, vendors supplying software that was tested to Versions 1.1 and 2.2 vendors have moved well beyond the capabilities of the software applications tested at that time.