Product Testing
Introduction to Testing
Software testing can be performed at three levels. Successive levels of testing provide increasing assurance that two applications will interoperate seamlessly.
- Compliance testing – Testing performed to assess whether the software meets the requirements of the specification. This type of testing has been performed by the NRECA-approved MultiSpeak testing laboratory on products submitted from a single vendor for MultiSpeak compliance.
- Interoperability testing – Such tests are performed between pairs of software applications to ensure that they work seamlessly together. It is possible that two software applications could both be compliant and yet not work seamlessly together without minor modifications.
- Site conformance testing – Tests performed at an individual utility site to ensure that pairs of products interoperate seamlessly given the utility’s specific data structures and application data. Even if two software applications have passed both compliance and interoperability testing, it is still possible that choices of data structure made by a specific utility or poor quality data stored in one of the applications may result in interface errors. Such errors may require work on the part of the software vendors or data scrubbing on the part of the utility.
If a product has passed either compliance testing or interoperability testing, it may be said to be MultiSpeak compatible. The best way to determine if a product has been tested for MultiSpeak compatibility, and the extent of that compatibility, is to check the tested products pages for Version 1, Version 2, or Version 3 products.
Although vendors may incorporate features of MultiSpeak compatibility in their products without submitting their products for compliance or interoperability testing, you can be assured that if the product is listed as compliant or interoperable on this web site, it has passed the necessary testing.
Past Testing Provided by MultiSpeak
For Versions 1.1 and 2.2, NRECA sponsored an independent testing organization to evaluate compatibility with the requirements of the MultiSpeak specification and a testing protocol was developed. The testing organization certified that products submitted for test met the requirements of the specification (compliance testing).
Certification is for products, not vendors; certification is by interface. Thus a particular product is deemed to be, for example, compliant with the requirements of the customer billing-geographic information system (CB-GIS) interface. If the submitted product passed the certification test for at least one defined interface it was declared to be compliant.
Compliance testing is no longer offered for newly developed Version 1.1 or Version 2.2 interfaces. No compliance testing will be offered for newly developed Version 3.0 interfaces.
Current Testing Provided by MultiSpeak
Interoperability testing is new to the effort with Version 3.0. In interoperability testing, two products (typically from different vendors) are submitted for a joint test. The vendors develop an assertions document that states the joint MultiSpeak functionality that they wish to claim and how those capabilities may be used to support utility business processes. The independent testing agency then verifies that the two products integrate in the manners described and that the integration is achieved using MultiSpeak. Verified assertions documents are available either from the vendors participating in the test or directly from the MultiSpeak Project Technical Coordinator. You may use these documents as a way to see, in detail, exactly what MultiSpeak-compatible integration is included in vendor product interfaces.
This type of testing is performed by the NRECA-approved testing agent when pairs of vendors specifically request and pay for such testing. It should be noted that, in some cases, products may be interoperable, but the vendors have not chosen to pay for testing to document this fact. For some vendors, whose products are expected to integrate with a large number of software products offered by different vendors, the cost of testing all possible combinations of software can be burdensome. Alternatively, a utility may choose to require interoperability testing for a pair of products that have not yet been jointly tested as a software quality control step during software procurement. In such a case, the cost of the test may be borne by the utility or by the vendor(s) of the software in question.
Will compliance (or interoperability) testing ensure that two software applications work together seamlessly at my utility?
For a number of reasons, such as non-standard formatting used in a particular utility or poor data quality, neither compliance nor interoperability testing can provide complete assurance that two software applications will work together seamlessly at your specific installation. Site conformance testing is required if it is necessary (i) to document the level of integration functionality, (ii) to determine the root cause of failure of two software applications to work together as expected, or (iii) to determine what customization is required to achieve the expected result in non-standard situations at a specific utility. Utilities may request that an independent party perform site conformance testing on their behalf. In addition, utilities may find it desirable to require site conformance testing as an acceptance criterion when purchasing new software products.