Ada has
become the first programming language to establish the process
for testing compilers' implementation as an international standard. Ada
compilers can be labeled as "certified as conforming to standard Ada"
(informally called "Validated Ada Compilers") only if they follow a
specific route of testing and certification. In October 1999, the
International Organization of Standards (ISO) approved the prescribed
path in Ada: Conformity Assessment of a Language Processor
(ISO/IEC-18009:1999).
The standard provides other programming languages with a compiler
certification model to follow. If the C community, for example, needs to
distinguish between compilers that translate standard C and those that
permit nonstandard extensions or that fail to handle the entire
language, the new ISO standard process for testing Ada compilers has
blazed a trail to that goal.
The Ada community will not notice much change in the testing procedures
except in the vocabulary (for instance, validation versus conformity
assessment) and in a streamlined bureaucracy. The new standard codifies
existing practices for testing Ada compilers that tool vendors have
followed since the early eighties.
The Ada community is well-qualified to pioneer the standard. Ada is
unique in insisting that software tools' conformity to the language's
international standard be an integral part of marketing compilers.
Facilities all over the world have tested standard Ada implementations
for over 15 years.
The language's designers realized that compiler testing is critical to
Ada's success. Testing compilers for translating "proper Ada" actually
predates the first Ada standard of 1983. The designers intended the
language-initially used in large embedded systems--to be dependable,
reusable, portable, maintainable, and legible. Many different
programmers working toward the same goal had to simultaneously write,
reuse, and integrate software components. After being fielded, the
systems were assumed to have a long life during which they would port to
many different hardware systems.
Ada 95 continues this emphasis on high reliability. The language is used
for most modern aircraft fly-by-wire controls and new air traffic
control systems, as well as rail transportation systems and satellites.
Ada has been used in the Chunnel and the subway systems in Paris, Hong
Kong, London, and New York; for manufacturing Volvos in Sweden; and for
controlling steel mills in West Virginia. The language is used for
smaller applications, such as network switching systems and e-commerce
applications, where adaptability and time to market are important
factors.
To realize its purpose, Ada must execute reliably across platforms and
national borders. In other words, it must be standardized. If Ada
compilers translated dialects, not only would safety-critical
verification of the software be difficult, but also the software would
not be reusable, universally legible, or as maintainable. Today, the Ada
Conformity Assessment Authority (ACAA) safeguards the procedures for
testing Ada compilers' translations against the ISO standard.
A history of Ada conformity assessment
The original big user of Ada, the US Department of Defense, first tested
compilers' conformity to standard Ada in 1984. In October 1998, the DoD
handed conformance testing over to an industry group, the Ada Resource
Association. The ARA consists of numerous Ada compiler and tool vendors
who work together to promote and support Ada's use in the commercial and
government marketplace.
Having Ada vendors controlling the process of testing Ada compilers is
very much like having the foxes guarding the hen house. The vendors
could easily decide to make the process less rigorous, reducing its
value for Ada users. Therefore, to safeguard the process, the Ada
community agreed to make the testing process an international standard.
This standard, ISO/IEC-18009, outlines the basic conformity assessment
process. The standard does not specify such details as what to test or
how to distribute the tests, but it does specify what is allowed and
disallowed in testing.
Why is a separate authority needed?
The conformity assessment standard defines an independent agency--the
ACAA--to manage the testing process. Although the Ada tool vendors
finance it through the ARA, the ACAA's real boss is the ISO, and its
charges are the testing laboratories.
Independent laboratories, or Ada Conformity Assessment Laboratories
(ACALs), test the compilers. Because compiler vendors pay testing fees,
they could put financial pressure on a lab to successfully complete
tests. If a laboratory certifies a processor that does not meet the Ada
standard, the testing becomes useless for users. The ACAA reduces this
risk by ensuring that labs use the same detailed procedure. The ACAA and
testing labs cooperate to develop the detailed procedures, with the ACAA
as the final arbiter. This lets laboratories compete on the basis of
price and service.
The ACAA enforces consistency by verifying that the ACALs follow the
procedures for each completed testing, maintaining the test suite, and
handling test disputes. Thus, the ACAA ensures that all labs use exactly
the same tests. The ACAA also maintains the single, common list of
successfully tested compilers (see www.adaic.org/compilers).
The Ada Conformity Assessment Test Suite (ACATS) includes both positive
tests, which check that the language's features work as defined by the
Ada 95 standard, and negative tests, which check that the compiler
rejects illegal Ada code. It is freely available to everyone from many
sources, including the ACAA's Web site, www.ada-auth.org/~acats.
The value of Ada conformity assessment
Ada users know that conformity assessment is the only objective presale
assurance that a compiler implements Ada correctly. They consistently
state that independent third-party testing is the most important feature
of Ada conformity assessment. Although vendors might have a strong
incentive to fudge results, the testing laboratory has verified that the
compiler passes the tests. The Ada conformity assessment standard
strengthens this advantage by providing an agency (the ACAA) to police
the laboratories, ensuring comparable results from different testing
laboratories. Moreover, the ACAA and the testing laboratories are
independent organizations, minimizing the possibility of collusion.
Finally, the free availability of the test suite and test reports also
makes test results more reliable, because any interested party can
repeat some or all of the testing. This provides yet another
disincentive to cheating.
Because the test suite (ACATS) is freely available, all Ada vendors use
it for regression testing and to judge the quality of their
implementations even before they contract with a lab for testing. This
inevitably improves the quality of Ada compilers because the test suite
detects many problems, which the vendor eliminates, long before users
are affected.
The common list of successfully tested compilers serves two purposes.
Not only can users verify the testing of a particular compiler, they can
also access the actual test reports. This is a more reliable way to
determine a compiler's test status than relying on a vendor's
information. Second, the list is complete and lets users find all
compiler vendors that target a certain processor. Because not all
compiler vendors are well-known, the list provides an Ada project
manager with the most choices.
The Ada conformity assessment process resembles the open source movement
in that the test source code and documentation are freely available,
while testing services cost money. The new standard goes further by
providing free maintenance to users of the process. No charge is made to
a vendor or user that requests a test modification or disputes a test's
results; the ACAA bears the cost. Thus, users face no political,
financial, or monolithic corporate obstacles in participating in the
maintenance of tests and procedures. Consequently, the test suite is
more likely to reflect real users' needs.
Status
Ada compiler testing continues unabated. Over 50 conformity assessments
have been performed in the last two years. The test suite now contains
more than 3,600 tests covering the full breadth of Ada 95. The suite
expands existing tests and adds new ones as user and vendor needs
evolve. New tests focus on recent corrections to the Ada standard-the
most likely areas for processor errors.
Software engineers have always been confident of the quality of Ada
compilers because the compilers were independently verified. The new ISO
standard assures them that the procedure is protected against weakening
by vendors. The standard is the product of 15 years of evolution of the
process of Ada conformity assessment. The process has evolved to benefit
everyone: users, vendors, and testing laboratories.
As demand for reliable software grows, other languages will need to
establish procedures for testing conformity of compilers to their
standard. Others could adapt the model and experience of the Ada
language for other languages, especially where a strong central
authority exists, as is the case with Java.
* * *
Randall Brukardt is the manager and Technical Agent of the Ada
Conformity Assessment Authority. He has been involved with the Ada
language for nearly twenty years, having been lead designer for a
popular PC Ada compiler, an Ada 9x distinguished reviewer, and now is
one of the editors for the Ada 95 standard. He still is Director of
Technical Operations at R.R. Software, Inc, leading development of their
compiler and Windows products. He occasionally finds time for travel and
photography. Contact him at agent@ada-auth.org.
