Ada Compiler Evaluation System
ACES 2.1
Primer
FINAL
Contract Number F33600-92-D-0125
CDRL A0014
Prepared for:
High Order Language Control Facility
Business Software Support Branch
88 CG/SCTL
Wright-Patterson AFB OH 45433-5707
Prepared by:
CTA INCORPORATED
5100 Springfield Pike, Suite 100
Dayton OH 45431
Availability of the Ada Compiler Evaluation System
The ACES software and documentation are available by anonymous FTP from the host "sw-eng.falls-church.va.us" in the directory "public/AdaIC/testing/aces/v2.0" and from other Ada-related hosts. Document files are included in PostScript format and as ASCII text. The ACES files are also accessible via the World-Wide Web. The home page URL is "http://sw-eng.falls-church.va.us/AdaIC/testing/aces/".
For further information about the ACES, contact the High Order Language Control Facility. As of 1 March 1995, the appropriate contact is:
Mr. Brian Andrews
88 CG/SCTL
3810 Communications, Suite 1
Wright-Patterson AFB, OH 45433-5707
(513) 255-4472
Go to: Section Summary or Table of Contents
This document serves as an introduction to the nature and use of the Ada Compiler Evaluation System (ACES), Version 2.1. The intent of the Primer is to provide the minimum information needed to assess an implementation of the Ada language using the ACES and/or the ACES Quick-Look Facility. It addresses only those issues that are expected to arise in a straight-forward application of the ACES. Other documents in the ACES documentation set should be referenced if issues arise that are not covered in the Primer.
The Primer is meant to be usable by any person who has knowledge of the operating systems and compilation system being tested. Previous ACES experience is not necessary.
The document consists of eight sections, each of which is briefly described in the following list:
* Section 1, "Overview of ACES"
Summarizes the purpose and structure of the ACES and gives a high-level view of its use.
* Section 2, "File Names and Installation"
Describes the ACES files, the default directory structure, and the installation procedures.
Describes the use of the automated Pretest process that must be completed before performance testing can begin.
* Section 4, "Performance Testing"
Describes the use of the Harness tool in executing and tracking the performance tests.
* Section 5, "Performance Analysis"
Describes the use of the Condense, Comparative Analysis, and Single System Analysis tools in analyzing the data produced by the performance tests.
Describes the use of the Assessor software and scenarios for determining system capacities and the capabilities of the symbolic debugger, the library management system, and the diagnostics system.
* Section 7, "Interpretation of Results"
Describes the most useful reports that are produced by the Analysis tools and the assessors.
Describes the use of the ACES Quick-Look facility which provides a subset of key ACES performance tests.
The user is expected to refer to other documents when issues arise that are not anticipated in straight-forward uses of the ACES. Information on the execution of the tests and tools is maintained in the User's Guide. Information on the use of the results, including discussions of analysis techniques, is maintained in the Reader's Guide.
If the user wishes to do only performance testing, then he/she should read Primer Sections 1 through 4. (Note that it is necessary to perform the steps described in Section 3 before doing performance testing.) If he/she then wishes to use the Analysis tools, Section 5 is important. Section 7 will be helpful in this case, also. If the user does not wish to use the ACES analysis techniques, then he/she should follow the instructions in the Condense subsection of Section 7, choosing to produce comma-delimited output. These files can then be imported into the analysis tools of the user's choice.
If the user wishes to use only the Assessors, then he/she should use Primer Sections 1 through 3 and Section 6. Note that some of the steps described in Section 3 must be performed before using the Assessors. The Section 7 paragraphs on the Assessors are also useful in this case.
If the user wishes to use the Quick-Look facility, he/she should use Primer Section 8.
LIST OF TABLES
The Ada Compiler Evaluation System (ACES) is a set of software tools, source code, scripted scenarios, and documentation. Its purpose is to help assess quality factors of Ada implementations. (An Ada implementation consists of a host computer, a target computer, operating systems for the two computers, and an Ada compilation system.) The ACES can provide compile-time performance data, run-time performance data, and system usability data. The ACES analysis programs can be used to compare the performance of two or more Ada implementations, or they can be used to indicate ways to get the best performance out of a particular implementation.
The ACES represents the merger of two evaluation systems: The Ada Compiler Evaluation Capability (ACEC) and the Ada Evaluation System (AES). Each of the systems boasted some unique strengths, yet there was some duplication between the two. The goal of the ACES merger was to take advantage of each systems' strengths and reduce the amount of duplication.
The Ada Joint Program Office (AJPO) of the United States Department of Defense (DoD) funded the development of the ACEC. The ACEC went through several releases, culminating with Version 3.1 (1992). The Air Force Wright Laboratory provided management, and the Boeing Defense and Space Group (Wichita, Kansas) performed the technical work. Boeing designed the ACEC to be portable to all validated Ada implementations with minimal adaptation. The Air Force distributed dozens of copies of the ACEC to U.S. and foreign Government organizations and their contractors.
The United Kingdom's Ministry of Defence (MoD) sponsored the AES. The British Standards Institute and Software Sciences Ltd. accomplished the development of AES versions 1.0 and 2.0. The AES test management tools were its strength. Evaluation services using the AES were provided for fee.
In June 1991, DoD and MoD agreed to merge the two evaluation suites and produce a single, internationally available suite. The AJPO provided funding for the merger effort, which was managed by the High Order Language Control Facility (HOLCF) at Wright-Patterson Air Force Base. In 1993, the HOLCF contractor, CTA INCORPORATED, delivered the ACES as the merger of the ACEC and the AES. Boeing Defense and Space Group again performed the actual development work under contract to CTA. In 1994, the HOLCF and CTA enhanced the ACES with this Primer, comma-delimited report files, and an automated process for the Pretest activity. The ACES was further enhanced in 1995 with a Quick-Look facility which provides a fast and easy (yet less complete) process for evaluating an Ada implementation.
The current version (2.1) provides more Ada 95 testing; allows the user more default choices for processing options, test selection, and analysis report selection; and provides a mechanism for incorporating user-defined performance tests into the ACES test selection and analysis processes.
From the user's point of view, the ACES consists of five components, as follows:
a. Support software and data
This component is composed of approximately 350 files, classified as Setup files, Harness files, and Analysis files.
These files are used to build and test a core set of library units, command scripts, and data files. These are used throughout the evaluation.
Approximately 90 of the support files contain source code and data for the Harness program. This tool aids the user in tracking the run-time status of the performance tests and in creating command scripts for compiling and executing the tests.
The remaining support files contain source code and data files for the Analysis tools. These programs are used to compare performance test results for two or more implementations and to evaluate the relative performance of various features of one implementation.
b. Performance test software
This component comprises approximately 2120 files. These files include source code for the tests and their support units along with sample command scripts for executing the tests. There are 21 groups of performance tests, each group addressing a particular set of performance issues.
c. Assessor software and data
This component is comprised of approximately 335 files, representing software modules, data files, and scripted user scenarios. Three of the assessors evaluate the usability of the diagnostic system, the library management system, and the symbolic debugger. The fourth determines compile-time and run-time capacity limits.
d. Quick-Look software and data
This component comprises approximately 155 files comprised of software modules and data files. These files contain the software necessary to setup the software tests, the software tests (selected subset of the full ACES), and reporting software.
e. Documentation
The ACES documentation set consists of this Primer, a User's Guide, a Reader's Guide, and a Version Description Document. Documents are distributed in both ASCII text form and PostScript form. With appendices, there are approximately 40 files. The User's Guide provides detailed instructions for the user. The Reader's Guide provides information on the analysis reports, the measurement and analysis techniques, and the interpretation of the results. The Version Description Document provides descriptions of the tests, the association between (long) test names and (short) file names, and descriptions of tests that are known to cause difficulties on some implementations.
Portability across validated Ada implementations was a primary design goal of the ACES. In particular, all user interface input and output are performed using Text_IO. No assumptions are made about terminal capabilities except those that apply to the least sophisticated terminals. Thus, users who are accustomed to windowing facilities, "hot-key" definitions, and point-and-click interfaces may perceive the ACES user interface as primitive.
The ACES was designed with three general evaluation goals:
a. Evaluation for implementation selection;
b. Evaluation of a selected implementation; and
c. Comparing releases of an implementation.
The ACES user must start with a set of clearly defined goals for each evaluation. These goals lead to evaluation criteria, which should be carefully documented. Whatever the goal(s), it is not appropriate to perform the testing, generate the analysis reports, read all of the data, and then decide what information is or is not important.
Figure 1-1 illustrates the paths through the ACES evaluation process. The first step is performing the Pretest activity. (The Pretest automation tool, ZP_SETUP, is useful here.) The results of the Pretest are recorded on the Pretest Report.
If the user wishes to run the performance tests, then the next step is the use of the Harness program (compiled during the Pretest activity). This program generates command scripts for compiling and executing the performance tests. The user then executes these scripts and captures the results in log files. The Harness program reads these log files and gives execution status information so that the user can choose to re-run selected tests or go on to other tests. This process is iterative.
When the user has run as many performance tests as desired, he/she runs the Analysis Menu program, first selecting the Condense tool. Condense reads the log files and produces the Analysis Database files. The user then selects either Comparative Analysis (in which case Analysis Database files for other programs must be available) or Single-System Analysis. These Analysis programs produce extensive reports, as directed by the user.
The user may elect to run the Assessors before, after, or independently of the performance tests, provided that at least part of the Pretest has been completed. The Assessors may be run in any order. The results from each are recorded on the appropriate Assessor Report form.
The user may elect to run the Quick-Look facility before, after, or independently of the performance tests. The Quick-Look facility provides a setup process very similar, yet less complex than that done as part of the Performance Testing and Assessor pretest activity.
Figure 1-1 Overview of the ACES Evaluation Process
The testing process varies according to the goals of the evaluation. For each of the goals described in Section 1.3, the corresponding paragraph below identifies some assumptions, suggests some questions that are useful in setting up evaluation criteria, identifies the activities appropriate to the goal, points to the Primer section that discusses each activity, and identifies decisions that must be made.
Assumptions
This goal assumes that a set of candidate implementations has been identified and made available. All activities except analysis must be performed on each implementation. The analysis step may be carried out on any one implementation.
Criteria
* Which implementation best meets the minimum performance levels required by the project (e.g., for task rendezvous time, task scheduling time, or input/output time)?
* Which implementation makes the tradeoffs (regarding compilation speed, linking speed, execution speed, and code size) that best fit the needs of the project?
* Which implementation provides the best combination (for the project) of functionality of the debugger, library management system, and diagnostic system?
* Which implementation supports the most important capacity limit (both compile-time and run-time) that are required by the project?
* Which implementation gives the best (in terms of project goals) performance in those language areas that are of particular importance to the project (e.g., generics or tasking)?
Activities
* Pretest/Setup (Section 3)
* Performance Testing (Section 4)
* Condense (Section 5.2)
* Comparative Analysis (Section 5.3)
* Assessors as determined by the evaluation criteria (Section 6)
* Interpretation (Section 7)
Decisions
* Whether, and how, to measure code size (Section 3.2.2, Section 5.2)
* Whether to measure elapsed or CPU time (Section 3.2.3)
* Which performance test groups to use (Section 4.1)
* Which comparison reports to request (Section 5.3)
Assumptions
This goal assumes that the organization or project has already selected an implementation and has it available for evaluation. The intent of the evaluation is to support the establishment of procedures that would make the most efficient use of the implementation.
Criteria
* Should the software be organized as many small units or fewer large units?
* Should complete system "builds" be done frequently or infrequently?
* Should the use of tasking or generics be avoided or encouraged?
* Should automatically generated code be manually optimized for certain cases or should the compilation system be expected to handle optimization?
* Should coding and optimization emphasize execution speed or code size?
* Should unchecked deallocation be used?
* Should particular data structures and data references be avoided or encouraged?
* Should exception handling be used generously or sparingly?
* Should if-then-elsif control be used in preference to case statements?
* Should subprogram calls be expanded in line?
* Should deployed code be compiled without checks?
Activities
* Pretest/Setup (Section 3)
* Performance Testing (Section 4)
* Condense (Section 5.2)
* Single System Analysis (Section 5.4)
* Assessors as determined by the evaluation criteria (Section 6)
* Interpretation (Section 7)
Decisions
* Whether, and how, to measure code size (Section 3.2.2, Section 5.2)
* Whether to measure elapsed or CPU time (Section 3.2.3)
* Which performance test groups to use (Section 4.1)
* Which analysis reports to request (Section 5.3)
Assumptions
This goal assumes that two releases of the same implementation are being compared to determine the performance gains or losses. The differences in the implementations may be due to hardware changes, operating system (OS) upgrades, or compiler upgrades.
Criteria
* Is the new hardware (or OS) faster or slower?
* Has compilation or linking speed changed?
* Has execution time changed?
* Has code size changed?
* Are new optimizations detected?
* Has the performance of particular language features changed?
Activities
* Pretest/Setup (Section 3)
* Performance Testing (Section 4)
* Condense (Section 5.2)
* Comparative Analysis (Section 5.4)
* Interpretation (Section 7)
Decisions
* Whether, and how, to measure code size (Section 3.2.2, Section 5.2)
* Whether to measure elapsed or CPU time (Section 3.2.3)
* Which comparison reports to request (Section 5.4)
As distributed, the ACES files are arranged in the directory structure outlined below. The numbers that follow behind the directory name indicate the current number of files (excluding sub-directory files) that are found in that directory.
aces (1)
assessrs
capacity (63)
debugger (79)
diagnost (101)
library (91)
docs (38) (documentation files)
qlook (156)
support (354) (Pretest, harness, and analysis files)
tests
ap ( 75)
ar (142)
cl ( 90)
do (106)
dr (290)
dt ( 51)
gn ( 28)
in ( 12)
io (131)
ms ( 18)
oo ( 10)
op (329)
po ( 98)
pt ( 10)
sr ( 65)
st ( 92)
su ( 82)
sy (296)
tk (131)
xh ( 61)
ud ( 1)
In this Primer, references to this directory structure will be given in UNIX style, rooted at the "aces" directory. For example, the files for the Classical (cl) group of performance tests are in the directory referred to as "aces/tests/cl".
The Primer uses the asterisk ("*") in file names as a "wild-card" specifier, matching any sequence of characters. The question mark ("?") is sometimes used as a limited wild-card character, matching any single character. Finally, the sharp symbol ("#") is sometimes used as a limited wild-card character, matching only those characters that are specified in the section using this character. Some examples follow:
Pattern Sample Matches
zc_*.com zc_lnk.com, zc_setli.com zg_glob3.el? zg_glob3.elg, zg_glob3.ell *.ada cl_sodum.ada, zp_setup.ada z?_*.el? zg_glob3.elg, zc_incld.ell aces/tests/# aces/tests/cl, aces/tests/ap
The ACES is distributed as a set of approximately 3000 files, most of whose names have the form "pp_ident.typ". The file names have been kept short and simple in order to achieve the goal of portability. The first two characters of the file name identify the category to which the file belongs, as discussed in subsections 2.1.1 through 2.1.7. In nearly all cases, the third character is an underscore, though there are a few exceptions. Characters 4 through 8 (or 3 through 8 if there is no underscore) identify the purpose of the file. This identification depends on the file category. The three-character suffix often identifies the type of contents of the file; for example, "ada" denotes syntactically complete Ada source code, "inc" denotes incomplete Ada code (the timing loop code must be included), and "txt" denotes a data file. In this Primer, we assume that all letters in files names are lowercase; however, some systems will translate them to uppercase.
These files are all in the "aces/support" directory. Each setup file name begins with the letter 'z'. File names in some other categories also begin with 'z', as discussed in later paragraphs. The first two characters of every setup file are one of the following combinations:
a. "zc" : Nearly all of these files (names ending in ".tpl") are templates for script files. The "setup" program (see Sections 3 and 8) uses them to generate implementation-specific scripts for fairly primitive operations (e.g., compile, link, copy source file, or delete library unit). The remaining files (names ending in ".cp", ".el", or ".ada") are source code files used in setting up the testing environment.
b. "zg" : Most of these files are source code files used in setting up the testing environment. These include a sample assembly language routine for determining an address (for VAX) and a sample Ada subprogram for determining CPU time (for VAX-Ada). A few of the "zg" files (names containing "start", "stop", and "verfy") are fragments of timing code that are automatically inserted into performance test programs before compilation.
c. "zm" : These files contain Ada source code providing access to certain mathematical functions, including a random number generator.
d. "zp" : These files contain Ada source code, sample data, and script templates for compiling, linking, and testing the common library units, the performance testing harness, and the analysis tools.
These files are all in the "aces/support" directory. Each harness file name begins with the letters "zh", and contains Ada units or data for creating, testing, and using the Harness program. Creating the Harness program is discussed in Section 3.2.11, and its use is discussed in Section 4.
There are 21 groups of performance test files, each with its own prefix. The Harness program is used to create command scripts for completing, compiling, linking, and executing all the tests in the group. Incomplete test files (names ending with ".inc") lack the timing code, which is inserted by running a preprocessor program ("zg_incld"). Performance test file prefixes are "ap", "ar", "cl", "do", "dr", "dt", "gn", "io", "in", "ms", "oo, "op", "po", "pt", "sr", "st", "su", "sy", "tk", "xh", and "ud". The files for the group with prefix "##" are in the directory "aces/tests/##". The performance test files are typically used in conjunction with the Harness program, whose use is discussed in Section 4.
There are four assessors, one each for the symbolic debugger ("yb*" files), compile-time and run-time capacities ("yc*" files), the diagnostics system ("yd*" files), and the library management system ("yl*" files). Each assessor includes Ada software files, sample command scripts, documentation files, and a reporting template. The files are in the appropriately named subdirectories of "aces/assessrs". In addition, the capacity assessor has a data file that can be edited to select testing limits. The assessors are discussed in Section 6.
There are four programs in the analysis set: a user interface (Menu), a data extraction program (Condense), Comparative Analysis (CA), and Single System Analysis (SSA). CA will be used when evaluating for selection or comparing releases of an implementation. SSA will be used when evaluating a selected implementation. All the analysis program file names, including Ada source files, command scripts, and data files, begin with "za". These files are in the directory "aces/support". Their use is discussed in Section 5.
There are 156 files in the Quick-Look set: Setup software and data files, testing software, reporting software, and documentation. Note that much of this code and the documentation duplicates that found in other directories. This was done so that the Quick-Look facility could be accessed early by the users.
All of the ACES documentation is distributed in two formats, ASCII and PostScript. The documentation file names all begin with "zd"; they are found in the directory "aces/docs". The ASCII versions have the extension ".txt" and the PostScript versions have the extension ".ps".
The versions of ACES that are available on the Internet are stored in 11 compressed files, using the for-public-use "zip" software. The "zip" software is subject to the following notice:
"Copyright (C) 1990-1993 Mark Adler, Richard B. Wales, Jean-loup Gailly, Kai Uwe Rommel, Igor Mandrichenko and John Bush. Permission is granted to any individual or institution to use, copy, or redistribute this software so long as all of the original files are included, that it is not sold for profit, and that this copyright notice is retained."
This compression program and its companion "unzip" program are compatible with recent versions of PKZIP for DOS systems. They are available for many platforms, including UNIX, VMS, DOS, Windows, OS/2, and Macintosh. The software and its documentation may be obtained from the following sites, among others:
ftp.uu.net:/pub/archiving/zip/...
wuarchive.wustl.edu:/mirrors/misc/unix/...
wuarchive.wustl.edu:/packages/compression/...
wuarchive.wustl.edu:/mirrors/misc/vaxvms/...
wuarchive.wustl.edu:/mirrors/msdos/zip/...
wuarchive.wustl.edu:/mirrors/msdos/windows3/...
ftp.informatik.tu-muenchen.de:/pub/comp/os/os2/archiver/...
sumex-aim.stanford.edu:/info-mac/util/mac-unzip-50pl.hqx
The directories containing the compressed ACES files also contain a plain-text file called "README". This file describes the compressed files and their contents. If the user does not have sufficient disk space to store all the ACES files, he/she may wish to download only those files needed for various steps. For example, one obtains all of the files necessary for the entire Pretest process by downloading and unzipping only the "support.zip" file. It contains all of the files needed for creation and initial testing of the ACES including the performance test files from the Sort subgroup of the Classical test group (cl_so*).
The compressed files are set up to create a subdirectory tree whose top level is named "aces", and to place files in the appropriate subdirectories. This subdirectory tree is depicted in Figure 2-1. Files needed for more than one purpose are duplicated. When "qlook.zip" is unzipped, it creates the "aces/qlook" directory. One of the files placed in this directory is named "ql_work.zip". This is a compressed file containing all the files necessary for compiling and executing the Quick-Look version of the Setup program (discussed in Section 8). Likewise, when "support.zip" is unzipped, the "aces/working" directory will contain a compressed file, "zp_work.zip", containing all the files necessary for compiling and executing the full testing version of the Setup program (discussed in Section 3).
The "aces" distribution directory should be protected against modification. The testing procedures described in this document (and the software that supports these procedures) assume that the user has established both "source" and "working" directories separate from the "distribution" directory. The software copies needed files from the "aces" distribution directory into the source directory, where they may be modified if required.
Figure 2-1 Distribution Directory Structure
Before beginning performance testing or assessor execution, the user must perform some preliminary activities known as the Pretest phase. The Pretest is designed to achieve the following goals:
(a) Create command scripts for performing basic operations such as invoking the compiler with various option combinations, invoking the linker, and removing units from the library;
(b) Compile support units that will be used throughout the evaluation process; and
(c) Test these support units.
The first task in the setup process is to build the SETUP executable program. The SETUP program queries the user for basic information concerning operating system commands, compiler and linker commands, and various conventions for command script files. Using this information, SETUP creates the command scripts described in (a) above. SETUP also creates command scripts for the Pretest steps. The user executes these command scripts, captures the results, and verifies the expected results. The user must occasionally make certain decisions based on the results.
The ACES user must be familiar with the commands that are used by the implementation's operating system(s) and the Ada compilation system. The SETUP program has predefined commands for a variety of compilers and operating systems. In addition, it provides some examples.
Should the user need a manual method for running the ACES, the symbols and commands are further explained in Appendix A of the ACES User's Guide. This process many need to be used if major complications arise that would prevent the use of the automated Setup process.
This setup section makes references to three different categories of directories, "distribution", "source", and "working". The "distribution" category refers to the various directories in which the original ACES files reside. These files must not be modified, and are kept separate for this reason. The "source" category refers to one specific directory into which copies of the "distribution" files are placed. These files are occasionally modified by the user. Therefore, it is necessary to keep this directory separate from the "distribution" directories. The "working" category refers to one specific directory from which programs are executed and into which the system generated files are placed. The system often makes copies of the "source" files and uses these "working" copies for the actual processing. The files in the working directory are often modified and deleted by the system. Therefore, it is necessary to keep this directory separate from the "distribution" and "source" directories. The user must specify a path name for both the "distribution" directory and the "source" directory while running the SETUP program (below).
The following sections discuss the use of SETUP and provide the details needed for each Pretest step.
At the beginning of each of the Pretest step sections is a list of the files that must be in the specified source and working directories in order for the step to run properly. The command scripts will automatically copy most files from the source directory into the working directory (exceptions are noted when they occur). The command scripts are set up to delete files from the working directory, and library units from the Ada library when they are no longer necessary for the Pretest.
The source code for the SETUP program is contained in the two distribution files "zp_setup.ada", and "zp_defs.ada". The program requires several input files and produces several output files.
Perform the following steps to prepare SETUP for execution:
(a) Create an Ada library and identify it as the current Ada library;
(b) Unzip the file "zp_work.zip" into the working directory. This file can be found in either the "support.zip" file or in the DISTRIBUTION directory "aces/support". The following files should be in the working directory:
zc_adack.tpl zc_adaho.tpl zc_adano.tpl zc_adaop.tpl zc_adasp.tpl zc_cmpck.tpl zc_cmpno.tpl zc_cmpop.tpl zc_cmpsp.tpl zc_cpada.tpl zc_cpyck.tpl zc_cpyno.tpl zc_cpyop.tpl zc_cpysp.tpl zc_delbo.tpl zc_delsb.tpl zc_delso.tpl zc_dlada.tpl zc_dlexc.tpl zc_dlhex.tpl zc_holi.tpl zc_horun.tpl zc_linho.tpl zc_link.tpl zc_link1.tpl zc_link2.tpl zc_lnk.tpl zc_time1.tpl zc_time2.tpl zc_run.tpl zc_setli.tpl zg_incld.tpl zh_usr.tpl zp_defs.ada zp_envrn.def zp_envrn.dgn zp_envrn.dns zp_envrn.dth zp_envrn.ngn zp_envrn.nth zp_envrn.nvr zp_envrn.udf zp_envrn.ugn zp_envrn.uns zp_envrn.uth zp_envrn.uvr zp_envrn.vns zp_envrn.vva zp_quest.txt zp_scrnm.txt zp_setup.ada zp_stp00.tpl zp_stp01.tpl zp_stp02.tpl zp_stp03.tpl zp_stp04.tpl zp_stp05.tpl zp_stp06.tpl zp_stp07.tpl zp_stp08.tpl zp_stp09.tpl zp_stp10.tpl zp_stp11.tpl zp_stp12.tpl zp_stp13.tpl zp_stp14.tpl zp_stp15.tpl
To make the Setup process faster and less cumbersome, various default files have been provided. The Setup program will access these files at the user's request during the initial question phase of the program.
Predefined information for your operating system/compiler implementation can be found in the following files. If yours is not listed you can edit one of the existing files for your use. The file "zp_envrn.udf" contains the symbols, but none of the values for Setup. It is provided for your convenience.
zp_envrn.dgn DOS/GNAT
zp_envrn.dns DOS/non-specified compiler
zp_envrn.dth DOS/Thomsoft (Alsys)
zp_envrn.ngn non-specified operating system/GNAT
zp_envrn.nth non-specified operating system/Thomsoft (Alsys)
zp_envrn.nvr non-specified operating system/Rational (Verdix VADS)
zp_envrn.udf user defined operating system and compiler
zp_envrn.ugn UNIX/GNAT
zp_envrn.uns UNIX/non-specified compiler
zp_envrn.uth UNIX/Thomsoft (Alsys)
zp_envrn.uvr UNIX/Rational (Verdix VADS)
zp_envrn.vns VMS/non-specified compiler
zp_envrn.vva VMS/Vax-Ada
The "zp_envrn.???" files contain defaults for various compiler/operating system combinations. The file "zp_envrn.udf" is a file that can be customized by the user for an implementation for which there is no default. The file "zp_envrn.def" contains general defaults that can be used by all systems when running a simple version of Setup. It is suggested for first time users.
(c) If necessary, rename "zp_setup.ada" and "zp_defs.ada" to meet any naming requirements of the implementation;
(d) Compile "zp_setup.ada";
(e) Compile "zp_defs.ada"; and
(f) Link the main program "SETUP" to form an executable.
Before executing SETUP, the user is advised to consider the choice of measurement techniques and mathematical function options.
On choosing measurement options:
There are four classes of choices that must be made during the execution of the Setup program, relating to the execution time measurement technique, the compilation and linking time measurement techniques, the code size measurement techniques, and the implementation of the mathematical functions. The following paragraphs discuss these choices.
(a) Execution time measurement: Elapsed or CPU
Execution time may be measured using elapsed (wall-clock) time or CPU time. Elapsed time measurements make use of the "Calendar.Clock" function. If the target (or self-targetted) system has more than one user or has many daemons contending for the processor, using elapsed time may result in many unstable measurements. In this case, CPU time measurements may be better. To obtain CPU time measurements, the user must modify the file (in the source directory) "zg_cpu.ada", so that the function "zg_cpu" returns accumulated CPU time (on the target system) as a value of type Duration. (This function must apply to the target system.)
(b) Compilation and linking time measurement: Elapsed or CPU
Again, these measurements may use "Calendar.Clock", representing elapsed time. If the user wishes to use CPU time, she/he must provide a host-based CPU time function. For a self-hosted system, this function must be in a file named "zg_cpu.ada". For a cross-compilation system, the required file name is "zg_hocpu.ada". In Pretest Step 4, a test is made to determine whether the user-defined "zg_cpu" function is appropriate for measuring compilation and link time.
NOTE: If elapsed time is chosen for execution-time measurements, then elapsed time MUST also be used for compilation and linking time.
(c) Code size measurement: Label'Address or Get Address
The user should initially choose the Label'Address method of measuring code size. If the result of Pretest Step 2 shows that this technique is not appropriate for the system being testing, then the answer to the corresponding Setup question should be changed (by re-running Setup and choosing Option One). In this case, the user must provide a Get Address function ("zg_getad") that returns the address from which the function was called. If code size measurements are not desired, this function may return a constant value, and all sizes will be reported as zero.
(d) Mathematical functions
There are several options for the mathematical functions required by the performance tests and the support programs. When running Setup, you will be asked questions that determine which option is used.
1. Ada 95: If the implementation is for Ada 95, then the mathematical functions required by the standard will be used, so no choices are necessary.
2. NUMWG: If an Ada 83 implementation supports the Generic_Elementary_Functions package recommended by the Numerics Working Group (NUMWG), then this packages should be used. Choose the NUMWG option. In the case of a cross-compilation system, this option may be chosen for either the target or the host, or both systems.
3. VENDOR: It is possible that an Ada 83 implementation supports all the functionality of the NUMWG recommendation, but does not use the same package specification, choose the VENDOR option. (In the cross-compiler case, the VENDOR option must be chosen for both the target and host system, or for neither.) In this case, "call-through" package files, "zm_math.ada" and "zm_dblma.ada" must be created. The files "zm_math.dec" and "zm_dblma.dec" are provided as examples (based on the Digital Equipment Corporation (DEC) VAX-Ada compiler).
4. PORTABLE: If an Ada 83 implementation supports neither the NUMWG nor the VENDOR option, then choose the PORTABLE option. In this case, the ACES mathematical packages (that should be portable to most implementations) are used. In the case of a cross-compilation system, it is possible to choose the PORTABLE option for either the target or the host, or both systems.
Run the SETUP executable. It first looks for the existence of the file "zp_envrn.txt". This file will not exist the first time that SETUP is executed from any given working directory. One of the results of running SETUP is the building of the "zp_envrn.txt" file. This file contains the questions and answers for a series of symbols that define a particular implementation's environment. If the file does not exist, the user is led through a series of questions that identify the implementation's environment. If the file does exist, the user is given the following options:
1. Modify some of the existing answers;
2. Use the existing values and regenerate the scripts;
3. Answer all the questions all over again.
Two special cases need to be discussed. Once the user determines that a compiler is either a cross compiler or a self-hosted compiler, the environment file is locked into that decision. The same is true for the Ada 83 vs. Ada 95 implementation question. The answers to these two questions determine another level of questions that are appropriate for the selections made. In order to change the values of either cross vs. self-hosted or Ada 83 vs. Ada 95, the user needs to rerun the SETUP executable and select Option 3. (Answer all questions.)
As described above, responses are stored in the file "zp_envrn.txt". This file may be edited to correct any mistakes, by rerunning Setup and Selecting Option 1, "modify some of the existing answers". After running SETUP, the following files can be deleted from the working directory:
zc_*.tpl all zp_envrn.* files OTHER THAN zp_envrn.txt
zp_quest.txt
zp_scrnm.txt
zp_setup.ada
zp_stp*.tpl
zh_usr.tpl
If SETUP ever needs to be run again, ensure that the files discussed in Section 3.1.1(b) are available in the working directory.
Run command scripts in batch mode (via the "submit" command) except when instructed to run a command script interactively. Use the "/log = [ ] log file name" notation on the "submit" command line to ensure that the log file is written in the working directory.
When SETUP prompts for a "special first line" respond with "SET NOON" so that the command scripts continue to run even if errors occur.
Be reminded that when submitting a batch job, the system first runs the "login.com" file, so it is necessary to set the default directory to the working directory. Issue the following commands in the "login.com" file:
$ if F$MODE ( ) .eq. "BATCH" then
$ SET DEFAULT working
Many Ada compilers for UNIX require that Ada source code files have the suffix ".a". If manually copying ".ada" files into the working directory, be sure to rename them appropriately. The step files will do this automatically.
Since there are both script and executable files "zg_incld", it is necessary to specify a file name suffix for script files. Since the ACES - distributed scripts use ".unx", this is a good choice.
The command scripts generated by SETUP and the Harness program will not normally be executable, until the UNIX "chmod" command is run to make these scripts executable. Note that the scripts supplied with the ACES have been tested under the C shell.
Scripts may be run in foreground, background, or batch mode. In any case, all of the output should be directed to a log file. For example, "zp_stp01.unx >& zp_stp01.log &", is a UNIX command to execute "zp_stp01.unx" in the background and write output to a log file named "zp_stp01.log".
In order to capture test output, use the ">>PRETEST.LOG" as suggested in the appropriate question asked by SETUP. To keep separate log files for separate steps, rename PRETEST.LOG appropriately after running each step.
For the Alsys compiler, when entering compile and link options as part of a batch command, do not depend on default values. Explicitly add each option and value. For example: when asked for the command to compile without optimization, include it as part of the command even though no optimization is the default.
The default environment files for GNAT contain invocations of three non-standard scripts, "gnat-comp", "gnat-compop", and "remove-unit". The GNAT user will have to replace these commands or supply appropriately named scripts. The purposes of these scripts are as follows:
gnat-comp Apply GNATCHOP and execute the resulting script to compile the file named by the parameter, using the default gcc options.
gnat-compop Apply GNATCHOP and execute the resulting script to compile the file named by the parameter, using a gcc option to optimize.
remove-unit Remove the specification and body of the unit named by the parameter from the "library".
Figure 3-1, 3-2, and 3-3 contain UNIX versions of these scripts, as used by the ACES development team. Note that the gnat-comp script (Figure 3-1) assumes that the file named in the parameter has no suffix.
#!/bin/csh
#
# Compile without optimization
echo Compiling $1
gnatchop -s -w $1
$1.sh
unalias rm
rm -f $1.sh
Figure 3-1 Sample "gnat-comp" script
#!/bin/csh
#
# Compile with optimization
echo Compiling $1
gnatchop -s -w $1
$1.sh -02
unalias rm
rm -f $1.sh
Figure 3-2 Sample "gnat-compop" script
#!/bin/csh
unalias rm
rm -f $1*.ads
rm -f $1*.adb
rm -f $1*.ali
rm -f $1*.o
Figure 3-3 Sample "remove-unit" script
The pretest subsections below follow the general scenario described here. Note that not all subsections are needed in all steps.
STEP NUMBER: [SECTION NUMBER] [STEP NUMBER] [STEP TITLE/NAME]
DESCRIPTION: What this step is designed to accomplish.
PREREQUISITE: Which steps or what other actions must have been performed before this step can be successfully executed.
REQUIRED/OPTIONAL: The Pretest steps are designed to be run sequentially, but occasionally some steps are not required. This section explains the status of this step.
FILES NEEDED IN SOURCE DIRECTORY: This should have been accomplished when executing Step 0.
FILES NEEDED IN WORKING DIRECTORY: Should have been accomplished by a previous, required step. When rerunning Setup, verify that needed files were not deleted by subsequent steps.
PREPARATION: Tasks that must be done and issues that need to be decided before running the step file.
EXECUTION: Actually issuing the command to run the step file and sending output to a log file. Note that each step may not produce a log file with each implementation. See Sections 3.1.3.1, 3.1.3.2, and 3.1.3.3 for suggestions on generating log files.
RESULTS EXPECTED: What you should find in the log file, on the screen or in the working directory (source directory for Step 0). If the results do not match the description of the expected results, see the discussion of the corresponding Pretest step in the User's Guide.
FOLLOW-UP NEEDED: Decisions that need to be made or actions that need to be taken as a result of that step's output.
FILES NO LONGER NEEDED IN WORKING DIRECTORY: Files that may be deleted from the working directory to make it less crowded. Remember that these files may be needed if Setup is rerun.
DESCRIPTION: This step will automatically copy files from the "distribution" to the "source" directory.
PREREQUISITE: Compiling, linking, and executing Setup.
REQUIRED/OPTIONAL: If this cannot be used for a specific implementation, either edit the command file to meet user needs or manually copy all of the files named in "zp_stp00.tpl" into the "source" directory.
FILES NEEDED IN SOURCE DIRECTORY: None.
FILES NEEDED IN WORKING DIRECTORY: zp_stp00.(command script)
PREPARATION: None.
EXECUTION: Run the file.
RESULTS EXPECTED: Copies of all of the files that are in either "support.zip" or "/aces/support/" should now be in "source".
FOLLOW-UP NEEDED: Verify that the files were copied.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp00 (command file)
zp_stp00 (log file)
DESCRIPTION: This step compiles two global packages into the library. It also compiles, links, and executes a simple program that tests the ability of Text_IO to output strings, integers, and floating-point values.
PREREQUISITE: Low-level command scripts must have been generated by SETUP and Step 0 must have been executed.
REQUIRED/OPTIONAL: This step MUST be completed before performance testing can be attempted.
FILES NEEDED IN SOURCE DIRECTORY:
zg_glob1.ada zp_talk.ada
zg_glob2.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp01 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp01" command script and send the results to a log file.
RESULTS EXPECTED: A log file with the following information:
Display floating point numbers 1.50000E+01 1.500E+1 15.0
Display floating point numbers -1.23450E+04 -1.235E+4 -12345.000
Display integers 100 22222
FOLLOW-UP NEEDED: Verify results.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp01 (command script)
zp_stp01 (log file)
DESCRIPTION: This step determines whether applying the 'ADDRESS attribute to a label can be used to measure code size for the implementation being assessed.
PREREQUISITE: Step 1 must have been completed.
REQUIRED/OPTIONAL: If the user already knows whether label'Address is supported, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_label.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp02 (command script)
PREPARATION: A subprogram in file "zp_label.ada" may require modification if the target system's type System.ADDRESS is not suitable for unchecked-conversion to a 32-bit integer type. If this situation arises, modify the function "pkg01_zp_label.Subtract_Address", which calculates the difference between two addresses and expresses the result as a floating point value.
EXECUTION: Execute the "zp_stp02" command script and send the results to a log file.
RESULTS EXPECTED: The program "zp_label" should have been compiled, linked, and executed, and one of the following results generated:
1. "Label'ADDRESS attribute works correctly
SIZE IN 8 BIT BYTES = (some number in the range 4 to 16)";
2. "The difference between two label'ADDRESSes is outside the expected range of 4 to 16 bytes. Please compare this result with a system map or machine code listing to verify accuracy. SIZE in 8 BIT BYTES = (number)"; or
3. "zp_label.ada" does not compile.
FOLLOW-UP NEEDED:
1. Verify the expected results.
In the first case, the ACES can use the default (and preferred) method of measuring code size (i.e., subtracting the results of applying the 'ADDRESS attribute to statement labels).
In the second and third cases, the ACES cannot use the default method of measuring code size. In these cases, the user must decide whether to provide an alternate method of measuring code size, as described below, or choose not to measure code size.
To provide an alternate method of measuring code size, the user must define the function "zg_getad". This function must return the address from which it was called. The Ada declaration of this function appears in the files "zg_glob3.elg" and "zg_glob3.cpg", one of which will be used in Step 5. In these files, it is assumed that the function's body is written in assembly code, and that a "pragma interface" and a "pragma import_function" are used to reference the body. Note that the use (or even the existence) of such a pragma is implementation dependent, and that modification may be necessary.
A sample function (for VAX systems) is given in the file "zg_getad.mar". In Step 5, it may be necessary to modify the pragmas in the "zg_glob3" file, and it may be necessary to modify the command script for Step 5 to account for translating the function body and making it available to the Ada library.
In order to understand the implementation's interface to assembly language routines, we recommend that a simple Ada function be compiled with the option to generate embedded assembly code. This code can serve as a starting point for writing the "zg_getad" function.
If the user chooses not to measure code size, there is another choice to be made. If zp_label reports that the size is 0, then the user may continue to use the default method, and all sizes will be reported as 0. If "zp_label.ada" fails to compile, then the user must define the function "zg_getad" such that it always returns the same address. This will result in all sizes being reported as 0.
2. Record the method of measuring code size, as follows:
During SETUP the user is asked to identify the code measurement method. The results of this step either confirm or invalidate the method declared. If this test step confirms the user's selection, the user can continue. If the declared method is invalid, the user must rerun SETUP and modify the value. To do this run the SETUP executable. Select Option one (i.e., modify the answers to some of the existing answers). Identify the question for Code Measurement Technique. Modify this answer appropriately.
The answer to this question is used as part of one or more file names. If your operating system is case-sensitive, you must use the same case as is used for the names of your source code files.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp02 (command file)
zp_stp02 (log file)
DESCRIPTION: This step compiles units that provide checks on the behavior of the CPU time function or the Calendar.Clock function, depending on whether CPU time or elapsed time is to be measured. The actual checking is performed in Step 4.
PREREQUISITE: Step 1 must have been completed.
REQUIRED/OPTIONAL: If CPU time measurements are NOT being used and the user knows that Calendar.Clock is accurate, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
Required for all cases:
zp_tcal1.ada
zp_tcal2.ada
Required for CPU time measurement cases:
zg_cpu.ada
zg_hocpu.ada (if you are using a cross-compiler)
zp_tcal3.ada
zp_tcal4.ada (includes source for zp_tcal5 also) only if CPU is used for compilation time)
FILES NEEDED IN WORKING DIRECTORY:
zp_stp03 (command script)
PREPARATION: The first task in this step is to decide whether to measure elapsed time or CPU time (see User's Guide, Section 5.1.2 for some discussion of the issues). If CPU time is to be used, a function that reports CPU time as a value of type DURATION must be supplied. For cross-compilers, it may be necessary to supply two functions, one for the host system and one for the target system. There is a VAX-Ada example in the file "zg_cpu.dec". (Note: This file "zg_cpu.dec" can be copied into your subdirectory and renamed, as needed for use on your system.) The file name for a self-hosted system must be "zg_cpu.ada". For a cross-compilation system, the file name for the host system must be "zg_hocpu.ada" and the file name for the target system must be "zg_cpu.ada".
If CPU time measurements are being used, the command script generated for this step is not complete. The user needs to insert commands to compile or assemble non-Ada units required by zg_cpu and to link programs zp_tcal3, zp_tcal4, and zp_tcal5. See User's Guide, Section 5.1.3 and the zp_stp03 command script file for more information.
When SETUP was executed, the user was asked to identify both the method used to measure compilation time and the method to measure execution time. The user was asked whether CPU (cp) time or elapsed wall-clock time (el) was to be used for each. These letters (i.e., the answers "cp", "el") are used as part of one or more file names. If your operating system is case-sensitive, you must use the same case as is used for the names of your source code files.
NOTE THAT IT IS NOT PERMISSIBLE TO MEASURE COMPILATION AS CPU TIME WHEN MEASURING EXECUTION WITH ELAPSED TIME.
If either compilation or execution time is to be measured in CPU time, modify the "zp_stp03" command script to account for compiling or assembling non-Ada code. It may also be necessary to edit the "link" command scripts (zc_lnk, zc_link, zc_link1, zc_link2, zc_linho, and zc_linkd) to account for linking any non-Ada units used in this process.
EXECUTION: Execute the "zp_stp03" command script.
RESULTS EXPECTED:
Executables in working directory:
zp_tcal1
zp_tcal2
Additional executable files expected in the working directory if using CPU time:
zp_tcal3
zp_tcal4
zp_tcal5
FOLLOW-UP NEEDED:
Verify the following:
Confirm that zp_tcal1 and zp_tcal2 were successfully compiled and linked for running on both the host and the target (or the common host/target).
If time is to be measured in CPU time, verify that zp_tcal3, zp_tcal4, and zp_tcal5 were successfully compiled and linked for running on the host (or the common host/target).
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp03 (command script)
zp_stp03 (log file)
DESCRIPTION: This step checks the accuracy of the CPU time measurement functions or the elapsed time measurement functions, depending on the method chosen for measuring compilation and execution time.
PREREQUISITE: Steps 1 and 3 must have been completed.
REQUIRED/OPTIONAL: If the user already knows that the selected timing functions are accurate, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY: Those copied in Step 0.
FILES NEEDED IN WORKING DIRECTORY:
zp_tcal1 executable
zp_tcal2 executable
zp_tcal3 executable (only if using CPU time)
zp_tcal4 executable (only if using CPU time)
zp_tcal5 executable (only if using CPU time)
zp_stp04 (command script)
PREPARATION:
Identify the paragraph below that describes the system being assessed and run the tests as described.
* Self-hosted systems:
+ Measuring elapsed time:
- Run zp_tcal1 and zp_tcal2 interactively, using a stop-watch to check the timings.
+ Measuring CPU time:
- Run zp_tcal1 and zp_tcal2 interactively, using a stop-watch to check the timings.
- Run zp_tcal3 interactively, preferably as the only process running on the system.
- If the system supports concurrent jobs, run two copies of zp_tcal3 at the same time. For UNIX systems, start one copy in the background using the "&" and immediately start another copy in the foreground. For VMS systems, start one copy in the background using the SPAWN/NOWAIT command and immediately start another copy in the foreground.
* Cross-compilation systems:
+ Measuring elapsed time:
- Run zp_tcal1 and zp_tcal2 interactively on the host system, using a stop-watch to check the timings.
- Run zp_tcal1 and zp_tcal2 interactively on the target system, using a stop-watch to check the timings.
+ Measuring CPU time:
- Run zp_tcal1 and zp_tcal2 interactively on the host system, using a stop-watch to check the timings.
- Run zp_tcal3 interactively on the host system, preferably as the only process running on the system.
- If the host system supports concurrent jobs, run two copies of zp_tcal3 on the host system at the same time. For UNIX systems, start one copy in the background using the "&" and immediately start another copy in the foreground. For VMS systems, start one copy in the background using the SPAWN/NOWAIT command and immediately start another copy in the foreground.
- Run zp_tcal1 and zp_tcal2 interactively on the target system, using a stop-watch to check the timings.
- Run zp_tcal3 interactively on the target system, preferably as the only process running on the system.
- If the target system supports concurrent jobs, run two copies of zp_tcal3 on the target at the same time (as suggested above).
EXECUTION: Execute the "zp_stp04" command script.
Note that this script does nothing if elapsed time measurements have been chosen for execution timings.
RESULTS EXPECTED:
For zp_tcal1 and zp_tcal2, a message should be displayed every 60 seconds for 15 minutes.
For zp_tcal3, the execution as the only process should report CPU time that is less than, but near, the elapsed time. During the second execution, each copy should report CPU time that is roughly half of that reported during the first execution.
For zp_tcal4 and zp_tcal5, see the following section.
FOLLOW-UP NEEDED:
For zp_tcal1 and zp_tcal2, a message should be displayed every 60 seconds for 15 minutes. If the intervals between messages vary noticeably, then Calendar.Clock is not reliable. See the User's Guide, Section 5.1.3.
For zp_tcal3, the execution as the only process should report CPU time that is less than, but near, the elapsed time. During the second execution, each copy should report CPU time that is roughly half of that reported during the first execution.
When the zp_stp04 command script is executed, it runs two programs: zp_tcal4 and zp_tcal5. The results from these programs are used to determine whether the CPU time function returns a value relative to the beginning of the process or relative to the beginning of the program. For more details see the User's Guide Section 5.1.3.
If the results of zp_tcal5 show that the CPU time function returns PROGRAM time (which is NOT appropriate for measuring compilation and linking time), then one option is to measure compilation and linking with elapsed time, rather than CPU time. If this option is chosen, rerun SETUP and modify the value of the answer. The other option is to measure compilation and linking with CPU time. If this option is chosen, it is necessary for the user to provide a different zg_cpu function that somehow retrieves process time.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp04 (command script) zp_tcal1 (executable)
zp_stp04 (log file) zp_tcal2 (executable)
zp_tcal3 (executable)
zp_tcal4 (executable)
DESCRIPTION: This step compiles the remaining global packages, a random number generator package, and a mathematics package. It also creates executables for producing baseline timing data, for including timing loop code into the distributed performance test sources, and for creating compilation and linking time stamps.
PREREQUISITE: Step 1 must have been completed.
REQUIRED/OPTIONAL: This step MUST be executed before performance testing can begin.
FILES NEEDED IN SOURCE DIRECTORY: The answers given by the user while running Setup put file names in the zp_stp00 command script to be copied to the "source" directory. Examine each of the following descriptions to ensure that the correct files, as described below were copied into "source".
1. For ALL cases:
zg_glob4.ada zg_glob5.ada zg_glob6.ada zm_math.ada zm_dblma.ada zc_data.ada
(Note: That is the VENDOR option was chosen, then the "zm_math.ada" and "zm_dblma.ada" files are the "call-through" packages discussed in Section 3.1.1.)
2. Random number generator (select one of the following):
If the implementation supports the declaration of 32-bit integer types:
zm_ran32.ada
If the implementation does NOT support the declaration of 32-bit integer types:
zm_ran16.ada
3. Math packages (select files if needed):
a. Ada 95 standard, the following files are required:
zm_math.a95 zm_dblma.a95
b. Ada 83 implementation:
1) If the PORTABLE option was chosen for a self-hosted system or for either HOST or TARGET of a cross-compilation system, then the following files are required:
zm_math.por zm_depen.por zm_genma.ada
2) If the PORTABLE option was chosen for a self-hosted system or for the TARGET or a cross-compilation system, then the following file is required:
zm_dblma.por
4. Cross/self-hosted compiler (select one of the following):
a. If the implementation is CROSS-compilation system, then the following file is required:
zg_iniso.cro
b. If the implementation is a SELF-hosted system, then the following file is required:
zg_iniso.sel
5. Measuring execution time and code size (select one of the following groups):
a. execution time: CPU
code size: Label'Address
The following files are required:
b. execution time: CPU
code size: Get Address
The following files are required:
zg_glob3.cpg zg_init.cpg zg_incld.cpg zc_time1.cp
zc_time2.cp zc_link1.cp zc_link2.cp
c. execution time: ELAPSED
code size: Label'Address
The following files are required:
zg_glob3.ell zg_init.ell zg_incld.ell zc_time1.el
zc_time2.el zc_link1.el zc_link2.el
d. execution time: ELAPSED
code size: Get Address
The following files are required:
zg_glob3.elg zg_init.elg zg_incld.elg zc_time1.el
zc_time2.el zc_link1.el zc_link2.el
6. If ELAPSED time measurements were chosen for COMPILATION and link time, then the following files are required:
zc_time1.el zc_time2.el zc_link1.el zc_link2.el
FILES NEEDED IN WORKING DIRECTORY:
zp_stp05 (command script)
PREPARATION: If some method other than label'Address is being used to measure code size (i.e., option "g"), then modify the "zp_stp05" command script to account for translating and linking the "zg_getad" units. See the comments in the zp_stp05 command script file for hints on how to do this.
EXECUTION: Execute the "zp_stp05" command script and send the results to a log file.
RESULTS EXPECTED:
For all CROSS-COMPILERs, the HOST library should contain the following units:
For a CROSS-COMPILER using the PORTABLE option for the HOST, the HOST library should contain the following unit:
zm_math.math_dep
* Note that this is a subunit, and may not be listed separately.
For all CROSS-COMPILERs, the TARGET library should contain the following units:
proc_spoil_glob4_variables zg_glob1 zm_math
zg_glob2 zm_ran
zg_glob3
zg_glob4
zg_glob5
zg_glob6
For a CROSS-COMPILER using the PORTABLE option for the TARGET, the TARGET library should contain the following units:
mathematical_exceptions zm_dblma.math_dep
zm_depen
zm_genma
zm_math.math_dep
* Note that the two ".math_dep" subunits may not be listed separately.
For all SELF-HOSTED COMPILERS the library should contain the following units:
proc_spoil_glob4_variables zg_glob1 zm_math
zg_glob2 zm_ran
zg_glob3
zg_glob4
zg_glob5
zg_glob6
For SELF-HOSTED COMPILERS using the PORTABLE option, the library should contain the following units:
mathematical_exceptions zm_dblma.math_dep
zm_depen
zm_genma
zm_math.math_dep
* Note that the two subunits ".math_dep" may not be listed separately.
The following executable to be run on the host should have been created in the working directory:
zc_link1 zc_time1 zg_incld
zc_link2 zc_time2
FOLLOW-UP NEEDED: If the implementation is a CROSS-COMPILER, read through the log file looking for instructions on how to extract the following files from the log files. These files are used when executing performance tests to check the consistency of timing measurements.
zg_cpy.op zg_cpy.no zg_cpy.ck zg_cpy.sp
If the implementation is a self-hosted compiler, verify that the following files were created:
zg_cpy.op zg_cpy.no zg_cpy.ck zg_cpy.sp
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp05 (command script)
zp_stp05 (log file)
DESCRIPTION: This step helps to determine the minimum number of times each test must be executed to eliminate the effects of clock "jitter" and other "noise" on the system. As a consequence of this step, it may be necessary to modify and re-compile the body of the global package "zg_glob3".
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user wishes to use the default minimum numbers, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY: The appropriate copy of the following file should be in the source directory:
zp_basic.??? The suffix of this file is either ".ell", ".elg", ".cpl", or ".cpg".
FILES NEEDED IN WORKING DIRECTORY: zp_stp06 (command script)
PREPARATION: Determine that the appropriate version at zp_basic is in the source directory. Verify it against the options selected in Setup and described in Section 3.1.1 and 3.2.5 (step 5).
EXECUTION: Execute the "zp_stp06" command script and send the results to a log file.
RESULTS EXPECTED: A log file with the results of the execution of the script.
FOLLOW-UP NEEDED:
Study the output of the command script. It may recommend changing the value of BASIC_ITERATION_COUNT. Normally, the default value of 2**17-1 is acceptable. If change appears necessary, follow the procedure below:
1. Get a copy of the version of zg_glob3 that was used in Step 5. Extract the body of package zg_glob3 into another file. Locate the beginning of the package body's statement-part. These are the first two lines:
BEGIN
excessive_time := 1800.0; -- 30 minutes
Immediately after the BEGIN, add statements assigning the appropriate values (see below) to BASIC_ITERATION_COUNT_LOWER and BASIC_ITERATION_COUNT_UPPER.
Suppose the desired value of BASIC_ITERATION_COUNT is 2**N - 1.
If N <= 14 then use
BASIC_ITERATION_COUNT_LOWER := 2**N - 1;
BASIC_ITERATION_COUNT_UPPER := 0;
otherwise, use
BASIC_ITERATION_COUNT_LOWER := 2**14 - 1;
BASIC_ITERATION_COUNT_UPPER := 2**(N - 14) - 1;
(Do not use "N"; instead, replace "N" with the value of the exponent as recommended in the output.)
2. Compile the body (only) of zg_glob3, as modified.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp06 (command script)
zp_stp06 (log file)
The source file (if any) containing the revised body of zg_glob3.
DESCRIPTION: This step tests the functionality of the PORTABLE version of the mathematics package.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: This step is optional, depending on the versions of "zm_math" used in Step 5. If the PORTABLE option was chosen for either the host or the target, this step should be performed to verify that the package "zm_depen" works properly on that system. If the user is already satisfied that the package works properly, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_dptst.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp07 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp07" command script and read the results to a log file.
RESULTS EXPECTED: Log file.
FOLLOW-UP NEEDED: Examine the output. Discrepancies are flagged with the string "<<<ERROR>>>". See User's Guide Section 5.1.6.2 for a discussion of errors output by the "zp_dptst" file. If it appears that too many errors occur, then re-do Step 5 with a different math option.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp07 (command script)
DESCRIPTION: This step tests the accuracy of the mathematical functions provided by the single precision math package.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied with the accuracy of the single precision math function, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_mtasn.ada zp_mtexp.ada zp_mtsin.ada
zp_mtatn.ada zp_mtln.ada zp_mtsqr.ada
zp_mtchn.ada zp_mtpwr.ada zp_mttan.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp08 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp08" command script and send the results to a log file.
RESULTS EXPECTED: Log file.
FOLLOW-UP NEEDED: Verify that the accuracy is sufficient. Errors are indicated in the log file with the string ">>> error". If there are several errors, it may be necessary to return to Step 5 and choose one of the other mathematics package options.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp08 (command script)
zp_stp08 (log file)
DESCRIPTION: This step tests the accuracy of the mathematical functions provided by the double precision math package.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied with the double precision math function, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_dmchn.ada zp_dmexp.ada zp_dmsin.ada
zp_dmasn.ada zp_dmln.ada zp_dmsqr.ada
zp_dmatn.ada zp_dmpwr.ada zp_dmtan.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp09 (command script)
PREPARATION: None.
EXECUTION: Execute the "zp_stp09" command script and send the results to a log file.
RESULTS EXPECTED: Log file.
FOLLOW-UP NEEDED: Verify that the accuracy is sufficient. Errors are indicated in the log file with the string ">>> error". If there are several errors, it may be necessary to return to Step 5 and choose one of the other mathematics package options.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp09 (command script)
zp_stp09 (log file)
DESCRIPTION: This step tests the executable that is responsible for including timing loop code into the distributed performance test sources.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: If the user has previously verified that the Include preprocessor is working correctly, then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
zp_intst.inc
One version of each of the following:
zg_stop2.### zp_intst.###
The suffix for these files is either ".ell", ".elg", ".cpl", or ".cpg".
FILES NEEDED IN WORKING DIRECTORY: zp_stp10 (command script)
PREPARATION: Determine that the correct files, appropriate for the options selected in Setup and described in Sections 3.1.1 and 3.2.5 (step 5), were copied into source.
EXECUTION: Execute the "zp_stp10" command script.
RESULTS EXPECTED:
- The working directory should contain the following files:
zp_intst (Ada source) zp_intst (".ell", ".elg", ".cpl", or ".cpg")
FOLLOW-UP NEEDED:
- Compare these two files, with an automated differencing tool or manually. There should be no differences.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_intst (Ada source) zp_stp10 (command script)
zp_intst (".ell", ".elg", ".cpl", or ".cpg") zp_stp10 (log file)
DESCRIPTION: This step compiles and tests the Harness program. The Harness program is used to create command scripts for groups of performance tests and to track their execution status. The command script file generated in this step copies the necessary files and builds the executable Harness. The user then runs Harness to create a command script for part of one performance test subgroup. Finally, the command script created by Harness runs the tests in the subgroup.
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: This step MUST be completed before performance test is begun.
FILES NEEDED IN SOURCE DIRECTORY:
zh_*.lib zh_*.txt (Ada 83 Implementation)
zh_*.tpl zh_*.a95 (Ada 95 Implementation)
zh_hrn*.ada zh_prfmi.txt (Ada 83 Implementation)
cl_so*.* zh_prfmi.a95 (Ada 95 Implementation)
zh_cosys.txt
One version of each of the following files:
zg_start.### zg_stop0.### zg_stop2.### zg_verfy.###
The extension of these files is either ".ell", ".elg", ".cpl", or ".cpg".
FILES NEEDED IN WORKING DIRECTORY:
zp_stp11 (command script)
PREPARATION: Determine that the files, appropriate for the options selected in Setup and described in Sections 3.1.1 and 3.2.5 (step 5), were copied into source.
EXECUTION: Execute the "zp_stp11" command script.
RESULTS EXPECTED:
The following files should have been generated in the working directory in all cases:
One version of each of the following files:
zg_start.### zg_stop0.### zg_stop2.### zg_verfy.###
The extension of these files is either ".ell", ".elg", ".cpl", ".cpg" as appropriate for the options selected in Setup and described in Sections 3.1.1 and 3.2.5 (step 5).
zh_ap.lib zh_in.lib zh_st.txt
zh_ap.txt zh_in.txt zh_su.lib
zh_ar.lib zh_io.lib zh_su.txt
zh_ar.txt zh_io.txt zh_tk.lib
zh_cl.lib zh_ms.lib zh_tk.txt
zh_cl.txt zh_ms.txt zh_ud.lib
zh_do.lib zh_op.lib zh_ud.txt
zh_do.txt zh_op.txt zh_xh.lib
zh_dr.lib zh_po.lib zh_xh.txt
zh_dr.txt zh_po.txt zh_sy.tpl
zh_dt.lib zh_prfmi.txt zh_sy.txt
zh_dt.txt zh_sr.lib zh_sy000.tpl
zh_gn.lib zh_sr.txt zh_sy_cu.tpl
zh_gn.txt zh_st.lib harness (executable)
If you are running an Ada 95 compiler, the following files should have been generated in the working directory:
zh_oo.lib zh_oo.txt zh_pt.lib zh_pt.txt
FOLLOW-UP NEEDED:
(a) Run the "harness" executable.
1. Acknowledge any "no existing database" messages by pressing RETURN.
2. At the first screen, select by group, subgroup and tests by entering "1".
3. At the first full screen, select the Classical group (enter "3"). When the screen is refreshed, note the "+" beside group 3.
4. Change to the Subgroups screen by entering "Show Subgroups" (or "ss").
5. Bring up the next page of the Subgroups screen by entering "Next". Continue, if necessary, until the Sort subgroup (12) is listed.
6. Select all tests in the Sort subgroup by entering "12/all" or "12/*". Note that the screen now shows the number of tests chosen is equal to 14.
7. Enter "Build" to build a command script for the selected tests.
8. Enter "6" to toggle the "save/delete library units" ("DELETED" is the preferred mode during this step).
9. Note the name (item 8) of the command script file to be generated.
10. Enter "Do" to actually start the build process.
11. Enter "Quit" (or "q") to exit the harness program. Answer "Yes" to the question regarding the saving of the selection information.
More details on running the harness may be found in Section 4 of this document. Still more details are available in the User's Guide, Section 6.
(b) Run the newly created command script file (see step 9 above), capturing the output in a log file named "zp_cl_so.log".
(c) Verify the following:
The log file should show the results of compiling the cl_so* tests, linking them into 5 main programs, and executing the main programs.
There should be time stamps marked with "\aces begin\" and "\aces end\", followed by either "el" or "cp" (elapsed time or CPU time).
There should also be an execution time report for each of the 14 test problems (provided that all of them were successfully compiled and executed).
(d) Run the "harness" executable interactively.
Acknowledge any "no existing database" messages by pressing RETURN.
At the first screen, select by group, subgroup, and test by entering "1".
Enter "Update" to read the log file just generated.
Identify the log file, by entering:
1 = zp_cl_so.log
(If the log file is not in the current directory, include a directory path in the file name.)
Enter "Do" to cause the log file to be read.
(e) Verify the reported status of the tests. Enter the "Show Chosen" ("SC") command and note the status values. There should be a "*1" in each column where a status could be determined for a test. "No data available" entries indicate tests that failed somehow. If all tests report "No data available", then something has probably gone wrong. See the User's Guide, Section 6.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp11 (command script)
zp_stp11 (log file)
Keep the "zp_cl_so.log" log file for use in Step 13.
DESCRIPTION: This step compiles and links the analysis tools (Condense, Comparative Analysis, Single System Analysis, and Menu).
PREREQUISITE: Step 5 must have been completed.
REQUIRED/OPTIONAL: This step may be deferred until the user is ready to analyze performance test results.
FILES NEEDED IN SOURCE DIRECTORY:
za_ca01.ada za_cn06.ada za_co16.ada za_mn04.ada
za_ca02.ada za_cn07.ada za_co17.ada za_mn05.ada
za_ca03.ada za_co01.ada za_co18.ada za_mn06.ada
za_ca04.ada za_co02.ada za_co19.ada za_sa01.ada
za_ca05.ada za_co03.ada za_co20.ada za_sa02.ada
za_ca06.ada za_co04.ada za_co21.ada za_sa03.ada
za_ca07.ada za_co05.ada za_co22.ada za_sa04.ada
za_ca08.ada za_co06.ada za_co23.ada za_sa05.ada
za_ca09.ada za_co07.ada za_co24.ada za_sa06.ada
za_ca10.ada za_co08.ada za_co25.ada za_sa07.ada
za_ca11.ada za_co09.ada za_co26.ada za_sa08.ada
za_ca12.ada za_co10.ada za_co27.ada za_sa09.ada
za_ca13.ada za_co11.ada za_co28.ada za_sa10.ada
za_cn01.ada za_co12.ada za_mn01.ada za_sa11.ada
za_cn02.ada za_co13.ada za_mn02.ada za_sa12.ada
za_cn03.ada za_co14.ada za_mn03.ada za_sa13.ada
za_cn04.ada za_co15.ada
za_cn05.ada
FILES NEEDED IN WORKING DIRECTORY:
zp_stp12 (command script)
PREPARATION:
There is an assumption that the host compilation system has sufficient capacity to compile and link all the units, an assumption that is known to fail for some systems. If it is not possible to do this, there are a couple of options. First, the analysis tools do not have to run on either the host or the target. They can be used on any Ada implementation that will support them, provided a mechanism exists for transferring the performance test log files to that implementation. Second, the analysis tools may be compiled and linked separately. See Section 9.1.1 of the User's Guide for details.
EXECUTION:
Execute the "zp_stp12" command script.
RESULTS EXPECTED: Menu executable.
FOLLOW-UP NEEDED:
Verify that the "menu" executable was created. If not, then the implementation may not have the capacity to compile and link all the analysis tools into one executable. In such case, see Section 9.1.1 of the User's Guide.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp12 (command script)
zp_stp12 (log file)
DESCRIPTION: This step tests the Condense tool that was built in Step 12.
PREREQUISITE: Steps 11 and 12 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied that Condense works properly (or if analysis is not being performed on this system), then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
za_cosys.txt za_cowgt.txt zp_sys_1.e00
FILES NEEDED IN WORKING DIRECTORY:
zp_cl_so.log (created in Step 11)
zp_stp13 (command script)
PREPARATION: Verify that the proper Log file exists.
EXECUTION:
Execute the "zp_stp13" command script.
RESULTS EXPECTED:
The following files should have been copied into the working directory:
za_cosys.txt za_cowgt.txt zp_sys_1.e00
FOLLOW-UP NEEDED:
(a) Run the "menu" executable interactively.
1. On the MAIN MENU screen, enter "1,4,Next" to select the Condense Option, interactively select reports, and go to the next screen.
2. On the SYSTEMS MENU screen, enter "1,Next" to select the first system (System 1), and go to the next screen.
3. On the CONDENSE: REPORT OPTIONS screen, enter "2,Next" to select the EXCEPTIONAL_DATA_REPORT, and go to the next screen.
4. On the RUN or SAVE REQUEST screen, check that the Current Selections are :
CONDENSE
System 1
EXCEPTIONAL_DATA_REPORT
Produce textual data
Append log to existing cmp database
Append log to existing exe database
5. Enter "1,Do" to start Condense.
6. Enter "Quit" to exit the "menu" executable.
(b) Verify that two "database" files ("zp_cl_so.c00" and "zp_cl_so.e00") were created. Compare the format of "zp_cl_so.e00" to that of "zp_sys_1.e00" to ensure that Condense formatted the data correctly. Note that the data will be different.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
zp_stp13 (command script)
zp_stp13 (log file)
DESCRIPTION: This step tests the Comparative Analysis tool that was built in Step 12. It uses sample database files that are included in the ACES distribution.
PREREQUISITE: Steps 12 and 13 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied that Comparative Analysis works properly (or if analysis is not being performed on this system), then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
za_caari.tim za_cacla.tim zp_sys_2.e00 zp_sys_3.e00
FILES NEEDED IN WORKING DIRECTORY: zp_stp14 (command script)
The following files should have been created in Step 13.
za_cosys.txt za_cowgt.txt zp_sys_1.e00
PREPARATION: Verify that the proper files exist.
EXECUTION: Execute the "zp_stp14" command script.
RESULTS EXPECTED:
The following files should be in the working directory:
za_caari.tim zp_sys_1.e00
za_cacla.tim zp_sys_2.e00
za_cosys.txt zp_sys_3.e00
za_cowgt.txt
FOLLOW-UP NEEDED:
(a) Run the "menu" executable interactively.
1. At the MAIN MENU screen, enter "2,4,Next" to go to select the Comparative Analysis Option, interactively select reports, and go to the next screen.
2. At the SYSTEMS MENU screen, enter "2,3,4,Next" (or "2..4,Next") to select systems "sys_1," "sys_2," and "sys_3", and go to the next screen.
3. At the METRICS MENU screen, enter "1,Next" to select the Execution Time results, and go to the next screen.
4. At the GROUPS MENU - PAGE ONE OF TWO screen, enter "2,3,Next" to select the Arithmetic and Classical groups, and go to the next screen.
5. At the GROUPS MENU - PAGE TWO OF TWO screen, enter "Next" to go to the next screen.
6. At the COMPARATIVE ANALYSIS (CA): REPORTS screen, enter "1,Next" to select Group Level Reports, and go to the next screen.
7. At the CA: GROUP-LEVEL OPTIONS - FOR REPORTS OF SELECTED METRICS screen, enter "Next" to acknowledge the default selection, and go to the next screen.
8. At the eighth screen, verify that the selections are correct:
PROGRAM : COMPARATIVE ANALYSIS
SYSTEMS : sys_1, sys_2, sys_3
METRICS : EXECUTION_TIME
GROUPS : ARITHMETIC, CLASSICAL
OPTIONS : Text Reports
Group-Level Report : full
Output line length : 80
To correct any mistakes, use the "Previous" option to move to the appropriate screen where corrections can be made.
9. Enter "1,Do" to cause the analysis to be performed immediately.
10. Enter "Quit" to exit the program.
(b) Verify that the following files have been created in the working directory:
arithm00.tim classi00.tim za_cadb.txt
(c) Verify that the data in "arithm00.tim" is essentially the same as that in the file "za_caari.tim" and that the data in "classi00.tim" is essentially the same as that in "za_cacla.tim". Expect differences in dates, and there may be rounding differences in other values, but verify that the numeric values are essentially the same.
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
arithm00.tim za_caari.tim zp_stp14 (command script) zp_sys_1.e00
classi00.tim za_cacla.tim zp_stp14 (log file) zp_sys_2.e00
za_cadb.txt zp_sys_3.e00
DESCRIPTION: This step tests the Single System Analysis tool that was built in Step 12. This test uses sample data provided as part of the ACES distribution.
PREREQUISITE: Steps 12 and 13 must have been completed.
REQUIRED/OPTIONAL: If the user is already satisfied that Single System Analysis works properly (or if analysis is not being performed on this system), then this step may be omitted.
FILES NEEDED IN SOURCE DIRECTORY:
za_salft.ssa za_saopt.ssa za_sasty.ssa za_sarts.ssa
za_smple.e00 za_smple.c00 zp_salft.ssa zp_smple.e00
zp_smple.toc zp_smple.rep
FILES NEEDED IN WORKING DIRECTORY:
za_cosys.txt za_cowgt.txt
PREPARATION: Verify that the proper files exist.
EXECUTION: Execute the "zp_stp15" command script.
RESULTS EXPECTED:
The following files have been created in the working directory:
za_salft.ssa za_smple.e00 zp_smple.toc zp_smple.rep
FOLLOW-UP NEEDED:
(a) Run the "menu" executable.
1. On the MAIN MENU screen, enter "3,4,Next" to select the Single System Analysis option, interactively select reports, and go to the next screen.
2. On the SYSTEM MENU screen, enter "5,Next" to select the Sample system, and go to the next screen.
3. On the SINGLE SYSTEM ANALYSIS (SSA): HIGH LEVEL SUMMARY REPORT OPTIONS screen, enter "Next" to ignore High Level Summary, and go to the next screen.
4. On the SINGLE SYSTEM ANALYSIS (SSA): MAIN REPORT OPTIONS screen, enter "1,3,10,Next" to select Main Report, Language Feature Overhead, and Write Statistical Tables, and go to the next screen.
5. On the RUN OR SAVE REQUEST screen, verify that the selections are correct:
PROGRAM: SINGLE_SYSTEM_ANALYSIS
SYSTEMS: Sample
OPTIONS: Main Report:
Sections: LANGUAGEFEATURES
Options: STATISTICALTABLES
To correct any mistakes, use the "Previous" option to move to the appropriate screen where corrections can be made.
6. Enter "1,Do" to cause the analysis to be performed immediately.
7. Enter "Quit" to exit the program.
(b) Verify that the following sets of files are identical. Expect differences in dates.
sample.toc and zp_smple.toc
sample.rep and zp_smple.rep
FILES NO LONGER NEEDED IN WORKING DIRECTORY:
sample.rep za_salft.ssa zp_smple.rep zp_stp15 (command file)
sample.toc za_smple.e00 zp_smple.toc zp_stp15 (log file)
This section assumes that the Pretest activity (Section 3) has been successfully executed.
Performance testing is supported by the Harness program that was created during the Pretest activity. If the implementation does not support the Harness, then command scripts (distributed with the ACES) will need to be adapted for this implementation. These command scripts have names of the form "##.unx" and "##.com", where "##" is the two-letter group abbreviation. Thus "ap.unx" and "ap.com" are sample command scripts for compiling and executing the Applications (ap) group. The ".unx" versions were built for use on UNIX systems (Sun Ada (VADS)). The ".com" versions were built for use on VMS systems (A VAX-Ada implementation was used to test these scripts).
For further details on running the performance tests without the Harness, see the User's Guide, Section 7.
The general pattern for running performance tests is described in the algorithm below. The steps marked (ZP_SETUP) have already been done by the ZP_SETUP program; the steps marked (Harness) are most easily executed using the HARNESS program.
Create low-level command scripts (ZP_SETUP)
Adapt Harness control files (ZP_SETUP)
REPEAT
Select a set of tests to be run
Copy the necessary test files into the working directory (Manual)
(These files are the files "##*.*", where "##" is
the two-letter group abbreviation)
Generate a command script for these tests (HARNESS)
Execute the command script and capture the results in
a log file
Examine the log file (HARNESS)
Check the status of the tests (HARNESS)
(Decide which ones to re-run)
UNTIL Finished OR Available time expires
The following paragraphs explain how to prepare for using the Harness, how to select tests, and how to generate command scripts for compiling and executing the selected tests. More details on using the Harness appear in Section 6 of the User's Guide.
The files for a selected test group need to be in your source subdirectory. Harness will automatically copy them into your working directory while changing their (".ada") extension to the system default extension you selected in Setup.
The Harness program should have already been compiled, linked, and tested (Pretest Step 11). As part of this process, a number of data files should have been generated or copied from the source directory, as discussed below:
1. Files with names of the form "zh_??.lib", where the "??" is replaced by the two letter abbreviation of each test group except for the Systematic Compile Speed (sy) group. The file "zh_??.lib" is used when a command script being generated for group "??" includes the deletion of library units. There are 17 for version 83 and 20 for version 95.
2. Files with names of the form "zh_??.txt", where the "??" is replaced by the two letter abbreviation of each test group except for the Systematic Compile Speed (sy) group. There are 17 for version 83 and 20 for version 95.
3. Two files, "zh_sy000.tpl" and "zh_sy_cu.tpl", are used to define processing of the Systematic Compile Speed group.
4. Two files are generated by ZP_SETUP during Pretest Step 11, named "zh_cosys.txt" and "zh_usr.txt". These files may be edited, if necessary, to change the default file names and sizes (in "zh_cosys.txt") or primitive symbols and low-level commands (in "zh_usr.txt").
If the implementation requires more than one command line for a process that is only allocated a single command in "zh_usr.txt", then create a command script containing the needed lines. In place of the single command line allowed in "zh_usr.txt", use the command to invoke this command script. For example, if using a cross-compiler under VMS, then running an executable might involve several steps: download the executable, start the target, and retrieve the results. Place these three commands in a command script (say "runtarg.com") and use "@runtarg.com" as the value for "Run_Command" in "zh_usr.txt". (Note the inclusion of '@', the VMS symbol for invoking a command script.)
Sets of tests are selected for two primary purposes: to generate command scripts for compiling, linking, and executing the tests or to view the status of tests that have already been attempted.
Among the commands that are accepted by the Harness program are the "Show Group" (SG), "Show Subgroup" (SS), "Show Tests" (ST) and "Show Chosen" (SC) commands. The screen that is displayed at the beginning of execution lists the groups (use the command "Next" (N) to view the next page of the list). This is the same screen that appears in response to the SG command. A group must first be selected in order to use the SS and ST commands:
Select a Group
While on the Groups screen, enter a group number to select a group. Precede the group number with a ('-') to deselect a group. The most recently selected group is the current group.
When on the Groups, Subgroups, or Tests screen, tests may be selected for viewing status or for building command scripts. The command for selecting tests depends on the screen. In each case, the actual tests may be indicated by a wild-card ("all" or "*"), by a range (e.g., "42..65"), or by a list (e.g., "42, 43, 44"). The following example assumes that Group 4 is the current group.
Select Tests
While on the Groups screen, enter "4/*/*" to select all tests in Group 4.
While on the Subgroups screen, enter "*/*" to select all tests in Group 4.
While on the Tests screen, enter "*" to select all tests in Group 4.
Tests may be deselected by using the minus ('-') as a prefix to the test number.
Any combination of tests may be selected, including sets of tests that cross group boundaries. However, if any test in a subgroup is selected and not all the tests in that subgroup are selected, then a command script that is built for that selection will not include any commands for collecting compilation/linking time data. This is especially important when re-running tests that did not give usable data on previous attempts.
In the rest of Section 4, it is assumed that the complete subgroups have been selected.
If a command script has already been executed and the status of the tests is being viewed (as discussed below), then the user may wish to select some of the tests, generate a new command script for just those, and re-run them. Again, Section 4 assumes that the user always selects a complete subgroup.
After a set of tests has been selected, the command "Build" brings up the "Building Command Files" screen. There are up to 9 fields that may be changed, as follows:
1. Entering "1" will toggle a "switch" that determines whether the generated script will contain commands that collect compilation and linking time.
2. Entering "2=<number>" will change the maximum number of test problems to be bound into a single executable. The default maximum is 9; if any other value is specified, the default grouping of test problems into executables is overridden, and compilation/linking time measurements will not be collected.
3. Entering "3" will toggle a "switch" that determines whether the script will contain commands to delete all source code files from the working directory after processing. This includes the files produced by the "Include" preprocessor, the subgroup "dummy" files generated by the Harness, and the support package files. (See Section 4.2 (2).)
4. Entering "4" will toggle a "switch" that determines whether the script will contain commands to delete all the "Include" files from the working directory after the "Include" preprocessor has been run on them. (See Section 4.2 (1).) This includes the unprocessed forms of the source files and the main program unit files generated by the harness.
5. Entering "5" will toggle a "switch" that determines whether the script will contain commands to delete the executables from the working directory after they are run. These executables are created when the command script is executed.
6. Entering "6" will toggle a "switch" that determines whether the script will contain commands to delete the library units from the Ada program library after the linking is completed.
7. This field appears only if tests from more than one group have been selected. Entering "7" will toggle a "switch" that determines whether one script will be generated to process all the tests, or a separate script will be generated to process the tests in each group.
8. If the user requests that more than one script file be generated, then entering "8=<string>" sets the file suffix for each script file to the value of <string>. In this case, the file name prefix for each script file will be the two-letter abbreviation of the group.
If a single script file is being generated, then entering "8=<string>" sets the entire file name (including suffix) to the value of <string>.
9. Entering "9=<directory path>" causes the command script(s) to be generated in the directory indicated by the value of <directory path>.
Enter "Do" to invoke the command script generator. When the generation is complete, one or more messages will be displayed concerning the database files for all selected groups, and the "Building command files" screen will reappear. The user may enter "Cancel" to return to the previous screen or "Quit" to exit the HARNESS program.
Once the appropriate command scripts have been generated, the user must execute the command scripts, capturing output in a log file for later analysis. Recall that the source files for the group should have been copied into the source directory BEFORE executing the command scripts. Certain tests require special treatment, as discussed below.
Some of the performance tests must not be run in background mode or batch mode, but must be run interactively from the keyboard. If a windowing terminal with cut-and-paste capabilities is available, the time and code measurement data can be cut from the output window and pasted into a "log" file window. If such a facility is not available, it will be necessary to copy the time and code measurement data as it is displayed on the screen. Note that it is not wise to suspend processing while copying the data for one test, since this may interfere with the timing of a later test in the same main program. For fast systems, it may be necessary to specify one test per main program when building the testing scripts. (Of course, compilation and linking time cannot be captured in this case.)
Table 4-1 lists the interactive tests, along with the files in which the test source code is contained and the main program (using the default assignment of tests to main programs) containing the executable code.
Table 4-1 Tests That Must Be Run Interactively
Test Name Source File Name Default Main Program Name
io_tx_async_01 io_tx01_.inc io_txm01
io_tx_async_02 io_tx02_.inc "
io_tx_cio_01 io_tx03_.inc io_txm02
io_tx_cio_02 io_tx04_.inc "
io_tx_cio_03 io_tx05_.inc "
io_tx_cio_04 io_tx06_.inc "
io_tx_cio_05 io_tx07_.inc "
io_tx_cio_06 io_tx08_.inc "
io_tx_cio_07 io_tx09_.inc "
io_tx_cio_08 io_tx10_.inc io_txm03
io_tx_cio_09 io_tx11_.inc "
io_tx_cio_10 io_tx12_.inc "
io_tx_cio_11 io_tx13_.inc "
io_tx_cio_12 io_tx14_.inc "
io_tx_cio_13 io_tx15_.inc "
io_tx_cio_14 io_tx16_.inc "
io_tx_io_24 io_tx33_.inc io_txm07
io_tx_io_25 io_tx34_.inc "
io_tx_io_26 io_tx35_.inc "
io_tx_io_27 io_tx36_.inc "
io_tx_io_28 io_tx37_.inc "
io_tx_io_29 io_tx38_.inc "
io_tx_io_30 io_tx39_.inc "
tk_la_async_04 tk_la02_.inc tk_lam02
The Input-Output ("io") tests measure the time required to write to the console (terminal). If this timing is important to the purpose of the evaluation, then these tests really should be executed from a terminal that is connected in the exact fashion that would be used for the project.
The Tasking ("tk") test examines the behavior of a program while one task waits for (asynchronous) input from the keyboard. If the test does not terminate within 10 minutes, the user is expected to enter the character 'X' to satisfy the waiting task and allow program termination. (This is a way to account for those systems in which the entire program is suspended while a task waits for keyboard input.)
The test "ms_il_interface_lang_assembly" requires that an assembly language module be provided. The test contains a PRAGMA INTERFACE that is expected to enable calling the assembly language procedure. The source code may have to be modified if the implementation requires additional information for a subprogram imported by PRAGMA INTERFACE, or if the name "assembler" is not the appropriate name for interfacing to assembly language. The user must examine the source code file ("ms_il01_.inc") to determine whether the pragma meets the requirements of the implementation. In addition, the command script controlling the compilation of this test may have to be modified to assemble the required module and make it known to the Ada library. Locate the string "User adaptation required" and modify the next two commands. The first command means to assembly the assembly language module in the file "ms_ilnul" (with whatever file-naming conventions are required.) The second creates an object library named "ms_illib", containing the newly assembled object file. For some implementations, the appropriate command would add the object file to the Ada library as a "foreign" unit. Note that the string "User adaptation required" appears again, followed by a link command that references the object library created above. The implementation under test may require quite a different syntax, or may not require a special link command.
There are seven tests that contain deliberately illegal symbols. In each case, the line containing the illegal symbol must be modified by the user. These tests, their source files, and their default main program names are given in Table 4-2.
Table 4-2 Tests Containing Illegal Symbols
Test Name Source File Name Default Main Program Name
dt_dp_delay_zero_01 dt_dp16_.inc dt_dpm04
dt_dp_delay_zero_02 dt_dp17_.inc "
dt_dp_delay_zero_03 dt_dp18_.inc "
dt_dp_delay_zero_04 dt_dp19_.inc dt_dpm05
dt_dp_delay_zero_05 dt_dp20_.inc "
dt_dp_delay_zero_06 dt_dp21_.inc "
tk_lf_task_48 tk_lf20_.inc tk_lfm20
In the Delay and Timing ("dt") tests, the symbol "<>" appears in an illegal context, requiring that the user modify the source file. The test's purpose is to examine the effect of time-sliced task scheduling. If the implementation supports a pragma for this purpose (such as the supplied example using PRAGMA TIME_SLICE), then the pragma should be used in place of the line containing the illegal symbol. Some implementations may achieve the same effect with compiler or linker options; in this case, the user should delete (or comment) the line containing the illegal symbol and also must modify the command script controlling the compilation, linking and execution of the test. The source code contains comments describing the size of the time-slice quantum that should be used. Note that in the last three of these tests, the specified time-slice quantum should be 0.0.
In the Tasking ("tk") test, the symbol "<>" is used illegally in two lines. In each case, the line must be replaced by an implementation-dependent method of associating a task entry with an interrupt. An example is included in the illegal line, using an address clause to associate the entry with the interrupt that is associated with the hexadecimal address 40. The actual address is highly implementation-dependent. Some implementations may require one or more pragmas in this context. Some implementations may simply not permit such an association.
The Systematic Compile Speed ("sy") tests pose several difficulties. First, it is not possible to generate a command script for just selected tests. The Harness program always generates three scripts ("sy<scripts>", "sy_cu<script>", and "sy_1000<script>", where <script> is the script file suffix specified by the user). The first of these compiles and executes all the "sy" tests except for the Compilation Unit Size subgroup. Responsibility for the tests in the omitted subgroup is divided between the command scripts "sy_cu<script>" and "sy_1000<script>". This division of control of the "sy" tests into three scripts is done because these tests may require more time and resources than the evaluator can afford.
Note that these tests have been observed to monopolize the CPU on multiuser systems, to consume all free disk space, and to crash some operating systems.
The second problem is that the command scripts, even those generated by the Harness program, require extensive user modification. In each of the scripts, modifications are indicated by comments containing the string "~~~~" (four occurrences of the "tilde" character - ASCII code 126 decimal). The adaptations are described in comments appearing before the modification flag. Adaptations that must be made include, but may not be limited to, the following:
* Compiler option to include the source code in the library
* Compiler option to generate warnings
* Method of submitting a list of files to the compiler at once
* Compiler option to generate a listing file
* Compiler option to check syntax only
* Compiler option to check syntax and semantics only
* Compiler option to suppress warnings
* Recompile command
* Recompile option to force closure
* Method of deleting deeply nested subunits
(where the expanded name is longer than a script line)
* Method of determining the total size of the program library
* Method of determining the size of selected library units
(with or without wild-card symbols)
* Method of obtaining a list of library units
* Method of determining whether a specific unit is in library
* Method of deleting library units using wild-card symbols
* Method of creating an Ada library
* Method of deleting an Ada library
* Method of deleting all but most recent version of a file
* Method of assigning a character value to a script variable
* Method of opening a file so a script can read from it
* Method of reading a value from a file into a script variable
* Method of writing the value of a script variable to a file
* Method of closing a file that was opened by a script
* Method of assigning the value of one script variable to another
* Method of comparing a script variable's value to a literal
(using an IF .. THEN .. ENDIF kind of structure)
* Method of checking the status of the previous command
(using an IF .. THEN .. ELSE .. ENDIF kind of structure)
In this section, it is assumed that the command scripts have been executed and that the test output has been captured in a "log" file.
To extract status data from a log file, enter "Update" (or "U") while on any Harness screen.
(Note that this command is not displayed in the list of available commands on a screen.)
On the next screen, specify the name of the log file by typing "1 = <name of log file>". Then enter "Do" to cause the status data to be extracted from the file.
After updating the status from the log file, the Groups screen (the opening screen, reached from other levels by the Show Groups command) will show the number of tests in each group having each status value. Use the Show Subgroups and Show Tests commands to determine which specific tests have particular status values. The following list gives the possible status values and their meanings.
Va Valid non-zero execution time (the desired result)
Nu (Null) Execution times of 0.0 (leave alone)
Dy Measures time for delay statement (leave alone)
Un (Unreliable) Timing measurement was variable (try again)
Vr Verification error (try again)
Ng Timing ERROR: large negative time (probably should give up)
Xc (Excessive time) Storage reclamation takes too long (leave alone)
Pk Packaging error - might be ok (run separately from default main)
Cm Compile time error (log file might show a solvable problem)
Ln Link time error (log file might show a solvable problem)
Rn Run time error (test problem did not complete executi