![]() |
Test & Analysis Project
|
The Test & Analysis project will provide tools for statistical testing of Geant4 physics and related quantities, both at unit and system level. It addresses primarily the areas of physics validation and regression testing, by means of comparison between simulation results and reference data.
The problem of | comparing Geant4 physics results with reference data or analytical functions |
affects | the Geant4 (electromagnetic) physics Working Groups |
the impact of which is | large amount of time spent in series of tedious, repetitive comparisons performed manually, or limited test performed, because of limited availability of resources |
A successful solution would be | being able to perform such testing by means of general tools for statistical comparisons |
The problem of | regression testing |
affects | the STT and the Geant4 (electromagnetic) physics coordinators |
the impact of which is | a significant workload to perform it manually, or a limitation of the test to very simple comparisons, such as the ASCII dump of the information of each event |
A successful solution would be | the availability of tools for automated statistical comparisons |
The Test&Analysis tool adopts a solid architectural approach and a rigorous software process:
Hacking some code quickly would hardly guarantee the quality, maintainability and extensibility needed, and in the end would result in a larger investment of resources.
To effectively provide a product that meet users' real needs, all stakeholders are invited to contribute to the Requirements Modeling process. The following sections highlight the key problems to be addressed by the proposed solution and the people involved; specific and detailed requirements are captured in the URD.
Name |
Description |
Responsibilities |
Geant4 STT Coordinator | Coordinates system testing | Ensures that the system meets the needs of Geant4 System Testing |
Geant4 physics coordinators | Coordinate Geant4 WGs | Ensure that the system meets the needs of Geant4 Physics WGs |
Geant4 TSB | Is responsible for Geant4 technical matters | Reviews the project, monitors progress |
Geant4-INFN Computing Committee | National Committee whom part of the project developers responds to; has 3 referees assigned to Geant4 | Recommends funding; reviews the project and monitors progress |
Student | Technical student at CERN and student at the Univ. of Genova | Is the main developer, should do work of value for her university degrees |
Name |
Responsibilities |
Physics developers | Provide and document requirements, provide feedback on prototypes, perform beta testing on preliminary releases of the product, provide use cases for acceptance testing |
STT | Provide and document requirements, perform formal acceptance testing for adoption in system testing |
Users of the product are expected to be:
Geant4 developers, as part of the testing process in the development and maintenance of the Geant4 Toolkit;
Coordinators of Geant4 Working Groups (especially in the physics domain), as part of the testing process at the package or subsystem level, especially in the context of integration;
Geant4 System Testing Team, as part of the system testing process.
In addition, other potential users may be:
users of the Geant4 Toolkit, wishing to verify the results of their applications with respect to reference data or their own experimental results;
other projects with similar requirements, or using the Geant4 Toolkit (such as, for instance, the LHC Computing Grid project).
The tests related to physics groups are usually performed on one platform by developers, and on two platforms at least by WG coordinators.
The STT does regression testing as part of system testing, in a fully automated process. They run the tests on all Geant4 supported platforms.
Need | Priority | Concerns | Current Solution |
Proposed Solutions |
Tool for regression testing |
? |
Degraded behaviour may slip into a Geant4 release | Done manually on limited set of observables (physics), or automatically on reference ASCII dump | Automated tool for regression testing |
Tool for comparison with reference data |
? |
Either the individual physics packages, or Geant4 as a whole, may not be subject to verification against reference data as part of the release testing. | Done manually, not all the test comparisons available are run regularly | General tool for physics validation |
Possible alternatives some of the stakeholders may consider are:
hacking code for ad-hoc solutions
a project limited to statistical regression testing for the STT, not encompassing the testing requirements specific to physics Working Groups
The first option would have the advantage of getting some specific problems solved quickly; however, it could not realistically cover all the requirements, nor be easily maintainable and extensible over a long time scale. The second option would produce some useful results in a critical area; however, by neglecting the commonality of some problems, it would not optimise the exploitation of the resources.
The Test & Analysis tool is expected to be self-contained for usage in physics Working Groups, while it should be integrated in the system testing environment.
The Test&Analysis system depends on AIDA and on the availability of at least one concrete implementation of it.
It assumes that there is a commonality of problems, and the interest to find common solutions, among Geant4 system testing and physics testing and validation.
The Test&Analysis system will offer the capability to compare a set of Geant4 results with reference distributions. Statistical comparisons will be performed, providing an evaluation of compatibility between the two sets compared.
Regression testing, in the form of comparison of equivalent histograms, will be automated.
The software will operate in the same environment as the development of the Geant4 Toolkit. It will be based on AIDA.
No concrete dependencies on specific AIDA implementations should appear in the code of the system tests.
A user-friendly interface should be provided, to allow for customisation, especially by non-expert users.
Date |
Version |
Description |
Author |
15 September 2002 |
1.0 |
Created |
M.G. Pia |
26 September 2002 | 1.1 | Corrections + requirement on independence from specific AIDA implementations by G. Cosmo | Corrections typed by MGP |
Maria Grazia Pia, 19 January 2003