Oasis have been working for a while on an office format named ODF. I became more interested when they started discussing testing, conformance and interop. this is the TC website for the group. This is what they are supposed to be doing. The deliverables seems somewhat short of the scope, but whatever. AFAIK they haven't delivered much so far. Just lots of talking.

I joined in heartily when the list was opened to discuss the formation of the TC. I've a fair background in test (twenty or more years), mostly on software, so I believed I could help. This blog entry is a brief summary of what I see as the shortcomings of ODF as a testable entity. I relate simple 'testing' and conformance, i.e. I test an implementation to assess its conformance to a requirement or specification. I do suffer from a history of developing under a waterfall methodology, so I cleanly relate a requirement to a design and to a test specification. Even post waterfall, I find that a fair way of looking at what Oasis are doing as far as ODF is concerned. My real doubt is that neither the TC nor Oasis think along these lines.

How I see a test architecture and approach

My definitions come from my background in test, working to ISO 9000 for software delivery to the UK MoD in the Aerospace Industry.

A system was specified to meet the customer requirement. A software requirement specification was written to meet that system specification. A design specifiction was written to meet all aspects of the requirement specification. A test specification was written stating how each requirement within the requirement specification (and implicit from regulatory aspects etc) was to be met via the design specifiction and tested via the test implementation. A test plan addressed coverage, ensuring that the full requirement specification was covered by the design specification. Each and every document spent time with our QA department for review and sign off. All testing was supervised by QA to ensure compliance to the test spec and coverage of the requirement and design specs.

Any test then could be simply mapped via the test plan back to the requirement specification. Each test was numbered sequentially for identification and reporting purposes. All tests executed in a test run were logged as part of the test report. If a full suite of tests were run and all tests passed, a pass was reported. Otherwise a list of failures was reported. Test data always included metadata such as the test operator, date/time, software versions etc.

Each test was executed(sequenced) automatically. Any test could be either manual or automatic. If a test was not objective it was not accepted as part of the test suite by the QA assessor. Manual intervention was generally kept to a minimum commensurate with the Unit Under Test parameters. The result of the test was always recorded. Measured tests were limited, generally with reference to the design specification or the requirement specification. The instrumentation used for limited values was traceable back to national standards and its calibration always checked.

Scope

It was always clear that a test specification had scope of the UUT design. With ODF a complexity arises insofar as what is specified sometimes includes just the XML file on disk, at others the behaviour of the implementation, e.g. how an end user might see changes to the presented data. In this manner ODF is weakly specified.

Test results

Having tested an application or implementation, a bean counter is unlikely to be impressed if all he has is the test operators word that everything is hunky-dory. Far more useful is a set of test results with a full suite of metadata about the test environment, the software under test and who witnessed the testing. This way it is not unreasonable to expect sign-off of the software under test as being compliant.

Test Environment

Running tests on a configured machine, of known hardware and software standards, makes tests repeatable. If someone upgrades the OS from XP to Windows 7, who knows what will happen to the tests? If OS based timing is being done then what is the accuracy? To address this, the entire hardware and software environment should be documented and not upgraded without reflecting on the impact on test execution. In a similar manner, if tests are written which depend on the underlying OS, then be aware of the re-write, re-test, validation and QA acceptance that might be implied.

Change Control

With an upgrade from 1.1 to 1.2 of the software under test, new tests will have to be written, changes coded and tests re-run. The traceability of changes is key to allowing tests to be carried forward without complete revalidation. Any software should record the trace back to the change order (or change in requirements) that caused it.

Summary

Test is a dirty word in software estimation. Always has been. Probably always will be. W3C (http://www.w3.org/TR/2010/NOTE-test-methodology-20100128/) are starting to take it seriously, quite possibly due to the wide variety of people who populate the working groups. Oasis still have some catching up to do I think.