ODF 1.1 and 1.2 are not written as system specifications or software requirement specifications, i.e. not written in a manner which naturally leads to validation and hence interoperability. Put simply, very few statements can be used to generate either automated or manual tests of an implementation against the specification, which is how an implementation should be judged, IMHO.

Some time back I spent sixteen years in a software test environment. The organization I worked for was AQAP, then ISO 9000 approved for the supply of software to the UK Ministry of Defence. I took a significant piece of software through from scratch to ISO 9000 approval over a period of six years. I'm a member of British Standards Institute, an ISO 9000 assessor and have a fair bit of knowledge and practical experience with software testing.

Most (quality) specifications provide clear instructions using those magic words SHALL, SHALL NOT, and MAY where those words have a defined meaning for an implementor. Paragraphs are clearly identified as either normative or informative. That way an implementor knows what they must and may implement to claim conformance against a specification. This approach has been well established over time as a sensible way for spec writers and implementors to work

That is the way quality specifications are written. For example, ISO/IEC's JTC 1 Directives (link to PDF) requires that international standards designed for interoperability "specify clearly and unambiguously the conformity requirements that are essential to achieve the interoperability."

With that clarity, conformance is testable and can provide confidence of interoperability. A suite of tests may be developed and applied to an implementation to determine which tests pass, which fail, and hence arrive at an objective pronouncement on conformance of an implementation against the entirety of the specification. An example, RenderX have implemented xsl-fo and publish both a list of implemented properties as well as a list of extensions which go beyond the standard. This kind of compliance table is easy with a well defined standard or specification.

In a quality specification, it should be feasible to select a normative paragraph, identify a conformance test for it, and make a clear statement that this test proves that an implementation meets (or fails to meet) that requirement. Call it a test plan: define the tests (test specification), define the expected set of results, and define what constitutes a "pass" of each test that establishes conformance. The plan then provides the matrix of test spec against requirement. Simple.

For a further example of that kind of test suite for a specification far less complex than ODF, see the W3C's WICD Mobile Test Suite. Note that each test is clearly related to a specific part of the specification, defines the test, and defines what a pass that establishes conformance is.

Of course it can be more complicated. The human-computer interface is an area where there needs generally to be some degree of manual intervention in an otherwise automated test environment. Rick Jelliffe recognises this in an article on ODF conformance testing here. Rick has a habit of asking awkward but generally astute questions on his blog. He nearly always makes me think hard!

Since most users are more interested in the visual presentation than the bits on disk view of an ODF file, the human-computer interface is clearly an aspect that ODF should address in its normative requirements. How? I've only tested for character level presentation, where a few hundreds of pages of text were required to be character perfect. For an office application this would need another layer of complexity. It is still possible though, given a clear specification

Does it matter what is in which menu? What does matter to a user? Functionality? Feature set? Ease of use? There are lots of conceivable dimensions, although I'd hazard a guess that it wouldn't take much to refine them to aspects that a real user would be interested in and to other dimensions that are really in the domain of the implementor such as items that make implementation A "better" or better suited, to user X than to user Y. On the other hand, if I am only using ODF as an interchange format, then I would quite possibly be uninterested in this aspect of testing. Alex has provided such validation as of today. This addresses validation of the files on disk, leaving alone the MMI aspects.

When Oasis announced its intent to review ODF from a conformance standpoint, I was eager to join what later became the ODF Interoperability & Conformance TC, OIC TC for short. So I participated in a mailing list meeting established to define a charter for the proposed TC, believing (rather naively) that its purpose was as stated.

Rob Weir of IBM chaired (apology for the misuse of that last word) the formation list and then simply announced what the charter would be rather than seeking consensus among the list participants. As part of this process before that charter was produced and while I still believed that consensus was a goal, I sat down with ODF 1.1 and did a paragraph-by-paragraph review for testability. The numbers were quite revealing. I completely reviewed only the first four major sections and found very few clear requirements.

The majority were mere statements with no normative language used to identify what was required or optional. Implementors would have to make their own interpretation. A good example of one that is testable (though only to emit a warning) is, in the format I used for recording my review:

 
 
<simplesect xml:id="tests.section3.3.1.1p1s1" xreflabel="3.1.1p1s1">
   <title>3.1.1 para 1 s1</title>          
   <para role='link'> <link xlink:href='&specx;3.1'>spec</link> </para> 
   <para role="spec">The <tag class="element">&lt;meta:generator>
   </tag>element contains a string that identifies the application or
   tool that was used to create or last modify the XML document. This
   string should match the definition for user-agents in the HTTP
   protocol a specified in section 14.43 of [RFC2616]. </para>
   <para role="test">Validate the contents of the <tag
   class="element">&lt;meta:generator> </tag>element against the
   syntax of <link
   xlink:href="http://www.ietf.org/rfc/rfc2616.txt">rfc2616,
   1999</link>.</para>
<literallayout>
   User-Agent     = "User-Agent" ":" 1*( product | comment )
   product         = token ["/" product-version]
   product-version = token
   comment        = "(" *( ctext | quoted-pair | comment ) ")"
       ctext          = &lt;any TEXT excluding "(" and ")">
</literallayout>
   <para role="precondition"><tag
   class="element">&lt;meta:generator></tag> is present. Strict
   version of schema in use.</para>
   <para role="postcondition">  </para>  
   <para role='remark'></para>   
   <o:test class='test' output="warning" type="auto" applicable="document"/>
</simplesect>

That relates to section 3.1.1, first paragraph, first section (even the spec isn't clear on individual requirements which is essential in my opinion).

An example fairly typical in its lack of clarity and not stated as a requirement:

 
 
<simplesect >
  <title>3.1.1 para 1 s2 </title> 
   <para role='link'> 
     <link  xlink:href='&specx;3.'>spec</link> </para> 
   <para role="spec">The generator string should allow product
   versions to differ between all released versions of a user agent,
   for instance by including build ids or patch level
   information.</para>
   <para role="test">-</para>   
   <para role="precondition">  </para>   
   <para role="postcondition">  </para>  
   <para role='remark'>No specification. A should clause. Unable to
   verify any form of compliance</para>
   <o:test class='untestable'/>
</simplesect>
 

Anyone concerned about development expense would simply not implement that. There is no requirement that they do so, only a recommendation.

Having done this work that took hundreds of hours, I basically gave up hope since the chair had by then made it crystal clear that he was not interested in standard-based interoperability and improving the specification so that a test suite could be developed.

The approach is illustrated by two emails. In the first email on 6 July 2008, I reported to the TC formation meeting:

Just ran some rough test counts.


[ODF] 1.1 Has 2578 'testable paragraphs'


Quite a task this TC has on its plate.

In the second email, the chair replied the same day:

This is less work than I expected. I was predicting 10,000 test cases.

It's ironic that the chair viewed as good news the fact that there were far fewer testable paragraphs than he had predicted. But his prediction of 10,000 test cases is probably far closer to how many testable paragraphs there should be; my counts were actually bad news.

I should also mention that "testable" does not equal "required." Options (MAY) can be testable. Paul Merrell developed some interesting statistics by running word counts against the ODF 1.1 specification for requirement keywords indicative of mandatory requirements. The count: only 227 occurrences.

All of the above leads to the interesting question of just how the chair expects to accomplish much that is useful in regard to ODF conformance testing before the specification is amended to tighten up the language and add clear requirements. The syntax conformity is already handled by validation against the schema, but the semantics are woefully under-specified.

But undeterred, the OIC TC Charter that the chair developed behind closed doors announces intent to produce, by 1 May 2009, "A conformity assessment methodology specification, detailing how each provision and recommendation in the ODF standard may be tested for conformance." Of course, recommendations have nothing at all to do with conformance, but what to do when there are so few conformance requirements to work with?

Summary: ODF 1.1 isn't verifiable as a specification. From a fairly cursory review of the latest draft, ODF 1.2 will follow the same path. With OASIS now being more demanding regarding conformance requirements on every specification and with ISO/IEC taking a closer interest in liaison with the ODF TC, I find it hard to see how the ODF TC co-chairs can maintain this view toward verification.

Only today I found this interop testing site for an ODF Plugfest of the kind Rob Weir is pushing. I was going to add a proposal for the testing but then realised that the 'wiki' aspect was untrue, in that a page that reads please add your proposals here. actually was (wiki wise) marked as read only! So travel to the Netherlands if you want to find out how to do ODF interoperability, I guess. The site owner failed to respond when I poitned this out.

This is definitely not how standards are supposed to work.