How to buy a standard in 10 days

by James Hogarth | 02 Apr 2008 | Permalink | Everything Else

Allan recently blogged about ‘Standards or Chaos’ and this seems very appropriate in light of the current tech news.

For those that missed 2007 let me set the stage:

The OASIS foundation (of which Microsoft is/was a part) after a lot of time and effort (many years) had finally released 1.0 of the ODF spec and it had just passed through the regular PAS ISO submission process to become recognised as the official international standard open for anyone to implement for documents (spreadsheets, word processed documents and presentations). It was a time of great joy knowing that documents saved in this format would be usable in any ISO26300 compliant application. Openoffice.org was first out the door (as was to be expected as the format had the original openoffice 1.x spec as it’s startpoint although with input from many organisations it changed a lot from that over the years to accommodate other applications and become as vendor neutral as possible) with Koffice, Lotus Symphony, Gnumeric, Abiword and Google Docs following to name but a few…

Microsoft Office users however were left high and dry for the large part… Microsoft announced they would be sponsoring another company (CleverAge) to create a plugin that will allow import and export of the ODF format from Microsoft Word 2007 (to be fair to them they have since further developed this to include Office 2003 along with Excel and Powerpoint). They, however, claimed that technical issues prevented them from making it part of the usual save/load options or making available as the default save format. There are substantial issues remaining with fidelity of conversion and it is ‘safer’ to just use .doc instead for the large part. Sun released a plugin for Office 2000/2003/2007 with corporate support and the ability to use ODF as an option in the standard load/save dialogs rather than an import/export option…. so far so good? Ultimately the best people to write the load/save filters for MS Office would be Microsoft themselves however they have stated that there is no demand from there customers for this and they have no plans to do so at this time…. the reasoning for this stance became clear soon thereafter….

Microsoft Office 2007 introduces a new file format for its documents. Rather than use the publicly available ODF specification (or even take part in creation of this new format) they elected to XML-ise their binary formats. As Office 2007 was nearing beta (December 2005 to be more precise) the ‘European Computer Manufacturers Association’ (abbreviated ECMA) created a working group (TC45) with the charter:

“The goal of the Technical Committee is to produce a formal standard for office productivity applications within the Ecma International standards process which is fully compatible with the Office Open XML Formats.”

It is worthwhile noting that the two chairmen for TC45 are Microsoft employees and here is a quote from an internal Microsoft document that was made public in the Comes Vs Microsoft anti-trust trial – please note that this is about stacking panels at conferences and not standards bodies. It is an extrapolation to view this in light of the happenings in the national bodies and it is up to the reader to decide – after following through with all accounts – how valid it may be in this case:

“A stacked panel, on the other hand, is like a stacked deck: it is packed with people who, on the face of things, should be neutral, but who are in fact strong supporters of our technology. The key to stacking a panel is being able to choose the moderator. Most conference organizers allow the moderator to select die panel, so if you can pick the moderator, you win.”

IBM objected to forming TC45 on the basis that ODF 1.0 had already been created and submitted to ISO as DIS26300 stating that any work to a standardised document format for office applications should be done on one specification and that rather than a new working group and standard be made it would be better to submit changes to the current OASIS specification. Obviously they were overruled by the other members of ECMA. For the final specification document IBM voted no to making it a standard. The rest (Apple, the British Library, Novell, etc having long standing MS connections or else announcing major MS projects during this time) voted yes to ratification of the draft. To a lack of surprise ECMA-376 was approved.

The specification weighed in at a grand total of 6000 pages or so and did not use any current standards but rather they wrote their own stuff (Office Math ML in place of MathML, VML in place of SVG, their own hasing functions in place os standard SHA-1 etc… the list goes on). In contrast ISO26300 is 722 pages and refers to existing standards where possible. ECMA, OASIS and XML Guild are all category A liaisons with ISO JTC1 and have the right to propose a standard go through the ‘Fast Track’ process for minimal edits and pretty much just a nod to “yup this should be an ISO standard.” Up till now this has been used for specifications already in use by multiple products and is pretty much a recognition of a documented, open de facto standard. Despite MS bloggers insisting so ODF did not take this route – only OOXML has done so as DIS29500.

As a fast track ISO P members of the appropriate committee (SC34 in this case) get only 30 days to review the specification before a 5 month ballot takes place. Despite the herculean effort involved many countries did indeed attempt to review this document within 30 days and came up with numerous questions, comments and objections. The five months ended in September 2007. Microsoft were documented in stuffing as many committees with friendly faces as possible. Leaks came out about Microsoft offering discounts to partners that wrote to their National Body in favour of the standard. Countries that had never participated in the standards process before signed up at the last moment to vote approve. Even with all this happening Microsoft failed to get approval DIS29500 was not approved. Over 1000 technical comments about the specification were raised.

And so we come to 2008:

Following the procedure to deal with this in a Fast Track process a Ballot Resolution Meeting (BRM) was set up for one week in February 2008. By this time ECMA was to have responses to the comments so that the countries interested could have delegations at the meeting to discuss these comments and instruct the editor for DIS29500. That’s right folks just one week to discuss 1000 comments not taking into account that if one comment makes a change it could create or affect another comment… what a mess. Many national bodies complained that this was not feasible and that DIS29500 should be taken off the fast track process and moved to the standard PAS submission process…. haha as if Microsoft would let that happen….

Microsoft had 20 people employed by them at the BRM as delegates for a variety of countries, in the capacity of ‘Head of Delegation’ for two countries, as part of ECMA and as assistant to the project editor. Given that the chairman of ECMA TC45 is an MS employee I can understand him there. A second technical person from Microsoft who worked on the ‘standard’ in TC45 (Brian Jones in this case) I can understand being there to answer specific technical questions arising from the document submitted by TC45. However I cannot even begin to fathom how 18 other Microsoft employees – 2 of them as delegation heads – could attend without a conflict of interest. Out of 32 countries attending MS had one representative in 17 of these. Surely the national bodies involved for these 17 countries could have sent another person in place of a Microsoft employee? Was there no others in those national bodies that approved of DIS29500 to go and discuss their positions on the comments? if not that alone should have told the world something….

At the BRM a grand total of 189 comments were discussed – including quick things like spelling mistakes. This works out to roughly 18.4% of all comments. The third day of the BRM – all accounts of this from all sides agree – it was obvious that there was no way to work through all the comments by Friday – I know the logical amongst you would suggest that day one that should have been obvious if not before. The guy running the BRM (Alex Brown) presented four options to the representatives present for any response to comment from ECMA not discussed:

1 – All rejected.

2 – All approved.

3 – The ITTF decides everything.

4 – Decision by vote in batch, with the possibility of declaration of each individual response vote and / or the definition of a overall vote.

At this point it is worth mentioning that according to reports nothing beyond a spelling or grammar mistake survived as a response from ECMA without additional changes from those present and it would appear that any response accepted would improve the specification by some amount – even if it doesn’t bring it up to the clarity and overall level expected from a ISO standard. Based upon these two things both option 1 and 2 would be foolish. Option 3 means that there would be no point having the expense of the BRM in the first place if a ‘neutral’ third party deals with all the comments/responses since it eliminates the NB representatives from discussing things. This leaves only option 4. The best in a bad bunch.

The result of the BRM was praised loudly by Microsoft and its affiliates as a ‘complete success’ – for those who have played Valve’s game Portal the end credit song springs to mind. According to them 98% of responses were approved – wow isn’t that incredible? Think about this for a moment….. A specification supposedly already standardised by a working group (ECMA TC45) and submitted to a process designed for de facto standards already in use to be quickly granted ISO approval had a minimum of 980 changes made to it – some as minor as a spelling mistake and some substantially more significant – and those only came from a very rushed review period. Imagine the result from a proper review through PAS or other submission procedures at JTC1? Personally I would be ashamed if I had lauded a piece of work as open, complete and finished for all to use and then to have others point out hundreds of mistakes or changes needed…..

To quote the US Head of Delegation:

“Eighty percent of the changes were not discussed, It’s like if you had a massive software project and 80% of it was not run through QA. It’s a big problem, I’ve never seen anything like this, and I’ve been doing this for 25 years.”

Now we are a company that develops a product – as I am sure you are aware – and I am sure our Product Managers would have a heart attack if QA skipped 80% of things in a release…

And so we come (in a roundabout and somewhat abbreviated fashion) to today. Following the BRM ending in February the national bodies of countries worldwide had the option of changing their votes from no to yes on the question of whether DIS29500 was ready to become a recognised ISO standard. Although an edited draft of DIS29500 with the responses from the BRM implemented is not available the ISO members have little choice but to vote on the original draft with what they believe will be written into the documentation by the project editor based upon the list of responses they have. This period for changing their vote ended last Saturday. The changes in detail for those who are interested can be witnessed here:

http://www.openmalaysiablog.com/2008/03/the-last-lap.html

Once again questions of corruption instigated by Microsoft come to light. The short story is that as of this moment in time DIS29500 will be approved as an ISO standard so long as no national bodies object. The long story is that once again Microsoft have been found trying to politically influence a technical process. Many national bodies seem to have voted for this becoming a standard despite their technical committees informing them it is not ready. The prevailing attitude to it seems to be along the lines of “We don’t believe it is ready really but we’ll vote yes and try to fix things in maintenance” or “We don’t believe it is ready but we have received letters from people saying they want it and are unsure what to do so we abstain from voting” – um guys if it is not ready then vote NO and have it fixed through the proper procedures in a submission process that is fit for a specification still needing development. France, Poland, Singapore, Germany, Norway and the Phillipines are just a few where it has come to light that Microsoft business partners have sent template letters through to their national body (even with spelling mistakes such as DIS29600 instead of DIS29500 intact – please if you use a template at least know what you are writing about – or with places holder [ASSOC HERE] items still in place) and cc’d their local Microsoft department or where the technical committee for that national body voted no or were unable to reach any consensus (often near 50/50) and yet a ‘YES’ was provided to ISO.

Given that I am in the UK I am, naturally, interested in the position of the British Standards Institute (BSI). Back in September they voted against DIS29500 becoming a ISO standard. The BSI had managed to do something outstanding back in May last year when they reviewed those 6000+ pages in just 30 days and produced no less than 635 comments – more than any other national body. The nearest was France with 592. Allowing for some duplication it would be fair to say these two countries probably made the bulk of the 1000 unique comments. We’ll come to our country we hold near and dear in a moment for France is an interesting tale first.

AFNOR (the national body for France) announced that France would be maintaining its NO vote to the acceptance of DIS29500. Given the French stance on FOSS in general at the moment this would not seem surprising. When this news leaked it appears that Microsoft France’s President Eric Boustouller sent AFNOR a letter. This letter told a tale about a wonderful future of ODF and OOXML working side by side and how a working group would be set up to make them more interoperable. It also had a ‘statement of support’ by HP attached to it in which the very generic message was:

“HP Position Statement on Standardization of Office Document Formats

Hewlett-Packard Company is in favor of standards for office document formats, including the Office Open XML (OOXML) and Open Document Format (ODF) standards. We believe that the two standards will co-exist interoperably, and that customers should have the opportunity to select the standards which best fit their needs.

HP believes that the international standardization process is working. This process has already significantly improved the OOXML specification and additional evolution of it will take place under control of the global community if the specification is ratified by ISO/IEC JTC 1.

In the current vote on OOXML at JTC 1, HP is supporting an affirmative vote in those national standards bodies in which HP is active.”

As a result of this the French position has been changed to Abstain instead of No since “in light of the new information, it seemed them that disapprove was not justified. But neither was a Yes vote, because there remain issues with OOXML.” If it is not ready guys why not maintain your NO vote? Then it can be passed to the appropriate sub-committee to make it ready as an international standard.

Ah so we come to our fine BSI. An organisation that many consider to be a reliable source of standards and the BSI mark is an industry – globally even – recognised mark of standard compliance and excellence. As mentioned the BSI voted no in September. The BSI has voted for DIS29500 with no comments about how the internal vote went or who comprised the committee for the vote. When questioned they only have this to say:

“It is not BSI’s policy to make public the details of how it votes on International Standards. BSI sent a delegation to the ballot resolution meeting which took place in Geneva in February 2008 and subsequently requested that its technical committee carry out a review of its position on ISO/IEC DIS 29500.”

Given that the British comments were too numerous to all be resolved at the BRM to a suitable standard it is curious how they come to the conclusion that DIS29500 should be approved. Mr. John Pugh, MP has tabled the follwoing question in parliament:

“Can the Minister reassure me that the British Standards Institute that operates as our National Body under a memorandum of understanding from the Government will not change its stance on the currently unavailable text for the proposed standardisation of OOXML without proper consultation.”

May I urge that the BSI uphold its admirable principles and does not approve DIS29500 in its present form. “

It will be the question has yet to be asked but seeing as approval was given I hope he can get to the bottom of why.

So what can be done now? The project editor has 30 days to produce a draft incorporating the result of the BRM. If that fails to appear it cannot be ratified by the ISO executive and DIS29500 will be dead in the water. As stated by ISO national bodies will be entitled to lodge a formal protest within 2 months that will have to be resolved first as well. It appears Norway might do that already – hopefully other countries will join them.

For my part? I will be writing to Mr. Pughs, MP giving my support to his questions. If he is right about suspicious activity in BSI then the result can be changed for 2 more months as a lodged complaint. I shall also be writing to my local MP pointing out his parliamentary question and international opinion. Finally I shall be writing to my MEP. Poland is being investigated by the EU commission and hopefully support will be lent to that.

If you are in the UK I’d encourage you to do the same – and elsewhere in the world to question your own national bodies and elected figures if you disagree your country’s vote.

As for what happens if this does pass? Well to be honest I doubt much will. Microsoft will no doubt proclaim loudly that their new XML Office format has been accepted world wide as a standard and lobby governments to use ISO29500. In all likelihood the promised ‘fixes in maintenance’ will never happen. Microsoft Office 2007 will probably not be changed to reflect ISO29500 anyway – but that will be unacknowledged by MS. IBM, Sun, the FOSS community and will keep developing ISO26300 and applications saving and loading to this format. Some people will have trouble exchanging documents and fall back to the binary MS Office formats for interchange of things to be edited whilst PDF (which is incidentally ISO 32000) will be used for storage/exchange with no changes to be made.

DISCLAIMER: As ever these are my personal views and do not necessarily correspond with those of my employer.

EDIT: Pamela Jones of Groklaw kindly reminded me that the Comes Vs Microsoft quote was about panels at conferences and not standards bodies. Edited this to make it clearer and fixed a few typos.