[Gd-hackers] Dylan documentation work

[Peter S. Housel]

Those who watch the gd-chatter mailing list or hang out in the #dylan IRC
channel may have noted the existence of a new documentation framework on the
gwydiondylan.org website, accessible via:

    http://www.gwydiondylan.org/cgi-bin/topic.cgi

This work is partially driven by the open-sourcing of Functional Developer
and its documentation, which was written using FrameMaker.  While a good
choice in a commercial environment, FrameMaker is not really appropriate for
an open-source project, especially given its high purchase price (USD
800 for Win32, if you must know).

I did some experimentation with converting the FunDev documents to DocBook
XML using FrameMaker's automatic document structuring tools, but discovered
that DocBook markup for API reference pages is not very appropriate for
Dylan (or other Lisp dialects), and found it hard to customize in a standard
way.

Instead, I decided to try the recently introduced DITA markup architecture.
DITA (see http://xml.coverpages.org/dita.html) is designed around two main
principles:

1. Topic orientation.  Documents are composed of individual topics, each
designed to be short and self-contained, effectively answering a single
question.  Larger document structures, such as articles, books, online help
systems, or websites can be composed out of multiple individual topics using
an external "map".  The context-independence of individual topics allows
them to be re-used in various different kinds of delivery contexts.  The
self-contained nature of topics also corresponds well to the way people use
manuals (especially API documentation) and online help.

2. Information typing.  Each topic has an information type specialized for
the sort of information it contains. In addition to the base information
type, <topic>, there are three pre-defined information types specialized
from it:

a. <concept> - answers "What is an X?"
b. <task> - answers "What steps do I take to do Y?"
c. <reference> - Provides a general reference for Z.

These topic types can be further specialized.  A generic specialization for 
API references was proposed and is described at

    http://groups.yahoo.com/group/dita-users/files/proposed_specializations/

I have further specialized this general framework to come up with document 
types for Dylan functions, classes, macros, variables, and constants.  These 
specializations can be found under documentation/topic/dtd in the 
repository.

Instances of these API reference topic types can be generated automatically 
for using the procedure described at:

    http://www.gwydiondylan.org/cgi-bin/topic.cgi/task/generating-api-documents.xml

I've already generated template API references for the "core" Open Dylan 
libraries, and pruned out multiply-exported modules and bindings.  They can 
be browsed at

    http://www.gwydiondylan.org/cgi-bin/topic.cgi/reference/lib

I'm gradually filling in these templates based on text from the Functional 
Developer documentation. To make this easier, I exported raw XML from 
FrameMaker and wrote a script that would turn this XML into individual topic 
files.  Each topic file just needs to be merged with its corresponding 
template, with a little bit of manual markup (and English) cleanup.

Converting things that aren't API reference entries takes a little more
brain work, categorizing topics and changing text to make it stand
better on its own.

Other than working through several hundred pages of printed docs and
converting them, what else is there to do?  Some ideas, in no particular
order:

- Navigation buttons.  It would be nice if someone would design
  a set of .png navigation buttons (prev/next/up, for-print view,
  raw xml source download, PDF download) to display at the head and
  foot of topic pages.  These should preferably fit in well with
  the rest of the website design.

- Map-based navigation.  Right now topic.cgi can only 
  browse the repository directory structure and display
  topics individually.  A way of navigating topics as
  components of a particular delivery map is also needed.
  This would make use of navigation buttons and possibly an
  ECMAscript table-of-contents tree display.

- Maps. After disassembling the manuals into individual topics,
  we need to be able to piece them back together again, possibly
  in new and interesting ways.

- Online help.  The DITA software already has tools for using a
  map to compose topics into Win32 HTML Help (CHM) files.

- Books.  Maps can also be used to compose text into PDF/printed
  manuals, using Apache FOP or some other XSL-FO processor.

- Smart API links.  It should be possible to look up the documentation
  for a particular Dylan binding using a query URL.  Knowledge of
  how to do this could be added to an IRC bot (similar to specbot
  in #lisp).

- Browsing by title (rather than just by repository hierarchy).

- Moving existing gwydiondylan.org web pages over to the new
  structure.  Topics are a good match for most of these pages, and
  it could be done gradually with little disruption.

- Annotations.  This would provide a simple text box that would allow
  people to "write in the margins" of a particular topic page, asking
  questions and supplying text for placeholder topics.  An editor
  could then periodically incorporate the annotations into the text
  itself.

- Replace the DRM.  There are API reference pages for all of the
  bindings in the dylan library, but no text to go with most of it.
  If we fill in these entries with our own text (possibly starting
  with the use of annotations), we can have the start of our own
  DRM without having to worry about the Apple copyright.

Who says documentation isn't exciting?
-- 
Peter S. Housel <housel at acm.org>