16. Site maps

16.1. How to write site maps

In this section, we explain how to use site maps in xhtml2to1. The detailed exposition on how they work can be found in Theory of operation.

The documentation writer must provide a site map document so that xhtml2to1 knows how to build a navigation menu for the set of documents it is about to process. The site map also indicates how sections of the documentation should be numbered.

The following gives an example.

16.1.1. Site map example

<html>

<head>
<title>Documentation for my amazing program</title>
</head>

<body>
<nl>
  <li href="intro.xml" />
  <li href="driver.xml" />
  <li href="document.xml" />
  <li href="structure.xml" />
  <li href="text.xml" />
  <li href="footnotes.xml" />
  <li href="hypertext.xml" />
  <li href="list.xml" />
  <li href="xref.xml" />
  <li href="copy-xhtml1.xml" />
  <li href="navigation.xml" />
  <li href="uri.xml" />
  <li href="quote-xml.xml" />
</nl>
</body>
</html>

16.1.2. Specifying cross reference labels

The s:label element defines the text generated for any cross reference to the resource given in its href attribute. So with the above example site map, if the XHTML 2.0 source reads:

<p>See <xref href="xref.xml" /> for details on cross referencing…</p>
then the XHTML 1.0 output would be:
<p>See <a href="xref.html">Resolving cross references</a> for
details on cross referencing…</p>

(Notice that we have changed “xref.xml” in the XHTML 2.0 href attribute to “xref.html”. xhtml2to1 can perform this substitution automatically — we will explain the details in a moment.)

The ctx attribute on s:label is not yet implemented. When it is, it specifies the context for which the given cross reference label can be used.

16.1.3. Specifying navigation menus

Next, we also want xhtml2to1 to create navigation menus for us. Since each section in the document are separate XML documents, they need to be explicitly linked together. We tell xhtml2to1 what goes in the navigation menu using XHTML 2.0’s nl element (meaning navigation list).

Each entry in the navigation list is introduced by the familiar li element. The nl navigation lists may be nested, to create hierarchical1

16.1.4. Expanding navigation menus

Now, it would be annoying if we had to key in all the entries in the navigation menu, including all the subsections. If a XHTML 2.0 file consists of nested sections like this:

<section>
  <h>Chapter 1</h>
  <section>
    <h>Section 1.1</h>
    <section>
      <h>Subsection 1.1.A</h>
    </section>
  </section>
</section>
we want the lower sections — namely, “Section 1.1” and “Section 1.1.A” in this example — to be entered into the navigation menu automatically.

There is a stylesheet in xhtml2to1, auto-nav.xsl, to do just this: it takes a site map file that you write, and expands it to include all the subsections.

xsl:stylesheet id="stylesheet" exclude-result-prefixes="lit #default" version="1.0" xml:lang="en"

xsl:output method="xml"

xsl:include href="uri.xsl"
xsl:include href="xsltsl/uri.xsl"


xsl:template match="@*|node()"
  xsl:copy
    xsl:apply-templates select="@*|node()"
  


xsl:template match="hs:label|hs:number"
  xsl:copy
    xsl:attribute name="href"
      xsl:call-template name="absolute-uri"
    
    xsl:apply-templates
  


xsl:template match="x2:li[@href]"
  xsl:variable name="absolute-uri"
    xsl:call-template name="absolute-uri"
  
  xsl:variable name="number"
    xsl:number label="multiple" count="x2:li"
  
  
  <hs:html-uri href="{$absolute-uri}">
    xsl:attribute name="to"
      xsl:call-template name="html-uri"
        xsl:with-param name="href" select="$absolute-uri"
      
    
  </hs:html-uri>

  xsl:variable name="t" select="document(@href, .)"
  xsl:if test="count($t)>0"
    xsl:apply-templates select="$t" mode="auto-nav"
      xsl:with-param name="base-uri" select="$absolute-uri"
      xsl:with-param name="number" select="$number"
    
  


xsl:template match="x2:section" mode="auto-nav"
  xsl:param name="base-uri"
  xsl:param name="number"

  xsl:variable name="href"
    xsl:value-of select="$base-uri"
    xsl:text#
    xsl:call-template name="id"
  

  <x2:li href="{$href}">
    xsl:apply-templates select="x2:h/node()"

    <hs:number href="{$href}">
      xsl:copy-of select="$number"
      xsl:text.
      xsl:number label="multiple" format="1." from="/x2:section" count="x2:section"
    </hs:number>
    <hs:label href="{$href}">
      xsl:apply-templates select="x2:h/node()"
    </hs:label>

    xsl:if test="x2:section"
      <x2:nl>
        xsl:apply-templates select="x2:section" mode="auto-nav"
          xsl:with-param name="base-uri" select="$base-uri"
          xsl:with-param name="number" select="$number"
        
      </x2:nl>
    
  </x2:li>

  xsl:text



xsl:template match="/|*" mode="auto-nav"
  xsl:param name="base-uri"
  xsl:param name="number"
  xsl:apply-templates mode="auto-nav"
    xsl:with-param name="base-uri" select="$base-uri"
    xsl:with-param name="number" select="$number"
  


xsl:template match="text()" mode="auto-nav"


16.2. Motivation for the xhtml2to1 linkage model

The XHTML 2.0 documents processed by xhtml2to1 are linked together differently from documents based on traditional SGML DTDs (such as DocBook), and linked more like HTML web pages.

16.1. Comparison with the DocBook model

For comparison, let us take an example DocBook book, containing some chapters and sections within them. We might organize our DocBook XML files like this:

  • document.xml
  • chapter1/
    • sect1.xml
    • sect2.xml
  • chapter2/
  • chapter3/
where document.xml would include all the other XML files using XML entities (the traditional way), or by XInclude (the new-fangled way).

So everything is conceptually under one XML document, which happens to have fragments spread over multiple external entities.

By contrast, Web pages do not work that way. Instead, each file, say chapter1/sect1.html is a complete XML document in itself. Hyperlinks are made to the other sections of the book by URI addresses.

The chief advantage of the DocBook model is that it is simple and straightforward. If you want to make a cross reference from one part of the book to another, you simply attach an ID to the element where the cross reference should go, and refer to that ID at other parts of the book.

16.3. DocBook documents can get too big

Another problem with using external entities or XInclude to incorporate sections of a book into one document is that the document can get big. This fact is undesirable in light of the fact that XSLT processors must load all of a document into memory before processing.

With modern computing capabilities, this seem to be a moot issue, but consider the situation that we want to serve XML documents directly to Web browsers (slowly becoming a real possibility). Obviously, we want to avoid loading in one-megabyte DocBook XML sources just to display one page of that document.

Or consider writing programs by XML-based literate programming. The program may be on the scale of ten thousand lines of code or more. If the literate programming document is all in one piece, then everything has to be processed to build the program. Consider that programmers have had incremental compilation and dynamic linking facilities available for a long time; we certainly do not want to take away this ability when writing in XML. In fact, the usual practice of not processing large XML documents incrementally ought to be considered as an anomaly.

16.4. Should we give up ID-based linking?

Unfortunately, we cannot just answer all the problems with DocBook-style ID-based linking by saying “just use URLs and the XHTML a element”. If we use the a element alone, we are basically forced to key in all the links and cross reference labels manually. We also lose automatic numbering of sections. These things, of course, we expect a computer program to for us automatically.

As the author found out while trying to write xhtml2to1, a solution to these problems is not at all trivial. At this point, we would be wise to question whether we should continue to use ID-based linking despite its problems.

A DocBook-style monolithic XML document is often viewed, very logically, as a tree with the document element at the tree root, with child elements and text nodes branching out. A subsection is a child node of a section node, which in turn is a child node of a chapter node, then a book node, and so on.

The problems with this model we are to consider are:

  1. What if we want to refer to a separate document tree?
  2. What if the document tree gets too big (to fit comfortably in memory)?

We could view these problems as only problems in practice only: just because we run into a few snags with the document-as-big-tree model does not mean we should throw out the elegant theoretical model.

16.1. XML databases as the solution?

To solve these problems, we could consider using XML databases. They can store many document trees of large size very efficiently, and retrieve and manipulate small parts of the tree efficiently too.

An obvious problem with XML databases is that they are too demanding for many users. Documents start out small, and many stay small (e.g. less than one megabyte in the size of the plain text representation). Requiring XML databases, along with their complexity, seems to be overkill. Many users (including this author) may want to write their XML documentation with vi, emacs or a graphical XML editor, or generate the documentation from comments in the source code. Most of these processing tools do not support XML databases. Many users may also want to publish their documentation as files, without depending (too much) on a fancy XML database and Web server up and running.

The other problem that even XML databases do not solve is isolating dependencies in a big document tree. A XML database knows how to “patch” a big document in small pieces, but it cannot know how these patches affect other parts of the document, because the effects depend on markup semantics. For example, if the writer changes the title of a section in the document, the display system needs to know that all the cross references to that section occurring elsewhere in the document must change their labels. If the display system naïvely uses the XML database, without keeping track of the cross reference markup, then it still has to process the whole document again just on changing one section label. Which is obviously no better than storing XML in flat text files.

16.2. Big-tree processing is slow

Actually, even for moderately-sized documents, DocBook already exhibits a (in)famous misfeature: documents are slow to process. As a simple back-of-the-hand calculation, the author has timed the running of the DocBook XSL stylesheets to create HTML documentation for his docbook2X package. On the author’s Pentium IV 1.7GHz machine with the libxslt processor (one of the fastest), it takes about 5.6 seconds. The documentation consists of 30 printed pages. Now, it is not uncommon to have computer books that are just a little less than 1000 pages thick. To generate HTML for such books would take about 3 minutes. Even that seems to be nothing alarming, if we only need to compile book-sized documents only occasionally (e.g. only when printing them).

But what if we want to revise a book, or a literate program, incrementally? Three minutes to compile the whole book after fixing a little typo in the source is ridiculous. That breaks the writer or programmer’s write-compile-test cycle, hurting productivity.

16.5. Summary

So, in summary, we would like a system that:

  1. is as almost as easy to use as flat XML files; and
  2. is scalable to large documents — formatting small changes should be nearly instant, not much slower than reloading a page in a Web browser.

It is a given that xhtml2to1 needs to support ID-based linking as well, for obviously, IDs are used for linking within a sub-document. But xhtml2to1 does not intend to include features specifically for working with large monolithic documents.

16.3. Theory of operation

In very general terms, a site map or navigation document is an index of all the relevant documents of a site and all the information needed for navigating through those documents. The information is needed to provide the navigation menu for the individual pages, to provide section numbering and to provide cross-reference labels for links between two documents.

There are many possible designs for what a site map or navigation document should be, and how the information inside it should be organized. For example, one could use a content management system, backed by a database, to keep track of the site map. In xhtml2to1, the site map scheme is a fairly simple design that can be efficiently implemented with XSLT.

We insist on the design being simple to minimize the inertia of the potential writer to write technical documentation. Thus there will necessarily be some trade-offs, such that certain advanced modes of navigation cannot be expressed in the current design. If those features are desired, then the only recourse is to invent a new kind of site map scheme; fortunately, xhtml2to1 is modular enough to allow this. We will have more to say about this topic later.

16.1. Structure of the site map

A xhtml2to1 site map is a XHTML 2.0 document, whose body element contains one or more nl navigation lists, usually written by the user. The navigation lists may be hierarchical, and entries in the navigation list are written in the form

<li href="uri" …

The xhtml2to1 stylesheets can automatically expand the navigation list to include the subsections under the document located at uri. The expansion always replaces the original li element itself, with another li element, with the same href attribute, and it contains the content of the expansion as its descendant nodes.

The expansion works this way so ease incremental builds of documents. If it is determined that the sub-document at uri has not changed, then the element content of li can simply be copied over for the next build. If the sub-document has changed, then replacing the li element yields a simple way of chucking out the old information for that sub-document, without considering more complicated data merging schemes.

A li element with no href attribute is always copied verbatim, so that the user may optionally put in links to external documents, for instance:

<li><a href="/">Back to home page</a></li>

16.2. Incremental builds

Despite the colorful criticism we made of large monolithic DocBook documents, we must not get too optimistic on how far incremental building of large documents can be pushed. If a change is made to a sub-document that affects all the other documents, such as changing a cross-reference label, or adding a new entry to the navigation menu, then all the sub-documents may still have to be generated.

But we can still obtain some benefits:

  1. Even if all the sub-documents have to be processed again at a certain stage, at least they are processed sequentially in small chunks. It is not necessary to load a five-megabyte XML document tree into memory all at once.
  2. Localized changes do not force all sub-documents to be processed again.
  3. To save on build time during development, the writer may allow some of the output files to be temporarily out-of-date (which only affects a small number of items on the navigation menu, and a few cross-reference labels). Only at the final stage does the writer do a full build of the documentation.
  4. If the writer’s Web browser supports loading XML documents directly and applying XSLT stylesheets to them2, then the sub-documents do not need to be generated all at once. For writing a book-size document, this saves a lot of time, as usually one works on only a few chapters at a time.

16.4. Resolving relative URIs

An essential function performed by the site map stylesheet is to resolve all relative links in the given site map into absolute links in the expanded site map.

16.4.1. Why?

Why do we need absolute URIs?

Consider generating a document, located in the file chapter1/sect1.xml, with the site map present in site-map.xml. The site map specifies cross reference labels and navigation lists as URIs; if these URIs are relative, then they are relative to the site map document site-map.xml. But relative links in chapter1/sect1.xml must necessarily be interpreted as relative to chapter1/sect1.xml.

So when we want to compare a href link in chapter1/sect1.xml versus a href link in site-map.xml, we must normalize them somehow. And the easiest way to do this is to make both of them into absolute URIs; then the two links point to the same thing if and only if the absolute URIs that they resolve to compare equal as character strings.


Notes

  1. Hierarchical menus are not always the best way to organize a document. But they are basic and familiar.
  2. Such as Mozilla Firefox.