The Template System -- Design Document

I. Essentials

• User directory -- none; www exists only for documentation.
• ACS administrator directory -- none.
• Subsite administrator directory -- none.
• Tcl script directory. Almost no procedures show up here. To minimize dependencies across packages, in particular on ad_proc from acs-kernel, this package uses proc.
• PL/SQL API -- none.
• Data model -- none. Templating does not depend on a database system at all. There's the one table ad_template_sample_users that some of the demonstrations use.
• Requirements document
• ER diagram -- none.
• Transaction flow diagram -- none.

II. Introduction

• What this package is intended to allow the user to accomplish.

  The overall goal of the templating system is to provide the publishing team with a set of tools for simplifying the development and maintenance of the user interface. In particular:

  • A common solution. Programmers and designers should only have to learn a single system that serves as a UI substrate for all the functionally specific modules used on a site. The system should not make any assumptions about how pages should look or function. Designers should be able to change the default presentation of any module using a single methodology with minimal exposure to code.

  • Separation of code (Tcl, Java and SQL) and layout (HTML). Programmers should be able to specify the data sources and other properties of the template independently of the HTML template used to present the data. HTML authors should be to able to write templates that reference the data sources and properties without further intervention from the programmer to produce a finished page.

  • Separation of page components. There should be provisions so that pages can be broken into discrete components to simplify maintenance of the HTML code and allow for reuse in different contexts. Examples of common page components include a navigation bar, a search box, or a section of a report or story. Another common example is a portal page that allows the user to choose from a palette of features to display.

  • Global control over presentation. There should be a way to define one or more standard master templates used by most pages on a site, so that changes to the overall look and feel of a site can be made in one place.

  • Dynamic selection of presentation style. Given that the same data may be presented in many different ways, there should be a general mechanism for selecting a specific presentation (including file format, layout, character set and language) for each page request, depending on characteristics such as user preference, location, browser type and/or device.

  • Usability. Programmers should be able to develop template specifications using their standard tools for writing and maintaining code on the server. HTML authors should be able to access information about template specifications and work on templates remotely without needing shell access to the server.

• What this package is not intended to allow users to accomplish.

  • Tcl "pages" that do not return anything visible to the user. Such pages may be, e.g., the action= target of a form. They typically call ad_returnredirect after completing their job.
  • Tcl scripts that are scheduled to run in the server without a connection to a user.

• The application domains where this package is most likely to be of use.

  User interface. Any application that delivers visible pages to a user. Any page that returns content (HTML or other) in response to an HTTP[S] request.

• How the package meets its requirements.

  • It supplies a set of custom markup tags.
  • The proc ad_page_contract (from the acs kernel) should be used to specify what makes the dynamic part of the page. There's also an API for creating forms and for creating and manipulating multirow data sources.
  • The mechanism for dynamically generating pages combines data and layout. It also allows coposition of modular pages from reusable widgets and skins. It is not limited to HTML.
  • The <master> tag specifies a master template. Its src attribute defaults to the site-wide master template.

III. Historical Considerations

Karl Goldstein designed the templating system. First it was called "Karl's Templates" or "The New Templating System" to distinguish it from the obsolescent templates or "Styles" by Philip Greenspun. An extended and improved version was named "Dynamic Publishing System". It wasn't part of the ACS yet, but client projects like iluvCAMP used it successfully. Newcomers were consistently puzzled by the .data files, which specified the datasources in an apparently unfamiliar XML syntax. (The .form files specified elements in an HTML form similarly.) To mitigate this initial shock, Karl redesigned templates to let the programmer specify datasources and forms in a .tcl script. The system is present as packages templates and form-manager in ACS 3.4. Both these packages are now merged and appear as acs-templating starting in ACS 4.0. The architecture of the package was changed several times to meet the emerging coding/style constraints of ACS 4.0.

V. Design Tradeoffs

As indicated above, the primary attribute that the page tries to achieve is the separation of code and layout. The primary sacrifice is simplicity; in the typical case there will be two files (a .adp templage and a .tcl script) instead of a single .tcl page.

Management of data sources. Through the various past versions of the package, it evolved that data sources should be set as Tcl variables and arrays. Earlier they were kept as lists of ns_sets, which was significantly less efficient. The datasources are not being copied around; they reside in a specific stack frame. Using the uplevel Tcl command, we make sure that the data file (tcl part of a page) executes in the same stack frame as the compiled template (the adp part), so the latter can make use of the data sources prepared by the former.
        Thus, we decided for performance, simplicity, and ease of use at the cost of using the (standard Tcl) commands upvar and uplevel, which is considered confusing and error-prone by reviewers (of 4.0). The use of these constructs has been reduced in 4.01, and the code is clearer now.

Other attributes are affected as follows. In parentheses the estimated priorities are listed, not the degree to which the attributes are being achieved:

• Performance (high): Early versions of the templating system were a compuational burden. This has been fixed. Thanks to compilation of .adp pages and caching of both .adp and .tcl parts as procs, templated pages are much faster now; the caching can in fact make a templated page faster than an old-style .tcl page, which is sourced and parsed on every request.
• Flexibility (high): the recursive composition of pages allows for a big deal of flexibility.
• Interoperability (low): The templating system must tie in with the request processor for delivery of pages and with ad_page_contract for the specification of the expected parameters (called query) and the datasources it will supply (called properties). The templating system is registered with the request processor as the handler for both adp and tcl extensions.
• Reliability and robustness (medium): Considering how many parts have to play together, one might not predict a very reliable system. In practice, the package works reliably. It is robust to user errors in the sense that it won't error out if a file is missing or such; rather it quietly proceeds. Error reporting to the user is not very sophisticated.
• Usability (high): Emphasis has been put on the easy use of the system. In particular, a graphics designer should only have to learn a small number of special markup tags.
• Maintainability (medium): The code is well structured in reasonably sized procedures, and well commented.
• Portability (high): Unlike most other parts of the ACS, the templating system can work standalone. It doesn't need the database nor the acs-kernel or any other part of the ACS. All you need is AOLserver with the fancy ADP parser.
• Reusability (low): Many parts of the templating system are actually generally reusable, and probably should be extracted into a common set of utility procs used by this package and the ACS; this would reduce code duplication. The API lets programmers call into the system at different level. The templating system will probably mostly deliver HTML pages, but it has also been used to format mass mailings (spam), and any other formal (e.g., XML) could be used.
• Testability(low): A demonstration sample page exercises most mechanisms of the templating system.

VI. API

Details are in the developer guide. Here we give an overview, and then the more obscure aspects of the current implementation.

The most important abstraction is the data source, of which there are several kinds:

• onevalue (string)
• onerow
• multirow
• onelist
• multilist

Currently ad_page_contract does not allow specifying the latter two.

Process Flow

1. The request processor gets a url and maps it to a .adp or .tcl file. As both invoke the same handler, it doesn't matter that adp take precedence.
2. If a .tcl file is present, its ad_page_contract in the -properties block indicates a set of data sources that will be made available to the template.
3. The rest of the Tcl script executes, defining these data sources. It may change the name of the page being served by calling template::set_file directly or through the wrapper ad_return_template.
4. The corresponding template (file stub.adp) is interpreted and the data sources from the .tcl script are interpolated into the template.
5. The HTML is streamed to the client by the handler.

Tcl Call Stack

Below is a diagram of the typical call stack when processing a page without dependent pages. To conform to the Tcl notion of what's up and down (as in upvar), the stack grows down.

Level	Procedure	Arguments
#1	rp_handler
#2	rp_serve_abstract_file	/web/service/www/page
#3	rp_serve_concrete_file	/web/service/www/page.adp
#4	adp_parse_ad_conn_file
#5	template::adp_parse	/web/service/www/page {}
(6)	template::adp_prepare
#5	template::code::tcl::/web/service/www/page

Levels #1 to #3 exposed here are request processor internals. In the case shown, datasources reside in level #5. Due to the uplevel command, the frame of the sixth procedure is not accessible in the call stack at this moment, and the seventh runs in stack frame #5. If the <include> or <master> tags are used, adp_parse will be invoked recursively. Datasources always reside in the stack frame of an instance of adp_parse.

To keep track of data sources of several page components, the templating system maintains a stack of their stack levels in the variable template::parse_level. In our case, it just contains 5. But if this page included another page or designated is as its master, the level of the next adp_parse would be pushed to the list, and popped when that proc returned. This next level will appear as #6, due to the repeated upleveling.

Caching and Template Compilation

To improve performance, adp pages are compiled into a Tcl proc, and thus cached for future use. Tcl pages are also cached in a proc; this saves the trouble of reading and parsing the file the next time. The template system remembers the modification times of the adp and Tcl sources, and re-processes any requested file if the cached version is no longer current. Consequently, this cacheing is transparent in normal use.

To emphasize that "normal" use essentially always applies, here's a scenario for abnormal use: Save version n of a file at 11:36:05.1; request a page that uses it at 11:36:05.3; modify and save version n+1 of the file at 11:36:05.9. If you work that fast (!), the new version will have the same modification time -- kept with 1 second resolution in Unix --, and will not be refreshed.

For timing measurements and performance tuning, you can set the parameter RefreshCache in section template to never or always. The former suppresses checking mtime and may improve performance on a production server, where the content pages don't change. The latter is only inteded for testing.

VII. Data Model Discussion

This packages doesn't need a data model.

It comes with its own database interfaces, one for using ns_ora, the Oracle driver from ArsDigita, and one for ns_db, the builtin database interface of the AOL server. If you are programming under the ACS, you should use neither of these, but rather the db_* interface, in particular db_multirow.

VIII. User Interface

This packages doesn't have a user interface. It is the substrate of all user interfaces, be it user or admin pages.

IX. Configuration/Parameters

      [ns/server/yourservername/acs/template]
      ; the site-wide Master template
      DefaultMaster=/www/default-master
      ; anything other than "never" or "always" means normal operation
      RefreshCache=as necessary
    

X. Future Improvements/Areas of Likely Change

Passing datasources by reference is new. The acs-templating syntax &formal="actual" is different from the independent ATS, which used formal="@actual.*@". The latter is phased out.

We intend to add a <which>, <switch>, or <case> tag, to complement sequential nested <if>/<else> constructs.

Authors

• System creator: Karl Goldstein
• System owners: Karl Goldstein and Christian Brechbühler
• Documentation authors: Karl Goldstein, Christian Brechbühler, and Bryan Quinn

XII. Revision History