2.2. Web Robot Detection Design Documentation

2.2.1. Essentials

• ACS Administrator directory: /robot-detection/admin

• Tcl API

  • robot-detection-procs.tcl

  • robot-detection-init.tcl

• PL/SQL file

  • robot-detection.create.sql

• Web Robot Detection Requirements

2.2.2. Introduction

The goal of this package is to expose some parts of the site that are normally not indexable by search engines. The package accomplishes this by performing the following functions:

• Storing and maintaining information about known robots on the web (found at Web Robots Database) in a robots table.

• Using postauth ad_register_filters to identify robots connecting to areas covered by robot detection by matching the UserAgent header with known robots.

• Redirecting the robots to a special area on the site.

This special area can either be a static page or a dynamic script that generates texts from the database which are suitable for search-engine indexing.

2.2.3. Historical Considerations

Previous versions of the robot detection package relied on the AOLServer procedure ns_register_filter, which is deprecated as of ACS4. The robot detection package now uses the analogous ad_register_filter provided by the request processor in ACS 4.

The queries are re-written to use the ACS 4 Database Access API. Other than that, this version is a pretty straightforward port.

2.2.4. Competitive Analysis

On Apache, it's possible to accomplish more or less the same thing with mod_rewrite, but that's a low-level module that requires a lot of coding to make it work like the robot detection package.

2.2.5. Design Tradeoffs

• Only suitable for ACS administrator use. What the web robots from search engines index depends on the content in the the directory specified with the RedirectURL parameter ("robot heaven"). Right now there's no automatic way to generate the static (or pseudo static) files in "robot heaven." The administrator should decide what sort of content to expose to search engines and generate the files accordingly.

• To keep things simple, admin page permission and parameter setting depends on the site-mapper. In particular, setting multiple values for the FilterPattern parameter can be kind of clumsy. Because ad_parameter only allows one value per key at this time, I chose to use to use CSV (comma separated values) string to represent a list of paths to be filtered. Fortunately, once robot-detection is setup, it doesn't require further administration.

• Refreshing the robots table entails dropping all rows from robots table and then re-populating the data from scratch. This ensures synchronization with the data from the web robots database.

• The raw data comes from one static file that must be retrieved in its entirety using http. However, building and maintaining real rdbms backed registry of web robots is not worth the effort because the number of robots is unlikely to grow very much.

2.2.6. API

The API for robot detection is quite simple: (You can also use the API Browser to view this)

2.2.7. Data Model Discussion

The data model has one table: robots. It mirrors a few critical fields of the schema laid out in Web Robots Database Schema. Only robot_id, robot_name, robot_details_url, and robot_useragent are used in the current implementation.

• robot_id is not generated from a sequence in Oracle, rather it is the robot_id assigned by Web Robot Database for each individual robot.

• robot_name and robot_details_url are used to display a list of robots in the /robot-detection/admin Robot Detection Administration page.

• robot_useragent is the most important field of the bunch. An incoming http request is identified as a robot if the UserAgent header matches of one of the robot_useragent in the robots table.

2.2.8. User Interface

There is one page for ACS Administrators. It displays the current parameters and a list of robots, plus a link to manually refresh the robots table.

2.2.9. Configuration/Parameters

2.2.10. Future Improvements/Areas of Likely Change

2.2.11. Authors

• System creator: Michael Yoon

• System owner: Roger Hsueh

• Documentation author: Roger Hsueh

2.2.12. Revision History