OpenACS Version 5.9.1
Release of OpenACS 5.9.1:
This is the announcement of the release of the OpenACS 5.9.1. This
release contains many security and performance improvements and
includes new functionalities.
The release of OpenACS 5.9.1 contains the 88 packages of the oacs-5-9
branch. These packages include the OpenACS core packages, the major
application packages (e.g. most the ones used on OpenACS.org), and
DotLRN 2.9.1.
The quanitative summary of the changes since the release of
OpenACS 5.9.0 is as follows:
3548 files changed, 113292 insertions(+), 90507 deletions(-)
contributed by 5 committers (Michael Aram, Gustaf Neumann, Antonio
Pisano, Hector Romojaro, Thomas Renner) and 8 patch/bugfix providers
(Frank Bergmann, Günter Ernst, Brian Fenton, Felix Mödritscher, Marcus
Moser, Franz Penz, Stefan Sobernig, Michael Steigman). All packages of
the release were tested with PostgreSQL 9.6.* and Tcl 8.5.*.
Below is a short summary of the changes in this release. For details,
consult the changelog of the release.
Refactoring of rich-text editor integration
- Driving force: Debian packaging (e.g. js minified code is not
allowed)
- Moved out code from acs-templating, provided interfaces to add
many different richtext editors as separate packages
- New OpenACS packages:
* richtext-xinha
* richtext-tinymce
* richtext-ckeditor4 (has ability to choose between CDN and
local installation via web interface)
Improving admin interface
- New theme manager:
* Goals:
+ Make it easier to keep track of themes with local
modifications
+ Make it easier to create local modification a new
themes and to update these
+ Show differences between default theme parameter (in DB)
and actual settings (in subsite parameters)
+ Allow to delete unused themes
+ Give site admin hints, which theme is used at which
subsite
+ Ease theme switching
* Added a subsite::theme_changed callback to be able to handle
theme changes in custom themes (was also necessary for proper
integration with DotLRN theming)
* Added support for these features under subsite admin (/admin/)
Improved support for themed templates via
[template::themed_template]
- Improved (broken) interface to define/manage groups over web
interface
- Allow to send as well mail, when membership was rejected
- New functions [membership_rel::get_user_id],
[membership_rel::get] and [membership_rel::get_group_id] to
avoid code duplication
- Added support to let user include %forgotten_password_url%
in self-registration emails (e.g. in message key
acs-subsite.email_body_Registration_password)
- Improved subsite/www/members
* Make it possible to manage members of arbitrary groups
* Improved performance for large groups
* Improved configurability: when ShowMembersListTo is set to
"3", show list to members only, when this is not the whole
subsite
- Improved user interface for /admin/applications for large number
of applications
- Various fixes for sitewide-admin pages (under /acs-admin)
- Update blueprint in "install from repository"
(currently just working in NaviServer)
SQL
- Further cleanup of .xql files (like what as done for acs-subsite
in OpenACS 5.9.0):
* 36 files deleted
* Removed more than 100 obsolete named queries
* Stripped misleading SQL statements
- Marked redundant / uncalled sql functions as deprecated
- Replaced usages of obsolete view "all_object_party_privilege_map"
by "acs_object_party_privilege_map"
- Removed type discrepancy introduced in 2002:
* acs_object_types.object_type has type varchar(1000), while
* acs_object_types.supertype has type varchar(100)
* ... several more data types are involved, using
acs_object_types.object_type as foreign key
- Simplified core sql functions by using defaults:
* Number of functions reduced by a factor of 2 compared to
OpenACS 5.9.0 (while providing compatibility for clients using
old versions),
* Reduced code redundancy
Affected functions:
+ Reduced content_item__new from 12 versions to 6,
+ Reduce content_revision__new from 7 to 4
+ Similar in image__new, image__new_revision,
content_item__copy, content_item__get_title,
content_item__move
+ PostgreSQL 9.5 supports named parameter in the same syntax
as in Oracle. Further reduction of variants will be
possible, once OpenACS requires at least PostgreSQL 9.5
- Reduced usage of deprecated versions of SQL functions
(mostly content repository calls)
- Reduced generation of dead tuples by combining multiple DML
statements to one (reduces costs of checkpoint cleanups in
PostgreSQL)
- Permission queries:
* Improved performance
* Support PACKAGE.FUNCTION notation for PostgreSQL to allow
calls permission queries exactly the same way as in Oracle
(e.g. "acs_permission.permission_p()"). This helps to reduce the
number of postgres specific .xql files.
- Modernize SQL:
* Use real Boolean types instead of character(1)
(done for new-portal, forums, faq, attachments, categories,
dotlrn, dotlrn-forums, evaluation)
* Use real enumeration types rather than check constraints
(done for storage_type text/file/lob)
CR hygienics (reduce cr bloat)
- Provided means to avoid insert/update/delete operations in the
search queue:
OpenACS adds for every new revision often multiple entries to the
search_queue, without providing any means to prevent this. This
requires for busy sites very short intervals between queue sweeps
(otherwise too many entries pile up). Another consequence is that
this behavior keeps the PostgreSQL auto-vacuum daemons
permanently active. Many of these operations are useless in cases
where the content repository is used for content that should not
be provided via search. The changed behavior should honors a
publish-date set to the future, since it will not add any content
with future publish dates to the search-queue.
- Reduced number of insert cr_child_rels operations, just when needed:
cr_child_rels provide only little benefit (allow to use roles in
a child-rel), but the common operation is a well available in
cr_items via the parent_id. cr_child_rels do not help for
recursive queries either. One option would be to add an
additional argument for content_item__new to omit child-rel
creation (default is old behavior) and adapt the other cases.
Security improvements
- Added support against [[CSRF]] (cross site request forgery)
* OpenACS maintains a per-request CSRF token that ensures that
form replies are coming just from sites that received the form
* CSRF support is optional for packages where CSRF is less
dangerous, and such requests are wanted (e.g. search and
api-browser)
- Added Support for W3C "Upgrade-Insecure-Headers"
(see https://www.w3.org/TR/upgrade-insecure-requests/):
For standard compliant upgrade for requests from HTTP to HTTPS
- Added support for W3C "Subresource Integrity" (SRI; see
https://www.w3.org/TR/SRI/)
- Added support for W3C "Content Security Policy"
([[CSP]]; see https://www.w3.org/TR/CSP/)
* Removed "javascript:*" links (all such urls are removed from
the 90 packages in oacs-5-9, excluding js libraries
(ajaxhelper) and richtext code)
* Removed "onclick", "onfocus", "onblur", "onchange" handlers
from all .adp and .tcl files in the 90 packages in oacs-5-9
(excluding js libraries (ajaxhelper) and richtext code)
* Added optional nonces to all <script> elements with literal
JavaScript content
- Removed "generic downloader", which allowed to download arbitrary
content items, when item_id was known (bug-fix)
- Improved protection against XSS and SQL-injection (strengthen
page contracts, add validators, added page_contract_filter
"localurl", improve HTML escaping, and URI encoding)
- Fixed for potential traversal attack (acs-api-documentation-procs)
Improvements for "host-node mapped" subsites
- Fixed links from host-node mapped subsite pages to swa-functions
(must be always on main subsite)
- Made "util_current_directory" aware of host-node-mapped subsites
- Added ability to pass "-cookie_domain" to make it possible to use
the same cookie for different domains
- Fixed result of affected commands "util_current_location",
"ad_return_url", "ad_get_login_url" and "ad_get_logout_url" for
HTTP and HTTPS, when UseHostnameDomainforReg is 0 or 1.
- Improved UI for host-node maps when a large number of site nodes
exists
Reform of acs-rels
- Made acs-rels configurable to give the developer the option to
specify, whether these are composable or not (default fully
backward compatible). This is required to control transitivity in
rel-segments
- The code changes are based on a patch provided by Michael
Steigmann. For details, see:
* http://openacs.org/forums/message-view?message_id=4031049
* http://openacs.org/forums/message-view?message_id=5330734
Improved status code handlers for AJAX scenarios
- Don't report data source errors with status code 200
(use 422 instead)
- Let "permission::require_permission" return forbidden (403) in
AJAX calls (determined via [ad_conn ajaxp])
Improved Internationalization
- Extended language catalogs for
* Russian (thanks to v v)
* Italian (thanks to Antonio Pisano)
* Spanish (thanks to Hector Romojaro)
* German (thanks to Markus Moser)
- Added (missing) message keys
- Improved wording of entries
- Added message keys for member_state changes, provide API via
group::get_member_state_pretty
Improved online documentation (/doc)
- Fixed many broken links
- Removed fully obsolete sections
- Improved markup (modernize HTML)
- Updated various sections
Misc code improvements:
- 18 issues from the OpenACS-bug-tracker fixed
- Made code more robust against invalid/incorrect input
(page_contracts, validators, values obtained from header fields
such as Accept-Language)
- Fixed quoting of message keys on many places
- Improved exception handling (often, a "catch" swallows to much,
e.g. script_aborts), introducing "ad_exception".
- Generalized handling of leading zeros:
* Fixed cases where leading zeros could lead to unwanted octal
interpretations
* Switch to use of " util::trim_leading_zeros" instead of
"template::util::leadingTrim", "dt_trim_leading_zeros" and
"template::util::leadingTrim", marked the latter as deprecated
- URL encoding
* "ad_urlencode_folder_path": new function to perform an
urlencode operation on the segments of the provided folder
path
* "export_vars": encode path always correctly, except
no_base_encode is specified
* Fixed encoding of the URL path in "ad_returnredirect"
- Improvements for "ad_conn":
* Added [ad_conn behind_proxy_p] to check, whether the request
is coming from behind a proxy server
* Added [ad_conn behind_secure_proxy_p] to check, whether the
request is coming from behind a secure proxy server
* Added [ad_conn ajax_p] to check, whether the request is an
AJAX requests (assumption: AJAX request sets header-field
Requested-With: XMLHttpRequest")
* Added [ad_conn vhost_url] to obtain the url of
host-node-mapped subsites
- Added various missing upgrade scripts (missing since many years)
of changes that were implemented for new installs to reduce
differences between "new"-and "old" (upgraded) installations
- Templating
* Get rid of various pesky "MISSING FORMWIDGET:
...formbutton:ok" messages
* Improved support for javascript event handlers in template::head
* New functions "template::add_event_listener" and
"template::add_confirm_handler"
* Fix handling, when "page_size_variable_p" is set (was broken
since ages)
- Improved location and URL handling:
* Refactored and commented "util_current_location" to address
security issues, handle IPv6 addresses, IP literal notation,
multiple drivers,
* Improved "security::get_secure_location" (align with documentation)
* New functions:
+ "util::configured_location"
+ "util::join_location", "util::split_location"
for working on HTTP locations to reduce scattered regexps
handling URL components
* Improved IPv6 support
* Use native "ns_parseurl" when available, provide backward
compatible version for AOLserver
- MIME types:
* Added more Open XML formats for MS-Office to allowed content types
* Modernized entries to IANA recommendations
* New function "cr_check_mime_type" centralizing the retrieval
of the mime_type from uploaded content
- Finalized cleanup of permissions (started in OpenACS 5.9.0):
Get rid of "acs_object_context_index " (and therefore on
acs_object_party_privilege_map " as well) on PostgreSQL.
Reasons:
* huge table,
* expensive maintenance, used only in a few places,
- Misc new functions:
* "lang::util::message_key_regexp": factor out scattered regexp
for detecting message keys
* "ns_md5" and "ns_parseurl": improve compatibility between
AOLserver and NaviServer
* "ad_dom_sanitize_html": allow to specify different sets of
tags, attributes and protocols and "ad_dom_fix_html", which is a
light weight tidy variant.
- Improved HTML rendering (acs-api-browser), provide width and
height to speed up rendering
- Improved ADP files (e.g. missing doc(title))
- Added usage of "ad_include_contract" on more occasions
- Modernize Tcl and HTML coding
- Reduced dependency on external programs (use Tcl functions instead)
- Improved robustness of "file delete" operations all over the code
- Improved documentation, fix demo pages
- Aligned usages of log notification levels (distinction between
"error", "warning" and "notice") with coding-standards
- Cleaned up deprecated calls:
* Removed usage of deprecated API functions
(e.g. "cc_lookup_email_user",
"cc_email_from_party", "util_unlist", ...)
* Moved more deprecated procs to acs-outdated
* Marked remaining (and unused) "cc_*" functions as well as
deprecated.
- Improved Oracle and windows support
- Fixed common spelling errors and standardize spelling of product
names all over the code (comments, documentation, ...)
- Many more small bug fixes
Packages:
- New Package Parameters
* acs-kernel:
+ MaxUrlLength: remove hard-coded constant in request
processor for max accepted url paths
+ SecureSessionCookie: Let site admin determine, whether or
not to use secured session cookies (useful, when not all
requests are over HTTPS)
+ CSPEnabledP: activate/deactivate CSP
* acs-kernel (recommended to be set via config file in section
"ns/server/${server}/>acs"
+ NsShutdownWithNonZeroExitCode: tell NaviServer to return
with a non-zero return code to cause restart (important under
windows)
+ LogIncludeUserId: include user_id in access log
* acs-api-browser:
+ ValidateCSRFP: make checking of CSRF optional (default 1)
* acs-content-repository:
+ AllowMimeTypeCreationP: Decides whether we allow unknown
mime types to be automatically registered (default: 0}
* news-portlet:
+ display_item_lead_p: Should we display news leads in the
portlet? (default 0)
* search:
+ ValidateCSRFP: make checking of CSRF optional (default 1)
* xotcl-request-monitor:
+ do_track_activity: turn activity monitoring on or off (default 0)
- New OpenACS packages:
* richtext-xinha
* richtext-tinymce
* richtext-ckeditor4 (has ability to choose between CDN and
local installation via Web interface)
* openacs-bootstrap3-theme (as used on openacs.org)
* dotlrn-bootstrap3-theme
- xotcl-core:
* Improved XOTcl 2.0 and NX support (e.g. api-browser)
* Added "-debug", "-deprecated" to ad_* defined methods
(such as e.g. "ad_instproc")
* Make use of explicit "create" statements when creating
XOTcl/NX objects (makes it easier to grab intentions and to
detect typos)
* Added parameter to "get_instance_from_db" to specify, whether
the loaded objects should be initialized
* Added support for PostgreSQL prepared statements of SQL
interface in ::xo::dc (nsdb driver)
- xowiki:
* Named all web-callable methods www-NAME (to make it clear,
what is called, what has to be checked especially carefully)
* Moved templates from www into xowiki/resources to avoid naming
conflicts
* Improved ckeditor support
* Added usage of prepared statements for common queries
* Improved error handling
* Better value checking for query parameter, error reporting via
ad_return_complaint
* Added option "-path_encode" to methods "pretty_link" and
"folder_path" to allow to control, whether the result should be
encoded or not (default true)
* Form fields:
+ Improved repeatable form fields (esp. composite cases),
don't require preallocation (can be costly in composite
cases)
+ Added signing of form-fields
+ Added HTML5 attributes such as "multiple" (for "file") or
"autocomplete"
+ Fixed generation of "orderby" attribute based on form-field
names
+ richtext: allow to specify "extraAllowedContent" via options
+ Improved layout of horizontal check boxes
* Menu bar:
+ Added dropzone (requires bootstrap): drag and drop file
upload
+ Added mode toggle (requires bootstrap)
+ Extended default policies for handling e.g. dropzone
(file-upload method)
+ Distinguish between "startpage" (menu.Package.Startpage)
and "table of contents" (menu.Package.Toc)
* Notifications:
+ Added support for better tailorable notifications:
introduced method "notification_render" (similar to
"search_render")
+ Added support for tailorable subject lines (method
"notification_subject")
* Improved bootstrap support, use "bootstrap" as
PreferredCSSToolkit
* Switched to ckeditor4 as PreferredRichtextEditor
* Improved handling of script-abort from within the payload of
::xowiki::Object payloads
* Added parameter to "get_all_children" to specify, whether the
child objects should be initialized
- xowf:
* Added property "payload" to "WorkflowConstruct" in order to
simplify customized workflow "allocate" actions
* Internationalized more menu buttons
- xotcl-request-monitor
* Added class "BanUser" (use. e.g. ip address to disallow
requests from a user via request monitor)
* Added support for optional user tracking in database
* Added support for monitoring response-time for certain urls
via munin
* Increased usage of XOTcl 2.0 variable resolver (potentially
speed improvement 4x)
* Performed some refactoring of response-time handling to allow
site-admin to make e.g. use of NaviServer's dynamic
connection pool management (not included in CVS)
* Added support for partial times in long-calls.tcl to ease
interpretation of unexpected slow calls
* last100.tcl: Don't report hrefs to URLs, except to SWAs
- chat:
* Introduced new options to set chat rooms so login and/or
logout message are not issued every time a user enters/exits a
chat-room (important for chats with huge number of participants)
* Parameterized viewing of chat-logs
* Fixed cases of over-/under-quoting
* Fixed JavaScript for IE, where innerHTML can cause problems
- file-storage:
* Don't show action keys, when user has no permissions
* Added support for copying of same-named files into a folder
(adding suffix)
* Fixed old bugs in connection with "views" package