acs-tcl.info

On openacs.org, we are experiencing numerous requests with

multiply very long and strange query variables like in the example

below. So far, it is not clear, whether these requests are the

consequence of a double encoding or a deliberate attack. Many (most)

of the requests contain the query variable names containing the

(decoded) pattern "*amp;*".

This is a relatively new phenomenon. I cannot exclude that this is a

bug introduced lately in OpenACS, or a bug in an external bot, or

whatever. The problem with these query variables is that OpenACS

propagates these further, e.g., when updating query variables in

ad_dimensional, via export_vars, or return_urls.

Since OpenACS never uses these query-variables, these can be safely

skipped, without loosing functionality in OpenACS. It is possible to

construct examples, where skipping such variables can change the

semantics. Therefore, the change introduces a single function

util::suspicious_query_variable where in case of problems, the

skipping feature can be deactivated.

GET /api-doc/proc-browse?amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3borderby=name&type=All&amp%3btype=All&amp%3bamp%3btype=All&amp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3borderby=name&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=Private&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3borderby=name&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All HTTP/1.1" 200 62378 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15" "1729029614.331581 0.109805 0.000434 0.004026 0.215927

This convenience function eases handling of ad_script_abort exceptions

in cases, where the OpenACS request processor is not involved.

From the documentation:

Execute the provided body in the callers' environment. When the

body does not raise an "ad_script_abort" exception, the

"non_abort_action" is also executed. This pattern is useful when

handling client requests and where the "non_abort_action" is used

to return results to the client. When "ad_script_abort" is

executed, the connection is usually closed, and any attempt to

talk to the client over the closed connection will fail.

The handling of script_abort exceptions is done usually in the

request processor. The function is useful when registering own

request procs (e.g., via "ns_register_proc") where the OpenACS

request processor is not involved.

bumped version to 5.10.1b12

Some functions are defined in the database with the same number of

arguments but different types, e.g., first argument "package_key"

(type text) or "package_id" (type integer). This is fine from the SQL

standpoint, but when calling from Tcl via bind-vars

(e.g. ":package_id"), everything is passed as a string, and

potentially, the wrong function is called.

Now, all the automatically generated subs are generated with casts,

when the integer based variant must be called.

Some examples:

Before:

set s [ns_pg_bind 0or1row $__DB {select apm__set_value(:package_id,:parameter_name,:attr_value)}]

set s [ns_pg_bind 0or1row $__DB {select apm__get_value(:package_id,:parameter_name)}]

Now:

set s [ns_pg_bind 0or1row $__DB {select apm__set_value(CAST(:package_id AS integer),:parameter_name,:attr_value)}]

set s [ns_pg_bind 0or1row $__DB {select apm__get_value(CAST(:package_id AS integer),:parameter_name)}]

- bumped version number to 5.10.1b11

bumped version number to 5.10.1b10

- Added convenience function "::util::resources::register_urns" to

register all URNs with CSP handling provided by a package (denoted

by its top level namespace)

- made parameter "version" in "check-installed" include optional

- bumped version number to 5.10.1b8

- New proc ::util::resources::check_vulnerability

- bumped verison number to 5.10.1b7

This eases the use of external JavaScript libraries by adding

the available version number and a link for vulnerability checks

on the site-wide admin pages (when this information is available)

- bumped version number to 5.10.1b6

When the dynamic cluster configuration has scale for certain

occasions, it makes sense to provide down scaling support when these

occasions are over, which does not rely on the configured cluster

disconnect timeout (ClusterAutodeleteInterval). The new code will send

automatically a disconnect request when a dynamic cluster node is

terminating gracefully.

Depending on the configuration, a new version of NaviServer will be

necessary to reliably execute disconnect requests. Appropriate changes

are in the NaviServer release/4.99 and main branches.

- Bumped version numbers:

* acs-tcl to 5.10.1b4

* acs-admin to 5.10.1b4

- New kernel parameter "ClusterAutodeleteInterval" to specify, when a

dynamic cluster node is not regarded as temporarily unavailable but

as definitely gone. The default value for this parameter is 2m (2

minutes).

- small refactoring to reduce duplicated logic

- Bumped version numbers:

* acs-kernel to 5.10.1b4

* acs-tcl to 5.10.1b3

* acs-admin to 5.10.1b3

- Improved visualization of cluster nodes that we lost contact to

- New kernel parameters to reduce hard-coded values and to make purpose more explicit

* new parameter ClusterHeartbeatInterval (default 20s),

was hardcoded before to the new default value

* renamed PreferredLocationRegexp -> ClusterPreferredLocationRegexp

* renamed EnableLoggingP -> ClusterEnableLoggingP

- Bumped version numbers:

* acs-kernel to 5.10.1b3

* acs-tcl to 5.10.1b2

* acs-admin to 5.10.1b2

acs-kernel acs-admin acs-tcl/

- logout from external identity provider, if logged in via it

- extend default login page via ADP include, when external

identity providers are configured.

Use the external identity provider for refresh of logins. When a user

is logged in via an external identity provider, use the same identity

provider for a refresh when it expires. The expiration time is

controlled via the classical OpenACS parameters.

Note that in general, the same user might be authenticated via a

classical OpenACS authority (e.g. local authority) and/or via an

external one (e.g. Microsoft Identity Platform (Azure) or GitHub).

For single-sign-ons, when the token is still valid, the redirect to

the external identity provider does not mean necessarily that the use

is shown the external identity provider's login page.

- When login happened via external registry, the logout should happen

there as well.

- let "sec_login_read_cookie" return a dict instead of a list

(eases future extension)

- bump version number to 5.10.1d31

- Support for dynamic cluster nodes:

Previous versions of OpenACS required to know the IP addresses of

the cluster nodes in advance, which is a show-stopper for many cloud

applications.

The new functionality allows arbitrary nodes to register as client

nodes at the canonical server, provided that these know a shared

secret. All messages of the intra-cluster talk are now

cryptographically signed using this shared secret.

In the current version, the shared secret key has to be specified in

the NaviServer configuration file (ClusterSecret). Later versions

will support the use of other measures such as generated keys, kept

as files.

Also in the new version it is still possible to use static peer

addresses (which do not have to register during startup).

- Support for multiple protocols for intra-cluster talk

Previous version of OpenACS required the communication of the

intra-cluster talk over HTTP. With these changes, multiple protocols

are supported. Most prominently HTTPS can be used, which is required

by some organization for all web communication, especially for cloud

applications. Other protocols might be added later (e.g. UDP).

- Nodes (such as the canonical server) can be specified via URL

locations. These location specifier have the following advantages

over classical IP address and port.

* support different protocols (http, https, udp)

* support of IP addresses or DNS names

* when DNS names map to multiple IP addresses

(e.g. multiple IPv4 and/or multiple IPv6 addresses)

the new parameter PreferredLocationRegexp can be use

to select the wanted. When this regexp does not match

any of the addresses, the first address is used.

* ports are now optional (except UDP)

* for location specifier: https://openacs.org

- Support for cluster communication statistics

(requires xotcl-request-monitor)

These statistics show the amount of intra-server

communication. The following statistics are collected

per node:

- number of broadcast operations

- number of send operations (active flushes)

- number of receive operation (passive flushes)

For N cluster nodes, the send operations are

basically the number of broadcast operations times

the number of nodes minus one.

These statistics can also be used prior to cluster

enabling, to obtain the number of potential broadcast

operations.

Kernel Parameters

CanonicalServer : location of the canonical server

ClusterAuthorizedIP : obsolete

ClusterEnabledP : unchanged

ClusterPeerIP : obsolete

EnableLoggingP : unchanged

PreferredLocationRegexp : new

DynamicClusterPeers : new, no not edit

Differences between old and new value for "CanonicalServer":

old:

- specify IP address with port

- example: 137.208.116.31:443

new:

- specify URL location

- support different protocols (http, https, udp)

- support of IP address or domain name

- when domain name maps to different IP addresses

(e.g. multiple IPv4 and/or multiple IPv6 addresses)

PreferredLocationRegexp can be use to select the wanted.

When the provided regexp does not match any of

the addresses, take the first address.

- port is optional

- example: https://openacs.org

Sample setup

Old:

CanonicalServer : 137.208.116.31:443

ClusterAuthorizedIP : 137.208.116.31

ClusterPeerIP : 137.208.116.31:443 137.208.116.31:8443

ClusterEnabledP : 1

EnableLoggingP : 1

New:

CanonicalServer : https://openacs.org

PreferredLocationRegexp : https://

ClusterEnabledP : 1

EnableLoggingP : 1

#

# Cluster secret for intra-cluster communications in NaviServer

# configuration file. Clustering will not be enabled if no value is

# provided.

#

ns_section ns/server/$server/acs {

# ...

ns_param ClusterSecret "please change me"

}

#-----------------------------------------------------------------------

# Support for munin statistics

#

# 1) install munin plugins for naviserver (munin-plugins-ns.git)

# 2) add link to the munin plugins (replace "openacs" by the name

# of your server in the munin configuration)

# /etc/munin/plugins/naviserver_openacs_count_cluster -> /usr/share/munin/plugins/naviserver_count

# 3) in the plugin plugin configuration (e.g., etc/munin/plugin-conf.d/naviserver)

# add a section like the following (again, replace "openacs" by the server name you used)

#

# [naviserver_openacs_count_cluster]

# env.title Cluster

# env.vars cluster:broadcast cluster:sent cluster:received

#

# 4) restart munin

See https://openacs.org/forums/message-view?message_id=5672076

This parameter can be used to adjust the time threshold for longdb

warnings in the log file. When SQL logging is turned on, it might

also adjust the threshold, unless it is already more sensitive.

Bump version number to 5.10.1d25.

Since "ns_mktemp" is deprecated (on the C level) and is prone

to vulnerabilities. This effects as well "ad_tmpnam" in OpenACS,

which uses "ns_mktemp".

Newer C-compilers complain about this more loudly:

Due to security concerns inherent in the design of mktemp(3),

it is highly recommended that you use mkstemp(3) instead.

The security concern is that when ns_mktemp() is used to generate a

(unique) file name, which is used for opening a file, an attacker can

intercept the running binary and sneak in a different file. Although

ns_mktemp() guarantees to return a unique file name, there is no

mechanism to prevent another process or an attacker from creating a

file with the same name before the application attempts to open it.

The problem with using mkstemp() instead is that it has different

semantics, since it returns the open file. So one cannot blindly

replace these calls, but it requires some refactoring. Unfortunately,

this also effects application code, since NaviServer offers

"ns_mktemp" on the Tcl level.

To make it short: one has to separate out different use_cases of

"ad_tmpnam":

(a) use it to obtain a name for creating a file, which is subsequently opened

(b) use it to obtain a name for creating a directory

(c) use it as a name, providing name as a unique name to some external programs.

Case (a) is similar to the "mkstemp(3)" recommendation above. For this

usage scenario, the call "file tmpfile..." in Tcl 8.6 can be used (but

it should also respect the configured tmp directory. This function

is also very similar to "ns_opentempdir" in NaviServer, which uses

as well "file tmpfile". Therefore, we have created a new API call

"ad_opentmpdir ..." which respects the OpenACS settings.

Case (b) can be addressed by "file tempdir" in Tcl 8.7, or by a function

in tcllib. The new API function "ad_mktmpdir" provides respects the

OpenACS settings, and works for Tcl 8.6 or newer.

Case (c) is somewhat different, since it just wants to create a unique name. This case has not received a special API so far

Check content of the string to identify potentially unsafe content

in the provided string. The content is unsafe, when it contains

externally provided content, which might be provided e.g. via

query variables, or via user values stored in the database. When

such content contains square braces, a "subst" command on

theses can evaluate arbitrary commands, which is dangerous.

The new API call is used in "::xo::Package->return_page", where the

"subst" command stripped from its command substitution capabilities.

In case, command subsitution is needed, perform this prior this call.

bumped acs-tcl to 5.10.1d23

bumped xotcl-core to 5.10.1d13

do not allow acs-subsite TmpDir parameter to define where the tmpfolder is located anymore. This MUST be the one configured in the server-wide configuration. Tmpfiles cannot be in a subfolder of the tmpfolder, they MUST be direct children instead. A tmpfile MUST exist beforehand and be owned, be readable and writable by the user running the nsd process. This complies with the definition of a tmpfile by AolServer/NaviServer when they are created to store content coming from a file upload.