• last updated 1 hour ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Use for new installation relative path names for external programs per default

Added support for automatic disconnect when a dynamic cluster node is shut down

When the dynamic cluster configuration has scale for certain

occasions, it makes sense to provide down scaling support when these

occasions are over, which does not rely on the configured cluster

disconnect timeout (ClusterAutodeleteInterval). The new code will send

automatically a disconnect request when a dynamic cluster node is

terminating gracefully.

Depending on the configuration, a new version of NaviServer will be

necessary to reliably execute disconnect requests. Appropriate changes

are in the NaviServer release/4.99 and main branches.

- Bumped version numbers:

* acs-tcl to 5.10.1b4

* acs-admin to 5.10.1b4

  1. … 4 more files in changeset.
Adding auto-deletion of dynamic cluster nodes and small refactoring

- New kernel parameter "ClusterAutodeleteInterval" to specify, when a

dynamic cluster node is not regarded as temporarily unavailable but

as definitely gone. The default value for this parameter is 2m (2

minutes).

- small refactoring to reduce duplicated logic

- Bumped version numbers:

* acs-kernel to 5.10.1b4

* acs-tcl to 5.10.1b3

* acs-admin to 5.10.1b3

  1. … 5 more files in changeset.
Small update for cluster support

- Improved visualization of cluster nodes that we lost contact to

- New kernel parameters to reduce hard-coded values and to make purpose more explicit

* new parameter ClusterHeartbeatInterval (default 20s),

was hardcoded before to the new default value

* renamed PreferredLocationRegexp -> ClusterPreferredLocationRegexp

* renamed EnableLoggingP -> ClusterEnableLoggingP

- Bumped version numbers:

* acs-kernel to 5.10.1b3

* acs-tcl to 5.10.1b2

* acs-admin to 5.10.1b2

acs-kernel acs-admin acs-tcl/

  1. … 7 more files in changeset.
bumped version numbers to 5.10.1b1

  1. … 85 more files in changeset.
External identity provider reform (part 3)

- logout from external identity provider, if logged in via it

- extend default login page via ADP include, when external

identity providers are configured.

  1. … 5 more files in changeset.
External identity provider reform (part 2)

Use the external identity provider for refresh of logins. When a user

is logged in via an external identity provider, use the same identity

provider for a refresh when it expires. The expiration time is

controlled via the classical OpenACS parameters.

Note that in general, the same user might be authenticated via a

classical OpenACS authority (e.g. local authority) and/or via an

external one (e.g. Microsoft Identity Platform (Azure) or GitHub).

For single-sign-ons, when the token is still valid, the redirect to

the external identity provider does not mean necessarily that the use

is shown the external identity provider's login page.

  1. … 3 more files in changeset.
Added preliminary support for secondary registries (e.g., MS Azure via oauth2)

- When login happened via external registry, the logout should happen

there as well.

- let "sec_login_read_cookie" return a dict instead of a list

(eases future extension)

- bump version number to 5.10.1d31

  1. … 1 more file in changeset.
bump version number to 5.10.1d30

Added proc "membership_rel::expire" so that all membership states are now covered. Note: The Oracle part is best effort only!

  1. … 10 more files in changeset.
cluster reform: added admin page and minor updates

  1. … 5 more files in changeset.
Cluster reform

- Support for dynamic cluster nodes:

Previous versions of OpenACS required to know the IP addresses of

the cluster nodes in advance, which is a show-stopper for many cloud

applications.

The new functionality allows arbitrary nodes to register as client

nodes at the canonical server, provided that these know a shared

secret. All messages of the intra-cluster talk are now

cryptographically signed using this shared secret.

In the current version, the shared secret key has to be specified in

the NaviServer configuration file (ClusterSecret). Later versions

will support the use of other measures such as generated keys, kept

as files.

Also in the new version it is still possible to use static peer

addresses (which do not have to register during startup).

- Support for multiple protocols for intra-cluster talk

Previous version of OpenACS required the communication of the

intra-cluster talk over HTTP. With these changes, multiple protocols

are supported. Most prominently HTTPS can be used, which is required

by some organization for all web communication, especially for cloud

applications. Other protocols might be added later (e.g. UDP).

- Nodes (such as the canonical server) can be specified via URL

locations. These location specifier have the following advantages

over classical IP address and port.

* support different protocols (http, https, udp)

* support of IP addresses or DNS names

* when DNS names map to multiple IP addresses

(e.g. multiple IPv4 and/or multiple IPv6 addresses)

the new parameter PreferredLocationRegexp can be use

to select the wanted. When this regexp does not match

any of the addresses, the first address is used.

* ports are now optional (except UDP)

* for location specifier: https://openacs.org

- Support for cluster communication statistics

(requires xotcl-request-monitor)

These statistics show the amount of intra-server

communication. The following statistics are collected

per node:

- number of broadcast operations

- number of send operations (active flushes)

- number of receive operation (passive flushes)

For N cluster nodes, the send operations are

basically the number of broadcast operations times

the number of nodes minus one.

These statistics can also be used prior to cluster

enabling, to obtain the number of potential broadcast

operations.

Kernel Parameters

CanonicalServer : location of the canonical server

ClusterAuthorizedIP : obsolete

ClusterEnabledP : unchanged

ClusterPeerIP : obsolete

EnableLoggingP : unchanged

PreferredLocationRegexp : new

DynamicClusterPeers : new, no not edit

Differences between old and new value for "CanonicalServer":

old:

- specify IP address with port

- example: 137.208.116.31:443

new:

- specify URL location

- support different protocols (http, https, udp)

- support of IP address or domain name

- when domain name maps to different IP addresses

(e.g. multiple IPv4 and/or multiple IPv6 addresses)

PreferredLocationRegexp can be use to select the wanted.

When the provided regexp does not match any of

the addresses, take the first address.

- port is optional

- example: https://openacs.org

Sample setup

Old:

CanonicalServer : 137.208.116.31:443

ClusterAuthorizedIP : 137.208.116.31

ClusterPeerIP : 137.208.116.31:443 137.208.116.31:8443

ClusterEnabledP : 1

EnableLoggingP : 1

New:

CanonicalServer : https://openacs.org

PreferredLocationRegexp : https://

ClusterEnabledP : 1

EnableLoggingP : 1

#

# Cluster secret for intra-cluster communications in NaviServer

# configuration file. Clustering will not be enabled if no value is

# provided.

#

ns_section ns/server/$server/acs {

# ...

ns_param ClusterSecret "please change me"

}

#-----------------------------------------------------------------------

# Support for munin statistics

#

# 1) install munin plugins for naviserver (munin-plugins-ns.git)

# 2) add link to the munin plugins (replace "openacs" by the name

# of your server in the munin configuration)

# /etc/munin/plugins/naviserver_openacs_count_cluster -> /usr/share/munin/plugins/naviserver_count

# 3) in the plugin plugin configuration (e.g., etc/munin/plugin-conf.d/naviserver)

# add a section like the following (again, replace "openacs" by the server name you used)

#

# [naviserver_openacs_count_cluster]

# env.title Cluster

# env.vars cluster:broadcast cluster:sent cluster:received

#

# 4) restart munin

  1. … 4 more files in changeset.
Deprecate acs_privacy:: api and inline the few usages in the dotlrn packages

See https://openacs.org/forums/message-view?message_id=5672076

  1. … 5 more files in changeset.
New package parameter for acs-tcl: DbLogMinDuration

This parameter can be used to adjust the time threshold for longdb

warnings in the log file. When SQL logging is turned on, it might

also adjust the threshold, unless it is already more sensitive.

Bump version number to 5.10.1d25.

  1. … 2 more files in changeset.
New API "ad_mktmpdir" and "ad_opentmpfile"

Since "ns_mktemp" is deprecated (on the C level) and is prone

to vulnerabilities. This effects as well "ad_tmpnam" in OpenACS,

which uses "ns_mktemp".

Newer C-compilers complain about this more loudly:

Due to security concerns inherent in the design of mktemp(3),

it is highly recommended that you use mkstemp(3) instead.

The security concern is that when ns_mktemp() is used to generate a

(unique) file name, which is used for opening a file, an attacker can

intercept the running binary and sneak in a different file. Although

ns_mktemp() guarantees to return a unique file name, there is no

mechanism to prevent another process or an attacker from creating a

file with the same name before the application attempts to open it.

The problem with using mkstemp() instead is that it has different

semantics, since it returns the open file. So one cannot blindly

replace these calls, but it requires some refactoring. Unfortunately,

this also effects application code, since NaviServer offers

"ns_mktemp" on the Tcl level.

To make it short: one has to separate out different use_cases of

"ad_tmpnam":

(a) use it to obtain a name for creating a file, which is subsequently opened

(b) use it to obtain a name for creating a directory

(c) use it as a name, providing name as a unique name to some external programs.

Case (a) is similar to the "mkstemp(3)" recommendation above. For this

usage scenario, the call "file tmpfile..." in Tcl 8.6 can be used (but

it should also respect the configured tmp directory. This function

is also very similar to "ns_opentempdir" in NaviServer, which uses

as well "file tmpfile". Therefore, we have created a new API call

"ad_opentmpdir ..." which respects the OpenACS settings.

Case (b) can be addressed by "file tempdir" in Tcl 8.7, or by a function

in tcllib. The new API function "ad_mktmpdir" provides respects the

OpenACS settings, and works for Tcl 8.6 or newer.

Case (c) is somewhat different, since it just wants to create a unique name. This case has not received a special API so far

  1. … 1 more file in changeset.
new API call util::potentially_unsafe_eval_p

Check content of the string to identify potentially unsafe content

in the provided string. The content is unsafe, when it contains

externally provided content, which might be provided e.g. via

query variables, or via user values stored in the database. When

such content contains square braces, a "subst" command on

theses can evaluate arbitrary commands, which is dangerous.

The new API call is used in "::xo::Package->return_page", where the

"subst" command stripped from its command substitution capabilities.

In case, command subsitution is needed, perform this prior this call.

bumped acs-tcl to 5.10.1d23

bumped xotcl-core to 5.10.1d13

  1. … 3 more files in changeset.
tmpfile page contract filter reform:

do not allow acs-subsite TmpDir parameter to define where the tmpfolder is located anymore. This MUST be the one configured in the server-wide configuration. Tmpfiles cannot be in a subfolder of the tmpfolder, they MUST be direct children instead. A tmpfile MUST exist beforehand and be owned, be readable and writable by the user running the nsd process. This complies with the definition of a tmpfile by AolServer/NaviServer when they are created to store content coming from a file upload.

  1. … 4 more files in changeset.
Add must_exist flag to enforce a safe tmpfile to already exist

  1. … 2 more files in changeset.
Introduce security::safe_tmpfile_p checking whether a file belongs to the configured tmpfolder and respects other constraints

The plan is to use it to improve input validations

  1. … 2 more files in changeset.
Generalized "version_dir" handling a little for download specs

The problem was that bootstrap5 uses a version directory, which

consists of the version plus an extra string element. The previous

version assumed that the version is always used as a directory name.

Whe track now the versionDir information in the resource_info dict and

use this, when available (otherwise the version number is used as before).

The resource_info dict contains now the following path components:

# Provide paths for loading either via /resources/ or CDN

#

# "resourceDir" is the absolute path in the filesystem

# "resourceUrl" is the URL path provided to the request processor

# "versionDir" is the version-specific element both in the

# URL and in the filesystem.

#

bumped acs-tcl to 5.10.1d19

  1. … 2 more files in changeset.
Update italian localization

  1. … 1 more file in changeset.
Add missing message key

  1. … 1 more file in changeset.
Export message keys from the UI to normalize sorting and quoting

  1. … 1 more file in changeset.
Fix array format in message keys

See acs-lang.message__format automated test, where the expected behavior of lang::message::format is documented

  1. … 11 more files in changeset.
bump version number to 5.10.1d14

  1. … 1 more file in changeset.
new API call ::security::csp::add_static_resource_header

The API call set the CSP rule on the current connection for a static

resource depending on the MIME type.

# Sample definition for custom CSP rules for static files in the

# OpenACS configuration file.

#

# ns_section ns/server/$server/acs {

# ...

# ns_param StaticCSP {

# image/svg+xml "script-src 'none'"

# }

# ...

# }

bump version number of acs-tcl to 5.10.1d13

  1. … 3 more files in changeset.
Support specification of allowed tags/attributes/protocols via global package parameters.

This change makes it possible to define for a package values for

AllowedTags, AllowedAttributes, or AllowedProtocols. If (some of)

these exist, use these for configuring "ad_html_security_check". If +

not, fall back to the default (site wide) definition.

This change is fully backward compatible and effects on sites defining

such global parameters. This change was induced by the need of large

sites, where it is not feasible to set these parameters on the package

instance level, since this site has 830000 packages instances.

Bump version number to 5.10.1d19

  1. … 2 more files in changeset.
make sure, the dependencies are aligned (drop of xo::site_node)

  1. … 1 more file in changeset.
Added new interface for calling database functions

This is tested for PostgreSQL and Oracle and the drivers nsdb and nsdbi.

For details, see https://openacs.org/xowiki/calling-database-functions

  1. … 2 more files in changeset.
util::file_content_check: new utility for checking content of a file

Check whether the provided file is of the requested type.

This function is more robust and protable than relying on

external programs and their output, but it does not work on

all possible file types. It checks a few common cases that

could lead to problems otherwise, like when uploading archives.

Bump version to 5.10.1d9

  1. … 2 more files in changeset.