• last updated 16 hours ago
Constraints
Constraints: committers
 
Constraints: files
Constraints: dates
Added code to skip suspicious looking query variables

On openacs.org, we are experiencing numerous requests with

multiply very long and strange query variables like in the example

below. So far, it is not clear, whether these requests are the

consequence of a double encoding or a deliberate attack. Many (most)

of the requests contain the query variable names containing the

(decoded) pattern "*amp;*".

This is a relatively new phenomenon. I cannot exclude that this is a

bug introduced lately in OpenACS, or a bug in an external bot, or

whatever. The problem with these query variables is that OpenACS

propagates these further, e.g., when updating query variables in

ad_dimensional, via export_vars, or return_urls.

Since OpenACS never uses these query-variables, these can be safely

skipped, without loosing functionality in OpenACS. It is possible to

construct examples, where skipping such variables can change the

semantics. Therefore, the change introduces a single function

util::suspicious_query_variable where in case of problems, the

skipping feature can be deactivated.

GET /api-doc/proc-browse?amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3borderby=name&type=All&amp%3btype=All&amp%3bamp%3btype=All&amp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3borderby=name&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=Private&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3borderby=name&amp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3bamp%3btype=All HTTP/1.1" 200 62378 "" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/605.1.15 (KHTML, like Gecko; compatible; FriendlyCrawler/1.0) Chrome/120.0.6099.216 Safari/605.1.15" "1729029614.331581 0.109805 0.000434 0.004026 0.215927

  1. … 6 more files in changeset.
Prefer unset to array unset when the whole array should be deleted

See https://wiki.tcl-lang.org/page/Dict+VS+Array+Speed

  1. … 7 more files in changeset.
merge with missing files

  1. … 1464 more files in changeset.
merge from oacs-5-10

  1. … 8099 more files in changeset.
whitespace changes

  1. … 15 more files in changeset.
Quote error message to better protect against XSS attacks

avoid double quoting

improved spelling

  1. … 10 more files in changeset.
Added support for form_parameter specs with value checkers

improve spelling

  1. … 8 more files in changeset.
Added support for passing parameter_name:value_constraint to xowiki::Package->get_parameter

- The get_parameter method can get values from query-parameters, therefore

we have to validate these.

- Use the new feature at several places (especially for boolean values)

- Still, more places should be checked

- bumped xowiki to 5.10.1d37

- bumped xotcl-core to 5.10.1d14

  1. … 10 more files in changeset.
reduce verbosity

Fixed bug with value-constraints query parameters

Previous versions had a problem with calls like

:query_parameter name:SOMECONSTRAINT

since the implementation clobbered the name variable. Versions of NSF

later than 2022-01-21 support "nsf::parseargs" with the "-asdict"

option, which can use all the nsf::parseargs options and avoids

clobbering.

Added tests of "xo::cc query_parameters" to the regression test suite.

  1. … 1 more file in changeset.
Require form_parameter array consistently in the commands that need it

whitespace changes

reduce verbosity

fix messed up commit

It seems that param has the minus, while declared_parameters do not

Fixes downstream pipeline

Export by default only the declared query variables

Varibles are declared via package specification or in the includelet

definition. Previously, all query variables where exported.

Make creation of ::xo::cc more robust

In case the argument processing of

ConnectionContext create ::xo::cc -p1 ... -p2 ....

::xo::cc destroy_on_cleanup

runs into an exception leading to an ad_script_abort, the half-in

initialized object without cleanup definition will survive. At later

times, this object might be reused, containing potentially

cached results from earlier runs, which can cause troubles in error

cases.

This change takes care that errors during argument processing will

not leave half-initialized objects behind.

improve documentation and streamline code

  1. … 2 more files in changeset.
Modernize parameter parsing (query and includelet paramters)

- handle invalid UTF-8 exception that might be triggered by

"ns_parsequery" in newer versions of NaviServer

- the new code is about 4 times faster than the old one

- replaced array by dict

- don't create helper procs on the fly but use "nsf::parseargs" if possible

- added icanuse rules for

* "nsf::parseargs -asdict"

* "ns_parsequery -charset"

  1. … 1 more file in changeset.
fix dirty buffer of last commit

Added support for multiplicity in value constraints of "query_parameter"

Calls like e.g.

set id [:query_parameter some_id:int32]

are actually equivalent to

set id [:query_parameter some_id:int32 ""]

and accept therefore as result also an empty value. By being able to

specify an explicit multiplicity, we can force non-empty values:

set id [:query_parameter some_id:int32,1..1]

This means effectively that the default multiplicity is "0..1".

avoid subst in export_vars by using xo::update_query

  1. … 1 more file in changeset.
Use built-in "ns_parsequery" instead of manual parsing of query parameters.

Note, that this change might alter query processing, since previously

query parameter without explicit values were treated as boolean values

(like in HTML), now these are treated like "x=" before. The new version is

about 10x faster.

Furthermore, new tests of the regression tests were added for query parameter

processing.

  1. … 1 more file in changeset.
improve spelling

  1. … 1 more file in changeset.
improve comments

  1. … 1 more file in changeset.
replace array by dict, provide error message, when someone tries to setup a connection context without a proper url

Provide a fallback empty URL when one tries to require the connection context without one and outside of a connection