oacs-5-10:gustafn:20230207175031Cluster reform
- Support for dynamic cluster nodes: Previous versions of OpenACS required to know the IP addresses of the cluster nodes in advance, which is a show-stopper for many cloud applications. The new functionality allows arbitrary nodes to register as client nodes at the canonical server, provided that these know a shared secret. All messages of the intra-cluster talk are now cryptographically signed using this shared secret. In the current version, the shared secret key has to be specified in the NaviServer configuration file (ClusterSecret). Later versions will support the use of other measures such as generated keys, kept as files. Also in the new version it is still possible to use static peer addresses (which do not have to register during startup). - Support for multiple protocols for intra-cluster talk Previous version of OpenACS required the communication of the intra-cluster talk over HTTP. With these changes, multiple protocols are supported. Most prominently HTTPS can be used, which is required by some organization for all web communication, especially for cloud applications. Other protocols might be added later (e.g. UDP). - Nodes (such as the canonical server) can be specified via URL locations. These location specifier have the following advantages over classical IP address and port. * support different protocols (http, https, udp) * support of IP addresses or DNS names * when DNS names map to multiple IP addresses (e.g. multiple IPv4 and/or multiple IPv6 addresses) the new parameter PreferredLocationRegexp can be use to select the wanted. When this regexp does not match any of the addresses, the first address is used. * ports are now optional (except UDP) * for location specifier: https://openacs.org - Support for cluster communication statistics (requires xotcl-request-monitor) These statistics show the amount of intra-server communication. The following statistics are collected per node: - number of broadcast operations - number of send operations (active flushes) - number of receive operation (passive flushes) For N cluster nodes, the send operations are basically the number of broadcast operations times the number of nodes minus one. These statistics can also be used prior to cluster enabling, to obtain the number of potential broadcast operations. Kernel Parameters CanonicalServer : location of the canonical server ClusterAuthorizedIP : obsolete ClusterEnabledP : unchanged ClusterPeerIP : obsolete EnableLoggingP : unchanged PreferredLocationRegexp : new DynamicClusterPeers : new, no not edit Differences between old and new value for "CanonicalServer": old: - specify IP address with port - example: 137.208.116.31:443 new: - specify URL location - support different protocols (http, https, udp) - support of IP address or domain name - when domain name maps to different IP addresses (e.g. multiple IPv4 and/or multiple IPv6 addresses) PreferredLocationRegexp can be use to select the wanted. When the provided regexp does not match any of the addresses, take the first address. - port is optional - example: https://openacs.org Sample setup Old: CanonicalServer : 137.208.116.31:443 ClusterAuthorizedIP : 137.208.116.31 ClusterPeerIP : 137.208.116.31:443 137.208.116.31:8443 ClusterEnabledP : 1 EnableLoggingP : 1 New: CanonicalServer : https://openacs.org PreferredLocationRegexp : https:// ClusterEnabledP : 1 EnableLoggingP : 1 # # Cluster secret for intra-cluster communications in NaviServer # configuration file. Clustering will not be enabled if no value is # provided. # ns_section ns/server/$server/acs { # ... ns_param ClusterSecret "please change me" } #----------------------------------------------------------------------- # Support for munin statistics # # 1) install munin plugins for naviserver (munin-plugins-ns.git) # 2) add link to the munin plugins (replace "openacs" by the name # of your server in the munin configuration) # /etc/munin/plugins/naviserver_openacs_count_cluster -> /usr/share/munin/plugins/naviserver_count # 3) in the plugin plugin configuration (e.g., etc/munin/plugin-conf.d/naviserver) # add a section like the following (again, replace "openacs" by the server name you used) # # [naviserver_openacs_count_cluster] # env.title Cluster # env.vars cluster:broadcast cluster:sent cluster:received # # 4) restart munin 07 Feb 23 gustafn oacs-5-10:gustafn:20230207175031 oacs-5-10:gustafn:20221229130248added multiple delivery methods to intra-server talk
Here is some background information for my experiments with the delivery methods. For this experiment, I compared 5 different means for this kind of communications - ns_http over HTTP (the standard setup, which is used in OpenACS 5.10) - ns_http over HTTPS - ns_conn over HTTP using persistent connections - ns_conn over HTTPS using persistent connections - ns_udp using UDP I tested the is in 2-node cluster to make measurements simple consisting of the canonical server and one node listening on the following protocols/ports: - http://127.0.0.1:8101 - https://127.0.0.1:8444 - udp://127.0.0.1:8101 The first test sends per call 1000 intra-server commands from the canonical server to the 2nd node over the various delivery methods: set times 1000 lappend _ ns_http-[time {::acs::CS_127.0.0.1_8101 message set x ns_http} $times] lappend _ ns_https-[time {::acs::CS_127.0.0.1_8444 message set x ns_https} $times] lappend _ ns_connchan-http-[time {::acs::CS_127.0.0.1_8101 message -delivery connchan set x ns_http} $times] lappend _ ns_connchan-https-[time {::acs::CS_127.0.0.1_8444 message -delivery connchan set x ns_https} $times] lappend _ ns_udp-[time {::acs::CS_127.0.0.1_8101 message -delivery udp set x udp} $times] join $_ \n This leads to the following results: ns_http 564.027083 microseconds per iteration ns_https 1483.478916 microseconds per iteration ns_connchan-http 147.688541 microseconds per iteration ns_connchan-https 68.480875 microseconds per iteration ns_udp 198.343416 microseconds per iteration Since the commands are sent in sequence, the variant with the persistent HTTP connection is the fastest, although this is Tcl implemented. The slowest is the version with HTTPS via ns_http without persistent connections. We see a factor of 20 in terms of performance. When using ns_udp with the "-noreply" option, we have would have a "fire and forget" solution, which might be ok when the packet loss rate is low. That would lead to 54 microseconds. Clearly, the numbers for persistent connections look the best, but it has as well some disadvantages compared to other solutions: - the server has to keep a socket open to every node (but no connection thread) - the keepalive setting of the server must set sufficiently long to gain advantage of persistent connections (e.g. 5 sec keepalive, heart beat frequency of 1s) - Since the whole communication goes over a single connection, it is necessary to serialize the requests to avoid that multiple connection threads write concurrently to the same connection and interfere with each other - It is probably necessary to have a separate thread handling the outgoing intra-server talk (implementing cmd queuing, async-handling, heart-beat, etc.). Since this has to be a Tcl-thread it will use up some memory (similar to a connection thread). - This intra-server talk thread requires queuing and event handling we have so far just in xotcl-core, so when implemented, it will require the xotcl-core package (maybe this can be put later to acs-core). As a second experiment, I've implemented a simple heart-beat service inside the request monitor that checks the liveliness of the nodes every second. So, in contrary to the back to back commands of the first experiment, these are single calls. Here are some random values for the 5 delivery methods: [27/Dec/2022:20:29:34.171376][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_http sent total 2.907ms [27/Dec/2022:20:29:34.182241][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x ns_https sent total 10.798ms [27/Dec/2022:20:29:34.183475][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_connchan sent total 1.161m [27/Dec/2022:20:29:34.183657][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x https-connchan sent total 0.086ms [27/Dec/2022:20:29:34.188564][::throttle] Notice: -cluster: udp://127.0.0.1:8101 set x udp sent total 4.861ms [27/Dec/2022:20:30:25.494080][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_http sent total 2.049ms [27/Dec/2022:20:30:25.516306][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x ns_https sent total 21.903ms [27/Dec/2022:20:30:25.517239][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_connchan sent total 0.814ms [27/Dec/2022:20:30:25.522957][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x https-connchan sent total 0.33ms [27/Dec/2022:20:30:25.534274][::throttle] Notice: -cluster: udp://127.0.0.1:8101 set x udp sent total 11.099ms [27/Dec/2022:20:31:54.993455][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_http sent total 2.431ms [27/Dec/2022:20:31:55.003036][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x ns_https sent total 9.499ms [27/Dec/2022:20:31:55.010100][::throttle] Notice: -cluster: http://127.0.0.1:8101 set x ns_connchan sent total 6.981ms [27/Dec/2022:20:31:55.010585][::throttle] Notice: -cluster: https://127.0.0.1:8444 set x https-connchan sent total 0.322ms [27/Dec/2022:20:31:55.017764][::throttle] Notice: -cluster: udp://127.0.0.1:8101 set x udp sent total 7.13ms We see in essence the same pattern. The approach with the persistent connections looks here the best as well. It is not clear to me, why HTTPS over connchan is the best, but the communication seems ok. Maybe some buffering/nagle algorithm is responsible for this. We see as well that the round-trip takes typically single to double-digit milliseconds. So when a single HTTP request to nsd triggers multiple cache-flush operations to multiple nodes, this will take some time. When e.g., the request issues 5 cash-flush operations, which are sent to 5 nodes, and every request with take 1ms, the cache flushing will make the original request about 25ms slower. This might also be an argument for a separate thread doing these operations asynchronously. 29 Dec 22 gustafn oacs-5-10:gustafn:20221229130248 |
Loading ...
|