HTTP/HTTPS server
In addition to local event queues, the event manager allows events to be forwarded to an HTTP/HTTPS server using POST requests with content type application/json. The content of each request consists of a single JSON array, with each element representing an event. Each event consists of a two-element array. The first element is an UUID identifying the event type. The second element is a JSON object containing the actual event data (see §Statistic Event Manager).
Configuration of the HTTP (client) event queue is primarily done using the following application properties:
-
egr.eai.fw.statistics.http_client.url– The http or https URL of the server; There is no default value. If the property is not defined, the HTTP event queue is not active. -
egr.eai.fw.statistics.http_client.certificate_validation– In case of an HTTPS server using a self-signed certificate, the property may be set tofalseto suppress certificate validation on the client side. The default value istrue.
If an HTTP (client) event queue has been configured, posted events are serialized into a message object. The message object will eventually be handed over to a separate upload thread using a queue of message objects. The upload thread sends the message to the server, using it as the content (entity body) of a POST request.
The message currently being constructed from newly posted events is completed and handed over to the upload thread if the age of the first event in the message, or the total size of the message after serialization of an event (or a group of events that have been posted together), exceeds the following configurable values:
-
egr.eai.fw.statistics.http_client.max_event_delay– the maximum time an event spends in a partially constructed message before the message is completed and handed over to the upload thread; The value must be specified as a number of seconds or as an ISO-8601 duration (limited to D, H, M, and S components). The default value is one hour. -
egr.eai.fw.statistics.http_client.target_message_size– The maximum message size limit in bytes. The default value is0x10000(64 KB). Note that this is not a hard limit (see above). The maximum allowed value is 1 GByte.
If this property is set to zero, each message consists of a single event (or of a group of events that have been posted together), and there is no delay between the serialization of events to the message and the handing over of the message to the upload thread.
The upload thread takes messages from the message queue fed by the event manager. The message queue may contain multiple messages if they are generated at a faster rate than they can be uploaded to the server. To avoid excessive growth of the message queues, the maximum size of the message queue can be configured with
egr.eai.fw.statistics.http_client.max_queue_size– the maximum size of the message queue in bytes. The default value is0x100000(1 MB).
If the event manager completes a message and attempts to feed it to the message queue, but doing so would result in the total size of all messages in the queue to exceed this value, the message is rejected by the queue. The event manager writes an INFO log message and discards the message.
Once the upload thread has taken a message from the message queue, it compares the age of the message with the value of property
egr.eai.fw.statistics.http_client.max_message_age– the maximum age of a message before it is discarded by the upload thread. The value must be specified as a number of seconds or as an ISO-8601 duration (limited to D, H, M, and S components). The default value is two hours.
The upload thread writes an INFO log message and discards a message taken from the message queue if the age of the message (the time elapsed since the posting of the first event in the message) exceeds the value configured by this property.
If the message is not expired, the upload thread establishes a connection to the server and constructs a request with POST as its request method. The request property Content-Type is set to application/json. The following application property can be used to define additional request properties:
egr.eai.fw.statistics.http_client.extra_request_properties– one additional request property to be sent to the server. The property value must conform to the production field-line of RFC 9112.
This application property may be used repeatedly to specify multiple request properties.
The request is sent to the server, using the message as the request’s content. The server should respond with a 200 (OK) status code or, preferably, with a 204 (No Content) status code (as any content sent as part of the response is discarded anyway).
If there is an IO error sending the request or receiving the response, or if the response contains a status code which does not indicate success, the upload thread either discards the message or puts it back at the head of the message queue. In general, the message is discarded unless
-
it can be said with absolute certainty that the server hasn’t received the complete message, or
-
the server responded with status code 404 (Not Found), 408 (Request Timeout), or 503 (Service Unavailable).
In either case (discard or put back), the upload thread pauses further processing for a configurable amount of time:
egr.eai.fw.statistics.http_client.error_retry_delay– the time to wait after a failed upload attempt encoded as a number of seconds or an ISO-8601 duration (limited to D, H, M, and S components); The default value is ten minutes.
While (mostly) sleeping for the configured amount time, the upload thread still removes expired messages (messages whose age is greater than the maximum message age) from the message queue so the event manager doesn’t have to discard messages it wants to hand over to the upload thread just because the message queue has filled up with expired messages.
-
egr.eai.fw.statistics.http_client.encoding– the method used by the HTTP client event queue to encode statistics event sent to the server; possible values are:-
json– data is sent as a single JSON array, with each element representing one event (the default) -
json-lines– data is sent using JSON Lines encoding, with each line containing one event -
cbor– data is sent as a single CBOR array, with each element representing one event -
cbor-records– data is sent using a proprietary binary record format, with each record containing one CBOR-encoded event
-
Depending on the encoding, the Content-Type of the POST request is set to either application/json, application/jsonl, application/cbor, or application/x.egr-cbor-records.
Use of JSON lines and CBOR records may be necessary (depending on the implementation) for the server to recover from decoding errors, as may happen (again depending on the implementation) in case of unknown event types.
Use of CBOR instead of JSON reduces the size of serialized data by about factor two. Most notably, this affects the amount of data which can be buffered by the client (EAIWS) in case of temporary unavailability of the server (assuming equal maximum queue size). It also reduces the frequency of POST requests issued by the client and/or content size of request messages (assuming equal maximum message size and event delay).
Compression allows more events to be buffered on the client (EAIWS) side in case of temporary server failure. If compression is enabled, the content of HTTP POST messages used to transfer events to the server is also compressed.
-
egr.eai.fw.statistics.http_client.compression– specifies the compression method:-
none– no compression (the default) -
deflate– “zlib” data format (RFC1950) containing a “deflate” compressed data stream (RFC1951) that uses a combination of the Lempel-Ziv (LZ77) compression algorithm and Huffman coding; The Content-Encoding header field of the POST request is set to deflate. -
gzip– an LZ77 coding with a 32-bit CRC that is commonly produced by the gzip file compression program (RFC1952). The Content-Encoding header field of the POST request is set to gzip.
-
-
egr.eai.fw.statistics.http_client.compression_level– the compression level used in case compression is enabled; The compression level must be a value between 1 and 9, both inclusive. The default value is 6.The configured compression level is ignored if compression is not enabled.
Given the difficulties to determine the final compressed message size while compression is still in progress (a considerable amount of the data about to be compressed is still buffered in the compressor), compressed messages use a different algorithm to compute the message size which compared against the maximum message size (which actually behaves as a target message size) as configured with application property egr.eai.fw.statistics.http_client.max_message_size. In case of compressed messages, the current estimated message size is computed as the larger of the number of bytes of compressed data produced so far, and the number of bytes of uncompressed data divided by the weighted average of recent compression rates.
Enabling of compression has no immediate effect on the performance of operations emitting statistics events because serialization of these events is done by a separate thread.