3.6 Error Message Handling (EMU)

The Error Message Utility (EMU) [125] is a CERN product which provides a unified system for handling alarm, error, warning, and informational messages from the Slow Controls and Data Acquisition systems. Messages, which can be injected anywhere on the network, are formatted by the EMU system according to a message description file. They are then sent to one of a number of logfiles or destination processes according to a message routing file.

Application programs inject messages into EMU using a short message name (i.e. set_error or clr_error) and usually some parameters (e.g. the channel name).

The EMU system consists of three processes, which in DELPHI all run on the central Slow Controls workstation, though in general they could each have many instances on different machines. The EMU DECnet server acts as an RPC server for the application programs, simply sending the messages without change via a mailbox to the EMU Decoder.

The Decoder attaches a description and properties to the message according to those listed in the message file for the given message name. The description will be used in the logfile and EMU display to clarify the message. The parameters sent by the application program are inserted at appropriate points to make an `English' sentence. The properties are used for routing the message, and for selections by the EMU display (see section 3.6.2).

The Decoder passes the message on to the EMU Router, which decides where to send the message: into one of a number of logfiles, or on to an application process. The routing can be based upon the name, properties (as attached by the decoder), or source of the message, as specified by logical expressions given in the routing file for each destination.

3.6.1 Use of EMU in DELPHI

Each Elementary Process or Data Acquisition program injects messages into EMU. The standard EPs use a limited set of message names (and hence they all can be associated with a single message description file). The message parameters are used to send specific information such as the channel name or the newly-read value.

In order to allow the EMU display to match an error message with its cancellation, an additional convention is observed [126]. The message names start with set (for raising) or clr (for cancelling messages), and significant text (such as the channel name, which clearly must be the same if the two messages are to refer to the same condition) is enclosed in square brackets. If these, the injecting program, and some additional properties match, then the EMU display is able to remove the message from the list of outstanding errors when the cancellation message is received.

Each detector partition (or other system, such as the central gas system) has one EMU logfile, which is normally only of interest to detector experts. All messages from that partition, or messages relevant to that partition from the gas or safety system, are sent to this logfile. Warning, error, and alarm messages are also sent into central logfiles (one for the Slow Controls and one for the Data Acquisition), which are watched by dedicated EMU displays on the operators' workstations. New versions of the logfiles are created every midnight, while the old versions are kept available for inspection by detector experts.

EMU messages corresponding to conditions that require rapid intervention from a detector expert are routed to a `beep-caller' program [127]. This uses the auto-dial facility of a modem to dial the telephone number that activates the pager carried by the appropriate detector expert.

3.6.2 EMU Display

The EMU Display [128] is a general configurable utility for showing outstanding conditions reported by EMU. It is based on the X-Windows/Motif graphical user interface (figure 3.4),

Figure 3.4: A typical screen from the EMU Display. On the left are the controls, with a button to select each partition's messages. Those that have unseen messages are coloured according to the highest severity of message. Above the controls are the messages for the Outer Detector (selected by clicking on the OD button). These show a number of trips of CAEN high voltage channels.
though a VT100-style terminal interface based on MHI is also provided.

The program watches for updates that EMU makes to a logfile (signalled by the VMS file system) and displays them according to category (e.g. by detector partition or severity), which may be selected for display separately. If not currently displayed, a new message is indicated by a colour change (according to its severity) on the button corresponding to its category. Cancelled messages are removed from the display, though they remain in the logfile. A logfile browsing facility is also provided.

In normal operation EMU displays are used by both Data Acquisition and Slow Controls operators to monitor warning, error, and alarm conditions.

The program is highly configurable (both by configuration file and interactively), allowing it also to be useful for detector experts to monitor or browse the partition-specific logfiles.

Tim Adye 2002-11-06