OCEOS/oceos fdir

From wiki
Jump to navigation Jump to search

OCEOS Fault Detection, Isolation and Recovery (FDIR)

FDIR Introduction

Hardware and software faults can occur that affect the behavior of the system. To detect such faults and anomalies, as scheduling proceeds OCEOS automatically checks that its key data areas have not been corrupted and that uses of resources are within their prescribed bounds. Parameters passed to directives are checked for consistency with the values originally declared in the application configuration.
Once detected, OCEOS provides five levels of responses to faults and anomalies:
Level 1: The status code returned by a directive indicates that there was a problem

The status code returned by a directive should always be checked.
This code identifies whether the directive succeeded and if not indicates the reason for the failure.
This is the only response made when the anomaly is an invalid parameter to a directive. It is combined with responses from other levels if a directive fails for some other reason.

Level 2: An appropriate log entry is added to the system log.

The log entry contains a code identifying the problem and a 32-bit timestamp.
This response occurs in addition to the Level 1 response if a directive fails due to an internal factor such as the maximum number of jobs for a task being already created or a data queue being full.
This response also occurs when a task misses its deadline or any unexpected behavior. This response is often accompanied by a Level 3 response.

Level 3: The system state variable is updated and a user defined problem handling function called.

The system state variable consists of 32 flags each indicating a particular problem type. It is reset by oceos_init(). A copy of this variable is also maintained. Both variables may be reset by the ASW. This response occurs also in most circumstances where the Level 2 response occurs.

Level 4: An ASW problem handling function is called by OCEOS.

A problem handling function can be identified in the application configuration structure passed to oceos_init() and if present may be called by OCEOS. This function can read the system log and system state variable, read the task timing and other information, enable and disable tasks, reset counting semaphores and data queues, and if necessary exit OCEOS. It can reset the system state variable, it is recommended that the copy of the system state variable not be reset so as to provide a longer term record of problems.
This response occurs also in most circumstances where the Level 3 response occurs

Level 5: OCEOS exits and returns to the ASW with an appropriate status code.

The status code in effect provides a Level 1 response to the oceos_start() directive used by the ASW to start OCEOS. In normal circumstances OCEOS never returns to the ASW.

This response by OCEOS only occurs when it detects that one of its essential internal elements has been corrupted. Depending on the severity of this corruption a Level 2 response may also be provided if possible, and the system state variable may also be updated. The user defined problem handling function is not called.

If OCEOS exits the ASW can inspect the system log and system status variable, and also the task timings and other information maintained by OCEOS. The ASW can decide what corrective action should be taken and may resume scheduling with a call to oceos_start().

The ASW can also inspect these items during scheduling, allowing it check the state of the system at any time,


This approach provides a graduated response by OCEOS to problems and allows some problems be isolated and recovered by, for example, disabling tasks or clearing a data queue if an appropriate problem handling function is provided as part of the ASW. The ASW may also check system information at any time to determine whether things are proceeding as expected.


Note: The ASW may set up a watchdog timer which will cause a system reset if a certain time elapses before it is reset. By using a lowest priority task to reset the watchdog failure of higher priority tasks to complete within their expected time can be detected. In OCEOS the lowest priority task, if it is the only task at that priority, is allowed to run forever and may be used to reset the watchdog. The watchdog timer is not itself part of OCEOS.

FDIR Configuration

API Functions

API Functions
Directive Description main task IRQ handler
oceos_log_add_entry() Add a log entry * * *
oceos_log_remove_entry() Read and remove the oldest unread log entry * * *
oceos_log_get_indexed_entry() Read the log entry at the given index * * *
oceos_log_reset() Clear all log entries and reset to empty * * *
oceos_log_get_size() Get the number of log entries * * *