Skip to content

Services

Functionalities under services are a bunch of standalone features, which are not alarms, metrics and notifications, but aiding devops daily tasks. These features turn WombatOAM to be an active agent. That means users can retrieve live information about running systems (such as the state of a gen_server), recover from outages for instance by forcing GC to save more memory, or fix misconfiguration issues by changing configuration parameters.

The services provided fall into 3 categories:

  • configurator: dealing with configuration management,
  • explorer: inspecting the subject on a read-only way,
  • executor: executed for their side-effect.

The scope of each service is either one node or one node family. Each service has zero or more arguments. The started instance of a service is called a request.

The provided services include

  • Configuration management.
  • Etop like process listing.
  • Inspecting processes.
  • Evaluating Erlang/Elixir expressions.
  • Soft purge modules.
  • Garbage Collect Processes.

There are some boolean properties of services worth mentioning:

  • Exclusiveness: if a service is exclusive only one request of the same service can be run on the same node or node family at the same time. This is because the request either uses some global resource or to prevent multiple instances using too much resources that would have impact on the managed node.
  • Periodicity: most services are executed only once with their result shown as output or executed for their side-effects. Periodic services, however, keep running until they are stopped regularly updating their output data (for example Processes list). A Running request can be stopped explicitly by the user or it also terminates if it is left unattended (i.e. no browser is subscribed to display its output). This inactivity timeout is one minute.

Configurator

Tailoring an application to fit a certain system is achieved by configuring the application. An application can be configured using its configuration parameters, which are often referred to as environmental variables or simply envs. A configuration parameter for an arbitrary application is a pair consisting of an identifier and a value. As applications can always retrieve the current value of a configuration parameter during runtime, users can change the system's behaviour immediately - without the need of stopping or rebuilding the system.

There are two kinds of configuration parameters, namely, a configuration parameter can be local or global. Local configuration parameters are specific to Erlang nodes, their values can vary among the nodes. The data's root directory of an Erlang node, the HTTP port on which the Erlang node listens are good examples for local configuration parameters. In contrary, global configuration parameters must have the same value set on all the nodes belonging to the same node family as they describe the property of the overall system. As examples, consider the IP address of the load balancer or the lifetime of sessions.

Services provided on node level

Services provided on node level are special WombatOAM plugins running on the nodes. They are super simple. Both their authority and their knowledge about its managed node are limited and restricted. They encapsulate the domain-specific knowledge they have and provide simple, well-defined services.

The Application tab in the Topology page provides the user with information about the loaded application, which includes the application's configuration parameters, the user can select any application and with the help of the provided service, to set the value of the parameter.

  • Set an application's configuration parameters. It changes the values of the given configs at once, Application should be a loaded Erlang application, all Keys should be atoms and all Values syntactically valid Erlang terms. Users may also want to store the new parameter permanently in the configuration file that the Erlang VM has been booted with, a dropdown menu allows the user to choose whether to persist the parameter or not.

By the above service, users can inspect and manage the current configuration of any application loaded to their managed nodes.

The plugin also detects some possible misconfigurations and raises an alarm for them:

  • A configuration file can't be read/parsed or contains multiple configurations for a single application (corrupt_config_file alarm).
  • A configuration file has changed - maybe the application should reload it (config_file_changed alarm).
  • A configuration parameter only exists in the configuration file, not in the application environment - maybe the configuration file should be reloaded ({config_only_in_file, <app-name>, <param-name>} alarm)
  • A configuration parameter has different values in the configuration file than in the application environment - maybe the configuration file should be reloaded ({config_difference, <app-name>, <param-name>} alarm, the alarm info contains the difference).

Services provided on node family level

Until that point configuration management was examined from the node's perspective. However, commands including configuration changes make sense on node family level. Performing a command on a node family usually means doing the same thing on all the nodes. Services are related to the global configurations of the nodes belonging to a certain node family. Facts:

  • Configuration parameters which are global configurations shall share the same value on all nodes.

  • If a certain global config has different values among the nodes belonging to the same node family then something went wrong, it is an anomaly.

  • If a certain global config is not set on some nodes of the node family then something went wrong, it is an anomaly.

  • Generally, the values of a global configuration should only be changed when the change affects all the nodes to keep the consistency of the system. However, node families having some nodes being down at the moment do not satisfy the previous condition. Despite of that, users may still want to change the global configuration related to such node families. Thus, the service is available with a clear notice on its current scope.

  • Global configurations are the subset of the local configurations. That means changing a global configuration only on a certain node (i.e. locally) is still allowed. However, if it happens, it is still an anomaly. (It is sometimes useful to try the new value only on one node before changing the value globally.)

WombatOAM maintains the set of the global configurations' keys and their values. This set is empty by default. A configuration becomes a global configuration if and only if the user requests it.

WombatOAM periodically scans for anomalies. An alarm will be raised whenever different values are set to any global configuration or whenever any global configuration is missing from any node of the family. These alarms will be cleared when the values of all the global configurations are the same and also set on all the nodes.

WombatOAM coordinates changing the values of global configurations on all the nodes. Requested changes will be carried out only on those nodes that have been listed in the description of the requested service.

Changing the value of a global configuration is a process that is carried out on the nodes sequentially. The process is aborted by any unexpected failure. While it is in progress there is an anomaly. Some of the nodes have already been updated and are using the new value, while the others are still using the old value. The anomaly can be tolerated, but only temporarily, until the end of the process. If the process managed to change the value on all the nodes, the anomaly will disappear, otherwise the anomaly remains.

WombatOAM provides the following services:

  • Set application parameter as global. Set the given configuration parameter, which is identified by Key and Application, as global configuration. The consistency of the config's values will be checked periodically among the nodes of the node family.

  • Set application parameter as local. Revert the given configuration parameter, which is identified by Key and Application, to be local again and therefore to be excluded from further inconsistency checks and global configs' changes.

  • List globals. List global configs of Application together with their values. The values of the global configs can be read from cache or can be retrieved from nodes.

  • Change global configs' values. Change the values of the given global configs on the nodes sequentially. The first unexpected failure aborts the whole process. The new value can be persisted in the configuration file of each node belonging to the family.

Explorer

Retrieve either static or periodically updated information about a certain process or about the VM. Has none or only a very small side-effect on the managed node.

The following services are available:

  • Process Manager. Presents information about Erlang processes similar to the information presented by "top" in UNIX. See details in the Process Manager section.

  • Process info. Shows the process information of the process given by its pid or registere name.

  • Process messages. Shows the message queue of the process given by its pid or registere name.

  • Process stack trace. Shows the stack trace of the process given by its pid or registere name.

  • Process dictionary. Shows the dictionary of the process given by its pid or registere name.

  • Process state. Shows the state (if available) of the process given by its pid or registere name.

  • Table Visualizer. Shows information about the ETS tables and their contents. See details in the Table Visualizer section.

Process Manager

For each process the following information is displayed:

  • Pid. The process identifier.

  • Reds. The number of reductions executed by the process.

  • Memory. The size of the process, in bytes, obtained by a call to process_info(Pid, memory).

  • MsgQ. The length of the message queue of the process.

Configuration possibilities are:

  • Refresh interval in seconds. Time interval (in seconds) between each update of the output.

  • Number of listed processes. Number of rows (processes) to display.

  • Order by column. Identifies what information to sort by.

  • Order direction. Identifies which direction to sort by (Ascending/Descending).

Various information can be retrieved by clicking on any Pid, such as:

  • Process info.

  • Process messages.

  • Process stack trace.

  • Process dictionary.

  • Process state.

Table Visualizer

For each ETS table the following information is displayed:

  • Id. The identifier of the table.

  • Name. The name of the table.

  • Type. The table type (set, ordered_set, bag or duplicate_bag).

  • Objects. The number of objects inserted in the table.

  • Size (words). The number of words allocated to the table.

  • Owner pid. The pid of the owner of the table.

  • Owner name. The registered name of the owner process of the table.

Configuration possibilities are:

  • Refresh interval in seconds. Time interval (in seconds) between each update of the output.

  • Order by column. Identifies what information to sort by.

  • Order direction. Identifies which direction to sort by (Ascending/Descending).

By clicking on the id of the ETS table, the contents of the table can be retrieved.

This service limits the size of the data that it displays. By default:

  • Only the first 200 ETS tables are shown.
  • Only the first 100 rows are shown in an ETS table.
  • Only the first 10 columns are shown in an ETS table.

These values can be modified in wombat.config:

1
2
3
  {set, wo_plugins, plugins, observer_ets, table_count_limit, 200}.
  {set, wo_plugins, plugins, observer_ets, row_limit, 100}.
  {set, wo_plugins, plugins, observer_ets, col_limit, 10}.

Executor

Real commands (e.g. Garbage Collect Processes, Evaluate Erlang/Elixir expressions). Executed for their side-effects. May not have any meaningful result as output. As these services give much freedom to the WombatOAM users to modify the monitored nodes for safety reasons it might be useful to disable all executors by adding the following line to the wombat.config.

1
{set, wo_services, enable_executor, false}.

The following services are provided:

  • Evaluate Erlang expression. It evaluates the user given Erlang expression on the managed node and returns the result.

  • Soft purge modules. Soft purge all modules from the managed node (see code:soft_purge/1).

  • Garbage Collect Processes . Force garbage collection on all processes in a controlled way. It asks the VM to perform garbage collection on GcGroupSize processes sequentially. Then, it sleeps for GcSleepTime milliseconds to weaken the negative effect of GC on the managed node.

  • Terminate process. Sends an exit signal with the given exit Reason to a process. The process can be given by PID or registered name.

  • Terminate shell processes. Sends an exit signal with the kill reason to all processes currently executing a function from the shell module. This kills all shells and the processes linked to them.

  • Recover from partitioned mnesia. If mnesia running on the current managed node is partitioned, this service restarts mnesia on all other nodes running this distributed mnesia and configures those nodes to load the table data from the current managed node to achieve consistent data in the tables. Warning: if there's different data in the tables in the other nodes (i.e. data written into the tables after mnesia was partitioned), that data is lost. Select carefully the node where you execute this service as the data of that node will be replicated.

The following additional service is provided on Elixir nodes:

  • Evaluate Elixir expression. It evaluates the user given Elixir expression on the managed node and returns the result.