From 74e058c8a010ef6feb539248a550d0bb169c1e94 Mon Sep 17 00:00:00 2001 From: ivan Date: Mon, 27 Dec 2010 00:04:44 +0000 Subject: import torrus 1.0.9 --- torrus/doc/devdoc/architecture.pod | 511 ++++++++++++++++++++++++++++ torrus/doc/devdoc/devdiscover.pod | 296 ++++++++++++++++ torrus/doc/devdoc/progstyle.pod | 138 ++++++++ torrus/doc/devdoc/reqs.0.0.pod | 166 +++++++++ torrus/doc/devdoc/reqs.0.1.pod | 210 ++++++++++++ torrus/doc/devdoc/torrus_roadmap.pod | 249 ++++++++++++++ torrus/doc/devdoc/wd.distributed.pod | 198 +++++++++++ torrus/doc/devdoc/wd.messaging.pod | 128 +++++++ torrus/doc/devdoc/wd.monitor-escalation.pod | 117 +++++++ torrus/doc/devdoc/wd.uptime-mon.pod | 162 +++++++++ 10 files changed, 2175 insertions(+) create mode 100644 torrus/doc/devdoc/architecture.pod create mode 100644 torrus/doc/devdoc/devdiscover.pod create mode 100644 torrus/doc/devdoc/progstyle.pod create mode 100644 torrus/doc/devdoc/reqs.0.0.pod create mode 100644 torrus/doc/devdoc/reqs.0.1.pod create mode 100644 torrus/doc/devdoc/torrus_roadmap.pod create mode 100644 torrus/doc/devdoc/wd.distributed.pod create mode 100644 torrus/doc/devdoc/wd.messaging.pod create mode 100644 torrus/doc/devdoc/wd.monitor-escalation.pod create mode 100644 torrus/doc/devdoc/wd.uptime-mon.pod (limited to 'torrus/doc/devdoc') diff --git a/torrus/doc/devdoc/architecture.pod b/torrus/doc/devdoc/architecture.pod new file mode 100644 index 000000000..4cf9c9ccb --- /dev/null +++ b/torrus/doc/devdoc/architecture.pod @@ -0,0 +1,511 @@ +# architecture.pod: The Torrus internals +# Copyright (C) 2002-2005 Stanislav Sinyagin +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. + +# $Id: architecture.pod,v 1.1 2010-12-27 00:04:37 ivan Exp $ +# Stanislav Sinyagin +# +# + +=head1 Torrus Framework Architecture + +=head2 Configuration Processing + +The XML configuration is compiled into the database representation by +operator's manual request. + +The compiled version of configuration is not a one-to-one +representation of the XML version. All aliases and templates are +expanded. The backwards restoration of XML from the database +is available with the snapshot utility. + +Aliases are the way to represent the data in a more convenient format. +An alias can point to a subtree or a leaf, and it works similarly as +a symbolic link in a filesystem. + +A template defines a piece of configuration which can be used in +multiple places. Templates can be nested. + +The configuration can consist of several XML files. They will be +processed in the specified order. Each new file is treated as an additive +information to the existing tree. + +The XML configuration compiler validates all the mandatory parameters. + + +=head2 Database Architecture + +All runtime data is stored in +B database environment (http://www.sleepycat.com). + +The compiled version of the configuration XML is stored in +the B and B. +These databases have similar structure, and +B keeps all datasource-related information. +C and C stand for the productive instance number, +and have values of 0 or 1. +Current productive instance numbers are stored in B +database. + +For each datasource tree, the database files are resided in +Ftree_nameE> directory. + +The runtime modules access the configuration via C objects. + +Each datasource subtree or leaf is identified by a I. +A token is a short alphanumeric string, generated internally. +Two types of tokens are recognized: single tokens and token sets. + +Single token starts with letter I. The rest is made with decimal digts. + +Token set name starts with letter I. The rest is an arbitrary sequence of +word characters. + +The special token I is reserved for tokensets list. Also tokenset +parameters are inherited from this token's parameters. + +View and monitor names must be unique, and must +start with a lower case letter. + +B is a I database, with keys of form +Ctree_nameE> or Ctree_nameE>, and 0 or 1 as +values. Also the compiler adds an entry Ctree_nameE> +during the compilation time, in order to avoid two compiler processes +running at the same time on the same tree. + +B is a I database, with the keys and values +defined as follows: + +=over 4 + +=item * tp:EpathE -- EtokenE + +Gives the token correspondig to the given element name. + +=item * pt:EtokenE -- EpathE + +Gives the path name by the given token. + +=item * c:EtokenE -- EctokenE,... + +For given subtree, contains the list of child tokens separated by comma. + +=item * p:EtokenE -- EptokenE + +For given subtree or leaf, contains the parent token. + +=item * P:EtokenE:EpnameE -- EvalueE + +Contains the parameter value for specified leaf or subtree. +Each leaf or subtree inherits parameters from its parent. +Thus, we must climb up the tree in order to get the parameter's +value if not defined locally. + +=item * Pl:EtokenE -- EpnameE,... + +Contains the list of parameter names for specified leaf or subtree. + +=item * a:EtokenE -- EtokenE + +If this subtree or leaf is an alias, specifies the reference to the real node. + +=item * ar:EtokenE -- EtokenE,... + +Specifies all alias subtrees or leaves pointing to this token. + +=item * d:EnameE -- EvalueE + +Definition value for the given name + +=item * D: -- EnameE,EnameE,... + +List of all known definitions + +=item * n:EtokenE -- EtypeE + +Defines a node type. Type is a number with the following values: +0 for subtree, 1 for leaf, 2 for alias. + +=back + +B is a I database, with the keys and values +defined as follows: + +=over 4 + +=item * ConfigurationReady -- 1:0 + +When nonzero, the configuration is ready for usage. +Otherwise, the configuration status is undefined. + +=item * P:EnameE:EpnameE -- EvalueE + +Contains the parameter value for specified view, monitor or action. + +=item * Pl:EnameE -- EpnameE,... + +Contains the list of parameter names for specified view, +monitor, or action. + +=item * V: -- EvnameE,... + +Specifies comma-separated list of all views defined. + +=item * v:EtokenE -- EvnameE,... + +Specifies comma-separated list of view names for the path given. +The first view in the list is interpreted as default. + +=item * M: -- EmnameE,... + +Specifies comma-separated list of all monitor names defined + +=item * A: -- EanameE,... + +Comma-separated list of actions defined + +=back + + + + +B is a I database for storing the +datasource parameter properties, such as expandable, list parameters, +searchable, etc.: + +=over 4 + +=item * EpnameE:EpropertyE -- EvalueE + +=back + + + + + +B is a I database with alias paths as keys +and target tokens as values. It is used for quick alias expansion. + +B is a I database containing the token sets. +The keys and values are as follows: + +=over 4 + +=item * S: -- EtokensetE,... + +Keeps the list of all known token sets. + +=item * s:EtokensetE -- EtokenE,... + +For given tokenset, keeps its contents. + +=item * o:EtokensetE:EtokenE -- EoriginE + +Defines the origin of the member. Currently two types of origin are known: +C and C + +=back + +B is a I database containing the cached +node parameter values. The keys and values are as follows: + +=over 4 + +=item * EnameE:EpnameE -- EstatusEEvalueE + +Keeps the status and the value for a given token and parameter. +Status is a one-byte prefix, with values C for undefined parameter, and +C for a parameter with value. + +=back + + +B is a I database that stores the mapping between +NodeID values and tokens. Database keys are NodeID strings, and values +are tokens. One NodeID corresponds to maximum one token. + + + +B is a I database which contains the informaton +about active processes which read the configuration. The configuration +compiler waits until all readers finish using the current configuration +database. Process IDs are used as keys, and the values contain timestamps. + +B is a I database containing various kinds of +timestamps. The timestamp name is the key, and the number of seconds +since epoch is the value. + +B keeps the status information about the graphs +ready to display. Last known timestamp of the configuration is +compared with the actual one. When the actual timestamp +differs from known, the renderer cache is cleaned up. +This is a I database, with the following +keys and values: + +=over 4 + +=item * EtokenE:EvnameE -- + Et_renderE:Et_expiresE:EfilenameE:Emime_typeE + +For the leaf/subtree and view name given, specifies two timestamps: the +moment of last rendering and expiration time. The filename is automatically +generated unique name in the spool directory. The contents type is determined +by the MIME type. + +=back + +B is a I database used in order to avoid the +unneccessary configuration tree walk. The keys are the leaf tokens, and +the values are comma-separated monitor names. At each monitor invocation, +the confguration timestamp is compared against the last known, and the +cache database is rebuilt if needed. + +B is a I database that keeps alarm status information +from previous runs of Monitor, with the keys and values as follows: + +=over 4 + +=item * EmnameE:EpathE -- +Et_setE:Et_expiresE:EstatusE: +Et_last_changeE + +Key consists of the monitor name and leaf path. In the value, B +is the time when the alarm was raised. If two subsequent runs of Monitor +raise the same alarm, B does not change. B is the +timestamp that shows when it's still important to keep the entry after the +alarm is cleared. B is 1 if the alarm is active, and 0 otherwise. +B is the timestamp of last status change. + +When B is 1, the record is kept regardless of timestamps. +When B is 0, and the current time is more than B, +the record is not reliable and may be deleted by Monitor. + +=back + +B is a I database used in order to avoid the +unneccessary configuration tree walk. X is the collector instance number, and +Y is the datasource configuration instance number. +Keys and values are as follows: + +=over 4 + +=item * EtokenE -- EperiodE:EoffsetE + +For each leaf token, period and time offset values are stored. + +=back + + +B is a I database which stores the runtime +statistics of Scheduler tasks. Each key is of structure +BtasknameE:EperiodE:EoffsetE:EvariableE>, +and the value is a number representing the current value of the variable. +Depending on variable purpose, the number is floating point or integer. + + +B is a I database containing user details, passwords, +and group membership: + +=over 4 + +=item * ua:EuidE:EattrE -- EvalueE + +User attributes, such as C (Common name) or C, are stored +here. For each user, there is a record consisting of the attribute C, +with the value equal to the user identifier. + +=item * uA:EuidE -- EattrE, ... + +Comma-separated list of attribute names for the given user. + +=item * gm:EuidE -- EgroupE, ... + +For each user ID, stores the comma-separated list of groups it belongs to. + +=item * ga:EgroupE:EattrE -- EvalueE + +Group attributes, such as group description. + +=item * gA:EgroupE -- EattrE, ... + +Comma-separated list of attribute names for the given group. + +=item * G: -- EgroupE, ... + +List of all groups + +=back + + +B is a I database containing group privileges information: + +=over 4 + +=item * u:EgroupE:EobjectE:EprivilegeE -- 1 + +The entry exists if and only if the group members have this privilege +over the object given. Most common privilege is C, where +the object is the tree name. + +=back + + +B is a I database containing properties +for each Service ID (exported collector information, usually stored in +an SQL database): + +=over 4 + +=item * a: EserviceidE,... + +Lists all known service IDs + +=item * t:EtreeE -- EserviceidE,... + +Lists service IDs exported by a given datasource tree. + +=item * p:EserviceidE:EparamE -- EvalueE + +Parameter value for a given service ID. Mandatory parameters are: +C, C, C. Optional: C. + +=item * P:EserviceidE -- EparamE, ... + +List of parameter names for a service ID. + +=back + + +B is a I database with DB_DUP and DB_DUPSORT flags. +It contains the search strings for the given tree: + +=over 4 + +=item * EkeywordE -- EpathE[:EparamE] + +For a given keyword, refer to a path of a node that contains this word. +If the node name matches the keyword, the I element +is omitted. Otherwise it refers to the parameter that matches the keyword. + +=back + + + +B is a I database with DB_DUP and DB_DUPSORT flags. +It contains the search strings for all trees: + +=over 4 + +=item * EkeywordE -- EtreeE:EpathE[:EparamE] + +For a given keyword, refer to a path of a node that contains this word. +If the node name matches the keyword, the I element +is omitted. Otherwise it refers to the parameter that matches the keyword. + +=back + + +B is a I database containing SNMP collector +failures information for a given collector instance for a tree. + +=over 4 + +=item * c:EcounterE -- ENE + +A counter with a name. Known names: I, I. + + +=item * h:EhosthashE -- EfailureE:EtimestampE + +SNMP host failure information. Hosthash is a concatenation of hostname, UDP +port, and SNMP community, separated by "|". Known failures: I, +I. Timestamp is a UNIX time of the event. + +=item * m:EhosthashE -- EpathE:EtimestampE + +MIB failures (I, I, and I) +for a given host, with the tree path of their occurence and the UNIX timestamp. + +=item * M:EhosthashE -- ENE + +Count of MIB failures per SNMP host. + +=back + + + + + + + +=head2 Modular Structure + +The Torrus framework consists of several functional modules: + +=over 4 + +=item * Configuration management + +Once the configuration XML files get changed, the configuration compiler +should be run manually. This guarantees that the actual framework +configuration is changed only when the files are ready. + +The configuration management module provides access methods for +enumeration and enquery of the configuratin objects. + +=item * Data Collector module + +Collector program runs as a separate process for each datasource tree. +Upon startup, it first runs all registered collectors. After that, +the collectors are grouped depending on period and time offset, and launched +periodically at the moments defined by formula: + + time + period - (time mod period) + timeoffset + +The datasources are grouped by collector type. +For SNMP collector type, the datasources are grouped by host. +SNMP requests are sent in non-blocking mode (see Net::SNMP Perl module +manual). + +For each SNMP host, system uptime is verified. For RRD datasource types +"COUNTER", if the device reload is +detected, the corresponding RRD file is updated with "undefined" +value at the calculated moment of reload. + +=item * Data threshold monitoring + +This module performs the monitoring tasks periodically, based on each +monitored leaf schedule. +It checks the conditions for each leaf having a monitor. +In case of the alarm, it executes the action instructions synchronously. + +=item * Rendering module + +Upon a request, this module generates the graph and HTML files for the +requested view and its subviews. It first checks availability of +cached objects and avoids unneeded regeneration. It must be possible +to force the renderer to flush the cache. + +=item * Web interface module + +Web interface module passes the Renderer output to an HTTP client. + + +=back + +=head1 Author + +Copyright (c) 2002-2005 Stanislav Sinyagin ssinyagin@yahoo.com diff --git a/torrus/doc/devdoc/devdiscover.pod b/torrus/doc/devdoc/devdiscover.pod new file mode 100644 index 000000000..8386c1755 --- /dev/null +++ b/torrus/doc/devdoc/devdiscover.pod @@ -0,0 +1,296 @@ +# devdiscover.pod - Guide to devdiscover +# Copyright (C) 2003 Shawn Ferry, Stanislav Sinyagin +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. + +# $Id: devdiscover.pod,v 1.1 2010-12-27 00:04:36 ivan Exp $ +# Shawn Ferry +# Stanislav Sinyagin +# + +=head1 Torrus SNMP Device Discovery Developer's Guide + +=head2 C overview + +C is an extensible, module based, SNMP device discovery +utility. It is intended to automatically generate Torrus configuration +files, based on SNMP discovery results and templates. + +See I for command usage and functionality overview. + +In general, C consists of the following files and functional +parts: + +=over 4 + +=item * C + +This file is installed as C in Torrus installation directory, +with certain variables substituted. The program provides all the commandline +functionality and options processing. Once the CLI options are processed and +verified, the control is passed to the C object. + +=item * C + +This Perl module is responsible for the SNMP discovery process organization: + +=over 8 + +=item * + +it registers the discovery modules; + +=item * + +establishes an SNMP session to the target host; + +=item * + +initiates a new C object for the target host; + +=item * + +stores the connection-specific parameters to the device object; + +=item * + +for each registered discovery module, executes C in +I order; + +=item * + +for those discovery modules which paid interest in this target host, +executes C in I order; + +=item * + +upon request from C, builds the configuration +XML tree, by calling C in I order for each +relevant discovery module for each target host. + +=back + +=item * C + +This Perl module is defined in F, and provides +the functionality to store the results of SNMP device discovery. + +=item * C + +This module is an encapsulation wrapper for XML configuration builder. +It provides methods for every element of Torrus configuration. + +=item * Discovery Modules + +These provide all the functionality for SNMP discovery. Normally +one module covers one MIB, or sometimes several vendor-specific MIBs, +and it is responsible for finding out the device details necessary +for Torrus configuration building. Usually a discovery module refers to one or +several I