diff options
Diffstat (limited to 'torrus/doc/devdoc/architecture.pod')
| -rw-r--r-- | torrus/doc/devdoc/architecture.pod | 511 |
1 files changed, 511 insertions, 0 deletions
diff --git a/torrus/doc/devdoc/architecture.pod b/torrus/doc/devdoc/architecture.pod new file mode 100644 index 000000000..4cf9c9ccb --- /dev/null +++ b/torrus/doc/devdoc/architecture.pod @@ -0,0 +1,511 @@ +# architecture.pod: The Torrus internals +# Copyright (C) 2002-2005 Stanislav Sinyagin +# +# This program is free software; you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation; either version 2 of the License, or +# (at your option) any later version. +# +# This program is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with this program; if not, write to the Free Software +# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA. + +# $Id: architecture.pod,v 1.1 2010-12-27 00:04:37 ivan Exp $ +# Stanislav Sinyagin <ssinyagin@yahoo.com> +# +# + +=head1 Torrus Framework Architecture + +=head2 Configuration Processing + +The XML configuration is compiled into the database representation by +operator's manual request. + +The compiled version of configuration is not a one-to-one +representation of the XML version. All aliases and templates are +expanded. The backwards restoration of XML from the database +is available with the snapshot utility. + +Aliases are the way to represent the data in a more convenient format. +An alias can point to a subtree or a leaf, and it works similarly as +a symbolic link in a filesystem. + +A template defines a piece of configuration which can be used in +multiple places. Templates can be nested. + +The configuration can consist of several XML files. They will be +processed in the specified order. Each new file is treated as an additive +information to the existing tree. + +The XML configuration compiler validates all the mandatory parameters. + + +=head2 Database Architecture + +All runtime data is stored in +B<Berkeley DB> database environment (http://www.sleepycat.com). + +The compiled version of the configuration XML is stored in +the B<ds_config_DSINST.db> and B<other_config_OINST.db>. +These databases have similar structure, and +B<ds_config_DSINST.db> keeps all datasource-related information. +C<DSINST> and C<OINST> stand for the productive instance number, +and have values of 0 or 1. +Current productive instance numbers are stored in B<db_config_instances.db> +database. + +For each datasource tree, the database files are resided in +F</var/torrus/db/sub/E<gt>tree_nameE<lt>> directory. + +The runtime modules access the configuration via C<ConfigTree> objects. + +Each datasource subtree or leaf is identified by a I<token>. +A token is a short alphanumeric string, generated internally. +Two types of tokens are recognized: single tokens and token sets. + +Single token starts with letter I<T>. The rest is made with decimal digts. + +Token set name starts with letter I<S>. The rest is an arbitrary sequence of +word characters. + +The special token I<SS> is reserved for tokensets list. Also tokenset +parameters are inherited from this token's parameters. + +View and monitor names must be unique, and must +start with a lower case letter. + +B<db_config_instances.db> is a I<Hash> database, with keys of form +C<ds:E<lt>tree_nameE<gt>> or C<other:E<lt>tree_nameE<gt>>, and 0 or 1 as +values. Also the compiler adds an entry C<compiling:E<lt>tree_nameE<gt>> +during the compilation time, in order to avoid two compiler processes +running at the same time on the same tree. + +B<ds_config_DSINST.db> is a I<Btree> database, with the keys and values +defined as follows: + +=over 4 + +=item * tp:E<lt>pathE<gt> -- E<lt>tokenE<gt> + +Gives the token correspondig to the given element name. + +=item * pt:E<lt>tokenE<gt> -- E<lt>pathE<gt> + +Gives the path name by the given token. + +=item * c:E<lt>tokenE<gt> -- E<lt>ctokenE<gt>,... + +For given subtree, contains the list of child tokens separated by comma. + +=item * p:E<lt>tokenE<gt> -- E<lt>ptokenE<gt> + +For given subtree or leaf, contains the parent token. + +=item * P:E<lt>tokenE<gt>:E<lt>pnameE<gt> -- E<lt>valueE<gt> + +Contains the parameter value for specified leaf or subtree. +Each leaf or subtree inherits parameters from its parent. +Thus, we must climb up the tree in order to get the parameter's +value if not defined locally. + +=item * Pl:E<lt>tokenE<gt> -- E<lt>pnameE<gt>,... + +Contains the list of parameter names for specified leaf or subtree. + +=item * a:E<lt>tokenE<gt> -- E<lt>tokenE<gt> + +If this subtree or leaf is an alias, specifies the reference to the real node. + +=item * ar:E<lt>tokenE<gt> -- E<lt>tokenE<gt>,... + +Specifies all alias subtrees or leaves pointing to this token. + +=item * d:E<lt>nameE<gt> -- E<lt>valueE<gt> + +Definition value for the given name + +=item * D: -- E<lt>nameE<gt>,E<lt>nameE<gt>,... + +List of all known definitions + +=item * n:E<lt>tokenE<gt> -- E<lt>typeE<gt> + +Defines a node type. Type is a number with the following values: +0 for subtree, 1 for leaf, 2 for alias. + +=back + +B<other_config_OINST.db> is a I<Btree> database, with the keys and values +defined as follows: + +=over 4 + +=item * ConfigurationReady -- 1:0 + +When nonzero, the configuration is ready for usage. +Otherwise, the configuration status is undefined. + +=item * P:E<lt>nameE<gt>:E<lt>pnameE<gt> -- E<lt>valueE<gt> + +Contains the parameter value for specified view, monitor or action. + +=item * Pl:E<lt>nameE<gt> -- E<lt>pnameE<gt>,... + +Contains the list of parameter names for specified view, +monitor, or action. + +=item * V: -- E<lt>vnameE<gt>,... + +Specifies comma-separated list of all views defined. + +=item * v:E<lt>tokenE<gt> -- E<lt>vnameE<gt>,... + +Specifies comma-separated list of view names for the path given. +The first view in the list is interpreted as default. + +=item * M: -- E<lt>mnameE<gt>,... + +Specifies comma-separated list of all monitor names defined + +=item * A: -- E<lt>anameE<gt>,... + +Comma-separated list of actions defined + +=back + + + + +B<paramprops_DSINST.db> is a I<Btree> database for storing the +datasource parameter properties, such as expandable, list parameters, +searchable, etc.: + +=over 4 + +=item * E<lt>pnameE<gt>:E<lt>propertyE<gt> -- E<lt>valueE<gt> + +=back + + + + + +B<aliases_DSINST.db> is a I<Btree> database with alias paths as keys +and target tokens as values. It is used for quick alias expansion. + +B<tokensets_DSINST.db> is a I<Hash> database containing the token sets. +The keys and values are as follows: + +=over 4 + +=item * S: -- E<lt>tokensetE<gt>,... + +Keeps the list of all known token sets. + +=item * s:E<lt>tokensetE<gt> -- E<lt>tokenE<gt>,... + +For given tokenset, keeps its contents. + +=item * o:E<lt>tokensetE<gt>:E<lt>tokenE<gt> -- E<lt>originE<gt> + +Defines the origin of the member. Currently two types of origin are known: +C<static> and C<monitor> + +=back + +B<nodepcache_DSINST.db> is a I<Btree> database containing the cached +node parameter values. The keys and values are as follows: + +=over 4 + +=item * E<lt>nameE<gt>:E<lt>pnameE<gt> -- E<lt>statusE<gt>E<lt>valueE<gt> + +Keeps the status and the value for a given token and parameter. +Status is a one-byte prefix, with values C<U> for undefined parameter, and +C<D> for a parameter with value. + +=back + + +B<nodeid_DSINST.db> is a I<Btree> database that stores the mapping between +NodeID values and tokens. Database keys are NodeID strings, and values +are tokens. One NodeID corresponds to maximum one token. + + + +B<config_readers.db> is a I<Hash> database which contains the informaton +about active processes which read the configuration. The configuration +compiler waits until all readers finish using the current configuration +database. Process IDs are used as keys, and the values contain timestamps. + +B<timestamps.db> is a I<Hash> database containing various kinds of +timestamps. The timestamp name is the key, and the number of seconds +since epoch is the value. + +B<render_cache.db> keeps the status information about the graphs +ready to display. Last known timestamp of the configuration is +compared with the actual one. When the actual timestamp +differs from known, the renderer cache is cleaned up. +This is a I<Hash> database, with the following +keys and values: + +=over 4 + +=item * E<lt>tokenE<gt>:E<lt>vnameE<gt> -- + E<lt>t_renderE<gt>:E<lt>t_expiresE<gt>:E<lt>filenameE<gt>:E<lt>mime_typeE<gt> + +For the leaf/subtree and view name given, specifies two timestamps: the +moment of last rendering and expiration time. The filename is automatically +generated unique name in the spool directory. The contents type is determined +by the MIME type. + +=back + +B<monitor_cache.db> is a I<Hash> database used in order to avoid the +unneccessary configuration tree walk. The keys are the leaf tokens, and +the values are comma-separated monitor names. At each monitor invocation, +the confguration timestamp is compared against the last known, and the +cache database is rebuilt if needed. + +B<monitor_alarms.db> is a I<Hash> database that keeps alarm status information +from previous runs of Monitor, with the keys and values as follows: + +=over 4 + +=item * E<lt>mnameE<gt>:E<lt>pathE<gt> -- +E<lt>t_setE<gt>:E<lt>t_expiresE<gt>:E<lt>statusE<gt>: +E<lt>t_last_changeE<gt> + +Key consists of the monitor name and leaf path. In the value, B<t_set> +is the time when the alarm was raised. If two subsequent runs of Monitor +raise the same alarm, B<t_set> does not change. B<t_expires> is the +timestamp that shows when it's still important to keep the entry after the +alarm is cleared. B<status> is 1 if the alarm is active, and 0 otherwise. +B<t_last_change> is the timestamp of last status change. + +When B<status> is 1, the record is kept regardless of timestamps. +When B<status> is 0, and the current time is more than B<t_expires>, +the record is not reliable and may be deleted by Monitor. + +=back + +B<collector_tokens_X_Y.db> is a I<Hash> database used in order to avoid the +unneccessary configuration tree walk. X is the collector instance number, and +Y is the datasource configuration instance number. +Keys and values are as follows: + +=over 4 + +=item * E<lt>tokenE<gt> -- E<lt>periodE<gt>:E<lt>offsetE<gt> + +For each leaf token, period and time offset values are stored. + +=back + + +B<scheduler_stats.db> is a I<Btree> database which stores the runtime +statistics of Scheduler tasks. Each key is of structure +B<E<lt>tasknameE<gt>:E<lt>periodE<gt>:E<lt>offsetE<gt>:E<lt>variableE<gt>>, +and the value is a number representing the current value of the variable. +Depending on variable purpose, the number is floating point or integer. + + +B<users.db> is a I<Hash> database containing user details, passwords, +and group membership: + +=over 4 + +=item * ua:E<lt>uidE<gt>:E<lt>attrE<gt> -- E<lt>valueE<gt> + +User attributes, such as C<cn> (Common name) or C<userPassword>, are stored +here. For each user, there is a record consisting of the attribute C<uid>, +with the value equal to the user identifier. + +=item * uA:E<lt>uidE<gt> -- E<lt>attrE<gt>, ... + +Comma-separated list of attribute names for the given user. + +=item * gm:E<lt>uidE<gt> -- E<lt>groupE<gt>, ... + +For each user ID, stores the comma-separated list of groups it belongs to. + +=item * ga:E<lt>groupE<gt>:E<lt>attrE<gt> -- E<lt>valueE<gt> + +Group attributes, such as group description. + +=item * gA:E<lt>groupE<gt> -- E<lt>attrE<gt>, ... + +Comma-separated list of attribute names for the given group. + +=item * G: -- E<lt>groupE<gt>, ... + +List of all groups + +=back + + +B<acl.db> is a I<Hash> database containing group privileges information: + +=over 4 + +=item * u:E<lt>groupE<gt>:E<lt>objectE<gt>:E<lt>privilegeE<gt> -- 1 + +The entry exists if and only if the group members have this privilege +over the object given. Most common privilege is C<DisplayTree>, where +the object is the tree name. + +=back + + +B<serviceid_params.db> is a I<Btree> database containing properties +for each Service ID (exported collector information, usually stored in +an SQL database): + +=over 4 + +=item * a: E<lt>serviceidE<gt>,... + +Lists all known service IDs + +=item * t:E<lt>treeE<gt> -- E<lt>serviceidE<gt>,... + +Lists service IDs exported by a given datasource tree. + +=item * p:E<lt>serviceidE<gt>:E<lt>paramE<gt> -- E<lt>valueE<gt> + +Parameter value for a given service ID. Mandatory parameters are: +C<tree>, C<token>, C<dstype>. Optional: C<units>. + +=item * P:E<lt>serviceidE<gt> -- E<lt>paramE<gt>, ... + +List of parameter names for a service ID. + +=back + + +B<searchwords.db> is a I<Btree> database with DB_DUP and DB_DUPSORT flags. +It contains the search strings for the given tree: + +=over 4 + +=item * E<lt>keywordE<gt> -- E<lt>pathE<gt>[:E<lt>paramE<gt>] + +For a given keyword, refer to a path of a node that contains this word. +If the node name matches the keyword, the I<param> element +is omitted. Otherwise it refers to the parameter that matches the keyword. + +=back + + + +B<globsearchwords.db> is a I<Btree> database with DB_DUP and DB_DUPSORT flags. +It contains the search strings for all trees: + +=over 4 + +=item * E<lt>keywordE<gt> -- E<lt>treeE<gt>:E<lt>pathE<gt>[:E<lt>paramE<gt>] + +For a given keyword, refer to a path of a node that contains this word. +If the node name matches the keyword, the I<param> element +is omitted. Otherwise it refers to the parameter that matches the keyword. + +=back + + +B<snmp_failures_X.db> is a I<Btree> database containing SNMP collector +failures information for a given collector instance for a tree. + +=over 4 + +=item * c:E<lt>counterE<gt> -- E<lt>NE<gt> + +A counter with a name. Known names: I<unreachable>, I<removed>. + + +=item * h:E<lt>hosthashE<gt> -- E<lt>failureE<gt>:E<lt>timestampE<gt> + +SNMP host failure information. Hosthash is a concatenation of hostname, UDP +port, and SNMP community, separated by "|". Known failures: I<unreachable>, +I<removed>. Timestamp is a UNIX time of the event. + +=item * m:E<lt>hosthashE<gt> -- E<lt>pathE<gt>:E<lt>timestampE<gt> + +MIB failures (I<noSuchObject>, I<noSuchInstance>, and I<endOfMibView>) +for a given host, with the tree path of their occurence and the UNIX timestamp. + +=item * M:E<lt>hosthashE<gt> -- E<lt>NE<gt> + +Count of MIB failures per SNMP host. + +=back + + + + + + + +=head2 Modular Structure + +The Torrus framework consists of several functional modules: + +=over 4 + +=item * Configuration management + +Once the configuration XML files get changed, the configuration compiler +should be run manually. This guarantees that the actual framework +configuration is changed only when the files are ready. + +The configuration management module provides access methods for +enumeration and enquery of the configuratin objects. + +=item * Data Collector module + +Collector program runs as a separate process for each datasource tree. +Upon startup, it first runs all registered collectors. After that, +the collectors are grouped depending on period and time offset, and launched +periodically at the moments defined by formula: + + time + period - (time mod period) + timeoffset + +The datasources are grouped by collector type. +For SNMP collector type, the datasources are grouped by host. +SNMP requests are sent in non-blocking mode (see Net::SNMP Perl module +manual). + +For each SNMP host, system uptime is verified. For RRD datasource types +"COUNTER", if the device reload is +detected, the corresponding RRD file is updated with "undefined" +value at the calculated moment of reload. + +=item * Data threshold monitoring + +This module performs the monitoring tasks periodically, based on each +monitored leaf schedule. +It checks the conditions for each leaf having a monitor. +In case of the alarm, it executes the action instructions synchronously. + +=item * Rendering module + +Upon a request, this module generates the graph and HTML files for the +requested view and its subviews. It first checks availability of +cached objects and avoids unneeded regeneration. It must be possible +to force the renderer to flush the cache. + +=item * Web interface module + +Web interface module passes the Renderer output to an HTTP client. + + +=back + +=head1 Author + +Copyright (c) 2002-2005 Stanislav Sinyagin ssinyagin@yahoo.com |