--- /dev/null
+# architecture.pod: The Torrus internals
+# Copyright (C) 2002-2005 Stanislav Sinyagin
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
+
+# $Id: architecture.pod,v 1.1 2010-12-27 00:04:37 ivan Exp $
+# Stanislav Sinyagin <ssinyagin@yahoo.com>
+#
+#
+
+=head1 Torrus Framework Architecture
+
+=head2 Configuration Processing
+
+The XML configuration is compiled into the database representation by
+operator's manual request.
+
+The compiled version of configuration is not a one-to-one
+representation of the XML version. All aliases and templates are
+expanded. The backwards restoration of XML from the database
+is available with the snapshot utility.
+
+Aliases are the way to represent the data in a more convenient format.
+An alias can point to a subtree or a leaf, and it works similarly as
+a symbolic link in a filesystem.
+
+A template defines a piece of configuration which can be used in
+multiple places. Templates can be nested.
+
+The configuration can consist of several XML files. They will be
+processed in the specified order. Each new file is treated as an additive
+information to the existing tree.
+
+The XML configuration compiler validates all the mandatory parameters.
+
+
+=head2 Database Architecture
+
+All runtime data is stored in
+B<Berkeley DB> database environment (http://www.sleepycat.com).
+
+The compiled version of the configuration XML is stored in
+the B<ds_config_DSINST.db> and B<other_config_OINST.db>.
+These databases have similar structure, and
+B<ds_config_DSINST.db> keeps all datasource-related information.
+C<DSINST> and C<OINST> stand for the productive instance number,
+and have values of 0 or 1.
+Current productive instance numbers are stored in B<db_config_instances.db>
+database.
+
+For each datasource tree, the database files are resided in
+F</var/torrus/db/sub/E<gt>tree_nameE<lt>> directory.
+
+The runtime modules access the configuration via C<ConfigTree> objects.
+
+Each datasource subtree or leaf is identified by a I<token>.
+A token is a short alphanumeric string, generated internally.
+Two types of tokens are recognized: single tokens and token sets.
+
+Single token starts with letter I<T>. The rest is made with decimal digts.
+
+Token set name starts with letter I<S>. The rest is an arbitrary sequence of
+word characters.
+
+The special token I<SS> is reserved for tokensets list. Also tokenset
+parameters are inherited from this token's parameters.
+
+View and monitor names must be unique, and must
+start with a lower case letter.
+
+B<db_config_instances.db> is a I<Hash> database, with keys of form
+C<ds:E<lt>tree_nameE<gt>> or C<other:E<lt>tree_nameE<gt>>, and 0 or 1 as
+values. Also the compiler adds an entry C<compiling:E<lt>tree_nameE<gt>>
+during the compilation time, in order to avoid two compiler processes
+running at the same time on the same tree.
+
+B<ds_config_DSINST.db> is a I<Btree> database, with the keys and values
+defined as follows:
+
+=over 4
+
+=item * tp:E<lt>pathE<gt> -- E<lt>tokenE<gt>
+
+Gives the token correspondig to the given element name.
+
+=item * pt:E<lt>tokenE<gt> -- E<lt>pathE<gt>
+
+Gives the path name by the given token.
+
+=item * c:E<lt>tokenE<gt> -- E<lt>ctokenE<gt>,...
+
+For given subtree, contains the list of child tokens separated by comma.
+
+=item * p:E<lt>tokenE<gt> -- E<lt>ptokenE<gt>
+
+For given subtree or leaf, contains the parent token.
+
+=item * P:E<lt>tokenE<gt>:E<lt>pnameE<gt> -- E<lt>valueE<gt>
+
+Contains the parameter value for specified leaf or subtree.
+Each leaf or subtree inherits parameters from its parent.
+Thus, we must climb up the tree in order to get the parameter's
+value if not defined locally.
+
+=item * Pl:E<lt>tokenE<gt> -- E<lt>pnameE<gt>,...
+
+Contains the list of parameter names for specified leaf or subtree.
+
+=item * a:E<lt>tokenE<gt> -- E<lt>tokenE<gt>
+
+If this subtree or leaf is an alias, specifies the reference to the real node.
+
+=item * ar:E<lt>tokenE<gt> -- E<lt>tokenE<gt>,...
+
+Specifies all alias subtrees or leaves pointing to this token.
+
+=item * d:E<lt>nameE<gt> -- E<lt>valueE<gt>
+
+Definition value for the given name
+
+=item * D: -- E<lt>nameE<gt>,E<lt>nameE<gt>,...
+
+List of all known definitions
+
+=item * n:E<lt>tokenE<gt> -- E<lt>typeE<gt>
+
+Defines a node type. Type is a number with the following values:
+0 for subtree, 1 for leaf, 2 for alias.
+
+=back
+
+B<other_config_OINST.db> is a I<Btree> database, with the keys and values
+defined as follows:
+
+=over 4
+
+=item * ConfigurationReady -- 1:0
+
+When nonzero, the configuration is ready for usage.
+Otherwise, the configuration status is undefined.
+
+=item * P:E<lt>nameE<gt>:E<lt>pnameE<gt> -- E<lt>valueE<gt>
+
+Contains the parameter value for specified view, monitor or action.
+
+=item * Pl:E<lt>nameE<gt> -- E<lt>pnameE<gt>,...
+
+Contains the list of parameter names for specified view,
+monitor, or action.
+
+=item * V: -- E<lt>vnameE<gt>,...
+
+Specifies comma-separated list of all views defined.
+
+=item * v:E<lt>tokenE<gt> -- E<lt>vnameE<gt>,...
+
+Specifies comma-separated list of view names for the path given.
+The first view in the list is interpreted as default.
+
+=item * M: -- E<lt>mnameE<gt>,...
+
+Specifies comma-separated list of all monitor names defined
+
+=item * A: -- E<lt>anameE<gt>,...
+
+Comma-separated list of actions defined
+
+=back
+
+
+
+
+B<paramprops_DSINST.db> is a I<Btree> database for storing the
+datasource parameter properties, such as expandable, list parameters,
+searchable, etc.:
+
+=over 4
+
+=item * E<lt>pnameE<gt>:E<lt>propertyE<gt> -- E<lt>valueE<gt>
+
+=back
+
+
+
+
+
+B<aliases_DSINST.db> is a I<Btree> database with alias paths as keys
+and target tokens as values. It is used for quick alias expansion.
+
+B<tokensets_DSINST.db> is a I<Hash> database containing the token sets.
+The keys and values are as follows:
+
+=over 4
+
+=item * S: -- E<lt>tokensetE<gt>,...
+
+Keeps the list of all known token sets.
+
+=item * s:E<lt>tokensetE<gt> -- E<lt>tokenE<gt>,...
+
+For given tokenset, keeps its contents.
+
+=item * o:E<lt>tokensetE<gt>:E<lt>tokenE<gt> -- E<lt>originE<gt>
+
+Defines the origin of the member. Currently two types of origin are known:
+C<static> and C<monitor>
+
+=back
+
+B<nodepcache_DSINST.db> is a I<Btree> database containing the cached
+node parameter values. The keys and values are as follows:
+
+=over 4
+
+=item * E<lt>nameE<gt>:E<lt>pnameE<gt> -- E<lt>statusE<gt>E<lt>valueE<gt>
+
+Keeps the status and the value for a given token and parameter.
+Status is a one-byte prefix, with values C<U> for undefined parameter, and
+C<D> for a parameter with value.
+
+=back
+
+
+B<nodeid_DSINST.db> is a I<Btree> database that stores the mapping between
+NodeID values and tokens. Database keys are NodeID strings, and values
+are tokens. One NodeID corresponds to maximum one token.
+
+
+
+B<config_readers.db> is a I<Hash> database which contains the informaton
+about active processes which read the configuration. The configuration
+compiler waits until all readers finish using the current configuration
+database. Process IDs are used as keys, and the values contain timestamps.
+
+B<timestamps.db> is a I<Hash> database containing various kinds of
+timestamps. The timestamp name is the key, and the number of seconds
+since epoch is the value.
+
+B<render_cache.db> keeps the status information about the graphs
+ready to display. Last known timestamp of the configuration is
+compared with the actual one. When the actual timestamp
+differs from known, the renderer cache is cleaned up.
+This is a I<Hash> database, with the following
+keys and values:
+
+=over 4
+
+=item * E<lt>tokenE<gt>:E<lt>vnameE<gt> --
+ E<lt>t_renderE<gt>:E<lt>t_expiresE<gt>:E<lt>filenameE<gt>:E<lt>mime_typeE<gt>
+
+For the leaf/subtree and view name given, specifies two timestamps: the
+moment of last rendering and expiration time. The filename is automatically
+generated unique name in the spool directory. The contents type is determined
+by the MIME type.
+
+=back
+
+B<monitor_cache.db> is a I<Hash> database used in order to avoid the
+unneccessary configuration tree walk. The keys are the leaf tokens, and
+the values are comma-separated monitor names. At each monitor invocation,
+the confguration timestamp is compared against the last known, and the
+cache database is rebuilt if needed.
+
+B<monitor_alarms.db> is a I<Hash> database that keeps alarm status information
+from previous runs of Monitor, with the keys and values as follows:
+
+=over 4
+
+=item * E<lt>mnameE<gt>:E<lt>pathE<gt> --
+E<lt>t_setE<gt>:E<lt>t_expiresE<gt>:E<lt>statusE<gt>:
+E<lt>t_last_changeE<gt>
+
+Key consists of the monitor name and leaf path. In the value, B<t_set>
+is the time when the alarm was raised. If two subsequent runs of Monitor
+raise the same alarm, B<t_set> does not change. B<t_expires> is the
+timestamp that shows when it's still important to keep the entry after the
+alarm is cleared. B<status> is 1 if the alarm is active, and 0 otherwise.
+B<t_last_change> is the timestamp of last status change.
+
+When B<status> is 1, the record is kept regardless of timestamps.
+When B<status> is 0, and the current time is more than B<t_expires>,
+the record is not reliable and may be deleted by Monitor.
+
+=back
+
+B<collector_tokens_X_Y.db> is a I<Hash> database used in order to avoid the
+unneccessary configuration tree walk. X is the collector instance number, and
+Y is the datasource configuration instance number.
+Keys and values are as follows:
+
+=over 4
+
+=item * E<lt>tokenE<gt> -- E<lt>periodE<gt>:E<lt>offsetE<gt>
+
+For each leaf token, period and time offset values are stored.
+
+=back
+
+
+B<scheduler_stats.db> is a I<Btree> database which stores the runtime
+statistics of Scheduler tasks. Each key is of structure
+B<E<lt>tasknameE<gt>:E<lt>periodE<gt>:E<lt>offsetE<gt>:E<lt>variableE<gt>>,
+and the value is a number representing the current value of the variable.
+Depending on variable purpose, the number is floating point or integer.
+
+
+B<users.db> is a I<Hash> database containing user details, passwords,
+and group membership:
+
+=over 4
+
+=item * ua:E<lt>uidE<gt>:E<lt>attrE<gt> -- E<lt>valueE<gt>
+
+User attributes, such as C<cn> (Common name) or C<userPassword>, are stored
+here. For each user, there is a record consisting of the attribute C<uid>,
+with the value equal to the user identifier.
+
+=item * uA:E<lt>uidE<gt> -- E<lt>attrE<gt>, ...
+
+Comma-separated list of attribute names for the given user.
+
+=item * gm:E<lt>uidE<gt> -- E<lt>groupE<gt>, ...
+
+For each user ID, stores the comma-separated list of groups it belongs to.
+
+=item * ga:E<lt>groupE<gt>:E<lt>attrE<gt> -- E<lt>valueE<gt>
+
+Group attributes, such as group description.
+
+=item * gA:E<lt>groupE<gt> -- E<lt>attrE<gt>, ...
+
+Comma-separated list of attribute names for the given group.
+
+=item * G: -- E<lt>groupE<gt>, ...
+
+List of all groups
+
+=back
+
+
+B<acl.db> is a I<Hash> database containing group privileges information:
+
+=over 4
+
+=item * u:E<lt>groupE<gt>:E<lt>objectE<gt>:E<lt>privilegeE<gt> -- 1
+
+The entry exists if and only if the group members have this privilege
+over the object given. Most common privilege is C<DisplayTree>, where
+the object is the tree name.
+
+=back
+
+
+B<serviceid_params.db> is a I<Btree> database containing properties
+for each Service ID (exported collector information, usually stored in
+an SQL database):
+
+=over 4
+
+=item * a: E<lt>serviceidE<gt>,...
+
+Lists all known service IDs
+
+=item * t:E<lt>treeE<gt> -- E<lt>serviceidE<gt>,...
+
+Lists service IDs exported by a given datasource tree.
+
+=item * p:E<lt>serviceidE<gt>:E<lt>paramE<gt> -- E<lt>valueE<gt>
+
+Parameter value for a given service ID. Mandatory parameters are:
+C<tree>, C<token>, C<dstype>. Optional: C<units>.
+
+=item * P:E<lt>serviceidE<gt> -- E<lt>paramE<gt>, ...
+
+List of parameter names for a service ID.
+
+=back
+
+
+B<searchwords.db> is a I<Btree> database with DB_DUP and DB_DUPSORT flags.
+It contains the search strings for the given tree:
+
+=over 4
+
+=item * E<lt>keywordE<gt> -- E<lt>pathE<gt>[:E<lt>paramE<gt>]
+
+For a given keyword, refer to a path of a node that contains this word.
+If the node name matches the keyword, the I<param> element
+is omitted. Otherwise it refers to the parameter that matches the keyword.
+
+=back
+
+
+
+B<globsearchwords.db> is a I<Btree> database with DB_DUP and DB_DUPSORT flags.
+It contains the search strings for all trees:
+
+=over 4
+
+=item * E<lt>keywordE<gt> -- E<lt>treeE<gt>:E<lt>pathE<gt>[:E<lt>paramE<gt>]
+
+For a given keyword, refer to a path of a node that contains this word.
+If the node name matches the keyword, the I<param> element
+is omitted. Otherwise it refers to the parameter that matches the keyword.
+
+=back
+
+
+B<snmp_failures_X.db> is a I<Btree> database containing SNMP collector
+failures information for a given collector instance for a tree.
+
+=over 4
+
+=item * c:E<lt>counterE<gt> -- E<lt>NE<gt>
+
+A counter with a name. Known names: I<unreachable>, I<removed>.
+
+
+=item * h:E<lt>hosthashE<gt> -- E<lt>failureE<gt>:E<lt>timestampE<gt>
+
+SNMP host failure information. Hosthash is a concatenation of hostname, UDP
+port, and SNMP community, separated by "|". Known failures: I<unreachable>,
+I<removed>. Timestamp is a UNIX time of the event.
+
+=item * m:E<lt>hosthashE<gt> -- E<lt>pathE<gt>:E<lt>timestampE<gt>
+
+MIB failures (I<noSuchObject>, I<noSuchInstance>, and I<endOfMibView>)
+for a given host, with the tree path of their occurence and the UNIX timestamp.
+
+=item * M:E<lt>hosthashE<gt> -- E<lt>NE<gt>
+
+Count of MIB failures per SNMP host.
+
+=back
+
+
+
+
+
+
+
+=head2 Modular Structure
+
+The Torrus framework consists of several functional modules:
+
+=over 4
+
+=item * Configuration management
+
+Once the configuration XML files get changed, the configuration compiler
+should be run manually. This guarantees that the actual framework
+configuration is changed only when the files are ready.
+
+The configuration management module provides access methods for
+enumeration and enquery of the configuratin objects.
+
+=item * Data Collector module
+
+Collector program runs as a separate process for each datasource tree.
+Upon startup, it first runs all registered collectors. After that,
+the collectors are grouped depending on period and time offset, and launched
+periodically at the moments defined by formula:
+
+ time + period - (time mod period) + timeoffset
+
+The datasources are grouped by collector type.
+For SNMP collector type, the datasources are grouped by host.
+SNMP requests are sent in non-blocking mode (see Net::SNMP Perl module
+manual).
+
+For each SNMP host, system uptime is verified. For RRD datasource types
+"COUNTER", if the device reload is
+detected, the corresponding RRD file is updated with "undefined"
+value at the calculated moment of reload.
+
+=item * Data threshold monitoring
+
+This module performs the monitoring tasks periodically, based on each
+monitored leaf schedule.
+It checks the conditions for each leaf having a monitor.
+In case of the alarm, it executes the action instructions synchronously.
+
+=item * Rendering module
+
+Upon a request, this module generates the graph and HTML files for the
+requested view and its subviews. It first checks availability of
+cached objects and avoids unneeded regeneration. It must be possible
+to force the renderer to flush the cache.
+
+=item * Web interface module
+
+Web interface module passes the Renderer output to an HTTP client.
+
+
+=back
+
+=head1 Author
+
+Copyright (c) 2002-2005 Stanislav Sinyagin ssinyagin@yahoo.com