1 # architecture.pod: The Torrus internals
2 # Copyright (C) 2002-2005 Stanislav Sinyagin
4 # This program is free software; you can redistribute it and/or modify
5 # it under the terms of the GNU General Public License as published by
6 # the Free Software Foundation; either version 2 of the License, or
7 # (at your option) any later version.
9 # This program is distributed in the hope that it will be useful,
10 # but WITHOUT ANY WARRANTY; without even the implied warranty of
11 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 # GNU General Public License for more details.
14 # You should have received a copy of the GNU General Public License
15 # along with this program; if not, write to the Free Software
16 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
18 # $Id: architecture.pod,v 1.1 2010-12-27 00:04:37 ivan Exp $
19 # Stanislav Sinyagin <ssinyagin@yahoo.com>
23 =head1 Torrus Framework Architecture
25 =head2 Configuration Processing
27 The XML configuration is compiled into the database representation by
28 operator's manual request.
30 The compiled version of configuration is not a one-to-one
31 representation of the XML version. All aliases and templates are
32 expanded. The backwards restoration of XML from the database
33 is available with the snapshot utility.
35 Aliases are the way to represent the data in a more convenient format.
36 An alias can point to a subtree or a leaf, and it works similarly as
37 a symbolic link in a filesystem.
39 A template defines a piece of configuration which can be used in
40 multiple places. Templates can be nested.
42 The configuration can consist of several XML files. They will be
43 processed in the specified order. Each new file is treated as an additive
44 information to the existing tree.
46 The XML configuration compiler validates all the mandatory parameters.
49 =head2 Database Architecture
51 All runtime data is stored in
52 B<Berkeley DB> database environment (http://www.sleepycat.com).
54 The compiled version of the configuration XML is stored in
55 the B<ds_config_DSINST.db> and B<other_config_OINST.db>.
56 These databases have similar structure, and
57 B<ds_config_DSINST.db> keeps all datasource-related information.
58 C<DSINST> and C<OINST> stand for the productive instance number,
59 and have values of 0 or 1.
60 Current productive instance numbers are stored in B<db_config_instances.db>
63 For each datasource tree, the database files are resided in
64 F</var/torrus/db/sub/E<gt>tree_nameE<lt>> directory.
66 The runtime modules access the configuration via C<ConfigTree> objects.
68 Each datasource subtree or leaf is identified by a I<token>.
69 A token is a short alphanumeric string, generated internally.
70 Two types of tokens are recognized: single tokens and token sets.
72 Single token starts with letter I<T>. The rest is made with decimal digts.
74 Token set name starts with letter I<S>. The rest is an arbitrary sequence of
77 The special token I<SS> is reserved for tokensets list. Also tokenset
78 parameters are inherited from this token's parameters.
80 View and monitor names must be unique, and must
81 start with a lower case letter.
83 B<db_config_instances.db> is a I<Hash> database, with keys of form
84 C<ds:E<lt>tree_nameE<gt>> or C<other:E<lt>tree_nameE<gt>>, and 0 or 1 as
85 values. Also the compiler adds an entry C<compiling:E<lt>tree_nameE<gt>>
86 during the compilation time, in order to avoid two compiler processes
87 running at the same time on the same tree.
89 B<ds_config_DSINST.db> is a I<Btree> database, with the keys and values
94 =item * tp:E<lt>pathE<gt> -- E<lt>tokenE<gt>
96 Gives the token correspondig to the given element name.
98 =item * pt:E<lt>tokenE<gt> -- E<lt>pathE<gt>
100 Gives the path name by the given token.
102 =item * c:E<lt>tokenE<gt> -- E<lt>ctokenE<gt>,...
104 For given subtree, contains the list of child tokens separated by comma.
106 =item * p:E<lt>tokenE<gt> -- E<lt>ptokenE<gt>
108 For given subtree or leaf, contains the parent token.
110 =item * P:E<lt>tokenE<gt>:E<lt>pnameE<gt> -- E<lt>valueE<gt>
112 Contains the parameter value for specified leaf or subtree.
113 Each leaf or subtree inherits parameters from its parent.
114 Thus, we must climb up the tree in order to get the parameter's
115 value if not defined locally.
117 =item * Pl:E<lt>tokenE<gt> -- E<lt>pnameE<gt>,...
119 Contains the list of parameter names for specified leaf or subtree.
121 =item * a:E<lt>tokenE<gt> -- E<lt>tokenE<gt>
123 If this subtree or leaf is an alias, specifies the reference to the real node.
125 =item * ar:E<lt>tokenE<gt> -- E<lt>tokenE<gt>,...
127 Specifies all alias subtrees or leaves pointing to this token.
129 =item * d:E<lt>nameE<gt> -- E<lt>valueE<gt>
131 Definition value for the given name
133 =item * D: -- E<lt>nameE<gt>,E<lt>nameE<gt>,...
135 List of all known definitions
137 =item * n:E<lt>tokenE<gt> -- E<lt>typeE<gt>
139 Defines a node type. Type is a number with the following values:
140 0 for subtree, 1 for leaf, 2 for alias.
144 B<other_config_OINST.db> is a I<Btree> database, with the keys and values
149 =item * ConfigurationReady -- 1:0
151 When nonzero, the configuration is ready for usage.
152 Otherwise, the configuration status is undefined.
154 =item * P:E<lt>nameE<gt>:E<lt>pnameE<gt> -- E<lt>valueE<gt>
156 Contains the parameter value for specified view, monitor or action.
158 =item * Pl:E<lt>nameE<gt> -- E<lt>pnameE<gt>,...
160 Contains the list of parameter names for specified view,
163 =item * V: -- E<lt>vnameE<gt>,...
165 Specifies comma-separated list of all views defined.
167 =item * v:E<lt>tokenE<gt> -- E<lt>vnameE<gt>,...
169 Specifies comma-separated list of view names for the path given.
170 The first view in the list is interpreted as default.
172 =item * M: -- E<lt>mnameE<gt>,...
174 Specifies comma-separated list of all monitor names defined
176 =item * A: -- E<lt>anameE<gt>,...
178 Comma-separated list of actions defined
185 B<paramprops_DSINST.db> is a I<Btree> database for storing the
186 datasource parameter properties, such as expandable, list parameters,
191 =item * E<lt>pnameE<gt>:E<lt>propertyE<gt> -- E<lt>valueE<gt>
199 B<aliases_DSINST.db> is a I<Btree> database with alias paths as keys
200 and target tokens as values. It is used for quick alias expansion.
202 B<tokensets_DSINST.db> is a I<Hash> database containing the token sets.
203 The keys and values are as follows:
207 =item * S: -- E<lt>tokensetE<gt>,...
209 Keeps the list of all known token sets.
211 =item * s:E<lt>tokensetE<gt> -- E<lt>tokenE<gt>,...
213 For given tokenset, keeps its contents.
215 =item * o:E<lt>tokensetE<gt>:E<lt>tokenE<gt> -- E<lt>originE<gt>
217 Defines the origin of the member. Currently two types of origin are known:
218 C<static> and C<monitor>
222 B<nodepcache_DSINST.db> is a I<Btree> database containing the cached
223 node parameter values. The keys and values are as follows:
227 =item * E<lt>nameE<gt>:E<lt>pnameE<gt> -- E<lt>statusE<gt>E<lt>valueE<gt>
229 Keeps the status and the value for a given token and parameter.
230 Status is a one-byte prefix, with values C<U> for undefined parameter, and
231 C<D> for a parameter with value.
236 B<nodeid_DSINST.db> is a I<Btree> database that stores the mapping between
237 NodeID values and tokens. Database keys are NodeID strings, and values
238 are tokens. One NodeID corresponds to maximum one token.
242 B<config_readers.db> is a I<Hash> database which contains the informaton
243 about active processes which read the configuration. The configuration
244 compiler waits until all readers finish using the current configuration
245 database. Process IDs are used as keys, and the values contain timestamps.
247 B<timestamps.db> is a I<Hash> database containing various kinds of
248 timestamps. The timestamp name is the key, and the number of seconds
249 since epoch is the value.
251 B<render_cache.db> keeps the status information about the graphs
252 ready to display. Last known timestamp of the configuration is
253 compared with the actual one. When the actual timestamp
254 differs from known, the renderer cache is cleaned up.
255 This is a I<Hash> database, with the following
260 =item * E<lt>tokenE<gt>:E<lt>vnameE<gt> --
261 E<lt>t_renderE<gt>:E<lt>t_expiresE<gt>:E<lt>filenameE<gt>:E<lt>mime_typeE<gt>
263 For the leaf/subtree and view name given, specifies two timestamps: the
264 moment of last rendering and expiration time. The filename is automatically
265 generated unique name in the spool directory. The contents type is determined
270 B<monitor_cache.db> is a I<Hash> database used in order to avoid the
271 unneccessary configuration tree walk. The keys are the leaf tokens, and
272 the values are comma-separated monitor names. At each monitor invocation,
273 the confguration timestamp is compared against the last known, and the
274 cache database is rebuilt if needed.
276 B<monitor_alarms.db> is a I<Hash> database that keeps alarm status information
277 from previous runs of Monitor, with the keys and values as follows:
281 =item * E<lt>mnameE<gt>:E<lt>pathE<gt> --
282 E<lt>t_setE<gt>:E<lt>t_expiresE<gt>:E<lt>statusE<gt>:
283 E<lt>t_last_changeE<gt>
285 Key consists of the monitor name and leaf path. In the value, B<t_set>
286 is the time when the alarm was raised. If two subsequent runs of Monitor
287 raise the same alarm, B<t_set> does not change. B<t_expires> is the
288 timestamp that shows when it's still important to keep the entry after the
289 alarm is cleared. B<status> is 1 if the alarm is active, and 0 otherwise.
290 B<t_last_change> is the timestamp of last status change.
292 When B<status> is 1, the record is kept regardless of timestamps.
293 When B<status> is 0, and the current time is more than B<t_expires>,
294 the record is not reliable and may be deleted by Monitor.
298 B<collector_tokens_X_Y.db> is a I<Hash> database used in order to avoid the
299 unneccessary configuration tree walk. X is the collector instance number, and
300 Y is the datasource configuration instance number.
301 Keys and values are as follows:
305 =item * E<lt>tokenE<gt> -- E<lt>periodE<gt>:E<lt>offsetE<gt>
307 For each leaf token, period and time offset values are stored.
312 B<scheduler_stats.db> is a I<Btree> database which stores the runtime
313 statistics of Scheduler tasks. Each key is of structure
314 B<E<lt>tasknameE<gt>:E<lt>periodE<gt>:E<lt>offsetE<gt>:E<lt>variableE<gt>>,
315 and the value is a number representing the current value of the variable.
316 Depending on variable purpose, the number is floating point or integer.
319 B<users.db> is a I<Hash> database containing user details, passwords,
320 and group membership:
324 =item * ua:E<lt>uidE<gt>:E<lt>attrE<gt> -- E<lt>valueE<gt>
326 User attributes, such as C<cn> (Common name) or C<userPassword>, are stored
327 here. For each user, there is a record consisting of the attribute C<uid>,
328 with the value equal to the user identifier.
330 =item * uA:E<lt>uidE<gt> -- E<lt>attrE<gt>, ...
332 Comma-separated list of attribute names for the given user.
334 =item * gm:E<lt>uidE<gt> -- E<lt>groupE<gt>, ...
336 For each user ID, stores the comma-separated list of groups it belongs to.
338 =item * ga:E<lt>groupE<gt>:E<lt>attrE<gt> -- E<lt>valueE<gt>
340 Group attributes, such as group description.
342 =item * gA:E<lt>groupE<gt> -- E<lt>attrE<gt>, ...
344 Comma-separated list of attribute names for the given group.
346 =item * G: -- E<lt>groupE<gt>, ...
353 B<acl.db> is a I<Hash> database containing group privileges information:
357 =item * u:E<lt>groupE<gt>:E<lt>objectE<gt>:E<lt>privilegeE<gt> -- 1
359 The entry exists if and only if the group members have this privilege
360 over the object given. Most common privilege is C<DisplayTree>, where
361 the object is the tree name.
366 B<serviceid_params.db> is a I<Btree> database containing properties
367 for each Service ID (exported collector information, usually stored in
372 =item * a: E<lt>serviceidE<gt>,...
374 Lists all known service IDs
376 =item * t:E<lt>treeE<gt> -- E<lt>serviceidE<gt>,...
378 Lists service IDs exported by a given datasource tree.
380 =item * p:E<lt>serviceidE<gt>:E<lt>paramE<gt> -- E<lt>valueE<gt>
382 Parameter value for a given service ID. Mandatory parameters are:
383 C<tree>, C<token>, C<dstype>. Optional: C<units>.
385 =item * P:E<lt>serviceidE<gt> -- E<lt>paramE<gt>, ...
387 List of parameter names for a service ID.
392 B<searchwords.db> is a I<Btree> database with DB_DUP and DB_DUPSORT flags.
393 It contains the search strings for the given tree:
397 =item * E<lt>keywordE<gt> -- E<lt>pathE<gt>[:E<lt>paramE<gt>]
399 For a given keyword, refer to a path of a node that contains this word.
400 If the node name matches the keyword, the I<param> element
401 is omitted. Otherwise it refers to the parameter that matches the keyword.
407 B<globsearchwords.db> is a I<Btree> database with DB_DUP and DB_DUPSORT flags.
408 It contains the search strings for all trees:
412 =item * E<lt>keywordE<gt> -- E<lt>treeE<gt>:E<lt>pathE<gt>[:E<lt>paramE<gt>]
414 For a given keyword, refer to a path of a node that contains this word.
415 If the node name matches the keyword, the I<param> element
416 is omitted. Otherwise it refers to the parameter that matches the keyword.
421 B<snmp_failures_X.db> is a I<Btree> database containing SNMP collector
422 failures information for a given collector instance for a tree.
426 =item * c:E<lt>counterE<gt> -- E<lt>NE<gt>
428 A counter with a name. Known names: I<unreachable>, I<removed>.
431 =item * h:E<lt>hosthashE<gt> -- E<lt>failureE<gt>:E<lt>timestampE<gt>
433 SNMP host failure information. Hosthash is a concatenation of hostname, UDP
434 port, and SNMP community, separated by "|". Known failures: I<unreachable>,
435 I<removed>. Timestamp is a UNIX time of the event.
437 =item * m:E<lt>hosthashE<gt> -- E<lt>pathE<gt>:E<lt>timestampE<gt>
439 MIB failures (I<noSuchObject>, I<noSuchInstance>, and I<endOfMibView>)
440 for a given host, with the tree path of their occurence and the UNIX timestamp.
442 =item * M:E<lt>hosthashE<gt> -- E<lt>NE<gt>
444 Count of MIB failures per SNMP host.
454 =head2 Modular Structure
456 The Torrus framework consists of several functional modules:
460 =item * Configuration management
462 Once the configuration XML files get changed, the configuration compiler
463 should be run manually. This guarantees that the actual framework
464 configuration is changed only when the files are ready.
466 The configuration management module provides access methods for
467 enumeration and enquery of the configuratin objects.
469 =item * Data Collector module
471 Collector program runs as a separate process for each datasource tree.
472 Upon startup, it first runs all registered collectors. After that,
473 the collectors are grouped depending on period and time offset, and launched
474 periodically at the moments defined by formula:
476 time + period - (time mod period) + timeoffset
478 The datasources are grouped by collector type.
479 For SNMP collector type, the datasources are grouped by host.
480 SNMP requests are sent in non-blocking mode (see Net::SNMP Perl module
483 For each SNMP host, system uptime is verified. For RRD datasource types
484 "COUNTER", if the device reload is
485 detected, the corresponding RRD file is updated with "undefined"
486 value at the calculated moment of reload.
488 =item * Data threshold monitoring
490 This module performs the monitoring tasks periodically, based on each
491 monitored leaf schedule.
492 It checks the conditions for each leaf having a monitor.
493 In case of the alarm, it executes the action instructions synchronously.
495 =item * Rendering module
497 Upon a request, this module generates the graph and HTML files for the
498 requested view and its subviews. It first checks availability of
499 cached objects and avoids unneeded regeneration. It must be possible
500 to force the renderer to flush the cache.
502 =item * Web interface module
504 Web interface module passes the Renderer output to an HTTP client.
511 Copyright (c) 2002-2005 Stanislav Sinyagin ssinyagin@yahoo.com