1 # webintf.pod - Torrus web interface reference
2 # Copyright (C) 2002 Stanislav Sinyagin
4 # This program is free software; you can redistribute it and/or modify
5 # it under the terms of the GNU General Public License as published by
6 # the Free Software Foundation; either version 2 of the License, or
7 # (at your option) any later version.
9 # This program is distributed in the hope that it will be useful,
10 # but WITHOUT ANY WARRANTY; without even the implied warranty of
11 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
12 # GNU General Public License for more details.
14 # You should have received a copy of the GNU General Public License
15 # along with this program; if not, write to the Free Software
16 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307, USA.
18 # $Id: reporting_setup.pod.in,v 1.1 2010-12-27 00:04:34 ivan Exp $
19 # Stanislav Sinyagin <ssinyagin@yahoo.com>
24 =head1 Torrus Reporting Setup Guide
26 In version 1.0.4, Torrus introduces a new and important functionality.
27 Now it is able to produce reports that may be used in billing or
28 resource planning. The reports are meant to be text output, telling
29 the bandwidth usage or volume: no graphs so far.
31 By default, the reporting functionality is not enabled, and the steps
32 required are described below.
37 The basic term in Torrus reporting is the B<Service ID>.
38 It denotes a single datasource that is represented by a single number.
39 For example, if you want to count input and output interface traffic,
40 this would make two different service IDs. Do not assign service IDs
41 to each and every interface in your network, as it would degrade
42 the performance of your Torrus installation significantly.
43 This functionality is designed for use with important datasources,
44 such as your customers' connection links or and ISP's upstream channels.
46 Each service ID must be B<unique> across the whole Torrus installation,
47 and across the database that stores them (it is possible to use
48 several Torrus installations with the same database).
50 B<Devdiscover>, the SNMP discovery engine, allows now to assign
51 service IDs to network interfaces of your SNMP-enabled devices.
52 However, this does not prevent you from assigning them to other Torrus
53 datasources, as it's done by simple configuration parameters.
55 B<Report>, when generated, is presented by a set of numeric values
56 in the SQL database. Torrus provides also tools for rendering these
57 values into HTML B<output>. In the future, other output formats
58 will be implemented. You can also build your own infrastructure that
59 reads the values from the reports database.
61 The produced reports may, and are primarily developed for using in
62 B<billing> process. As it is stated in the GNU General Public License
63 text, this program is provided "as is" and B<without warranty of any kind>.
64 It is up to the users of Torrus software to rely or not to rely on
65 the generated numeric data, and the Torrus software developers disclaim
66 any responsibility for the data accuracy and usability.
69 B<New in version 1.0.7:> You can assign the list of trees where the reports
70 should be generated and shown. B<Warning:> after changing the destination tree,
71 the compiler may complain about I<duplicate service IDs>. Then you need to
72 stop all the torrus processes, including Apache, and then remove the file
73 F<serviceid_params.db> from Torrus database directory, then recompile the
74 trees and start the processes.
77 =head2 Install Perl modules
79 Install the following Perl modules from CPAN:
82 DBD::mysql or other RDBMS driver
86 On many platforms, DBI is already pre-installed. You need to make sure
87 that appropriate DBD driver is installed for your database engine.
88 The setup was tested with MySQL, SQL syntax is compatible with Postgres,
89 and in theory it should run also with Oracle and probably Sybase or DB2.
90 No idea about MSSQL - if you're brave enough, let me know if it works :)
93 =head2 Enable the External Storage and specify the SQL connection
95 In F<torrus-siteconfig.pl>, add the following lines. The first one
96 tells the collector to use the module for storing the collector results in
97 SQL database. Next lines define the database connection. By default,
98 it refers to the MySQL database named C<torrus> on C<localhost>,
99 with username and password set to C<imiF1oih>.
102 push(@Torrus::Collector::loadModules, 'Torrus::Collector::ExternalStorage');
103 $Torrus::SQL::connections{'Default'}{'dsn'} =
104 'DBI:mysql:database=torrus;host=dbhost.example.com';
105 $Torrus::SQL::connections{'Default'}{'username'} = 'torrus';
106 $Torrus::SQL::connections{'Default'}{'password'} = 'imiF1oih';
108 In addition to the default connection, you can specify different connections
109 for different data structures: for example, keep 5-minutes samples on
110 a bulky storage server, and store the reports on a high-availability cluster.
111 See the comments in F<torrus-config.pl> for more details.
114 =head2 Create SQL tables
116 With your RDBMS client, create the following tables. We assume here that
117 the same database is used for all tables. The SQL syntax is verified with
118 MySQL 4.x and Postgres 8.x. It is brought as much as possible to
119 the standard and platform-independent SQL syntax. Please let me know
120 if it causes problems with your RDBMS.
123 /* Collector export table. It usually grows at several megabytes
124 per month, and is updated every 5 minutes */
125 CREATE TABLE srvexport
127 srv_date DATE NOT NULL, /* date and time of the data sample */
128 srv_time TIME NOT NULL,
129 serviceid VARCHAR(64) NOT NULL, /* unique service ID per counter */
130 value DOUBLE PRECISION NOT NULL, /* collected rate or gauge value */
131 intvl INTEGER NOT NULL /* collection interval -
132 for counter volume calculation */
134 CREATE INDEX srvexp_date ON srvexport (srv_date);
135 CREATE INDEX srvexp_srvid ON srvexport (serviceid);
138 /* Tables for (currently monthly only) report contents.
139 These are updated usually once per month, and read at the moment of
140 rendering the report output (HTML now, PDF or XML or Excel or whatever
143 /* DBIx::Sequence backend, theplatform-independent inplementation
145 CREATE TABLE dbix_sequence_state
147 dataset varchar(50) NOT NULL,
148 state_id INTEGER NOT NULL,
149 CONSTRAINT pk_dbix_sequence PRIMARY KEY (dataset, state_id)
152 CREATE TABLE dbix_sequence_release
154 dataset varchar(50) NOT NULL,
155 released_id INTEGER NOT NULL,
156 CONSTRAINT pk_dbi_release PRIMARY KEY (dataset, released_id)
160 /* Each report is characterized by name, date and time.
161 Monthly reports are automatically assigned 00:00 of the 1st day
162 in the month. The report contains fields for every service ID
163 defined across all datasource trees. */
166 id INTEGER PRIMARY KEY,
167 rep_date DATE NOT NULL, /* Start date of the report */
168 rep_time TIME NOT NULL, /* Start time of the report */
169 reportname VARCHAR(64) NOT NULL, /* Report name, such as MonthlyUsage */
170 iscomplete INTEGER NOT NULL, /* 0 when the report is in progress, */
171 /* 1 when it is ready */
172 UNIQUE(rep_date, rep_time, reportname)
174 CREATE INDEX rep_date_idx ON reports (rep_date);
177 /* Each report contains fields. For each service ID,
178 the report may contain several fields for various statistics.
179 Each field contains information about the units of the value it
181 CREATE TABLE reportfields
183 id INTEGER PRIMARY KEY,
185 name VARCHAR(64) NOT NULL, /* name of the field, such as AVG or MAX */
186 serviceid VARCHAR(64) NOT NULL, /* service ID */
187 value DOUBLE PRECISION NOT NULL, /* Numeric value */
188 units VARCHAR(64) NOT NULL DEFAULT '', /* Units, such as bytes or Mbps */
189 UNIQUE (rep_id, name, serviceid)
193 =head2 Devdiscover parameters
195 Currently devdiscover allows you to assign service IDs to network interfaces'
196 input and output traffic counters. Other types of datasources may be
197 implemented in the future.
201 =item * C<RFC2863_IF_MIB::external-serviceid>
203 This discovery parameter specifies which service IDs are assigned to which
204 interface names. The interface names whould be specified in the form of
205 their subtree names in Torrus configuration. The example below is
206 typical for a Cisco IOS router. The value of the parameter consists of
207 comma-separated strings. The values in each string are separated by colons,
208 and they correspond to the service ID, interface name, counter type,
209 and optional destination tree names separated by the pipe symbol (|).
210 The interface name can be prefixed by hostname and slash (/),
211 the same way as in C<RFC2863_IF_MIB::tokenset-members>.
212 All strings are case-sensitive. Three counter types are supported: C<In>,
213 C<Out>, and C<Both>. When set to C<Both>, the service ID is appended with
214 C<_IN> or C<_OUT> postfix accordingly, for input and output byte counters.
216 <!-- global parameter that will match specific routers -->
217 <param name="RFC2863_IF_MIB::external-serviceid">
218 CUSTOMERONE:nyc01br01/GigabitEthernet9_2_1:Both,
219 CUSTOMERTWO:wdc01br02/GigabitEthernet6_3_1:Both,
223 <param name="snmp-host" value="lsn01br01"/>
224 <!-- host-specfic parameter -->
225 <param name="RFC2863_IF_MIB::external-serviceid">
226 UPSTREAM1_IN:FastEthernet0_0_0:In:upstreams,
227 UPSTREAM1_OUT:FastEthernet0_0_0:Out:upstreams,
228 UPSTREAM2:GigabitEthernet0_1_1:Both:upstreams,
229 CUST1:GigabitEthernet0_2_2:Both:customers|cust1,
233 =item * C<CiscoIOS_MacAccounting::external-serviceid>
235 The same format as for the parameter listed above, but instead of
236 interface names, you can specify the MAC accounting peer, in one
237 of the following formats: AS number (I<AS12345>), IP address, or peer's MAC
238 address (I<0x0003319c4200>).
244 =head2 Torrus XML configuration parameters
246 You can skip this section if Devdiscover provides all desired functionality.
247 It explains parameters additional to those described in I<Torrus XML
248 Configuration Guide>.
250 The collector's ExternalStorage module is designed for storing the data to
251 a generic external storage, and the default external storage is the SQL
256 =item * C<storage-type>
258 Mandatory parameter for datasource type C<collector>. Comma-separated list
259 of storage types. The collected value is duplicated on every storage listed.
260 Supported values are: C<rrd>, C<ext>. For C<ext> (external storage),
261 see the I<Reporting Setup Guide>.
263 =item * C<ext-dstype>
265 Mandatory datasource type to be used in external storage. Accepted
266 values are: C<GAUGE>, C<COUNTER32>, C<COUNTER64>.
268 =item * C<ext-service-id>
270 Mandatory service ID for the external storage.
272 =item * C<ext-service-units>
274 Optional units for the collected value. The only supported value is
277 =item * C<ext-service-trees>
279 (B<New in version 1.0.7>) Optional list of comma-separated tree names.
280 If specified, the report generator will produce report in corresponding trees.
281 By default it's the tree which runs the collector process.
287 =head2 Enable displaying of the reports in the web interface
289 First, enable the reports displaying in <torrus-siteconfig.pl>:
291 $Torrus::Renderer::displayReports = 1;
293 Second, configure the ACL for your users that are allowed to see the reports
294 in the web interface:
296 torrus acl --modgroup=admin --permit=DisplayReports --for=mytree
298 In this example, members of the group C<admin> will be prompted
299 with the C<[Reports]> shortcut in the web interface's bottom of the page
300 for a given tree. For easier setup, the tree name may be replaced with
301 asterisk (*) to allow this option for all trees.
304 =head2 Generate reports
306 Report generation is usually a CPU-intensive task. A monthly report calculation
307 for one service ID may take several dozens of seconds of CPU time.
308 This uit's recommended to use the C<nice> command to lower the process
311 The C<torrus genreport> (or simply C<torrus report>) command-line utility
312 is designed for two different tasks: calculation of a single report
313 for the period chosen, and generation of output HTML for all reports
316 The example below generates the monthly usage report for September 2005:
318 nice torrus report --report=MonthlyUsage --date=2005-09-01 --debug
320 The next example generates HTML output for all reports that are found
323 nice torrus report --genhtml --tree=customers
325 It makes sence to set up a monthly cron job and generate the reports on
326 the first day of every month, with the command like follows:
328 nice torrus report --report=MonthlyUsage \
329 --date=`perl -e 'use Date::Format;
330 print time2str("%Y-%m-%d", time()-1728000)'`
332 The HTML output is optimized for printing, and is quite easy
333 to navigate. The overview page provides the list of years. Clicking to the
334 year leads you into the list of monthly reports available.
335 Each monthly report consists of a table for each report name.
336 For C<MonthlyUsage>, the data is organized in five columns:
337 the service ID, average monthly rate, 95th percentile of the rates,
338 maximum rate throughout the month, unavailable samples (how many 5-minutes
339 intervals are missed in the collected data), and monthly volume (which
340 is roughly less than the actual volume by the percentage of missed samples).
341 Also clicking a service ID leads to its monthly report throughout the year.
344 =head2 Getting the sum or maximum value from several service IDs
346 A new utility has been sponsored by nexellent ag (www.nexellent.ch), a
347 managed hosting solution provider near Zurich.
349 The utility C<srvderive> can be used to generate a
350 new service ID which would contain sum or maximum of values of other
351 service IDs. You would usually run this utlity monthly, just before generating
354 torrus srvderive --verbose --start=20080801 --month \
355 --out=JSMITH_SUM_OUT --func=SUM JSMITH01_OUT JSMITH02_OUT
357 torrus srvderive --verbose --start=20080801 --month \
358 --out=JSMITH_SUM_IN --func=SUM JSMITH01_IN JSMITH02_IN
365 Copyright (c) 2005-2008 Stanislav Sinyagin E<lt>ssinyagin@yahoo.comE<gt>