1 # BEGIN BPS TAGGED BLOCK {{{
5 # This software is Copyright (c) 1996-2014 Best Practical Solutions, LLC
6 # <sales@bestpractical.com>
8 # (Except where explicitly superseded by other copyright notices)
13 # This work is made available to you under the terms of Version 2 of
14 # the GNU General Public License. A copy of that license should have
15 # been provided with this software, but in any event can be snarfed
18 # This work is distributed in the hope that it will be useful, but
19 # WITHOUT ANY WARRANTY; without even the implied warranty of
20 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
21 # General Public License for more details.
23 # You should have received a copy of the GNU General Public License
24 # along with this program; if not, write to the Free Software
25 # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA
26 # 02110-1301 or visit their web page on the internet at
27 # http://www.gnu.org/licenses/old-licenses/gpl-2.0.html.
30 # CONTRIBUTION SUBMISSION POLICY:
32 # (The following paragraph is not intended to limit the rights granted
33 # to you to modify and distribute this software under the terms of
34 # the GNU General Public License and is only of importance to you if
35 # you choose to contribute your changes and enhancements to the
36 # community by submitting them to Best Practical Solutions, LLC.)
38 # By intentionally submitting any modifications, corrections or
39 # derivatives to this work, or any other work intended for use with
40 # Request Tracker, to Best Practical Solutions, LLC, you confirm that
41 # you are the copyright holder for those contributions and you grant
42 # Best Practical Solutions, LLC a nonexclusive, worldwide, irrevocable,
43 # royalty-free, perpetual, license to use, copy, create derivative
44 # works based on those contributions, and sublicense and distribute
45 # those contributions and any derivatives thereof.
47 # END BPS TAGGED BLOCK }}}
58 RT::Shredder - Permanently wipeout data from RT
65 rt-shredder --force --plugin 'Tickets=query,Queue="General" and Status="deleted"'
69 RT::Shredder is extension to RT which allows you to permanently wipeout
70 data from the RT database. Shredder supports the wiping of almost
71 all RT objects (Tickets, Transactions, Attachments, Users...).
74 =head2 "Delete" vs "Wipeout"
76 RT uses the term "delete" to mean "deactivate". To avoid confusion,
77 RT::Shredder uses the term "Wipeout" to mean "permanently erase" (or
78 what most people would think of as "delete").
81 =head2 Why do you want this?
83 Normally in RT, "deleting" an item simply deactivates it and makes it
84 invisible from view. This is done to retain full history and
85 auditability of your tickets. For most RT users this is fine and they
86 have no need of RT::Shredder.
88 But in some large and heavily used RT instances the database can get
89 clogged up with junk, particularly spam. This can slow down searches
90 and bloat the size of the database. For these users, RT::Shredder
91 allows them to completely clear the database of this unwanted junk.
93 An additional use of Shredder is to obliterate sensitive information
94 (passwords, credit card numbers, ...) which might have made their way
98 =head2 Command line tools (CLI)
100 L<rt-shredder> is a program which allows you to wipe objects from
101 command line or with system tasks scheduler (cron, for example).
102 See also 'rt-shredder --help'.
105 =head2 Web based interface (WebUI)
107 Shredder's WebUI integrates into RT's WebUI. You can find it in the
108 Configuration->Tools->Shredder tab. The interface is similar to the
109 CLI and gives you the same functionality. You can find 'Shredder' link
110 at the bottom of tickets search results, so you could wipeout tickets
111 in the way similar to the bulk update.
114 =head1 DATA STORAGE AND BACKUPS
116 Shredder allows you to store data you wiped in files as scripts with SQL
119 =head3 Restoring from backup
121 Should you wipeout something you did not intend to the objects can be
122 restored by using the storage files. These files are a simple set of
123 SQL commands to re-insert your objects into the RT database.
125 1) Locate the appropriate shredder SQL dump file. In the WebUI, when
126 you use shredder, the path to the dump file is displayed. It also
127 gives the option to download the dump file after each wipeout. Or
128 it can be found in your C<$ShredderStoragePath>.
130 2) Load the shredder SQL dump into your RT database. The details will
131 be different for each database and RT configuration, consult your
132 database manual and RT config. For example, in MySQL...
134 mysql -u your_rt_user -p your_rt_database < /path/to/rt/var/data/shredder/dump.sql
136 That's it.i This will restore everything you'd deleted during a
137 shredding session when the file had been created.
141 =head2 $DependenciesLimit
143 Shredder stops with an error if the object has more than
144 C<$DependenciesLimit> dependencies. For example: a ticket has 1000
145 transactions or a transaction has 1000 attachments. This is protection
146 from bugs in shredder from wiping out your whole database, but
147 sometimes when you have big mail loops you may hit it.
149 Defaults to 1000. To change this (for example, to 10000) add the
150 following to your F<RT_SiteConfig.pm>:
152 Set( $DependenciesLimit, 10_000 );>
155 =head2 $ShredderStoragePath
157 Directory containing Shredder backup dumps; defaults to
158 F</opt/rt4/var/data/RT-Shredder> (assuming an /opt/rt4 installation).
160 To change this (for example, to /some/backup/path) add the following to
161 your F<RT_SiteConfig.pm>:
163 Set( $ShredderStoragePath, "/some/backup/path" );>
165 Be sure to specify an absolute path.
167 =head1 Database Indexes
169 We have found that the following indexes significantly speed up
170 shredding on most databases.
172 CREATE INDEX SHREDDER_CGM1 ON CachedGroupMembers(MemberId, GroupId, Disabled);
173 CREATE INDEX SHREDDER_CGM2 ON CachedGroupMembers(ImmediateParentId,MemberId);
174 CREATE INDEX SHREDDER_CGM3 on CachedGroupMembers (Via, Id);
176 CREATE UNIQUE INDEX SHREDDER_GM1 ON GroupMembers(MemberId, GroupId);
178 CREATE INDEX SHREDDER_TXN1 ON Transactions(ReferenceType, OldReference);
179 CREATE INDEX SHREDDER_TXN2 ON Transactions(ReferenceType, NewReference);
180 CREATE INDEX SHREDDER_TXN3 ON Transactions(Type, OldValue);
181 CREATE INDEX SHREDDER_TXN4 ON Transactions(Type, NewValue)
183 CREATE INDEX SHREDDER_ATTACHMENTS1 ON Attachments(Creator);
185 =head1 INFORMATION FOR DEVELOPERS
189 L<RT::Shredder> is an extension to RT which adds shredder methods to
190 RT objects and classes. The API is not well documented yet, but you
191 can find usage examples in L<rt-shredder> and the
192 F<lib/t/regression/shredder/*.t> test files.
194 However, here is a small example that do the same action as in CLI
195 example from L</SYNOPSIS>:
198 RT::Shredder::Init( force => 1 );
199 my $deleted = RT::Tickets->new( RT->SystemUser );
200 $deleted->{'allow_deleted_search'} = 1;
201 $deleted->LimitQueue( VALUE => 'general' );
202 $deleted->LimitStatus( VALUE => 'deleted' );
203 while( my $t = $deleted->Next ) {
208 =head2 RT::Shredder class' API
210 L<RT::Shredder> implements interfaces to objects cache, actions on the
211 objects in the cache and backups storage.
215 our $VERSION = '0.04';
220 # I can't use 'use lib' here since it breakes tests
221 # because test suite uses old RT::Shredder setup from
224 ### after: push @INC, qw(@RT_LIB_PATH@);
225 use RT::Shredder::Constants;
226 use RT::Shredder::Exceptions;
230 require RT::Shredder::Record;
232 require RT::Shredder::ACE;
233 require RT::Shredder::Attachment;
234 require RT::Shredder::CachedGroupMember;
235 require RT::Shredder::CustomField;
236 require RT::Shredder::CustomFieldValue;
237 require RT::Shredder::GroupMember;
238 require RT::Shredder::Group;
239 require RT::Shredder::Link;
240 require RT::Shredder::Principal;
241 require RT::Shredder::Queue;
242 require RT::Shredder::Scrip;
243 require RT::Shredder::ScripAction;
244 require RT::Shredder::ScripCondition;
245 require RT::Shredder::Template;
246 require RT::Shredder::ObjectCustomFieldValue;
247 require RT::Shredder::Ticket;
248 require RT::Shredder::Transaction;
249 require RT::Shredder::User;
252 our @SUPPORTED_OBJECTS = qw(
267 ObjectCustomFieldValue
277 RT::Shredder::Init( %default_options );
279 C<RT::Shredder::Init()> should be called before creating an
280 RT::Shredder object. It iniitalizes RT and loads the RT
283 %default_options are passed to every C<<RT::Shredder->new>> call.
298 my $shredder = RT::Shredder->new(%options);
300 Construct a new RT::Shredder object.
302 There currently are no %options.
309 my $self = bless( {}, ref $proto || $proto );
317 $self->{'opt'} = { %opt, @_ };
318 $self->{'cache'} = {};
319 $self->{'resolver'} = {};
320 $self->{'dump_plugins'} = [];
323 =head4 CastObjectsToRecords( Objects => undef )
325 Cast objects to the C<RT::Record> objects or its ancesstors.
326 Objects can be passed as SCALAR (format C<< <class>-<id> >>),
327 ARRAY, C<RT::Record> ancesstors or C<RT::SearchBuilder> ancesstor.
329 Most methods that takes C<Objects> argument use this method to
330 cast argument value to list of records.
332 Returns an array of records.
336 my @objs = $shredder->CastObjectsToRecords(
337 Objects => [ # ARRAY reference
338 'RT::Attachment-10', # SCALAR or SCALAR reference
339 $tickets, # RT::Tickets object (isa RT::SearchBuilder)
340 $user, # RT::User object (isa RT::Record)
346 sub CastObjectsToRecords
349 my %args = ( Objects => undef, @_ );
352 my $targets = delete $args{'Objects'};
354 RT::Shredder::Exception->throw( "Undefined Objects argument" );
357 if( UNIVERSAL::isa( $targets, 'RT::SearchBuilder' ) ) {
358 #XXX: try to use ->_DoSearch + ->ItemsArrayRef in feature
359 # like we do in Record with links, but change only when
360 # more tests would be available
361 while( my $tmp = $targets->Next ) { push @res, $tmp };
362 } elsif ( UNIVERSAL::isa( $targets, 'RT::Record' ) ) {
364 } elsif ( UNIVERSAL::isa( $targets, 'ARRAY' ) ) {
365 foreach( @$targets ) {
366 push @res, $self->CastObjectsToRecords( Objects => $_ );
368 } elsif ( UNIVERSAL::isa( $targets, 'SCALAR' ) || !ref $targets ) {
369 $targets = $$targets if ref $targets;
370 my ($class, $id) = split /-/, $targets;
371 RT::Shredder::Exception->throw( "Unsupported class $class" )
372 unless $class =~ /^\w+(::\w+)*$/;
373 $class = 'RT::'. $class unless $class =~ /^RTx?::/i;
374 eval "require $class";
375 die "Couldn't load '$class' module" if $@;
376 my $obj = $class->new( RT->SystemUser );
377 die "Couldn't construct new '$class' object" unless $obj;
379 unless ( $obj->id ) {
380 $RT::Logger->error( "Couldn't load '$class' object with id '$id'" );
381 RT::Shredder::Exception::Info->throw( 'CouldntLoadObject' );
383 die "Loaded object has different id" unless( $id eq $obj->id );
386 RT::Shredder::Exception->throw( "Unsupported type ". ref $targets );
393 =head4 PutObjects( Objects => undef )
395 Puts objects into cache.
397 Returns array of the cache entries.
399 See C<CastObjectsToRecords> method for supported types of the C<Objects>
407 my %args = ( Objects => undef, @_ );
410 for( $self->CastObjectsToRecords( Objects => delete $args{'Objects'} ) ) {
411 push @res, $self->PutObject( %args, Object => $_ )
417 =head4 PutObject( Object => undef )
419 Puts record object into cache and returns its cache entry.
421 B<NOTE> that this method support B<only C<RT::Record> object or its ancesstor
422 objects>, if you want put mutliple objects or objects represented by different
423 classes then use C<PutObjects> method instead.
430 my %args = ( Object => undef, @_ );
432 my $obj = $args{'Object'};
433 unless( UNIVERSAL::isa( $obj, 'RT::Record' ) ) {
434 RT::Shredder::Exception->throw( "Unsupported type '". (ref $obj || $obj || '(undef)')."'" );
437 my $str = $obj->_AsString;
438 return ($self->{'cache'}->{ $str } ||= { State => ON_STACK, Object => $obj } );
441 =head4 GetObject, GetState, GetRecord( String => ''| Object => '' )
443 Returns record object from cache, cache entry state or cache entry accordingly.
445 All three methods takes C<String> (format C<< <class>-<id> >>) or C<Object> argument.
446 C<String> argument has more priority than C<Object> so if it's not empty then methods
447 leave C<Object> argument unchecked.
449 You can read about possible states and their meanings in L<RT::Shredder::Constants> docs.
461 if( $args{'String'} && $args{'Object'} ) {
463 Carp::croak( "both String and Object args passed" );
465 return $args{'String'} if $args{'String'};
466 return $args{'Object'}->_AsString if UNIVERSAL::can($args{'Object'}, '_AsString' );
470 sub GetObject { return (shift)->GetRecord( @_ )->{'Object'} }
471 sub GetState { return (shift)->GetRecord( @_ )->{'State'} }
475 my $str = $self->_ParseRefStrArgs( @_ );
476 return $self->{'cache'}->{ $str };
479 =head3 Dependencies resolvers
481 =head4 PutResolver, GetResolvers and ApplyResolvers
483 TODO: These methods have no documentation.
496 unless( UNIVERSAL::isa( $args{'Code'} => 'CODE' ) ) {
497 die "Resolver '$args{Code}' is not code reference";
502 $self->{'resolver'}->{ $args{'BaseClass'} } ||= {}
503 )->{ $args{'TargetClass'} || '' } ||= []
505 unshift @$resolvers, $args{'Code'};
519 if( $args{'TargetClass'} && exists $self->{'resolver'}->{ $args{'BaseClass'} }->{ $args{'TargetClass'} } ) {
520 push @res, @{ $self->{'resolver'}->{ $args{'BaseClass'} }->{ $args{'TargetClass'} || '' } };
522 if( exists $self->{'resolver'}->{ $args{'BaseClass'} }->{ '' } ) {
523 push @res, @{ $self->{'resolver'}->{ $args{'BaseClass'} }->{''} };
532 my %args = ( Dependency => undef, @_ );
533 my $dep = $args{'Dependency'};
535 my @resolvers = $self->GetResolvers(
536 BaseClass => $dep->BaseClass,
537 TargetClass => $dep->TargetClass,
540 unless( @resolvers ) {
541 RT::Shredder::Exception::Info->throw(
543 error => "Couldn't find resolver for dependency '". $dep->AsString ."'",
548 BaseObject => $dep->BaseObject,
549 TargetObject => $dep->TargetObject,
550 ) foreach @resolvers;
559 foreach my $cache_val ( values %{ $self->{'cache'} } ) {
560 next if $cache_val->{'State'} & (WIPED | IN_WIPING);
561 $self->Wipeout( Object => $cache_val->{'Object'} );
570 die "Couldn't begin transaction" unless $RT::Handle->BeginTransaction;
571 $mark = $self->PushDumpMark or die "Couldn't get dump mark";
572 $self->_Wipeout( @_ );
573 $self->PopDumpMark( Mark => $mark );
574 die "Couldn't commit transaction" unless $RT::Handle->Commit;
578 $RT::Handle->Rollback('force');
579 $self->RollbackDumpTo( Mark => $mark ) if $mark;
580 die $error if RT::Shredder::Exception::Info->caught;
581 die "Couldn't wipeout object: $error";
588 my %args = ( CacheRecord => undef, Object => undef, @_ );
590 my $record = $args{'CacheRecord'};
591 $record = $self->PutObject( Object => $args{'Object'} ) unless $record;
592 return if $record->{'State'} & (WIPED | IN_WIPING);
594 $record->{'State'} |= IN_WIPING;
595 my $object = $record->{'Object'};
597 $self->DumpObject( Object => $object, State => 'before any action' );
599 unless( $object->BeforeWipeout ) {
600 RT::Shredder::Exception->throw( "BeforeWipeout check returned error" );
603 my $deps = $object->Dependencies( Shredder => $self );
605 WithFlags => DEPENDS_ON | VARIABLE,
606 Callback => sub { $self->ApplyResolvers( Dependency => $_[0] ) },
608 $self->DumpObject( Object => $object, State => 'after resolvers' );
611 WithFlags => DEPENDS_ON,
612 WithoutFlags => WIPE_AFTER | VARIABLE,
613 Callback => sub { $self->_Wipeout( Object => $_[0]->TargetObject ) },
615 $self->DumpObject( Object => $object, State => 'after wiping dependencies' );
618 $record->{'State'} |= WIPED; delete $record->{'Object'};
619 $self->DumpObject( Object => $object, State => 'after wipeout' );
622 WithFlags => DEPENDS_ON | WIPE_AFTER,
623 WithoutFlags => VARIABLE,
624 Callback => sub { $self->_Wipeout( Object => $_[0]->TargetObject ) },
626 $self->DumpObject( Object => $object, State => 'after late dependencies' );
631 sub ValidateRelations
636 foreach my $record( values %{ $self->{'cache'} } ) {
637 next if( $record->{'State'} & VALID );
638 $record->{'Object'}->ValidateRelations( Shredder => $self );
642 =head3 Data storage and backups
644 =head4 GetFileName( FileName => '<ISO DATETIME>-XXXX.sql', FromStorage => 1 )
646 Takes desired C<FileName> and flag C<FromStorage> then translate file name to absolute
649 * Default value of the C<FileName> option is C<< <ISO DATETIME>-XXXX.sql >>;
651 * if C<FileName> has C<XXXX> (exactly four uppercase C<X> letters) then it would be changed with digits from 0000 to 9999 range, with first one free value;
653 * if C<FileName> has C<%T> then it would be replaced with the current date and time in the C<YYYY-MM-DDTHH:MM:SS> format. Note that using C<%t> may still generate not unique names, using C<XXXX> recomended.
655 * if C<FromStorage> argument is true (default behaviour) then result path would always be relative to C<StoragePath>;
657 * if C<FromStorage> argument is false then result would be relative to the current dir unless it's already absolute path.
659 Returns an absolute path of the file.
662 # file from storage with default name format
663 my $fname = $shredder->GetFileName;
665 # file from storage with custom name format
666 my $fname = $shredder->GetFileName( FileName => 'shredder-XXXX.backup' );
668 # file with path relative to the current dir
669 my $fname = $shredder->GetFileName(
671 FileName => 'backups/shredder.sql',
674 # file with absolute path
675 my $fname = $shredder->GetFileName(
677 FileName => '/var/backups/shredder-XXXX.sql'
685 my %args = ( FileName => '', FromStorage => 1, @_ );
688 my $file = $args{'FileName'} || '%t-XXXX.sql';
689 if( $file =~ /\%t/i ) {
691 my $date_time = POSIX::strftime( "%Y%m%dT%H%M%S", gmtime );
692 $file =~ s/\%t/$date_time/gi;
695 # convert to absolute path
696 if( $args{'FromStorage'} ) {
697 $file = File::Spec->catfile( $self->StoragePath, $file );
698 } elsif( !File::Spec->file_name_is_absolute( $file ) ) {
699 $file = File::Spec->rel2abs( $file );
703 if( $file =~ /XXXX[^\/\\]*$/ ) {
704 my( $tmp, $i ) = ( $file, 0 );
708 $tmp =~ s/XXXX([^\/\\]*)$/sprintf("%04d", $i).$1/e;
709 } while( -e $tmp && $i < 9999 );
715 die "File '$file' exists, but is read-only";
718 unless( File::Spec->file_name_is_absolute( $file ) ) {
719 $file = File::Spec->rel2abs( $file );
723 my $dir = File::Spec->join( (File::Spec->splitpath( $file ))[0,1] );
724 unless( -e $dir && -d _) {
725 die "Base directory '$dir' for file '$file' doesn't exist";
728 die "Base directory '$dir' is not writable";
731 die "'$file' is not regular file";
739 Returns an absolute path to the storage dir. See
740 L</$ShredderStoragePath>.
742 See also description of the L</GetFileName> method.
748 return scalar( RT->Config->Get('ShredderStoragePath') )
749 || File::Spec->catdir( $RT::VarPath, qw(data RT-Shredder) );
752 my %active_dump_state = ();
755 my %args = ( Object => undef, Name => 'SQLDump', Arguments => undef, @_ );
757 my $plugin = $args{'Object'};
759 require RT::Shredder::Plugin;
760 $plugin = RT::Shredder::Plugin->new;
761 my( $status, $msg ) = $plugin->LoadByName( $args{'Name'} );
762 die "Couldn't load dump plugin: $msg\n" unless $status;
764 die "Plugin is not of correct type" unless lc $plugin->Type eq 'dump';
766 if ( my $pargs = $args{'Arguments'} ) {
767 my ($status, $msg) = $plugin->TestArgs( %$pargs );
768 die "Couldn't set plugin args: $msg\n" unless $status;
771 my @applies_to = $plugin->AppliesToStates;
772 die "Plugin doesn't apply to any state" unless @applies_to;
773 $active_dump_state{ lc $_ } = 1 foreach @applies_to;
775 push @{ $self->{'dump_plugins'} }, $plugin;
782 my %args = (Object => undef, State => undef, @_);
783 die "No state passed" unless $args{'State'};
784 return unless $active_dump_state{ lc $args{'State'} };
786 foreach (@{ $self->{'dump_plugins'} }) {
787 next unless grep lc $args{'State'} eq lc $_, $_->AppliesToStates;
788 my ($state, $msg) = $_->Run( %args );
789 die "Couldn't run plugin: $msg" unless $state;
793 { my $mark = 1; # XXX: integer overflows?
797 foreach (@{ $self->{'dump_plugins'} }) {
798 my ($state, $msg) = $_->PushMark( Mark => $mark );
799 die "Couldn't push mark: $msg" unless $state;
805 foreach (@{ $self->{'dump_plugins'} }) {
806 my ($state, $msg) = $_->PushMark( @_ );
807 die "Couldn't pop mark: $msg" unless $state;
812 foreach (@{ $self->{'dump_plugins'} }) {
813 my ($state, $msg) = $_->RollbackTo( @_ );
814 die "Couldn't rollback to mark: $msg" unless $state;
824 =head2 Database transactions support
826 Since 0.03_01 RT::Shredder uses database transactions and should be
827 much safer to run on production servers.
831 Mainstream RT doesn't use FKs, but at least I posted DDL script that creates them
832 in mysql DB, note that if you use FKs then this two valid keys don't allow delete
833 Tickets because of bug in MySQL:
835 ALTER TABLE Tickets ADD FOREIGN KEY (EffectiveId) REFERENCES Tickets(id);
836 ALTER TABLE CachedGroupMembers ADD FOREIGN KEY (Via) REFERENCES CachedGroupMembers(id);
838 L<http://bugs.mysql.com/bug.php?id=4042>
840 =head1 BUGS AND HOW TO CONTRIBUTE
842 We need your feedback in all cases: if you use it or not,
843 is it works for you or not.
847 Don't skip C<make test> step while install and send me reports if it's fails.
848 Add your own tests, it's easy enough if you've writen at list one perl script
849 that works with RT. Read more about testing in F<t/utils.pl>.
853 Send reports to L</AUTHOR> or to the RT mailing lists.
857 Many bugs in the docs: insanity, spelling, gramar and so on.
858 Patches are wellcome.
862 Please, see Todo file, it has some technical notes
863 about what I plan to do, when I'll do it, also it
864 describes some problems code has.
868 Since RT-3.7 shredder is a part of the RT distribution.
869 Versions of the RTx::Shredder extension could
870 be downloaded from the CPAN. Those work with older
871 RT versions or you can find repository at
872 L<https://opensvn.csie.org/rtx_shredder>
876 Ruslan U. Zakirov <Ruslan.Zakirov@gmail.com>
880 This program is free software; you can redistribute
881 it and/or modify it under the same terms as Perl itself.
883 The full text of the license can be found in the
888 L<rt-shredder>, L<rt-validator>