[freeside.git] / TODO

$Id: TODO,v 1.56 2000-12-26 23:51:40 ivan Exp $

If you are interested in helping with any of these, please join the
*development* mailing list (send a blank message to
ivan-freeside-devel-subscribe@sisd.com) to avoid duplication of effort.

---

Use this for email checking:
libemail-valid-perl - Check validity of Internet email addresses
.                                                                
This module determines whether an email address is well-formed, and 
optionally, whether a mail host exists for the domain.


wishlist from drenalin@ultimanet.com:
* delete button for customers
- 15th of the month billing
- field for customer referrals (naming who it is) and automatically crediting
+that account
- ability to edit referrals
- catch expired credit cards and notify via email when they are expiring
* show passwords
- set default shell to /bin/false when adding ppp account
- import list of POPs from Megapop (see www.ultimanet.com and click on locations
+from text in index page)


wishlist wrt projects/consulting from jivko@ijs.com:
>The other thing, which is the serious part, is the following: We do not
>offer dial-up services and that part of the system does not have to work,
>but we have many customer, for which we do both hosting and consulting work.
.
>The hosting can be handled by freeside fine, but in the little time I spent 
>on it I could not see how exactly to solve the consulting problem. What I
>would love to see there is a way to create 'projects' for each customer and
>ability to follow up on those (I mean to be able to add new comments,
>descriptions, progress updates etc.). You already have the feature, which
>allows multiple people to access the system with different access rights so
>it should be possible to make it keep track of who and when updated given
>'project' so we could have the developers access the database and update
>the projects they are working on.
.
Hmm.  Is this accurate:  Each project is related to a single customer, and
zero or more packages (billing items).  Each project has a name, due date,    
and zero or more log entries consisting of date, user, and a text field
(for "new comments, descriptions, progress updates etc." - would you like
anything more specific?)

In conjunction with that it looks like you need the ability to create
"one-off" packages on the fly for arbitrary charges.

> Additionally it would be nice to have a page, where all open projects can
> be listed and not only that but be able list them by date too. (Say I want
> to find out which projects are due today or two days from today, or which
> projects are running late)
.
Browse/search project by date and customer.
.
> Finally it would be real nice if the billing script could add the cost for
> all completed projects (or work done by the hour) to the monthly bill along   
> with the hosting information and using the description of the work from the
> 'project' table. 
.
Can you be more specific about your needs here?
.
> This is a lot of work, unless you disagree :-), and I would not expect you
> to do all of it - we can work on it too - but I would hope for you to lead 
> the effort.
.
I'm trying to get a handle on specifically what you need.
.
> Oh, and there is one more thing, but that should be simple to do. I would
> like to be able to give the customers access to selected information from
> their account. I want them to be able to monitor the progress of their
> projects and the status of their accounts. 
.
Ok.
.
> Does all this make sense? It must be close to what you need at your office
> though or at least I think so.
.
Yes, but we're small and have been tracking projects manually.
.
.
>You may have this already .. what I have in mind is the following. Say you
>have a customer for which you do hosting and some consulting (CGI, JAVA
>etc) At the end of the month you need to bill the customer for the hosting,
>35hours of Java programming and 10 hours of HTML authoring. The last two
>are not because the project is over but because the work was done during
>that month.
>
>So the way I see it, and it could be wrong, is that during the month the
>people who work on particular project enter their comments and time spent
>on the project. At the end of the month the billing script generates
>invoices in which besides the hosting charges there are also charges for
>work done during the period.
>
>Currently we do this in a very stupid way by maintaining a customer file,
>in which a script saves things to be invoiced. At the end of the month a
>script goes through these files and generates invoices with all items from
>the files, which have not been flagged as 'paid'. The convenience of this
>is that no one needs to worry about the invoices containing the items they
>need to contain.



first package select field in edit/cust_main.cgi isn't sticky on errors, yuck

> 1. A Web Form to the user get his account added automatically . The
> /etc/raddb/users and /etc/passwd would be updated automatically (these
> file are on the same machine Freeside is). I guess the the Add
> Customer
> page with some customization could be the work.
> 2. A Canceling Account Web Form available to user cancel his account   
> at
> any time he wants.
> 3. A Password Changing Web Form where the user could change your
> password by sending your username, old and new password.
> 4. Additional POP Accounts Creation where the user supply only his
> main
> username/password (probably provided from a PPP Account already
> created)
> and his POP username/password for this specific account.
(actually need something more general i guess - probably need some way to say
which accounts (svc_acct) can log in and make changes per customer (cust_main) )

this is awfully vauge, perhaps email for more info?  don't want to alientate
someone with accounting experience, but i just don't have time right now.
.
Hi Ivan,
.
Thanks for the information. It took me a little while to figure it out, but I
+was
able to create an invoice. As a lamer and an accountant, may I make a few
suggestions?
.
On the Customer view page:
1) Layout the information in the manner the invoicing process is conducted.
    a) Customer information                                                
    b) Package Ordering    
    c)  Add a "Create & Edit Invoice" section - this allows auditing of  the
invoice     before it is processed.
    d) then put the billing section to actually process the invoice
    e) then put payment history information                        
.
Thanks,
Mark Roberts <mroberts@iopenenterprises.com>


new overdue code might be overzealous.  test carefully.

Port Freeside to use the modules that were abstracted from it:
Net::SSH, Net::SCP, DBIx::DataSource and DBIx::DBSchema, at least.

"first package" and email invoice (?) not sticky on errors in new/edit customer
screen.

http://www.ipmeter.com/ integration would be useful

http://tangram.sourceforge.net/
Tie::DBI

mmm, http://pootpoot.com/~dlowe/DBIx-Table/

The cybercash links in htdocs/docs/config.html are b0rken.

Yes.  Which is what I've been trying to tell you.  (destination user,
destination domainname) would be represented by the svcnum of a record in
svc_acct.  (In retrospect, using the uid instead of the svcnum was a bad
choice).  (domuser, domain name) would be part of the svc_acct_sm record,
as domuser and domsvc.
. 
Upon further consideration, I'll probably eliminate the svc_acct_sm table
completely, and just add a field for the svcnum of a domain and the svcnum
of a destination mailbox to svc_acct (or perhaps a one-to-many
relationship - multiple svcnum(s) for multiple destination mailboxes.
hmm.)

> > > > Longer-term, I need to do something about the length of the             
> > > > column names -                                                          
> > > > The SQL1992 standard defines 18 character column names,                 
> > > > which would be a                                                        
> > > > reasonable goal.                                                        
> > >                                                                           
> > > Maybe the thing to do would be to separate these out to separate          
> > > tables.                                                                   
> > > then we could simply use something like id, attrib_name,                  
> > attrib_value                                                                
> > > to                                                                        
> > > store all of the radius values.                                           
> >                                                                             
> > Hmm, no, that's not quite right.  You don't want to store the actual        
> > strings in the records for each account.  Probably need a table that        
> > corresponds to the RADIUS dictionary file, with a list of                   
> > attributes and                                                              
> > attribute id's, then reference the attribute by id and not name.            
> >                                                                             
> > The more difficult bit is handling the service definitions                  
> > correctly -                                                                 
> > part_svc and it's effects.                                                  
>                                                                               
> Yes that could be a bit tricky. Perhaps just one "radius" field in            
> part_svc that held a list of id's from the radius table that applied          
> that particular service?                                                      
. 
No, that wouldn't let you set defaults or fixed values for each attribute,
like part_svc currently does.
 


hmm - if you delete an account in svc_acct somehow that a mail alias points to,
svc_acct_sm.export will fail.  make sure this can't be done using
the web interface.

Bug: during the linking process apparantly you can link too many services to 
a package.  *sigh*

zip code i18n is not very good :( but at least US, CA, HU, ?

ut_phonen doesn't check data length.  several
orthogonal cleanup projects here.  *sigh*

update the cybercash links in config.html and in the homepage.  add more explicit support for other payment types.  etc.

From "Tim Jung" <tjung@igateway.net>
.
Automatic CC Decline Notices via email
Block Time billing for prepaid internet cards (400 hours or whatever block
time of hours)
Trouble Ticket System with simple search function for Knowledge Base feature
Employee Timecards
Credit Card System support for Red Hat's CCVS, and other Linux credit card
programs
Contact Manager/Lead Tracking
Pre-Sale Quotes
Cisco NetFlow Account for IP traffic billing
VoIP and FoIP billing
Roaming per minute billing based on Radius Proxy support or number they
dialed into from Radius logs (800# etc)
Support for OpenSRS Domain Registration
Support for TUCOWS signup CDs


dbdef-create for postgres

pro-rating, fiddling dates

It looks like svc_acct.import doesn't deal well with comments and
multi-attribute lines.

ivan@rootwood:~/freeside_current$ rgrep -r domuid * | cut -d: -f1 | sort |
uniq
CVS/Base/TODO
TODO
bin/fs-setup
bin/svc_acct_sm.export
bin/svc_acct_sm.import
htdocs/docs/man/svc_acct_sm.txt
htdocs/docs/schema.html
htdocs/edit/CVS/Base/part_svc.cgi
htdocs/edit/part_svc.cgi
htdocs/edit/process/svc_acct_sm.cgi
htdocs/edit/svc_acct_sm.cgi
htdocs/search/svc_acct_sm.cgi
htdocs/view/svc_acct_sm.cgi
site_perl/svc_acct_sm.pm

rootwood:COMPLETEHOST/TODO

Currently, you set a value in the %FS::UID::callback hash with a coderef  
of the code you want to execute, but this was bad design and will shortly
be changed to a subroutine to which you pass your coderef to register it.

This is not a bug, it is how the system is currently designed.  After 
defining a new package, you need to specifically allow agent types to
purchase packages.  You can do this from the edit screen of the new agent
type.
.
It wouldn't be a bad idea to add a configuration value that makes new
packages available for all currently existing agent types automatically.
.
On Fri, Apr 07, 2000 at 10:50:35AM -0500, David Morton wrote:
> For some reason, the type_pkgs table didn't get updated when I tried to
> create a domain service and package...  insert into type_pkgs values   
> (4,1);  fixed it up so that I could actually order the package.


From: Chuck Cochems <zaphod@tdl.com>
.
1) automated generating of late notices.  So far the only way I know to
generate one is to re-invoice manually.
. 
2) smart credit card rejection handling.  It should e-mail a note at
least, explainig that it did't go through, and why.
. 
3)coment field in the main customer record.
.
4) streamlined payment entry feature. the current scheme has it fairly
buried.
. 
5) queries to show al customers who owe money,and such stuff as that.


credits aren't counted against past invoices?  hmm.  something needs to be
done to "apply payment" genericly. and zero out invoices etc.

more email which should make it into a more organized TODO list:
.
I would also love to see Freeside support bandwidth billing by reading the
Cisco NetFlow Accounting data so we and other ISP's could automatically bill
co-located servers and even potentially other virtually hosted sites like
MUD, Palace Chat, IRC Chat, etc based on the bandwidth they use or average
sustained rates or whatever. I'm not much of a programmer so I don't know
what all this entails but I did download a NetFlow client agent/whatever for
Linux though.
 .
It would also be nice to see Freeside be able to read Apache log files and
bill customers for web traffic that way as an option also. Plus an option to
bill for excessive disk usage without having to use quotas if you didn't
want to, would be a nice feature as well. So you could monitor with a script
or something to see how much disk space a user was using then get some
average and charge a certain amount for anything above some preset limit for
that account type. I might be able to hack something like this up, but I'm
not 100% sure where to start or if there is something out there that could
be modified or not.
.
Do you think that you will ever support the HKS CCVS (Hell's Kitchen
Software Credit Card Verification Software) since Red Hat bought them out
and is going to be including that for credit card processing when you buy
the professional version? What about possibly supporting the OpenCCVS which
is a GNU/GPL version of a credit card program? I haven't had time to comb
through the Freeside code to see how hard it would be to add support for
these as externally called programs.
.
Also any thoughts on help desk, and knowledge base stuff? Any thoughts on
this stuff, and how possible and what kinds of work or time frame would be
involved?
.
Tim Jung
System Admin
Internet Gateway Inc.
tjung@igateway.net



           CVS via SSH (Score:1)
           by platinum (jedgar at fxp dot org) on Thursday September 30, @07:13PM EDT (#4)
           (User Info) http://www.fxp.org/~jedgar
           The links above are your best bet for basic cvs server configuration. Once you have a pserver set up, you
           may consider using cvs via ssh. All you need to do is the following: 

           1) Have ssh and sshd set up on the client and server machines 
           2) Set CVSROOT="joe@example.com:/home/ncvs" 
           3) Set CVS_RSH="/usr/local/bin/ssh" 
           4) Use cvs normally 
           (Obviously, insert the proper host/paths above) 

           You can also set up cvs/ssh to not need a password every time (similiar to an initial 'cvs login'): 

           1) run ssh-keygen on client machine using no passphrase. 
           2) copy/add ~/.ssh/identity.pub on the client to ~/.ssh/authorized_keys on the server 
           (man ssh for more details) 

           The main reason I mentio
           CVS via SSH (Score:1)
           by platinum (jedgar at fxp dot org) on Thursday September 30, @07:13PM EDT (#4)
           (User Info) http://www.fxp.org/~jedgar


There's no way to do this currently, though it would be pretty
straightforward to modify the source - specifically, the _collect_
subroutine in FS::cust_main.
>
On Mon, Dec 13, 1999 at 04:05:32PM -0700, Jeff Garner wrote:
> Freeside will e-mail my users when they are billed if they are setup as
> billing.
>
> When setup by credit card it does not e-mail them.  Is there a way to
> turn on e-mail invoices for those who are billed via credit card?  Kind
> of like a recipt of billing....


doc: http://www.softagency.co.jp/mysql/qmail.en.html

sql1992.txt standard is 18 character column names.  shoot for that, not just
32 (postgresql default)

Also note that (AFAIK) Freeside won't display any package that has more
than one service on the "Add Customer" page.  The reason for this is
because there's no way to dynamically alter the displayed html form
based on which package is selected. One possible solution would be to
make an additional page that's used in the signup process that would
display the form for each package, like a MS-style "Wizard".  The first
page lets you select the package, then the second page has the custom
form with fields for each service in that package and a save button. Of
course, that would require a little perl work.
.
Later,
Scott Cruzen <sic@boernenet.com>


Your suggested script with back up /usr/local/etc/freeside, but will miss
any database not named `freeside'.  Both of our scripts are specific to
MySQL.  If you're interested in contributing to Freeside, maybe you could
work on a script which: reads the mapsecrets configuration file and then
each secrets file to find out what specific database engine(s) (MySQL,
PostgreSQL, etc.) and database(s) need to be backed up, then does so,
serializing backups of the same engine, i.e. stop mysql, do all the mysql
backups, start mysql, stop postgresql, do all the postgresql backups,
start postgresql, etc.
> #!/bin/sh                                                                     
> apachectl stop                                                                
> mysqldump -t freeside > fs-backup.sql                                         
> apachectl start                                                               
> tar -Pzcvf fs-backup-`date +%y%m%d%H%M%S`.tgz fs-backup.sql /usr/local/etc/freeside/                                                       
> rm fs-backup.sql                                                              

I chose to use counters in the filesystem because there is no standard way
to get the value of an auto-incrementing keyfield which is common across  
all databases (as seen through DBI/DBD).
.
It certainly wouldn't be a bad idea to use the database-specific methods,
when available.

htdocs/edit/svc_acct.cgi:
(Does the `*HIDDEN*' show up when you are adding a new account, and 
specify the password, then receive an error and are returned to the form?)

more DOC:
Thought some of you might be interested in this:
  
<ftp://ftp.minivend.com/pub> has CyberCash compatibility modules for 
Paymentnet <http://www.paymentnet.com> and Authorizenet 
<http://www.authorizenet.com> which should allow you process transactions
using those services as well as CyberCash.

The files are named CCLib.pm.paymentnet and CCLib.pm_authorizenet,
respectively, and are installed by renaming to CCLib.pm and moving to your
site_perl directory.  Otherwise, follow the directions for Cybercash v2 in
htdocs/docs/config.html

DOC:
fs_passwd/ is a client-server replacement for the `passwd', `chfn' and    
`chsh' commands that updates the Freeside database.  (so for that to be   
useful, you'd have to be exporting that data periodically)
  
fs_radlog/ is a client-server RADIUS log parser that stuffs the data into
SQL.  It isn't finished, and probably won't be unless someone who I can't
convince to use one of the RADIUS daemons that logs to SQL directly pays  
me money or something.
  
fs_signup/ is a client-server signup server.  i'm just finishing it up    
now; probably isn't on your machine yet.


http://www.sisd.com/freeside/list-archive/msg00812.html

package definitions should be implicit allow wrt agent types, not implicit deny
(with the old behavior possible via a config file)

> So is there anyway it could be setup to allow you to select a "primary
> service" from each package?  This service would be the one you were prompted
> for.  Could the signup server then be expanded to allow users to go into
> their package and "turn-on" the remaining non-primary services(using the
> primary account.)

take the GPL'ed whois proxy stuff at www.geektools.com and turn it into
intelligence for Net::Whois.

A web version of the fs_passwd stuff would be nifty.   

If you have Cistron authenticating directly from MySQL, you can replicate
in real-time instead of exporting periodically.  See 
<http://www.mysql.com/Manual_chapter/manual_Common_problems.html#Replication>.

these go in docs:
<http://www.sisd.com/freeside/list-archive/msg00541.html> (was 546), and

and http://www.sisd.com/freeside/list-archive/msg00421.html (was 423)

> > 5: Is there anyway to get freeside to send a sysadmin a warning when a
> > credit card has expired?
No, but there should be.

Put this in the doc (quoting Mark Wells <mark@pc-intouch.com>):
>Of course, thanks to the sheer coolness of SQL and MyODBC, you can do
>whatever reports you want in basically whatever application you want.
>There's no need for Freeside itself to do any reports at all.

middle names and titles

On Wed, Jul 07, 1999 at 01:11:40PM -0400, Frank Nazario wrote:
> Playing and entering information to Freeside i encountered the following
> missing reports:
> 
> View Customers by Agent
> 
> View Pending Invoices
> 

grep 'uncomment this to encrypt password immediately' site_perl/svc_acct.pm
Not to say that it shouldn't be a configurable option.

in site_perl/cust_main_invoice.pm (elsewhere?), error out if mydomain config file is gone
(at least until the idea of a default domain goes away)

FS::Record::qsearch does an eval every loop iteration (which is itself not
guaranteed to work across all DBD's and should be fixed).  This has got to be
slow.  Fix it.  (I think recent Perls might have a way to accept a variable
there, no eval needed?)

Could you have added /bin/sync, /sbin/shutdown, and /bin/halt to the
`shells' configuration file before importing, and removed them afterwords?
(even better if svc_acct.import did that automatically - it could just    
munge and restore @FS::svc_acct::shells... hmm.)

> BTW, Ivan, I am trying to verify in an additional database table that a
> particular user doesn't exist. This database is used to store email aliases a$
> additional POP boxes for our customers (kinda like AOL allows). I have toyed  
> with the idea of just writing the aliases to an email only svc_acct that
> doesn't write to the password file, but that isn't really how I want to do it.

Actually, I think that's a pretty good way to do it.  Cerkit contributed
support a little while back for svc_acct.pm and svc_acct.export for
multiple export targets.  It needs to be cleaned up and documented, which
I'll try to get to soon.  For this to work correctly, the svc_acct_sm
table should go away, along with the concept of a "default" domain.  

default setting for new packages should allow all agents to purchase them...
with a config file for the old behaviour

fix or replace Term::Query (Quiz::Question doesn't do what i need)

Check config file reading stuff from CPAN
 
Authorizenet module from CPAN!

<http://www.math.fu-berlin.de/~leitner/mutt/faq.html> has a good y2k complience
statement!

    I'm hoping Freeside can support arbitrailly complex pricing plans
    because of a simple concept: all prices are perl expressions.  So if
    you use `19.95' for example, perl evalates that to be `19.95'.  But if
    you need to do a complex pricing scheme, you just need to write an
    appropriate perl expression, which will most likely pull data from the
    database to return pricing.  Some things will already log to SQL; for
    example most RADIUS servers can or have a patch available.  Getting
    Freeside to bill based on any sort of data then becomes a matter of
    importing the data into the database.
 
    There are some issues involved with pro-rating, partial month charges,
    that sort of thing.  Expressions will need a standard way to have the
    applicable time/data ranges passed to them.  Also the expressions are
    currently running under the Safe perl module, and the opmask might not
    be right in all situations.  I'll try to spend some time working on
    this if you are using it.

> 2. can customers view their bills on-line.                                    
 
Not yet; it needs to be proxied from Freeside to a customer web server in
a secure way using something not completely unlike the fs_passwd,
fs_passwdd, fs_passwd_server trio.


> Lastly, if someone over pays on an invoice, the credit part does not flow
> over to other invoices..

The total balance flows over correctly, but individual payments don't.
The code you're looking for is in FS::cust_pay::insert

The question of what to do with overpayments that don't have another 
invoice to flow into (yet.. or possibly not) is still an open one.    The
legacy system Freeside replaced long ago had a separate place for this
(payments waiting for an invoice) for each customer, and it gave our
bookeeper fits.


option to relax username uniqueness in favor of username+domain or mail/shell
vs. radius to ease import for isp's with namespace problems or who buy others.

do i have to store anything for radius realms besides regular radius attributes
(which are handled fine now)?

warn or complain or something when invoice_from is empty (and we use it)

Right now Freeside uses the `freq' field of a package definition as a 
number of months.  The specific section of code you're looking for is in  
FS::cust_main::bill:
        
        #change this bit to use Date::Manip?
        #$sdate=$cust_pkg->bill || time;
        #$sdate=$cust_pkg->bill || $time;
        $sdate = $cust_pkg->bill || $cust_pkg->setup || $time;
        my ($sec,$min,$hour,$mday,$mon,$year) =
          (localtime($sdate) )[0,1,2,3,4,5];
        $mon += $part_pkg->getfield('freq');
        until ( $mon < 12 ) { $mon -= 12; $year++; }
        $cust_pkg->setfield('bill',
          timelocal($sec,$min,$hour,$mday,$mon,$year));
        $cust_pkg_mod_flag = 1;
  
..and when I went poking for this, looks like it tells us just what needs   
to be done!  Hehehe...

Date::Manip can handle cool things like "+ 1 month" (actually the current
case of /^(\d+)$/ would have to be added as a special case of "+ $1 
month") and "+ 30 days" (what you need) and even "+ 5 business days" !


On Wed, Apr 28, 1999 at 08:38:16PM +0000, Kristian Hoffmann wrote:
> I can't quite seem to figure out how this exporting works.  From what I
> understand, when you run svc_acct.export, it rewrites the /etc/passwd,
> /etc/shadow, etc. files.  Is this only for initial setups with the
> export hooks being in the pm's?
 
You can use both, or just one method.  The configuration files control    
this.  One of the things in the TODO is to take out the last few things   
that aren't customizable wrt this and put them in config files.

http://www.daemonnews.org/199905/user-mgmt.html

Term::Query doesn't install out of the box from CPAN.  Fix it (and get it 
submitted upstream!), or remove requirement from bin/svc_acct.import and
bin/svc_acct_sm.import and take it out of the install instructions.

use this cool link to explain the Freeside API
ftp://cpan.nas.nasa.gov/pub/perl/CPAN/doc/FMTEYEWTK/easy_objects.html

Multiple tax rates by geographic region (county, state, and county) are   
supported; just choose View/Edit tax rates from the main menu.

Multiple tax rates by package are not (yet) supported.

On Wed, Jul 07, 1999 at 12:13:36PM -0400, Shaun Batterton wrote:
> How would you handle something like multiple tax rates and multiple   
> states?  For example in Connecticut, they just changed computer and/or
> data processing services to 3% (whatever that is), and everything else
> is 6%.


> Second, when trying to add a new user I get two types of errors; first one
> is when I place an e-mail address and select the "submit," Freeside
> complains with "Error: Unknown local account (specified literally)"
.  
You probably put a (local) email address in for email invoices.  Freeside
stores these as references to the database records, so (for example) they
follow username changes.
.
I'm guessing you put in the email address that you were creating, and got
that error because it didn't exist yet.  Sounds like a buglet to me.  I'll
try to fix that soon; in the meantime you can add the invoicing email 
address afterwords.

(workaround for postgres before 6.5) (unlikely to ever be implemented)
In the mean-time, this could probably be fixed with the reverse of        
the kludge in FS::Record::new.  This removes `$' and `,' from money fields
coming out of the database.  Something which fixed up the data so Postgres
was happy with it could go in FS::Record::_quote, for data going into the
database.

our data display problem might be a Freeside problem wrt not using 
Oracle-compatible DBI syntax (fixed using the return value from $sth->execute
as a number of rows - something else?).

hooks for arbitrary commands out of configuration files
svc_acct.pm svc_acct_sm.pm etc.

Add this to a FAQ, along with doing it for middle names:


>  What I'm finding difficult is how to easily 
> customize fields.  For example, I am trying to add a "middle name" field
> to the Customer Edit, view, etc.  If I'm going about it right, it appears
> I have to edit the cust_main.cgi under edit and edit/process and the
> site_perl/cust_main.pm, as well as other things.  Perhaps you could shed
> some light on the best way of doing this.

You have the basic idea.  To implement that completely, I would:
- Add the new field to bin/fs-setup for new users
- Document the field in htdocs/docs/schema.html
- Document the change in a new file, htdocs/docs/upgrade4.html
* Run bin/dbdef-create
* Add the new field to edit/cust_main.cgi and edit/process/cust_main.cgi

For bonus points, I'd grep around for the various bits which use "$first 
$last" or "$last, $first" and replace them with a method call in cust_main.pm,
 , like search/cust_main.cgi

document security model:
Don't forget about Apache usernames - since, via the mapsecrets file, each
user can login to the SQL database with a different username and
password, you can utitilize the security model of the SQL database as
well.  Also, each username here can point to a different configuration
directory where you could store user-specific configuration info.  Then   
you could link each username to one-to-many agents. 
(The web demo works using a trivial version of this.)

Yes, queue processing or the equivalent via checkpoint fields on various  
talbes (which you pick up via a pretty simple SELECT) would be really 
nice too.

default (and ordering) state/county/country config file
expand the
cust_main_county table to provide a preferred ordering, so the most common
entries would be at the top of the selection box.  automatically, based on
recent selections?

hmm... maybe svc_acct__shell should check off the legal shells list if
applicable?  yeah... cool.

payinfo field should me much larger than 16


[Mon Apr 12 20:31:21 1999] [error] [Mon Apr 12 20:31:21 1999] null: Error closing true: Broken pipe at /usr/local/lib/site_perl/FS/cust_main.pm line 615.

javascript (yuck!) "are you sure?" confirmation on cancelations, etc.
(view/cust_pkg and view/svc_*)

get rid of time2str("%D") which formats dates in a non-y2k-safe looking fashion
(all the actual date handling uses UNIX timestamps and is fine)

uncomment expire in view/cust_pkg.cgi and find the expire cron from fsold

(Test this)
one-time/per-customer/? changes in rates and descriptions ('remembered
invoices'): implement by creating a new package on the fly... but it isn't 
associated with any agent types so it won't show up for other customers to buy.
(but also... make sure they go away when the customer does! - need this? :
 one-off package edits! : need a cust_pkgs or cust_part_pkgs or something table,
 with custnum and partpkg (like type_pkgs)
(what happens if you hit "custom pricing" but the pricing is already custom?)

Lay out any remaining ugly forms better.

remove "records identical" warning?  gets in the way of more important stuff.
or fix logic which tries to update identical records??
1.2 should be quiet enough that the error log is useful, hopefully.

Postgres has a maximum column length of 31 characters (but see NAMEDATALEN in
postgres_ext.h).  part_svc has columns like: svc_acct__radius_Attribute_flag
(22 characters!)  It seems that stuff over the limit is silently ignored,
so we get 4 characters back.  So, Radius_Attributes are max 13 characters with
stock Postgres.  see rfc2138 for what's affected
What's a good fix?  (besides recompiling postgres with NAMEDATALEN 64)
(mysql has a 64 character max column length.  others?)

[Mon Mar 29 06:57:56 1999] -e: Use of uninitialized value at /usr/lib/perl5/Date/Format.pm line 333.
(when sending mail in cust_main.pm::bill or svc_domain.pm)

look at DBIx::Recordset!  (and Tie::DBI, and...)

undefined conf/lpr gives this uninfomative error:
[Fri Feb 26 16:42:36 1999] bill.cgi: Can't do bidirectional pipe at
/usr/lib/per
l5/site_perl/FS/cust_main.pm line 629.
[Fri Feb 26 16:42:38 1999] bill.cgi: Error closing : Broken pipe at
/usr/lib/per
l5/site_perl/FS/cust_main.pm line 631.
So give a meaningful error!

password and slipip stuff in svc_acct.pm store need to be split into two fields or something, so the silliness in svc_acct.pm and svc_acct.export with looking at the data to decide what to do with it can be fixed

i10n: Apache::Language

Apache::Session?  Other useful Apache::* ?

email invoices are only sent for the BILL payby.  If setup, should statements
(since they're not invoices) be sent for COMP and CARD as well?

$cgi->keywords is causing the (hard to trace) error:
        Use of uninitialized value at (eval 5) line 5

edit/cust_main.cgi gives an uninformative error message:
> Can't call method "agentnum" without a package or object reference at   
> /usr/local/apache-ssl/htdocs/freeside/edit/cust_main.cgi line 116.
if there are no agents.

(is this missing on any web screens?  (easy with $cust_svc->label)
Add the ability for services to filter information up to the package level
for web screens, so you can select a particlar package based
on username or domain name, etc.

Allow a cancelled/suspended/active status from packages to bubble up to
the customer lists.  Put active, then suspended, then cancelled accounts.
Similar ordering on the package listing inside a single customer.

false laziness: edit/cust_main.cgi got some parts copied from edit/svc_acct.cgi
the web interface in general needs to be redone in a more abstract way.

false laziness: some of search/svc_acct_sm.cgi was copied to search/svc_domain.cgi.  but web interface in general needs to be rewritten in a mucho cleaner way.

subroutine the where clause (eventually all SQL) as OO perhaps (has anyone done this?)

add a select method to FS::Record?

one-time/per-customer/? changes in rates and descriptions ('remembered
invoices'): implement by creating a new package on the fly... but it isn't 
associated with any agent types so it won't show up for other customers to buy.
(but also... make sure they go away when the customer does! - need this? :
 one-off package edits! : need a cust_pkgs or cust_part_pkgs or something table,
 with custnum and partpkg (like type_pkgs)
(what happens if you hit "custom pricing" but the pricing is already custom?)

You can't delete the stuff under administration yet.  Add this,
_including_ making sure the thing you are deleting is not in use!

add links on view/cust_main.cgi to setup services, like view/cust_pkg.cgi

FS::cust_pkg _require_'s FS::$svc, but this won't work with %FS::UID::callback
loading of configuration.  (pry need same idea, but will run immediately if
context allows).  Looks like error is masked by 'use FS::cust_svc' which in
turn 'use's FS::{svc_acct, svc_acct_sm, svc_domain}' which is now explicit
w/comments in source

Allow a cancelled/suspended/active status from packages to bubble up to
the customer lists.  Put active, then suspended, then cancelled accounts.
Similar ordering on the package listing inside a single customer.

svc_domain.pm mail sending uses Date::Format which doesn't seem to pick up 
correct timezone.

view/svc_domain.cgi needs to know the domain might be unaudited (cosmetic)

remove whois_hack set to 1 for svc_domain.pm?  add all known registries and
whois accordingly.
.us domains and others!
site_perl/svc_domain.cgi (hmm... or maybe should have a button?  or maybe svc_domain.pm should handle this) should set $whois_hack for non-internic domains, so you can add them...

turn on the depriciation warnings for [e]idiot in FS::CGI. Stop using [e]idiot
the last places it is (htdocs/search/ htdocs/misc/ htdocs/misc/process)

(test cust_main.pm with cybercash v2 and v3, especially with the callback
 stuff AND with mod_perl w/cybercash v2 kludge in package main)
(callback stuff should be eliminated by now)

bah, table/itable/*table in FS::CGI is silly.

doc Apache::AuthDBI as well
..
Provide sample httpd.conf files.

hey look: Tie::DBI!  Check that out.  Override its commit with something that
does perl-side caching for ? a performance improvement and as an emulation
layer to plug in f.ex mysql's atomic transactions
..
Record.pm uses does some non-portable DBI things.  MySQL and Pg seem fine.
Fix it anyway unless we migrate to Tie::DBI.

faq

cust_bill.pm uses '==' comparison on dates because they're currently ints

config file for allowed card types

write instructions for adding new services w/svc_Common.pm.  Get rid of all
places where svc_* tables are hardcoded (rename svc_acct_pop to part_pop so
we can do that)

test and document libapache-dbi-logger (woo!)

radius logfile parsing and perl expression check.

Fix in cust_bill BUGS: 
There is an off-by-one error in print_text which causes a visual error (Page 1
of 2 printed on some single-page invoices).

fields should be a method against a FS::Record or derived object, as well as
being something you can call as FS::Record::fields('tablename').  Might
even be able to handle both in the same routine (that would be neato).

Immediate removal of incorrectly entered check payments (can't take too
long to do this, or accounting is fubared).

Add code to move from one service to another (POP to SLIP/PPP, etc.).
This _should_ be possible by working off the rules in part_svc rather than
hardcoding anything in.  The rules in part_svc may need some elaboration,
perhaps.

Use ut_ FS::Record methods in all derived classes (possibly some from dbdef?... eventually all from dbdef??? - but then `dbdef-create' would be impossible as there would be metadata we couldn't ask the backend for.  hmm.)

(bring back from fsold, ) Generalize config-sending stuff and make more configurable.
Expand the HylaFAX interface (also possibly generalize for other fax
softwar ie .comfaxe); allow things like arbitrary faxes of sales
literature, specific troubleshooting documents and so on.  Maybe even
allow users to do this (though that might not belong in Freeside).
misc/sendconfig.cgi
misc/process/sendconfig.cgi
Configure fax recipients via a separate box rather than using the finger
name or first+last from cust_main.

move all phone number logic out of Freeside - let HylaFAX or whatever
handle it.

soundex searches for customer name and company?  where are free soundex tools? (standard Text::Soundex duh) - I could have sworn I saw Text::Soundex on CPAN?!

should be able to link on some field in email alias (right now you can link
on username or domain with a fallback to svcnum)

generalize and make configurable new invoice printing scheme in FS::cust_main::collect (past due)

deleting an svc_domain should delete all associated svc_acct_sm records.
same with a svc_acct.

periodic password encrypter

Automated, configurable notification, suspension and cancellation of
defunct accounts.
...
expire cron job
...
Allow for a future setup date on accounts.

sub AUTOLOAD in FS::Record should warn? die? if used with a non-existant column
name?

edit (not just import, export and allow default/fixed) arbitrary radius stuff
in svc_acct
edit/svc_acct.cgi and edit/process/svc_acct.cgi should deal with arbitrary radius stuff

radius import should take DEFAULT entry and put it in /var/spool/freeside/conf/radius-default ; svc_acct.export should use it (and doc)

in UI, s/State/State\/Provence/go and s/County/County\/Locality/go

what else (besides l10n) for i18n? (money!)

audit htdocs/* for things that should be libraried and things that should be
new methods on the objects (need to do this before implementing a new UI)
all the big things are done

some places we die() where we should &FS::CGI::idiot (and perhaps vice-versa).
Decide based on whether or not the "error" should show up in logs.

all .cgi's should use standard header/footer and idiot() subroutines.  maybe HTML:: perl modules
for HTML creation.  Maybe Embperl or something along those lines.  ?

When running bin/bill, Fix this (Annoying but harmless):
Use of uninitialized value at /usr/local/lib/site_perl/FS/cust_pkg.pm line 99, <ADDRESS> chunk 4.
Use of uninitialized value at /usr/local/lib/site_perl/FS/cust_pkg.pm line 102, <ADDRESS> chunk 4.
Use of uninitialized value at /usr/local/lib/site_perl/FS/cust_pkg.pm line 105, <ADDRESS> chunk 4.

all cgi (but internal to the isp) places where package names are listed should also have
comment (like agent_type)

clean up $recref and other silliness and use -> calls where possible, or
one other alternative.  clean up everything else.
should FS::Record use Tie::Hash?  That would be very clean, but where do we
store the other information?  Maybe you could ask any FS::Record object for a
tied hash?

change all file access from regular open(FILE,) stuff to OO, because of 
problems scoping and passing filehandles like that.

the web interface should create a new object and use it instead of a blank
form for new records.  the create method of svc_ objects should set defaults
(from part_svc).

sub check in man FS::table_name should be rewriteen.  Get rid of $recref
stuff.  Make sure all fields that refer to other database are checked.

Integration with signup disks (are there any free ones?  Netscape?).

One-button cancel (+refund) for lusers who can't get online.

Keep information on virtual web servers (hostname, IP, host machine,
directory, etc.) and export this information for importation into the ISPs
web farm. 

Remove requirement that the first mail alias be the catchall?  Still make
sure only one catchall per domain is defined in any case, of course. 

Ability to move cust_pkg records from one customer to another? (proably
will need to cancel the old and create a new like when we move services
between packages). 

Auto-increment expired cards one year, and try again?

More non-US stuff - zip codes, country codes, foreign currencies, etc.

cust_refund.{cgi.pm} need to do cards xaxtions.  (now we only have cust_credit)

Nicer set of integrated reporting possibilities, like weekly sales totals
by customer, package, agent, referral, etc., aging reports sorted by lots
of different things, and so on.

Client/server setup for users to modify their own passwords, shells, etc,
via passwd or secure web interface (prelminary passwd/chfn/chsh
replacement done).  Complicated by the fact that we don't want to allow
incoming connections to the machine running Freeside, so we probably need
to have a daemon on each external shell or web machine that is contacted
by the Freeside machine.  Be very very careful for both traditional
security issues and DoS problems. 

An extension of the above to allow users to modify selected parts of their
own information, order and cancel services.  A web interface for new
customers.

Expand domain name stuff to house all domain information.  Export
named.boot/named.conf (primary and secondary) and named.{domain} files.
Add more registries (not just InterNIC's com org net edu)

Nice postscript paper invoices, rather than current ASCII invoices.

think about race-condititions in FS::Record and derived ->check ->insert
and so on, uid and username checks in svc_acct, etc.

Move to rsync over ssh file exportation rather than scp.

check 'n fix the proactive password checker. (cracklib?)

refunds of "BILL" payments: generate pseudo-check.

write batch senders and batch parsers for the different credit card processors
people use/
More CC processors/methods.

you should be able to fiddle the setup date in cust_pkg. (at least initially)

delete options in administration section

write a generic batch senders and batch parsers.

need a way to override svc_acct export on a per-machine basis; just use config files based on machine name i suppose; document that. (no, import desync_hosts
type stuff from cerkit)
...
add a table with column of export services (passwd, shadow, master.passwd, .qmail file update, dns update, etc.) and rows machine groups and whether or not to export that (and any necessary parameters).  wasn't matt (vunderkid, not matt@michweb) working on this?  find him?  each machine goes in a group of its own as well as a group based on function.  add a table with only svcpart and machine group.  now, when you import from each machine, it can get its own accounts with one svcpart and universal accounts with another svcpart.  (though that does make the username duplicate checking more interesting)

you should be able to get column types as a method against an FS::Record object
as well as dbdef->table($table)->column($column)->type

move to perl module for fuzzy and soundex searching.

package view needs to list extraneous services; we need to prevent the
creation of them so this never happens (and mark it as such in the source)
(the creation problem should be fixed - though they will still happen if people
fsck around in the data manually, so list them anyway)

add attribute dictionary to fs-setup as a menu, plus analyze users file to
decide automatically

Check for and report on duplicate billing accounts (cust_main, though many
will have a need for these so probably don't disable them outright.)

create a ->warn as well as a ->check method for all FS::table classes?
(see above)

something to automate making a release and updating the web demo

export a debian-style (also redhat and?) /etc/group file aswell!

svc_acct_sm.import qmail import should pull in recipientmap people too.

.pm's like svc_acct.pm which need to do time-consuming things like ssh remotely
should fork and do them in a child.

i18n/l10n: take ALL messages and catalog them in english.txt or in database or something, so we can eventually go int'l.  int'l currency support would be a help aswell.

get some of { city, county, state, zip } from the missing bits if
possible (where can i get the data to do this?  usps.gov?)

additional interfaces (perltk?  java?)

Put the GPL notice in all files.

integrate w/IDEA's signup server

$cust_bill->owed database field to be eliminated, replaced by a method call
that calculates on the fly.  make sure to grep for ->(get|set)field('owed') 
same for cust_credit->credited

Export quota information.

move all configuration to a central place.  maybe in blob's in the
database.  maybe even things like the code to execute when a username is
changed can be in there, so less of the distributed scripts change between
different sites.

Implement setup and recurring fees as Safe perl expressions rather than
numbers, to allow for variable-rate services.  Backwards compatibility is
obtained because { 43 } in perl is still 43.  :)  Define API to pass
starting and ending dates and any other necessary data to expression
(fees are currently evaluated as Safe expressions but more work needs to
be done to define an opmask for various needs, write examples
(usage-based billing, etc.) and so on).
...
Add the ability to modify the next billing date in cust_pkg, and take
appropriate action.  This will allow the implementation of pro-rate/1st of
the month billing as well as the ability to manually fiddle with
anniversary dates in cust_pkg, so you can sync a customer's anniversary
date even if you're using anniversary billing (manually or automatically).
(now with above, we need to have a way to automatically pro-rate /^(\d+)$/
charges - anything more complicated should figure it out itself given
starting and ending dates [document that!])
...
Daily Radius log parsing into database; other logfile formats?
...
Callbacks to enforce hourly limits on accounts (suspend until the end of
the billing period?), for those who limit customers rather than tack on
extra charges.

Flag packages (part_pkg) as taxable or non/taxable as some ISPs (for
example) need to charge tax on equipment but not service (separate flags
for setup and recurring fee... or perhaps a setup_tax, setup_notax,
recur_tax and recur_notax fees, and possibly something more flexible if
there is need).

Allow for a variable number of invoices for customers who need multiple
copies.

Add a mail alias service with table svc_acct (not domain mail aliasing
which is domain with svc_acct_sm)

(bring back from fsold) Change customer comment field from its current kludge to something more
workable.

Better work orders with more information.  Should eventually open a ticket
when we have such a thing.
edit/svc_wo.cgi
edit/process/svc_wo.cgi
Call tracking and trouble tickets.

More accoutability for complimentary accounts: approval, expiration, term
(no more than x months in advance) and notification.
Flag particular users (or all users, for that matter) as having their
passwords hidden and/or locked from users of Freeside (maybe need Freeside
security levels first?). 
...
Better Freeside-level configurable access, for those ISP's who have
employees they can't trust.  Right now you're "stuck" with setting up
.htaccess stuff yourself.  This should really just be integrated. 

configuration/setup should get web interface
...
/usr/local/etc/freeside should be configurable
...
(probably as part of some automated installation process?)

This requires some serious magic in FS::Record:
ok, if date_type in fs-setup is to be something besides int,
now we need to create wrappers
for them so they behave identically across RDBMS's, ie date pops out as as
UNIX timestamp (or an object of some sort? maybe even a blessed $obj which
is a string not a hashref for backwards compatibility?) and so on. (remember
to treat '0' as Not a Date instead of 1/1/70.

Add Freeside-level transactions for RDBMS's which don't support
transcations?  (Currently we assume a minimal RDBMS which has no rollback,
transactions or atomic updates).  Or just require a RDBMS that supports
rollback and/or atomic updates and get rid of the work-arounds?  The /rdb
interface had this kludge on top of it but is a technical dead-end in most
other ways, unless it can gain an SQL parser and DBD interface.
...
if i'm really bored, find the /rdb interface in fsold and port it to NoSQL,
and while I'm add it add interfaces for AnyDBM_File tied hash.. hmm.  Shouldn't
an FS::Record have something to do with a tied hash?  But we don't want
performance to go gaga... maybe something with commit to help out here?
...
Ok: FS::Record gives you a tied hash, and you get methods for commit, etc.

Better automated comparison of our CC records with processors (CyberCash,
at least, has not always had 100% accuracy, though recent versions are
much better) 

Expect or other pty based login check, where we actually connect to a
terminal server or shell machine and test logging in as the user (if we
are keeping a cleartext password for that user)  (This is something tech
support often needs for new customers)

Use cust_main table for pre-sales tracking as well?

Automatic commision report and check generation via freq and prog (to
become a Safe perl expression) fields in agent table, and possibly others.

Database and add a mailed-out date and method for disk/CD mailing, so a
customer can call and you can say, "sent on xx/xx/xx via {US Mail, Fedex,
UPS, etc}" 

Inventory tracking for physical items such as routers (for sale or
lease... probably doesn't make a difference in the ordering... but if you
cancel a router lease the inventory should come back.  hmm.)

-- Matt's wishlist ---

From matt@michweb.net Fri Feb 20 16:39:53 1998
Date: Thu, 19 Feb 1998 23:20:11 -0500
From: Matt Simerson <matt@michweb.net>
Reply-To: quadran-developer@netgoth.com
To: quadran-developer@netgoth.com
Subject: Re: Welcome to quadran-developer

>Whats it based on and what is it supposed to do?  I'm interested, but
>unfortunatly, I don't have that much time to help on the project (I'm busily
>working on one of my own based around MySQL and Qt right now -- don't know
>if it will be GPL'ed or not yet -- we'll probably just use it in house since
>it is designed around our system)...

That's what I set out to find, but didn't find anything on the web site.
I'm looking for something that will do the following:

Single point of entry for users on a secure system:
        Creates account on user (public) systems
                update /etc/passwd/master.passwd file
                update radius database (if necessary)
        Set up up disk quotas (although I hacked adduser to do this)
        Option for adding user to a mailing list(s)
        Export of new user info to customizable report (for automated entry
into
                accounting software, etc...)

Automated billing:
        Export credit card info for batch processing and have hooks built
                in for other forms of electronic processing.
        Batch-Payment (apply payments from formatted text file).
        Customizable reports for manual entry/importing into Accounting
software
        Email or laser print invoices
        Sanity checks credit card numbers before processing (code available)

Simple method for disabling an account.
        Arbitrary Expiration Dates (on a given day, in x days)
        Remove from radius.
        Changing password to '*'
        Virtual customers disabling dns, http server, log processing, etc..

Billing for different account types:
        Dialup monthly flat rate. Prorates for partial months.
        Dialup monthly flat rate for x hours + hourly usage.
        Dialup email only
        Email only accounts
        Virtual Web accounts - w/multiple mailboxes
        Leased line accounts
        Disk space used over quota.
        Tech support minimum + hourly charges
        Other for misc stuff (modem, RAM, etc...)

Per user definable RADIUS attributes (ties in with above)
        Fixed IP
        Simultaneous Use
        IP filters (for dialup email only)

Keep logs of modem usage generated daily from radius accounting logs stored
on multiple radius servers.

Keep logs of disk usage generated from quota.

Method of adding virtual domains to your system:
        Automatically grabs an IP address from a preassigned pool.
        Creates a domain.com database file from database fields
        Updates /etc/named.conf or /etc/named.boot and reloads named.
        Add's virtual.com to /etc/sendmail.cw or qmail control files.
        Edits your web servers httpd.conf file and restarts http server.
        An optional section for adding vif's can be added if the users OS
                supports adding them on the fly. Otherwise it's up to the end
                user. Make a hook that can run a custom script that the user
                tweaks for his system.
        Update or create the config file your web stats analyzer needs. I've
                done this for analog (free) and http-analyze. Probably
                should only officially support analog and let users hack
                it to their hearts desire.
I've already written scripts that do most of the virtual web stuff on my
system...in bash. Shouldn't be hard for a perlmeister to convert. In fact,
as long as all the info was stored in the database (username, domain name,
and ip pool) this could easily just be run as an external script that the
user tweaks to match his system.

We use a great accounting software (M.Y.O.B) that does all the AP, AR,
Payroll, Tax stuff, and most everything else we could need. It's already
set up for the type of checks we have, etc, etc... I just need something to
do the billing part. I can import/export sales and payments directly once
the billing part is done. You can't write accounting software as good as
M.Y.O.B. for $120.