#
# Welcome to ipt-parse a tool to manage log files created
# by custom iptables in tomato firmware.
#
# Original Author: Mark Vejvoda
# mark_vejvoda@hotmail.com
# www.soft-haus.com/blog
#
# Copyleft: Do whatever you please with this code.
#
# This package of files contains my first attempt at 
# adding support for bandwidth monitoring per IP Address
# in conjuction with the latest tomato firmware.
#
# This documentation is now out-dated and has been superceded by a Wiki Page
# located at: http://www.soft-haus.com/wiki/index.php5?title=IPTables_Bandwidth_Monitor
#
# While the contents of this README may be of some use, the latest and most
# accurate content will be maintained at the above mentioned Wiki.
#
# There are two approaches to using this bandwidth monitor.
# 
# Type #1 - Logfiles Configuration
# This method simply saves the output of iptables into logfiles
# at the specified interval of time that you choose. The ipt-parse
# program reads these files (parses them) and produces a summary output view.
#
# Type #2 - SQLITE Configuration
# This method saves the output of iptables into a SQLITE database file
# at the specified interval of time that you choose. The ipt-parse
# program reads this data from this file and produces a summary output view.
# This method has many advantages and allows for a multitude of statistical
# analysis possibilities.
#
# Special Note: for the two input date parameters you may use the following special
# notations:
# today              (meaning use todays date) 
# today-4            (meaning use todays date minus 4 days as the date) 
# bw_startday=21     (meaning the 21st is the bandwidth monthly start
#                     date. if today > x then use current month and change day 
#                     of month to x, if today < x use last months month and change 
#                     day of month to x)
#
#
# Type #1 - Logfiles Configuration Installation
#
# Here are the steps required to setup your tomato router
# (or really ANY device or operating system that supports iptables)
# *NOTE: It is assumed that you have enough free space accessible 
# to the tomato scripts (like a cifs share) for log files to be archived
#
# Step #1:
#
# In Tomato Setup goto Basic / Static DHCP
# and configure a static IP address for each client
# that you wish to monitor bandwidth for. 
#
# Step #2:
#
# In Tomato Setup goto Administration / Scripts / Firewall
# and enter the following: (repeat the last two lines for each 
# static IP address that you configured in the previous step)
#
# iptables -N traffic_in
# iptables -N traffic_out
# iptables -I FORWARD 1 -j traffic_in
# iptables -I FORWARD 2 -j traffic_out
# iptables -A traffic_in -d 192.168.0.100   
# iptables -A traffic_out -s 192.168.0.100
#
# Step #3:
#
# In Tomato Setup goto Administration / Scheduler / Custom 1
# and enable the custom scheduled script (at a minimum of every hour
# or every minute if you want to ensure you don't lose much data
# if your router loses power or reboots) and select Everyday.
#
# Enter the following script (which periodically archives bandwidth
# usage statistics:
#
# cd /cifs1  #change this line to the location accessible to tomato for saving logs
# iptables -L traffic_in -vn > usagelogs/traffic_in_`date '+%Y%m%d%H%M%S'`
# iptables -L traffic_out -vn > usagelogs/traffic_out_`date '+%Y%m%d%H%M%S'`
# ./ipt-parse 1 today-7 today usagelogs/ traffic_in_ traffic_out_ > weeklybandwidth.html
#
# Step #4:
#
# Copy the attached ipt-parse binary to /cifs1 and ensure it has execute permission
# from within tomato (telnet to tomato and from the shell type: chmod 777 /cifs1/ipt-parse). 
# *NOTE: the ipt-parse-ubuntu is the same program compiled
# under Ubuntu 9.04 (Jaunty) and should be able to run from any regular linux distro.
# 
# ALL DONE.
#
# Now depending on how frequently you told the script to execute it will generate
# and html page in /cifs1/weeklybandwidth.html showing you the bandwidth usage
# for the last 7 days.
#
#
#
#
#
# Type #2 - SQLITE Configuration Installation
#
# Here are the steps required to setup your tomato router
# (or really ANY device or operating system that supports iptables)
# *NOTE: It is assumed that you have enough free space accessible 
# to the tomato scripts (like a cifs share) for the SQLITE database file 
# to be saved.
#
# Step #1:
#
# In Tomato Setup goto Basic / Static DHCP
# and configure a static IP address for each client
# that you wish to monitor bandwidth for. 
#
# Step #2:
#
# In Tomato Setup goto Administration / Scripts / Firewall
# and enter the following: (repeat the last two lines for each 
# static IP address that you configured in the previous step)
#
# *NOTE: You may ask ipt-parse to create bandwidth rules using IP addresses
# that you have already configured in the Static DHCP page. To do this
# you should NOT manually enter the values'iptables xxx' entries shown below
# but instead add this line to your firewall script (assuming ipt-parse is on cifs1):
#
# /cifs1/ipt-parse 6
#
# **NOTE: If you use this way to auto configure the bandwidth monitoring rules
# then for step #3 below do not pass the additional commandline parameters ie:
#
# ./ipt-parse 2 BANDWIDTH
#
# should replace the line in step #3 that looks like:
#
# ./ipt-parse 2 BANDWIDTH "iptables -L traffic_all -vnx" "iptables -Z traffic_all"
#
# If you did not enter the line above in your firewall script you will need to
# enter the items below.
#
iptables -N traffic_all
iptables -I FORWARD 3 -j traffic_all
iptables -A traffic_all -d 192.168.0.100
iptables -A traffic_all -s 192.168.0.100
#
# Step #3:
#
# In Tomato Setup goto Administration / Scheduler / Custom 1
# and enable the custom scheduled script (at a minimum of every hour
# or every minute if you want to ensure you don't lose much data
# if your router loses power or reboots) and select Everyday.
#
# Enter the following script (which periodically archives bandwidth
# usage statistics, I have mine set to every 5 minutes)
#
# cd /cifs1 #change this line to the location accessible to tomato for saving logs
# ./ipt-parse 2 BANDWIDTH "iptables -L traffic_all -vnx" "iptables -Z traffic_all"
# ./ipt-parse 4 today today BANDWIDTH > dailybandwidthlive.html
# ./ipt-parse 4 today-7 today BANDWIDTH > weeklybandwidthlive.html
#
# Step #4:
#
# Copy the attached ipt-parse and BANDWIDTH binaries to /cifs1 and ensure they 
# have execute and read/write permission from within tomato (telnet to tomato 
# and from the shell type: chmod 777 /cifs1/ipt-parse)
# *NOTE: the ipt-parse-ubuntu is the same program compiled
# under Ubuntu 9.04 (Jaunty) and should be able to run from any regular linux distro.
# Notice that BANDWIDTH is a standard sqlite database that may be queried using
# the standard sqlite3 binary from any platform (we have attached one for
# both tomato and Ubuntu 9.04).
# ALL DONE.
#
# Now depending on how frequently you told the script to execute it will generate
# and html page in /cifs1/weeklybandwidth.html showing you the bandwidth usage
# for the last 7 days.
#
#
#
# General Notes:
# To query using up to the minute intervals (if you are recording stats that frequently)
# you may pass dates with times to produce a more finite data report like this: 
#
# ./ipt-parse 3 "20090511 22:15" "20090512 01:45" BANDWIDTH
#
# ***NOTICE: the format MUST be exactly as above, with a space between date and time portion)
#
# If you have existing iptables logfiles, and it is assumed their values are stored 
# in the format: 
#
# iptables -L traffic_out -vn
#
# then you may import these logfiles into a sqlite database using this commandline:
#
# ./ipt-parse 0 20090401 20090501 usagelogs/ traffic_in_ traffic_out_ flags=import=BANDWIDTH
#
# where all parameters are the same as if you were generating a statistical
# output for a given date range with the addition of the following parameter:
#
# flags=import=BANDWIDTH
#
# where BANDWIDTH is the path and location fo the sqlite database to import into.
#
# It should be safe to run multiple times (Even for the same date range) because
# there are validations to ensure that the same data does not get imported more than once.
#
# It is possible to shrink the size of the database IF, and ONLY IF you are willing to
# remove granularity by time for a specific date range (for example for and old
# date range where you really don't care what happened throughout the day)
# To shrink the database for a range of dates (compress intra-day values into
# daily values) use this syntax:
#
# ./ipt-parse 5 20090301 20090510 ~/Code/sqlite-3.6.13/BANDWIDTH
#
# The commandline above will scan all data between March 1st, 2009 - May 10, 2009
# and summarize ststs daily, then delete old records for that date range and insert 
# the new daily summarized values, and resize the database.
#
# If you want to see additional host information you may add the optional commandline parameter:
#
# flags=morehostinfo
#
# For CONSOLE output it will append extra host infomation to the IP Address field,
# while for HTML output it will turn the IP Address fields into links with pop-up hints
# showing the additional host information for each IP Address. For example you
# can generate using a SQLITE database with extra host info as follows:
#
# ./ipt-parse 4 today today BANDWIDTH flags=morehostinfo > dailybandwidthlive.html
#
# HOW TO Modify the Tomato UI and add new content:
#
# Add the following line to the firewall rules, if you want to make use of the
# modified UI attached with this package: 
#
# ./SetupUI.sh > SetupUI.log 
#
# You will need to copy the folder tomato_dynamic_files and the 
# file SetupUI.sh to /cifs1/ and possibly edit the contents to suit your paths etc.
# You'll also need to ensure it has execute access by executing (from tomato shell)
#
# chmod 777 /cifs1/SetupUI.sh
#
# Eventually I will look at creating my own firmware mod with these items more fully
# integrated into tomato unless somebody else beats me to it first.
# 

