bordercheck release v0.2b

Border-Check/README.md

@@ -0,0 +1,158 @@
+Border Check v0.2 - 2015
+========================
+
+Border Check (BC) is a browser extension that illustrates the physical and political realities of the internet's infrastructure using free software tools.
+
+As one surfs the net, data packets are sent from the user's computer to the target server. These data packets go on a journey hopping from server to server, potentially crossing multiple countries, until the packets reach the desired website. In each of the countries that are passed different laws and practices can apply to the data, influencing whether or not authorities can inspect, store or modify that data.
+
+In realtime BC lets you know which countries you surf through as you browse the web. Additionally BC will illustrate this process on a world map and (where available) provide you with contextualizing information on that country's laws and practices regarding your data.
+
+BC runs on OSx and Unix systems.
+
+Currently supporting the following browsers on OSX and Unix systems: Firefox, Chromium, Chrome, Safari
+
+NOTE: Browser history has to be enabled.
+
+
+### Installing BC:
+
+BC v0.2 (2015) provides a "Wizard" installer that makes installing the required libraries and packages for BC easier. 
+
+The wizard runs automatically the first time you run BC. It will check if your system has all the dependencies met for BC and if not the wizard will download and install them automagically. During the wizard BC will also install the required maps and databases.
+
+Border Check has the following dependencies:
+
+### Python:
+
+* [pygeoip](https://pypi.python.org/pypi/pygeoip/0.2.7) 
+* [lxml](https://pypi.python.org/pypi/lxml/3.2.3)
+* [biplist](https://pypi.python.org/pypi/biplist/0.5) (if you plan to use Safari)
+
+For **Debian**-based systems (ex: Ubuntu), first run:
+
+`sudo apt-get install python-pip` to install [pip](https://pypi.python.org/pypi/pip/), python's super usefull package manager.
+
+If you already have pip: `pip install lxml` and `pip install pygeoip`
+
+On **OSX** it's recommended you install [Homebrew](http://brew.sh/) first and use that to get [python + pip](https://github.com/mxcl/homebrew/wiki/Homebrew-and-Python).
+
+
+### LFT
+
+BC uses [LFT v3.73 (08/2014)](http://pwhois.org/lft/) for tracerouting.
+
+On **Debian** and **Ubuntu** it needs to be built from source and requires libpcap [`sudo apt-get install python-libpcap`]
+
+To make this process more easy, BC (v0.2 2015) provides you a binary with the source. It is called `lft.linux` and is on folder `bin`.
+
+On **OSX** you can use Homebrew: `brew install lft` which will automatically download and build the package.
+
+#### Note on provided LFT binary
+
+We do provide a static compiled binary of lft in the bin directory.
+
+Please consider that this is for convenience only, you are invited to make your own.
+
+
+### GeoIP databases and js libraries
+
+BC will automatically unpack the newest geoip databases on the first run. 
+
+Javascript mapping libraries are included in the package.
+
+
+## Options and features:
+
+See the included examples for usage.
+
+```
+bc [OPTIONS] 
+
+  --version             show program's version number and exit
+  -h, --help            show this help message and exit
+  -d, --debug           debug mode
+  -l LFT_PATH           path to lft (fetch from source or use provided binary)
+  --xml=EXPORT_XML      export traces to xml (ex: --xml foo.xml)
+  --load=IMPORT_XML     import traces (non root required) (ex: --load bar.xml)
+  --bh=BROWSER_HISTORY  set browser's history path
+  -b BROWSER            set browser type to be used: F = Firefox / C = Chrome / S = Safari / Ch = Chromium / N = None
+
+```
+
+#### Examples of usage:
+
+Normal BC run:
+
+`$ python bc`
+
+Debug mode:
+
+`$ python bc --debug`
+
+Export 'tracing' results to xml:
+
+`$ python bc --xml "mytravel.xml"`
+
+
+#### More options (set browsers, paths, etc):
+
+Use Firefox browser: 
+
+`$ python bc -b F`
+
+Use Chromium browser: 
+
+`$ python bc -b Ch` 
+
+Set browser's history path manually, on Galeon/Unix:
+
+`$ python bc --bh ".galeon/mozilla/galeon/places.sqlite"`
+
+Set browser's history path manually, on Chrome/OSx:
+
+`$ python bc --bh "Library/Application Support/Google/Chrome/Default/History"`
+
+Set browser's history path manually, on Safari/OSx:
+
+`$ python bc --bh "Library/Safari/History.plist"` 
+
+Set browser's history path manually, on Firefox/Unix:
+
+`$ python bc --bh "Library/Safari/History.plist"` 
+
+Import 'traces' from xml:
+
+`$ python bc --load "mytravel.xml"`
+
+Set lft path to be used by BC traces:
+
+`$ python bc -l "/usr/bin/lft"`
+
+
+### License
+
+Border Check is free software, and may be redistributed under [GPL v3].
+
+
+### Contribute: 
+
+If you want to contribute to BC development, reporting a bug, providing a patch, commenting 
+on the code base or simply need to find help to run it, please go to:
+
+      irc.freenode.net / channel: #BorderCheck
+
+If nobody gets back to you, then drop an e-mail.
+
+To make donations use the following hashes:
+  
+      - Bitcoin: 1Q63KtiLGzXiYA8XkWFPnWo7nKPWFr3nrc
+      - Ecoin: ETtSteMWxjY7RKWZGMNSkX7eC3BJ21VYXE
+
+
+### Contact
+
+Please report any problems you encounter using/installing Border Check to:
+
+ - Roel Roscam Abbing (rscmbbng@riseup.net)
+ - psy (epsylon@riseup.net)
+

bc → Border-Check/bc

@@ -1,8 +1,8 @@
-#!/usr/bin/env python
+#!/usr/bin/env python2
 # -*- coding: iso-8859-15 -*-
 """
 BC (Border-Check) is a tool to retrieve info of traceroute tests over website navigation routes.
-GPLv3 - 2013 by psy (epsylon@riseup.net)
+GPLv3 - 2013-2014-2015 by psy (epsylon@riseup.net)
 """
 from main import bc
 

doc/CHANGELOG → Border-Check/doc/CHANGELOG

@@ -2,6 +2,19 @@
 Changelog: Border-Check
 ==============================
 
+=================
+May 13, 2015:
+=================
+
+- added "wizard" installer
+- added LFT binary to source
+- added more maps
+- added metadata
+- added travelling feature on map
+- added history of urls visited on map
+- added controllers to map (play|hide|stop)
+- fixed some minor bugs
+- public release: Border Check v0.2
 
 =================
 October 4, 2013:

doc/INSTALL → Border-Check/doc/INSTALL

@@ -1,6 +1,6 @@
-========================================================
-BC - Border Check - 2013/2014
-========================================================
+========================
+Border Check v0.2 - 2015
+========================
 
 Border Check (BC) is a browser extension that illustrates the physical and political realities of the internet's infrastructure using free software tools.
 
@@ -90,10 +90,5 @@ Please report any problems you encounter using/installing Border Check to:
  - Roel Roscam Abbing (rscmbbng@riseup.net)
  - psy (epsylon@riseup.net)
 
-Or visit IRC Community:
-
- - Server: irc.freenode.net 
- - Channel: #BorderCheck
-
 =========
 

Border-Check/doc/README

@@ -0,0 +1,96 @@
+Border Check v0.2 - 2015
+========================
+
+Border Check (BC) is a browser extension that illustrates the physical and political realities of the internet's infrastructure using free software tools.
+
+As one surfs the net, data packets are sent from the user's computer to the target server. These data packets go on a journey hopping from server to server, potentially crossing multiple countries, until the packets reach the desired website. In each of the countries that are passed different laws and practices can apply to the data, influencing whether or not authorities can inspect, store or modify that data.
+
+In realtime BC lets you know which countries you surf through as you browse the web. Additionally BC will illustrate this process on a world map and (where available) provide you with contextualizing information on that country's laws and practices regarding your data.
+
+Currently supporting the following browsers on OSX and Unix systems: Firefox, Chromium, Chrome, Safari
+
+NOTE: Browser history has to be enabled.
+
+See the included examples for usage.
+
+```
+bc [OPTIONS] 
+
+  --version             show program's version number and exit
+  -h, --help            show this help message and exit
+  -d, --debug           debug mode
+  -l LFT_PATH           path to lft (fetch from source or use provided binary)
+  --xml=EXPORT_XML      export traces to xml (ex: --xml foo.xml)
+  --load=IMPORT_XML     import traces (non root required) (ex: --load bar.xml)
+  --bh=BROWSER_HISTORY  set browser's history path
+  -b BROWSER            set browser type to be used: F = Firefox / C = Chrome / S = Safari / Ch = Chromium / N = None
+
+```
+
+#### Examples of usage:
+
+Normal BC run:
+
+`$ python bc`
+
+Debug mode:
+
+`$ python bc --debug`
+
+Export 'tracing' results to xml:
+
+`$ python bc --xml "mytravel.xml"`
+
+
+#### More options (set browsers, paths, etc):
+
+Use Firefox browser: 
+
+`$ python bc -b F`
+
+Use Chromium browser: 
+
+`$ python bc -b Ch` 
+
+Set browser's history path manually, on Galeon/Unix:
+
+`$ python bc --bh ".galeon/mozilla/galeon/places.sqlite"`
+
+Set browser's history path manually, on Chrome/OSx:
+
+`$ python bc --bh "Library/Application Support/Google/Chrome/Default/History"`
+
+Set browser's history path manually, on Safari/OSx:
+
+`$ python bc --bh "Library/Safari/History.plist"` 
+
+Set browser's history path manually, on Firefox/Unix:
+
+`$ python bc --bh "Library/Safari/History.plist"` 
+
+Import 'traces' from xml:
+
+`$ python bc --load "mytravel.xml"`
+
+Set lft path to be used by BC traces:
+
+`$ python bc -l "/usr/bin/lft"`
+
+
+### Contribute: 
+
+If you want to contribute to BC development, reporting a bug, providing a patch, commenting 
+on the code base or simply need to find help to run it, please go to:
+
+      irc.freenode.net / channel: #BorderCheck
+
+If nobody gets back to you, then drop an e-mail:
+
+    - Roel Roscam Abbing (rscmbbng@riseup.net)
+    - psy (epsylon@riseup.net)
+
+To make donations use the following hashes:
+  
+      - Bitcoin: 1Q63KtiLGzXiYA8XkWFPnWo7nKPWFr3nrc
+      - Ecoin: ETtSteMWxjY7RKWZGMNSkX7eC3BJ21VYXE
+

Border-Check/doc/THANKS

@@ -0,0 +1 @@
+UFO & Ikujam

options.py → Border-Check/options.py

@@ -2,25 +2,34 @@
 # -*- coding: iso-8859-15 -*-
 """
 BC (Border-Check) is a tool to retrieve info of traceroute tests over website navigation routes.
-GPLv3 - 2013 by psy (epsylon@riseup.net)
+GPLv3 - 2013-2014-2015 by psy (epsylon@riseup.net)
 """
 import optparse
+import json
 
 class BCOptions(optparse.OptionParser):
     def __init__(self, *args):
         optparse.OptionParser.__init__(self, 
                            prog='bc.py',
-			   version='\nBC (Border-Check) 0.1v - 2013 - (GPLv3.0)\n',
+			   version='\nBC (Border-Check) v0.2 - 2015 - (GPLv3.0)\n',
                            usage= '\n\nbc [OPTIONS]')
-
+ 
+        self.add_option("-w", "--wizard", action="store_true", dest="wizard", help="wizard installer")
         self.add_option("-d", "--debug", action="store_true", dest="debug", help="debug mode")
+        self.add_option("-l", action="store", dest="lft_path", help="path to lft (fetch from source or use provided binary)")
         self.add_option("--xml", action="store", dest="export_xml", help="export traces to xml (ex: --xml foo.xml)")
-        self.add_option("--load", action="store", dest="import_xml", help="import traces to show (ex: --load bar.xml)")
+        self.add_option("--load", action="store", dest="import_xml", help="import traces (non root required) (ex: --load bar.xml)")
         self.add_option("--bh", action="store", dest="browser_history", help="set browser's history path")
-        self.add_option("-b", action="store", dest="browser", help="set browser manually: F = Firefox / C = Chrome / S = Safari / Ch = Chromium")
+        self.add_option("-b", action="store", dest="browser", help="set browser type to be used: F = Firefox / C = Chrome / S = Safari / Ch = Chromium / N = None")
         #self.add_option("--proxy", action="store", dest="proxy", help="set proxy server")
-
+        self._options={}
+        
     def get_options(self, user_args=None):
         (options, args) = self.parse_args(user_args)
+        self._options=options
         return options
 
+
+    def save_options(self):
+        optionfile=open("options.json","w")
+        json.dump(self._options,optionfile)

Border-Check/web/ajax.py

@@ -0,0 +1,116 @@
+#!/usr/bin/env python
+# -*- coding: iso-8859-15 -*-
+"""
+BC (Border-Check) is a tool to retrieve info of traceroute tests over website navigation routes.
+GPLv3 - 2013-2014-2015 by psy (epsylon@riseup.net)
+"""
+from xml.dom.minidom import parseString
+import xml.etree.ElementTree as ET
+import re
+import traceback
+reload=False
+
+try:
+  bc_status_file = open('bc.status')
+  s = str(bc_status_file.read())
+  if s =='fresh':
+    with open('bc.status', 'w') as file:
+      file.write("old")
+    reload=True
+except:
+  pass
+
+# todo :
+# * call ajax.py from here
+# * save last url timestamp as get parameter
+# * do parsing only if timestamp changes
+# * ui integration
+
+#function to split ISP company names from ASN
+def ASN_Split(asn):
+  name_parts = []
+  for i in asn.split():
+    if re.match(r'AS\d{1,6}$', i):
+      asn = i
+    elif not re.match(r'AS\d{1,6}$', i):
+      name_parts.append(i)
+  company = ' '.join(name_parts)
+  return (asn, company)
+
+
+def extract_data():
+  url=""
+  hop_list = []
+  hop_ip_list =[]
+  geoarray = []
+  latlong= []
+  asn_list =[]
+  server_name_list = []
+  timestamp_list = []
+  telco_list = []
+  country_code_list = []
+  last_hop = 0
+  n_hops=0
+  unique_country_code_list = []
+  # extract data from a xml file
+  f = open('data.xml', 'r')
+  f2 = open('data.xml', 'r')
+  xml = ET.parse(f)
+  data = f2.read()
+  dom = parseString(data.encode('utf-8'))
+  f.close()
+  f2.close()
+  last_hop = int(xml.findall('hop')[-1].text)
+  n_hops = dom.getElementsByTagName('hop')[-1].toxml().replace('<hop>', '').replace('</hop','')
+
+
+  for counter in range(0, last_hop+1):
+    url = xml.getroot().text
+    hop_element = parseString(dom.getElementsByTagName('hop')[counter].toxml().encode('utf-8'))
+    hop = xml.findall('hop')[counter].text
+    server_name = hop_element.getElementsByTagName('server_name')[0].toxml().replace('<server_name>','').replace('</server_name>','')
+    asn = hop_element.getElementsByTagName('asn')[0].toxml().replace('<asn>','').replace('</asn>','')
+    hop_ip = hop_element.getElementsByTagName('hop_ip')[0].toxml().replace('<hop_ip>','').replace('</hop_ip>','')
+    longitude = hop_element.getElementsByTagName('longitude')[0].toxml().replace('<longitude>','').replace('</longitude>','')
+    latitude = hop_element.getElementsByTagName('latitude')[0].toxml().replace('<latitude>','').replace('</latitude>','')
+    timestamp = hop_element.getElementsByTagName('timestamp')[0].toxml().replace('<timestamp>','').replace('</timestamp>','')
+    country_code = hop_element.getElementsByTagName('country_code')[0].toxml().replace('<country_code>','').replace('</country_code>','')
+
+    latlong = [float(latitude.encode('utf-8')), float(longitude.encode('utf-8'))]
+    geoarray.append(latlong)
+    asn_list.append(ASN_Split(asn.encode('utf-8'))[0])
+    telco_list.append(ASN_Split(asn.encode('utf-8'))[1])
+    hop = int(hop) +1
+    hop_list.append(str(hop))
+    hop_ip_list.append(hop_ip.encode('utf-8'))
+    server_name_list.append(server_name.encode('utf-8'))
+    timestamp_list.append(float(timestamp))
+    country_code_list.append(country_code.encode('utf-8'))
+
+    unique_country_code_list = set(country_code_list)
+  return """
+<script type="text/javascript">
+    counter_max = """+str(last_hop)+"""
+    latlong = """+str(geoarray)+"""
+    asn_list = """+str(asn_list)+"""
+    hop_ip_list = """+str(hop_ip_list)+"""
+    telco_list = """+str(telco_list)+"""
+    server_name_list = """+str(server_name_list)+"""
+    timestamp_list = """+str(timestamp_list)+"""
+    country_code_list = """+str(country_code_list)+"""
+    unique_country_code_list = """+str(list(unique_country_code_list))+"""
+    bcHistory.add('"""+url+"""',Array(counter_max,latlong,asn_list, hop_ip_list,telco_list,server_name_list,timestamp_list,country_code_list,unique_country_code_list))
+</script>
+"""
+
+try:
+  if reload:
+    output=extract_data()
+
+except:
+  args['error']="No data available"
+  traceback.print_exc()
+  reload=False
+
+if 'error' in args:
+  output=output+'<script>$("#error").html("'+args['error']+'")</script>'