APP AdsMonster loves Ads

Joe

Thành viên VIP
21/1/13
3,075
1,342
113
Hi,
I have showed you how to implement a CookiesMonster that only loves Browser cookies and thrives on cookies. Today I show you how to turn a CookiesMonster to an AdsMonster. What is it, an AdsMonster? Well, an AdsMonster loves Advertisings and thrives on them. When an AdsMonster sees an advertising nothing can keep him back from storming to the ad and to munch it with delight. Together with the cookies, of course.

Why you need an AdsMonster? All the ads are superfluous to the content of the page, but they have to be loaded. And that takes a lot of time, causes congestion on the gateway and delays the response time (or completion time). And
  1. because you are tired of all the bling-bling ads that divert your concentration from reading an interesting article
  2. because your computer or laptop is powered by a weak processor, or it leaks of memory
  3. because some Ads Providers are too stupid to save money by employing only mediocre web developers and web designers who are cheap and can only create ads with unoptimized pages or images
Case 1: You have to get rid of the ads by elimination them. But How?
Case 2: You have to upgrade your hardware and that could cost you some hundred bucks. And the result is certainly not foolproof that your upgraded computer or laptop would run faster.
Case 3: The stupidity is a phenomenon that even God can do nothing against it.

So, three cases seem to be hopeless and don't provide you any satisfying solution, right? No, it isn't right. As an IT developer you have to find the way to eliminate the ads from the pages you are interested. It's the HOW of case 1). For case 3) you can do nothing because you have no access to the sources or the images of these stupid ads providers. What's about case 2) ? Forget it. It costs money and shows minimal effect, not even perceivable that your computer or laptop runs faster.

Well, back to case 1). Do you know that software optimization is difficult, but the most efficient way to speed up the performance more than 100%, while hardware optimization could achieve only up to 100% if you were really lucky. Software process is always fast, very fast even it is executed by an One-Core Intel processor. But software optimization is, as mentioned, difficult because it requires some imaginative and inventive perception from the software developer. Imaginative Perception is the understanding of the working and processing mechanism. Inventive Perception is the venture into the unknown based on the imaginative perception.

The working and processing mechanism of Client/Server is simple: the Client sends a request to a server and the Server replies to the client. Bidirectional. Now it comes to the evil tricks of the advertisers who "corrupt" the server with their ads. The ads are either visible (images, texts) or invisible (tracking). The server sends back the required pages to the Client WITH the paid ads. It doesn't matter for the server whether the client wants or not.

With this knowledge you have two choices:
  1. accept and ignore the bling-bling ads
  2. try to get rid of the bling-bling ads with something
The inventive perception is this "something". And this something is an app, a software tool, that intercepts the Server replies, filters the ads that are listed in, for example, a blacklist and finally forwards the "sanitized" pages to the Client. All that sounds complicated, but it is in reality simple and as common as water. The App is no other thing than a Proxy Server.

YouAndTheWeb.jpg

To do that you need to know the web names of all the advertisers whom you want to get rid. Example: blacklist.txt. To save comparing time the list should contain only an unique part of the name. For example: adserv is sufficient enough for the name www.adserver.com.

How to get the Ads-Server name? On Windows 10, for example, you move the mouse cursor to the ad, click the right button, select Copy Link Address, then paste it on an editor (e.g. notepad) to attain the full link. Example: https://www.trivago.com/en-US/srl?_wr=3&tc=26&cip=49030703040509& ....

With the link you are able to pick an unique part out of the link. Here: trivago for your blacklist. Easy, isn't it?
Code:
#---------------------------------------------------+
# Comment begins with a HASH (#) at first position  |
#---------------------------------------------------+
adfarm
adnxs
adserv
adtima
admicro
adnxs
ads-
ads.
tracking
trackjs
#
trivago
#
yield
Note that the CookiesList.txt is slightly different from the CookiesList.txt of CookiesMonster.
Code:
#-------------------------------------------------------------------------------
# WINDOWS 10.
COOKIES_0=/AppData/Roaming/Opera Software/Opera Stable/cookies
COOKIES_1=/AppData/Roaming/Opera Software/Opera Stable/cookies-journal
COOKIES_2=/AppData/Local/Google/Chrome/User Data/Default/cookies
COOKIES_3=/AppData/Local/Google/Chrome/User Data/Default/cookies-journal
COOKIES_4=/AppData/Local/Google/Chrome/User Data/Default/Safe Browsing cookies
COOKIES_5=/AppData/Local/Google/Chrome/User Data/Default/Safe Browsing cookies-journal
COOKIES_6=/AppData/Local/Microsoft/Edge/User Data/Default/cookies
COOKIES_7=/AppData/Roaming/Mozilla/Firefox/Profiles/usqz5p89.default/cookies.sqlite
#
OPERA=/AppData/Roaming/Opera Software/Opera Stable
EDGE=/AppData/Local/Microsoft/Edge/User Data/Default
CHROME=/AppData/Local/Google/Chrome/User Data/Default
FIREFOX=/AppData/Roaming/Mozilla/Firefox/Profiles/usqz5p89.default
# min. 1000
SWEEP = 1000
# min. 5000
POOL_SIZE = 5000
#
BLACKLIST = blacklist.txt
#-------------------------------------------------------------------------------
and here is the AdsMonster.java. AdsMonster is an extension of CookiesMonster.
Java:
import java.io.*;
import java.net.*;
import java.nio.*;
import java.util.*;
import java.nio.file.*;
import java.nio.channels.*;
import java.util.concurrent.*;
//
import javax.swing.*;
// Joe T. Nartca (C)
public class AdsMonster extends Thread {
  // Constructor
  public AdsMonster(String file, String sPort) throws Exception {
    port = Integer.parseInt(sPort); // get AdsMonster port
    // load the config file for existed browsers
    // and "sweeping" time
    FileInputStream fin = new FileInputStream(file);
    prop = new Properties();
    prop.load(fin);
    fin.close();
    start( );
  }
  @SuppressWarnings("unchecked")
  public void run( ) {
    // Cookies sweeping time (milliSeconds)
    String home = System.getProperty("user.home");
    String tmp = prop.getProperty("SWEEP");
    if (tmp != null) { // sweeping time
      time = Integer.parseInt(tmp);
      if (time < 1000) time = 1000;
    }
    int size = 5000;
    tmp = prop.getProperty("POOL_SIZE");
    if (tmp != null) { // Pool size
      size = Integer.parseInt(tmp);
      if (size < 5000) time = 5000;
    }
    // set FixedThreadPool
    pool = Executors.newFixedThreadPool(size);
    fLst = new ArrayList<>(); // CookiesList
    for (int i = 0; ; ++i) {
      tmp = prop.getProperty("COOKIES_"+i);
      if (tmp == null) break;
      fLst.add((home+tmp).replace("/", File.separator));
      String fN = fLst.get(i);
      pool.submit(()-> { // start monitoring this file
        // check for Cookie entry by listening to the WatchKey (see CookiesMonster)
        RandomAccessFile raf = null;
        try {
          raf = new RandomAccessFile(fN, "rw");
          int p = fN.lastIndexOf(File.separator);
          WatchKey wk = Paths.get(fN.substring(0, p)).register(FileSystems.getDefault().newWatchService(),
                                  StandardWatchEventKinds.ENTRY_CREATE, StandardWatchEventKinds.ENTRY_MODIFY);
          String fil = fN.substring(p+1);
          OUT: while (!end) {
            for (WatchEvent<?> event : wk.pollEvents()) {
              WatchEvent.Kind<?> kind = event.kind();
              if (((WatchEvent<Path>)event).context().toString().equalsIgnoreCase(fil) &&
                  (kind == StandardWatchEventKinds.ENTRY_MODIFY || kind == StandardWatchEventKinds.ENTRY_CREATE)) {
                if (raf.length() > 0) try {
                  raf.setLength(0); // reset cookies
                  break;
                } catch (Exception ex) { }
              }
              if (end) break OUT;
            }
            wk.reset();
            try { Thread.sleep(time); }
            catch (Exception ie) { }
          }
        } catch (Exception ex) {
          ex.printStackTrace();
          if (raf != null) try {
            raf.close();
          } catch (Exception e) { }
        }
      });
    }
    try { // ProxyServer
      blList = new ArrayList<>();
      tmp = prop.getProperty("BLACKLIST");
      if (tmp != null) { // BlackList existed
        Object[] objs = Files.readAllLines((new File(tmp)).toPath()).toArray();
        for (int i = 0; i < objs.length; ++i) // ignore all lines start with '#'
        if (((String)objs[i]).charAt(0) != '#') blList.add((String)objs[i]);
        blist = blList.toArray(new String[blList.size()]);
        blList.clear();
      }
      server = ServerSocketChannel.open();
      server.socket().bind(new InetSocketAddress(port));
      server.setOption(StandardSocketOptions.SO_RCVBUF, 32768);
      while (!end) pool.execute(new Monster(server.accept()));
    } catch (Exception ex) {
      if (!end) ex.printStackTrace();
      System.exit(0);
    }
  }
  //
  public void halt( ) {
    end = true;
    cleanNow( );
    if (pool != null) pool.shutdownNow();
    System.exit(0);
  }
  // force to clean up all cookies of the listed browsers
  public void cleanNow( ) {
    if (prop != null) try {
      if (fLst != null) for (String cookies:fLst) {
        RandomAccessFile raf = new RandomAccessFile(cookies, "rw");
        raf.setLength(0);
        raf.close();
      }
      String home = System.getProperty("user.home");
      String opera = prop.getProperty("OPERA");
      if (opera != null) { // Opera
        opera = (home+opera).replace("/", File.separator) + File.separator;
        FileOutputStream fou = new FileOutputStream((opera+"/opera_autoupdate.log").replace("/", File.separator), false);
        fou.close();
        File[] files = new File(opera).listFiles();
        for (File file : files) if (file.isFile()) {
          String fn = file.getName(); // remove all ssdfp... files
          if (fn.indexOf("ssd") == 0 && fn.lastIndexOf(".lock") < 0 || fn.lastIndexOf(".old") > 0) file.delete();
        }
      }
      String chrome = prop.getProperty("CHROME");
      if (chrome != null) { // Chrome
        chrome = (home+chrome).replace("/", File.separator) + File.separator;
        File[] files = new File(chrome).listFiles();
        for (File file : files) {
          String fn = file.getName();
          if (file.isFile() && (fn.lastIndexOf(".tmp") > 0 || fn.lastIndexOf(".old") > 0)) file.delete();
        }
      }
      String firefox = prop.getProperty("FIREFOX");
      if (firefox != null) {
        firefox = (home+firefox).replace("/", File.separator) + File.separator;
        File[] files = new File(firefox).listFiles();
        for (File file : files) {
          String fn = file.getName();
          if (file.isFile() && (fn.lastIndexOf(".tmp") > 0 || fn.lastIndexOf(".bak") > 0)) file.delete();
        }
      }
    } catch (Exception ex) { }
  }
  public void printList( ) {
    if (blON) {
      if (blList.size() > 0) for (String bl : blList) System.out.println("BlackListed: "+bl);
      else if (blON) System.out.println("NO site is blacklisted.");
      else System.out.println("Blacklist is disabled.");
    }
  }
  // Proxy between Browser and Host
  private class Monster implements Runnable {
    public Monster(SocketChannel soc) {
      this.soc = soc;
    }
    public void run() {
      String webHost = null;
      try {
        ByteBuffer wbuf = ByteBuffer.allocateDirect(32768);
        int len = soc.read(wbuf);
        byte[] buf = new byte[len+64];
        ((ByteBuffer)wbuf.flip()).get(buf, 0, len);
        // HTTPS with CONNECT
        if (buf[0] == 'C') { // CONNECT: 8 letters
          for (int i = 8; i < len; ++i) if (buf[i] == ':') {
            webHost = new String(buf, 8, i-8);
            if (isBlacklisted(webHost)) return;
            break;
          }
          // SSL Port 443
          webSoc = SocketChannel.open(new InetSocketAddress(webHost, 443));
          soc.write(ByteBuffer.wrap("HTTP/1.1 200 OK\r\n\r\n".getBytes()));
        } else // HTTP with PUT/GET/POST/
        if (buf[0] == 'G' || buf[0] == 'P' && buf[1] != 'A') {
          int i, b = buf[3] == ' '? 4 : 5; // 4:GET/PUT, 5:POST
          for(i = b+7; i < len; ++i) // 7:' http//'
          if (buf[i] == '/' || buf[i] == ' ') {
            webHost = new String(buf, b+7, i-b-7);
            if (isBlacklisted(webHost)) return;
            break;
          }
          len -= (i-b); // remove http://host
          System.arraycopy(buf, i, buf, b, len);
          for (i += 9; i < len; ++i) // HTTP/1.1\r -> 9
          if (buf[i] == '\n' && buf[i+1] == 'P' && buf[i+5] == 'y') {
            len -= 6; // remove Proxy-connection: keep-alive
            System.arraycopy(buf, i+7, buf, i+1, len);
            break;
          }
          webSoc = SocketChannel.open(new InetSocketAddress(webHost, 80));
          webSoc.write(ByteBuffer.wrap(buf, 0, len));
        } else { // ignore PATCH, TRACE, DELETE, OPTIONS, etc.
          soc.close();
          return;
        }
        // prepare for the dataXchange
        soc.socket().setTcpNoDelay(true);
        webSoc.socket().setTcpNoDelay(true);
        soc.socket().setSendBufferSize(32768);
        webSoc.socket().setReceiveBufferSize(32768);
        // Browser -> Host
        pool.execute(()-> {
          try {
            wbuf.clear();
            while (soc.read(wbuf) > 0) {
              webSoc.write((ByteBuffer)wbuf.flip());
              wbuf.clear();
            }
            setDone();
          } catch (Exception e) { }
        });
        // Host -> Browser
        ByteBuffer rbuf = ByteBuffer.allocateDirect(32768);
        while (webSoc.read(rbuf) > 0) {
          soc.write((ByteBuffer)rbuf.flip());
          rbuf.clear();
        }
        setDone();
      } catch (Exception e) { }
    }
    private boolean isBlacklisted(String host) throws Exception {
      if (blON) for (int l = 0; l < blist.length; ++l) {
        if (host.indexOf(blist[l]) >= 0) {
          synchronized (this) {
             blList.add(host);
          }
          soc.write(ByteBuffer.wrap("HTTP/1.1 200 OK\r\n\r\n".getBytes()));
          soc.close();
          return true;
        }
      }
      return false;
    }
    private void setDone() throws Exception {
      if (done) {
        soc.close();
        webSoc.close();
      } else done = true;
    }
    private SocketChannel soc, webSoc;
    private volatile boolean done = false;
  }
  //
  private Properties prop;
  private ExecutorService pool;
  private int time = 500, port;
  private String[] blist = {   };
  private List<String> fLst, blList;
  private ServerSocketChannel server;
  private static AdsMonster monster;
  private static  boolean blON = true;
  private volatile boolean end = false;
  //----------------------------------------------------------------------------------------------------
  public static void main(String... argv) throws Exception {
    UIManager.setLookAndFeel("com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel");
    JFrame jf = new JFrame("AdsMonster - Joe Nartca / Congdongjava.com (C) -");
    JLabel msg = new JLabel("Enter Port then CookiesList and press ENTER to start AdsMonster");
    jf.setDefaultCloseOperation(JFrame.DO_NOTHING_ON_CLOSE);
    //
    JTextField tport = new JTextField("30751");
    JRadioButton rad = new JRadioButton("BlackList ON");
    rad.setSelected(true);
    rad.addActionListener(e -> {
      rad.setText("BlackList "+(!blON?"ON":"OFF"));
      if (tport.isEditable()) tport.grabFocus();
      blON = !blON;
    });
    JLabel lp = new JLabel("ProxyPort");
    JTextField conf = new JTextField(argv.length > 0? argv[0]:"CookiesList.txt");
    tport.addActionListener(e -> {
      conf.grabFocus();
    });
    conf.addActionListener(e -> {
      if (monster == null) try {
        monster = new AdsMonster(conf.getText(), tport.getText());
        msg.setText("AdsMonster is lurking...");
        tport.setEditable(false);
        conf.setEditable(false);
        rad.grabFocus();
      } catch (Exception ex) { }
    });
    JLabel labC = new JLabel("CookiesList file");
    JButton but = new JButton("CLEAN COOKIES NOW");
    but.addActionListener(e -> {
      if (monster != null) {
        monster.cleanNow();
        msg.setText("All cookies were swept away");
      }
    });
    JButton stop = new JButton("EXIT & CLEAN UP");
    stop.addActionListener(e -> {
      if (monster != null) monster.halt();
      else System.exit(0);
    });
    JButton print = new JButton("PRINT BLACKLISTED");
    print.addActionListener(e -> {
      if (monster != null) monster.printList();
    });
    jf.addWindowListener(new java.awt.event.WindowAdapter() {
       public void windowClosing(java.awt.event.WindowEvent ew){
        if (monster != null) monster.halt();
        else System.exit(0);
       }
    });
    JPanel north = new JPanel();
    north.add(lp); north.add(tport); north.add(labC); north.add(conf); north.add(rad);
    JPanel center = new JPanel();
    center.add(but); center.add(print); center.add(stop);
    JPanel south = new JPanel();
    south.add(msg);

    jf.add("North", north);
    jf.add("Center", center);
    jf.add("South", south);
    jf.pack();
    jf.setVisible(true);
  }
}
As you see, the AdsMonster is relatively simple an sufficient has only 320 lines (incl. some comments). To run this app you have to set your computer to proxy. On Windows 10 for example:
  • Settings
  • Network and Internet
  • Proxy
  • Proxy Server ON
  • Type localhost and port of your choice (e.g. 6789)
  • Save
Note: If you don't want to run your Nrowser thru AdsMonster you have to turn off: Proxy Server OFF
Here is the result with an "incredible" quick response ;)

AdsMonster.jpg

Enjoy
Joe
 
Sửa lần cuối: