Tag: phantomsql

EventEmitter

Our EventEmitter in PhantomSQL is based on NodeJS version so they should be compatible. Here are couple examples on how to use the emitter.

Basic usage of registering and listening to an event.

"use strict";

const {EventEmitter} = require('events');

// Dump all the args
em.on('hello-event', (...arg)=> {console.info("Hello event handler : " + arg)});
// Handler without args
em.on('hello-event', ()=> {console.info("Another handler")});
// passed in arguments
em.on('hello-event', (id, val)=> {console.info("Handler :"+id +", "+ val)});

// emit event
em.emit('hello-event', 123, 'ABC');

A more typical example would be to extend via prototype.

"use strict";
const {EventEmitter} = require('events');

function HelloService()
{
	// Extends via prototype
	Object.setPrototypeOf(HelloService.prototype, EventEmitter.prototype);
	
	this.hello = function()
	{
		console.info("Hello service called");
		this.emit('hello');
	}
}  

const service = new HelloService();

service.on('hello', ()=> {console.info("Hello Handler called")});
service.hello();

May 30, 2017

PhantomSQL 2.0
Since the beginning of this project, I have learned many lessons.

First of all the syntax of the language was to closely related to SQL, as this was the language that I wanted to mimic and fit into this framework. Main reason for that was that I wanted something familiar to the end user, I was hoping for easier adoption rate. Secondly, the lack of Web 2.0+ futures. They were not thought out in the beginning and shoehorning them in just made no sense and even more complicated the language.

Example : Grab data from woot (v1)
```
foreach @srcUrl : [http://www.woot.com/, http://home.woot.com/, http://shirt.woot.com/, http://kids.woot.com/]
begin
	set @doc = select 
			xpaths("//*[@class='amount']/text()"),
			xpaths("//h2[@class='fn']/text()"),
			xpaths("//*[@class='lightBox' and @rel='sale']/img/@src"),			
			xpaths("//*[@class='wootOffProgressBarValue']/@style")
		 from @srcUrl
		
	set @images = @document.xpaths("//*[@class='lightBox' and @rel='sale']/@href")
	set @first = @doc[0]
	set @price = @first[0]	
	set @title = @first[1]
	set @image = @first[2]	
	set @wootoff = @first[3].split(":")
	set @wootoff = @wootoff[1].replace("%", "")

	insert  products (@srcUrl, @title, @price, @wootoff, @image, @images)
end
```
Couple problems that this introduced, first of all, we were constrained by the original language structure. Let’s take simple snippet
```
set @price = @first[0]
```
It might not look like much but this type of declaration is very declarative and it could easily be removed from the syntax.

Another drawback was the lack of extensibility. Many of the basic functions have been hardcoded into the parser, thus adding any new language extension was dependent on recreating the AST. Only later and ‘dynamic’ methods have been added but again they were an afterthought and seem more like a hack.

There was no way to reuse existing libraries, basically, this means I had to reimplement all the features that someone already did. This only became apparent to me when I needed to run a md5 checksum on one of the downloaded files.

Another issue for me was that there was no good support for collections (List, Set, Dictionary). There was the basic support for ‘list’ and then later ‘dictionary ‘ but again it was not thought out and missed many features.

As we can see there were quite a few design issues that were not thought out in the initial version. Many of this thing have been learned though running the system on real word problems. As long as we are able to take this and adapt I think we will be in good shape.

Roadmap
- Reworking language syntax
  - Based on modern languages / frameworks (EcmaScript / Python / Rubby / SQL/ Node.js)
  - Web 2.0 (Support for extracting data from dynamic websites)
- Distributed Processing(Will run on a distributed clustering framework)
- Package Manager (PhantomSQL Package Manager(PPM))
January 23, 2017

PhantomSQL released

Finally I have released my PhantomSQL project.

PhantomSQL is a domain specific language designed for mining content from static and dynamic sources, It closely resembles SQL with features borrowed from other popular dynamic languages.
It can be run as a interpreter or ‘server’ mode, it comes with type 4 JDBC driver for ease of integration with java applications.

Sample Queries

Here are just few examples taken from the project site to display some of the syntax of the language.

Hello World

Following example illustrates how to query google.com for some blog search results.


 select first css("#ires a") as title from https://www.google.com/search
  using get with {'q': "josh bloch", 'tbm':"blg"}
  
  print @title +" : "+ @title['href']

Crawling Flicker.com

Flicker Integration
This query does nothing more than query flicker.com for ‘nabilishes’ and then crawl the site using GET while the ‘css(“a.Next”)’ condition matches, at the end it prints how many results have been found.

 @result = select css(".pc_img") from http://www.flickr.com/search 
 using get with {'q': "nabilishes", 'm' : "text"}
 crawl(css("a.Next")) 
 print @result.length()

Extracting images from Flicker.com search results

Here we build on previous example by adding the ‘WHILE-SELECT’ construct and actually saving the image with the ‘save’ function.

while select css(".pc_img", "src", true) as img from http://www.flickr.com/search  
       using get with {'q': "nabilishes", 'm' : "text"}
       crawl(css("a.Next")) 
 begin
   save (@img)
 end

Retrieving Binary Content

Following two examples are equivalent.

Following example illustrates how to retrieve binary content from a site and save it on the filesystem when running via interpreter.


select first css("#main_image", "src", true) as item from https://1saleaday.com
save(@item)

Following example illustrates how to retrieve binary content from a site with JDBC Driver and dump the file to file system.

    public void retieveFile()
    {
        try
        {
            try
            {
                Class.forName("com.gbltech.phantomsql.driver.PhantomDriver");
            }
            catch (Exception e)
            {
                e.printStackTrace();
            }
            Connection conn = DriverManager.getConnection("jdbc:phantomsql://localhost?characterEncoding=utf8");
            Statement statement = conn.createStatement();
            ResultSet resultSet = statement
                .executeQuery("select first css(\"#main_image\", \"src\", true) as item from https://1saleaday.com");
            if (resultSet.next())
            {
                OutputStream out = null;
                Blob blob = resultSet.getBlob(1);
                try
                {
                    out = new FileOutputStream(new File("./test.jpg"));
                    InputStream is = blob.getBinaryStream();
                    byte[] buff = new byte[1024];
                    int read = 0;
                    while ((read = is.read(buff, 0, buff.length)) != -1)
                    {
                        out.write(buff, 0, read);
                    }
                    out.flush();
                }
                catch (FileNotFoundException e)
                {
                    e.printStackTrace();
                }
                catch (IOException e)
                {
                    e.printStackTrace();
                }
                finally
                {
                    if (out != null)
                    {
                        try
                        {
                            out.close();
                        }
                        catch (IOException e)
                        {
                            e.printStackTrace();
                        }
                    }
                }
            }
        }
        catch (SQLException e)
        {
            e.printStackTrace();
        }
    }

This is just a taste of what the PhantomSQL can do, there is much more so go check it out.
I am looking for feedback, criticism and bug reports to let me make the project better, so if you have something drop me a line.

November 21, 2012