How to pass Selenium pages to DEiXToBot

Recently we talked about Selenium and its potential combination with DEiXTo. It is a truly remarkable browser automation tool with numerous uses and applications. For those of you wondering how to programmatically pass pages fetched with Selenium to DEiXToBot on the fly, then here is a way (provided you are familiar with Perl programming):

# suppose that you have already fetched the target page with the WWW::Selenium object ($sel variable)
my $content = $sel->get_html_source(); # get the page source code

my ($fh,$name); # create a temporary file containing the page's source code
do { $name = tmpnam() } until $fh = IO::File->new($name, O_RDWR|O_CREAT|O_EXCL);
print $fh $content;
close $fh;

$agent->get("file://$name"); # load the temporary file/page with the DEiXToBot agent using the file:// scheme

unlink $name; # delete the temporary file, it is not needed any more

if (! $agent->success) { die "Could not fetch the temp file!"; }

$agent->load_pattern('pattern.xml'); # load the pattern built with the GUI tool

$agent->build_dom(); # build the DOM tree of the page

$agent->extract_content(); # apply the pattern 

my @records = @{$agent->records};
for my $record (@records) { # loop through the data/ records scraped
....

    Therefore, you can create temporary HTML files, in real time, containing the source code of the target pages (after the WWW::Selenium object gets these pages) and pass them to the DEiXToBot agent to do the scraping job. Another interesting scenario is to download the pages locally with Selenium and then read/ scrape them directly from the disk at a later stage. We hope the above snippet helps. Please do not hesitate to contact us for any questions or feedback!

Cloudify your browser testing (and scraping) with Sauce!

For quite some time now, along with our DEiXTo scraping software, we have been using Selenium which is perhaps the best web browser automation tool currently available. It's really great and has helped us a lot in a variety of web data extraction cases (we published another post about it recently). We tried it locally as well as on remote GNU/Linux servers and we wrote code for a couple of automated tests and scraping tasks. However, it was not that easy to set everything up and get things running; we came across various difficulties (ranging from installation to stability issues e.g. sporadic timeout errors) although we were finally able to surpass most of them.

    Wouldn't it be great though if there was a robust framework that would provide you with the necessary infrastructure and all possible browser/OS combinations and allow you to run your Selenium tests in the cloud? You would not have to worry about setting a bunch of things up, installing updates, machines management, maintenance, etc. Well, there is! And it offers a whole lot more.. Its name is Sauce Labs and it provides an amazing set of tools and features. Admittedly they have done awesome work and they bring great products to software developers. Moreover, their team seems to share some great values: pursuit of excellence, innovation and open source culture (among others).
    They offer a variety of pricing plans (a bit expensive in my opinion though) while the free account includes 100 automated code minutes for Win, Linux and Android, 40 automated code minutes for Mac and iOS and 30 Minutes of manual testing. And for those contributing to an open source project that needs testing support, Open Sauce Plan is just for you (unlimited minutes without any cost!). Please note that the Selenium project is sponsored by Sauce Labs.

    Being a Perl programmer, I could not resist signing up and writing some Perl code to run a test on the ondemand.saucelabs.com host! I was already familiar with the WWW::Selenium CPAN module, so it was quite easy and straightforward. It should be noted that they provide useful guidelines and various examples online for multiple languages e.g. Python, Java, PHP and others. Overall my test script worked pretty well but it was a bit slow (compared to running the same code locally). However, one could improve speed by deploying lots of processes in parallel (if the use case scenario is suitable) and by disabling video (the script's execution and browser activity is recorded for easier debugging). Furthermore, Sauce's big advantage is that it can go large scale, which would be especially suited for complex cases with heavy requirements.

     The bottom line is that the "Selenium - Sauce Labs" pair is remarkable and can be very useful in a wide range of cases and purposes. Sauce in particular offers developers an exciting way to cloudify and manage their automated browser testing (although we personally focus more on the scraping capabilities that these tools provide). Their combination with DEiXTo extraction patterns could definitely be very fertile and open new, interesting potentials. In conclusion, the uses and applications of web scraping are limitless and Selenium turns out to be a powerful tool in our quiver!

Selenium: a web browser automation companion for DEiXTo

 Selenium is probably the best web browser automation tool we have come across so far. Primarily it is intended for automated testing of web applications but it's certainly not limited to that; it provides a suite of free software tools to automate web browsers across many platforms. The range of its use case scenarios is really wide and its usefulness is just great.

    However, as scraping experts, we inevitably focus on using Selenium for web data extraction purposes. Its functionality-rich client API can be used to launch browser instances (e.g. Firefox processes) and simulate, through the proper commands, almost everything a user could do on a web site/ page. Thus, it allows you to deploy a fully-fledged web browser and surpass the difficulties that pop up from heavy JavaScript/ AJAX use. Moreover, via the virtual framebuffer X server (Xvfb), one could automate browsers without the need for an actual display and create scripts/ services running periodically or at will on a headless server e.g. on a remote GNU/Linux machine. Therefore, Selenium could successfully be used in combination with DEiXToBot, our beloved Mechanize scraping module. 

    For example, the Selenium-automated browser could fetch a target page after a couple of steps (like clicking a button/ hyperlink, selecting an item from a drop-down list, submitting a form, etc.) and then pass it to DEiXToBot (which lacks JavaScript support) to do the scraping job through DOM-based tree patterns previously generated with the GUI DEiXTo tool. This is particularly useful for complex scraping cases and opens new potential for DEiXTo wrappers.

    The Selenium Server component (formerly the Selenium RC Server) as well as the client drivers that allow you to write scripts that interact with the Selenium Server can be found here. We have used it quite a few times for various cases and the results were great. In conclusion, Selenium is an amazing "weapon" added to our arsenal and we strongly believe that along with DEiXTo it boosts our scraping capabilities. If you have an idea/ project that involves web browser automation or/ and web data extraction, we would be more than glad to hear from you!