Can DEiXTo power mobile apps? Yes, it can!

Web content scraped with DEiXTo can be presented in a wide variety of formats. However, the most common choice is probably XML since it facilitates heavy post-processing and further transformations so as to make the data suit your needs. A potentially interesting scenario would be to output bits of interest from a target website into an XML file and then transform it to HTML through XSLT (Extensible Stylesheet Language Transformations). This could be very practical and useful for creating in real time a customized, "shortened" version of a target web page specifically for mobile devices (e.g. Android and iOS devices).
    As you all know smartphones and tablets over the last few years have changed the computer world. So, we thought it would be challenging and hopefully useful to build a web service capable of repurposing specified pages on the fly (through the use of a DEiXTo-based agent), keeping only the important/ interesting stuff and returning it in a mobile-compatible fashion, suitable to fit small screens by harnessing XML, XSLT and CSS. We did not fully implement the service but we got a simple prototype ready to try our idea. And the results were quite encouraging!

    For the needs of our demo we used greektech-news, a technology news blog covering a plethora of interesting and fun topics around the IT industry. In the context of the demo, we supposed that we wanted to scrape the articles of the home page. So, we built a quick test scraper able to extract all the records found and generate an XML document with the data captured. With the use of an elegant XSLT and a CSS we achieved a nice, usable and easy to navigate structure, suitable for a smartphone screen (illustrated in the picture above). You can see live how the output XML file (containing 15 sample headlines) looks like on an online iPhone simulator at the following address:

http://deixto.com/repurposed-data-smartphones

    The concept of the proposed web service is the following: suppose that you are an app developer or a website owner/ administrator and that you need to display content inside your app or the mobile version of your site, either from a website of your control (meaning that you have access to its backend) or from another, "external" site (with respect to copyright and access restrictions). Often, though, it's not easy to retrieve data from the target website or simply you don't know how to do it.. Therefore, a service that could listen to requests for certain pages/ URIs and return their important data in a suitable form could potentially be very useful. For example, ideally an HTTP request like http://deixto.com/webservice.pl?uri="http://example.com/.." would result in a good-looking XML chunk (formatted with XSLT and CSS) containing the data scraped from the original page (specified with the uri parameter).
    Finally, we would like to bring forward the fact that DEiXToBot contains best of breed Perl technology and allows extensive customization. Thus, it facilitates tailor-made solutions so as to make the data captured fully fit your project's needs. And towards this direction, deploying XSLT and XML-related technologies in general can really boost the utility and value of scraping and DEiXTo in particular!

APIs vs Scraping - Cl@rity & Yperdiavgeia

Typically there are two main mechanisms to search and retrieve data from a website: either through an Application Programming Interface commonly known as an API (if available) or via screen scraping. The first one is better, faster and more reliable. However, there is not always a search API available or perhaps even if there exists one, it may not fully cover your needs. In such cases, web robots, also called agents, are usually used in order to simulate a person searching the target website/ online database through a web browser and capture bits of interest by utilizing scraping techniques.

An API that has attracted some attention over the last few months in Greece is the Opendata API that the "Cl@rity" program ("Διαύγεια" in Greek) is offering. Since the 1st of October 2010, all Greek Ministries are obliged to upload their decisions and expenditure on the Internet, through the Cl@rity program. Cl@rity is one of the major transparency initiatives of the Ministry of Interior, Decentralization and e-Government. Each document uploaded is digitally signed and given a transaction unique number automatically by the system.

The Opendata API offers a variety of search parameters such as organization, type, tag (subject), ada (the unique number assigned), signer and date. However, there are still a lot of parameters and functionality missing such as full text search as well as searching by certain criteria like beneficiary's name, VAT registration number (ΑΦΜ in Greek), document title and other metadata fields.

A remarkable alternative for searching effectively through the documents of the Greek public organizations is yperdiavgeia.gr ("ΥπερΔιαύγεια" in Greek), a web-based platform built by the expert in digital libraries and institutional repositories Vangelis Banos. Yperdiavgeia is a mirror site of Cl@rity that gets updated on a daily basis and it provides a powerful and robust OpenSearch API which is far more usable and easy to harness. Its great advantage is that it facilitates full text searching. Currently, it lacks some parameters support but it seems that they are going to be added soon since it is under active development.

Even though both APIs mentioned above are really remarkable (especially for communicating and exchanging data with third party programs) there is still some room for utilizing scraping techniques and coming up with some "magic". In a previous post we had described in detail an application we developed mostly for downloading a user-specified number of the latest PDF documents of a specific organization uploaded to Cl@rity. We believe that this little utility we created (offering both a GUI as well as a command line version) can be quite useful for many people working in the public sector and potentially save a lot of time and effort. For further information about it, please check out this post (although it is written in Greek).

So, in this short post we just wanted to point out that there are quite a lot of great APIs out there, provided mostly by large organizations (e.g., firms, governments, cultural institutions and digital libraries - collections) as well as the major players of the IT industry such as Google, Amazon, etc, offering amazing features and functionality. Nevertheless, scraping the native web interface of a target site can still be useful and sometimes come up with a solution that overpasses difficulties or/and inefficiencies of APIs and yield an innovative outcome. Moreover, there are numerous websites that do not offer an API, thus a scraper could perhaps be deployed in case data searching, gathering or exporting would be needed. Therefore, the "battle" between APIs and scraping still rages.. and we are eager to see how things will evolve. Truth be told, we love them both!

DSpace & Institutional Repositories

Institutional Repositories (IRs) have emerged over the last few years and became very popular in the academic library world. The system that has dominated the "market" is DSpace, an exciting, functionality-rich, open source software package that is installed at over 1.100 institutions around the globe. It offers an OAI-PMH web service for harvesting the metadata of the repository and getting its entire contents in Dublin Core format. However, OAI-PMH does not provide advanced search by certain criteria such as title, author, supervisor, etc. Even the REST API which is under construction does not facilitate searching with these metadata fields, at least to the best of our knowledge. Moreover, the default DSpace OpenSearch support still seems incomplete and a bit buggy. Therefore, a potential solution for searching in real time a DSpace repository could be submitting a query and scraping the results returned through its native web interface. This could probably be useful for building a federated search engine or perhaps for creating a mobile app (currently there is no mobile version for DSpace).

Having in mind the lack of an "advanced", multiple-criteria enabled, search API / mechanism by DSpace, we thought it would be interesting and perhaps useful to write a test scraper that could submit queries to a DSpace repository and fetch the search results through its website. So, we built a simple, DOM-based, extraction rule (wrapper) with the DEiXTo GUI tool and then wrote a short DEiXToBot-based script that submits a sample query to Psepheda (the IR of University of Macedonia) and scrapes the results returned. The following picture illustrates the 10 first results for a sample query by title.

To get a better idea of how a Perl, DEiXToBot-based script works, below you can find the code that scrapes the 10 first items containing "programming" in title. The pattern used captures five metadata fields: detailed URL, title, date, authors and supervisor, and prints them on the screen. Of course this script can be easily extended to submit user specified queries as well as navigate through all the result pages by following the Next page link ("επόμενη" is the inner text of this link in Greek).

use DEiXToBot;
use Encode;
my $agent = DEiXToBot->new();
$agent->get('http://dspace.lib.uom.gr/simple-search?query=((title:programming))');

$agent->load_pattern('dspace_pattern.xml');
$agent->ignore_tags( [ 'em' ] );

$agent->build_dom();

$agent->extract_content();

for my $record (@{$agent->records}) {

    print encode_utf8(join("\n",@{$record})),"\n\n";

}

DEiXToBot is written is Perl, thus it is portable and can run on multiple operating systems provided you have all the prerequisite Perl modules installed. You can download the lines of code given above along with the necessary pattern by clicking here. This short script serves as a good, simple example for utilizing the power and flexibility of DEiXToBot (a Mechanize agent object, essentially a browser emulator, which is able to execute patterns/ extraction rules previously built with the GUI tool).

Generally, IRs have huge potential and in the next few years they are expected to play an increasingly important role in storing and preserving digital content, academic or not. By the way, a great federated search engine harvesting numerous Greek digital libraries and institutional repositories is openarchives.gr which is mostly based upon OAI-PMH. It harnesses innovative technologies and has grown a lot since 2006 when it was first launched.

Last but not least, DEiXTo was used quite long ago by "Pantou", the federated search engine of the University of Macedonia, in order to scrape (in real time) multiple online resources simultaneously via their web interface/ site. It is worth noting that a predecessor of the current DEiXToBot module, back in 2007, was included in the official dbWiz distribution, a remarkable, open source, federated search software package upon which pantou was built.

myVisitPlanner & DEiXTo

We are very happy to announce that DEiXTo is going to power myVisitPlanner, an exciting project funded by the Greek Ministry of Education and Lifelong Learning, under the national action "COOPERATION 2011". myVisitPlanner is coordinated by the University of Macedonia and it is aiming at creating a personalized system for cultural itineraries planning. The consortium participants include Athena - Research and Innovation Center, GNOMON Informatics SA, the Ethnological Museum of Thrace and the West Macedonia Development Company (ANKO) SA. 

DEiXTo-based wrappers are going to be deployed in order to automatically retrieve regional cultural itineraries and points of interest from various, heterogenous target websites. For a pre-defined set of webpages, DEiXTo scrapers will run periodically and monitor them for new content and coming cultural events. However, besides the traditional DOM-based tree patterns, new “smart” and innovative techniques, such as text mining and NLP algorithms, will be used so as to also contend with sites of unknown structure and layout.

myVisitPlanner officially started a few months ago and its duration is 36 months. We are really glad that we are participating in this challenging and exciting project and we hope that DEiXTo will help myVisitPlanner towards implementing its ambitious goals.

TEL-MAP & DEiXTo

DEiXTo was recently used successfully in the context of the TEL-MAP FP7 project in order to scrape the metadata of several European Technology Enhanced Learning (TEL) projects from the CORDIS website as well as from the European Commission website.

TEL-MAP is a Coordination and Support Action funded by the European Commission under the Technology-Enhanced Learning programme. It is coordinated by the Brunel University of London and it focuses on exploratory / roadmapping activities for fundamentally new forms of learning to support the adoption of those new forms, via awareness building and knowledge management on the results of EU RTD projects in TEL and socio-economic evaluations in education.

Currently, the TEL-MAP team is building a new portal called "Learning Frontiers". This portal aspires to become a widely recognized, single-point-of-access source of information for all European TEL. Among others, it offers a Projects space that contains a lot of detailed information about numerous TEL EU-funded projects and their participants (essentially the data scraped from the target web pages). It is really worth noting that through using the geo-location info of each participant / organization, Learning Frontiers provides an interactive map of TEL in Europe!

We are really happy that we have helped the Learning Frontiers portal a little towards the implementation of its ambitious vision to increase EU-wide and global dissemination, adoption and impact of EU TEL. We wish them best success!

DEiXTo & Athos Memory

In the context of our collaboration with the awarded Veria Central Public Library, a really remarkable Greek, online digital collection, Athos Memory, has been scraped through DEiXTo in order to be added to the European Library. Athos Memory has been a giant effort of the monastic community of the Holy Mountain to preserve and disseminate the unique religious tradition of the Eastern Orthodox Church on this peninsula of Chalcidice. Numerous people have worked tirelessly for years to make this endeavour possible. We would really like to congratulate and thank them for their great efforts and for providing open access to this magnificent collection.

The metadata of 27.223 photographs, documents and digitalized manuscripts from Athos, the Sacred Mountain of Christianity, have been transformed into Europeana Semantic Elements (ESE) format so that they could then be inserted into the Hellenic Aggregator's database.

To give you a better idea of the transformation process, check out the picture below. It's a screenshot of a typical item of Athos Memory archives.

Now, this record, after extracting and repurposing its metadata based on Dublin Core, gets the following form, suitable for exporting it:

Finally, it should be noted that this was the fourth digital library that was included in the Europeana with the help of DEiXTo. And we are eager to add more online resources and help Europeana enrich further its huge cultural and scientific collection!

DEiXTo & Veria Central Public Library

One of DEiXTo's most important success stories is our collaboration with Veria Central Public Library in the context of the EuropeanaLocal project. Veria Central Public Library is a really remarkable library that embraces technology and constitutes a successful model for libraries in Greece and around the world. That's why it received a 1$ Million international award from Bill & Melinda Gates Foundation in 2010.

DEiXTo powers the Hellenic Aggregator for Europeana, created by Veria Central Public Library. DEiXToBot based Perl scripts have enabled the metadata extraction of the Music Library of Greece “Lilian Voudouri”, the Greek Educational TV and Corgialenios Digital Library in a format suitable for further processing. Once extracted, their rich content was repurposed through customized Perl code and transformed into Europeana Semantic Elements (ESE) format so that it could then be inserted into the aggregator's database.

This is the reason why DEiXTo was cited at the Symposium "Europeana in Greece" that took place on 19 October 2010 in Athens, Greece, as well as at the 19th Hellenic Academic Libraries Conference (3-5 November 2010, Athens).

Hopefully, more digital libraries/archives will use DEiXTo in the next few months in order to be able to export their metadata to the great Europeana collection (more than 15 million items from 1.500 institutions!). And we are more than glad to help Europeana enrich its content even more!