Skip to main content

Screen scraping with Nokogiri

So a few months ago I put together a side project at http://www.myrecipesavour.com/ . Basically, the site allows you to put in the URL of a cooking recipe page and will then parse the recipe for your collection.

So it turns out, reading data from another site is very easy with Nokogiri.

The source code is available here https://github.com/abreckner/MyRecipeSavour

There is a lot I am going to cover in the next few posts based on this code base (like Devise and Heroku), but for now we are focussed on this file https://github.com/abreckner/MyRecipeSavour/blob/master/app/models/site.rb

So we are going to look at the add_recipe method.

First we need to require a few packages
require 'open-uri'
require 'rubygems'
require 'nokogiri'
Unfortunately, I haven't yet figured out a heuristic for separating a recipe web page into a recipe's components (Title, Ingredients, Instructions, Amounts, etc...) but as a workaround, I maintain a catalogue of CSS selectors which define these elements per domain. When I read the page, I use NokoGiri to parse those elements for me using the CSS selectors

i.e.
html = Nokogiri::HTML(open(url).read) # open the page
  title = html.css(site.title_selector).text.strip # read the title

I then populate a recipe object with these pieces


recipe = Recipe.new
recipe.name = title
...
recipe.save


My code around the ingredients and instructions is a little more complex as my Recipe model has many Ingredients and Instructions (eventually I am going to allow users to manipulate them individually). Each ingredient/instruction is parsed based on a line break, so I need to pull in the ingredient array from Nokogiri and then merge it into a string separated by line breaks.

ingredients = html.css(site.ingredient_selector).children.inject(''){|sum, n| sum + n.text + "\n"
...
Ingredient.multi_save(ingredients, recipe)

The reason I convert it to a string and then back into an array is so that the user can later edit the ingredients via a textarea. It's fair to say that I actually write the multi_save code from a textarea for input before I did the screen scrape and I wanted to reuse it.

The other interesting piece of this add_recipe method is that I store a new Site in case the user tries to add a recipe from an "uncatalogued" site. This automatically builds up a list of the sites people are interested in saving recipes from and allows me to catalogue it at a later date


site_domain = URI.parse(url).host
    site = Site.find_by_domain site_domain
    if site.nil?
      site = Site.new
      site.domain = site_domain
      site.url = url
      site.user = current_user
      site.save!
      false
    else
... #Nokogiri scraping code goes here
end


Comments

Popular posts from this blog

Freezing Gems

What is a gem and why would you want to freeze it?

In Ruby, there are times when you want to access pieces of functionality that other people of written (3rd party libraries) and you normally have 2 options. You can install a plug in or install a gem. Normally the method you use is determined by which ever is made available by the author.

Gems are installed on the host machine and are pretty handy when you want to run things in the command line or else across lots of projects, but their downside is that if you use a gem in a Rails project there is no automatic publishing mechanism when you deploy your site. You will need to log onto the remote host machine and install the gem manually.

Plugins are specific to Rails and are similar to gems in that they are also 3rd party libraries. However they are associated with your Rails project as opposed to your machine so they will get posted to the server on a regular deploy.

Freezing a gem is the process of transforming a gem into a plug in. Essen…

Unit/Functional Testing RubyAMF

One of my current projects is using RubyAMF to communicate with Flash (http://rubyforge.org/projects/rubyamf/). On the whole this is really nice because it allows you to transfer Ruby objects directly to ActionScript ones (as opposed to translating the object into XML, sending the XML and then recreating the object in ActionScript).
However, Rails does not provide a built in transport mechanism for AMF, so we cannot run functional testing directly on the data call (as we could for an XML or HTML transport layer). This is a show stopper for a lot of people (Rails w/o Unit testing = a big mess of trouble when something goes wrong).
We can though serve both the HTML and the AMF formats depending on the request format. This means that we can test the object instantiation logic and make sure there are no errors in the controllers (though we cannot check the actual format of the data being served). In the controller, instead of rendering AMF alone, do the following respond_to do |format|

Responsive Web Design

I wanted to go over Responsive Web Design using CSS.

In the old days of web development, we had to code to common screen sizes (i.e. 800 X 600, 1024 X 768) and we patiently waited for people to upgrade their computers to have a decent amount of screen real estate so we could design things the way we really wanted. We also took on semi stretchy web layouts etc to expand and contract appropriately.

Then about 2 or 3 years ago, Apple released this little device called an iPhone with a 320 X 480 resolution which took the world by storm and suddenly a lot of people were viewing your website on a tiny screen again...

Anyways, as it can be difficult to design a site which looks good on 320 X 480 AND 1680 X 1050, we need to come up with some kind of solution.

One way is to sniff the client and then use an appropriate stylesheet, but then you are mixing CSS with either JavaScript or server side programming and also potentially maintaining a list of appropriate clients and stylesheets. Also, you ca…