Image Search

The Task: What’s the name of that flower?

We want to build an app that recognizes flowers using the cell phone camera. The workflows is that:

  • We take a picture of a flower using the cell phone camera
  • We match the picture against a large database of images of flowers and their names
  • We output the name of the best matching flowers

Building a database of flower images

We use the bing search engine api on Azure to gather images. First, we create a list of flower names. Each flower can have multiple names, for example one in Latin and one in English. We can also construct multiple queries for disambiguation. The following are reasonable queries:

  • flower
  • plant

We use bing image search to retrieve the urls of images of flowers on the web. At a later stage we use a script to download those images.

Issues: Sometimes the bing search for flower name retrieves cartoon characters (because those are named after a flower), drawings of flowers rather than pictures. Some files are too large and it makes little sense to download and process them.

Scripts:

  • bing image search
  • download images

Image preprocessing

Image search is usually based on explicit features extracted from the images. Those features are in the following categories:

  • Color based features, usually a normalized color histogram

  • Texture based features

  • Shape-based

Practical Stuff: OpenCV with Python

It seems that the easiest way to build an image search is to use OpenCV from Python (import cv2 library)

One needs to install this library from source. The latest version is 3.0.

Installing OpenCV

This is a bit tricky.

First one needs to decide if the stable or the development version will be used.

Consult this page for the steps: link (or better install-opencv-3-0-and-python-2-7-on-ubuntu).

  • Install the prerequisites:
[compiler] sudo apt-get install build-essential
[required] sudo apt-get install cmake git libgtk2.0-dev pkg-config libavcodec-dev libavformat-dev libswscale-dev
[optional] sudo apt-get install python-dev python-numpy libtbb2 libtbb-dev libjpeg-dev libpng-dev libtiff-dev libjasper-dev libdc1394-22-dev

  • Decide if you want only python2 support (old Python) or Python3 support. If python3 is required install python3 and python3-dev (python3-numpy?). I built without Python3

  • Decide if you want opencv_contrib modules. If so clone the repository. According to this question as of 3.0 SURF and SIFT are in contrib (see also Where did sift and surf go in opencv 3).

  • Prepare the arguments to cmake

cmake CMAKE_BUILD_TYPE=RELEASE \
-D BUILD_PYTHON_SUPPORT=ON \
-D BUILD_EXAMPLES=ON \
-D INSTALL_C_EXAMPLES=ON \
-D INSTALL_PYTHON_EXAMPLES=ON \
-D BUILD_NEW_PYTHON_SUPPORT=OFF \
-D CMAKE_INSTALL_PREFIX=/opencvbin \
-D OPENCV_EXTRA_MODULES_PATH=../opencv_contrib/modules \
../opencv/
</pre>

Important: also consult this page install-opencv-3-0-and-python-2-7-on-ubuntu. It suggests more libraries to install (depending on the purpose of usage), 
a fortan library for optimizing the routines and a virtual python environment.