LBP_multiresolution gray scale and rotation invariant

What are Local Binary Patterns?

Local Binary Patterns, or LBPs for short, are a texture descriptor made popular by the work of Ojala et al. in their 2002 paper, Multiresolution Grayscale and Rotation Invariant Texture Classification with Local Binary Patterns (although the concept of LBPs were introduced as early as 1993).

Unlike Haralick texture features that compute a global representation of texture based on the Gray Level Co-occurrence Matrix, LBPs instead compute a local representation of texture. This local representation is constructed by comparing each pixel with its surrounding neighborhood of pixels.

The first step in constructing the LBP texture descriptor is to convert the image to grayscale. For each pixel in the grayscale image, we select a neighborhood of size r surrounding the center pixel. A LBP value is then calculated for this center pixel and stored in the output 2D array with the same width and height as the input image.

For example, let’s take a look at the original LBP descriptor which operates on a fixed 3 x 3neighborhood of pixels just like this:

Figure 1: The first step in constructing a LBP is to take the 8 pixel neighborhood surrounding a center pixel and threshold it to construct a set of 8 binary digits.

In the above figure we take the center pixel (highlighted in red) and threshold it against its neighborhood of 8 pixels. If the intensity of the center pixel is greater-than-or-equal to its neighbor, then we set the value to 1; otherwise, we set it to 0. With 8 surrounding pixels, we have a total of 2 ^ 8 = 256 possible combinations of LBP codes.

From there, we need to calculate the LBP value for the center pixel. We can start from any neighboring pixel and work our way clockwise or counter-clockwise, but our ordering must be kept consistent for all pixels in our image and all images in our dataset. Given a 3 x 3neighborhood, we thus have 8 neighbors that we must perform a binary test on. The results of this binary test are stored in an 8-bit array, which we then convert to decimal, like this:

Figure 2: Taking the 8-bit binary neighborhood of the center pixel and converting it into a decimal representation. (Thanks to Bikramjot of Hanzra Tech for the inspiration on this visualization!)

In this example we start at the top-right point and work our way clockwise accumulating the binary string as we go along. We can then convert this binary string to decimal, yielding a value of 23.

This value is stored in the output LBP 2D array, which we can then visualize below:

Figure 3: The calculated LBP value is then stored in an output array with the same width and height as the original image.

This process of thresholding, accumulating binary strings, and storing the output decimal value in the LBP array is then repeated for each pixel in the input image.

Here is an example of computing and visualizing a full LBP 2D array:

Figure 4: An example of computing the LBP representation (right) from the original input image (left).

The last step is to compute a histogram over the output LBP array. Since a 3 x 3 neighborhood has 2 ^ 8 = 256 possible patterns, our LBP 2D array thus has a minimum value of 0 and a maximum value of 255, allowing us to construct a 256-bin histogram of LBP codes as our final feature vector:

Figure 5: Finally, we can compute a histogram that tabulates the number of times each LBP pattern occurs. We can treat this histogram as our feature vector.

A primary benefit of this original LBP implementation is that we can capture extremely fine-grained details in the image. However, being able to capture details at such a small scale is also the biggest drawback to the algorithm — we cannot capture details at varying scales, only the fixed 3 x 3 scale!

To handle this, an extension to the original LBP implementation was proposed by Ojala et al. to handle variable neighborhood sizes. To account for variable neighborhood sizes, two parameters were introduced:

1. The number of points p in a circularly symmetric neighborhood to consider (thus removing relying on a square neighborhood).
2. The radius of the circle r, which allows us to account for different scales.

Below follows a visualization of these parameters:

Figure 6: Three neighborhood examples with varying p and r used to construct Local Binary Patterns.

Lastly, it’s important that we consider the concept of LBP uniformity. A LBP is considered to be uniform if it has at most two 0-1 or 1-0 transitions. For example, the pattern 00001000  (2 transitions) and 10000000  (1 transition) are both considered to be uniform patterns since they contain at most two 0-1 and 1-0 transitions. The pattern 01010010 ) on the other hand is not considered a uniform pattern since it has six 0-1 or 1-0 transitions.

The number of uniform prototypes in a Local Binary Pattern is completely dependent on the number of points p. As the value of p increases, so will the dimensionality of your resulting histogram. Please refer to the original Ojala et al. paper for the full explanation on deriving the number of patterns and uniform patterns based on this value. However, for the time being simply keep in mind that given the number of points p in the LBP there are p + 1 uniform patterns. The final dimensionality of the histogram is thus p + 2, where the added entry tabulates all patterns that are not uniform.

So why are uniform LBP patterns so interesting? Simply put: they add an extra level of rotation and grayscale invariance, hence they are commonly used when extracting LBP feature vectors from images.

Local Binary Patterns with Python and OpenCV

Local Binary Pattern implementations can be found in both the scikit-image and mahotaspackages. OpenCV also implements LBPs, but strictly in the context of face recognition — the underlying LBP extractor is not exposed for raw LBP histogram computation.

In general, I recommend using the scikit-image implementation of LBPs as they offer more control of the types of LBP histograms you want to generate. Furthermore, the scikit-image implementation also includes variants of LBPs that improve rotation and grayscale invariance.

Before we get started extracting Local Binary Patterns from images and using them for classification, we first need to create a dataset of textures. To form this dataset, earlier today I took a walk through my apartment and collected 20 photos of various textures and patterns, including an area rug:

Figure 7: Example images of the area rug texture and pattern.

Notice how the area rug images have a geometric design to it.

I also gathered a few examples of carpet:

Figure 8: Four examples of the carpet texture.

Notice how the carpet has a distinct pattern with a coarse texture.

I then snapped a few photos of the keyboard sitting on my desk:

Figure 9: Example images of my keyboard.

Notice how the keyboard has little texture — but it does demonstrate a repeatable pattern of white keys and silver metal spacing in between them.

Finally, I gathered a few final examples of wrapping paper (since it is my birthday after all):

Figure 10: Our final texture we are going to classify — wrapping paper.

The wrapping paper has a very smooth texture to it, but also demonstrates a unique pattern.

Given this dataset of area rug, carpet, keyboard, and wrapping paper, our goal is to extract Local Binary Patterns from these images and apply machine learning to automatically recognize and categorize these texture images.

Let’s go ahead and get this demonstration started by defining the directory structure for our project:

Local Binary Patterns with Python & OpenCV

Shell

 

We’ll be creating a pyimagesearch  module to keep our code organized. And within thepyimagesearch  module we’ll create localbinarypatterns.py , which as the name suggests, is where our Local Binary Patterns implementation will be stored.

Speaking of Local Binary Patterns, let’s go ahead and create the descriptor class now:

Local Binary Patterns with Python & OpenCV

Python

 

We start of by importing the feature  sub-module of scikit-image which contains the implementation of the Local Binary Patterns descriptor.

Line 5 defines our constructor for our LocalBinaryPatterns  class. As mentioned in the section above, we know that LBPs require two parameters: the radius of the pattern surrounding the central pixel, along with the number of points along the outer radius. We’ll store both of these values on Lines 8 and 9.

From there, we define our describe  method on Line 11, which accepts a single required argument — the image we want to extract LBPs from.

The actual LBP computation is handled on Lines 15 and 16 using our supplied radius and number of points. The uniform  method indicates that we are computing the rotation and grayscale invariant form of LBPs.

However, the lbp  variable returned by the local_binary_patterns  function is not directly usable as a feature vector. Instead, lbp  is a 2D array with the same width and height as our input image — each of the values inside lbp  ranges from [0, numPoints + 2], a value for each of the possible numPoints + 1 possible rotation invariant prototypes (see the discussion of uniform patterns at the top of this post for more information) along with an extra dimension for all patterns that are not uniform, yielding a total of numPoints + 2 unique possible values.

Thus, to construct the actual feature vector, we need to make a call to np.histogram  which counts the number of times each of the LBP prototypes appears. The returned histogram is numPoints + 2-dimensional, an integer count for each of the prototypes. We then take this histogram and normalize it such that it sums to 1, and then return it to the calling function.

Now that our LocalBinaryPatterns  descriptor is defined, let’s see how we can use it to recognize textures and patterns. Create a new file named recognize.py , and let’s get coding:

Local Binary Patterns with Python & OpenCV

Python

 

We start off on Lines 2-6 by importing our necessary command line arguments. Notice how we are importing the LocalBinaryPatterns  descriptor from the pyimagesearch  sub-module that we defined above.

From there, Lines 9-14 handle parsing our command line arguments. We’ll only need two switches here: the path to the --training  data and the path to the --testing  data.

In this example, we have partitioned our textures into two sets: a training set of 4 images per texture (4 textures x 4 images per texture = 16 total images), and a testing set of one image per texture (4 textures x 1 image per texture = 4 images). The training set of 16 images will be used to “teach” our classifier — and then we’ll evaluate performance on our testing set of 4 images.

On Line 18 we initialize our LocalBinaryPattern  descriptor using a numPoints=24and radius=8.

In order to store the LBP feature vectors and the label names associated with each of the texture classes, we’ll initialize two lists:  data  to store the feature vectors and labels  to store the names of each texture (Lines 19 and 20).

Now it’s time to extract LBP features from our set of training images:

Local Binary Patterns with Python & OpenCV

Python

 

We start looping over our training images on Line 23. For each of these images, we load them from disk, convert them to grayscale, and extract Local Binary Pattern features. The label (i.e., texture name) is then extracted from the image path and both our labels  anddata  lists are updated, respectively.

Once we have our features and labels extracted, we can train our Linear Support Vector Machine on Lines 35 and 36 to learn the difference between the various texture classes.

Once our Linear SVM is trained, we can use it to classify subsequent texture images:

Local Binary Patterns with Python & OpenCV

Python

 

Just as we looped over the training images on Line 22 to gather data to train our classifier, we now loop over the testing images on Line 39 to test the performance and accuracy of our classifier.

Again, all we need to do is load our image from disk, convert it to grayscale, extract Local Binary Patterns from the grayscale image, and then pass the features onto our Linear SVM for classification (Lines 42-45).

I’d like to draw your attention to hist.reshape(1, -1) on Line 45. This reshapes our histogram from a 1D array to a 2D array allowing for the potential of multiple feature vectors to run predictions on.

Lines 48-51 show the output classification to our screen.

Results

Let’s go ahead and give our texture classification system a try by executing the following command:

Local Binary Patterns with Python & OpenCV

Shell

 

And here’s the first output image from our classification:

Figure 11: Our Linear SVM + Local Binary Pattern combination is able to correctly classify the area rug pattern.

Sure enough, the image is correctly classified as “area rug”.

Let’s try another one:

Figure 12: We are also able to recognize the carpet pattern.

Once again, our classifier correctly identifies the texture/pattern of the image.

Here’s an example of the keyboard pattern being correctly labeled:

Figure 13: Classifying the keyboard pattern is also easy for our method.

Finally, we are able to recognize the texture and pattern of the wrapping paper as well:

Figure 14: Using Local Binary Patterns to classify the texture of an image.

While this example was quite small and simple, it was still able to demonstrate that by using Local Binary Pattern features and a bit of machine learning, we are able to correctly classify the texture and pattern of an image.