Texture
Texture is a remarkable example of visual invariance. There are infinitely many different images that all look like the texture of bark (or wood, or brick, or sand). But the individual images can be very different pixel-by-pixel. What makes them all the same texture?
In a recent breakthrough, Javier Portilla and Eero Simoncelli showed that textures could be represented using a hierarchical statistical model (see this page for code and relevant publications). According to the model, image pairs will appear to be composed of the same texture if they match with respect to a small collection of image statistics. These statistics are based on a multi-scale, oriented image representation, and include both marginal statistics (e.g., energy in each band) and correlations (between neighboring filter responses in space, orientation and scale). Intuitively, these statistics correspond to the amount of image content in the image at different orientations (vertical, horizontal diagonal) and different spatial scales (coarse or fine), and how similar this content is across different orientations and scales. The image pairs to the left show original photographic textures (left column) and textures synthesized from the model (right column). The synthesized textures are remarkably similar to the originals, showing that the model captures important structure in visual texture.
For an intuition behind the model, consider Van Gogh’s paintings (like “Wheat Field,” shown at the lower left). In many of his paintings, Van Gogh used simple strokes of different orientations and widths (and color) to convey dramatic variations in texture. In “Wheat Field” he uses long narrow vertical strokes to represent the grass, slightly wider, more angled strokes to represent the wheat waving in the wind, short and thick strokes to represent the fruit, and thick heavy strokes that smoothly vary in angle across the image to represent the stormy sky. These correspond to the structures in images that we use to model texture.
We are using our model to investigate how the brain represents texture. How does our brain know that five different images of bark are all the same texture? It has been hypothesized that the brain extracts the same statistics that the model uses in order to distinguish textures, but this has never been tested. Using the model, we can synthesize a large number of different naturalistic images that are texturally equivalent (with respect to the model’s statistics). We are using neuroimaging to measure neural activity while people view naturalistic texture, and we are linking the different statistics of the model to different aspects of neural processing (to be presented at the Society for Neuroscience conference, Chicago, IL, 2009). This work is in collaboration with Luke Hallum, David Heeger, and Michael Landy.
I am also currently exploring novel extensions to the texture model, with applications to peripheral vision, crowding, and visual invariance. This work is in collaboration with Eero Simoncelli.