3.1. Part 1: Working with images, regions and fixations

Welcome to the first part of our GridFix tutorial. In this part, we will get to know the different objects provided by GridFix to facilitate interactive use of the toolbox in fixation analysis by loading some images, creating a grid parcellation and viewing image fixations.

# Import GridFix toolbox and related modules

%matplotlib inline
import numpy as np
import matplotlib as mp
import matplotlib.pyplot as plt

from gridfix import *

3.1.1. Loading images and feature maps: the ImageSet object

An ImageSet holds a collection of images or feature maps, each addressed by an individual imageid which is used throughout the toolbox to identify images. As a first step, we are going to load some example images using a CSV file containing image filenames and IDs, then print some information about the resulting ImageSet. It is also possible to load images from a folder and specify imageids explicitly - we are going to do that at a later stage.

images = ImageSet('images/tutorial_images.csv', label='tutorial')
print(images)

# Number of images available using len() and image size as (width, height)
print(len(images))
print(images.size)

# The imageids attribute contains a list of all defined imageids in the set
print(images.imageids)

<gridfix.ImageSet "tutorial", 15 images, size=(800, 600)>
15
(800, 600)
['6', '9', '37', '52', '58', '67', '85', '97', '106', '107', '111', '112', '129', '149', '150']

Images or feature maps are stored as numpy arrays. To see a table (actually, a pandas DataFrame) of all images in the set and their corresponding IDs, you can query the info attribute.

images.info

	imageid	filename	width	height	channels	type	mat_var
0	6	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
1	9	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
2	37	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
3	52	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
4	58	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
5	67	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
6	85	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
7	97	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
8	106	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
9	107	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
10	111	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
11	112	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
12	129	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
13	149	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG
14	150	/home/mo/Seafile/PythonProjects/gridfix-dev/tu...	800	600	3	PNG

Image data (i.e., the raw floating-point matrices) can be retrieved by passing the corresponding imageid to the ImageSet.images method or simply indexing using [‘imageid’]. Most gridfix objects also support a plot() method to render images and feature maps/values as a matplotlib figure.

Note: in an interactive IPython/Jupyter session, plotted figures will be displayed immediately and no figure object will be returned. Use plt.savefig() to save the most recent figure. When using the gridfix module in a script, plot() will return a figure object for further processing.

# Raw image data array (uncomment to display)
#print(images['149'])

# Show an image interactively
example_img = '149'
images.plot(example_img)

# To save the previous figure output to file, use:
#plt.savefig('image10.pdf')

3.1.2. Defining image parcellations (e.g., regular grid) using a RegionSet

GridFix segments images using a RegionSet, which simply holds a collection of binary masks that have the same dimensions as the image(s) they will be applied to. Each mask therefore selects a region from the image. Regions do not have to be contiguous or non-overlapping, and the sum of all regions (available in the RegionSet.mask attribute) is not required to cover the whole image area.

In this example, we will use the same regular grid parcellation used in [1]. To define a rectangular grid parcellation such as the 8-by-6 grid used in the above article, we create a GridRegionSet - a special type of RegionSet object which will automatically create the individual masks for each grid cell. Creating a GridRegionSet takes at least two arguments: the size of the corresponding image(s), which we can get from our previously created ImageSet, and the grid dimensions as a (width, height) tuple.

grid = GridRegionSet(size=images.size, gridsize=(8,6), label='testgrid')
print(grid)

<gridfix.GridRegionSet (testgrid), size=(800, 600), 8x6 grid, 48 cells, memory=22500.0 kB>

To quickly visualize a RegionSet (especially when dealing with more complicated parcellations than a simple rectangular grid), each RegionSet object supports the plot() method that will create a visual representation in the size of the input images.

grid.plot()

Even when no fixation data is loaded, a RegionSet can be used to segment an image into subregions, for example to perform image statistics on each region or visualize which parts of an image are selected by a collection of regions of interest. The apply() methods returns a list of image arrays, each containing the image patch(es) corresponding to each mask. If you add crop=True to the apply() call, the resulting output is cropped to the minimal bounding box of each image patch.

Note: For a GridRegionSet, mask numbering starts at the upper left corner and progresses row-wise.

# Applying the grid to our previously used example image
ex_img = images[example_img] # this yields the image data array
slices = grid.apply(ex_img, crop=True) # slices now contains the selected image data
print(len(slices))

48

# Display some of the image patches using matplotlib
f = plt.figure()
f1 = f.add_subplot(1,2,1)
plt.imshow(slices[17], interpolation='none')
f2 = f.add_subplot(1,2,2)
plt.imshow(slices[23], interpolation='none')

<matplotlib.image.AxesImage at 0x7f75c86f8dd8>

Note: any RegionSet object allows to export the generated image patches as files, either using .export_patches() for a single image array or using .export_patches_from_set() to export all image regions from a complete image set.

# This will export all 48 patches selected from the example image when uncommented
#grid.export_patches(images)

3.1.3. Loading and viewing image fixations

GridFix stores fixation data in a Fixations() object and expects data in table form, e.g. imported from a CSV text file. Fixation data should be in long format (one fixation per row), and the file must at least include the following columns:

• imageid: unique ID for each image (string)
• fixid: unique ID for each fixation per trial, e.g. sequential fixation number
• x, y: horizontal and vertical fixation position in pixels

Column names do not have to be as defined above - if your columns are named differently, the corresponding column names can be submitted when creating the Fixations object (see example below).

For the tutorial, we will a small fixation data set containing fixations by 8 subjects on our example images:

fix = Fixations('tutorial_fixations.csv', imageid='image_id', fixid='CURRENT_FIX_INDEX',
                x='CURRENT_FIX_X', y='CURRENT_FIX_Y', imageset=images)
print(fix)

<gridfix.Fixations data set, 2556 samples, 15 images>
Images:
        <gridfix.ImageSet "tutorial", 15 images, size=(800, 600), normalized>

Note: if the parameter imageset= is added on creation of a Fixations dataset, the toolbox will check that all imageids found in the fixation data have a corresponding image in the ImageSet.

Now that we have successfully read the data, let’s look at the fixations our subjects made on the example image selected above (using white circles as the fixation markers) - note that this call takes the imageid as argument, not an image array, so that appropriate fixations can be selected from the dataset:

fix.plot(example_img, plotformat='wo')

This displays all fixations on this image, regardless of which subject was viewing the image at the time. Let’s find out which subject IDs are in out dataset and plot only fixations by one of them. The original dataset is available under the .data attribute as a DataFrame and supports all DataFrame operations.

Moreover, Fixations objects support a select() operation which takes a dictionary as argument, with each dictionary key specifying one column from the original data file and each value specifying a list of values to select from this column.

Note: ``select_fix()`` does not automatically add the corresponding image id variable to its filter variables, so be sure to include it - otherwise fixations for multiple images are returned! If you are only interested in the plot, you can also add select={'some_column': ['some', 'values']} to the arguments of plot() without explicitly selecting fixations into a new dataset - this will return only fixations matching the selected image.

# Find all unique subject IDs in the dataset
print(fix.data['subject_number'].unique())

# ...and create a subset of the fixation data for subject #203
subj_fix = fix.select_fix(select={'subject_number': 203, 'image_id': example_img})

# Now plot only the fixations of subject #203
subj_fix.plot(example_img, plotformat='wo')

[201 203 204 205 206 207 208 209]

Note: if necessary, a Fixations object can also return an array of fixated/non-fixated pixels, either binary using e.g. subj_fix.location_map(), or containing the count of fixations for each pixel using .count_map().

3.1.4. Using a RegionSet to evaluate fixated locations

After fixation data has been loaded and selected in this way, the resulting fixations can now be evaluated on a previously created RegionSet parcellation, returning the fixated and non-fixated regions (for the example grid: fixated and non-fixated grid cells):

# Determine fixated and non-fixated grid cells
fix_on_image = grid.fixated(subj_fix)
print(fix_on_image)

# Number of returned values - this should match number of grid cells
print(len(fix_on_image))

[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  1.  1.  0.  0.  0.  0.  0.
  1.  1.  1.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  1.
  1.  0.  0.  0.  0.  0.  1.  0.  0.  1.  0.  0.]
48

The result of this operation yields a binary value for each region in our previously created GridRegionSet: the respective cell has been fixated (1) or not (0). To visualize this result on the grid, we can simply plot the GridRegionSet as we did above, but this time pass the resulting fixations to the plotting function using the values= argument:

grid.plot(values=fix_on_image)

When GridFix is used to create a predictor file for GLMM analysis, it is this vector of fixated and non-fixated regions that will be calculated alongside the other predictors and used as the dependent variable in the GLMM.

In case that not the fixated/non-fixated status of a cell but rather the absolute number of fixations is of interest, this information can simply be calculated and plotted by adding count=True when calculating the fixated regions:

# Calculate number of fixations per cell...
count_fix = grid.fixated(subj_fix, count=True)

# ...and visualize, specifying a colormap for easier visualization
grid.plot(values=count_fix, cmap='hot')
print(count_fix)

[ 0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  0.  4.  2.  0.  0.  0.  0.  0.
  2.  1.  3.  0.  0.  0.  0.  0.  0.  1.  0.  0.  0.  0.  0.  0.  0.  3.
  2.  0.  0.  0.  0.  0.  1.  0.  0.  1.  0.  0.]

3.1.5. Concluding Remarks

This concludes the first part of our GridFix tutorial. Since Jupyter notebooks are interactive when opened on your local machine, feel free to play around with the code in the above tutorial, for example by selecting a different image from the ImageSet or using a grid of different dimensions, then selecting “Cell > Run All” to recreate all results using new data.

3.1.6. References

[1] Nuthmann, A., & Einhäuser, W. (2015). A new approach to modeling the influence of image features on fixation selection in scenes. Annals of the New York Academy of Sciences, 1339(1), 82-96. http://dx.doi.org/10.1111/nyas.12705