reverse correlation stories
Summer 2019, Winter 2023
I became interested in the artistic application of reverse correlation - a technique in the field of visual perception. In this project, I use this methodology to illustrate simple stories. In this way, not only have I written the story, but I get a rough idea of how I, the author, mentally visualize the scene in my mind.
Simply put, reverse correlation involves utilizing images distorted by randomly generated noise to discern how individuals perceive abstract concepts such as 'sad,' 'happy,' and 'dangerous.' The method involves showing participants a series of paired images, a singular ‘base image’ that has been distorted twice by a random noise pattern. Then, they are asked to choose which image most illustrates the concept. By averaging across the user’s answers, researchers get an idea of how these concepts appear in physical characteristics.
In 2019, as I wrapped up my bachelor's degree in cognitive science, I served as an undergraduate research assistant at the Whitney Lab for Perception and Action, working under Allison Yaminashi-Leib. During this time, she guided us through the intricacies of conducting experiments and introduced me to this concept in visual perception. Both of the papers I pulled from were recommended by her during this time.
In order to achieve the results of this project, I duplicated the methodology used in the papers referenced below to create faces distorted by sine wave grating noise. By doing so, it’s possible to generate and track the distortion level on each face. The original papers use the coding language ‘R’ to do this, but for my application, I’ve used Python. I only ran the experiment on myself with a limited number of trials (varying from 200-300).
Papers referenced:
‘Visualising mental representations: A primer on noise-based reverse correlation in social psychology’ by L. Brinkman, A. Todorov, & R. Dotsch
‘Reverse Correlating Social Face Perception’ by Ron Dotsch and Alexander Todorov
A screenshot of the experiment running in pygame on my computer. I conducted about two hundred trials on myself for each aspect of the story.
The story: At dawn, a man thinks deeply. The sun appears. Melancholy sets in.
To illustrate this short story, I ran the experiment five 5times. Because the noise pattern used to generate the results was tracked, it was also possible to calculate the inverse of the images.
The story: A man is sad at night. He looks down and finds something bright. He smiles.
The story: A man in love finds himself utterly full so he cries softly.
I employed the same image used by Todorov and Dotsch in their original methodology. They describe its origin by writing, 'the base face was a gray scale average of all male faces in the Karolinska Face Database.' citing Lundqvist, Flykt, and O¨hman's nineteen ninety eight paper 'The Karolinska Directed Emotional Faces'
By overlaying the noise, a distorted version of the original image was created...
... and also, it was possible to create the inverse of the image as well
Hundreds of unique noise patterns were generated and overlaid on the base image. It's interesting to see already how the stimuli already have a range of expressions just with this simple modification. Using the selection of hundreds of stimuli pairs, the resulting images are formed.