WYSIWYG - What You See Is What You Get

2022

Machine Learning

Béatrice Lartigue - Lab212

A new framework of perception: past forms activated by present WYSIWYG explores how machines are making sense of the world.

Disruptive technologies reshape our paradigms: machine learning is becoming a design material. WYSIWYG investigates new tools that try to extend the viewer's field of vision.
Alternative worlds : crossing formats Through the lens of an object detection algorithm, the model generates complete sentences from an input image: the result is computed automatically. It creates unconventional connections between the input and the output.

The description highlights a shift between the representation of the work and its subject, and what is perceived by the machine. The issue of these representations produces an alternate reality, provoking a temporal move.
Data driven: amplified by technology WYSIWYG uses object recognition and detection algorithms applied to datasets of images and sentences in English describing these images. A dataset is a selection of data you have selected: your own interpretation of the world.

To keep improving the accuracy of the model, it is training to maximize the likelihood of the sentence given the image. The produced descriptions are one of many possible image interpretations.

These interpretations are often partially correct, absurd or even poetic: some characters appear, others disappear from the compositions, alternative viewpoints are outlined… Above all, they reveal a specific period of time and are perpetually reconsidered as they are inherent to the development of both technology and the data at stake.
Show and Tell: a neural and probabilistic framework « Automatically describes the content of an image is a challenging task.’im2txt’ captures not only the objects contained in an image, but also must express how theses objects relate to each other. » WYSIWYG uses the 'im2txt’ model developed by Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan in 2015, and published on the RunwayML platform. This model is at the intersection of deep learning, computer vision and natural languages.