Concept
An interactive, immersive installation in which poetry, sound, visual marks, coloured light and human movement become different expressions of the same underlying information. Rather than using generative AI to freely reinterpret a poem, the work employs a shared mathematical representation and deterministic transformation rules to translate between modalities. Each translation preserves certain properties while altering or losing others.
The central artistic question: what remains of a poem when its language becomes sound, its rhythm becomes movement, its movement becomes light, and the resulting sensory experience becomes language again?
Artistic proposition and system logic
The installation treats a poem not only as written language, but as a structured field of relationships: rhythm, duration, repetition, density, intensity, pauses, contrasts, spatial patterns and changes over time. These properties are encoded into a common, parameterised representation. Explicit mappings translate this representation into different sensory forms: words and phonetic structures become sound; temporal and semantic structures become lines, strokes and visual compositions; selected parameters drive colour, brightness and spatial light behaviour; and rhythmic, directional and dynamic features become movement.
Conversely, captured movement, sound, light or visual marks can be analysed and mapped back into a constrained poetic structure. The system does not claim that these modalities are equivalent: it makes their correspondences, mismatches and losses perceptible. Every transformation is reproducible under the same rules, so audiences encounter the consequences of the mapping rather than an opaque algorithm’s interpretation.
The three-act audience experience
- Receive. A poem becomes an environment. Its encoded structure generates a coordinated audiovisual and spatial response: sound evolves, projected lines emerge and transform, and programmable coloured light shifts in response to the poem’s parameters. A projected human silhouette or movement sequence may embody its temporal and spatial qualities.
- Compose. The audience becomes the source. A participant moves, makes sounds or creates marks within the installation. Camera-based motion tracking extracts position, velocity, acceleration, direction, curvature, joint angles, repetition, stillness and duration, and translates them through predefined rules into sound, visual strokes, colour changes and a constrained poem. No musical or artistic skill is required.
- Transform. Meaning passes through a loop. The reconstructed poem is fed back into the same translation system and interpreted again. Across iterations, the audience observes what persists, what mutates and what disappears. Translation loss is not a technical failure: it is the work’s principal artistic material.
Technical architecture
A modular, locally operated prototype: Python with NumPy/SciPy for feature extraction and mappings; MediaPipe or OpenCV for camera-based movement analysis; TouchDesigner for real-time visual composition and projection; Ableton Live for sound. A shared intermediate representation of normalised, time-indexed features lets each modality act as both input and output. A programmable LED-strip installation (potentially ESP32-based WLED) provides responsive physical colour, while projection carries detailed lines and imagery. The core transformations run locally, with no dependence on cloud generative AI.
Development begins with one poem, one tracked movement input, one visual output and one controllable light zone, before expanding to bidirectional translation and iterative loops.
Space
A small, enclosed, controllable room or black box, up to about 4 × 4 metres for the first prototype. Portable, modular and relatively low-cost: a computer, camera, projector, speaker, programmable lighting and minimal mounting. Physical light establishes spatial atmosphere; projected light renders fine-grained visual transformation.
Why it matters
Situated at the intersection of experimental poetry, media art, computational aesthetics, sound, embodied interaction and perception, the work asks whether meaning can travel between unlike sensory systems, and how the limits of translation reveal the distinctive qualities of each. Its contribution is a transparent, repeatable system in which the audience can experience and examine the transformation process itself.
One structure, many expressions; every translation reveals both a correspondence and a loss.