Object: flute
silent
1Dataset
Each pair is built from one real clip: the named object is segmented with a text-prompted video segmenter and inpainted out, and its sound is separated out of the soundtrack. A pair is used in both directions — original → edited for removal, edited → original for addition. 24,444 training / 1,389 validation / 1,425 test pairs; the test set covers 374 object descriptions.
Object:
Object: brass instrument
silent
Object: train engine
silent
Object: birds chirping
silent
Object: animal
silent
Object: white-tailed eagle
silent
Object: individual
silent
Object: helicopter
silent
2Best results
Our model edits the picture and the soundtrack jointly. These showcase clips reappear in the baseline comparisons of sections 3 and 4 under the same tags.
2.1Text-guided removal
Text prompt only. The object is clearly visible and audible in the input and is gone from both the picture and the soundtrack in our output.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
2.2Text + first-frame addition
Text prompt plus the first frame of the target. The input contains neither the object nor its sound; our output adds both, in sync.
Add flute
Add train engine
Add running water
Add white-tailed eagle
Add helicopter
3Comparison with unimodal editing baselines
Unimodal editors change one modality and leave the other untouched, and none of them supports addition, so this comparison is on removal (clips R1–R5). Each baseline is judged in its own modality; one cascade of a video editor and an audio editor follows.
3.1Video-only baselines
ROSE, VideoPainter and LGVI edit the picture only. All clips in this block are silent, so only the visual edit is compared.
Remove brass instrument
silent
silent
silent
silent
silent
silent
Remove birds chirping
silent
silent
silent
silent
silent
silent
Remove animal
silent
silent
silent
silent
silent
silent
Remove individual
silent
silent
silent
silent
silent
silent
Remove dogs
silent
silent
silent
silent
silent
silent
3.2Audio-only baselines
ZEUS and AUDIT edit the soundtrack only. Audio players only, so only the audio edit is compared.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
3.3Cascade: ROSE + ZEUS
The video edited by ROSE combined with the audio edited by ZEUS — two independent models, neither aware of the other's edit.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
4Comparison with joint audio-visual editing baselines
InstructAV2AV and ours generate both streams. AVI-Edit generates the video and is given the target audio track; CoherentAVEdit generates the audio and is given the edited video.
4.1Removal
Clips R1–R5. All methods receive the input clip and the text prompt.
Remove brass instrument
Remove birds chirping
Remove animal
Remove individual
Remove dogs
4.2Addition
Clips A1–A5. Every method that generates video is additionally given the first frame of the target.
Add flute
Add train engine
Add running water
Add white-tailed eagle
Add helicopter
5More examples
Further results of our model, selected automatically: an audio-visual judge (a multimodal LLM) must confirm the edit in the video, in the audio, and their synchronisation.
5.1Removal
Text prompt only (R6–R29).
Remove birds
Remove engine
Remove a person
Remove high-revving engine
Remove birds
Remove birds
Remove birds
Remove trumpets
Remove running water
Remove birds
Remove knife
Remove clarinet
Remove chirping birds
Remove birds
Remove gunshots
Remove birds
Remove animal
Remove dishes
Remove owl
Remove worker
Remove idling engine
Remove livestock
Remove birds
Remove man
5.2Addition
Text prompt plus first frame (A6–A30).
Add didgeridoo
Add didgeridoo
Add clarinet
Add birds
Add bass guitar
Add pianist
Add male performer
Add accordion
Add person
Add drum kit
Add saxophonist
Add bald man
Add individual
Add birds
Add birds
Add birds
Add individual
Add cow
Add birds
Add crowd
Add brass instrument
Add birds
Add individual
Add individual
Add helicopter
6Failure cases
Typical failures of the dataset and of our model, grouped by type. They were found with the audio-visual judge, which is asked whether the object is visible, whether it is audible, and whether sound and picture are in sync; its explanation is quoted under each example.
6.1Dataset failures
Errors in how the pairs were built; they bound what any model trained or scored on this data can reach.
6.1.1Sound separation leaves the object audible
The separated soundtrack of the edited clip still contains the object's sound.
Object: idling engine
JudgeThe idling engine sound is audibly present and remains consistent without interruption across the clip.
Object: drumming
JudgeThe sound of drumming is clearly audible and remains consistent throughout the clip.
6.1.2Inpainting leaves the object (or a remnant) visible
The inpainted picture of the edited clip still shows the object or a recognisable remnant.
Object: truck engine
JudgeThe engine is clearly visible and remains consistent in its appearance and operation across the entire clip.
Object: toilet
JudgeThe toilet is consistently visible as the camera pulls back and focuses.
6.1.3The named object is heard but never seen in the original clip
The prompt names a sound source that is off-screen, or does not match what the picture shows.
Object: motorcycle engine
JudgeThe video shows a car engine, not a motorcycle engine; the target is entirely missing.
Object: engine
JudgeThe target engine is not visually present at any point in the clip.
6.2Removal failures
Our model, text prompt only, on clips whose input clearly shows and sounds the object.
6.2.1The object is left in place
The output is close to the input: the object is still fully visible.
Remove truck engine
JudgeThe engine remains visible throughout the entire clip.
Remove toilet
JudgeThe target object (toilet) remains visible as a close-up surface for the entire duration.
6.2.2A ghost or blurred remnant stays behind
The object is only partly erased and leaves a blurred or ghost-like shape.
Remove train
JudgeA massive, moving artifact retaining the shape and color of the train is visible for the entire clip.
Remove train
JudgeA large, moving ghost image of the train remains visible on the tracks for the entire clip.
6.2.3The object leaves the picture but its sound stays
The visual removal succeeds while the soundtrack keeps the object's sound, although the reference audio is clean.
Remove electronic device
JudgeThe buzzing sound of the electronic device remains clearly audible for the entire duration.
Remove motorcycle engine
JudgeThe sound of a motorcycle engine is clearly audible continuously from the beginning to the end of the clip.
6.3Addition failures
Our model, text prompt plus first frame, on clips whose input has neither the object nor its sound.
6.3.1The object drifts away from the first frame (identity or shape changes)
The object starts from the given first frame but its shape or identity changes over time.
Add cargo ship
JudgeThe ship structure is highly distorted and morphs continuously, lacking consistency.
Add animal
JudgeThe animal severely distorts and loses its initial identity, becoming an unrecognizable shape starting around 0.5s.
6.3.2The object fades out after the first frame
The object is present at the first frame and then disappears.
Add male vocalist
JudgeNo singing action is visible, and the person's head disappears entirely at 00:01.000.
Add canine
JudgeThe dog arbitrarily disappears and blends into the floor texture at 00:01.000 and 00:04.000.
6.3.3Object and sound are both added, but out of sync
Both streams contain the object, but sound events do not line up with the visible action.
Add barking dog
JudgeThe audible barks do not match the visible actions of the dog, which appears to be sniffing or eating rather than barking, especially noticeable around 00:03 and 00:04.
Add rooster
JudgeThe rooster crowing sound occurs between 00:01.8 and 00:03.0, but the visible rooster's beak remains closed and it does not perform the physical action of crowing.
No example matches this filter.