This repository was archived by the owner on Jan 1, 2025. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 0
/
Copy pathindex.html
504 lines (431 loc) · 23.3 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
<!DOCTYPE HTML>
<html lang="en"><head><meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Motion-Conditioned Image Animation for Video Editing</title>
<style type="text/css">
</style>
<meta name="author" content="Wilson Yan">
<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/css/bootstrap.min.css">
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.5.1/jquery.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/4.0.0/js/bootstrap.min.js"></script>
<style>
/* Apply styles to the container div */
.center-container {
text-align: center; /* Center text within the container */
}
/* Apply styles to the paragraph inside the container */
.center-container p {
display: inline-block; /* Make the paragraph a block element */
font-size: 24px;
font-family: "Gill Sans", sans-serif;
}
.method-container p {
display: inline-block;
font-size: 18px;
}
.red-text {
color: red; /* Text with class 'red-text' set to red */
}
</style>
</head>
<body>
<div class="jumbotron jumbotron-fluid text-center">
<div class="container"></div>
<h2>
Motion-Conditioned Image Animation for Video Editing
</h2>
<br>
<p class="authors">
<a href="https://wilson1yan.github.io/">Wilson Yan<sup>1,2</sup></a>,
<a href="https://scholar.google.com/citations?user=WXUm4uUAAAAJ&hl=en">Andrew Brown<sup>1</sup></a>,
<a href="https://people.eecs.berkeley.edu/~pabbeel/">Pieter Abbeel<sup>2</sup></a>,
<a href="https://rohitgirdhar.github.io/">Rohit Girdhar<sup>1</sup></a>,
<a href="https://scholar.google.com/citations?user=X0EXfT8AAAAJ&hl=en">Samaneh Azadi<sup>1</sup></a>
</p>
<p class="authors">
<sup>1</sup>GenAI, Meta </a>
<sup>2</sup>UC Berkeley</a>
</p>
<div class="btn-group" role="group" aria-label="Top menu">
<a class="btn btn-primary" href="https://arxiv.org/pdf/2311.18827.pdf" role="button">Paper</a>
<a class="btn btn-primary" href="https://github.com/facebookresearch/MoCA#videoedit-benchmarking-data" role="button">Dataset</a>
</div>
</div>
<div class="container">
<div class="section">
<h2> Abstract </h2>
<hr>
<p>
We introduce MoCA, a <b>Mo</b>tion-<b>C</b>onditioned Image <b>A</b>nimation approach for video editing.
It leverages a simple decomposition of the video editing problem into image editing followed by
motion-conditioned image animation. Furthermore, given the lack of robust evaluation datasets for video editing,
we introduce a new benchmark that measures edit capability across a wide variety of tasks, such as object
replacement, background changes, style changes, and motion edits. In addition, we present a comprehensive
human evaluation of the latest video editing methods along with \ours, on our proposed benchmark. MoCA establishes
a new state-of-the-art, outperforming notable recent approaches including Dreamix (63%), MasaCtrl (75%),
and Tune-A-Video (72%), with especially significant improvements for motion edits.
</p>
</div>
<!-- style: 27, 60, 83, 95 ,113
background: 41, 53, 68, 133
object: 39, 57, 88, 223
multi-spatial: 78, 86
motion: 162, 175, 184, 226, 253, 255, 258
multi-motion: 164, 197, 204, 218, 230, 246, 250, 260 -->
<div class="section">
<hr>
<h3>Style Edits</h3>
<div style="width: 100%; overflow: hidden;">
<div style="width: 49%; float: left;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000083.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Several goldfish swim in a tank.<br> ↓ <br> Several goldfish swim in a tank, <span class="red-text">impressionist style.</span></p>
</div>
</div>
<div style="width: 2%;"> </div>
<div style="width: 49%; float: right;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000060.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Three cows graze together in a green pasture. <br> ↓ <br> Three cows graze together in a green pasture, <span class="red-text">pointilism style.</span></p>
</div>
</div>
</div>
<hr>
<h3>Background Edits</h3>
<div style="width: 100%; overflow: hidden;">
<div style="width: 49%; float: left;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000041.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A ship sails on the sea during sunset. <br> ↓ <br> A ship sails on the <span class="red-text">lunar surface</span> during sunset.</p>
</div>
</div>
<div style="width: 2%;"> </div>
<div style="width: 49%; float: right;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000133.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>An airplane flies through the blue sky leaving a contrail behind it. <br> ↓ <br> An airplane flies <span class="red-text">over the Taj Mahal </span>leaving a contrail behind it.</p>
</div>
</div>
</div>
<hr>
<h3>Object Edits</h3>
<div style="width: 100%; overflow: hidden;">
<div style="width: 49%; float: left;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000057.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A beautiful lotus in river water on a rainy day. <br> ↓ <br> A beautiful <span class="red-text">poppy flower</span> in river water on a rainy day.</p>
</div>
</div>
<div style="width: 2%;"> </div>
<div style="width: 49%; float: right;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000088.mp4" type="video/mp4">
</video>
<div class="center-container">
<p> A static shot of red roses in sunlight, gently swaying in the breeze. <br> ↓ <br> A static shot of <span class="red-text">rainbow</span> roses in sunlight, gently swaying in the breeze.</p>
</div>
</div>
</div>
<hr>
<h3>Multi-Spatial Edits (Combo of style / background / object edits)</h3>
<div style="width: 100%; overflow: hidden;">
<div style="width: 49%; float: left;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000078.mp4" type="video/mp4">
</video>
<div class="center-container">
<p> The sun setting with clouds moving around it. <br> ↓ <br> The <span class="red-text">moon </span> setting with clouds moving around it, <span class="red-text">reflecting on a flooded road.</span></p>
</div>
</div>
<div style="width: 2%;"> </div>
<div style="width: 49%; float: right;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000086.mp4" type="video/mp4">
</video>
<div class="center-container">
<p> Rain falling on a stone pathway in super slow motion. <br> ↓ <br> <span class="red-text">Snow</span> falling on a fantasy landscape in super slow motion, <span class="red-text">with dramatic lighting.</span></p>
</div>
</div>
</div>
<hr>
<h3>Motion Edits</h3>
<div style="width: 100%; overflow: hidden;">
<div style="width: 49%; float: left;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000162.mp4" type="video/mp4">
</video>
<div class="center-container">
<p> Riding a boat over the ocean<br> ↓ <br> <span class="red-text">Huge waves crash while </span>riding a boat over the ocean</p>
</div>
</div>
<div style="width: 2%;"> </div>
<div style="width: 49%; float: right;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000184.mp4" type="video/mp4">
</video>
<div class="center-container">
<p> A duckling looking for food in grass <br> ↓ <br> <span class="red-text">Zooming out from </span>a duckling looking for food in grass</p>
</div>
</div>
</div>
<hr>
<h3>Multi-Motion Edits (Combo of motion + spatial edits)</h3>
<div style="width: 100%; overflow: hidden;">
<div style="width: 49%; float: left;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000204.mp4" type="video/mp4">
</video>
<div class="center-container">
<p> A bright pink flower<br> ↓ <br> <span class="red-text">Timelapse of an orange lily fully blooming</span></p>
</div>
</div>
<div style="width: 2%;"> </div>
<div style="width: 49%; float: right;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/000230.mp4" type="video/mp4">
</video>
<div class="center-container">
<p> Fresh apricots hanging off a tree <br> ↓ <br> <span class="red-text">Ripe apples fall off </span>a tree</p>
</div>
</div>
</div>
<br>
<h2>Comparisons Against Baselines</h2>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/FPU3MkaT_9k_00_04.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A panda sitting down and eating from a pile of bamboo → A panda sitting down and <span class='red-text'>playing in</span> a pile of <span class='red-text'>colorful ribbons</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/nA1jBUEzgSQ_00_03.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A duck swimming on water → A duck <span class='red-text'>shaking wings</span> on water</p>
</div>
</div>
<hr>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/sharks-swimming_03.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Two grey sharks swim in the blue ocean on a coral reef. → Two grey sharks swim in the blue ocean on a coral <span class='red-text'>reef, surrounded by various colorful fish.</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/audi-snow-trail_02.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>An audi q7 goes on a snow trail. → An audi q7 goes on a <span class='red-text'>desert</span> trail.</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/wNITe1mUNxw_00_01.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A white jeep driving down a gravel road → A white jeep driving down a <span class='red-text'>dirt</span> road</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/T0DV3BriqZs_00_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A turtle laying on a green floor → A turtle laying on a green floor <span class='red-text'>retracts into its shell</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/O7HE0dvNx8c_00_04.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A scene of a calm lake → A scene of a lake <span class='red-text'>with crashing waves</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/setting-sun_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>The sun setting with clouds moving around it. → The sun setting with clouds moving around <span class='red-text'>it, 2d vector animation.</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/ferris-wheel-timelapse_02.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A time lapsed video of a ferris wheel moving. → A <span class='red-text'>time-lapsed</span> video of a ferris wheel <span class='red-text'>moving with</span> a <span class='red-text'>sunset in the background.</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/nA1jBUEzgSQ_00_04.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A duck swimming on water → A duck swimming on water <span class='red-text'>with big waves</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/zYBq6V-3BpM_01_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Fresh apricots hanging off a tree → <span class='red-text'>Ripe</span> apricots <span class='red-text'>fall</span> off a tree</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/WmmGvnuAF18_00_02.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A duckling looking for food in grass → A duckling <span class='red-text'>jumping on a pile of leaves</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/cows-grazing_01.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Three cows graze together in a green pasture. → Three <span class='red-text'>sheep</span> graze together in a green pasture.</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/rreEpKo3o_Y_00_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A bright pink flower → <span class='red-text'>Timelapse of a</span> bright pink flower <span class='red-text'>fully blooming</span></p>
</div>
</div>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/nA1jBUEzgSQ_00_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A duck swimming on water → A duck <span class='red-text'>dunks its head underwater</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/ship-sailing_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A ship sails on the sea during sunset. → A ship sails on the sea during <span class='red-text'>sunset, 2d vector art.</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/6mRgGoa08AQ_00_04.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A bird sitting on a rock in a river → A <span class='red-text'>red cardinal takes flight off</span> a rock</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/pLAHIjC8MIs_00_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A boar sniffs the dirt looking for food → A boar <span class='red-text'>digs a hole in</span> the <span class='red-text'>ground</span></p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/raindrops_01.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Rain falling on a stone pathway in super slow motion. → <span class='red-text'>Snow</span> falling on a stone pathway in super slow motion.</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/UmtZLpHPRCs_00_01.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Bright red leaves on a tree during autumn → Bright red leaves <span class='red-text'>fall off</span> a tree during autumn</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/nA1jBUEzgSQ_00_01.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A duck swimming on water → A <span class='red-text'>swan</span> swimming on water</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/q4yHz3ysMUk_00_01.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Goldfish swimming around in a lake → <span class='red-text'>Blue fish</span> swimming around in a lake</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/UmtZLpHPRCs_00_00.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>Bright red leaves on a tree during autumn → Bright red leaves on a tree during <span class='red-text'>autumn, windy day, rustling</span> leaves</p>
</div>
</div>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/O7HE0dvNx8c_00_01.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A scene of a calm lake → A scene of a <span class='red-text'>left to right pan</span> of a calm lake</p>
</div>
</div>
<hr>
<hr>
<div style="width: 100%; overflow: hidden;">
<video width="100%" onloadstart="this.playbackRate = 0.75" autoplay loop playsinline muted="">
<source src="./videos/comparisons/O7HE0dvNx8c_00_03.mp4" type="video/mp4">
</video>
<div class="center-container">
<p>A scene of a calm lake → A <span class='red-text'>time-lapse</span> of a calm lake</p>
</div>
</div>
<hr>
</body>
</html>