otiotool command line utility

[jminor]

This PR adds a new multi-purpose command line utility, otiotool. Patterned after the amazingly helpful oiiotool command line utility included with OpenImageIO, this utility provides a collection of modular, reusable, small operations which can be combined into high level operations on the command line.

As an overview of what otiotool does, here is the command line usage statement:

usage: otiotool [-h] -i INPUT [INPUT ...] [-v] [-a] [--only-tracks-with-name [ONLY_TRACKS_WITH_NAME ...]] [--only-tracks-with-index [ONLY_TRACKS_WITH_INDEX ...]]
                [--only-clips-with-name [ONLY_CLIPS_WITH_NAME ...]] [--only-clips-with-name-regex [ONLY_CLIPS_WITH_NAME_REGEX ...]] [--remove-transitions]
                [-t TRIM TRIM] [-f {video,audio,all}] [--keep-flattened-tracks] [-s] [-c] [--copy-media-to-folder COPY_MEDIA_TO_FOLDER] [--redact] [--stats]
                [--list-clips] [--list-tracks] [--list-media] [--verify-media] [--list-markers] [--inspect [INSPECT ...]] [-o OUTPUT]

otiotool = a multi-purpose command line utility for working with OpenTimelineIO.

This tool works in phases, as follows:
1. Input
    Input files provided by the "--input <filename>" argument(s) are read into
    memory. Files may be OTIO format, or any format supported by adapter
    plugins.

2. Filtering
    Options such as --video-only, --audio-only, --only-tracks-with-name,
    -only-tracks-with-index, --only-clips-with-name,
    --only-clips-with-name-regex, --remove-transitions, and --trim will remove
    content. Only the tracks, clips, etc. that pass all of the filtering options
    provided are passed to the next phase.

3. Combine
    If specified, the --stack, --concat, and --flatten operations are
    performed (in that order) to combine all of the input timeline(s) into one.

4. Relink
    If specified, the --copy-media-to-folder option, will copy or download
    all linked media, and relink the OTIO to reference the local copies.

5. Redact
    If specified, the --redact option, will remove all metadata and rename all
    objects in the OTIO with generic names (e.g. "Track 1", "Clip 17", etc.)

6. Inspect
    Options such as --stats, --list-clips, --list-tracks, --list-media,
    --verify-media, --list-markers, and --inspect will examine the OTIO and
    print information to standard output.

7. Output
    Finally, if the "--output <filename>" option is specified, the resulting
    OTIO will be written to the specified file. The extension of the output
    filename is used to determine the format of the output (e.g. OTIO or any
    format supported by the adapter plugins.)

optional arguments:
  -h, --help            show this help message and exit
  -i INPUT [INPUT ...], --input INPUT [INPUT ...]
                        Input file path(s). All formats supported by adapter plugins are supported. Use '-' to read OTIO from standard input.
  -v, --video-only      Output only video tracks
  -a, --audio-only      Output only audio tracks
  --only-tracks-with-name [ONLY_TRACKS_WITH_NAME ...]
                        Output tracks with these name(s)
  --only-tracks-with-index [ONLY_TRACKS_WITH_INDEX ...]
                        Output tracks with these indexes (1 based, in same order as --list-tracks)
  --only-clips-with-name [ONLY_CLIPS_WITH_NAME ...]
                        Output only clips with these name(s)
  --only-clips-with-name-regex [ONLY_CLIPS_WITH_NAME_REGEX ...]
                        Output only clips with names matching the given regex
  --remove-transitions  Remove all transitions
  -t TRIM TRIM, --trim TRIM TRIM
                        Trim from <start> to <end> as HH:MM:SS:FF timecode or seconds
  -f {video,audio,all}, --flatten {video,audio,all}
                        Flatten multiple tracks into one.
  --keep-flattened-tracks
                        When used with --flatten, the new flat track is added above the others instead of replacing them.
  -s, --stack           Stack multiple input files into one timeline
  -c, --concat          Concatenate multiple input files end-to-end into one timeline
  --copy-media-to-folder COPY_MEDIA_TO_FOLDER
                        Copy or download all linked media to the specified folder and relink all media references to the copies
  --redact              Remove all metadata, names, etc. leaving only the timeline structure
  --stats               List statistics about the result, including start, end, and duration
  --list-clips          List each clip's name
  --list-tracks         List each track's name
  --list-media          List each referenced media URL
  --verify-media        Verify that each referenced media URL exists (for local media only)
  --list-markers        List summary of all markers
  --inspect [INSPECT ...]
                        Inspect details of clips with names matching the given regex
  -o OUTPUT, --output OUTPUT
                        Output file. All formats supported by adapter plugins are supported. Use '-' to write OTIO to standard output.

Examples:

Combine multiple files into one, by joining them end-to-end:
otiotool -i titles.otio -i feature.otio -i credits.otio --concat -o full.otio

Layer multiple files on top of each other in a stack:
otiotool -i background.otio -i foreground.otio --stack -o composite.otio

Verify that all referenced media files are accessible:
otiotool -i playlist.otio --verify-media

Inspect specific audio clips in detail:
otiotool -i playlist.otio --only-audio --list-tracks --inspect "Interview"

In addition to being a helpful tool, the code within otiotool is made up of many small, clear, examples of performing common operations with OTIO. These small functions aim to be useful as a starting point for working with OTIO in larger applications.

For example, the question "How can I stack multiple timelines on top of each other?" can be answered with a link to one of these functions: https://github.com/jminor/OpenTimelineIO/blob/otiotool2/src/py-opentimelineio/opentimelineio/console/otiotool.py#L455

Note: This PR needs unit tests. If you would like to help with that, feel free to volunteer :)

[apetrynet]

This is awesome!

[codecov-commenter]

Codecov Report

Merging #1375 (0dbbf82) into main (6cd4161) will decrease coverage by 0.25%.
The diff coverage is 79.31%.

@@            Coverage Diff             @@
##             main    #1375      +/-   ##
==========================================
- Coverage   86.27%   86.02%   -0.26%     
==========================================
  Files         196      200       +4     
  Lines       19865    20859     +994     
  Branches     2309     2459     +150     
==========================================
+ Hits        17138    17943     +805     
- Misses       2161     2315     +154     
- Partials      566      601      +35     

Flag	Coverage Δ
py-unittests	86.02% <79.31%> (-0.26%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...-opentimelineio/opentimelineio/console/__init__.py	100.00% <ø> (ø)
...-opentimelineio/opentimelineio/console/otiotool.py	73.11% <73.11%> (ø)
tests/test_console.py	94.42% <97.16%> (+2.00%)	⬆️
...pentimelineio/opentimelineio/adapters/otio_json.py	92.30% <0.00%> (-7.70%)	⬇️
src/opentimelineio/typeRegistry.cpp	76.59% <0.00%> (-5.76%)	⬇️
tests/test_serializable_object.py	92.85% <0.00%> (-5.45%)	⬇️
src/opentimelineio/serialization.cpp	80.17% <0.00%> (-2.41%)	⬇️
src/opentimelineio/anyDictionary.h	98.03% <0.00%> (-1.97%)	⬇️
...melineio/opentimelineio-bindings/otio_bindings.cpp	98.27% <0.00%> (-1.73%)	⬇️
src/opentimelineio/serializableObject.cpp	62.06% <0.00%> (-1.47%)	⬇️
... and 17 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6cd4161...0dbbf82. Read the comment docs.

[jminor]

Anyone want to help add tests for this and/or port it to work in Python 2.7?

[apetrynet]

I can take a look at python 2 support. What's the time frame for when you'd like it done?
I have a couple of other projects going as well.

[jminor]

That would be great @apetrynet - no hurry at all.

[meshula]

Josh, in addition to the command line arguments, it could make sense to also accept a response file as input. The thought would be have arguments decomposed one per line, perhaps with a simple ${1} ${2} style syntax, in order to ease repetitive or common invocations.

trivial example:

no-video.txt

--no-video
${1}

otiotool no-video.txt cute_cats.otio

Perhaps ${n} isn't necessary, and positional logic could suffice.

otiotool no-video.txt cute_cats.otio dogs_bouncy_house.otio

I don't mean to predicate landing the tool on such a feature, it's a suggestion for a follow on feature.

[jminor]

@meshula that's a good suggestion. I sort of wonder, also, if it could be extended in the future to allow for more complex chains of operations (ala ffmpeg's pipelines) but I opted to keep it fixed-pipeline for now for simplicity. You can chain multiple executions together via the -i and -o options: otiotool -i foo ... -o - | otiotool -i - ... -o bar.
On the other hand, if you want to do something long and complicated, then the Python API is right there waiting for you ;)

[ssteinbach]

Do we list these packed-in applications anywhere? I wonder if we should list them in the README or make a "included console programs" doc page or something just so that folks know they are there. Or is this obvious? I feel like I'm kind of too deep in to have a sense for how discoverable or not these scripts are.

[meshula]

@ssteinbach we could really use a front door that looks like OCIO's. https://opencolorio.org that would be a great place to provide pointers to bundled tools. but also listing them in the root README.md makes sense.

[ssteinbach]

oh yeah wow, thats really nice. I haven't looked at that in a while, good suggestion. So maybe we defer that to a later PR?

[ssteinbach]

This is great!

Its great you added all these tests. I was recently profiling the unit tests out of curiosity and noticed that the shellout tests are some of the slowest because its slow to call out to the shell -- it makes me wonder, especially if we're adding another big hunk of tests -- if we should disable the shell out tests by default, and enable them with an env var (so that CI can still run them that way). That would help with local iteration.

Thats more of a thought for the future though.