FMV with KLV Support

[BryonLewis]

Support for full motion video with Key-Length-Value packets that are used to describe the positioning of the camera and the orientation to generate a ground-based mapping for each frame of the video. I.E from drone footage it will map the image onto a map and change the size/orientation and other data based on the metadata in the video file.

TODO:

• Get sample video files for testing
• Initial script running kwiver/klv-dump to extract metadata from sample files
• Docker Worker Setup
  • Create a new docker configuration file that utilize the kitware/kwiver as a base file and installs the django celery tasks
  • In docker-compose create a new Queue for 'kwiver'
  • Tasks that use kwiver will now run through this container
• Create ORM Models
  • FMV Layer inherits from AbstractMap layer
    • implement new bounds calcualation for ti
    • FMVVideo - S3Filefield (source video)
    • geojson_file = S3FileField (geojson for the bboxes with frame index properties, the camera positioning with frame index properties, and the union gemoetry for the entire camera view
  • FMV Metadata Data
    • ground_frames = geoJSON Polygons with a property for frameId and a property of type == 'ground_frame'
    • ground_union = Large geoJSON Polygon with property of type == 'ground_union'
    • flight_path = geoJSON points with property for frameId and a property of type == 'flight_path
  • FMVVectorFeature - similar to VectorFeature but it has a foreignkey to the FMVLayer
    • This will store the features for the ground_frames
• Create FMV KLV Processing task
  • Task runs kwiver klv-dump similar to initial script to output the geojson file required
  • Creates a new FMV Layer from the data and creates an FMV Metadata Field and populates all of the polygon/point fields
  • Add to this a way to run the importing script to process this data and load it
• REST Endpoints
  • Update existing functions/endpoints to handle a new AbstractMapLayer type FMVLayer
    • MapLayerViewSet.create
    • MapLayerViewSet.list_all_map_layers
    • MapLayerViewSet.list
    • MapLayerViewSet.map_layer_bbox
    • MapLayerViewSet.update_name
  • Add New Rest Endpoints for FMVLayerViewSet
    • fmv-layer/tiles/{x}/{y}/{z} - Need a tile server for the geojson data for the footprint, the camera positioning and basic frames. Use the FMVVectorFeature to render
    • GET fmv-layer/{id} - returns back the fmv-layer information like the name. Presigned URL of the fmv media file, a mapping of frames to bounding boxes and the other data
• Basic Client Support
  • Support for new Abstract Map Layer type 'fmv-layer'
    • Requires a bit of changes in a bunch of areas to support a new type
  • Layer rendering support
    • VectorLayer
      • Footprint, flightplan, bbox frames rendered with filters for scrubbing through the values
      • support for syncing with video or independent frame slider
    • FMV Video Layer
      • Support for repositioning bbox based on the current frame index
      • Seeking/Playing/Pausing the video and using keyboard controls for it
      • Make sure it can handle what to do if the frame isn't there yet
  • FMV Layer Config Controls
    • Play/pause seeking, beginning, end icons for playback of the video
    • Ability to toggle on/off video, flight_path, union area, individual frames
    • Can filter based on frames or render all of the values at once
    • Add opacity settings for the video layer
    • Lock Camera Options
      • Lock to video with bounds multiple
    • Connect frame scrubber to the seeking of the video
    • Make it so there is an indicator of missing bbox values if there doesn't happen to some.
  • Playback improvements
    • Use requestAnimationFrame instead of events on HTMLVideoElement
    • Look into animations to smooth movement of the adjustment of the video
  • Include other keys in the flight_path points
  • Update Point selection so it displays the selected point

Future Upgrades:

• Legend Controls
  • Standard opacity controls
  • FrameSlider and play/pause/advance frames
• Frame-Scrubber doesn't update the video frame (rendering is a limitation of map-libre, may require upstream patch)
• Locking to the video location + the flight path location

[BryonLewis]

Issues:

• Provide Aashish with instructions on how to launch and run the FMV version
  • Include docker compose manage.py initialization instructions
• Update the FMV corner values so they properly are aligned based on the rotation angle of the drone.
  • I should be able to use Platform Heading Angle (degrees) to determine the orientation of the drone and that can be used to adjust the corners for the video.
• Add playback speed controls 0.25, 0.5, 1, 2, 4, 8
• See if I can get playback and seeking to work at the same time
• Remove animation from the zoom locking