Improve make_texture for very large images #3829
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
`make_texture()` already took a configuration option "maketx:forcefloat" (defaults to 1), which if set to 0 disables the conversion to a float buffer for intermediate computation. For `oiiotool -otex`, there was no way to specify this. This patch allows it to be disabled with the new optional modifier `-otex:forcefloat=0`. Also, in make_texture, this option only controlled the initial conversion of the whole buffer to float, but the downsized levels all were float either way. This patch now uses the forcefloat config option to also control the buffers used for downsizing. Disabling forcefloat like this has a speed penalty (the downsizing math on float buffers is faster than going in and out of float for every indvidual value) and loses some precision in the process of repeatedly downsizing to generate the MIP levels. But it can sure save a lot of memory if you have a ginormous uint8 image and want to avod the 4x balooning of memory. I don't recommend disabling forcefloat except for images that can't fit into memory for the texture creation without it. But when this is the right hammer to use, it can be the difference between being able to create the texture, or not. Some other improvements: * `oiiotool -otex` wasn't outputting ANY status output during texture output as maketx does, and for that matter, -v and -debug were not being influencing the behavior of make_texture at all despite it having such support. So fix to pass down the right flags. Now there are proper verbose status and debug messages when requested. * Eliminate useless call to is_monochrome by calling it only when the previously-computed pixel stats indicate that the average value of all the channels is the same (if they don't have the same average, there's no way the pixel-by-pixel values will be the same in all channels for all pixels). This saves an extra traversal of all the pixel memory during texture creation. * Use synchronized/flushed status messages so users can see them as they are issued for very long maketx tasks. This also included some conversion from old iostream stuff to fmt-based print. * For verbose/debug, more fine-grained timing for make_texture, in particular, print the time spent downsizing and writing each MIP level.
lgritz
added a commit
to lgritz/OpenImageIO
that referenced
this pull request
May 8, 2023
…#3829) `make_texture()` already took a configuration option "maketx:forcefloat" (defaults to 1), which if set to 0 disables the conversion to a float buffer for intermediate computation. For `oiiotool -otex`, there was no way to specify this. This patch allows it to be disabled with the new optional modifier `-otex:forcefloat=0`. Also, in make_texture, this option only controlled the initial conversion of the whole buffer to float, but the downsized levels all were float either way. This patch now uses the forcefloat config option to also control the buffers used for downsizing. Disabling forcefloat like this has a speed penalty (the downsizing math on float buffers is faster than going in and out of float for every indvidual value) and loses some precision in the process of repeatedly downsizing to generate the MIP levels. But it can sure save a lot of memory if you have a ginormous uint8 image and want to avod the 4x balooning of memory. I don't recommend disabling forcefloat except for images that can't fit into memory for the texture creation without it. But when this is the right hammer to use, it can be the difference between being able to create the texture, or not. Some other improvements: * `oiiotool -otex` wasn't outputting ANY status output during texture output as maketx does, and for that matter, -v and -debug were not being influencing the behavior of make_texture at all despite it having such support. So fix to pass down the right flags. Now there are proper verbose status and debug messages when requested. * Eliminate useless call to is_monochrome by calling it only when the previously-computed pixel stats indicate that the average value of all the channels is the same (if they don't have the same average, there's no way the pixel-by-pixel values will be the same in all channels for all pixels). This saves an extra traversal of all the pixel memory during texture creation. * Use synchronized/flushed status messages so users can see them as they are issued for very long maketx tasks. This also included some conversion from old iostream stuff to fmt-based print. * For verbose/debug, more fine-grained timing for make_texture, in particular, print the time spent downsizing and writing each MIP level.
lgritz
added a commit
to lgritz/OpenImageIO
that referenced
this pull request
May 15, 2023
…#3829) `make_texture()` already took a configuration option "maketx:forcefloat" (defaults to 1), which if set to 0 disables the conversion to a float buffer for intermediate computation. For `oiiotool -otex`, there was no way to specify this. This patch allows it to be disabled with the new optional modifier `-otex:forcefloat=0`. Also, in make_texture, this option only controlled the initial conversion of the whole buffer to float, but the downsized levels all were float either way. This patch now uses the forcefloat config option to also control the buffers used for downsizing. Disabling forcefloat like this has a speed penalty (the downsizing math on float buffers is faster than going in and out of float for every indvidual value) and loses some precision in the process of repeatedly downsizing to generate the MIP levels. But it can sure save a lot of memory if you have a ginormous uint8 image and want to avod the 4x balooning of memory. I don't recommend disabling forcefloat except for images that can't fit into memory for the texture creation without it. But when this is the right hammer to use, it can be the difference between being able to create the texture, or not. Some other improvements: * `oiiotool -otex` wasn't outputting ANY status output during texture output as maketx does, and for that matter, -v and -debug were not being influencing the behavior of make_texture at all despite it having such support. So fix to pass down the right flags. Now there are proper verbose status and debug messages when requested. * Eliminate useless call to is_monochrome by calling it only when the previously-computed pixel stats indicate that the average value of all the channels is the same (if they don't have the same average, there's no way the pixel-by-pixel values will be the same in all channels for all pixels). This saves an extra traversal of all the pixel memory during texture creation. * Use synchronized/flushed status messages so users can see them as they are issued for very long maketx tasks. This also included some conversion from old iostream stuff to fmt-based print. * For verbose/debug, more fine-grained timing for make_texture, in particular, print the time spent downsizing and writing each MIP level.
lgritz
added a commit
to lgritz/OpenImageIO
that referenced
this pull request
Jun 7, 2023
…#3829) `make_texture()` already took a configuration option "maketx:forcefloat" (defaults to 1), which if set to 0 disables the conversion to a float buffer for intermediate computation. For `oiiotool -otex`, there was no way to specify this. This patch allows it to be disabled with the new optional modifier `-otex:forcefloat=0`. Also, in make_texture, this option only controlled the initial conversion of the whole buffer to float, but the downsized levels all were float either way. This patch now uses the forcefloat config option to also control the buffers used for downsizing. Disabling forcefloat like this has a speed penalty (the downsizing math on float buffers is faster than going in and out of float for every indvidual value) and loses some precision in the process of repeatedly downsizing to generate the MIP levels. But it can sure save a lot of memory if you have a ginormous uint8 image and want to avod the 4x balooning of memory. I don't recommend disabling forcefloat except for images that can't fit into memory for the texture creation without it. But when this is the right hammer to use, it can be the difference between being able to create the texture, or not. Some other improvements: * `oiiotool -otex` wasn't outputting ANY status output during texture output as maketx does, and for that matter, -v and -debug were not being influencing the behavior of make_texture at all despite it having such support. So fix to pass down the right flags. Now there are proper verbose status and debug messages when requested. * Eliminate useless call to is_monochrome by calling it only when the previously-computed pixel stats indicate that the average value of all the channels is the same (if they don't have the same average, there's no way the pixel-by-pixel values will be the same in all channels for all pixels). This saves an extra traversal of all the pixel memory during texture creation. * Use synchronized/flushed status messages so users can see them as they are issued for very long maketx tasks. This also included some conversion from old iostream stuff to fmt-based print. * For verbose/debug, more fine-grained timing for make_texture, in particular, print the time spent downsizing and writing each MIP level.
birsoyo
pushed a commit
to birsoyo/oiio
that referenced
this pull request
Jun 8, 2023
…#3829) `make_texture()` already took a configuration option "maketx:forcefloat" (defaults to 1), which if set to 0 disables the conversion to a float buffer for intermediate computation. For `oiiotool -otex`, there was no way to specify this. This patch allows it to be disabled with the new optional modifier `-otex:forcefloat=0`. Also, in make_texture, this option only controlled the initial conversion of the whole buffer to float, but the downsized levels all were float either way. This patch now uses the forcefloat config option to also control the buffers used for downsizing. Disabling forcefloat like this has a speed penalty (the downsizing math on float buffers is faster than going in and out of float for every indvidual value) and loses some precision in the process of repeatedly downsizing to generate the MIP levels. But it can sure save a lot of memory if you have a ginormous uint8 image and want to avod the 4x balooning of memory. I don't recommend disabling forcefloat except for images that can't fit into memory for the texture creation without it. But when this is the right hammer to use, it can be the difference between being able to create the texture, or not. Some other improvements: * `oiiotool -otex` wasn't outputting ANY status output during texture output as maketx does, and for that matter, -v and -debug were not being influencing the behavior of make_texture at all despite it having such support. So fix to pass down the right flags. Now there are proper verbose status and debug messages when requested. * Eliminate useless call to is_monochrome by calling it only when the previously-computed pixel stats indicate that the average value of all the channels is the same (if they don't have the same average, there's no way the pixel-by-pixel values will be the same in all channels for all pixels). This saves an extra traversal of all the pixel memory during texture creation. * Use synchronized/flushed status messages so users can see them as they are issued for very long maketx tasks. This also included some conversion from old iostream stuff to fmt-based print. * For verbose/debug, more fine-grained timing for make_texture, in particular, print the time spent downsizing and writing each MIP level.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
make_texture()already took a configuration option "maketx:forcefloat" (defaults to 1), which if set to 0 disables the conversion to a float buffer for intermediate computation.For
oiiotool -otex, there was no way to specify this. This patch allows it to be disabled with the new optional modifier-otex:forcefloat=0.Also, in make_texture, this option only controlled the initial conversion of the whole buffer to float, but the downsized levels all were float either way. This patch now uses the forcefloat config option to also control the buffers used for downsizing.
Disabling forcefloat like this has a speed penalty (the downsizing math on float buffers is faster than going in and out of float for every indvidual value) and loses some precision in the process of repeatedly downsizing to generate the MIP levels. But it can sure save a lot of memory if you have a ginormous uint8 image and want to avod the 4x balooning of memory. I don't recommend disabling forcefloat except for images that can't fit into memory for the texture creation without it. But when this is the right hammer to use, it can be the difference between being able to create the texture, or not.
Some other improvements:
oiiotool -otexwasn't outputting ANY status output during texture output as maketx does, and for that matter, -v and -debug were not being influencing the behavior of make_texture at all despite it having such support. So fix to pass down the right flags. Now there are proper verbose status and debug messages when requested.Eliminate useless call to is_monochrome by calling it only when the previously-computed pixel stats indicate that the average value of all the channels is the same (if they don't have the same average, there's no way the pixel-by-pixel values will be the same in all channels for all pixels). This saves an extra traversal of all the pixel memory during texture creation.
Use synchronized/flushed status messages so users can see them as they are issued for very long maketx tasks. This also included some conversion from old iostream stuff to fmt-based print.
For verbose/debug, more fine-grained timing for make_texture, in particular, print the time spent downsizing and writing each MIP level.