Add parallax mapping to bevy_pbr

Add a [parallax mapping] shader to bevy. Please note that
this is a 3d technique, NOT a 2d sidescroller feature.

- Add related fields to `StandardMaterial`
- update the pbr shader
- Add an example taking advantage of parallax mapping

A pre-existing implementation exists at:
https://github.com/nicopap/bevy_mod_paramap/

The implementation is derived from:
https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28

Further discussion on literature is found in the `bevy_mod_paramap` README.

Limitations
-----------

- The mesh silhouette isn't affected by the depth map.
- The depth of the pixel does not reflect its visual position, resulting
  in artifacts for depth-dependent features such as fog or SSAO
- GLTF does not define a height map texture, so somehow the user will
  always need to work around this limitation, though [an extension is in
  the works][gltf]

Future work
-----------

- It's possible to update the depth in the depth buffer to follow the
  parallaxed texture. This would enable interop with depth-based
  visual effects, it also allows `discard`ing pixels of materials when
  computed depth is higher than the one in depth buffer
- Cheap lower quality single-sample method using [offset limiting]
- Add distance fading, to disable parallaxing (relatively expensive)
  on distant objects
- GLTF extension to allow defining height maps. Or a workaround
  implemented through a blender plugin to the GLTF exporter that
  uses the `extras` field to add height map.
- [Quadratic surface vertex attributes][oliveira_3] to enable parallax
  mapping on bending surfaces and allow clean silhouetting.
- noise based sampling, to limit the pancake artifacts.
- Cone mapping ([GPU gems], [Simcity (2013)][simcity]). Requires
  preprocessing, increase depth map size, reduces sample count greatly.
- [Quadtree parallax mapping][qpm] (also requires preprocessing)
- Self-shadowing of parallax-mapped surfaces by modifying the shadow map

https://user-images.githubusercontent.com/26321040/223563792-dffcc6ab-70e8-4ff9-90d1-b36c338695ad.mp4

---

- Add a `depth_map` field to the `StandardMaterial`, it is a greyscale
  image where white represents bottom and black the top. If `depth_map`
  is set, bevy's pbr shader will use it to do [parallax mapping] to
  give an increased feel of depth to the material. This is similar to a
  displacement map, but with infinite precision at fairly low cost.
- The fields `parallax_mapping_method`, `parallax_depth` and
  `max_parallax_layer_count` allow finer grained control over the
  behavior of the parallax shader.
- Add the `parallax_mapping` example to show off the effect.

[parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping
[oliveira_3]: https://www.inf.ufrgs.br/~oliveira/pubs_files/Oliveira_Policarpo_RP-351_Jan_2005.pdf
[GPU gems]: https://developer.nvidia.com/gpugems/gpugems3/part-iii-rendering/chapter-18-relaxed-cone-stepping-relief-mapping
[simcity]: https://community.simtropolis.com/omnibus/other-games/building-and-rendering-simcity-2013-r247/
[offset limiting]: https://raw.githubusercontent.com/marcusstenbeck/tncg14-parallax-mapping/master/documents/Parallax%20Mapping%20with%20Offset%20Limiting%20-%20A%20Per-Pixel%20Approximation%20of%20Uneven%20Surfaces.pdf
[gltf]: KhronosGroup/glTF#2196
[qpm]: https://www.gamedevs.org/uploads/quadtree-displacement-mapping-with-height-blending.pdf

Co-authored-by: Robert Swain <[email protected]>

Cargo.toml

@@ -561,6 +561,17 @@ description = "Demonstrates use of Physically Based Rendering (PBR) properties"
 category = "3D Rendering"
 wasm = true
 
+[[example]]
+name = "parallax_mapping"
+path = "examples/3d/parallax_mapping.rs"
+required-features = [ "jpeg" ]
+
+[package.metadata.example.parallax_mapping]
+name = "Parallax Mapping"
+description = "Demonstrates use of a normal map and height map for parallax mapping"
+category = "3D Rendering"
+wasm = true
+
 [[example]]
 name = "render_to_texture"
 path = "examples/3d/render_to_texture.rs"

crates/bevy_pbr/src/lib.rs

@@ -6,6 +6,7 @@ mod environment_map;
 mod fog;
 mod light;
 mod material;
+mod parallax;
 mod pbr_material;
 mod prepass;
 mod render;
@@ -16,6 +17,7 @@ pub use environment_map::EnvironmentMapLight;
 pub use fog::*;
 pub use light::*;
 pub use material::*;
+pub use parallax::*;
 pub use pbr_material::*;
 pub use prepass::*;
 pub use render::*;
@@ -32,6 +34,7 @@ pub mod prelude {
         fog::{FogFalloff, FogSettings},
         light::{AmbientLight, DirectionalLight, PointLight, SpotLight},
         material::{Material, MaterialPlugin},
+        parallax::ParallaxMappingMethod,
         pbr_material::StandardMaterial,
     };
 }
@@ -80,6 +83,8 @@ pub const PBR_FUNCTIONS_HANDLE: HandleUntyped =
     HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 16550102964439850292);
 pub const PBR_AMBIENT_HANDLE: HandleUntyped =
     HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 2441520459096337034);
+pub const PARALLAX_MAPPING_SHADER_HANDLE: HandleUntyped =
+    HandleUntyped::weak_from_u64(Shader::TYPE_UUID, 17035894873630133905);
 
 /// Sets up the entire PBR infrastructure of bevy.
 pub struct PbrPlugin {
@@ -148,6 +153,12 @@ impl Plugin for PbrPlugin {
             "render/pbr_prepass.wgsl",
             Shader::from_wgsl
         );
+        load_internal_asset!(
+            app,
+            PARALLAX_MAPPING_SHADER_HANDLE,
+            "render/parallax_mapping.wgsl",
+            Shader::from_wgsl
+        );
 
         app.register_asset_reflect::<StandardMaterial>()
             .register_type::<AmbientLight>()

crates/bevy_pbr/src/parallax.rs

@@ -0,0 +1,20 @@
+use bevy_reflect::{FromReflect, Reflect};
+
+/// The parallax mapping method to use to compute a displacement based on the
+/// material's [`depth_map`].
+///
+/// See the `parallax_mapping.wgsl` shader code for implementation details
+/// and explanation of the methods used.
+///
+/// [`depth_map`]: crate::StandardMaterial::depth_map
+#[derive(Debug, Copy, Clone, PartialEq, Eq, Default, Reflect, FromReflect)]
+pub enum ParallaxMappingMethod {
+    /// A simple linear interpolation, using a single texture sample.
+    #[default]
+    ParallaxOcclusionMapping,
+    /// A discovery of 5 iterations of the best displacement
+    /// value. Each iteration incurs a texture sample.
+    ///
+    /// The result has fewer visual artifacts than `ParallaxOcclusionMapping`.
+    ReliefMapping { n_steps: u32 },
+}

crates/bevy_pbr/src/pbr_material.rs

@@ -1,6 +1,6 @@
 use crate::{
-    AlphaMode, Material, MaterialPipeline, MaterialPipelineKey, PBR_PREPASS_SHADER_HANDLE,
-    PBR_SHADER_HANDLE,
+    AlphaMode, Material, MaterialPipeline, MaterialPipelineKey, ParallaxMappingMethod,
+    PBR_PREPASS_SHADER_HANDLE, PBR_SHADER_HANDLE,
 };
 use bevy_asset::Handle;
 use bevy_math::Vec4;
@@ -231,6 +231,84 @@ pub struct StandardMaterial {
     ///
     /// [z-fighting]: https://en.wikipedia.org/wiki/Z-fighting
     pub depth_bias: f32,
+
+    /// The depth map used for [parallax mapping].
+    ///
+    /// It is a greyscale image where white represents bottom and black the top.
+    /// If this field is set, bevy will apply [parallax mapping].
+    /// Parallax mapping, unlike simple normal maps, will move the texture
+    /// coordinate according to the current perspective,
+    /// giving actual depth to the texture.
+    ///
+    /// The visual result is similar to a displacement map,
+    /// but does not require additional geometry.
+    ///
+    /// Use the [`parallax_depth`] field to control the depth of the parallax.
+    ///
+    /// ## Limitations
+    ///
+    /// - It will look weird on bent/non-planar surfaces.
+    /// - The depth of the pixel does not reflect its visual position, resulting
+    ///   in artifacts for depth-dependent features such as fog or SSAO.
+    /// - For the same reason, the the geometry silhouette will always be
+    ///   the one of the actual geometry, not the parallaxed version, resulting
+    ///   in awkward looks on intersecting parallaxed surfaces.
+    ///
+    /// ## Performance
+    ///
+    /// Parallax mapping requires multiple texture lookups, proportional to
+    /// [`max_parallax_layer_count`], which might be costly.
+    ///
+    /// Use the [`parallax_mapping_method`] and [`max_parallax_layer_count`] fields
+    /// to tweak the shader, trading graphical quality for performance.
+    ///
+    /// To improve performance, set your `depth_map`'s [`Image::sampler_descriptor`]
+    /// filter mode to `FilterMode::Nearest`, as [this paper] indicates, it improves
+    /// performance a bit.
+    ///
+    /// To reduce artifacts, avoid steep changes in depth, blurring the depth
+    /// map helps with this.
+    ///
+    /// Larger depth maps haves a disproportionate performance impact.
+    ///
+    /// [this paper]: https://www.diva-portal.org/smash/get/diva2:831762/FULLTEXT01.pdf
+    /// [parallax mapping]: https://en.wikipedia.org/wiki/Parallax_mapping
+    /// [`parallax_depth`]: StandardMaterial::parallax_depth
+    /// [`parallax_mapping_method`]: StandardMaterial::parallax_mapping_method
+    /// [`max_parallax_layer_count`]: StandardMaterial::max_parallax_layer_count
+    #[texture(11)]
+    #[sampler(12)]
+    pub depth_map: Option<Handle<Image>>,
+
+    /// How deep the offset introduced by the depth map should be.
+    ///
+    /// Default is `0.1`, anything over that value may look distorted.
+    /// Lower values lessen the effect.
+    ///
+    /// The depth is relative to texture size. This means that if your texture
+    /// occupies a surface of `1` world unit, and `parallax_depth` is `0.1`, then
+    /// the in-world depth will be of `0.1` world units.
+    /// If the texture stretches for `10` world units, then the final depth
+    /// will be of `1` world unit.
+    pub parallax_depth: f32,
+
+    /// Which parallax mapping method to use.
+    ///
+    /// We recommend that all objects use the same [`ParallaxMappingMethod`], to avoid
+    /// duplicating and running two shaders.
+    pub parallax_mapping_method: ParallaxMappingMethod,
+
+    /// In how many layers to split the depth maps for parallax mapping.
+    ///
+    /// If you are seeing jaggy edges, increase this value.
+    /// However, this incurs a performance cost.
+    ///
+    /// Dependent on the situation, switching to [`ParallaxMappingMethod::ReliefMapping`]
+    /// and keeping this value low might have better performance than increasing the
+    /// layer count while using [`ParallaxMappingMethod::ParallaxOcclusionMapping`].
+    ///
+    /// Default is `16.0`.
+    pub max_parallax_layer_count: f32,
 }
 
 impl Default for StandardMaterial {
@@ -260,6 +338,10 @@ impl Default for StandardMaterial {
             fog_enabled: true,
             alpha_mode: AlphaMode::Opaque,
             depth_bias: 0.0,
+            depth_map: None,
+            parallax_depth: 0.1,
+            max_parallax_layer_count: 16.0,
+            parallax_mapping_method: ParallaxMappingMethod::ParallaxOcclusionMapping,
         }
     }
 }
@@ -302,6 +384,7 @@ bitflags::bitflags! {
         const TWO_COMPONENT_NORMAL_MAP   = (1 << 6);
         const FLIP_NORMAL_MAP_Y          = (1 << 7);
         const FOG_ENABLED                = (1 << 8);
+        const DEPTH_MAP                  = (1 << 9); // Used for parallax mapping
         const ALPHA_MODE_RESERVED_BITS   = (Self::ALPHA_MODE_MASK_BITS << Self::ALPHA_MODE_SHIFT_BITS); // ← Bitmask reserving bits for the `AlphaMode`
         const ALPHA_MODE_OPAQUE          = (0 << Self::ALPHA_MODE_SHIFT_BITS);                          // ← Values are just sequential values bitshifted into
         const ALPHA_MODE_MASK            = (1 << Self::ALPHA_MODE_SHIFT_BITS);                          //   the bitmask, and can range from 0 to 7.
@@ -341,6 +424,13 @@ pub struct StandardMaterialUniform {
     /// When the alpha mode mask flag is set, any base color alpha above this cutoff means fully opaque,
     /// and any below means fully transparent.
     pub alpha_cutoff: f32,
+    /// The depth of the [`StandardMaterial::depth_map`] to apply.
+    pub parallax_depth: f32,
+    /// In how many layers to split the depth maps for Steep parallax mapping.
+    ///
+    /// If your `parallax_depth` is >0.1 and you are seeing jaggy edges,
+    /// increase this value. However, this incurs a performance cost.
+    pub max_parallax_layer_count: f32,
 }
 
 impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
@@ -367,6 +457,9 @@ impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
         if self.fog_enabled {
             flags |= StandardMaterialFlags::FOG_ENABLED;
         }
+        if self.depth_map.is_some() {
+            flags |= StandardMaterialFlags::DEPTH_MAP;
+        }
         let has_normal_map = self.normal_map_texture.is_some();
         if has_normal_map {
             if let Some(texture) = images.get(self.normal_map_texture.as_ref().unwrap()) {
@@ -407,15 +500,19 @@ impl AsBindGroupShaderType<StandardMaterialUniform> for StandardMaterial {
             reflectance: self.reflectance,
             flags: flags.bits(),
             alpha_cutoff,
+            parallax_depth: self.parallax_depth,
+            max_parallax_layer_count: self.max_parallax_layer_count,
         }
     }
 }
 
+/// The pipeline key for [`StandardMaterial`].
 #[derive(Clone, PartialEq, Eq, Hash)]
 pub struct StandardMaterialKey {
     normal_map: bool,
     cull_mode: Option<Face>,
     depth_bias: i32,
+    relief_mapping: bool,
 }
 
 impl From<&StandardMaterial> for StandardMaterialKey {
@@ -424,6 +521,10 @@ impl From<&StandardMaterial> for StandardMaterialKey {
             normal_map: material.normal_map_texture.is_some(),
             cull_mode: material.cull_mode,
             depth_bias: material.depth_bias as i32,
+            relief_mapping: matches!(
+                material.parallax_mapping_method,
+                ParallaxMappingMethod::ReliefMapping { .. }
+            ),
         }
     }
 }
@@ -435,11 +536,14 @@ impl Material for StandardMaterial {
         _layout: &MeshVertexBufferLayout,
         key: MaterialPipelineKey<Self>,
     ) -> Result<(), SpecializedMeshPipelineError> {
-        if key.bind_group_data.normal_map {
-            if let Some(fragment) = descriptor.fragment.as_mut() {
-                fragment
-                    .shader_defs
-                    .push("STANDARDMATERIAL_NORMAL_MAP".into());
+        if let Some(fragment) = descriptor.fragment.as_mut() {
+            let shader_defs = &mut fragment.shader_defs;
+
+            if key.bind_group_data.normal_map {
+                shader_defs.push("STANDARDMATERIAL_NORMAL_MAP".into());
+            }
+            if key.bind_group_data.relief_mapping {
+                shader_defs.push("RELIEF_MAPPING".into());
             }
         }
         descriptor.primitive.cull_mode = key.bind_group_data.cull_mode;

crates/bevy_pbr/src/render/parallax_mapping.wgsl

@@ -0,0 +1,105 @@
+#define_import_path bevy_pbr::parallax_mapping
+
+fn sample_depth_map(uv: vec2<f32>) -> f32 {
+    return textureSample(depth_map_texture, depth_map_sampler, uv).r;
+}
+
+// An implementation of parallax mapping, see https://en.wikipedia.org/wiki/Parallax_mapping
+// Code derived from: https://web.archive.org/web/20150419215321/http://sunandblackcat.com/tipFullView.php?l=eng&topicid=28
+fn parallaxed_uv(
+    depth: f32,
+    max_layer_count: f32,
+    // The original uv
+    uv: vec2<f32>,
+    // The vector from camera to the surface of material
+    V: vec3<f32>,
+) -> vec2<f32> {
+    var uv = uv;
+    if max_layer_count < 1.0 {
+        return uv;
+    }
+
+    // Steep Parallax Mapping
+    // ======================
+    // Split the depth map into `layer_count` layers.
+    // When V hits the surface of the mesh (excluding depth displacement),
+    // if the depth is not below or on surface including depth displacement (textureSample), then
+    // look forward (-= delta_uv) according to V and distance between hit surface and
+    // depth map surface, repeat until below the surface.
+    //
+    // Where `layer_count` is interpolated between `min_layer_count` and
+    // `max_layer_count` according to the steepness of V.
+
+    let view_steepness = abs(dot(vec3<f32>(0.0, 0.0, 1.0), V));
+    // We mix with minimum value 1.0 because otherwise, with 0.0, we get
+    // a nice division by zero in surfaces parallel to viewport, resulting
+    // in a singularity.
+    let layer_count = mix(max_layer_count, 1.0, view_steepness);
+    let layer_height = 1.0 / layer_count;
+    var delta_uv = depth * V.xy / V.z / layer_count;
+
+    var current_layer_height = 0.0;
+    var current_height = sample_depth_map(uv);
+
+    // This at most runs layer_count times
+    while true {
+        if (current_height <= current_layer_height) {
+            break;
+        }
+        current_layer_height += layer_height;
+        uv -= delta_uv;
+        current_height = sample_depth_map(uv);
+    }
+
+#ifdef RELIEF_MAPPING
+    // Relief Mapping
+    // ==============
+    // "Refine" the rough result from Steep Parallax Mapping
+    // with a binary search between the layer selected by steep parallax
+    // and the next one to find a point closer to the depth map surface.
+    // This reduces the jaggy step artifacts from steep parallax mapping.
+    let MAX_STEPS: i32 = 5;
+
+    delta_uv *= 0.5;
+    var delta_height = 0.5 * layer_height;
+    uv += delta_uv;
+    current_layer_height -= delta_height;
+    for (var i: i32 = 0; i < MAX_STEPS; i++) {
+        // Sample depth at current offset
+        current_height = sample_depth_map(uv);
+
+        // Halve the deltas for the next step
+        delta_uv *= 0.5;
+        delta_height *= 0.5;
+
+        // Step based on whether the current depth is above or below the depth map
+        if (current_height > current_layer_height) {
+            uv -= delta_uv;
+            current_layer_height += delta_height;
+        } else {
+            uv += delta_uv;
+            current_layer_height -= delta_height;
+        }
+    }
+#else    
+    // Parallax Occlusion mapping
+    // ==========================
+    // "Refine" Steep Parallax Mapping by interpolating between the
+    // previous layer's height and the computed layer height.
+    // Only requires a single lookup, unlike Relief Mapping, but
+    // may incur artifacts on very steep relief.
+    let previous_uv = uv + delta_uv;
+    let next_height = current_height - current_layer_height;
+    let previous_height = sample_depth_map(previous_uv) - current_layer_height + layer_height;
+
+    let weight = next_height / (next_height - previous_height);
+
+    uv = mix(uv, previous_uv, weight);
+
+    current_layer_height += mix(next_height, previous_height, weight);
+#endif
+
+    // Note: `current_layer_height` is not returned, but may be useful
+    // for light computation later on in future improvements of the pbr shader.
+    return uv;
+}