vkCreateInstance should return an error if a layer can't be loaded

[davidlunarg]

If the json file for the trace layer is missing or otherwise can't be found, vkCreateInstance returns VK_ERROR_LAYER_NOT_PRESENT.

If the shared library for the trace layer is missing otherwise can't be found, vkCreateInstance returns VK_SUCCESS. It seems it should return VK_ERROR_LAYER_NOT_PRESENT.

Quoting from the vkCreateInstance manual page:

vkCreateInstance verifies that the requested layers exist. If not, vkCreateInstance will return VK_ERROR_LAYER_NOT_PRESENT.

This issue needs to be resolved so that LunarG/VulkanTools#501 can work correctly.

[charles-lunarg]

This is because when the loader does the validation of the layers, it doesn't open the layer library, it only validates that the json is correct, and that ppEnableLayerNames is in the list of layers available on the system.

The actual loading of the layer library is only done when setting up the dispatch chain, and as I found, it simply skips over the layer if it fails to load the library, but doesn't report an error.

loader.c, line 5883

lib_handle = loaderOpenLayerFile(inst, "instance", layer_prop);
            if (!lib_handle) {
                continue;
            }

While it might make sense to return VK_ERROR_LAYER_NOT_PRESENT where the continue is now, it would be less intrusive if in the layer validation there was a check to see if the layer library exists (which doesn't require loading it, as that may be a potentially expensive operation).