Refactor custom shaders to abstract binding differences. Closes #2458

* Newly written shaders and any updated shaders can now use pre-defined macros
  to abstract away binding differences between APIs, so custom shaders will be
  more portable in particular shaders written in HLSL for D3D or GLSL on OpenGL
  won't break on vulkan because they refer to incorrect binds.
This commit is contained in:
baldurk
2022-01-31 19:14:08 +00:00
parent 0d7dcabdac
commit 3ac4bd0ebd
37 changed files with 1382 additions and 613 deletions
+161 -176
View File
@@ -8,9 +8,9 @@ Introduction
The basic process of setting up the custom shader involves writing a shader file that will be compiled and used by RenderDoc. Note that you can use any language that is either natively accepted by the graphics API used, or any language that can be compiled to an accepted shader.
For example on D3D11 or D3D12, hlsl is the only language usable by default, but on Vulkan you can use hlsl or glsl as long as a compiler is available.
For example on D3D11 or D3D12, hlsl is the only language usable by default. On OpenGL only glsl can be used, but on Vulkan you can use glsl or use hlsl as long as a compiler is available.
There are several special global variables that can be specified and will be filled in with values by RenderDoc.
There are several special global helpers that can be declared and used, which will be implemented and return values by RenderDoc. In addition there are automatic macros that can be used to bind resources and still write shaders which work on different APIs with different bindings.
Your pixel shader defines an operation that transforms the raw value from the input texture into a value that will then be displayed by the texture viewer. The usual texture viewer controls for range adaption and channels will still be available on the resulting texture.
@@ -18,7 +18,11 @@ To set up your shader, it's recommended that you use the UI defined in the docum
.. note::
Since ``.glsl`` is used for both Vulkan and OpenGL shaders, you can differentiate by the pre-defined macro ``VULKAN`` if you are writing a shader to be used for both.
Since ``.glsl`` is used for both Vulkan and OpenGL shaders, you can differentiate by the pre-defined macro ``VULKAN`` if you are writing a shader to be used for both.
.. warning::
Previously the custom shaders allowed more direct binding without helper functions or binding macros. These shaders will continue to work as backwards compatibility is maintained, however be aware that these bindings are API specific and so e.g. a shader written for OpenGL in glsl will not work on Vulkan unless care has been taken, or an HLSL shader between Vulkan and D3D. If portability is desired please update and use the new helpers and binding macros, or else be careful only to use custom shaders with the API they were written for.
Predefined inputs
-----------------
@@ -27,7 +31,7 @@ There are several pre-defined inputs that can either be taken as parameters to t
.. warning::
Type and capitalisation is important for these variables, so ensure you use the right declaration!
Type and capitalisation is important for these variables, so ensure you use the right declaration!
The shader editor when using the UI can be used to insert these snippets for you, with the right type and spelling. For GLSL these snippets are inserted at the top of the file just after any ``#version`` statement.
@@ -37,19 +41,19 @@ UV co-ordinates
.. highlight:: c++
.. code:: c++
/* HLSL */
float4 main(float4 pos : SV_Position, float4 uv : TEXCOORD0) : SV_Target0
{
return 1;
}
/* HLSL */
float4 main(float4 pos : SV_Position, float4 uv : TEXCOORD0) : SV_Target0
{
return 1;
}
/* GLSL */
layout (location = 0) in vec2 uv;
/* GLSL */
layout (location = 0) in vec2 uv;
void main()
{
// ...
}
void main()
{
// ...
}
This input is defined in HLSL as a second parameter to the shader entry point. The first defines the usual ``SV_Position`` system semantic, and the first two components of the second ``TEXCOORD0`` parameter gives UV co-ordinates from 0 to 1 in each dimension over the size of the texture (or texture slice).
@@ -59,31 +63,12 @@ You can also use the auto-generated system co-ordinates - ``SV_Position`` or ``g
.. note::
You must bind these parameters like this in this order to ensure the linkage with the vertex shader matches.
You must bind these parameters like this in this order to ensure the linkage with the vertex shader matches.
Constant Parameters
~~~~~~~~~~~~~~~~~~~
There are several constant parameters available, each detailed below with the values they contain. Where possible these are bound by name as globals for convenience, but in Vulkan all variables must be contained within a single uniform buffer. The parameters correspond with the GLSL documentation but are contained within a uniform buffer at binding 0, with a structure given as so:
.. highlight:: c++
.. code:: c++
layout(binding = 0, std140) uniform RENDERDOC_Uniforms
{
uvec4 TexDim;
uint SelectedMip;
uint TextureType;
uint SelectedSliceFace;
int SelectedSample;
uvec4 YUVDownsampleRate;
uvec4 YUVAChannels;
float SelectedRangeMin;
float SelectedRangeMax;
} RENDERDOC;
In this way you can access the properties as ``RENDERDOC.TexDim`` instead of ``RENDERDOC_TexDim``.
There are several constant parameters available, each available via a helper function. They are detailed below with the values they contain.
Texture dimensions
~~~~~~~~~~~~~~~~~~
@@ -91,14 +76,16 @@ Texture dimensions
.. highlight:: c++
.. code:: c++
uint4 RENDERDOC_TexDim; // hlsl
uniform uvec4 RENDERDOC_TexDim; // glsl
uint4 RD_TexDim(); // hlsl
uvec4 RD_TexDim(); // glsl
uint4 RENDERDOC_YUVDownsampleRate; // hlsl / vulkan glsl only
uint4 RENDERDOC_YUVAChannels; // hlsl / vulkan glsl only
uint4 RD_YUVDownsampleRate(); // hlsl
uvec4 RD_YUVDownsampleRate(); // vulkan glsl only
uint4 RD_YUVAChannels(); // hlsl
uvec4 RD_YUVAChannels(); // vulkan glsl only
``RENDERDOC_TexDim`` will be filled out with the following values:
``RD_TexDim`` will return the following values:
* ``.x`` Width
* ``.y`` Height (if 2D or 3D)
@@ -106,7 +93,7 @@ Texture dimensions
* ``.w`` Number of mip levels
``RENDERDOC_YUVDownsampleRate`` will be filled out with the following values:
``RD_YUVDownsampleRate`` will return the following values:
* ``.x`` Horizontal downsample rate. 1 for equal luma and chroma width, 2 for half rate.
* ``.y`` Vertical downsample rate. 1 for equal luma and chroma height, 2 for half rate.
@@ -114,7 +101,7 @@ Texture dimensions
* ``.w`` Number of bits per component, e.g. 8, 10 or 16.
``RENDERDOC_YUVAChannels`` will be filled out an index indicating where each channel comes from in the source textures. The order is ``.x`` for ``Y``, ``.y`` for ``U``, ``.z`` for ``V`` and ``.w`` for ``A``.
``RD_YUVAChannels`` will return an index indicating where each channel comes from in the source textures. The order is ``.x`` for ``Y``, ``.y`` for ``U``, ``.z`` for ``V`` and ``.w`` for ``A``.
The indices for channels in the first texture in the normal 2D slot are ``0, 1, 2, 3``. Indices from ``4`` to ``7`` indicate channels in the second texture, and so on.
@@ -126,11 +113,10 @@ Selected Mip level
.. highlight:: c++
.. code:: c++
uint RENDERDOC_SelectedMip; // hlsl
uniform uint RENDERDOC_SelectedMip; // glsl
uint RD_SelectedMip(); // hlsl or glsl
This variable will be filled out with the selected mip level in the UI.
This will return the selected mip level in the UI.
Selected Slice/Face
~~~~~~~~~~~~~~~~~~~
@@ -138,8 +124,7 @@ Selected Slice/Face
.. highlight:: c++
.. code:: c++
uint RENDERDOC_SelectedSliceFace; // hlsl
uniform uint RENDERDOC_SelectedSliceFace; // glsl
uint RD_SelectedSliceFace(); // hlsl or glsl
This variable will be filled out with the selected texture array slice (or cubemap face) in the UI.
@@ -150,8 +135,7 @@ Selected Multisample sample
.. highlight:: c++
.. code:: c++
int RENDERDOC_SelectedSample; // hlsl
uniform int RENDERDOC_SelectedSample; // glsl
int RD_SelectedSample(); // hlsl or glsl
This variable will be filled out with the selected multisample sample index as chosen in the UI. If the UI has 'average value' selected, this variable will be negative and with an absolute value equal to the number of samples.
@@ -165,14 +149,11 @@ Selected RangeMin, RangeMax
.. highlight:: c++
.. code:: c++
float RENDERDOC_SelectedRangeMin; // hlsl
float RENDERDOC_SelectedRangeMax; // hlsl
uniform int RENDERDOC_SelectedRangeMin; // glsl
uniform int RENDERDOC_SelectedRangeMax; // glsl
float2 RD_SelectedRange(); // hlsl
vec2 RD_SelectedRange(); // glsl
These variables will be filled out with the current Minimum and Maximum values for the Range-selector in the Texture Viewer.
This function will return a pair, with the current Minimum and Maximum values for the Range-selector in the Texture Viewer.
Current texture type
@@ -181,166 +162,170 @@ Current texture type
.. highlight:: c++
.. code:: c++
uint RENDERDOC_TextureType; // hlsl
uniform uint RENDERDOC_TextureType; // glsl
uint RD_TextureType(); // hlsl or glsl
This variable will be set to a given integer value, depending on the type of the current texture being displayed. This can be used to sample from the correct resource.
.. note::
The value varies depending on the API this shader will be used for, as each has different resource bindings.
The value varies depending on the API this shader will be used for, as each has different resource bindings. You should use the defines below to check, which will be portable across APIs
D3D11 or D3D12 / HLSL
^^^^^^^^^^^^^^^^^^^^^
D3D11 or D3D12
^^^^^^^^^^^^^^
#. 1D texture
#. 2D texture
#. 3D texture
#. Depth
#. Depth + Stencil
#. Depth (Multisampled)
#. Depth + Stencil (Multisampled)
#. Legacy: used to be cubemap, removed as it's unused
#. 2D texture (Multisampled)
* ``RD_TextureType_1D`` - 1D texture
* ``RD_TextureType_2D`` - 2D texture
* ``RD_TextureType_3D`` - 3D texture
* ``RD_TextureType_Depth`` - Depth
* ``RD_TextureType_DepthStencil`` - Depth + Stencil
* ``RD_TextureType_DepthMS`` - Depth (Multisampled)
* ``RD_TextureType_DepthStencilMS`` - Depth + Stencil (Multisampled)
* ``RD_TextureType_2DMS`` - 2D texture (Multisampled)
OpenGL / GLSL
^^^^^^^^^^^^^
In all cases on D3D the bindings can be used for arrays or not, interchangeably.
#. 1D texture
#. 2D texture
#. 3D texture
#. Cubemap
#. 1D array texture
#. 2D array texture
#. Cubemap array
#. Rectangle
#. Buffer texture
#. 2D texture (Multisampled)
OpenGL
^^^^^^
Vulkan / GLSL
^^^^^^^^^^^^^
* ``RD_TextureType_1D`` - 1D texture
* ``RD_TextureType_2D`` - 2D texture
* ``RD_TextureType_3D`` - 3D texture
* ``RD_TextureType_Cube`` - Cubemap
* ``RD_TextureType_1D_Array`` - 1D array texture
* ``RD_TextureType_2D_Array`` - 2D array texture
* ``RD_TextureType_Cube_Array`` - Cube array texture
* ``RD_TextureType_Rect`` - Rectangle texture
* ``RD_TextureType_Buffer`` - Buffer texture
* ``RD_TextureType_2DMS`` - 2D texture (Multisampled)
* ``RD_TextureType_2DMS_Array`` - 2D array texture (Multisampled)
#. 1D texture
#. 2D texture
#. 3D texture
#. 2D texture (Multisampled)
OpenGL has separate types and bindings for arrayed and non-arrayed textures.
Samplers (D3D11/D3D12 only)
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Vulkan
^^^^^^
* ``RD_TextureType_1D`` - 1D texture
* ``RD_TextureType_2D`` - 2D texture
* ``RD_TextureType_3D`` - 3D texture
* ``RD_TextureType_2DMS`` - 2D texture (Multisampled)
In all cases on Vulkan the bindings can be used for arrays or not, interchangeably.
Samplers
~~~~~~~~
.. highlight:: c++
.. code:: c++
SamplerState pointSampler : register(s0);
SamplerState linearSampler : register(s1);
These samplers are provided to allow you to sample from the resource as opposed to doing straight loads. They are bound by slot and not by variable name - so this means you can name them as you wish but you must specify the register binding explicitly.
/* HLSL */
SamplerState pointSampler : register(RD_POINT_SAMPLER_BINDING);
SamplerState linearSampler : register(RD_LINEAR_SAMPLER_BINDING);
/* GLSL */
#ifdef VULKAN
layout(binding = RD_POINT_SAMPLER_BINDING) uniform sampler pointSampler;
layout(binding = RD_LINEAR_SAMPLER_BINDING) uniform sampler linearSampler;
#endif
These samplers are provided to allow you to sample from the resource as opposed to doing straight loads. Samplers are not available on OpenGL, so it is recommended to protect the glsl definitions with ``#ifdef VULKAN`` as shown.
Resources
~~~~~~~~~
D3D11 or D3D12 / HLSL
^^^^^^^^^^^^^^^^^^^^^
HLSL
^^^^
.. highlight:: c++
.. code:: c++
Texture1DArray<float4> texDisplayTex1DArray : register(t1);
Texture2DArray<float4> texDisplayTex2DArray : register(t2);
Texture3D<float4> texDisplayTex3D : register(t3);
Texture2DArray<float2> texDisplayTexDepthArray : register(t4);
Texture2DArray<uint2> texDisplayTexStencilArray : register(t5);
Texture2DMSArray<float2> texDisplayTexDepthMSArray : register(t6);
Texture2DMSArray<uint2> texDisplayTexStencilMSArray : register(t7);
Texture2DMSArray<float4> texDisplayTex2DMSArray : register(t9);
Texture2DArray<float4> texDisplayYUVArray : register(t10);
// Float Textures
Texture1DArray<float4> texDisplayTex1DArray : register(RD_FLOAT_1D_ARRAY_BINDING);
Texture2DArray<float4> texDisplayTex2DArray : register(RD_FLOAT_2D_ARRAY_BINDING);
Texture3D<float4> texDisplayTex3D : register(RD_FLOAT_3D_BINDING);
Texture2DMSArray<float4> texDisplayTex2DMSArray : register(RD_FLOAT_2DMS_ARRAY_BINDING);
Texture2DArray<float4> texDisplayYUVArray : register(RD_FLOAT_YUV_ARRAY_BINDING);
Texture1DArray<uint4> texDisplayUIntTex1DArray : register(t11);
Texture2DArray<uint4> texDisplayUIntTex2DArray : register(t12);
Texture3D<uint4> texDisplayUIntTex3D : register(t13);
Texture2DMSArray<uint4> texDisplayUIntTex2DMSArray : register(t19);
// only used on D3D
Texture2DArray<float2> texDisplayTexDepthArray : register(RD_FLOAT_DEPTH_ARRAY_BINDING);
Texture2DArray<uint2> texDisplayTexStencilArray : register(RD_FLOAT_STENCIL_ARRAY_BINDING);
Texture2DMSArray<float2> texDisplayTexDepthMSArray : register(RD_FLOAT_DEPTHMS_ARRAY_BINDING);
Texture2DMSArray<uint2> texDisplayTexStencilMSArray : register(RD_FLOAT_STENCILMS_ARRAY_BINDING);
Texture1DArray<int4> texDisplayIntTex1DArray : register(t21);
Texture2DArray<int4> texDisplayIntTex2DArray : register(t22);
Texture3D<int4> texDisplayIntTex3D : register(t23);
Texture2DMSArray<int4> texDisplayIntTex2DMSArray : register(t29);
// Int Textures
Texture1DArray<int4> texDisplayIntTex1DArray : register(RD_INT_1D_ARRAY_BINDING);
Texture2DArray<int4> texDisplayIntTex2DArray : register(RD_INT_2D_ARRAY_BINDING);
Texture3D<int4> texDisplayIntTex3D : register(RD_INT_3D_BINDING);
Texture2DMSArray<int4> texDisplayIntTex2DMSArray : register(RD_INT_2DMS_ARRAY_BINDING);
OpenGL / GLSL
^^^^^^^^^^^^^
// Unsigned int Textures
Texture1DArray<uint4> texDisplayUIntTex1DArray : register(RD_UINT_1D_ARRAY_BINDING);
Texture2DArray<uint4> texDisplayUIntTex2DArray : register(RD_UINT_2D_ARRAY_BINDING);
Texture3D<uint4> texDisplayUIntTex3D : register(RD_UINT_3D_BINDING);
Texture2DMSArray<uint4> texDisplayUIntTex2DMSArray : register(RD_UINT_2DMS_ARRAY_BINDING);
GLSL
^^^^
.. highlight:: c++
.. code:: c++
// Unsigned int samplers
layout (binding = 1) uniform usampler1D texUInt1D;
layout (binding = 2) uniform usampler2D texUInt2D;
layout (binding = 3) uniform usampler3D texUInt3D;
// skip cube = 4
layout (binding = 5) uniform usampler1DArray texUInt1DArray;
layout (binding = 6) uniform usampler2DArray texUInt2DArray;
// skip cube array = 7
layout (binding = 8) uniform usampler2DRect texUInt2DRect;
layout (binding = 9) uniform usamplerBuffer texUIntBuffer;
layout (binding = 10) uniform usampler2DMS texUInt2DMS;
// Float Textures
layout (binding = RD_FLOAT_1D_ARRAY_BINDING) uniform sampler1DArray tex1DArray;
layout (binding = RD_FLOAT_2D_ARRAY_BINDING) uniform sampler2DArray tex2DArray;
layout (binding = RD_FLOAT_3D_BINDING) uniform sampler3D tex3D;
layout (binding = RD_FLOAT_2DMS_ARRAY_BINDING) uniform sampler2DMSArray tex2DMSArray;
// Int samplers
layout (binding = 1) uniform isampler1D texSInt1D;
layout (binding = 2) uniform isampler2D texSInt2D;
layout (binding = 3) uniform isampler3D texSInt3D;
// skip cube = 4
layout (binding = 5) uniform isampler1DArray texSInt1DArray;
layout (binding = 6) uniform isampler2DArray texSInt2DArray;
// skip cube array = 7
layout (binding = 8) uniform isampler2DRect texSInt2DRect;
layout (binding = 9) uniform isamplerBuffer texSIntBuffer;
layout (binding = 10) uniform isampler2DMS texSInt2DMS;
// YUV textures only supported on vulkan
#ifdef VULKAN
layout(binding = RD_FLOAT_YUV_ARRAY_BINDING) uniform sampler2DArray texYUVArray[2];
#endif
// Floating point samplers
layout (binding = 1) uniform sampler1D tex1D;
layout (binding = 2) uniform sampler2D tex2D;
layout (binding = 3) uniform sampler3D tex3D;
layout (binding = 4) uniform samplerCube texCube;
layout (binding = 5) uniform sampler1DArray tex1DArray;
layout (binding = 6) uniform sampler2DArray tex2DArray;
layout (binding = 7) uniform samplerCubeArray texCubeArray;
layout (binding = 8) uniform sampler2DRect tex2DRect;
layout (binding = 9) uniform samplerBuffer texBuffer;
layout (binding = 10) uniform sampler2DMS tex2DMS;
// OpenGL has more texture types to match
#ifndef VULKAN
layout (binding = RD_FLOAT_1D_BINDING) uniform sampler1D tex1D;
layout (binding = RD_FLOAT_2D_BINDING) uniform sampler2D tex2D;
layout (binding = RD_FLOAT_CUBE_BINDING) uniform samplerCube texCube;
layout (binding = RD_FLOAT_CUBE_ARRAY_BINDING) uniform samplerCubeArray texCubeArray;
layout (binding = RD_FLOAT_RECT_BINDING) uniform sampler2DRect tex2DRect;
layout (binding = RD_FLOAT_BUFFER_BINDING) uniform samplerBuffer texBuffer;
layout (binding = RD_FLOAT_2DMS_BINDING) uniform sampler2DMS tex2DMS;
#endif
Vulkan / GLSL
^^^^^^^^^^^^^
// Int Textures
layout (binding = RD_INT_1D_ARRAY_BINDING) uniform isampler1DArray texSInt1DArray;
layout (binding = RD_INT_2D_ARRAY_BINDING) uniform isampler2DArray texSInt2DArray;
layout (binding = RD_INT_3D_BINDING) uniform isampler3D texSInt3D;
layout (binding = RD_INT_2DMS_ARRAY_BINDING) uniform isampler2DMSArray texSInt2DMSArray;
.. highlight:: c++
.. code:: c++
#ifndef VULKAN
layout (binding = RD_INT_1D_BINDING) uniform isampler1D texSInt1D;
layout (binding = RD_INT_2D_BINDING) uniform isampler2D texSInt2D;
layout (binding = RD_INT_RECT_BINDING) uniform isampler2DRect texSInt2DRect;
layout (binding = RD_INT_BUFFER_BINDING) uniform isamplerBuffer texSIntBuffer;
layout (binding = RD_INT_2DMS_BINDING) uniform isampler2DMS texSInt2DMS;
#endif
// Floating point samplers
// Unsigned int Textures
layout (binding = RD_UINT_1D_ARRAY_BINDING) uniform usampler1DArray texSInt1DArray;
layout (binding = RD_UINT_2D_ARRAY_BINDING) uniform usampler2DArray texSInt2DArray;
layout (binding = RD_UINT_3D_BINDING) uniform usampler3D texSInt3D;
layout (binding = RD_UINT_2DMS_ARRAY_BINDING) uniform usampler2DMSArray texSInt2DMSArray;
// binding = 5 + RENDERDOC_TextureType
layout(binding = 6) uniform sampler1DArray tex1DArray;
layout(binding = 7) uniform sampler2DArray tex2DArray;
layout(binding = 8) uniform sampler3D tex3D;
layout(binding = 9) uniform sampler2DMS tex2DMS;
layout(binding = 10) uniform sampler2DArray texYUV;
// Unsigned int samplers
// binding = 10 + RENDERDOC_TextureType
layout(binding = 11) uniform usampler1DArray texUInt1DArray;
layout(binding = 12) uniform usampler2DArray texUInt2DArray;
layout(binding = 13) uniform usampler3D texUInt3D;
layout(binding = 14) uniform usampler2DMS texUInt2DMS;
// Int samplers
// binding = 15 + RENDERDOC_TextureType
layout(binding = 16) uniform isampler1DArray texSInt1DArray;
layout(binding = 17) uniform isampler2DArray texSInt2DArray;
layout(binding = 18) uniform isampler3D texSInt3D;
layout(binding = 19) uniform isampler2DMS texSInt2DMS;
#ifndef VULKAN
layout (binding = RD_UINT_1D_BINDING) uniform usampler1D texSInt1D;
layout (binding = RD_UINT_2D_BINDING) uniform usampler2D texSInt2D;
layout (binding = RD_UINT_RECT_BINDING) uniform usampler2DRect texSInt2DRect;
layout (binding = RD_UINT_BUFFER_BINDING) uniform usamplerBuffer texSIntBuffer;
layout (binding = RD_UINT_2DMS_BINDING) uniform usampler2DMS texSInt2DMS;
#endif
These resources are bound sparsely with the appropriate type for the current texture. With a couple of exceptions there will only be one texture bound at any one time.
These resources are bound sparsely with the appropriate type for the current texture. With a couple of exceptions there will only be one texture bound at any one time. Different APIs have different texture type matching requirements, so e.g. OpenGL has separate bindings for array and non-array texures, which will be reflected in the different ``RD_TextureType`` return values.
When a cubemap texture is bound, it is bound both to the 2D Array as well as the Cube Array. If a depth-stencil texture has both components, the relevant depth and stencil resources will both be bound at once.
@@ -351,7 +336,7 @@ Usually the float textures are used, but for unsigned and signed integer formats
As with the samplers, these textures are bound by slot and not by name, so while you are free to name the variables as you wish, you must bind them explicitly to the slots listed here.
.. note::
YUV textures may have additional planes bound as separate textures - for D3D this is ``texDisplayYUVArray`` and for Vulkan it's ``texYUV`` above. Whether to use these planes or not is specified in the texture dimension variables.
YUV textures may have additional planes bound as separate textures - for D3D this is ``texDisplayYUVArray`` and for Vulkan it's ``texYUVArray`` above. Whether to use these planes or not is specified in the texture dimension variables.
See Also
--------