renderdoc/docs/HowTo/HowToCustomShader.aml

<?xml version="1.0" encoding="utf-8"?>
<topic id="a00964a1-733f-4288-a79e-c28238bc6018" revisionNumber="1">
  <developerConceptualDocument xmlns="http://ddue.schemas.microsoft.com/authoring/2003/5" xmlns:xlink="http://www.w3.org/1999/xlink">
    <introduction>
      <para>This page details how to set up a custom shader for visualisation. This can be used
      in the <link xlink:href="2c540574-0b81-4a40-8119-ba0283fddf41" /> to unpack or decode complex
      formats, or simply apply a custom complex transformation to the image beyond the default controls.</para>
    </introduction>
    <section address="intro">
      <title>Introduction</title>
      <content>
        <para>The basic process of setting up the custom shader involves writing a .hlsl file that
        will be compiled and used by RenderDoc.</para>
        <para>There are several special global variables that can be specified and will
        be filled in with values by RenderDoc.</para>
        <para>Your pixel shader defines an operation that transforms the raw value from the input
        texture into a value that will then be displayed by the texture viewer. The usual texture
        viewer controls for range adaption and channels will still be available on the resulting
        texture.</para>
        <para>Multisampled textures will be resolved before being passed to your function.
        Depth and stencil textures will be bound separately and passed as multisampled resources.</para>
        <para>To set up your shader, it's recommended that you use the UI defined in the documentation for the
        <link xlink:href="2c540574-0b81-4a40-8119-ba0283fddf41" />, but you can manually create a .hlsl file in
        <codeInline>%APPDATA%\RenderDoc\</codeInline>. The file must contain an hlsl function main() that
        returns float4, and uses any of the below inputs. These shaders are loaded when RenderDoc
        loads a logfile, and RenderDoc watches for any changes to the files (either externally or in
        the shader editor in RenderDoc) and automatically reloads them.</para>
      </content>
    </section>
    <section address="inputs">
      <title>Predefined inputs</title>
      <content>
        <para>There are several pre-defined inputs that can either be taken as parameters to
        the shader entry point, or defined as global variables with a particular name that will
        then be filled in. There are also definitions for the different type of input textures.</para>
        
<alert class="caution">
  <para>Type and capitalisation is important for these variables, so ensure you use the right
  declaration!</para>
</alert>

<table>
  <tableHeader>
    <row>
      <entry><para>Input</para></entry>
      <entry><para>Explanation</para></entry>
    </row>
  </tableHeader>
  <row>
    <entry><para>UV co-ordinates</para></entry>
    <entry>
<code language="hlsl">
float4 main(<markup><u>float4 pos : SV_Position</u></markup>, <markup><u>float4 uv : TEXCOORD0</u></markup>) : SV_Target0
{
  return 1;
}
</code>
<para>This input is defined as two parameters to the shader entry point. The first defines the usual
SV_Position system semantic, and the first two components of the second TEXCOORD0 parameter gives UV
co-ordinates from 0 to 1 in each dimension over the size of the texture (or texture slice).</para>
    
<alert class="note">
  <para>You must bind these parameters like this in this order to ensure the linkage with the
  vertex shader matches.</para>
</alert>
</entry>
  </row>
  <row>
    <entry><para>Texture dimensions</para></entry>
    <entry>
<code language="hlsl">
uint4 RENDERDOC_TexDim;
</code>
<para>This variable will be filled out with the following values:</para>
    
<list class="bullet">
  <listItem><para><codeInline>.x = </codeInline> Width</para></listItem>
  <listItem><para><codeInline>.y = </codeInline> Height (if 2D or 3D)</para></listItem>
  <listItem><para><codeInline>.z = </codeInline> Depth if 3D or array size if an array</para></listItem>
  <listItem><para><codeInline>.w = </codeInline> Number of mip levels</para></listItem>
</list>

</entry>
  </row>
  <row>
    <entry><para>Selected Mip level</para></entry>
    <entry>
<code language="hlsl">
uint RENDERDOC_SelectedMip;
</code>
<para>This variable will be filled out with the selected mip level in the UI.</para></entry>
  </row>
  <row>
    <entry><para>Current texture type</para></entry>
    <entry>
<code language="hlsl">
uint RENDERDOC_TextureType;
</code>
<para>This variable will be set to a given integer value, depending on the type of
    the current texture being displayed. This can be used to sample from the correct resource.</para>
    
<list class="ordered">
  <listItem><para>1D texture</para></listItem>
  <listItem><para>2D texture</para></listItem>
  <listItem><para>3D texture</para></listItem>
  <listItem><para>Depth</para></listItem>
  <listItem><para>Depth + Stencil</para></listItem>
  <listItem><para>Depth (Multisampled)</para></listItem>
  <listItem><para>Depth + Stencil (Multisampled)</para></listItem>
  <listItem><para>Cubemap</para></listItem>
  <listItem><para>2D texture (Multisampled)</para></listItem>
</list>

</entry>
  </row>
  <row>
    <entry><para>Samplers</para></entry>
    <entry>
<code language="hlsl">
SamplerState pointSampler : register(s0);
SamplerState linearSampler : register(s1);
</code>
<para>These samplers are provided to allow you to sample from the resource
    as opposed to doing straight loads. They are bound by slot and not by variable name -
    so this means you can name them as you wish but you must specify the register binding
    explicitly.</para></entry>
  </row>
  <row>
    <entry><para>Resources</para></entry>
    <entry>
<code language="hlsl">
Texture1DArray&lt;float4&gt; texDisplayTex1DArray : register(t1);
Texture2DArray&lt;float4&gt; texDisplayTex2DArray : register(t2);
Texture3D&lt;float4&gt; texDisplayTex3D : register(t3);
Texture2DArray&lt;float2&gt; texDisplayTexDepthArray : register(t4);
Texture2DArray&lt;uint2&gt; texDisplayTexStencilArray : register(t5);
Texture2DMSArray&lt;float2&gt; texDisplayTexDepthMSArray : register(t6);
Texture2DMSArray&lt;uint2&gt; texDisplayTexStencilMSArray : register(t7);
Texture2DArray&lt;float4&gt; texDisplayTexCubeArray : register(t8);
Texture2DMSArray&lt;float4&gt; texDisplayTex2DMSArray : register(t9);

Texture1DArray&lt;uint4&gt; texDisplayUIntTex1DArray : register(t11);
Texture2DArray&lt;uint4&gt; texDisplayUIntTex2DArray : register(t12);
Texture3D&lt;uint4&gt; texDisplayUIntTex3D : register(t13);
Texture2DMSArray&lt;uint4&gt; texDisplayUIntTex2DMSArray : register(t19);

Texture1DArray&lt;int4&gt; texDisplayIntTex1DArray : register(t21);
Texture2DArray&lt;int4&gt; texDisplayIntTex2DArray : register(t22);
Texture3D&lt;int4&gt; texDisplayIntTex3D : register(t23);
Texture2DMSArray&lt;int4&gt; texDisplayIntTex2DMSArray : register(t29);
</code>
<para>These resources are bound sparsely with the appropriate type for the
    current texture. With a couple of exceptions there will only be one texture bound
    at any one time.</para>
    <para>When a cubemap texture is bound, it is bound both to the 2D Array as well
    as the Cube Array. If a depth-stencil texture has both components, the relevant depth and
    stencil resources will both be bound at once.</para>
    <para>To determine which resource to sample from you can use the RENDERDOC_TexType variable
    above.</para>
    <para>Usually the float textures are used, but for unsigned and signed integer formats,
    the relevant integer resources are used.</para>
    <para>As with the samplers, these textures are bound by slot and not by name, so while
    you are free to name the variables as you wish, you must bind them explicitly to the
    slots listed here.</para></entry>
  </row>
</table>

      </content>
    </section>
    <relatedTopics>
      <link xlink:href="2c540574-0b81-4a40-8119-ba0283fddf41" />
    </relatedTopics>
  </developerConceptualDocument>
</topic>