Add feature to globally hook processes to inject indirectly.

* This allows you to hook into processes that are difficult to launch
  directly with the existing functionality in RenderDoc.
* This is rather risky, as it modifies the AppInit_DLLs registry key to
  inject a small shim dll that checks for the desired process and injects
  the full renderdoc.dll. If that registry key got left, or if there was
  some incompatibility with the shim dll, you could have problems. It
  should only ever be used as a last resort if there's no other way to
  capture.
This commit is contained in:
baldurk
2014-10-05 13:57:11 +01:00
parent a6d68e5598
commit 9460fbd970
29 changed files with 1292 additions and 110 deletions
+16
View File
@@ -289,5 +289,21 @@
</content>
</section>
<section address="FAQ13">
<title>I can't launch my program for capture directly. Can I capture it anyway?</title>
<content>
<para>
There is an option for capturing programs using RenderDoc where you can't easily set
up a direct launch of the process.
</para>
<para>
More details can be found in the
<link xlink:href="D1612D25-C8BA-4349-9CE2-1E57D60F98C5">capture options</link> page which
details how to use it, however you should take care to read the warnings! This option
isn't without its risks, so you need to be sure you know what you're doing before
using it. It should always be used as a last resort when there is no other option.
</para>
</content>
</section>
</developerConceptualDocument>
</topic>
+2
View File
@@ -126,6 +126,8 @@
<HelpKeywords>
<HelpKeyword index="K" term="Window, Capture Window" />
<HelpKeyword index="K" term="Window, Attach to Process" />
<HelpKeyword index="K" term="Capture All Processes" />
<HelpKeyword index="K" term="Global Hook" />
</HelpKeywords>
</Topic>
<Topic id="16D2B42E-65B0-40D1-AB91-AD2B156F5DA0" visible="True" title="Event Browser">
+36
View File
@@ -201,6 +201,42 @@
</content>
</section>
<section address="globalhook">
<title>Global Process Hook</title>
<content>
<alert class="caution">
<para>This option is risky and should not be used lightly. Know what you're doing and
use it as a last resort.</para>
</alert>
<para>To expose this option you have to enable it in
<link xlink:href="B1826EEE-2ED1-44E4-8202-37CD8B3FEEB5">the settings</link>,
to prevent it being used accidentally.</para>
<para>When you've entered a path, or filename, in the executable text at the top
of the window, this option will then insert a global hook that causes <legacyBold>every</legacyBold> new
process created to load a very small shim dll.</para>
<para>The shim dll will load, create a thread that checks to see if the process
matches the path or filename specified, and then unload. If the process matches
it will also inject RenderDoc and capturing will continue as normal. At this
point you should <legacyItalic>first disable the global hook</legacyItalic>, then
you can use the 'Attach to running instance' menu option to continue as normal.</para>
<para>RenderDoc implements this behaviour by modifying the
<externalLink>
<linkText>AppInit_DLLs</linkText>
<linkAlternateText>AppInit_DLLs on MSDN</linkAlternateText>
<linkUri>http://support2.microsoft.com/kb/197571</linkUri>
</externalLink> registry key to reference RenderDoc's dlls. This is not a particularly safe
method but it's the only reliable method to do what we want. The shim dll is
deliberately made as small and thin as possible, referencing only kernel32.dll, to
minimise any risks.
</para>
<para>If RenderDoc crashes or something otherwise goes wrong while these registry
keys are modified, the shim dll will continue to be injected into every process which
is certainly not desireable. Should anything go wrong, RenderDoc writes a .reg
file that restores the registry to its previous state in %TEMP%.</para>
<para>Again, <legacyBold>this method should be a last resort</legacyBold>. Given
the risks you should always try to capture directly in some way before trying this.</para>
</content>
</section>
<relatedTopics>
<link xlink:href="f2f4fbfa-6127-4831-84e7-07309bd85908" />
<link xlink:href="14048aef-0053-4e38-98cd-541f4d15d12e" />
+9
View File
@@ -96,6 +96,15 @@
recommended that you manually check for updates as new versions will be made available regularly with bugfixes.</para></entry>
<entry><para>Enabled</para></entry>
</row>
<row>
<entry><para>Allow global process hooking</para></entry>
<entry><para>This option enables the functionality allowing capturing of programs that aren't
launched directly from RenderDoc, but are launched from somewhere else.</para>
<para>This option <legacyBold>can be dangerous</legacyBold> which is why you have to deliberately enable it
here. Be careful when using this and only do so when necessary - more details can be found in the
<link xlink:href="D1612D25-C8BA-4349-9CE2-1E57D60F98C5">capture options</link> page.</para></entry>
<entry><para>Disabled</para></entry>
</row>
</table>
</content>