Files
rustdesk/libs
Mariano Abad 0d49ead0c3 fix(drm): turn a captured frame only when the plane did not rotate it (#16345)
* fix(drm): turn a captured frame only when the plane did not rotate it

A compositor rotates an output either in hardware, setting the primary
plane's `rotation` property, or in software, drawing the scanout already
turned. `wl_output` cannot tell the two apart, and `frame_transform`
guessed: 90/270 turned, 180 left alone, which is right for i915 + mutter
(rotate-180 in hardware, scanout upright) and wrong wherever the
compositor drew the scanout turned, where 180 comes out upside down.

Measured before changing it. virtio-gpu (no rotation property, mutter 46):
the scanout dump at 180 is the desktop upside down, at 90 and 270 the
logical desktop turned inside the native mode framebuffer. i915 + mutter
(GNOME 50) at 180: the plane reports rotate-180 and the dump is upright,
identical to the unrotated one. amdgpu + KWin 6 at 180: the plane stays at
rotate-0 and the dump is upside down.

The producer reads `drmtap_plane_rotation()` (libdrmtap 0.5.8, optional
symbol) right after each grab and stamps it on the frame, in the dma-buf
descriptor and in the CPU frame header, both `serde(default)` so an older
producer still parses. A plane that rotated scans out the logical desktop
whatever the output transform says: the compositor programs it from the
CRTC transform, which also folds in the connector's panel orientation that
`wl_output` never carries, so the consumer leaves such a frame alone. A
plane at rotate-0 scanned out what the compositor drew, and the frame is
turned by the whole output transform. `None` (a library before 0.5.8, or
nothing it could read) keeps the previous rule, so nothing changes until
the library says otherwise. The session is still sized from the output
transform: a plane that rotated 90 in hardware hands over the portrait
scanout those dims already name.

* build: pin libdrmtap 0.5.8 (rustdesk-org/libdrmtap 95d4d7454)

The rotation fix reads drmtap_plane_rotation(), added in libdrmtap 0.5.8. The mirror carries that commit since 2026-09-25, so the pin moves there.

* drm: pin the decoding of a frame from a producer older than the plane rotation

The root service and the --server are upgraded separately, so a frame
header or a dma-buf descriptor without `plane_rotation` must still
decode, as None, which keeps the previous rule. The tests built both
messages with the field set to None; this one decodes payloads that omit
it, and one that carries it (greptile on #16345). Serde already reads a
missing Option as None, so what the test pins is the wire name and the
None on absence, for both messages.

Mutation: renaming the field on the wire, in either message, turns it red.

* drm: take the plane rotation right after the grab, not after the copies

`drmtap_plane_rotation()` reads the rotation property of the plane the
last grab came from at the moment it is called. The cpu path asked only
after `DrmReader::grab()` had copied the frame into its buffer and the
producer had copied it again into the message, so on a large frame the
compositor had time to change the plane and the frame went out with the
rotation of the next one; `frame_transform` turns that frame by it with
nothing to absorb the skew (zhou, review of 26-sep).

`DrmReader` now reads the rotation right after a successful
`drmtap_grab_mapped` or `drmtap_grab_desc`, before any copy or release,
and `plane_rotation()` returns that snapshot, so both paths stamp a frame
with the rotation its own grab saw.

Test: a_frame_keeps_the_rotation_its_plane_had_when_it_was_grabbed, in
`plane_rotation_tests` (which CI already runs), drives the reader through
a fake library whose frame release turns the plane: the rotation read at
the grab survives on the cpu grab and on the dma-buf export. Mutations:
no snapshot on the cpu grab, none on the export, and the cpu snapshot
taken after the release, each turn it red.
2026-09-28 15:54:55 +08:00
..
2026-09-23 17:30:22 +08:00