Linux Window System Integration as a Layer (almost)

Linux has a lot of window systems. If you wind the clock back 10 years or so, there was only one: X11. Now, we have X11, Wayland, Mir, Android, and several other one-off window systems that pop up from time to time. Trying to support all these window systems is annoying because there are a lot of them and because window system issues have this way of weaving themselves throughout your driver in weird ways. One of the holy grails of window system integration is in trying to get it out of the driver. Android almost achieves this by having most of the interesting bits in a wrapper EGL layer and the drivers talk to the Android window system through the ANativeWindow struct. For X11 and Wayland, however, we’re not really in the position of being able to mandate an interface the drivers must use so everything is defined as GLX and EGL extensions and the implementation has to live in the individual drivers. This leaves everyone with having to carry around full GLX and EGL stacks that support all of the window systems.

With Vulkan, some work has already been done to start to solve this problem. In the world of VR, every vendor (Valve, Oculus, etc.) have their own VR compositors and they need some way to talk to the VR app running in side of them. To solve this, we at Khronos have developed a set of Vulkan extensions for sharing images, memory objects, and synchronization primitives between processes. The result is that you can implement an entire VR compositor with just those Vulkan extensions and no custom driver code. Unfortunately, these extensions come with some fairly heavy-handed limitations on driver version matching which prevent them from being suitable for more general-purpose compositors such as X11 and Wayland.

In the last few months, I’ve been working on a bunch of different window system integration items related to our efforts to use DRM format modifiers throughout the stack. As part of that, we needed to plumb DRM format modifiers for Vulkan WSI. The Vulkan WSI code, as it stood, wasn’t really ready for such a change. It was a fairly ad-hock abstraction created when we pulled it out of the Intel driver and into common code and it only got messier when prime support was added to radv.

Before making even more of a mess of things by adding modifiers support, it was time to clean things up a bit. A few weeks ago, Dave Airlie and I were talking on IRC and we both started hacking away at trying to clean up the interface. The result was a series of patches which landed today which reworks window system integration to look as much like a Vulkan layer as possible. Instead of using an ad-hock interface, we’ve tried to re-use as much of the Vulkan API as possible.

Previously, WSI image creation used a special WSI-specific function call which would create an image and allocate memory at the same time. To replace this, I added some new chain-in structs which look and feel like a Vulkan extension even though there’s nothing official. This pseudo-extension adds a chain-in struct for vkCreateImage that lets the WSI code create legacy window system images which may have additional tiling and alignment constraints. It also adds a chain-in for vkAllocateMemory which lets the WSI code enable GEM implicit synchronization. Between these two, we can now create “legacy” window system images with just vkCreateImage and vkAllocateMemory.

For prime support, we instead create a vanilla VkImage (without using the special chain-ins) for the WSI image and create a VkBuffer for the linear shadow copy. We then create a set of command buffers (one per queue family) which perform a vkCmdCopyImageToBuffer to blit from the image to the linear shadow copy. By doing this, the driver becomes completely unaware that prime is even happening and it just thinks that it’s rendering to a regular old VkImage.

With this refactoring done, the WSI code now uses entirely standard Vulkan entrypoints for image creation, memory allocation, and memory import/export. There are a few assumptions we have to make such as assuming that you can do vkCmdCopyImageToBuffer on an image in the VK_IMAGE_LAYOUT_PRESENT_SRC_KHR layout but the assumptions we have to make are not onerous. The functions provided to the driver by the WSI code also now look a lot more like the Vulkan entrypoints they are used to implement and the WSI entrypoints in the drivers are basically just wrappers around common code to make sure they end up in the dispatch table. In short, the WSI code in mesa is starting to look very much like a Vulkan layer.

Going forward, however, things are going to look a little different. With DRM format modifiers, we are in the process of drafting a Vulkan extension to expose DRM format modifers support in the driver for both image creation and import. Once that extension is finished, it will take the place of the WSI pseud-extension for image creation at least on Intel. Similarly, we’re also working on plumbing explicit synchronization through the X server using Linux sync files. Once this is done, we can drop the implicit synchronization portion of the pseudo-extension and start using VK_KHR_external_semaphore_fd instead.

One day, I would love to see Linux window-system integration pulled entirely into a proper Vulkan layer that gets shared by mesa and closed-source drivers alike but we’re not quite there yet. In the mean time, we have a much cleaner abstraction inside of mesa and the modifiers work can start moving forward again.