This aricle is NOT BY ME, but found originally posted on http://ktown.kde.org/~fredrik/composite_howto.html and now unavailable. If anyone does know who wrote it, I'd love to give correct attribution for it.

The article targets the Qt Widget toolkit but should make sense for every widget toolkit that exposes some basic X11 protocol. It's been cloned here (ignoring formatting changes and minor corrections) from http://trinity.netcat.be/blog/composite-tutorial (that webserver is also gone) to prevent its loss.

Composite tutorial

Index

Introduction
1. About this tutorial
2. When a windows contents are made available by Composite
3. Reasons for using Xrender to access the contents
Checking if the server supports the composite extension
Redirecting windows to offscreen pixmaps
Referencing the window contents
1. Getting information about the window
2. Creating a Render picture so we can access the window contents
3. Handling shaped windows correctly
Drawing the window on a QWidget or a QPixmap
Tracking damage and other changes to a window
1. Intercepting X events as they're received
Advanced concepts
1. Using a transformation matrix: scaling and rotating
2. Preventing the backing pixmap from being freed when the window is hidden/destroyed
3. Converting the window contents to a QImage
4. Differences between automatic and manual redirection
Credits
References

Introduction

About this tutorial

Now that composite has appeared in an X.Org release, a lot of people will be interested in using it, not just from a user point of view for transparency and eye candy, but also from a developers point of view, for accessing the contents of covered windows.

This tutorial is about using Composite and the Xrender extension for doing the latter. Tracking changes to window contents with Xdamage is tightly coupled to accessing window contents, so that's also covered in this tutorial.

This tutorial is mainly aimed at those that are interested in using Composite for providing thumbnails in a desktop pager, or for those working on Exposé like features. But the concepts introduced here are applicable to Composite managers as well. This tutorial is not meant to be the one and only reference you'll ever need for everything having to do with the Composite extension however, but I've tried to cover all aspects of its usage for the above mentioned purposes. It should also help you avoid all the pitfalls you're likely to run into when using the Composite extension.

You don't need to have any previous experience from using X directly from within a Qt application, but it helps if you have a general idea on how X works, and how an application communicates with an X server. Previous experience with Xlib programming helps too.

When a windows contents are made available by Composite

Before we begin I feel I need to clear up a misconception about when the contents of a window is made available by composite. A backing pixmap is allocated when the window is shown, and is deallocated when it becomes hidden. What this means in other words is that when a window is minimized, or not on the current desktop, the window contents aren't available.

There are basically two reasons why composite works like this. The first is to minimize the amount of video ram needed at a given time by the backing store. The second is that with the traditional X design, drawing on a hidden window is effectively a NOOP. Clever applications and toolkits know this, so they'll never try to draw on a minimized window anyway.

This probably comes as a dissapointment nonetheless to those that are hoping that composite would enable them to e.g. hover over a taskbar button, and get a tooltip with a thumbnail of a minimized window or a window on another desktop (to see if the window has been updated).

The best thing a taskbar/pager that wants to do this can do is to keep a cached thumbnail of the contents of the window before it was unmapped. Composite actually provides an easy way of doing that, as explained in this tutorial.

Note that while the contents of a minimized window will never be available, the contents of a window on an inactive desktop may be, depending on the WM design. If the WM uses a virtual root window for each desktop, and don't unmap the inactive ones, then the contents of those windows on those desktops will still be available. KWin and Metacity aren't designed like that however.

Reasons for using Xrender to access the contents

I'll also briefly explain the motivation for using Xrender to access the window contents. It is perfectly possible to create a GC for a window and use XCopyArea() to copy the contents of the window if you want to use the core protocol, but since the Composite extension exposes new visuals (ones with alpha channels e.g.), there's no guarantee that the format of the source drawable will match that of the destination. With the core protocol that situation will result in a match error, something that won't happen with the Xrender extension.

In addition the core protocol has no understanding of alpha channels, which means that it can't composite windows that use the new ARGB visual. When the source and destination have the same format, there's also no performance advantage to using the core protocol as of X11R6.8. That release is also the first to support the new Composite extension.

So in conclusion there are no drawbacks, and only advantages to choosing Xrender over the core protocol for these operations.

Checking if the X server supports the composite extension

The first you thing you need to do is to do a runtime check to make sure the X server you're connected to actually supports the composite extension. You'll need a compile time check too of course, but that'll only tell you that the composite lib is available, not that the X server the application will be using has the server side bits.

Before you can make any calls to Xlib functions, you need to obtain a pointer from Qt to the Display struct, which is the first argument to each Xlib function. The Display struct contains information about the connection to the X server, such as the socket number. This pointer is obtained by calling the static function QPaintDevice::x11AppDisplay(), If you need a pointer to the Display struct from inside a class that inherits QPaintDevice, you can call x11Display() instead.

Display *dpy = QPaintDevice::x11AppDisplay();

Note that the KApplication object must be created before asking Qt for the Display pointer, since it won't be initialized until QApplication has established a connection to the X server. Once you have a pointer to the Display struct we can proceed with the runtime check for the extension:

bool hasNamePixmap = false;
int event_base, error_base;
if ( XCompositeQueryExtension( dpy, &event_base, &error_base ) )
{
    // If we get here the server supports the extension
    
    int major = 0, minor = 2; // The highest version we support
    XCompositeQueryVersion( dpy, &major, &minor );	
    
    // major and minor will now contain the highest version the server supports.
    // The protocol specifies that the returned version will never be higher
    // then the one requested. Version 0.2 is the first version to have the
    // XCompositeNameWindowPixmap() request.
    if ( major > 0 || minor >= 2 )
        hasNamePixmap = true;
}

Redirecting all toplevel windows to offscreen pixmaps

Once you've made sure the X server supports the Composite extension, you need to make sure that the contents of the window you want to grab are available in a backing pixmap.

This is accomplished by calling XCompositeRedirectSubwindows() once for each root window, specifying that you want automatic redirection:

for ( int i = 0; i < ScreenCount( dpy ); i++ )
    XCompositeRedirectSubwindows( dpy, RootWindow( dpy, i ),
CompositeRedirectAutomatic );

This will redirect all current and future toplevel windows on each screen to offscreen storage. There will only be more than one screen on a multihead (but not on a Xinerama) system.

You don't need to worry about unredirecting the windows, since this will be done automatically when your application shuts down. Also if a composite manager has already redirected the windows, this call will be a NOOP, so you don't need to worry about this disrupting e.g. xcompmgr.

The windows themselves are oblivious to the fact that they've been redirected to offscreen storage, and no modifications need to be made to existing applications for this to work. Both from the users point of view, and from that of the applications, everything will just continue to work as it did before the windows were redirected.

Note that if you know you won't be dealing with more than one window, it's better to just redirect the window you're interested in using XCompositeRedirectWindow(). You can then unredirect it when you're done (using XCompositeUnredirectWindow()).

Referencing the window contents

If you want access to the contents of a window, you must first know its ID. It might be tempting to create a QPixmap and simply copy the window contents to it, but remember that the window is already a pixmap, and if you want to store the contents of multiple windows, creating a QPixmap for each one of them would mean that each window ends up being stored twice in video RAM. So in other words that isn't a good idea. If you just want to store thumbnails of each window, it might be a good idea to cache those in QPixmaps however.

Getting information about the window

Okay so we know the window ID, but we need to find out some other things about the window, such as its size, the format of the pixels in the backing pixmap, and whether the window has an alpha channel or not.

It turns out we can find out most of these things by calling XGetWindowAttributes():

// We need to find out some things about the window, such as its size, its position
// on the screen, and the format of the pixel data
XWindowAttributes attr;
XGetWindowAttributes( dpy, wId, &attr );

XGetWindowAttributes is a synchronous call (it blocks), so you don't want to call this function too often. In a real world application you'll also need to check the return value, since due to the asynchronous nature of X, the window may have been destroyed by the time the call is made.

Once XGetWindowAttributes() has filled in the members of the XWindowAttributes struct, we'll extract the data we need:

XRenderPictFormat *format = XRenderFindVisualFormat( dpy, attr.visual );
bool hasAlpha             = ( format->type == PictTypeDirect && format->direct.alphaMask );
int x                     = attr.x;
int y                     = attr.y;
int width                 = attr.width;
int height                = attr.height;

Creating a Render picture so we can access the window contents

Okay, so now we know the size of the window, the pixel format, and whether the window has an alpha channel or not. What we need to do now is to create an XRender Picture for the window, which we'll need to draw it with the Render extension. A picture is a basically a handle to a server side struct with some additional information about a drawable (in this case a window), such as its format, which clipping region should be used when drawing it (if any), whether it should be tiled etc.

Creating a picture for a drawable is done by calling XRenderCreatePicture(). Unlike XGetWindowAttributes() this function isn't synchronous, so it won't return an error code if the window doesn't exist. Instead an X error will be sent to the application when the X server processes the request, and the returned picture handle will be invalid (which will result in further X errors if it's used in subsequent function calls).

Responding to those errors correctly is quite complex, and beyond the scope of this tutorial. But the errors are harmless and will only result in an error message being printed on the console. When your application returns to the event loop you'll recieve a message about the window having been deleted.

// Create a Render picture so we can reference the window contents.
// We need to set the subwindow mode to IncludeInferiors, otherwise child widgets
// in the window won't be included when we draw it, which is not what we want.
XRenderPictureAttributes pa;
pa.subwindow_mode = IncludeInferiors; // Don't clip child widgets

Picture picture = XRenderCreatePicture( dpy, wId, format, CPSubwindowMode, &pa );

To avoid having to go through each of these steps each time you want to draw the window, you'll probably want to stick this information in an class for quick access. You could also put a draw() method in the same class for drawing the window on a QPixmap/QWidget.

Handling shaped windows correctly

A windows backing pixmap is always rectangular, but if the window has a non-rectangular shape we don't want to end up copying pixels that aren't a part of the window when we draw it (those pixels are undefined). To avoid doing that we'll set the clip region for the picture to the windows shape region. Doing this requires using the XFixes extension, which you'll need to query for, just like you did with composite.

// Create a copy of the bounding region for the window
XserverRegion region = XFixesCreateRegionFromWindow( dpy, wId, WindowRegionBounding );

// The region is relative to the screen, not the window, so we need to offset
// it with the windows position
XFixesTranslateRegion( dpy, region, -x, -y );
XFixesSetPictureClipRegion( dpy, picture, 0, 0, region );
XFixesDestroyRegion( dpy, region );

It might be useful to know that when you draw a pixmap with Xrender, you provide pictures both for the source and the destination drawables, and both the source and destination pictures can have clip regions set. So if you're not planning on scaling the window when you draw it, you could set the clip region in the destination picture instead.

Don't forget that since the region is only a copy, you'll need to update it when the window is resized or when the window shape is changed.

The XShape extension can provide notifications when the window shape changes, but it will only do this if you explicitly ask it to. Telling XShape to send such notifications is done by calling XShapeSelectInput():

XShapeSelectInput( dpy, wId, ShapeNotifyMask );

See the section about intercepting X events for information on how to recieve the actual events.

Drawing the window on a QWidget or QPixmap

We now have all the information we need in order to be able to draw the window using the Xrender extension, and we've created and prepared a source picture for the window for this purpose.

The Xrender function we'll use to draw the window is XRenderComposite(), which is defined like this in the Xrender header file:

void XRenderComposite (Display   *dpy,
                       int       op,
                       Picture   src,
                       Picture   mask,
                       Picture   dst,
                       int       src_x,
                       int       src_y,
                       int       mask_x,
                       int       mask_y,
                       int       dst_x,
                       int       dst_y,
                       unsigned int  width,
                       unsigned int  height);

As you can see this function takes a source picture, a destination picture and an optional mask picture. In our case we have need for a mask, but we will need a destination picture however.

We want to draw the window on a QWidget or a QPixmap, and it turns out that objects of both these types already have render pictures (if Qt was built with Xft/Xrender support). The picture is accessed by calling the x11RenderHandle() method in the QWidget or QPixmap.

Another important parameter for XRenderComposite() is the second, op, which specifies how the source and destination pixels should be combined. For our purposes there are only two render operations that are of interest - PictOpSrc and PictOpOver.

PictOpSrc specifies that the destination pixels should be replaced with the source pixels (dst = src), including the alpha values. PictOpOver corresponds to the Porter/Duff Over operator, which specifies that Xrender should use the alpha values in the source pixels to blend them with the destination pixels (dst = src Over dst).

So PictOpSrc won't blend the window, while PictOpOver will. PictOpSrc is faster, and is almost always guaranteed to be accelerated, so when the window doesn't have an alpha channel we'll want to use that. When it has an alpha channel we'll want to use PictOpOver.

In the following example dest must be a QWidget or a QPixmap. destX and destY are the X and Y coordinates in the widget or pixmap where you want the window to be drawn.

// [Fill the destination widget/pixmap with whatever you want to use as a background here]
XRenderComposite( dpy, hasAlpha ? PictOpOver : PictOpSrc, picture, None,
                  dest.x11RenderHandle(), 0, 0, 0, 0, destX, destY, width, height );

Tracking damage and other changes to a window

If you're only interested in creating a one time snapshot of a window, and you're not interested in updating the snapshot when the window changes, you can skip this section.

The first thing we need to do if we want to track damage to a window is to query the X server for the damage extension, and this time we need to save the event base for later. Read on to find out why.

int damage_event, damage_error; // The event base is important here
XDamageQueryExtension( dpy, &damage_event, &damage_error );

Once we've made sure the X server supports the damage extension, we need to create a damage handle for each window we're interested in. There are a number of ways Xdamage can report changes to the window, in this case we'll specify that we want an event whenever the window state changes from not damaged to damaged. The allocated handle must be destroyed by calling XDamageDestroy() when damage events for the window are no longer needed.

// Create a damage handle for the window, and specify that we want an event whenever the
// damage state changes from not damaged to damaged.
Damage damage = XDamageCreate( dpy, wId, XDamageReportNonEmpty );

One thing you'll notice here is that unlike most extensions that provide notifications, Xdamage provides damage notification objects rather than exposing an XDamageSelectInput() request. You'll recall from the section about shaped windows that the XShape extension provides an XShapeSelectInput() request to request shape change notifications for a window.

Another interesting thing to note about Xdamage is that it doesn't just track damage to windows, it can also track damage to pixmaps, if you want to use it for that.

Intercepting X events as they're received

X continually sends events to the application over the network socket, where they're demarshalled and inserted into an event queue. An application pulls the events out of the queue one at a time by calling XNextEvent().

In a Qt application all this is handled by QApplication, but in this particular case we want to take a peek at each received X event before QApplication processess it.

It turns out there's a way to do that - two ways in fact. In a Qt application this is done by reimplementing QApplication::x11EventFilter( XEvent * ), which is called each time an event is pulled out of the queue, but before it's processed by Qt. We even have a choice of whether we should swallow the event (by returning true), or telling Qt it should go ahead and process the event (by returning false).

In a KDE application we also have the option of calling KApplication::installX11EventFilter( QWidget * ), which tells KApplication to forward each received X event to the x11Event( XEvent * ) member function in the widget you specify.

Now I mentioned before that it was important to save the event base for the damage extension. It's now time to explain what you need it for. Each X event has a unique number that identifies that event, with the numbers for the events in the core protocol starting with the base number zero.

Since there can be any number of X extensions and each extension can add any number of events, the base number for the events provided by an extension depends on the X implementation, and must be obtained from the X server at runtime.

The event base must be added to the constants identifying the events from the extension to compute the actual event number. You'll see how this works in the example below.

For each X event there's a corresponding structure, and XEvent is union of all possible event structures. The first member, which is common for all the structs, is type which contains the event number, which tells us which type of event the struct contains. The XEvent struct must be cast to the appropriate struct matching the event, e.g. if type is ConfigureNotify, the XEvent should be cast to XConfigureEvent.

Here's a sample implementation of a function that receives X events, checks the type member, and processes damage, shape and configure events:

bool x11EventFilter( XEvent *event )
{
    if ( event->type == damage_event + XDamageNotify ) {
        XDamageNotifyEvent *e = reinterpret_cast<XDamageNotifyEvent*>( event );
        // e->drawable is the window ID of the damaged window
        // e->geometry is the geometry of the damaged window	
        // e->area     is the bounding rect for the damaged area	
        // e->damage   is the damage handle returned by XDamageCreate()
        
        // Subtract all the damage, repairing the window.
        XDamageSubtract( dpy, e->damage, None, None );
    }
    
    else if ( event->type == shape_event + ShapeNotify ) {
        XShapeEvent *e = reinterpret_cast<XShapeEvent*>( event );
        // It's probably safe to assume that the window shape region
        // is invalid at this point...	
    }
    
    else if ( event->type == ConfigureNotify ) {
        XConfigureEvent *e = &event->xconfigure;	
        // The windows size, position or Z index in the stacking
        // order has changed
    }
    
    return false;			
}

Keep in mind that there may be several events of the same type for the same window in the queue, so you should wait until you've processed all of them before taking any action. This is also important because there could be a DestroyNotify event for the window in the queue, after e.g. a damage event.

Note that in the above example all the damage is subtracted from the window, but the actual damage region is thrown away. In this example the damage region is retrieved from the damage object, and set as a clip region for the picture:

// Create an empty region
XserverRegion region = XFixesCreateRegion( dpy, 0, 0 );

// Copy the damage region to region, subtracting it from the windows' damage
XDamageSubtract( dpy, e->damage, None, region );

// Offset the region with the windows' position
XFixesTranslateRegion( dpy, region, e->geometry.x, e->geometry.y );

// Set the region as the clip region for the picture
XFixesSetPictureClipRegion( dpy, picture, 0, 0, region );

// Free the region
XFixesDestroyRegion( dpy, region );

This will result in only the damaged pixels being copied when the window is drawn, but this may not be an option if you're scaling the window as you're drawing it. Scaling the window is covered in the next section.

Advanced concepts

Using a tranformation matrix: scaling and rotating

If we want to create a thumbnail of a window we'll need to scale it, so I'll mention briefly how to do that with the render extension. Doing it with Xrender has the advantage that it's done server side, so there's no image transport involved. This is quite important, especially for remote X connections.

What we'll do is set a transformation matrix for the picture that'll cause the contents to be scaled to the size we want when it's drawn. XTransform works much in the same way as QWMatrix.

double scale = .5; // We'll scale the window to 50% of its original size

// Scaling matrix
XTransform xform = {{
    { XDoubleToFixed( 1 ), XDoubleToFixed( 0 ), XDoubleToFixed(     0 ) },
    { XDoubleToFixed( 0 ), XDoubleToFixed( 1 ), XDoubleToFixed(     0 ) },
    { XDoubleToFixed( 0 ), XDoubleToFixed( 0 ), XDoubleToFixed( scale ) }
}};

XRenderSetPictureTransform( dpy, picture, &xform );

Since XTransform is a projective transformation matrix, scaling isn't the only possible transformation. E.g. below is an example of a matrix that rotates the picture 30 degrees clockwise.

double angle = M_PI / 180 * 30; // 30 degrees
double sina = std::sin( angle );
double cosa = std::cos( angle );

// Rotation matrix
XTransform xform = {{
    { XDoubleToFixed(  cosa ), XDoubleToFixed( sina ), XDoubleToFixed( 0 ) },
    { XDoubleToFixed( -sina ), XDoubleToFixed( cosa ), XDoubleToFixed( 0 ) },
    { XDoubleToFixed(     0 ), XDoubleToFixed(    0 ), XDoubleToFixed( 1 ) }
}};

Note that the center of rotation is the upper left corner, not the center of the picture, so you'll need to offset the picture when you draw it to compensate for that.

Since the resulting image comes out looking rather jagged, you might want to use a filter when doing the transformation. All render implementations are required to support two filters - nearest neighbor and bilinear, but may support any number of filters, such as gaussian or even arbitrary convolution filters. XRenderQueryFilters() returns a list of all filters supported by the render implementation.

In this example we'll tell render to use a bilinear filter. When you use a filter you'll probably want to use PictOpOver as the render op, regardless of whether the source picture has an alpha channel or not, since the edges may end up having alpha values after the filter has been applied.

XRenderSetPictureFilter( dpy, picture, FilterBilinear, 0, 0 );

You should keep in mind that the transformation is applied in real time each time the picture is rendered, so there's some value in caching the resulting images.

Preventing the backing pixmap from being freed when the window is hidden/destroyed

If you want the window contents to still be available after the window has been destroyed, or after the window has been resized (but not yet redrawn), you can increment the backing pixmaps ref count to prevent it from being deallocated:

Pixmap windowPix = XCompositeNameWindowPixmap( dpy, wId );

As you can see this function also returns a handle to the backing pixmap the window is currently using, which is different from the window ID. The window ID, unlike the returned pixmap handle won't be valid after the window has been destroyed.

For this function to be useful you must call it before XRenderCreatePicture(), and substitute the wId parameter with windowPix in that call.

It's important to keep in mind that after doing this the picture will reference this particular backing pixmap, rather than whatever the current backing pixmap is for the window (which can always be referred to by the window ID).

The caveat is that when the window is resized, composite allocates a new backing pixmap for the window with the new size, and when that happens the picture will continue refer to the now stale backing pixmap containing the window image as it looked before it was resized.

You'll therefore have to track resize events for the window, and when it's resized you have to deref the backing pixmap (using XFreePixmap()) and destroy the picture, then call XCompositeNameWindowPixmap() to get a handle to the new backing pixmap, and recreate the picture.

To get resize events for the window you'll have to make this call to tell the X server you're interested in those types of events:

XSelectInput( dpy, wId, StructureNotifyMask );

See the section about intercepting X events in a KDE application, for information on how to receive the actual events. The event you're looking for isConfigureNotify.

Don't forget to deref the window pixmap when the window is destroyed, otherwise you'll leak the backing pixmap.

Converting the window contents to a QImage

In some cases it might be interesting to take a snapshot of a window and save it to disk. One way of doing that is to create a QPixmap of the same size as the window, copy the contents to the pixmap, and then use QPixmap::convertToImage(). The problem with this approach is that due to the limited support in Qt3 (and in Qt4 TP1) for alpha channels, the alpha channel in the window will be lost when doing this.

When the window has an alpha channel, the only option currently is to use XGetImage() to get the window contents into an XImage, and then manually convert the XImage to a QImage, using the information in the XRenderPictFormat. When doing this keep in mind that the ARGB32 visual is a premultiplied alpha format, while the QImage format isn't.

// Convert the window contents to an XImage
XImage *image = XGetImage( dpy, wId, 0, 0, width, height, AllPlanes, ZPixmap );

// [Convert image to a QImage]

XDestroyImage( image );

The exact details on how to do the XImage to QImage conversion is beyond the scope of this tutorial.

When doing the conversion you'll need to take the windows shape region into account, in case the window has a non-rectangular shape. XFixesFetchRegion() will return an XserverRegion in the form of an array of XRectangles. You can also use XShapeGetRectangles() if you want to avoid creating an XserverRegion. XGetImage(), XFixesFetchRegion() and XShapeGetRectangles() are all synchronous calls.

Differences between automatic and manual redirection

In this tutorial we've been using automatic redirection, since the goal has been to demonstrate how we can use Composite to access window contents, regardless of whether they're covered by other windows or not.

An application that isn't only interested in doing that, but also in constructing the screen image as it's presented to the user will want to use manual redirection. The difference between automatic and manual redirection is, that with manual redirection the window contents will be redirected to offscreen storage, but not automatically updated on the screen when they're modified.

If you're writing a composite manager you'll therefore want to use manual redirection. In order to create the screen presentation you'll also need to create a render picture for the root window, and draw the windows on it manually, taking the window hierarchy into account. Doing this you'll have total control over the presentation, which enables you to draw additional decorations, such drop shadows or lense glares. This is the topic of another tutorial however.

Credits

Thanks to Richard Moore and Chris Lee for their feedback.

fredrik's X11 Composite Tutorial