SharePoint Search 2007 – hacking the SiteData web service – Part I

When I started preparing this posting I realized that it would be too long, so I decided to split it into two parts – the first one being more introductory and explaining some aspects of the inner workings of the SharePoint search engine, and the second one concentrating on the actual implementation of the “hack”. Then when I started the first part, which you are now reading, I felt that the posting’s title itself already raises several questions, so it would be a good idea to start with a brief Q & A which will help you get into the discussed matter. This is a short list of questions that you may have also asked yourself two sentences into the posting:

1. What is the relation between the SharePoint search service and the SharePoint SiteData web service in the first place?
2. Why would I need to change the working of the SharePoint search, what can be the reasons and motives for that?
3. Is it a wise idea and would you recommend using this hack?

And the answers come promptly:

1. To answer this one we need to have a closer look at the internal workings of the SharePoint search index engine. If you are not familiar with some core concepts and basic terminology like index engine, content sources, filter daemon, protocol handlers, IFilters, I would recommend that you first check these two MSDN articles – here (for a high level architecture overview) and here (for a high level overview of the protocol handlers). Let me start with several words about the protocol handlers – these are basically responsible for crawling the different types of content sources. They are implemented as COM components written in unmanaged code (C or C++). If you are familiar with COM Interop you will know that it is also possible to create COM components using .NET and managed code and in fact there is a sample .NET protocol handler project in CodePlex. I am not sure though how wise it is to create your own protocol handler with managed code (apart from the fact that it is quite complex to start with) knowing that all existing protocol handlers by Microsoft and third party vendors are written in unmanaged code.
   You can check the available index engine protocols and their matching protocols handlers for your SharePoint installation in the Windows registry:
   
   
   
   You can see that there are different protocol handlers for the different types of content sources – SharePoint sites, external web sites, file shares, BDC, etc. The name of the protocol handler (the “Data” column in the image above) is actually the ProgID (in COM terms) of the COM component that implements the handler.
   In this posting we are only interested in just one of the protocol handlers – this is the one for the Sts3 protocol, which is responsible for crawling the content from SharePoint sites. The same handler is also used for the Sts3s protocol (see the image) which is again for SharePoint sites but which use the HTTPS (SSL) scheme. And now the interesting part – how does the Sts3 protocol handler traverse the content from SharePoint. The answer is actually also the answer of the first question in the list above – it calls the standard SharePoint SiteData web service (/_vti_bin/SiteData.asmx). If you wonder why for instance it doesn’t use the SharePoint object model directly – the main reason I think is for greater scalability (not to mention that it would be at best challenging to call managed from unmanaged code). The better scalability comes from the fact that the handler can be configured to call the SiteData web service from all available web front servers in the SharePoint farm, which can distribute better the workload and utilize better the resources of the farm. Later in the posting I will give you more details about how you can check and monitor the calls to the SiteData web service from the crawl engine and also some additional information about the exact methods of the SiteData service that are used for traversing the content of the SharePoint sites.
2. As I already mentioned in the answer for the first question, this posting deals specifically with the search functionality that targets SharePoint content. So, the motives to come to this hack are directly related to the using and querying of SharePoint data. The reasons and motives for these changes can be separated in two groups – the first one is more general - why use SharePoint search and not some other available alternative method. The second one is more specific – what is not available or well implemented in the SharePoint search query engine that needs to be changed or improved.
   Let me start with the first group – out of the available methods to query and aggregate SharePoint content in the form of SharePoint list item and document metadata – SharePoint search doesn’t even come as the first or preferred option. Normally you would use the SharePoint object model with the SPList, SPListItem and SPQuery classes (for a single SharePoint list) or the SPSiteDataQuery class with the SPWeb.GetSiteData method (or alternatively the CrossListQueryInfo and CrossListQueryCache classes if you use the publishing infrastructure) – for querying and retrieving data from many lists in one site collection. The cross list query functionality is actually directly used in the standard SharePoint content by query web part (CQWP), so even without using custom code you may have experienced certain issues with it. Probably the biggest one is performance – maybe you’ve never seen it or you are well aware of it. This is because it becomes a real issue only if the size of your site collection in terms of the number of sub-sites becomes very big and you use queries that aggregate data from most of the available sub-sites. You can add to these two conditions the number of list items in the individual SharePoint lists which further degrades the performance. So, when does this become a visible issue – you can have various combinations of the above said conditions, but if you query more than one hundred sub-sites and/or you have more than several thousand items in every list (or many of the lists) you may see page loading times ranging from several seconds to well above a minute in certain extreme cases. And … this is an issue even with the built-in caching capabilities of the cross list query classes. As to why the caching doesn’t solve always the performance issue – there are several reasons (and cases) for that: first – there’re specific CAML queries for which the caching is not used at all (e.g. queries that contain the <UserID /> element); secondly – even if the caching works well, you have the first load that populates the cache which will still be slow, etc.
   Let me now briefly explain why the cross list query has such performance issues (only in the above mentioned cases). The main reason is the fact that the content database contains all list data (all list items in the whole site collection, it may also contain more than one site collections) in a highly denormalized table called AllUserData. This design solution was totally deliberate because it allows all flexibility that we know with SharePoint lists in terms of the ability to add, modify and customize fields, which unfortunately comes with a price in some rare cases like this one. Let’s see how the cross list query works from a database perspective with a real example – let’s say that we have a site collection with one hundred sub-sites each containing an “announcements” list with two custom fields “expiration date” and “publication date”. On the home page of the root site we want to place a CQWP that displays the latest five announcements (aggregated from all sub-sites) ordered by publication date and for which the expiration date is in the future. Knowing that all list item data is contained in a single database table you may think that it may be possible to retrieve the aggregated data in a single SQL query but, alas, this is not the case. If you have a closer look at the AllUserData table you will find out that it contains columns whose names go: nvarchar1, nvarchar2, …, int1, int2, …, datetime1, datetime2, … – these are the underlying storage placeholders for the various types of SharePoint fields in your lists. Obviously the “publication date” and “expiration date” will be stored in two of the “datetimeN” SQL columns but the important thing is that for the different lists the mappings may be totally different, e.g. for list 1 “publication date” and “expiration date” may map to datetime1 and datetime2 respectively, whereas for list 2 they can map to datetime3 and datetime4 respectively. This heterogeneous storage pattern makes the retrieval much more complex and time costly – the object model first needs to extract the metadata for all target lists in these one hundred sites (which contains the mappings for the fields) and after that retrieve the items from all one hundred lists one by one making a SQL union with the correct field to SQL columns mappings and applying the filtering and sorting after that. If you are interested in checking that yourself you can use the SQL profiler tool that comes with the MS SQL management studio.
   Having seen the performance issues that may arise with the usage of the cross list query built-in functionality of SharePoint, it is quite natural to check what SharePoint Search can offer as an alternative. Obviously it performs much faster in these cases and allows data retrieval and metadata filtering but the results and functionality it has are not exactly identical to the ones of the cross list query. And here we come to the second group of motives for implementing this kind of hack that I mentioned in the beginning of this paragraph. So let’s see some of the things that we’re missing in SharePoint search – from a data retrieval perspective – the text fields, especially the ones that contain HTML are returned by the search query with the mark-up stripped out (this is especially embarrassing for the publishing Image field type, whose values are stored as mark-up and get retrieved virtually empty by the search query); the “content type id” field is never crawled and cannot be used as a crawled and managed property; for the “lookup” field type (and derivative field types as the “user” type) – these are retrieved as plain text, with the lookup item ID contained in the field value stripped out; etc. From filtering and sorting perspective, you have pretty much everything needed – you can perform comparison operations on the basic value types – text, date, integer and float and perform the correct sorting based on the respective field type. What is missing is for instance the filtering on “lookup” (including “user”) fields based not on the textual value but on the integer (lookup ID) value – this is because this part of the lookup field value is simply ignored by the search crawler (we’ll come to that in the next part of the posting). For the same reason you cannot filter on the “content type id” field.
   The next question is of course is it possible to achieve these things with the SharePoint search – the answer is yes, and the hack that is a subject of this posting does exactly that.
3. And lastly the third and most serious one – most of the time I am overly critical towards my own code and solutions, so I would normally not recommend using this hack (I will publish the source code in the second part of the posting), at least not in production environments. I would only suggest that you use it very limitedly in development/testing or small intranet environments if at all. I suppose that the material in the posting about some of the inner workings of the indexing engine and the SiteData web service would be interesting and useful by itself.

So, let’s now see how the index engine or more precisely the Sts3 protocol handler calls the SiteData web service. Basically you can track the SiteData.asmx invocations by simply checking the IIS logs of your web front server or servers (you have to have IIS logging enabled beforehand). If you first run a full crawl on one of your “SharePoint Site” content sources from the SSP admin site and after it completes open the latest IIS log file you will see that there will be many request to _vti_bin/SiteData.asmx and also to all pages and documents available in the SharePoint sites that were listed in the selected content source. It is logical to conclude that the protocol handler calls the SiteData web service to traverse the existing SharePoint hierarchy and to also fetch the available metadata for the SharePoint list items and documents and then it also opens every page and document and scans/indexes their contents so that they are later available for the full text search queries.

The checking of the IIS logs was in fact the first thing that I tried when I began investigating the SiteData-SharePoint Search relation but I was also curious to find out what method or methods exactly of the SiteData web service get called when the crawler runs. If you have a look at the documentation of the SiteData web service you will see that some of its methods like GetSite, GetWeb, GetListCollection, GetList, GetListItems, etc. look like ideal candidates for traversing the SharePoint site hierarchy starting from the site collection level down to the list item level. The IIS logs couldn’t help me here because they don’t track the POST body of the HTTP requests, which is exactly the place where the XML of the SOAP request is put. So I needed a little bit more verbose tracking here and I quickly came up with a bit ugly but working solution – I simply modified the global.asax of my test SharePoint web application like this:

<%@ Assembly Name="Microsoft.SharePoint"%><%@ Application Language="C#" Inherits="Microsoft.SharePoint.ApplicationRuntime.SPHttpApplication" %>

<%@ Import Namespace="System.IO" %>

<script RunAt="server">

 

    protected void Application_BeginRequest(object sender, EventArgs e)

    {

        TraceUri();

    }

 

    protected void TraceUri()

    {

        const string path = @"c:\temp\wssiis.log";

        try

        {

            HttpRequest request = HttpContext.Current.Request;

            DateTime date = DateTime.Now;

            string httpMethod = request.HttpMethod;

            string url = request.Url.ToString();

            string soapAction = request.Headers["SoapAction"] ?? string.Empty;

            string inputStream = string.Empty;

 

            if (string.Compare(httpMethod, "post", true) == 0)

            {

                request.InputStream.Position = 0;

                StreamReader sr = new StreamReader(request.InputStream);

                inputStream = sr.ReadToEnd();

                request.InputStream.Position = 0;

            }

 

            string msg = string.Format("{0}, {1}, {2}, {3}, {4}\r\n", date, httpMethod, url, soapAction, inputStream);

 

            File.AppendAllText(path, msg);

        }

        catch { }

    }

</script>

The code is pretty simple – it hooks onto the BeginRequest event of the HttpApplication class which effectively enables it to track several pieces of useful information for every HTTP request made against the target SharePoint web application. So, apart from the date and time of the request, the requested URL and the HTTP method (GET, POST or some other) I also track the “SoapAction” HTTP header which contains the name of the SOAP method for a web service call and also the POST content of the HTTP request which contains the XML of the SOAP request (in the case of a web service call). The SOAP request body contains all parameters that are passed to the web service method call, so by tracking this I could have everything I wanted – the exact web service method being called and the exact values of the parameters that were being passed to it. Just to quickly make an important note about this code – don’t use it for anything serious, I created it only for testing and quick tracking purposes.

With this small custom tracking of mine enabled I ran a full crawl of my test web application again and after the crawl completed I opened the log file (the tracking code writes to a plain text file in a hard-coded disc location) and to my surprise I saw that only two methods of the SiteData web service were called – GetContent and GetURLSegments. Actually the real job was obviously done by the GetContent method – there were about 30-35 calls to it, and only one call to GetURLSegments. You can see the actual trace file that I had after running the full crawl here. My test web application was very small containing only one site collection with a single site, so the trace file is very small and easy to follow. The fourth column contains something that looks like an URL address but this is in fact the value of the “SoapAction” HTTP header – the last part of this “URL” is in fact the actual method that was called in the SiteData web service. The fifth column contains the XML of the SOAP request that was used for the web service calls – you can see the parameters that were passed to the SiteData.GetContent method inside. If you check the MSDN documentation about the SiteData.GetContent method you will see that its first parameter is of type “ObjectType” which is an enumeration. The possible values of this enumeration are: VirtualServer, ContentDatabase, SiteCollection, Site, Folder, List, ListItem, ListItemAttachments. As one can deduce from this enumeration, the GetContent method is designed and obviously used for hierarchy traversing and metadata retrieval (the MSDN article explicitly mentions that in the yellow note box at the bottom). If you check the sample trace file from my test site again you will see that the calls made by the crawler indeed start with a call using ObjectData.VirtualServer and continue down the hierarchy with ObjectData.ContentDatabase, ObjectData.SiteCollection, etc. You may notice something interesting – after the calls with ObjectData.List there’re no calls with ObjectData.ListItem. Actually in the trace file there is only one call to GetContent using ObjectData.ListItem and it is invoked for the corresponding list item of the home (welcome) page of the site, which in my case was a publishing page. The other method of the SiteData web service – GetURLSegments is also called for the home page only – it basically returns the containing site and list of the corresponding list item by providing the URL of the page. And if you wonder which option is used for retrieving list items – it is neither the ObjectData.List nor the ObjectData.ListItem. The former returns an XML fragment containing mostly the list metadata and the latter the metadata of a single list item. The option that actually returns the metadata of multiple list items is ObjectData.Folder. Even though the name is a bit misleading this option can be used in two cases – to retrieve the files from a folder that is not in a SharePoint list or library (e.g. the root folder of a SharePoint site) or to retrieve the list items from a SharePoint list/library. If you check the sample trace file you will see that the GetContent method is not called with ObjectData.Folder for all lists – this is because the crawler is smart enough and doesn’t call it for empty lists (and most of the lists in my site were empty). And the crawler knows that a particular list is empty by the preceding GetContent (ObjectData.List) call which returns the ItemCount property of the list. There is one other interesting thing about how the crawler uses the GetContent with ObjectData.Folder – if the list contains a big number of items, the crawler doesn’t retrieve all of them with one call to GetContent but instead reads them in chunks of two thousand items each (the logic in SharePoint 2010 is even better – it determines the number of items in a batch depending on the number of fields that the items in the particular list have). And … about the return value of the GetContent method – it is in all cases an XML document that contains the metadata for the requested object or objects. It is interesting to note that the XML also contains the permissions data associated with the object which is obviously used by the indexing engine to maintain ACL-s for the various items in its index which allows the query engine to apply appropriate security trimming based on the permissions of the user that issues the search query. For the purposes of this posting we are mostly interested in the result XML for the ObjectData.List and ObjectData.Folder GetContent invocations – here’re two sample XML fragments from GetContent (List) and GetContent (Folder) calls. Well, indeed they seem quite … SharePoint-ish. Except for the permissions parts, the GetContent (Folder) yields pretty much the same XML as the standard Lists.GetListItems web service method. Have a look at the attributes containing the field values in the list items – these start with the well-known “ows_” prefix, which is the very same prefix that we see in the crawled properties associated with SharePoint content. Another small detail to note is that the GetContent (Folder)’s XML is not exactly well formed – for example it contains not properly escaped new line characters inside attribute values (not that this prevents it from rendering normally in IE) – I will come again to this point in the second part of this posting.

So far so good, but the results above are from a full crawl. And what happens when we run an incremental crawl? Have a look at the sample trace file that I got when I ran an incremental crawl on my test web application after i had changed several list items and had created a new sub-site and several lists in it. You can see that it contains again several calls to SiteData.GetContent, one call to SiteData.GetURLSegments and this time one call to SiteData.GetChanges. If you wonder why there is only one call to SiteData.GetChanges – a quick look at the result XML of this method will explain most of it. If you open the sample XML file you will see that the XML is something like a merged document from the results of the GetContent method for all levels from “ContentDatabase” down to “ListItem” … but containing only the parts of the SharePoint hierarchy whose leaf descendants (that is list items) got changed since the time of the last crawl. So basically, with one call the crawler can get all the changes in the entire content database … well, almost. Unless there are too many changes – in this cases the method is called several times each time retrieving a certain number of changes and then continuing after the reached book-mark. If you check the documentation of the GetChanges method in MSDN you will see that its first parameter is again of type ObjectData. Unlike the GetContent method however, you can use it here only with the “ContentDatabase” and “SiteCollection” values (the rest of the possible values of the enumeration are ignored and the returned XML if you use them is the same as with the “ContentDatabase” option). And one last thing in the case of the incremental crawl – the calls to the GetContent method are only for new site collections, sites and lists (which is normally to expect). The metadata for new, updated and deleted list items in existing lists is retrieved with the call to the GetChanges method.

So, this was in short the mechanism of the interaction between the SharePoint Search 2007 indexing engine (the Sts3 protocol handler) and the SharePoint SiteData web service. In the second part of this posting I will continue with explaining how I got to hack the SiteData web service and what the results of this hack were for the standard SharePoint search functionality.

Pluggable Content Query web part

Sub-classing the standard SharePoint ContentByQueryWebPart is a common approach for extending the default set of functionality that this web part offers. There are many examples for sub-classing the CQWP on the internet – for instance this sample by Andrew Connell that handles URL query parameters for adding dynamic filtering in the web part, or the paging enabled CQWP version by Waldek Mastykarz, just to name a few.

In most cases the need to sub-class the CQWP comes with some extra requirements for filtering or modifying the result set (SharePoint list items or documents) that the CQWP is supposed to present. The standard implementation of the CQWP of course has many filtering options itself but if you want some additional filtering based on dynamic conditions (as it is the case with the URL query parameters) you will have to consider the extending of the standard implementation by sub-classing the ContentByQueryWebPart class. As for the URL query parameters example, the SharePoint 2010 version of the CQWP has some limited built-in support for handling query parameters, but this still may be far from adequate in many cases.

I myself have sub-classed the CQWP in a handful of SharePoint projects of mine, so now I have maybe at least ten different implementations of cases where I had to handle some similar or not that similar requirements all based on the premises of having to have some sort of modification of the original CQWP result set. And some time ago I brain-stormed a bit the whole issue coming up with the idea that the whole problem can be handled in a bit more generalized way – if you for instance separate the web part implementation and the result set filtering/modifying logic implementation in a sort of pluggable architecture. Basically, the idea is pretty simple and further facilitated by the fact that you can have as many custom persistable properties in a web part as you need. The “plugin” can be as simple as a .NET class implementing a predefined .NET interface and the web part can be configured with a dedicated string property containing the fully qualified type name of this class, which will allow the web part to create instances of it using reflection. The web part will then be implemented to call on the methods of the plugin class which will be implementing the “contract” specified in the predefined .NET interface that I mentioned. The whole idea is that on the side of the extended CQWP there will be only a single implementation instead of many different ones for all possible cases that you may have and the whole effort would be for the part with the filtering/modification “plugin” class (classes) where it will be possible to concentrate on the retrieving/filtering specifics instead of having to pay attention to web part and presentation details.

And now to the more interesting part – how is it actually possible to hook onto the standard CQWP web part with your inheriting class so that you can modify the result set that the CQWP normally produces. Basically, there are a few options available here – it can be as simple as setting or modifying the values of the built-in filtering properties of the CQWP – for instance: FilterField1, FilterValue1, FilterType1, FilterOperator1, Filter1ChainingOperator, ListsOverride, QueryOverride, etc. Another approach is to use the handy ProcessDataDelegate property to which you can provide a delegate referencing a custom method receiving a DataTable parameter with a return type a DataTable as well. The idea of this delegate property is, that you can provide a custom method of yours to it, and it will be called with a DataTable parameter that will contain the freshly retrieved result set that the CQWP is about to display and you will have the chance to modify this DataTable instance (and its contents) or to even create a new DataTable instance and reuse, modify or filter the items from the original DataTable. The modified or the newly created DataTable then you will return from the custom method to the CQWP so that it goes to the presentation handling logic (check Waldek’s posting at the top for more details).

Having briefly mentioned these two approaches for implementing some extra filtering in the CQWP and before I continue with describing the actual approach in my implementation I think that it will be a good moment here to give you some more details on the internal workings of the CQWP and more specifically on its data retrieving logic so that you can better understand the whole hooking procedure in the inheriting “pluggable” CQWP. So, the first thing here, which is probably familiar to most of you is that the CQWP uses internally the  CrossListQueryCache and CrossListQueryInfo classes combo for retrieving aggregated cross list data. These two classes are actually a thin wrapper on top of the SPWeb.GetSiteData method and the SPSiteDataQuery class in which you provide the CAML query details for the cross list data call. What the CrossListQueryCache and CrossListQueryInfo classes add on top of the base implementation is some caching support and support for audience filtering and grouping. In theory this is pretty simple, the using of the cross list classes or the SPWeb.GetSiteData method is pretty common in many cases even not related to the CQWP at all. Apart from this there are several important facts giving some further details about the exact implementation of the cross list query invocation in the CQWP:

• The call of CrossListQueryCache.GetSiteData method that retrieves the list data for the CQWP takes place in the GetXPathNavigator virtual method (actually in a private method called by the latter) of the web part. Note that the GetXPathNavigator is a virtual method which means that you can override it thus it is an ideal candidate for hooking onto the base implementation of the CQWP, when it comes to modifying its data retrieval logic.
• The DataTable instance returned by the CrossListQueryCache.GetSiteData method is assigned to the Data public property of the CQWP. This property has a public getter and setter and even more importantly – the CQWP issues the CrossListQueryCache.GetSiteData call only if the Data property is not initialized and has the default null value.
• The CrossListQueryCache class requires an instance of the CrossListQueryInfo class in its constructor. The CrossListQueryInfo has properties like the Query, Lists, Webs, RowLimit, etc. in which you specify CAML fragments that determine what list data you want to retrieve from the target site collection. The CQWP creates and initializes a CrossListQueryInfo instance based on its filtering properties, some of which I mentioned above: FilterField1, FilterValue1, FilterType1, FilterOperator1, Filter1ChainingOperator, ListsOverride, QueryOverride, etc. The good news here is that the CQWP provides a public method which you can use to get a CrossListQueryInfo instance with the exactly same configuration as the one that the web part uses for its internal cross list query call. This method is BuildCbqQueryVersionInfo – it actually returns an instance of the CbqQueryVersionInfo class whose CbqQueryVersionInfo.VersionCrossListQueryInfo property contains the CrossListQueryInfo instance that we need.
• And lastly – the call to the delegate ProcessDataDelegate property that I mentioned above also takes place in the CQWP’s GetXPathNavigator virtual method. Actually this happens at the same moment when the web part gets the list data in the DataTable instance from the CrossListQueryCache.GetSiteData call.

With all these facts it is easy now to devise a strategy for implementing the hooking logic in the inheriting class. It is obvious that the whole logic can be placed in the overridden version of the GetXPathNavigator method, just before you call the base CQWP implementation (base.GetXPathNavigator();) in the method body. For the plugin logic I thought of three alternative types of overriding the standard CQWP data retrieving and filtering logic:

• The first type of override is the most radical one – you directly circumvent the standard logic for retrieving the list data in the CQWP and produce your own DataTable instance in some custom way. The only thing that you will need to do after producing the DataTable instance is to assign it to the CQWP’s “Data” property. This way you will ensure that when you call the base GetXPathNavigator method, the CQWP won’t issue its standard cross list query call, thus there won’t be any performance issues because of the double data retrieval. As of how you are going to get your DataTable instance – you have many possible choices here depending on your particular case. I can think of for instance using the SharePoint search functionality, using data in an external SQL database or even using again the CrossListQueryCache.GetSiteData method but applying some custom caching logic instead of relying on the built-in one if it doesn’t fit in your requirements.
• The second type of override is to manipulate just the CrossListQueryInfo instance that will be used for the CrossListQueryCache.GetSiteData call. This instance will contain the initial configuration of the web part and you can add some additional filtering or scoping logic to it based on your requirements. Then the inheriting web part will issue the call to the CrossListQueryCache.GetSiteData method and will again set the CQWP’s Data property, so that the base implementation doesn’t make a second call. Note also that in this type of overriding the CQWP’s filtering you have to deal only with a CrossListQueryInfo instance and manipulate the CAML fragments in its properties (merge some extra filtering CAML, etc.) instead of having to modify directly the CQWP’s properties like the FilterField1, FilterField2, FilterField3, etc. which the end user may have already set values to using the SharePoint UI.
• The third type of override is to simply use the standard delegate ProcessDataDelegate property override mechanism – the CQWP will make the cross list query call itself and we will only manipulate the DataTable instance (or create a new instance using the rows from the latter) that it has already retrieved.

And now, let’s move on to some code snippets which will better illustrate the implementation of the pluggable CQWP. You can download the project containing the pluggable CQWP from here. Let me first show you the code of the .NET interface that should be implemented by the custom plugin classes for the pluggable CQWP:

    public interface IContentQueryPlugin

    {

        // should return true if the pluging can generate the data for the web part - the GetDataResults method should produce a DataTable instance

        bool CanGetDataResults(PluggableContentByQuery part);

        // should return true if the plugin needs to modify the CrossListQueryInfo instance that the CQWP uses for its internal cross list query - the ProcessQueryInfo should be implemented in this case

        bool CanProcessQueryInfo(PluggableContentByQuery part);

        // should return true if the plugin needs to modify the DataTable returned by the CQWP cross list query - the ProcessData method should be implemented

        bool CanProcessData(PluggableContentByQuery part);

 

        // implement this if you want the retrieve the data without using the CQWP built-in retrieval mechanism

        DataTable GetDataResults(PluggableContentByQuery part, CrossListQueryInfo queryInfo);

        // implement this if you want to modify the CrossListQueryInfo instance before the CQWP issues its cross list query

        CrossListQueryInfo ProcessQueryInfo(PluggableContentByQuery part, CrossListQueryInfo queryInfo);

        // implement this if you want to modify the already retrieved by the CQWP DataTable with the results of the cross list query

        DataTable ProcessData(PluggableContentByQuery part, DataTable data);

    }

As you see the interface contains six methods, but actually the three types of overrides that I mentioned above are implemented by only three of the methods: GetDataResults, ProcessQueryInfo and ProcessData. The other three methods are sort of auxiliary methods in the sense that you have one auxiliary method corresponding to one of the implementation methods – the auxiliary methods all start with the verb “can” prefix. Their purpose is pretty transparent – since the types of overrides that you can implement are optional (and even mutually exclusive – the first one versus the second two) with the auxiliary methods you can simply specify whether you are implementing a particular override. For instance if you want to take care of retrieving the data results yourself you are going to implement the GetDataResults method and also you will have to implement the CanGetDataResults auxiliary method by simply returning “true” in its implementation (for the other “can” methods you will have to return “false”).

Let’s now have a closer look at the CQWP overrides implementing methods:

• the GetDataResults method – as I mentioned you go for this method if you want to have your custom data retrieving logic and circumvent the CQWP’s normal data retrieving routine altogether. As you see, the method’s return type is a DataTable, meaning that you will have to gather your result items in some custom way and then produce a DataTable instance that you will pass back to the CQWP. The method accepts two parameters – a reference to the “parent” CQWP and a CrossListQueryInfo object. The latter will contain all CAML fragments for the cross list query that the web part would have issued – you can use that for the filtering against your custom data source if you use one. Further you can use the reference to the web part to inspect the properties of the latter and make use of their values in some way or another.
• the ProcessQueryInfo method – you can use this method if you want to only add some additional CAML markup or make some other modifications to the CrossListQueryInfo object that the CQWP constructs for its cross list query call. This means that you are opting only for some additional filtering but the data retrieval method remains the standard one for the CQWP. The method accepts CrossListQueryInfo parameter and also returns a CrossListQueryInfo object. You can either make your modifications to the input CrossListQueryInfo instance and then return the same instance, or you can optionally create a brand new CrossListQueryInfo instance, copy some or all of the properties of the source instance and then return the new one.
• the ProcessData method – if you opt for this override it will mean that you don’t want the change the CAML filtering and will only want to modify the DataTable instance that the CQWP produces with its standard cross list query call. You can perform all sorts of changes to the DataTable instance like adding new columns to it, add or remove data rows, etc. The removing of data rows will effectively be equivalent to applying some extra filtering, which in many cases will be better if implemented in CAML, meaning that probably there will be a better solution if using the ProcessQueryInfo to inject the filtering in the CrossListQueryInfo’s CAML.

Let me now show you how the IContentQueryPlugin interface fits in the overriding procedure of the CQWP. As I mentioned above, the only hooking point that we need to change the data retrieving logic of the CQWP is the virtual GetXPathNavigator method, and this is its overridden version in the pluggable CQWP:

        protected override XPathNavigator GetXPathNavigator(string viewPath)

        {

            // This method is the single point where we need to hook to get the whole plugin thing working.

            // The standard CQWP issues the cross list query in its implementation of the GetXPathNavigator method

            // setting the CQWP's Data property with a DataTable instance holding the results of the query.

            // The CQWP issues its query only if the Data property is null, so we can even set it with

            // a customly retrieved DataTable if we want before calling base.GetXPathNavigator(viewPath).

 

            // first get the plugin instance

            IContentQueryPlugin plugin = this.GetQueryPlugin();

            if (plugin != null)

            {

                // check if the plugin wants to generate the data results first

                if (plugin.CanGetDataResults(this))

                {

                    // if so, provide it with a CrossListQueryInfo instance and set the Data property of the web part

                    CrossListQueryInfo queryInfo = this.BuildCbqQueryVersionInfo().VersionCrossListQueryInfo;

                    this.Data = plugin.GetDataResults(this, queryInfo);

                }

 

                // then check if the plugin wants to modify the original CrossListQueryInfo instance (check this only if the Data property is still null)

#if SHAREPOINT2010

                if (this.Results == null && plugin.CanProcessQueryInfo(this))

#else

                if (this.Data == null && plugin.CanProcessQueryInfo(this))

#endif

                {

                    // get the CrossListQueryInfo instance and pass it to the plugin

                    CrossListQueryInfo queryInfo = this.BuildCbqQueryVersionInfo().VersionCrossListQueryInfo;

                    queryInfo = plugin.ProcessQueryInfo(this, queryInfo);

                    // after this run the cross list query to populate the Data property

                    IssueQuery(queryInfo);

                }

 

                // set the CQWP ProcessDataDelegate property if the plugin needs to post-process the results DataTable

                if (plugin.CanProcessData(this))

                {

                    this.ProcessDataDelegate = (dt) =>

                    {

                        return plugin.ProcessData(this, dt);

                    };

                }

            }

 

            // only after we finish the plugin stuff call the base CQWP GetXPathNavigator implementation

            return base.GetXPathNavigator(viewPath);

        }

You can see that the implementation is actually pretty concise and straight-forward. Basically the implementation of the overriding GetXPathNavigator method simply dispatches the calls for modifying the data retrieving logic of the CQWP to the plugin class that implements the IContentQueryPlugin interface. The creating of the plugin class instance is also a pretty trivial thing – the pluggable CQWP has a custom string property called PluginClassName, which as its name suggests should be set to contain the fully qualified type name (plus the full assembly name if the plugin class is not in the same assembly as the assembly of the pluggable CQWP) of the plugin class. The pluggable CQWP creates the plugin class instance using reflection (there’s also the option to cache the plugin class instance – check the source code for details). Note also that you can compile the web part for SharePoint 2010 too but you will need to define an extra Debug symbol in the settings of the Visual Studio project of the web part (check the conditional compiling statements in the source code).

The Visual Studio project with the pluggable CQWP (download from here) also contains a sample plugin class implementing the IContentQueryPlugin interface. It implements only the IContentQueryPlugin.ProcessQueryInfo meaning that it only modifies the original CAML of the CQWP’s cross list query call. This is the source code of its implementation of the ProcessQueryInfo method:

        public CrossListQueryInfo ProcessQueryInfo(PluggableContentByQuery part, CrossListQueryInfo queryInfo)

        {

            // check the FilterFieldN and FilterValueN query parameters 0 through 9

            string[][] args = (new int[10]).Select((i, idx) => new string[] { HttpContext.Current.Request.QueryString["FilterField" + idx], HttpContext.Current.Request.QueryString["FilterValue" + idx] })

                .Where (arr => !string.IsNullOrEmpty(arr[0]) && !string.IsNullOrEmpty(arr[1]))

                .ToArray();

            // if none found or their values are empty just exit

            if (args.Length == 0) return queryInfo;

 

            // get the CAML query in an XElement

            XElement queryElement = XElement.Parse("<Query>" + queryInfo.Query + "</Query>");

            // retrieve the Where clause or create a new one if missing

            XElement whereEelement = queryElement.Element("Where");

            if (whereEelement == null)

            {

                whereEelement = new XElement("Where");

                queryElement.Add(whereEelement);

            }

 

            // get the existing where clause if present and append all comparison clauses from the query parameters

            string[] clauses = whereEelement.Elements().Select(el => el.ToString())

                .Union(args.Select(arg => FormatComparisonQuery(arg[0], arg[1], "Eq")))

                .ToArray();

 

            // change the contents of the Where XElement performing a CAML 'And' on the clauses array

            whereEelement.RemoveAll();

            whereEelement.Add(XElement.Parse(FormatLogicalOpQuery("And", clauses)));

 

            // set the Query property of the CrossListQueryInfo instance

            queryInfo.Query = string.Concat (queryElement.Elements().Select(el => el.ToString()).ToArray());

            // return the modified CrossListQueryInfo

            return queryInfo;

        }

Some bits of the full implementation are missing (there are also several helper private methods in the plugin class – check the source code in the project for details) but you can see that the method simply “injects” some extra CAML in the Query property of the source CrossListQueryInfo parameter (the Query property may already contain some CAML that comes from the particular query configuration of the CQWP). The additional filtering that the sample plugin class is dynamically determined by the presence of certain URL query parameters that you use to call the page containing the web part. Actually these are no other than “FilterField1” and “FilterValue1”, the same as the standard LV and XLV web parts use and also the same as Andrew Connell uses in his sample of sub-classing the CQWP that I mentioned in the beginning.

In order that you can see the plugin class working with the pluggable CQWP you will need to set its fully qualified type name in the PluginClassName property of the web part. You can easily do that from the standard SharePoint UI – you will have to put this value for the sample plugin class: “Stefan.SharePoint.WebParts.QueryStringPlugin, Stefan.SharePoint.WebParts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=f9bc508a219e6843”.

New version of the Site Template Configurator utility

This posting is a brief announcement of the new and enhanced version of my Site Template Configurator utility, which I first introduced several months ago in another posting of mine. You can download the source code of the utility from here.

Here is a short list of the changes and enhancements in this new version:

• The tool is now capable of generating “wsp” packages and also has commands for “wsp” installing and uninstalling, both in two flavors – local (for a single machine farm) and farm (launching timer jobs on all machines in the farm). The “wsp” generation and handling functionality neatly packs the custom site templates that you create and make them easy to deploy to other SharePoint farms (see the notes about the “wsp” generation below).

• The “Generate artifacts…” and the “wsp” export and install/uninstall commands are now available in the context menu of the “project/root” node of the left hand pane hierarchy. The “project” menu in the program’s toolbar now contains only the file handling commands for the “template project” file.
• The “project/root” node now contains an extra property – “SiteTemplatePrefix”. This allows you to specify a common prefix for the newly generated site templates’ folders in the “TEMPLATE/SiteTemplates” system folder. If you don’t specify one the tool will use long and ugly Guid containing names as in the previous version.
• The source code comes with two “sln” files – one for Visual Studio 2008 and one for Visual Studio 2010. The tool also compiles for both SharePoint 2007 and SharePoint 2010.
• The tool now displays a “close warning” prompt if you have modified your “template project” and want to exit without saving the changes.

And several handy notes about the tool:

• The tool uses the “makecab” Windows utility to create “wsp” packages, so you need to have it installed on the machine where you will use the utility. Alternatively on machines with SharePoint 2010, the tool may be compiled to use the SharePoint 2010 built-in CAB file support (the functionality is not publicly accessible, but the tool’s code uses a couple of reflection tricks to make use of it). In order that the tool is compiled to use this, you need to specify the “SHAREPOINT2010” compilation symbol in the project settings for the projects in the Visual Studio solution.
• Specify appropriate values in the “SiteTemplatePrefix” and “StartTemplateID” properties of the “project/root” node. The latter specifies the starting ID value of the “TEMPLATE” elements in the generated WEBTEMP*.xml file – make sure, that you don’t have several projects that use one and the same starting or overlapping ID values (these should be unique in the SharePoint farm). Use the “Generate artifacts …” command of the “project/root” node and then carefully inspect the configuration files (and the subfolders that they are placed in), that the tool generates – make sure that when these get copied to the TEMPLATE/SiteTemplates and TEMPLATE/1033 system folders, they won’t overwrite existing files there.
• Check the “IncludeLists” and “IncludeModules” properties of the “template” nodes (the nodes with blue icons). These have the value “false” by default, which means that the “List” and “Module” elements from the ONET.XML of the base template won’t get executed for your custom template. If you don’t have the “IncludeModules” property enabled, and you don’t have a custom feature that provisions another default/welcome page for your site template, the sites based on this custom template will end up with no default/welcome page.

Automatically publishing files provisioned with Sandboxed Solutions

I couple of days ago I saw this posting on Waldek Mastykarz’s blog (a great blog by the way) and now you are reading my posting with the very same name, which treats also the very same issue, but provides a different solution to it. So, what exactly is the issue with file provisioning with sandbox solutions? In short – the problem occurs when you use “Module” elements in a feature for provisioning files to SharePoint document libraries. And not to any document library but to a library which is configured with one of the following:

1. Check out for files is required
2. Content approval is enabled
3. Minor versions are enabled

The files will be of course provisioned to the target library but they will be either checked out and/or in a draft or pending state depending on the enabled configurations and the actual combination of the three that is enabled on the library. And this will mean that the files will be present but they won’t be accessible for the regular users of the site. This behavior is obviously intended and by design – I suppose that the main reason for it is the ability for the site and site collection administrators to review and approve or reject the content provisioned by the sandbox solution.

Having said that, I should mention that I knew about this issue for quite some time myself and even had tackled one isolated case of it in a previous posting of mine – it is about provisioning of publishing pages with sandbox solutions. The problem with publishing pages was even more serious – not only the pages end up in a draft state and checked out, but the web parts defined in the “Module” manifest file never get provisioned (check the posting itself for more details). The solution for publishing pages was pretty neat and simple  (it doesn’t need a feature receiver and code with it) but unfortunately it is applicable only for publishing pages.

As for the solution itself – it is known that the SPFile.CheckIn, SPFile.Publish and SPFile.Approve methods are available in sandbox solutions too, so with a feature receiver and several lines of code the solution would be pretty trivial. But obviously in this case the best solution would be if you have a reusable feature receiver that you write just once and then can use everywhere without changing or adjusting its code. The universal feature receiver will save you a lot of time and efforts since the usage of “Module” features is pretty frequent in SharePoint. The main technical challenge for this universal feature receiver is that the code in the receiver should be “aware” of which files exactly its feature provisions to the target library (libraries). This type of “awareness” is provided OOB by the built in SPFeatureDefinition.GetElementDefinitions method but unfortunately this method is not available in the sandbox subset of the object model. So we need a different solution and I will start with briefly explaining the solution from Waldek’s blog – it is pretty simple and requires very little additional effort – you just add an extra “Property” element to every “File” element in your “Module” manifest file like that:

<File Path="Images\someimage.png" Url="someimage.png" Type="GhostableInLibrary">

  <Property Name="FeatureId" Value="$SharePoint.Feature.Id$" Type="string"/>

</File>

The idea is as follows – since you won’t have a column name with the name “FeatureId” in your target library, the value of this property element will be saved in the property bag of the underlying SPFile instance of the provisioned file. And the value of the property element is none other but the Guid of the parent feature (note the smart usage of the Visual Studio token syntax). So basically by adding this extra “Property” element to all “File” elements in the manifest file you literally “mark” all files that your feature provisions. The logic then of the feature receiver will be to iterate all files in the target library and after finding all marked files to check them in, approve or publish them where appropriately. Additionally you will have to provide somehow the name/names of the target library/libraries to the feature receiver, which is possible by using feature properties in the feature.xml file of your feature.

And, now to the reasons as to why I started this article and thought of a different solution – the solution above is basically pretty neat and straight-forward and involves very little extra implementation effort, but what I didn’t like about it was the iteration of all files in the target library(libraries), which may happen to contain thousands of files (which in fact is very improbable but this thought always gives me the creeps when I think of some aspects of SharePoint performance). Of course there’re several ways to optimize the iteration – for instance, to iterate the SPList.Items collection instead of the SPFolder/SPFile hierarchy (the former is generally faster) and use the contents of the internal system MetaInfo list column which contains the serialized property bag of the underlying SPFile instance (unfortunately you cannot use CAML filtering on this column). Further, a very simple but effective optimization would be to use CAML filtering on the “Created”, “_Level” and “_ModerationStatus” fields so that you can fetch only items just recently created (say you can put a margin of 5 or 10 minutes in the past; note that even if the file exists already in the library, when the feature provisions it again, its “Created” date will also get updated) and are either checked out or in a draft or pending state.

Thus far so good, but still I wanted to find a solution that is as close to the “self awareness” approach with the SPFeatureDefinition.GetElementDefinitions method, so that the feature receiver knows exactly which files its feature provisions without it being necessary to traverse the target library or libraries and search for them. And what occurred to me was that if we need the contents of the manifest file in the feature receiver why can’t we simply provision it to the target site (it is available in the feature definition as an element file but there is nothing wrong if it is also referenced in a “File” element in the manifest file), then read its contents from the feature receiver, and finally when the files are published the feature receiver can safely delete it. And let me show you a sample “Module” manifest file so that you can get a better idea of the trick that I just explained:

<Elements xmlns="http://schemas.microsoft.com/sharepoint/">

  <Module Name="TestModule" Url="Style Library">

    <File Path="TestModule\Sample.txt" Url="Sample.txt" Type="GhostableInLibrary" />

    <File Path="TestModule\Sample2.txt" Url="Sample2.txt" Type="GhostableInLibrary" />

    <File Path="TestModule\Sample3.txt" Url="test/Sample3.txt" />

  </Module>

  <Module Name="TestModule2">

    <File Path="TestModule\Elements.xml" Url="Elements_$SharePoint.Feature.Id$.xml" Type="Ghostable" />

  </Module>

</Elements>

You can see that the manifest file (whose name is the commonplace “elements.xml”) contains two “Module” elements – the first one provisions several files to the standard “Style Library” library (a pretty recurring task in SharePoint development) and check carefully the second one – this is the only extra bit in the manifest file that you need (the other bit is the reusable feature receiver) – this is a “Module” element that provisions the manifest file itself. Note two things – the manifest file gets provisioned to the root folder of the target site – we don’t want to put the file in a document library because this way it will be directly visible to the site users. And secondly – check the URL of the file – it contains the Guid of the feature and the feature receiver will use exactly this so that it can locate the file when it gets to be executed. The pattern of the URL should be like this: [any number of characters that are unique among the “Module” manifest files in the parent feature]_$SharePoint.Feature.Id$.xml (the part from the underscore character on should be always the same) – basically we want to get a unique target name for every “Module” manifest file in this feature. And here is the code of the feature receiver itself:

    public class TestFeatureEventReceiver : SPFeatureReceiver

    {

        // The SharePoint elements file namespace

        private static readonly XNamespace WS = "http://schemas.microsoft.com/sharepoint/";

 

        public override void FeatureActivated(SPFeatureReceiverProperties properties)

        {

            // make it work for both 'Site' and 'Web' scoped features

            SPWeb web = properties.Feature.Parent as SPWeb;

            if (web == null && properties.Feature.Parent is SPSite) web = ((SPSite)properties.Feature.Parent).RootWeb;

            if (web != null) CheckinFiles(web, properties.Feature.DefinitionId);

        }

 

        private void CheckinFiles(SPWeb web, Guid featureID)

        {

            // create a regular expression pattern for the manifest files

            string pattern = string.Format(@"^.+_{0}.xml$", featureID);

            Regex fileNameRE = new Regex(pattern, RegexOptions.Compiled | RegexOptions.IgnoreCase);

 

            // get the manifest files from the root folder of the site

            SPFile[] manifestFiles = web.RootFolder.Files.Cast<SPFile>().Where(f => fileNameRE.IsMatch(f.Name)).ToArray();

            try

            {

                // iterate the manifest files

                foreach (SPFile manifestFile in manifestFiles)

                {

                    // load the contents of the manifest file in an XDocument

                    MemoryStream mStream = new MemoryStream(manifestFile.OpenBinary());

                    StreamReader reader = new StreamReader(mStream, true);

                    XDocument manifestDoc = XDocument.Load(reader, LoadOptions.None);

 

                    // iterate over the 'Module' and 'File' elements in the XDocument, concatenating their Url attributes in a smart way so that we grab the site relative file Url-s

                    string[] fileUrls = manifestDoc.Root.Elements(WS + "Module")

                        .SelectMany(me => me.Elements(WS + "File"), (me, fe) => string.Join("/", new XAttribute[] { me.Attribute("Url"), fe.Attribute("Url") }.Select(attr => attr != null ? attr.Value : null).Where(val => !string.IsNullOrEmpty(val)).ToArray()))

                        .ToArray();

 

                    // iterate the file url-s

                    foreach (string fileUrl in fileUrls)

                    {

                        // get the file

                        SPFile file = web.GetFile(fileUrl);

                        // depending on the settings of the parent document library we may need to check in and/or (publish or approve) the file

                        if (file.Level == SPFileLevel.Checkout) file.CheckIn("", SPCheckinType.MajorCheckIn);

                        if (file.Level == SPFileLevel.Draft)

                        {

                            if (file.DocumentLibrary.EnableModeration) file.Approve("");

                            else file.Publish("");

                        }

                    }

                }

            }

            finally

            {

                // finally delete the manifest files from the site root folder

                foreach (SPFile manifestFile in manifestFiles) manifestFile.Delete();

            }

        }

    }

As you can see, the receiver’s code is quite small and straight-forward (check also the comments inside the code). In short what it does is as follows: it first iterates the files in the site root folder, matching their names against the above mentioned name pattern, then it loads the manifest files one by one in an XDocument object and extracts the URL-s of the files in every manifest file. After that the provisioned files are being checked in, published or approved if necessary (depending on their state and the settings of the containing document library). The final step is to delete the manifest file or files in the site root folder, because they are no longer needed, of course.

Feature stapling in sandbox solutions

In a recent posting of mine about the WebProvisioned event receiver in SharePoint 2010, I made a comparison between the latter and the feature stapling functionality which has been available since SharePoint 2007. In that posting I had the wrong assumption that feature stapling only works at farm level, but after a prompting comment I had to correct this. After another check of the SharePoint SDK, I saw that feature stapling can actually be scoped at three levels: farm, web application and site collection – which provides not only extra flexibility but also better isolation when necessary. When I understood that one can create a site collection scoped “feature stapling” feature, I asked myself whether it’s possible to provision such a feature in a sandbox solution (that’s the highest possible feature scope for a sandbox solution). I created a small POC project in a couple of minutes, deployed it and it turned out that feature stapling does work in sandbox solutions too.

The setup in my POC project was very simple – it contained two features – one site collection scoped (Scope="Site") and one site scoped (Scope="Web"). The latter was a dummy feature, containing no feature elements – its purpose was to be “stapled” by the stapling feature. The stapling feature was the site collection scoped one (naturally) – it contained one feature element file with two “FeatureSiteTemplateAssociation” elements – both were targeting the standard “blank site” site definition (STS#1), the first one specifying the dummy site scoped feature in the same sandbox solution and the second one – the standard “WikiPageHomePage” feature (this is the standard feature that creates the wiki page library in the standard “team site” and sets its default page to be the “home” page in the wiki page library). I used two feature template associations, because I wanted to make sure that both sandbox and farm solution features get stapled by a “feature stapling” feature in a sandbox solution – which turned out to be exactly so.

And let’s see what the possible benefits from the feature stapling (site collection scoped) in a sandbox solution may be compared to the “normal” feature stapling in farm solutions – I can see these two positive/advantageous points:

• the first one stems from a general feature of the sandbox solutions – sandbox solutions can be installed and activated by site collection administrators (no need for farm administrator’s rights) and you don’t “pollute” the 14 hive with extra features.
• the second one is better isolation and manageability – if you have the feature stapling features in a farm solution with the “Hidden” attribute set to FALSE, so that site collection administrators can see them and manage them (activate/deactivate) in the standard site settings pages, the stapling features will appear in the settings pages of all site collections in the SharePoint farm. If you on the other hand decide to have the stapling features with the “Hidden” attribute set to TRUE, then the site collection administrators won’t be able to manage them using the standard SharePoint UI. And on the other hand with a stapling feature in a sandbox solution you can safely leave the feature to be visible, since this visibility will be limited only to the site collection in which you have the sandbox solution activated. And even if the stapling feature is hidden, the site collection administrator will still be able to deactivate the containing sandbox solution and thus deactivate the stapling feature itself.

And now let me make another comparison of the feature stapling (site collection scoped), this time with the WebProvisioned event receiver (which was introduced in SharePoint 2010 – check this posting of mine for more details about the WebProvisioned event receiver) – here are some of the differences between the two:

• With feature stapling you basically activate extra features to sites based on standard and custom site definitions and you configure the whole thing with XML in a feature element file, while with the WebProvisioned event receiver you write custom code which gets executed after the new site is created. In the WebProvisioned event receiver you can also activate features but you have to do that with code using the SharePoint object model.
• When you use feature stapling you can use feature properties for the features that you staple. If you want to activate features in the WebProvisioned event receiver with code, you cannot provide feature properties – there is no public method in the SharePoint object model that allows that.
• The lowest possible scope for feature stapling is the site collection level – meaning that all sub-sites in the site collection with the targeted site definitions will be affected. For the WebProvisioned event receiver you have the site collection scope but also the site scope. With the latter you target only the immediate children of the site to which you attach the WebProvisioned event receiver (this site may be the root site of the site collection but may also be a sub-site).
• The time of the activation of the stapled features and the time of the execution of the code of the WebProvisioned event receiver in the timeline of the site provisioning differs and this may seriously impact your solution depending on the specifics of your case. Here is the order of activation of the standard elements in the ONET.XML of a site definition/web template:
  
  1. site collection scoped features from the onet.xml
  2. site collection scoped stapled features
  3. site scoped features from the onet.xml
  4. site scoped stapled features
  5. “List” elements from the onet.xml
  6. “Module” elements from the onet.xml
  
  In the case of site collection scoped “feature stapling” features we can obviously staple only site scoped (Scope="Web") features, but as you see the time of execution of the site scoped stapled features is before the activation of the “List” and “Module” elements, which means that when your stapled feature gets activated your site is not fully provisioned and you may miss some of the SharePoint lists and some of the files/pages in the site depending on the site definition that is used for the site.  On the other hand the custom code of the WebProvisioned event receiver gets executed after the target site is fully provisioned and you will have all artifacts in the site already created which you can further modify/adjust with the code of the receiver.