Pluggable Content Query web part

Sub-classing the standard SharePoint ContentByQueryWebPart is a common approach for extending the default set of functionality that this web part offers. There are many examples for sub-classing the CQWP on the internet – for instance this sample by Andrew Connell that handles URL query parameters for adding dynamic filtering in the web part, or the paging enabled CQWP version by Waldek Mastykarz, just to name a few.

In most cases the need to sub-class the CQWP comes with some extra requirements for filtering or modifying the result set (SharePoint list items or documents) that the CQWP is supposed to present. The standard implementation of the CQWP of course has many filtering options itself but if you want some additional filtering based on dynamic conditions (as it is the case with the URL query parameters) you will have to consider the extending of the standard implementation by sub-classing the ContentByQueryWebPart class. As for the URL query parameters example, the SharePoint 2010 version of the CQWP has some limited built-in support for handling query parameters, but this still may be far from adequate in many cases.

I myself have sub-classed the CQWP in a handful of SharePoint projects of mine, so now I have maybe at least ten different implementations of cases where I had to handle some similar or not that similar requirements all based on the premises of having to have some sort of modification of the original CQWP result set. And some time ago I brain-stormed a bit the whole issue coming up with the idea that the whole problem can be handled in a bit more generalized way – if you for instance separate the web part implementation and the result set filtering/modifying logic implementation in a sort of pluggable architecture. Basically, the idea is pretty simple and further facilitated by the fact that you can have as many custom persistable properties in a web part as you need. The “plugin” can be as simple as a .NET class implementing a predefined .NET interface and the web part can be configured with a dedicated string property containing the fully qualified type name of this class, which will allow the web part to create instances of it using reflection. The web part will then be implemented to call on the methods of the plugin class which will be implementing the “contract” specified in the predefined .NET interface that I mentioned. The whole idea is that on the side of the extended CQWP there will be only a single implementation instead of many different ones for all possible cases that you may have and the whole effort would be for the part with the filtering/modification “plugin” class (classes) where it will be possible to concentrate on the retrieving/filtering specifics instead of having to pay attention to web part and presentation details.

And now to the more interesting part – how is it actually possible to hook onto the standard CQWP web part with your inheriting class so that you can modify the result set that the CQWP normally produces. Basically, there are a few options available here – it can be as simple as setting or modifying the values of the built-in filtering properties of the CQWP – for instance: FilterField1, FilterValue1, FilterType1, FilterOperator1, Filter1ChainingOperator, ListsOverride, QueryOverride, etc. Another approach is to use the handy ProcessDataDelegate property to which you can provide a delegate referencing a custom method receiving a DataTable parameter with a return type a DataTable as well. The idea of this delegate property is, that you can provide a custom method of yours to it, and it will be called with a DataTable parameter that will contain the freshly retrieved result set that the CQWP is about to display and you will have the chance to modify this DataTable instance (and its contents) or to even create a new DataTable instance and reuse, modify or filter the items from the original DataTable. The modified or the newly created DataTable then you will return from the custom method to the CQWP so that it goes to the presentation handling logic (check Waldek’s posting at the top for more details).

Having briefly mentioned these two approaches for implementing some extra filtering in the CQWP and before I continue with describing the actual approach in my implementation I think that it will be a good moment here to give you some more details on the internal workings of the CQWP and more specifically on its data retrieving logic so that you can better understand the whole hooking procedure in the inheriting “pluggable” CQWP. So, the first thing here, which is probably familiar to most of you is that the CQWP uses internally the  CrossListQueryCache and CrossListQueryInfo classes combo for retrieving aggregated cross list data. These two classes are actually a thin wrapper on top of the SPWeb.GetSiteData method and the SPSiteDataQuery class in which you provide the CAML query details for the cross list data call. What the CrossListQueryCache and CrossListQueryInfo classes add on top of the base implementation is some caching support and support for audience filtering and grouping. In theory this is pretty simple, the using of the cross list classes or the SPWeb.GetSiteData method is pretty common in many cases even not related to the CQWP at all. Apart from this there are several important facts giving some further details about the exact implementation of the cross list query invocation in the CQWP:

• The call of CrossListQueryCache.GetSiteData method that retrieves the list data for the CQWP takes place in the GetXPathNavigator virtual method (actually in a private method called by the latter) of the web part. Note that the GetXPathNavigator is a virtual method which means that you can override it thus it is an ideal candidate for hooking onto the base implementation of the CQWP, when it comes to modifying its data retrieval logic.
• The DataTable instance returned by the CrossListQueryCache.GetSiteData method is assigned to the Data public property of the CQWP. This property has a public getter and setter and even more importantly – the CQWP issues the CrossListQueryCache.GetSiteData call only if the Data property is not initialized and has the default null value.
• The CrossListQueryCache class requires an instance of the CrossListQueryInfo class in its constructor. The CrossListQueryInfo has properties like the Query, Lists, Webs, RowLimit, etc. in which you specify CAML fragments that determine what list data you want to retrieve from the target site collection. The CQWP creates and initializes a CrossListQueryInfo instance based on its filtering properties, some of which I mentioned above: FilterField1, FilterValue1, FilterType1, FilterOperator1, Filter1ChainingOperator, ListsOverride, QueryOverride, etc. The good news here is that the CQWP provides a public method which you can use to get a CrossListQueryInfo instance with the exactly same configuration as the one that the web part uses for its internal cross list query call. This method is BuildCbqQueryVersionInfo – it actually returns an instance of the CbqQueryVersionInfo class whose CbqQueryVersionInfo.VersionCrossListQueryInfo property contains the CrossListQueryInfo instance that we need.
• And lastly – the call to the delegate ProcessDataDelegate property that I mentioned above also takes place in the CQWP’s GetXPathNavigator virtual method. Actually this happens at the same moment when the web part gets the list data in the DataTable instance from the CrossListQueryCache.GetSiteData call.

With all these facts it is easy now to devise a strategy for implementing the hooking logic in the inheriting class. It is obvious that the whole logic can be placed in the overridden version of the GetXPathNavigator method, just before you call the base CQWP implementation (base.GetXPathNavigator();) in the method body. For the plugin logic I thought of three alternative types of overriding the standard CQWP data retrieving and filtering logic:

• The first type of override is the most radical one – you directly circumvent the standard logic for retrieving the list data in the CQWP and produce your own DataTable instance in some custom way. The only thing that you will need to do after producing the DataTable instance is to assign it to the CQWP’s “Data” property. This way you will ensure that when you call the base GetXPathNavigator method, the CQWP won’t issue its standard cross list query call, thus there won’t be any performance issues because of the double data retrieval. As of how you are going to get your DataTable instance – you have many possible choices here depending on your particular case. I can think of for instance using the SharePoint search functionality, using data in an external SQL database or even using again the CrossListQueryCache.GetSiteData method but applying some custom caching logic instead of relying on the built-in one if it doesn’t fit in your requirements.
• The second type of override is to manipulate just the CrossListQueryInfo instance that will be used for the CrossListQueryCache.GetSiteData call. This instance will contain the initial configuration of the web part and you can add some additional filtering or scoping logic to it based on your requirements. Then the inheriting web part will issue the call to the CrossListQueryCache.GetSiteData method and will again set the CQWP’s Data property, so that the base implementation doesn’t make a second call. Note also that in this type of overriding the CQWP’s filtering you have to deal only with a CrossListQueryInfo instance and manipulate the CAML fragments in its properties (merge some extra filtering CAML, etc.) instead of having to modify directly the CQWP’s properties like the FilterField1, FilterField2, FilterField3, etc. which the end user may have already set values to using the SharePoint UI.
• The third type of override is to simply use the standard delegate ProcessDataDelegate property override mechanism – the CQWP will make the cross list query call itself and we will only manipulate the DataTable instance (or create a new instance using the rows from the latter) that it has already retrieved.

And now, let’s move on to some code snippets which will better illustrate the implementation of the pluggable CQWP. You can download the project containing the pluggable CQWP from here. Let me first show you the code of the .NET interface that should be implemented by the custom plugin classes for the pluggable CQWP:

    public interface IContentQueryPlugin

    {

        // should return true if the pluging can generate the data for the web part - the GetDataResults method should produce a DataTable instance

        bool CanGetDataResults(PluggableContentByQuery part);

        // should return true if the plugin needs to modify the CrossListQueryInfo instance that the CQWP uses for its internal cross list query - the ProcessQueryInfo should be implemented in this case

        bool CanProcessQueryInfo(PluggableContentByQuery part);

        // should return true if the plugin needs to modify the DataTable returned by the CQWP cross list query - the ProcessData method should be implemented

        bool CanProcessData(PluggableContentByQuery part);

 

        // implement this if you want the retrieve the data without using the CQWP built-in retrieval mechanism

        DataTable GetDataResults(PluggableContentByQuery part, CrossListQueryInfo queryInfo);

        // implement this if you want to modify the CrossListQueryInfo instance before the CQWP issues its cross list query

        CrossListQueryInfo ProcessQueryInfo(PluggableContentByQuery part, CrossListQueryInfo queryInfo);

        // implement this if you want to modify the already retrieved by the CQWP DataTable with the results of the cross list query

        DataTable ProcessData(PluggableContentByQuery part, DataTable data);

    }

As you see the interface contains six methods, but actually the three types of overrides that I mentioned above are implemented by only three of the methods: GetDataResults, ProcessQueryInfo and ProcessData. The other three methods are sort of auxiliary methods in the sense that you have one auxiliary method corresponding to one of the implementation methods – the auxiliary methods all start with the verb “can” prefix. Their purpose is pretty transparent – since the types of overrides that you can implement are optional (and even mutually exclusive – the first one versus the second two) with the auxiliary methods you can simply specify whether you are implementing a particular override. For instance if you want to take care of retrieving the data results yourself you are going to implement the GetDataResults method and also you will have to implement the CanGetDataResults auxiliary method by simply returning “true” in its implementation (for the other “can” methods you will have to return “false”).

Let’s now have a closer look at the CQWP overrides implementing methods:

• the GetDataResults method – as I mentioned you go for this method if you want to have your custom data retrieving logic and circumvent the CQWP’s normal data retrieving routine altogether. As you see, the method’s return type is a DataTable, meaning that you will have to gather your result items in some custom way and then produce a DataTable instance that you will pass back to the CQWP. The method accepts two parameters – a reference to the “parent” CQWP and a CrossListQueryInfo object. The latter will contain all CAML fragments for the cross list query that the web part would have issued – you can use that for the filtering against your custom data source if you use one. Further you can use the reference to the web part to inspect the properties of the latter and make use of their values in some way or another.
• the ProcessQueryInfo method – you can use this method if you want to only add some additional CAML markup or make some other modifications to the CrossListQueryInfo object that the CQWP constructs for its cross list query call. This means that you are opting only for some additional filtering but the data retrieval method remains the standard one for the CQWP. The method accepts CrossListQueryInfo parameter and also returns a CrossListQueryInfo object. You can either make your modifications to the input CrossListQueryInfo instance and then return the same instance, or you can optionally create a brand new CrossListQueryInfo instance, copy some or all of the properties of the source instance and then return the new one.
• the ProcessData method – if you opt for this override it will mean that you don’t want the change the CAML filtering and will only want to modify the DataTable instance that the CQWP produces with its standard cross list query call. You can perform all sorts of changes to the DataTable instance like adding new columns to it, add or remove data rows, etc. The removing of data rows will effectively be equivalent to applying some extra filtering, which in many cases will be better if implemented in CAML, meaning that probably there will be a better solution if using the ProcessQueryInfo to inject the filtering in the CrossListQueryInfo’s CAML.

Let me now show you how the IContentQueryPlugin interface fits in the overriding procedure of the CQWP. As I mentioned above, the only hooking point that we need to change the data retrieving logic of the CQWP is the virtual GetXPathNavigator method, and this is its overridden version in the pluggable CQWP:

        protected override XPathNavigator GetXPathNavigator(string viewPath)

        {

            // This method is the single point where we need to hook to get the whole plugin thing working.

            // The standard CQWP issues the cross list query in its implementation of the GetXPathNavigator method

            // setting the CQWP's Data property with a DataTable instance holding the results of the query.

            // The CQWP issues its query only if the Data property is null, so we can even set it with

            // a customly retrieved DataTable if we want before calling base.GetXPathNavigator(viewPath).

 

            // first get the plugin instance

            IContentQueryPlugin plugin = this.GetQueryPlugin();

            if (plugin != null)

            {

                // check if the plugin wants to generate the data results first

                if (plugin.CanGetDataResults(this))

                {

                    // if so, provide it with a CrossListQueryInfo instance and set the Data property of the web part

                    CrossListQueryInfo queryInfo = this.BuildCbqQueryVersionInfo().VersionCrossListQueryInfo;

                    this.Data = plugin.GetDataResults(this, queryInfo);

                }

 

                // then check if the plugin wants to modify the original CrossListQueryInfo instance (check this only if the Data property is still null)

#if SHAREPOINT2010

                if (this.Results == null && plugin.CanProcessQueryInfo(this))

#else

                if (this.Data == null && plugin.CanProcessQueryInfo(this))

#endif

                {

                    // get the CrossListQueryInfo instance and pass it to the plugin

                    CrossListQueryInfo queryInfo = this.BuildCbqQueryVersionInfo().VersionCrossListQueryInfo;

                    queryInfo = plugin.ProcessQueryInfo(this, queryInfo);

                    // after this run the cross list query to populate the Data property

                    IssueQuery(queryInfo);

                }

 

                // set the CQWP ProcessDataDelegate property if the plugin needs to post-process the results DataTable

                if (plugin.CanProcessData(this))

                {

                    this.ProcessDataDelegate = (dt) =>

                    {

                        return plugin.ProcessData(this, dt);

                    };

                }

            }

 

            // only after we finish the plugin stuff call the base CQWP GetXPathNavigator implementation

            return base.GetXPathNavigator(viewPath);

        }

You can see that the implementation is actually pretty concise and straight-forward. Basically the implementation of the overriding GetXPathNavigator method simply dispatches the calls for modifying the data retrieving logic of the CQWP to the plugin class that implements the IContentQueryPlugin interface. The creating of the plugin class instance is also a pretty trivial thing – the pluggable CQWP has a custom string property called PluginClassName, which as its name suggests should be set to contain the fully qualified type name (plus the full assembly name if the plugin class is not in the same assembly as the assembly of the pluggable CQWP) of the plugin class. The pluggable CQWP creates the plugin class instance using reflection (there’s also the option to cache the plugin class instance – check the source code for details). Note also that you can compile the web part for SharePoint 2010 too but you will need to define an extra Debug symbol in the settings of the Visual Studio project of the web part (check the conditional compiling statements in the source code).

The Visual Studio project with the pluggable CQWP (download from here) also contains a sample plugin class implementing the IContentQueryPlugin interface. It implements only the IContentQueryPlugin.ProcessQueryInfo meaning that it only modifies the original CAML of the CQWP’s cross list query call. This is the source code of its implementation of the ProcessQueryInfo method:

        public CrossListQueryInfo ProcessQueryInfo(PluggableContentByQuery part, CrossListQueryInfo queryInfo)

        {

            // check the FilterFieldN and FilterValueN query parameters 0 through 9

            string[][] args = (new int[10]).Select((i, idx) => new string[] { HttpContext.Current.Request.QueryString["FilterField" + idx], HttpContext.Current.Request.QueryString["FilterValue" + idx] })

                .Where (arr => !string.IsNullOrEmpty(arr[0]) && !string.IsNullOrEmpty(arr[1]))

                .ToArray();

            // if none found or their values are empty just exit

            if (args.Length == 0) return queryInfo;

 

            // get the CAML query in an XElement

            XElement queryElement = XElement.Parse("<Query>" + queryInfo.Query + "</Query>");

            // retrieve the Where clause or create a new one if missing

            XElement whereEelement = queryElement.Element("Where");

            if (whereEelement == null)

            {

                whereEelement = new XElement("Where");

                queryElement.Add(whereEelement);

            }

 

            // get the existing where clause if present and append all comparison clauses from the query parameters

            string[] clauses = whereEelement.Elements().Select(el => el.ToString())

                .Union(args.Select(arg => FormatComparisonQuery(arg[0], arg[1], "Eq")))

                .ToArray();

 

            // change the contents of the Where XElement performing a CAML 'And' on the clauses array

            whereEelement.RemoveAll();

            whereEelement.Add(XElement.Parse(FormatLogicalOpQuery("And", clauses)));

 

            // set the Query property of the CrossListQueryInfo instance

            queryInfo.Query = string.Concat (queryElement.Elements().Select(el => el.ToString()).ToArray());

            // return the modified CrossListQueryInfo

            return queryInfo;

        }

Some bits of the full implementation are missing (there are also several helper private methods in the plugin class – check the source code in the project for details) but you can see that the method simply “injects” some extra CAML in the Query property of the source CrossListQueryInfo parameter (the Query property may already contain some CAML that comes from the particular query configuration of the CQWP). The additional filtering that the sample plugin class is dynamically determined by the presence of certain URL query parameters that you use to call the page containing the web part. Actually these are no other than “FilterField1” and “FilterValue1”, the same as the standard LV and XLV web parts use and also the same as Andrew Connell uses in his sample of sub-classing the CQWP that I mentioned in the beginning.

In order that you can see the plugin class working with the pluggable CQWP you will need to set its fully qualified type name in the PluginClassName property of the web part. You can easily do that from the standard SharePoint UI – you will have to put this value for the sample plugin class: “Stefan.SharePoint.WebParts.QueryStringPlugin, Stefan.SharePoint.WebParts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=f9bc508a219e6843”.