Stefan Stanev's SharePoint blog

SharePoint WCM HTML clean-up

2012-01-20T11:43:00.001-08:00

SharePoint is a great web content management system, it is fast, scalable, reliable, comes with lots of out-of-the-box components, web parts, etc. that often make the life of the content managers much easier. There are certain aspects of the WCM capabilities of SharePoint though that sometimes need a little more time or some hacking to get them to work properly, or the way you may want them to work. One such thing is the HTML code that appears on your SharePoint pages - there are several things in the HTML generated by the SharePoint WCM system which make it look not quite neat and tidy. The SharePoint UI is built on top of the asp.net Web Forms technology, so SharePoint actually inherits some of the HTML issues directly from its asp.net foundation. The problem here is that when you use asp.net and Web Forms you don't have full control over the HTML that is going to be generated in your page. With the advent of the asp.net MVC this was one of the arguments in favor of the latter, because with asp.net MVC the developer indeed has full control over the generated HTML code. Unfortunately SharePoint doesn't utilize the MVC framework, so many of us at one or another point have had to struggle with the extra HTML bits that get injected in the SharePoint aspx page. Some examples for such bits that come directly from asp.net are the many system hidden fields that appear in the "form" element, the infamous "ViewState" field among them which can grow very big in size, the inline JavaScript blocks with "form" submit helpers, etc. Several intrinsically SharePoint items that further inflate your HTML are for instance the inclusion of the two "core" files: "core.js" and "core.css" (quite big both of them), the many nested HTML "table" elements around your web parts which are rendered by the containing WebPartZone controls, especially in cases when you want your HTML to contain only nice looking "div" elements, etc. The dilemma here is that because SharePoint utilizes in-place page content editing and it is a single aspx file that handles both the editing process and the actual displaying of the page to the end user, the items (web controls in most cases) responsible for these extra (but necessary) HTML artifacts cannot be removed directly from the page. So, we need them for the page content editing, but on the other hand we need to somehow get rid of them, or hide them, or at least suppress the extra HTML that they generate so that the page in display mode shows only the bare minimum of HTML that needs to render the page contents. In the WCM context, I assume here that the SharePoint site is publicly accessible or at least allows anonymous access within some internal network, so the hiding of the extra artifacts is necessary only when the pages are being accessed anonymously. This is a pretty broad scenario and this particular setup is quite popular in the WCM function of SharePoint. The next question is how many of the "extra" HTML SharePoint artifacts may be unwanted in your scenario. If it is about simple content pages with SharePoint field controls only or standard content editor web parts you actually won't need any of the above mentioned bits in display mode with anonymous access. This is especially true when your HTML design is very different from the standard SharePoint page design.
So, after several years and several partial solutions I decided to wrap up the whole thing in a single solution. And it turned out that the solution was pretty easy and simple to develop, and luckily - very easy to use too. It is actually a single user control that you need to place in one place only in your master page. And that's all. The control has several public properties that can be used to configure it, so that it suppresses some of the SharePoint artifacts that it can handle but not others (I will explain these in detail shortly). I chose to create the control as a user control (and there is no code behind assembly, the code is placed inline in the ascx file directly) because this way you have the two deployment options - to either place it in the TEMPLATE/CONTROLTEMPLATES folder of the SharePoint hive, or to upload it to the Master Page Gallery of your site and reference it in your master page from there (I explained this technique in this recent posting of mine). Of course, the code can be easily transferred to a simple web control and put in an assembly of yours.
You can download the user control that I named appropriately "HideSPArtifacts.ascx" from here.
Provided that you have uploaded it to the Master Page Gallery of your site collection (/_catalogs/masterpage) you will need the following lines of code to place it in your master page file:
First you need the "Register" directive at the top of your master page

<%@ Register="" TagPrefix="MyControls" TagName="HideSPArtifacts" Src="~SiteCollection/_catalogs/masterpage/HideSPArtifacts.ascx" %>

The second bit is to place the control declaration in the page mark-up:

<MyControls:HideSPArtifacts runat="server" RemoveCoreJS="false" RemoveCoreCss="false" RemoveHeadCss="false" RemoveForm="false" AddBodyOnLoadDummy="false" EnablePageViewState="true" RemoveZoneHeaders="false"/>

Two very important notes here: 1) if you upload the user control to the Master Page Gallery of your site collection you will have to make additionally certain modifications to your web.config file (check the previous posting that I mentioned above). 2) You need to place the control's declaration (MyControls:HideSPArtifacts) immediately after the opening "html" element of your page and before the "head" HTML element.
One other thing that you should check in your master page is whether you have a "head" element and whether it has the runat="server" attribute (if you have used one of the SharePoint master pages as a base for your master page you will have these). If this condition is not satisfied the control won't be able to remove some of the SharePoint artifacts from the page.
So, after you have the user control in your master page and open a page from your site anonymously (if you view the page as authenticated user the control will do nothing) and you have the values of the control's properties as they are in the snippet above you will see ... no changes in the HTML code of the page. This is because all "Remove.." properties are set to "false". I will now give you a list with the properties of the control and will briefly explain what changes in the generated HTML you will see after setting each property:

RemoveCoreJS - as the name suggests, if this property is set, the include script declaration for the SharePoint's "core.js" file is removed from the page
RemoveCoreCss - when set, this property causes the core.css style sheet include to be removed from the page. Note that if you use alternate style sheets, these won't be rendered either. This is because the HideSPArtifacts control will block the rendering of the standard SharePoint CssLink control (if available).
RemoveHeadCss - when you use certain web controls like the TreeView control, the AspMenu control and some other controls, the asp.net page generates an inline CSS block in its "head" HTML element. If you don't want this inline style sheet to appear in the page (check if this doesn't affect any of the controls that you use on the page), set this property to true.
RemoveZoneHeaders - the WebPartZone controls that contain your web parts have the bad habit of creating several nested "table" HTML elements. The web parts' chrome (frame) which you most often set to "none", because you don't need it in WCM public sites also renders a "table" element. If you don't want any of these "table" elements and want to have only the HTML markup directly rendered by your web parts set this property to true. Note that even if you set the "ChromeType" property of the web part, no chrome will be actually rendered (in anonymous mode only).
EnablePageViewState - the default value of this property is true and in this case the control will change nothing on the page. If you set this property to false it will simply set the EnableViewState property of the containing page to false (only when the page is viewed anonymously). The net effect will be that ... you will still have the "ViewState" hidden field in your page, but it will contain only thirty or so bytes of data.
RemoveForm - when set this property removes the "form" element from your page. Actually it does something much more radical - it removes also all system hidden fields including the ViewState field and all inline JavaScript blocks that were included using the methods of Page.ClientScript - e.g. RegisterClientScriptBlock, RegisterStartupScript, etc. You will get rid of a ton of HTML and JavaScript in your page, which you wouldn't need if you don't have controls and logic that need to do POST submits of the page. If your pages (at least the pages using the master page with the HideSPArtifacts control) contain only SharePoint field controls and content editor web parts this will be a perfect choice. Note however that you will need to carefully check all your pages - some controls (like Button, LinkButton, etc) directly crash if there is no "form" rendered on the page. Other controls may stop function properly because they will miss some JavaScript code that won't get rendered. Bottom line - use cautiously.
AddBodyOnLoadDummy - the default value of this property is true. It has a visible effect only when the "RemoveForm" property is set to true. It adds a small JavaScript block with several empty JavaScript functions. One of these functions is called "_spBodyOnLoadWrapper". This function appears in the "onload" attribute of the "body" element of the default SharePoint master page. Since the "RemoveForm" property removes all inline JavaScript associated with the page's "form" element, the real definition of this JavaScript function won't be available on the page, and you will see a JavaScript error in your browser when the page loads. This is the reason why this property causes the adding of this small JavaScript block with empty definitions of this and two other system JS functions.

And now, let me briefly explain how the trick with hiding a control without hiding its contents is possible. Actually the idea is to hide the control itself (or at least parts of it) but display its child controls. This technique is used in the implementation of the "RemoveForm", "RemoveHeadCss" and "RemoveZoneHeaders" properties. The following steps are executed:

in its OnInit override the HideSPArtifacts hooks onto the parent page's InitComplete event
in the InitComplete event handler an empty control (class Control) is created in the "Controls" collection of the parent control of the control which we want to hide. The new control is inserted in the "Controls" collection of the parent control right after the target control.
The SetRenderMethodDelegate method of the new empty control is called - this method receives a single delegate parameter, which you use to provide a method to be called right after the control's "Render" method exits. The idea is to use the empty control as a place-holder to inject some HTML right after the control that we want to hide.
In the "Render" method override of the HideSPArtifacts control the "Visible" property of the control that we want to hide is set to false. Since we have placed the HideSPArtifacts right after the beginning of the master page its Render method is guaranteed to be called first in the child controls' chain. This way the control whose Visible property is set to false will not get rendered.
in the render method passed as the render delegate parameter of the SetRenderMethodDelegate method, the Controls collection of the target control is iterated and all child controls are rendered using the Control.RenderControl method. This way we have the target control itself not rendered but all its children get actually rendered within the empty control that was injected right after it. This is how the goal of hiding the control itself but not its child controls is achieved.

You can check the source code of the HideSPArtifacts control for the details of the actual implementation.

Dynamic class system in C# 4.0

2011-11-26T09:08:00.001-08:00

The "dynamic" type (rather pseudo-type or keyword) is a new feature in the latest version of C# - 4.0. Its purpose is to enable an easier interoperability with dynamic type languages and COM components and is part of the new DLR system introduced in the .NET Framework 4.0. In addition to the new "dynamic" type there are two interesting new built-in .NET classes that allow you to do some sort of dynamic type programming with C# - the ExpandoObject class and the DynamicObject class. Using the "dynamic" type/keyword and the ExpandoObject you can do things like this:

dynamic sampleObject = new ExpandoObject();

sampleObject.number = 10;

sampleObject.Increment = (Action)(() => { sampleObject.number++; });

sampleObject.Increment();

You can add dynamically member variables to the "dynamic" instance and also methods by assigning delegates or anonymous methods to instance members (you can also assign and use events in the same manner). These tricks are possible and the C# compiler doesn't raise compilation errors exactly because of the "dynamic" keyword. It ensures that these assignments, member variable usages and method invocations are not checked during compile time but all that happens at run time. The dynamic dispatching semantics are implemented by the ExpandoObject class. It implements a special .NET interface called IDynamicMetaObjectProvider which is used as a bridge by the DLR for the correct routing of all these dynamic invocations and references at run time. It is possible to create custom classes implementing this interface but it is actually easier to do that by inheriting the second class that I mentioned - the DynamicObject class. The DynamicObject class is an abstract class and if you inherit your class from it you will be able to easily implement your custom dynamic member dispatching logic by overriding one or more of its virtual methods.
I will not go into further detail about the "dynamic" type/keyword and the ExpandoObject as this is quite a broad subject and there are many in depth articles already available in MSDN. Instead I will go directly to the subject of this posting which is the simple dynamic class system for C# 4.0 that I implemented.
First, let me briefly explain as to why I came with the idea of implementing a dynamic class system in C#. The main reason was to do it just for fun and to prove (at least to myself) the expression power of C#. As a side note this means that this implementation is not intended for any serious usage, I coded the whole thing in just a couple of hours, so it lacks any performance optimizations whatsoever, it doesn't support multi threading and is not thoroughly tested. So, basically the answer to the question, why would there be any need to add dynamic classes support to a powerful static type language like C# is the plainly sounding - just to demonstrate that it is possible.
And the immediate reason to do this was that when I was reading the MSDN documentation about the ExpandoObject class I started wondering why they made it only possible to add dynamically members to a single instance/object only instead of providing support for dynamic classes with which to enable the creation of many instances having a predefined set of members simultaneously. Of course, this can be somehow emulated with the ExpandoObject itself - one only needs to create a "factory" method that creates and returns instances of the ExpandoObject "initialized" with the desired set of members but I wanted something more coherent and self-contained. And also some real class support with at least some of the standard class system features like inheritance, polymorphism, etc.
You can download the implementation of the dynamic class system from here - this is a small console application that defines two public classes - "Class" and "ClassDefinition" (quite descriptive names) which contain the whole logic of the dynamic class system. The application contains also a big demonstration section with lots of verbose comments describing the details of the usage of the dynamic classes - I will have this entire section pasted at the end of this posting, so that you can get an idea of the syntactic peculiarities of these "dynamic classes". But before doing that let me quickly list the main features of this custom dynamic class system:

uses only C# 4.0 syntax without any external or extra language declarations or configurations
supports simple class inheritance - a class can inherit only one base class
classes are defined with imperative statements (as opposed to the declarative syntax for normal/static C# classes)
supports simple polymorphism - all methods are treated as virtual
supports class constructors and constructor overloading
supports method overloading - you can have two or more member methods with the same name but with different set of parameters
supports the "self" keyword for referencing member variables/methods from member methods (analogue to the C#'s native "this" keyword)
supports the "super" keyword for referencing methods defined in the parent class (if available)
objects (class instances) are "open" - you can initialize and use additional member variables and methods that are not defined in the class (similarly to the objects in JavaScript and the ExpandoObject itself)
class definitions are also "open" - you can extend the definition of any class at any later moment adding new member variables and methods (slightly influenced by Ruby). The new methods are immediately available in all class instances even those created before the extension of the class definition.
relaxed rules for uninitialized member variables and methods. You can reference not initialized member variables and this will not raise an error - but the value returned will be null. You can call not initialized (not existing) methods and this will not raise an error either - the invocation will do nothing and will return null.
doesn't support events
doesn't support access modifiers - all members are treated as public and accessible from any context

And here is the demonstration section describing the "syntax" of the dynamic class system:

// 'Class' is a static class with one static method - you can define dynamic classes with it: Class.DefineClass

// you specify the class name, the base class (null for none) and an Action delegate (lambda expression) that defines the class' members

Class.DefineClass(new ClassDefinition()

{

    ClassName = "BaseClass", 

    Variables = (def) =>     {

         // two member variables

         def.Register1 = 1;

         def.Register2 = 2;

     },

     Methods = (def, self) =>     {

         // the Init method is used as a constructor

         def.Init = new Action<int, int>((a, b) => { self.Register1 = a; self.Register2 = b; });

         // define an Add method

         def.Add = new Func<int, int, int>((a, b) => a + b);

         // define an overload of the Add method that sums the member properties Register1 and Register2 - note the usage of the "self" lambda parameter

         def.Add = new Func<int>(() => self.Register1 + self.Register2);

         def.ProxyAdd = new Func<int>(() => { Console.Write("BaseClass.ProxyAdd calling Add(): "); return self.Add(); });

     } 

});

        

// now have a look at the static property "New" of the Class class, which is of type ... dynamic

// and we can call methods on this "New" dynamic object which will return instances of our custom dynamic classes with the same name // let's create an instance of the dynamic class Test calling its constructor that accepts a string paramter

// but ... we haven't defined a class named "Test" let alone a constructor of the "Test" class

// still the local variable c1 gets initialized with a "dynamic" instance which behaves pretty much like the standard ExpandoObject class

dynamic c1 = Class.New.Test("test");

// it doesn't have any properties but we can add one

c1.var1 = "test";

// this will print "test"

Console.WriteLine(c1.var1);

// what about the "var2" property - it hasn't been initialized so c1.var2 is simply null

Console.WriteLine(c1.var2 ?? "this is null");

// we don't have any methods in c1, but we can call whatever methods we want - they will all return null (and will do nothing of course)

Console.WriteLine(c1.test(1, 2) ?? "this is null");

// let's now create an instance of the "BaseClass" class, which we have already declared. This would call the parameterless Init method of the class (if it existed)

dynamic c2 = Class.New.BaseClass();

// call its Add(int, int) method with two integer parameters - result is 2 + 2 = 4

Console.WriteLine(c2.Add(2, 2));

// call the Add() overload with no parameters - this sums the object's properties Register1 and Register2 which have their initial values: 1 + 2 = 3

Console.WriteLine(c2.Add());

// let's create another instance of the BaseClass class. This will call its Init(int, int) method that takes two integer parameters.

dynamic c3 = Class.New.BaseClass(7, 8);

// calling the parameterless Add() overload will sum the Register1 and Register2 properties which were initialized to 7 and 8 respectively: 7 + 8 = 15

Console.WriteLine(c3.Add());

// the same: 7 + 8 = 15

Console.WriteLine(c3.ProxyAdd());

// now let's define a new class that inherits the BaseClass class

Class.DefineClass(new

ClassDefinition() {

     ClassName = "DerivedClass",

     BaseClassName = "BaseClass",

     Variables = (def) =>     {

         // declare a new property

         def.Register3 = 3;

     },

     Methods = (def, self) =>     {

         // declare a constructor that accepts three integers

         // note the usage of the self.super expression - we can call methods from the base class this way (not necessary in this case since the base method accepts two parameters and the new method - three, so the new method doesn't hide the overload in the base class and it can be called simply with self.Init(int, int))

         // note that the constructor of the base class is called explicitly

         def.Init = new Action<int, int, int>((a, b, c) => { self.super.Init(a, b); self.Register3 = c; });

         // redefining the parameterless Add() method, this one will hide the parameterless Add() method in BaseClass - to call the base class implementation we need to use self.super.Add()

         // we call both Add overloads of BaseClass - first to sum Register1 and Register2 and then to sum the result with Register3

         def.Add = new Func<int>(() => self.super.Add(self.super.Add(), self.Register3));

     }

});

// here is an instance of the DerivedClass class (no constructor will be called since we don't have a parameterless Init() method)

dynamic c4 = Class.New.DerivedClass();

// this will invoke the DerivedClass.Add() implementation which sums Register1, Register2 and Register3: 1 + 2 + 3 = 6

Console.WriteLine(c4.Add());

// this will invoke the BaseClass.Add method(int, int) with two integer parameters (it hasn't been redefined in DerivedClass): 4 + 4 = 8

Console.WriteLine(c4.Add(4, 4));

// create an instance of DerivedClass ... using BaseClass.Init (int, int), which hasn't been redefined in DerivedClass

dynamic c5 = Class.New.DerivedClass(10, 11);

// calling DerivedClass.Add() will return: 10 + 11 + 3 = 24

Console.WriteLine(c5.Add());

// create another instance of DerivedClass - this will call Init(int, int, int) which was declared in DerivedClass

dynamic c6 = Class.New.DerivedClass(100, 200, 300);

// calling DerivedClass.Add() will return: 100 + 200 + 300 = 600

Console.WriteLine(c6.Add());

// calling BaseClass.ProxyAdd which calls Add() which is the DerivedClass.Add() version since c6 is DerivedClass

Console.WriteLine(c6.ProxyAdd());

// Declare the DerivedClass class again? No, this will only extend the definition of DerivedClass - additional members can be added. No need to specify the base class again.

Class.DefineClass(new ClassDefinition {

     ClassName = "DerivedClass",

     Variables = (def) =>      {

         // add a new member property

         def.Register4 = 4;

     },

     Methods = (def, self) =>     {

         // add a new constructor accepting 4 integer parameters - this one calls the constructor (with three integer parameters) from the first DerivedClass declaration

         def.Init = new Action<int, int, int, int>((a, b, c, d) => { self.Init(a, b, c); self.Register4 = d; });

         // redefine again the parameterless Add method - this one will sum all properties: Register1, Register2, Register3 and Register4

         // note the tree calls to self.super.Add - two to self.super.Add(int, int) and one to self.super.Add()

         def.Add = new Func<int>(() => self.super.Add(self.super.Add(), self.super.Add(self.Register3, (self.Register4 ?? 0))));

     }

});

// the c6 instance was created before the extending of the DerivedClass definition - but it will use the new version of DerivedClass.Add()

// it doesn't have the Register4 property initialized - it's value will be null (the Add() will add it as 0)

// 100 + 200 + 300 + 0 = 600

Console.WriteLine(c6.Add());

// set Register4 explicitly

c6.Register4 = 400;

// 100 + 200 + 300 + 400 = 1000

Console.WriteLine(c6.Add());

// again 100 + 200 + 300 + 400 = 1000

Console.WriteLine(c6.ProxyAdd());

// c7 is created calling the constructor DerivedClass.Init(int, int, int) declared in the first definition of DerivedClass

dynamic c7 = Class.New.DerivedClass(1000, 2000, 3000);

// its Add method will now (as defined in the extension of DerivedClass) sum Register1, Register2, Register3 and Register4: 1000 + 2000 + 3000 + 4 = 6004

Console.WriteLine(c7.Add());

// c8 is created with the constructor DerivedClass.Init(int, int, int, int) defined in the second definition (extension) of DerivedClass

dynamic c8 = Class.New.DerivedClass(1000, 2000, 3000, 4000);

// its Add method will now sum Register1, Register2, Register3 and Register4: 1000 + 2000 + 3000 + 4000 = 10000

Console.WriteLine(c8.Add());

// we can call the parameterless constructor without the parenthesis

dynamic c9 = Class.New.DerivedClass;

// it's possible to call the Init method afterwards

c9.Init(10000, 20000);

// let's check that it's a different instance from c9 - see below

dynamic c10 = Class.New.DerivedClass;

Console.WriteLine();

// a built-in diagnostics method - dumping all member properties of the dynamic objects

Console.WriteLine(c1.dump());

Console.WriteLine(c2.dump());

Console.WriteLine(c3.dump());

Console.WriteLine(c4.dump());

Console.WriteLine(c5.dump());

Console.WriteLine(c6.dump());

Console.WriteLine(c7.dump());

Console.WriteLine(c8.dump());

Console.WriteLine(c9.dump());

Console.WriteLine(c10.dump());

Deploy user controls in the SharePoint content database

2011-10-30T04:30:00.000-07:00

Normally user controls (*.ascx files) in SharePoint are deployed with farm solutions and the preferred system (hive) folder for that is the CONTROLTEMPLATES one. User controls can be used for various purposes – for form templates in SharePoint lists, with the new visual web part in SharePoint 2010 or simply to provide reusable visual bits that can be placed in more than one master page files of page layouts. User controls can be quite handy since they provide a nice separation of the presentation logic which is not directly available in the regular web controls – you can use the WebForms markup to produce your HTML output instead of having to deal with that in the code itself.

And to the subject of this posting – is it possible to have user controls (*.ascx) files directly in the content database of your SharePoint web application (site) instead of in the SharePoint hive folder (12/TEMPLATE/CONTROLTEMPLATES or 14/TEMPLATE/CONTROLTEMPLATES). The answer to this question is yes and I am going to demonstrate that shortly. But before that I would want to briefly discuss the motives and reasons that could justify the usage of user controls in the content database and also some possible advantages and disadvantages of this approach. This topic is actually a bit broader than suggested by the posting’s title and it is about the possibility of having a SharePoint custom application that doesn’t use custom assemblies and files deployed to the SharePoint hive folder. Sounds a bit like the new sandbox solutions available in SharePoint 2010 – this is to some extent so, though the code that can be executed is the normal farm version of the SharePoint object model. Unlike the sandbox solutions, there is also a big security implication that I will discuss shortly. The main motivation here is to have a way to place your code directly in your master pages or page layouts by using say the SharePoint designer. So, opposed to the usual way of deploying SharePoint farm solutions with “wsp” files, this is sorts of “SharePoint designer” development and deployment (the boundary between development and deployment with the SharePoint designer is almost non-existing). The other advantage here is that you can push all you code and code updates using the simple content deployment paths and jobs built-in functionality. The bottom line here is that if you use extensively the SharePoint designer and have a SharePoint environment with a publishing and production server using content deployment paths for content synchronization you can consider using this approach.

As for the question of how to use code directly in your master page files and page layouts (maybe you’ve done that many times already with the SharePoint designer) – the answer is simple – inline code blocks:

<script runat="server">

Two big notes here – the first one is the security issue that I mentioned above. Since it is very easy to insert a code block in a page using the SharePoint designer, there is an internal protection in SharePoint – this is the so called “safe mode” for parsing and processing of un-ghosted pages (pages that are in the content database). By default if you have a code block in a page that is un-ghosted (or was created directly in the content database) you will receive an error if you try to open the page. This can be overridden by a setting in the web.config file:

<PageParserPaths><PageParserPath VirtualPath="/_catalogs/masterpage/*" CompilationMode="Always" AllowServerSideScript="True" IncludeSubFolders="true" />
</PageParserPaths>

but keep in mind that this can be a serious security hole. Note the value of the “VirtualPath” attribute containing the location of the Master Page gallery with a wild card meaning that all your master page files and page layouts will be allowed to have code blocks. If you have a site collection under this server relative URL – “/sites/test-site” you will have to specify the full path to its Master Page gallery: “/sites/test-site/_catalogs/masterpage/*” (unless you don’t want to put something as unsecure as “/*”). For a detailed treatment of the SharePoint “safe mode” page processing, the configuration of the “PageParsePath” elements (and also of the “SafeControl” elements) check this MSDN article.

The second note is that the usage of code blocks as opposed to having the code in a code-behind assembly is not considered the very best and recommendable code practice. Apart from that the SharePoint designer is far from Visual Studio in terms of providing good IDE support for code development.

The placing of code blocks inside master pages and page layouts is maybe nothing new for most of you but the main idea of this posting is to demonstrate the using of user controls inside master pages and page layouts. And it will be the user control that will contain the inline code block in this case. Remember that the ascx file of the user control will reside in the content database (it can be uploaded to the Master Page gallery for instance), so it will be subject to the SharePoint “safe mode” processing mode too. And you will need to add some extra configuration bits to the web.config file so that your pages are allowed to use the user controls. You have two ways to enable this in the web.config – the first one is to add a “SafeControl” element like this one:

<SafeControl Src="/_catalogs/masterpage/*" IncludeSubFolders="True" Safe="True" AllowRemoteDesigner="True" SafeAgainstScript="True" />

Note that the value of the “Src” attribute should contain the full server relative URL of the target library in your site collection (e.g. for the “/sites/test-site site collection it will be /sites/test-site/_catalogs/masterpage/*”). The other way (much more unsecure) is to add an extra attribute to the “PageParsePath” element:

AllowUnsafeControls="True"

In this case you won’t need an extra “SafeControl” element.

Note also that although you can deploy your code files (pages and user controls) directly to the content database unlike the SharePoint 2010 sandbox solutions you will need to modify the web.config file of the containing web application, which has its own serious security implication, that I already pointed out.

And let me now demonstrate a sample user control that can be deployed (uploaded) to the Master Page gallery and used by page layouts (wpzone.ascx):

<%@ Assembly Name="Microsoft.SharePoint, Version=14.0.0.0,Culture=neutral,PublicKeyToken=71e9bce111e9429c" %>
<%@ Assembly Name="Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
 
<%@ Control Language="C#"  %>
<%@ Import Namespace="System.Collections.Generic" %>
<%@ Import Namespace="Microsoft.SharePoint" %>
<%@ Import Namespace="Microsoft.SharePoint.WebPartPages" %>
 
<script runat="server">
    protected string _wpZoneID;

    public string WPZoneID { get { return _wpZoneID; } set { _wpZoneID= value; } }

    protected override void OnLoad (EventArgs args)

         base.OnLoad(args);

    protected override void Render (HtmlTextWriter writer)

         if (string.IsNullOrEmpty(_wpZoneID)) return;

         Control c = this.NamingContainer.FindControl (_wpZoneID);

         WebPartZone _zone = c as WebPartZone;

         if (_zone == null) return;

         SPWebPartManager mngr = SPWebPartManager.GetCurrentWebPartManager(this.Page) as SPWebPartManager;

         if (mngr == null) return;

         if (!mngr.GetDisplayMode().AllowPageDesign)

             // if we are in display mode - hide the zone control itself

             _zone.Visible = false;

             // and render the web parts directly

             foreach (WebPart part in _zone.WebParts)

                 part.RenderControl(writer);

     }
script>

And here is how you can use this control in a page layout (the technique is identical for master pages and regular web part pages) – first you need the control “Register” directive at the top of the page:

<%@ Register TagPrefix="Test" TagName="WPZone" Src="~SiteCollection/_catalogs/masterpage/wpzone.ascx" %>

Note the value of the “Src” attribute – you can use the handy “~SiteCollection” URL token here instead of having to hard-code the server relative URL of the containing site collection. And then the declaration of the user control’s tag inside the markup of the page:

<Test:WPZone runat="server" WPZoneID="TopZone"/>

This simple user control modifies the default rendering of the WebPartZone control whose ID is specified in its “WPZoneiD” property. When the page is in display mode the control hides the web part zone and renders only the web parts that belong to this web part zone. This effectively hides the markup that is produces by the web part zone (several nested HTML table elements) and also the chrome (or frame) headers of the web parts. You should place the control just before the declaration of the web part zone control whose rendering you want to modify.

It is also possible to use user controls just as regular classes with helper methods that can be reused in various places – in this case the user control will not have visual behavior. For example you can create a user control like this (mylib.ascx):

<%@ Assembly Name="Microsoft.SharePoint, Version=14.0.0.0,Culture=neutral,PublicKeyToken=71e9bce111e9429c" %>
<%@ Assembly Name="Microsoft.SharePoint.Publishing, Version=14.0.0.0, Culture=neutral, PublicKeyToken=71e9bce111e9429c" %>
 
<%@ Control Language="C#" ClassName="MyLib" %>
<script runat="server">
 public static string SayHello(string who)
 {
  return "Hello " + who;
 }
 
 public string SayHi (string who)
 {
  return "Hi " + who;
 }
script>

Note the “ClassName” attribute in the “Control” directive – this specifies the name of the class that will be generated from the ascx file. You will be able to use this generated class by this name in the code blocks of the pages that use the control (or in other user controls). To use the user control in this way you will only have to put the control “Register” directive at the top of the page. And then in a code block you can have something like:

<script runat="server">
protected override void OnLoad(EventArgs args)
{
 base.OnLoad(args);
 this.txtBox.Text = MyLib.SayHello("John");
}
script>

SharePoint 2010–activate features with feature properties

2011-09-11T07:44:00.001-07:00

There are two ways to achieve this in SharePoint – specify the feature properties in the ONET.XML file of your site definition or web template (the new SharePoint 2010 feature) or put the properties directly in the FEATURE.XML definition file of the feature. The latter is almost useless since it almost defeats the purpose of having the type of parameterization that you can achieve with feature properties – which is a very simple and powerful concept – use one and the same feature which can act differently in different contexts. As for the ONET.XML usage of feature properties – I think that this is probably the only thing that justifies the usage of site templates at least for me. In this sense site definitions are not merely a set or a grouping of features that should be activated to a site, provisioning some SharePoint artifacts, but rather a grouping of specifically configured features with this configurability being possible with the usage of feature properties. So despite the numerous issues that one may encounter with site definitions in SharePoint especially when it comes to updating the definition in existing sites, site definitions still have this important and pretty useful bit to offer.

The next question is can this same thing be achieved programmatically. In the SharePoint object model we have the SPFeatureCollection.Add method which allows you to activate features to a site collection or a site. It actually comes with three overloads but none of them allows you to provide specific feature properties that should be used for the feature activation. Of course if you are the “hacker” type of developer and use frequently the .NET Reflector utility you will probably already know that there is an internal method in the aforementioned SPFeatureCollection class that actually accepts a parameter in which you can specify the feature properties for activation. If you Google this you will find that there are actually several blog articles describing this technique. There is one big problem with reflection though – apart from the fact that it is not one of the best practices to use, there is always the risk to get things broken with the next service pack or version of SharePoint. In fact the notation of this exact method changed in SharePoint 2010 as compared to SharePoint 2007.

But is reflection the only option that we have here? It turns out that there is one other way to get this working and you can get an idea of how this is possible again by using the .NET Reflector tool (quite popular for SharePoint development actually). If you use the “analyze” command for the SPFeatureCollection.AddInternal method (the internal method, that I mentioned above) in the .NET Reflector you will see that it is used directly by an internal class called FeatureSerializer from the Microsoft.SharePoint.Deployment namespace. The latter is a namespace in the core Microsoft.SharePoint.dll which contains the SharePoint Deployment API and this is an important clue. If you have done export and import operations for SharePoint site collections or sites you’ve probably noticed that the imported site collections and sites are pretty much exact replicas of the source site collections and sites and that they contain all features with exactly the same feature properties activated as their source counterparts. It is obvious that the deployment API handles not only SharePoint artifacts like SharePoint lists and list items but for sites and site collections the associated features as well. This is where the SPFeatureCollection.AddInternal internal method comes into play invoked by the deployment API.

So, we know that if you have an export package of a site or a site collection (created with the STSADM utility or PowerShell) it will activate the very same set of features in a target site or site collection (or will create them if they don’t exist) but it will also import a ton of other things like all SharePoint lists and list items in them. We face two questions here – one is – is it possible to create an export package that can activate only features to a target site or site collection and the second is – is it legal and allowed by SharePoint standards to create Deployment API packages manually, i.e. without using the Deployment API itself. Starting with the second – this is perfectly OK, the schema of the XML files in a SharePoint deployment package is described officially in MSDN - http://msdn.microsoft.com/en-us/library/bb249989.aspx – the idea is that it can be used by third party providers who want to import content in SharePoint. If you check the article in MSDN you will see that there are eight deployment schema files that a deployment package should contain (although not all of them are actually required). For all deployment schema files there are XSD files which you can find in the 14/TEMPLATE/XML folder (all starting with “Deployment”) and which you can use to validate the XML of the deployment schema files that you create.

And to answer the first question – this can be achieved by creating three small deployment schema files that look like this:

ExportSettings.xml:

<ExportSettings xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" SiteUrl="http://mysite/sites/2" FileLocation="" BaseFileName="" IncludeSecurity="None" IncludeVersions="LastMajor" ExportPublicSchema="true" ExportFrontEndFileStreams="true" ExportMethod="ExportAll" ExcludeDependencies="false" xmlns="urn:deployment-exportsettings-schema">
  <ExportObjects>
    <DeploymentObject Id="ea0d5873-cf3a-4015-b015-db74f39db27a" Type="Web" ParentId="00000000-0000-0000-0000-000000000000" Url="" ExcludeChildren="false" IncludeDescendants="None" />
  </ExportObjects>
</ExportSettings>

SystemData.xml:

<SystemData xmlns="urn:deployment-systemdata-schema">
  <SchemaVersion Version="14.0.0.0" Build="14.0.4762.1000" DatabaseVersion="3683" SiteVersion="0" ObjectsProcessed="1" />
  <ManifestFiles>
    <ManifestFile Name="Manifest.xml" />
  </ManifestFiles>
  <SystemObjects></SystemObjects>
  <RootWebOnlyLists />
</SystemData>

Manifest.xml:

<SPObjects xmlns="urn:deployment-manifest-schema">
  <SPObject Id="04a18cb4-39d8-4371-8a21-d11f2b54a333" ObjectType="SPFeature" ParentId="" ParentWebId="" xmlns="urn:deployment-manifest-schema">
    <Feature Id="04a18cb4-39d8-4371-8a21-d11f2b54a333" FeatureDefinitionName="WTSample_SiteScopedFeature" Version="0.0.0.0" IsUserSolutionFeature="false" Properties="&lt;Properties&gt;&#xD;&#xA;  &lt;Property Key=&quot;a&quot; Value=&quot;b&quot; /&gt;&#xD;&#xA;&lt;/Properties&gt;" />
  </SPObject>
  <SPObject Id="4edea5b8-7da5-46cc-b1c6-7a8ce4bcb899" ObjectType="SPFeature" ParentId="" ParentWebId="ea0d5873-cf3a-4015-b015-db74f39db27a" xmlns="urn:deployment-manifest-schema">
    <Feature Id="4edea5b8-7da5-46cc-b1c6-7a8ce4bcb899" FeatureDefinitionName="WTSample_WTMarkFeature" Version="1.0.0.0" IsUserSolutionFeature="false" WebId="ea0d5873-cf3a-4015-b015-db74f39db27a" Properties="&lt;Properties&gt;&#xD;&#xA;  &lt;Property Key=&quot;c&quot; Value=&quot;d&quot; /&gt;&#xD;&#xA;&lt;/Properties&gt;" />
  </SPObject>
</SPObjects>

You see that the important bits are in the “manifest.xml” schema file – you can see two “SPObject” elements in it with one “Feature” element directly beneath each – these correspond to two features – one “Site” and one “Web” scoped. Have a look at the “Properties” attribute of the “Feature” element – you can see there a familiarly looking “Properties” XML fragment, exactly the same as you have in a standard feature.xml file.

After you have these three schema deployment files you can simply place them in a disc folder and use the deployment API with the SPImport.Run method to import the package as any normal deployment package (note that you will have an uncompressed package as opposed to the compressed packages that STSADM and PowerShell normally output).

An important note here – you can activate only features that are not already activated on the target site or site collection in this manner. If the features are already activated they won’t get reactivated as you can do with the STSADM utility or PowerShell using the “-force” parameter.

If you want to use this approach, instead of formatting the deployment schema files yourselves you can use a handy helper method that I created which nicely wraps the whole logic of the deployment package creation and the running of the SPImort.Run method. You can download the sample solution containing this method from here.

You can also see the source code of the helper method here (it uses some additional private helpers and helper classes whose definitions you can see in the full source code from the above link):

        // provide the target SPWeb and one or several FeatureInfo instances each containing a SPFeatureDefinition and Dictionary<string, string> (for the feature properties) pair
        // you can provide features with Site and Web scopes, Site scoped features are valid only for a root SPWeb
        public void EnableFeature(SPWeb web, params FeatureInfo[] infos)
        {
            // check for valid feature scopes
            bool valid = infos.All(info => info.FeatureDefinition.Scope == SPFeatureScope.Web || (info.FeatureDefinition.Scope == SPFeatureScope.Site && web.IsRootWeb));
            if (!valid) throw new ArgumentException("Invalid feature scope.");
 
            // load the template SystemData.xml, ExportSettings.xml, Manifest.xml files which are stored as embedded resources
            XDocument systemDataDoc = XDocument.Parse(this.GetManifestResource("WTSample.Core.Xml.SystemData.xml"));
            XDocument exportSettingsDoc = XDocument.Parse(this.GetManifestResource("WTSample.Core.Xml.ExportSettings.xml"));
            XDocument manifestDoc = XDocument.Parse(this.GetManifestResource("WTSample.Core.Xml.Manifest.xml"));
 
            XNamespace ns = systemDataDoc.Root.Name.Namespace;
            XElement schemaVersionElement = systemDataDoc.Root.Element(ns + "SchemaVersion");
            // set the Build and DatabaseVersion attributes in the SystemData.xml
            schemaVersionElement.SetAttributeValue("Build", SPFarm.Local.BuildVersion.ToString());
            schemaVersionElement.SetAttributeValue("DatabaseVersion", web.Site.ContentDatabase.Version.ToString(CultureInfo.InvariantCulture));
 
            ns = exportSettingsDoc.Root.Name.Namespace;
            IXmlNamespaceResolver resolver = exportSettingsDoc.Root.CreateNavigator();
            // set the SiteUrl and Id attributes in the ExportSettings.xml
            exportSettingsDoc.Root.SetAttributeValue("SiteUrl", web.Url);
            XElement deploymentObject = exportSettingsDoc.Root.Descendants(ns + "DeploymentObject").FirstOrDefault();
            if (deploymentObject != null) deploymentObject.SetAttributeValue("Id", web.ID);
 
            ns = manifestDoc.Root.Name.Namespace;
            // extract the existing SPObject element from the Manifest.xml - we will use it as a template for adding the feature SPObject nodes
            XElement featureElement = manifestDoc.Root.Elements(ns + "SPObject").First();
            string featureElementStr = featureElement.ToString();
            featureElement.Remove();
 
            foreach (FeatureInfo info in infos)
            {
                // create a new feature SPObject element for every feature definition
                featureElement = XElement.Parse(featureElementStr);
                manifestDoc.Root.Add(featureElement);
 
                // set its Id and ParentWebId (the latter only for Web scoped features)
                featureElement.SetAttributeValue("Id", info.FeatureDefinition.Id.ToString());
                if (info.FeatureDefinition.Scope == SPFeatureScope.Web) featureElement.SetAttributeValue("ParentWebId", web.ID.ToString());
 
                // get the feature's name and the feature's parent solution type
                bool isUserSolution = false;
                string rootDirectory = info.FeatureDefinition.RootDirectory;
                int startIndex = rootDirectory.LastIndexOf(@"\") + 1;
                if (startIndex >= 0) { rootDirectory = rootDirectory.Substring(startIndex); }
                if (rootDirectory.Length == 0)
                {
                    isUserSolution = true;
                    rootDirectory = info.FeatureDefinition.GetTitle(CultureInfo.InvariantCulture) ?? "";
                }
 
                // create the child Feature element
                XElement contentElement = featureElement.Element(ns + "Feature");
                // set its Id, FeatureDefinitionName, Version, IsUserSolutionFeature and WebId attributes
                contentElement.SetAttributeValue("Id", info.FeatureDefinition.Id.ToString());
                contentElement.SetAttributeValue("FeatureDefinitionName", rootDirectory);
                contentElement.SetAttributeValue("Version", info.FeatureDefinition.Version.ToString(4));
                contentElement.SetAttributeValue("IsUserSolutionFeature", isUserSolution.ToString().ToLower());
                if (info.FeatureDefinition.Scope == SPFeatureScope.Web) contentElement.SetAttributeValue("WebId", web.ID.ToString());
                if (info.Properties != null && info.Properties.Count > 0)
                {
                    // create Properties XML fragment
                    XElement propertiesElement = new XElement("Properties", info.Properties.Select(kv => new XElement("Property", new XAttribute("Key", kv.Key), new XAttribute("Value", kv.Value))));
                    // and assign its value to the Properties attribute
                    contentElement.SetAttributeValue("Properties", propertiesElement.ToString());
                }
 
                if (string.IsNullOrEmpty(contentElement.Attribute("WebId").Value)) contentElement.Attribute("WebId").Remove();
                if (string.IsNullOrEmpty(contentElement.Attribute("Properties").Value)) contentElement.Attribute("Properties").Remove();
            }
            // create a temporary folder to store the export schema files
            string tempFolder = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());
            Directory.CreateDirectory(tempFolder);
            try
            {
                // create the export schema files
                systemDataDoc.Save(Path.Combine(tempFolder, "SystemData.xml"));
                exportSettingsDoc.Save(Path.Combine(tempFolder, "ExportSettings.xml"));
                manifestDoc.Save(Path.Combine(tempFolder, "Manifest.xml"));
                // set up the SPImportSettings for the created export batch
                SPImportSettings importSettings = new SPImportSettings()
                {
                    FileCompression = false,
                    FileLocation = tempFolder,
                    SiteUrl = web.Site.Url,
                    WebUrl = web.Url
                };
                // run the SPImport.Run methods to import the batch
                SPImport import = new SPImport(importSettings);
                import.Run();
            }
            finally
            {
                Thread.Sleep(0);
                // clean up - delete the temporary folder
                try { Directory.Delete(tempFolder, true); }
                catch { }
            }
        }

The code contains pretty verbose comments, so that you can easily see how the three deployment schema XML files are formatted and how the input bits from the target SPWeb instance and the feature or features to be activated are fitted in XML elements or attributes inside them. The method accepts a SPWeb input parameter but you can use it for activating “Site” scoped features as well, as long as the SPWeb instance represents the root site of the target site collection.

One last thing to mention – this approach (at least its implementation) may seem a bit complicated and overly verbose as opposed to the one-liner solution when you use reflection. Another thing is that the deployment schema files can hardly be seen as a sort of a contract and we cannot expect that their format won’t get changed in the future in some next release of version of SharePoint. Still, one thing that I find important is that the very same approach of constructing manually deployment packages can as well be used to provision other types of SharePoint artifacts that we thought so far are possible only with XML in feature element files which I think can reveal the great potential of the SharePoint deployment API in this field.

SharePoint Search 2007 – hacking the SiteData web service – Part II

2011-07-30T07:08:00.001-07:00

In the first part of this posting I outlined the physical architecture of the SharePoint search engine mostly concerning the part of the Sts3 protocol handler and the role that the “SiteData” web service plays in the crawling process. The very fact that the search engine uses a web service to access and retrieve SharePoint list and site data gave me the first clue as to how I can “hack” the search crawling process. My idea was simple – since the web service is hosted in IIS, of course, I can use some basic URL rewriting techniques so that the call to the real web service is “covertly” redirected to a custom web service which will either implement the original logic of the standard service adding some additional functionality or will simply serve as a proxy to the real web service but will do some modifications to either the input or the output of the latter. Out of these two options the first one seemed more than complex and the second one actually was pretty sufficient as to the goals that I had with the implementation of the “hack”. The thing is that the output XML of the SiteData.GetContent method contains all relevant SharePoint list item and schema data – the “List” option for the ObjectType parameter returns the schema of the SharePoint list and the “Folder” option – the list item data (see the sample XML outputs of the web service in the first part of the posting). The problem is that the Sts3 protocol handler “interprets” the data from the output XML in its own specific way which results in the well-known limitations of the crawled properties and the retrieved for them data in the search index that we have for SharePoint content. So what I decided to do was to create a small custom web service which implements the SiteData.GetContent and SiteData.GetChanges methods (with the exact same parameters and notation). Since I wanted to use it as a proxy to the real SiteData web service I needed somehow to pass the call to it. The simplest option here would have been to simply issue a second web service call from my web service, but the better solution was to just instantiate an instance of the SiteData web service class (Microsoft.SharePoint.SoapServer.SiteData from the STSSOAP assembly which is in the _app_bin subfolder of the SharePoint web application) and call its corresponding method. The last trick of the “hack” was to get the XML output from the SiteData GetContent and SiteData.GetChanges methods and modify it (actually add some additional stuff to it) so that I can get the extra crawled properties in the search index that I needed.

So, before going into details about the implementation of the “hack”, I want to point out several arguments as to why you should consider twice before starting using it (it’s become a habit of mine trying to dissuade people from using my own solutions) and I would rather not recommend using it in bigger production environments:

It tampers with the XML output of the standard SiteData web service – this may lead to unpredictable behavior of the index engine and result in it being not able to crawl your site(s). The standard XML output of the SiteData service is itself not quite well-formed XML so before getting the right way to modify it without losing its original formatting I kept receiving crawler errors which I could find in the crawl log of my SSP admin site.
There will be a serious performance penalty compared to using just the standard SiteData service. The increased processing time comes from the added parsing of the output XML and the extra modifications and additions added to it.
The general argument that this is indeed a hack which gets inside the standard implementation of the SharePoint search indexing which won’t sound good to both managers and Microsoft guys alike.

Having said that (and if you are still reading) let me give you the details of the implementation itself. The solution of the “hack” can be downloaded from here (check the installation notes below).

The first thing that I will start with is the URL rewriting logic that allows the custom web service to be invoked instead of the standard SiteData web service. In IIS 7 there is a built-in support for URL rewriting, but because I was testing on a Windows 2003 server with IIS 6 and because I was a bit lazy to implement a full proxy for the SiteData web service I went to the other approach … Which is to use a custom .NET HTTP module (the better solution) or to simply modify the global.asax of the target SharePoint web application (the worse but easier to implement solution) – which is the one that I actually used. The advantage of using a custom URL rewriting logic as opposed to using the built in URL rewriting functionality in IIS 7 is that in the former you can additionally inspect the HTTP request data and apply the URL rewriting only for certain methods of the web service. So in the modified version of the global.asax I do an explicit check for the web service method being called and redirect to the custom web service only if I detect the GetContent or GetChanges methods (all other methods will hit directly the standard SiteData service and no URL rewriting will take place). You can see the source code of the global.asax file that I used below:

<%@ Application Language="C#" Inherits="Microsoft.SharePoint.ApplicationRuntime.SPHttpApplication" %>
<script language="C#" runat="server">
protected void Application_BeginRequest(Object sender, EventArgs e) 
{
    CheckRewriteSiteData();
}
protected void CheckRewriteSiteData()
{
    if (IsGetListItemsCall())
    {
        string newUrl = this.Request.Url.AbsolutePath.ToLower().Replace("/sitedata.asmx", "/stefansitedata.asmx");
        HttpContext.Current.RewritePath(newUrl);
    }
}
protected bool IsGetListItemsCall()
{
    if (string.Compare(this.Request.ServerVariables["REQUEST_METHOD"], "post", true) != 0) return false;
    if (!this.Request.Url.AbsolutePath.EndsWith("/_vti_bin/SiteData.asmx", StringComparison.InvariantCultureIgnoreCase)) return false;
 
    if (string.IsNullOrEmpty(this.Request.Headers["SOAPAction"])) return false;
 
    string soapAction = this.Request.Headers["SOAPAction"].Trim('"').ToLower();
    if (!soapAction.EndsWith("getcontent") && !soapAction.EndsWith("getchanges")) return false;
    if (string.Compare(ConfigurationManager.AppSettings["UseSiteDataRewrite"], "true", true) != 0) return false;
 
    return true;
}
</script>

Note also that in the code I check a custom “AppSettings” key in the web.config file whether to use or not URL rewriting logic. This way you can easily turn on or off the “hack” with a simple tweak in the configuration file of the SharePoint web application.

And this is the code of the custom “SiteData” web service:

[WebServiceBinding(ConformsTo = WsiProfiles.BasicProfile1_1), WebService(Namespace = "http://schemas.microsoft.com/sharepoint/soap/")]
public class SiteData
{
    [WebMethod]
    public string GetContent(ObjectType objectType, string objectId, string folderUrl, string itemId, bool retrieveChildItems, bool securityOnly, ref string lastItemIdOnPage)
    {
        try
        {
            SiteDataHelper siteDataHelper = new SiteDataHelper();
            return siteDataHelper.GetContent(objectType, objectId, folderUrl, itemId, retrieveChildItems, securityOnly, ref lastItemIdOnPage);
        }
        catch (ThreadAbortException) { throw; }
        catch (Exception exception) { throw SoapServerException.HandleException(exception); }
    }
    [WebMethod]
    public string GetChanges(ObjectType objectType, string contentDatabaseId, ref string LastChangeId, ref string CurrentChangeId, int Timeout, out bool moreChanges)
    {
        try
        {
            SiteDataHelper siteDataHelper = new SiteDataHelper();
            return siteDataHelper.GetChanges(objectType, contentDatabaseId, ref LastChangeId, ref CurrentChangeId, Timeout, out moreChanges);
        }
        catch (ThreadAbortException) { throw; }
        catch (Exception exception) { throw SoapServerException.HandleException(exception); }
    }
}

As you see, the custom “SiteData” web service implements only the GetContent and GetChanges methods. We don’t need to implement the other methods of the standard SiteData web service because the URL rewriting will redirect to the custom web service only in case these two methods are being invoked. The two methods in the custom service have the exact same notation as the ones in the standard SiteData web service. The implementation of the methods is a simple delegation to the corresponding methods with the same names of a helper class: SiteDataHelper. Here is the source code of the SiteDataHelper class:

using SP = Microsoft.SharePoint.SoapServer;
namespace Stefan.SharePoint.SiteData
{
    public class SiteDataHelper
    {
        public string GetChanges(ObjectType objectType, string contentDatabaseId, ref string startChangeId, ref string endChangeId, int Timeout, out bool moreChanges)
        {
            SP.SiteData siteData = new SP.SiteData();
            string res = siteData.GetChanges(objectType, contentDatabaseId, ref startChangeId, ref endChangeId, Timeout, out moreChanges);
            try
            {
                ListItemXmlModifier modifier = new ListItemXmlModifier(new EnvData(), res);
                res = modifier.ModifyChangesXml();
            }
            catch (Exception ex) { Logging.LogError(ex); }
            return res;
        }
 
        public string GetContent(ObjectType objectType, string objectId, string folderUrl, string itemId, bool retrieveChildItems, bool securityOnly, ref string lastItemIdOnPage)
        {
            SPWeb web = SPContext.Current.Web;
            SP.SiteData siteData = new SP.SiteData();
            string res = siteData.GetContent(objectType, objectId, folderUrl, itemId, retrieveChildItems, securityOnly, ref lastItemIdOnPage);
            try
            {
                EnvData envData = new EnvData() { SiteId = web.Site.ID, WebId = web.ID, ListId = objectId.TrimStart('{').TrimEnd('}') };
                if ((objectType == ObjectType.ListItem || objectType == ObjectType.Folder) && !securityOnly)
                {
                    ListItemXmlModifier modifier = new ListItemXmlModifier(envData, res);
                    res = modifier.ModifyListItemXml();
                }
                else if (objectType == ObjectType.List)
                {
                    ListItemXmlModifier modifier = new ListItemXmlModifier(envData, res);
                    res = modifier.ModifyListXml();
                }
            }
            catch (Exception ex) { Logging.LogError(ex); }
            return res;
        }
    }
}

The thing to note here is that the two methods in the SiteDataHelper helper class create an instance of the SiteData web service class directly (note that this is not a generated proxy class, but the actual web service class implemented in the standard STSSOAP.DLL). The GetContent and GetChanges methods are called on this instance respectively and the string result of the calls is stored in a local variable. The string value that these methods return actually contains the XML with the list schema or list item data (depending on the “ObjectType” parameter being “List” or “Folder”). This XML data is then provided to an instance of the custom ListItemXmlModifier class which handles all XML modifications for both the GetContent and GetChanges methods. Note that for the GetContent method, the XML results are passed for modification only if the “ObjectType” parameter has the “ListItem”, “Folder” or “List” values. I am not going to show the source code of the ListItemXmlModifier class directly in the posting (it is over 700 lines of code) but instead I will briefly explain to you what are the changes in the XML from the GetContent and GetChanges methods that this class implements. The modifications to the XML are actually pretty simple and semantically there are only two types of changes – these correspond to the result XML-s of the GetContent (ObjectType=List) and GetContent (ObjectType=Folder) methods (the result XML of the GetChanges method has a more complex structure but contains the above two fragments (in one or more occurrences) where list and list item changes are available).

Let’s start with a sample XML from the standard SiteData.GetContent(ObjectType=List) method (I’ve trimmed some of the elements for brevity):

<List>
  <Metadata ID="{1d53a556-ae9d-4fbf-8917-46c7d97ebfa5}" LastModified="2011-01-17 13:24:18Z" Title="Pages" DefaultTitle="False" Description="This system library was created by the Publishing feature to store pages that are created in this site." BaseType="DocumentLibrary" BaseTemplate="850" DefaultViewUrl="/Pages/Forms/AllItems.aspx" DefaultViewItemUrl="/Pages/Forms/DispForm.aspx" RootFolder="Pages" Author="System Account" ItemCount="4" ReadSecurity="1" AllowAnonymousAccess="False" AnonymousViewListItems="False" AnonymousPermMask="0" CRC="699748088" NoIndex="False" ScopeID="{a1372e10-8ffb-4e21-b627-bed44a5130cd}" />
  <ACL>
    <permissions>
      <permission memberid='3' mask='9223372036854775807' />
      ....
    </permissions>
  </ACL>
  <Views>
    <View URL="Pages/Forms/AllItems.aspx" ID="{771a1809-e7f3-4c52-b346-971d77ff215a}" Title="All Documents" />
    ....
  </Views>
  <Schema>
    <Field Name="FileLeafRef" Title="Name" Type="File" />
    <Field Name="Title" Title="Title" Type="Text" />
    <Field Name="Comments" Title="Description" Type="Note" />
    <Field Name="PublishingContact" Title="Contact" Type="User" />
    <Field Name="PublishingContactEmail" Title="Contact E-Mail Address" Type="Text" />
    <Field Name="PublishingContactName" Title="Contact Name" Type="Text" />
    <Field Name="PublishingContactPicture" Title="Contact Picture" Type="URL" />
    <Field Name="PublishingPageLayout" Title="Page Layout" Type="URL" />
    <Field Name="PublishingRollupImage" Title="Rollup Image" Type="Note" TypeAsString="Image" />
    <Field Name="Audience" Title="Target Audiences" Type="Note" TypeAsString="TargetTo" />
    <Field Name="ContentType" Title="Content Type" Type="Choice" />
    <Field Name="MyLookup" Title="MyLookup" Type="Lookup" />
    ....
  </Schema>
</List>

The XML contains the metadata properties of the queried SharePoint list, the most important part of which is contained in the Schema/Field elements – the simple definitions of the fields in this list. It is easy to deduce that the fields that the index engine encounters in this part of the XML will be recognized and appear as crawled properties in the search index. So what if we start adding fields of our own – this won’t solve the thing by itself because we will further need list items with values for these “added” fields (we’ll see that in the second XML sample) but it is the first required bit of the “hack”. The custom service implementation will actually add several extra “Field” elements like these:

    <Field Name='ContentTypeId.Text' Title='ContentTypeId' Type='Note' />
    <Field Name='Author.Text' Title='Created By' Type='Note' />
    <Field Name='Author.ID' Title='Created By' Type='Integer' />
    <Field Name='MyLookup.Text' Title='MyLookup' Type='Note' />
    <Field Name='MyLookup.ID' Title='MyLookup' Type='Integer' />
    <Field Name='PublishingRollupImage.Html' Title='Rollup Image' Type='Note' />
    <Field Name='PublishingPageImage.Html' Title='Page Image' Type='Note' />
    <Field Name='PublishingPageContent.Html' Title='Page Content' Type='Note' />
    <Field Name='Env.SiteId' Title='Env.SiteId' Type='Text' />
    <Field Name='Env.WebId' Title='Env.WebId' Type='Text' />
    <Field Name='Env.ListId' Title='Env.ListId' Type='Text' />
    <Field Name='Env.IsListItem' Title='Env.IsListItem' Type='Integer' />

You can immediately notice that these “new” fields are actually related to already existing fields in the SharePoint list in the schema XML that’s being modified. You can see that I used a specific naming convention for the “Name” attribute – with a dot and a short suffix. Actually the crawled properties that the index engine will generate will also contain the dot and the suffix, so it will be easy for you to locate them in the “crawled properties” page in the SSP admin site. From the “Name” attribute you can immediately see which the related original fields for the new fields are. In short the rules for creating these new fields are:

For every original lookup field (both single and multiple lookup columns and all derived lookup columns, e.g. user fields) two additional fields are added – with the suffixes “.ID” and “.Text” and field “Type” attribute “Integer” and “Note” respectively.
For every original publishing “HTML” and “Image” field one extra field with the “.Html” suffix is added.
For all lists the “ContentTypeId.Text” extra field is added with “Type” attribute set to “Note”
For all lists the additional fields “Env.SiteId”, “Env.WebId”, “Env.ListId”, “Env.IsListItem” are added.

So, we have already extra fields in the list schema, the next step is to have them in the list item data populated with the relevant values. Let me first show you a sample of the standard unmodified XML output of the GetContent(ObjectType=Folder) method (I trimmed some of the elements and reduced the values of some of the attributes for brevity):

<Folder>
  <Metadata>
    <scope id='{5dd2834e-902d-4db0-8db2-4a1da762a620}'>
      <permissions>
        <permission memberid='1' mask='206292717568' />
        ....
      </permissions>
    </scope>
  </Metadata>
  <xml xmlns:s='uuid:BDC6E3F0-6DA3-11d1-A2A3-00AA00C14882' xmlns:dt='uuid:C2F41010-65B3-11d1-A29F-00AA00C14882' xmlns:rs='urn:schemas-microsoft-com:rowset' xmlns:z='#RowsetSchema'>
    <s:Schema id='RowsetSchema'>
      <s:ElementType name='row' content='eltOnly' rs:CommandTimeout='30'>
        <s:AttributeType name='ows_ContentTypeId' rs:name='Content Type ID' rs:number='1'>
          <s:datatype dt:type='int' dt:maxLength='512' />
        </s:AttributeType>
        <s:AttributeType name='ows__ModerationComments' rs:name='Approver Comments' rs:number='2'>
          <s:datatype dt:type='string' dt:maxLength='1073741823' />
        </s:AttributeType>
        <s:AttributeType name='ows_FileLeafRef' rs:name='Name' rs:number='3'>
          <s:datatype dt:type='string' dt:lookup='true' dt:maxLength='512' />
        </s:AttributeType>
        ....
      </s:ElementType>
    </s:Schema>
    <scopes>
    </scopes>
    <rs:data ItemCount='2'>
      <z:row ows_ContentTypeId='0x010100C568DB52D9D0A14D9B2FDCC96666E9F2007948130EC3DB064584E219954237AF390064DEA0F50FC8C147B0B6EA0636C4A7D400E595F4AC9968CC4FAD1928288BC9885A' ows_FileLeafRef='1;#Default.aspx' ows_Modified_x0020_By='myserver\sstanev' ows_File_x0020_Type='aspx' ows_Title='Home' ows_PublishingPageLayout='http://searchtest/_catalogs/masterpage/defaultlayout.aspx, Welcome page with Web Part zones' ows_ContentType='Welcome Page' ows_PublishingPageImage='' ows_PublishingPageContent='some content' ows_ID='1' ows_Created='2010-12-20T18:53:18Z' ows_Author='1;#Stefan Stanev' ows_Modified='2010-12-26T12:45:31Z' ows_Editor='1;#Stefan Stanev' ows__ModerationStatus='0' ows_FileRef='1;#Pages/Default.aspx' ows_FileDirRef='1;#Pages' ows_Last_x0020_Modified='1;#2010-12-26T12:45:32Z' ows_Created_x0020_Date='1;#2010-12-20T18:53:19Z' ows_File_x0020_Size='1;#6000' ows_FSObjType='1;#0' ows_PermMask='0x7fffffffffffffff' ows_CheckedOutUserId='1;#' ows_IsCheckedoutToLocal='1;#0' ows_UniqueId='1;#{923EEE29-44AB-4D1B-B65B-E3ECEAE1353E}' ows_ProgId='1;#' ows_ScopeId='1;#{5DD2834E-902D-4DB0-8DB2-4A1DA762A620}' ows_VirusStatus='1;#6000' ows_CheckedOutTitle='1;#' ows__CheckinComment='1;#' ows__EditMenuTableStart='Default.aspx' ows__EditMenuTableEnd='1' ows_LinkFilenameNoMenu='Default.aspx' ows_LinkFilename='Default.aspx' ows_DocIcon='aspx' ows_ServerUrl='/Pages/Default.aspx' ows_EncodedAbsUrl='http://searchtest/Pages/Default.aspx' ows_BaseName='Default' ows_FileSizeDisplay='6000' ows_MetaInfo='...' ows__Level='1' ows__IsCurrentVersion='1' ows_SelectTitle='1' ows_SelectFilename='1' ows_Edit='0' ows_owshiddenversion='26' ows__UIVersion='6656' ows__UIVersionString='13.0' ows_Order='100.000000000000' ows_GUID='{2C80A53D-4F38-4494-855D-5B52ED1D095B}' ows_WorkflowVersion='1' ows_ParentVersionString='1;#' ows_ParentLeafName='1;#' ows_Combine='0' ows_RepairDocument='0' ows_ServerRedirected='0' />
      ....
    </rs:data>
  </xml>
</Folder>

The list item data is contained below the “rs:data” element – there is one “z:row” element for every list item. The attributes of the “z:row” element contain the field values of the corresponding list item. You can see here that the attributes already have the “ows_” prefix as all crawl properties in the “SharePoint” category. You can notice that the attributes for lookup fields contain the unmodified item field data but the publishing “HTML” and “Image” columns are already modified – all HTML markup has been removed from them (for the “Image” type column this means that they become empty, since all the data they contain is in markup).

And let’s see the additional attributes that the custom web service adds to the “z:row” elements of the list item data XML:

ows_ContentTypeId.Text='0x010100C568DB52D9D0A14D9B2FDCC96666E9F2007948130EC3DB064584E219954237AF3900242457EFB8B24247815D688C526CD44D0005E464F1BD83D14983E49C578030FBF6' 
ows_PublishingPageImage.Html='&lt;img border="0" src="/PublishingImages/newsarticleimage.jpg" vspace="0" style="margin-top:8px" alt=""&gt;' 
ows_PublishingPageContent.Html='&lt;b&gt;some content&lt;/b&gt;' 
ows_Author.Text='1;#Stefan Stanev' 
ows_Author.ID='1' 
ows_Editor.Text='1;#Stefan Stanev' 
ows_Editor.ID='1' 
ows_Env.SiteId='ff96067a-accf-4763-8ec1-194f20fbf0f5' 
ows_Env.WebId='b2099353-41d6-43a7-9b0d-ab6ad87fb180' 
ows_Env.ListId='Pages' 
ows_Env.IsListItem='1' 

Here is how these fields are populated/formatted (as I mentioned above all the data is retrieved from the XML itself or from the context of the service request):

the lookup derived fields with the “.ID” and “.Text” suffixes – both get their values from the “parent” lookup column – the former is populated with the starting integer value of the lookup field, the latter is set with the unmodified value of the original lookup column. When the search index generates the corresponding crawled properties the “.ID” field can be used as a true integer property and the “.Text” one although containing the original lookup value will be treated as a simple text property by the index engine (remember that in the list schema XML the type of this extra field was set to “Note”). So what will be the difference between the “.Text” field and the original lookup column in the search index. The difference is that the value of the original lookup column will be trimmed in the search index and will contain only the text value without the integer part preceding it. And if you issue an SQL search query against a managed property mapped to the crawled property of a lookup field you will be able to retrieve only the textual part of the lookup value (this holds also for the filtering and sorting operation for this field type). Whereas with the “.Text” derived field you will have access to the original unmodified value of the lookup field.
the fields derived from the “HTML” and “Image” publishing field type with the “.Html” suffix – they are populated with the original values of the corresponding fields with the original markup intact. Since the values of the original fields in the list item data XML are already trimmed the original values are retrieved with a simple trick. The “z:row” element for every list item contains the “ows_MetaInfo” attribute which contains a serialized property bag with all properties of the underlying SPFile object for the current list item. This property bag happens to contain all list item field values which are non-empty. So what I do in this case is to parse the value of the ows_MetaInfo attribute and retrieve the unmodified values for all “Html” and “Image” fields that I need. An important note here – the ows_MetaInfo attribute (and its corresponding system list field – MetaInfo) is available only for document libraries and is not present in non-library lists, which means that this trick is possible only for library-type lists.
the ows_ContentTypeId.Text field gets populated from the value of the original ows_ContentTypeId field/attribute. The difference between the two is that the derived one is defined in the schema as a “Note” field so its value is treated by the search index as a text property.
the ows_Env.*, fields get populated from service contextual data (see the implementation of the SiteDataHelper class). For the implementation of the XML modifications for the SiteData.GetChanges method these values are retrieved from the result XML itself. The value of the ows_Env.IsListItem is always set to 1 (its purpose is to be used as a flag defining a superset of the standard “isdocument” managed property).

Installation steps for the “hack” solution

Download and extract the contents of the zip archive.
Build the project in Visual Studio (it is a VS 2008 project).
The assembly file (Stefan.SharePoint.SiteData.dll) should be deployed to the GAC
The StefanSiteData.asmx file should be copied to {your 12 hive root}\ISAPI folder
The global.asax file should be copied to the root folder of your SharePoint web application. Note that you will have to backup the original global.asax file before you overwrite it with this one.
Open the web.config file in the target SharePoint web application and add an “appSettings” “add” element with key “UseSiteDataRewrite” and value “true”.
Note that if you have more than one front end servers in your farm you should repeat steps 3-6 on all machines.
After the installation is ready you can start the search crawler (full crawl) from the SSP admin site. It is a good idea if you have a content source only for the web application for which the custom SiteData service is enabled, so that you can see immediately the results of the custom service.
After the crawling is complete you should check the crawl log for errors – check whether there’re unusual errors which were not occurring before the installation of the custom SiteData service.
If there’re no errors in the crawl log you can check the “crawled properties” page in the SSP admin site – the new “dotted” crawled properties should be there and you can now create new managed properties that can be mapped to them.
Note that the newly created managed properties are not ready for use before you run a second full crawl for the target content source.

SharePoint Search 2007 – hacking the SiteData web service – Part I

2011-05-24T12:10:00.001-07:00

When I started preparing this posting I realized that it would be too long, so I decided to split it into two parts – the first one being more introductory and explaining some aspects of the inner workings of the SharePoint search engine, and the second one concentrating on the actual implementation of the “hack”. Then when I started the first part, which you are now reading, I felt that the posting’s title itself already raises several questions, so it would be a good idea to start with a brief Q & A which will help you get into the discussed matter. This is a short list of questions that you may have also asked yourself two sentences into the posting:

What is the relation between the SharePoint search service and the SharePoint SiteData web service in the first place?
Why would I need to change the working of the SharePoint search, what can be the reasons and motives for that?
Is it a wise idea and would you recommend using this hack?

And the answers come promptly:

To answer this one we need to have a closer look at the internal workings of the SharePoint search index engine. If you are not familiar with some core concepts and basic terminology like index engine, content sources, filter daemon, protocol handlers, IFilters, I would recommend that you first check these two MSDN articles – here (for a high level architecture overview) and here (for a high level overview of the protocol handlers). Let me start with several words about the protocol handlers – these are basically responsible for crawling the different types of content sources. They are implemented as COM components written in unmanaged code (C or C++). If you are familiar with COM Interop you will know that it is also possible to create COM components using .NET and managed code and in fact there is a sample .NET protocol handler project in CodePlex. I am not sure though how wise it is to create your own protocol handler with managed code (apart from the fact that it is quite complex to start with) knowing that all existing protocol handlers by Microsoft and third party vendors are written in unmanaged code.
You can check the available index engine protocols and their matching protocols handlers for your SharePoint installation in the Windows registry:

You can see that there are different protocol handlers for the different types of content sources – SharePoint sites, external web sites, file shares, BDC, etc. The name of the protocol handler (the “Data” column in the image above) is actually the ProgID (in COM terms) of the COM component that implements the handler.
In this posting we are only interested in just one of the protocol handlers – this is the one for the Sts3 protocol, which is responsible for crawling the content from SharePoint sites. The same handler is also used for the Sts3s protocol (see the image) which is again for SharePoint sites but which use the HTTPS (SSL) scheme. And now the interesting part – how does the Sts3 protocol handler traverse the content from SharePoint. The answer is actually also the answer of the first question in the list above – it calls the standard SharePoint SiteData web service (/_vti_bin/SiteData.asmx). If you wonder why for instance it doesn’t use the SharePoint object model directly – the main reason I think is for greater scalability (not to mention that it would be at best challenging to call managed from unmanaged code). The better scalability comes from the fact that the handler can be configured to call the SiteData web service from all available web front servers in the SharePoint farm, which can distribute better the workload and utilize better the resources of the farm. Later in the posting I will give you more details about how you can check and monitor the calls to the SiteData web service from the crawl engine and also some additional information about the exact methods of the SiteData service that are used for traversing the content of the SharePoint sites.
As I already mentioned in the answer for the first question, this posting deals specifically with the search functionality that targets SharePoint content. So, the motives to come to this hack are directly related to the using and querying of SharePoint data. The reasons and motives for these changes can be separated in two groups – the first one is more general - why use SharePoint search and not some other available alternative method. The second one is more specific – what is not available or well implemented in the SharePoint search query engine that needs to be changed or improved.
Let me start with the first group – out of the available methods to query and aggregate SharePoint content in the form of SharePoint list item and document metadata – SharePoint search doesn’t even come as the first or preferred option. Normally you would use the SharePoint object model with the SPList, SPListItem and SPQuery classes (for a single SharePoint list) or the SPSiteDataQuery class with the SPWeb.GetSiteData method (or alternatively the CrossListQueryInfo and CrossListQueryCache classes if you use the publishing infrastructure) – for querying and retrieving data from many lists in one site collection. The cross list query functionality is actually directly used in the standard SharePoint content by query web part (CQWP), so even without using custom code you may have experienced certain issues with it. Probably the biggest one is performance – maybe you’ve never seen it or you are well aware of it. This is because it becomes a real issue only if the size of your site collection in terms of the number of sub-sites becomes very big and you use queries that aggregate data from most of the available sub-sites. You can add to these two conditions the number of list items in the individual SharePoint lists which further degrades the performance. So, when does this become a visible issue – you can have various combinations of the above said conditions, but if you query more than one hundred sub-sites and/or you have more than several thousand items in every list (or many of the lists) you may see page loading times ranging from several seconds to well above a minute in certain extreme cases. And … this is an issue even with the built-in caching capabilities of the cross list query classes. As to why the caching doesn’t solve always the performance issue – there are several reasons (and cases) for that: first – there’re specific CAML queries for which the caching is not used at all (e.g. queries that contain the <UserID /> element); secondly – even if the caching works well, you have the first load that populates the cache which will still be slow, etc.
Let me now briefly explain why the cross list query has such performance issues (only in the above mentioned cases). The main reason is the fact that the content database contains all list data (all list items in the whole site collection, it may also contain more than one site collections) in a highly denormalized table called AllUserData. This design solution was totally deliberate because it allows all flexibility that we know with SharePoint lists in terms of the ability to add, modify and customize fields, which unfortunately comes with a price in some rare cases like this one. Let’s see how the cross list query works from a database perspective with a real example – let’s say that we have a site collection with one hundred sub-sites each containing an “announcements” list with two custom fields “expiration date” and “publication date”. On the home page of the root site we want to place a CQWP that displays the latest five announcements (aggregated from all sub-sites) ordered by publication date and for which the expiration date is in the future. Knowing that all list item data is contained in a single database table you may think that it may be possible to retrieve the aggregated data in a single SQL query but, alas, this is not the case. If you have a closer look at the AllUserData table you will find out that it contains columns whose names go: nvarchar1, nvarchar2, …, int1, int2, …, datetime1, datetime2, … – these are the underlying storage placeholders for the various types of SharePoint fields in your lists. Obviously the “publication date” and “expiration date” will be stored in two of the “datetimeN” SQL columns but the important thing is that for the different lists the mappings may be totally different, e.g. for list 1 “publication date” and “expiration date” may map to datetime1 and datetime2 respectively, whereas for list 2 they can map to datetime3 and datetime4 respectively. This heterogeneous storage pattern makes the retrieval much more complex and time costly – the object model first needs to extract the metadata for all target lists in these one hundred sites (which contains the mappings for the fields) and after that retrieve the items from all one hundred lists one by one making a SQL union with the correct field to SQL columns mappings and applying the filtering and sorting after that. If you are interested in checking that yourself you can use the SQL profiler tool that comes with the MS SQL management studio.
Having seen the performance issues that may arise with the usage of the cross list query built-in functionality of SharePoint, it is quite natural to check what SharePoint Search can offer as an alternative. Obviously it performs much faster in these cases and allows data retrieval and metadata filtering but the results and functionality it has are not exactly identical to the ones of the cross list query. And here we come to the second group of motives for implementing this kind of hack that I mentioned in the beginning of this paragraph. So let’s see some of the things that we’re missing in SharePoint search – from a data retrieval perspective – the text fields, especially the ones that contain HTML are returned by the search query with the mark-up stripped out (this is especially embarrassing for the publishing Image field type, whose values are stored as mark-up and get retrieved virtually empty by the search query); the “content type id” field is never crawled and cannot be used as a crawled and managed property; for the “lookup” field type (and derivative field types as the “user” type) – these are retrieved as plain text, with the lookup item ID contained in the field value stripped out; etc. From filtering and sorting perspective, you have pretty much everything needed – you can perform comparison operations on the basic value types – text, date, integer and float and perform the correct sorting based on the respective field type. What is missing is for instance the filtering on “lookup” (including “user”) fields based not on the textual value but on the integer (lookup ID) value – this is because this part of the lookup field value is simply ignored by the search crawler (we’ll come to that in the next part of the posting). For the same reason you cannot filter on the “content type id” field.
The next question is of course is it possible to achieve these things with the SharePoint search – the answer is yes, and the hack that is a subject of this posting does exactly that.
And lastly the third and most serious one – most of the time I am overly critical towards my own code and solutions, so I would normally not recommend using this hack (I will publish the source code in the second part of the posting), at least not in production environments. I would only suggest that you use it very limitedly in development/testing or small intranet environments if at all. I suppose that the material in the posting about some of the inner workings of the indexing engine and the SiteData web service would be interesting and useful by itself.

So, let’s now see how the index engine or more precisely the Sts3 protocol handler calls the SiteData web service. Basically you can track the SiteData.asmx invocations by simply checking the IIS logs of your web front server or servers (you have to have IIS logging enabled beforehand). If you first run a full crawl on one of your “SharePoint Site” content sources from the SSP admin site and after it completes open the latest IIS log file you will see that there will be many request to _vti_bin/SiteData.asmx and also to all pages and documents available in the SharePoint sites that were listed in the selected content source. It is logical to conclude that the protocol handler calls the SiteData web service to traverse the existing SharePoint hierarchy and to also fetch the available metadata for the SharePoint list items and documents and then it also opens every page and document and scans/indexes their contents so that they are later available for the full text search queries.

The checking of the IIS logs was in fact the first thing that I tried when I began investigating the SiteData-SharePoint Search relation but I was also curious to find out what method or methods exactly of the SiteData web service get called when the crawler runs. If you have a look at the documentation of the SiteData web service you will see that some of its methods like GetSite, GetWeb, GetListCollection, GetList, GetListItems, etc. look like ideal candidates for traversing the SharePoint site hierarchy starting from the site collection level down to the list item level. The IIS logs couldn’t help me here because they don’t track the POST body of the HTTP requests, which is exactly the place where the XML of the SOAP request is put. So I needed a little bit more verbose tracking here and I quickly came up with a bit ugly but working solution – I simply modified the global.asax of my test SharePoint web application like this:

<%@ Assembly Name="Microsoft.SharePoint"%><%@ Application Language="C#" Inherits="Microsoft.SharePoint.ApplicationRuntime.SPHttpApplication" %>
<%@ Import Namespace="System.IO" %>
<script RunAt="server">
 
    protected void Application_BeginRequest(object sender, EventArgs e)
    {
        TraceUri();
    }
 
    protected void TraceUri()
    {
        const string path = @"c:\temp\wssiis.log";
        try
        {
            HttpRequest request = HttpContext.Current.Request;
            DateTime date = DateTime.Now;
            string httpMethod = request.HttpMethod;
            string url = request.Url.ToString();
            string soapAction = request.Headers["SoapAction"] ?? string.Empty;
            string inputStream = string.Empty;
 
            if (string.Compare(httpMethod, "post", true) == 0)
            {
                request.InputStream.Position = 0;
                StreamReader sr = new StreamReader(request.InputStream);
                inputStream = sr.ReadToEnd();
                request.InputStream.Position = 0;
            }
 
            string msg = string.Format("{0}, {1}, {2}, {3}, {4}\r\n", date, httpMethod, url, soapAction, inputStream);
 
            File.AppendAllText(path, msg);
        }
        catch { }
    }
</script>

The code is pretty simple – it hooks onto the BeginRequest event of the HttpApplication class which effectively enables it to track several pieces of useful information for every HTTP request made against the target SharePoint web application. So, apart from the date and time of the request, the requested URL and the HTTP method (GET, POST or some other) I also track the “SoapAction” HTTP header which contains the name of the SOAP method for a web service call and also the POST content of the HTTP request which contains the XML of the SOAP request (in the case of a web service call). The SOAP request body contains all parameters that are passed to the web service method call, so by tracking this I could have everything I wanted – the exact web service method being called and the exact values of the parameters that were being passed to it. Just to quickly make an important note about this code – don’t use it for anything serious, I created it only for testing and quick tracking purposes.

With this small custom tracking of mine enabled I ran a full crawl of my test web application again and after the crawl completed I opened the log file (the tracking code writes to a plain text file in a hard-coded disc location) and to my surprise I saw that only two methods of the SiteData web service were called – GetContent and GetURLSegments. Actually the real job was obviously done by the GetContent method – there were about 30-35 calls to it, and only one call to GetURLSegments. You can see the actual trace file that I had after running the full crawl here. My test web application was very small containing only one site collection with a single site, so the trace file is very small and easy to follow. The fourth column contains something that looks like an URL address but this is in fact the value of the “SoapAction” HTTP header – the last part of this “URL” is in fact the actual method that was called in the SiteData web service. The fifth column contains the XML of the SOAP request that was used for the web service calls – you can see the parameters that were passed to the SiteData.GetContent method inside. If you check the MSDN documentation about the SiteData.GetContent method you will see that its first parameter is of type “ObjectType” which is an enumeration. The possible values of this enumeration are: VirtualServer, ContentDatabase, SiteCollection, Site, Folder, List, ListItem, ListItemAttachments. As one can deduce from this enumeration, the GetContent method is designed and obviously used for hierarchy traversing and metadata retrieval (the MSDN article explicitly mentions that in the yellow note box at the bottom). If you check the sample trace file from my test site again you will see that the calls made by the crawler indeed start with a call using ObjectData.VirtualServer and continue down the hierarchy with ObjectData.ContentDatabase, ObjectData.SiteCollection, etc. You may notice something interesting – after the calls with ObjectData.List there’re no calls with ObjectData.ListItem. Actually in the trace file there is only one call to GetContent using ObjectData.ListItem and it is invoked for the corresponding list item of the home (welcome) page of the site, which in my case was a publishing page. The other method of the SiteData web service – GetURLSegments is also called for the home page only – it basically returns the containing site and list of the corresponding list item by providing the URL of the page. And if you wonder which option is used for retrieving list items – it is neither the ObjectData.List nor the ObjectData.ListItem. The former returns an XML fragment containing mostly the list metadata and the latter the metadata of a single list item. The option that actually returns the metadata of multiple list items is ObjectData.Folder. Even though the name is a bit misleading this option can be used in two cases – to retrieve the files from a folder that is not in a SharePoint list or library (e.g. the root folder of a SharePoint site) or to retrieve the list items from a SharePoint list/library. If you check the sample trace file you will see that the GetContent method is not called with ObjectData.Folder for all lists – this is because the crawler is smart enough and doesn’t call it for empty lists (and most of the lists in my site were empty). And the crawler knows that a particular list is empty by the preceding GetContent (ObjectData.List) call which returns the ItemCount property of the list. There is one other interesting thing about how the crawler uses the GetContent with ObjectData.Folder – if the list contains a big number of items, the crawler doesn’t retrieve all of them with one call to GetContent but instead reads them in chunks of two thousand items each (the logic in SharePoint 2010 is even better – it determines the number of items in a batch depending on the number of fields that the items in the particular list have). And … about the return value of the GetContent method – it is in all cases an XML document that contains the metadata for the requested object or objects. It is interesting to note that the XML also contains the permissions data associated with the object which is obviously used by the indexing engine to maintain ACL-s for the various items in its index which allows the query engine to apply appropriate security trimming based on the permissions of the user that issues the search query. For the purposes of this posting we are mostly interested in the result XML for the ObjectData.List and ObjectData.Folder GetContent invocations – here’re two sample XML fragments from GetContent (List) and GetContent (Folder) calls. Well, indeed they seem quite … SharePoint-ish. Except for the permissions parts, the GetContent (Folder) yields pretty much the same XML as the standard Lists.GetListItems web service method. Have a look at the attributes containing the field values in the list items – these start with the well-known “ows_” prefix, which is the very same prefix that we see in the crawled properties associated with SharePoint content. Another small detail to note is that the GetContent (Folder)’s XML is not exactly well formed – for example it contains not properly escaped new line characters inside attribute values (not that this prevents it from rendering normally in IE) – I will come again to this point in the second part of this posting.

So far so good, but the results above are from a full crawl. And what happens when we run an incremental crawl? Have a look at the sample trace file that I got when I ran an incremental crawl on my test web application after i had changed several list items and had created a new sub-site and several lists in it. You can see that it contains again several calls to SiteData.GetContent, one call to SiteData.GetURLSegments and this time one call to SiteData.GetChanges. If you wonder why there is only one call to SiteData.GetChanges – a quick look at the result XML of this method will explain most of it. If you open the sample XML file you will see that the XML is something like a merged document from the results of the GetContent method for all levels from “ContentDatabase” down to “ListItem” … but containing only the parts of the SharePoint hierarchy whose leaf descendants (that is list items) got changed since the time of the last crawl. So basically, with one call the crawler can get all the changes in the entire content database … well, almost. Unless there are too many changes – in this cases the method is called several times each time retrieving a certain number of changes and then continuing after the reached book-mark. If you check the documentation of the GetChanges method in MSDN you will see that its first parameter is again of type ObjectData. Unlike the GetContent method however, you can use it here only with the “ContentDatabase” and “SiteCollection” values (the rest of the possible values of the enumeration are ignored and the returned XML if you use them is the same as with the “ContentDatabase” option). And one last thing in the case of the incremental crawl – the calls to the GetContent method are only for new site collections, sites and lists (which is normally to expect). The metadata for new, updated and deleted list items in existing lists is retrieved with the call to the GetChanges method.

So, this was in short the mechanism of the interaction between the SharePoint Search 2007 indexing engine (the Sts3 protocol handler) and the SharePoint SiteData web service. In the second part of this posting I will continue with explaining how I got to hack the SiteData web service and what the results of this hack were for the standard SharePoint search functionality.

Pluggable Content Query web part

2011-03-17T07:36:00.001-07:00

Sub-classing the standard SharePoint ContentByQueryWebPart is a common approach for extending the default set of functionality that this web part offers. There are many examples for sub-classing the CQWP on the internet – for instance this sample by Andrew Connell that handles URL query parameters for adding dynamic filtering in the web part, or the paging enabled CQWP version by Waldek Mastykarz, just to name a few.

In most cases the need to sub-class the CQWP comes with some extra requirements for filtering or modifying the result set (SharePoint list items or documents) that the CQWP is supposed to present. The standard implementation of the CQWP of course has many filtering options itself but if you want some additional filtering based on dynamic conditions (as it is the case with the URL query parameters) you will have to consider the extending of the standard implementation by sub-classing the ContentByQueryWebPart class. As for the URL query parameters example, the SharePoint 2010 version of the CQWP has some limited built-in support for handling query parameters, but this still may be far from adequate in many cases.

I myself have sub-classed the CQWP in a handful of SharePoint projects of mine, so now I have maybe at least ten different implementations of cases where I had to handle some similar or not that similar requirements all based on the premises of having to have some sort of modification of the original CQWP result set. And some time ago I brain-stormed a bit the whole issue coming up with the idea that the whole problem can be handled in a bit more generalized way – if you for instance separate the web part implementation and the result set filtering/modifying logic implementation in a sort of pluggable architecture. Basically, the idea is pretty simple and further facilitated by the fact that you can have as many custom persistable properties in a web part as you need. The “plugin” can be as simple as a .NET class implementing a predefined .NET interface and the web part can be configured with a dedicated string property containing the fully qualified type name of this class, which will allow the web part to create instances of it using reflection. The web part will then be implemented to call on the methods of the plugin class which will be implementing the “contract” specified in the predefined .NET interface that I mentioned. The whole idea is that on the side of the extended CQWP there will be only a single implementation instead of many different ones for all possible cases that you may have and the whole effort would be for the part with the filtering/modification “plugin” class (classes) where it will be possible to concentrate on the retrieving/filtering specifics instead of having to pay attention to web part and presentation details.

And now to the more interesting part – how is it actually possible to hook onto the standard CQWP web part with your inheriting class so that you can modify the result set that the CQWP normally produces. Basically, there are a few options available here – it can be as simple as setting or modifying the values of the built-in filtering properties of the CQWP – for instance: FilterField1, FilterValue1, FilterType1, FilterOperator1, Filter1ChainingOperator, ListsOverride, QueryOverride, etc. Another approach is to use the handy ProcessDataDelegate property to which you can provide a delegate referencing a custom method receiving a DataTable parameter with a return type a DataTable as well. The idea of this delegate property is, that you can provide a custom method of yours to it, and it will be called with a DataTable parameter that will contain the freshly retrieved result set that the CQWP is about to display and you will have the chance to modify this DataTable instance (and its contents) or to even create a new DataTable instance and reuse, modify or filter the items from the original DataTable. The modified or the newly created DataTable then you will return from the custom method to the CQWP so that it goes to the presentation handling logic (check Waldek’s posting at the top for more details).

Having briefly mentioned these two approaches for implementing some extra filtering in the CQWP and before I continue with describing the actual approach in my implementation I think that it will be a good moment here to give you some more details on the internal workings of the CQWP and more specifically on its data retrieving logic so that you can better understand the whole hooking procedure in the inheriting “pluggable” CQWP. So, the first thing here, which is probably familiar to most of you is that the CQWP uses internally the CrossListQueryCache and CrossListQueryInfo classes combo for retrieving aggregated cross list data. These two classes are actually a thin wrapper on top of the SPWeb.GetSiteData method and the SPSiteDataQuery class in which you provide the CAML query details for the cross list data call. What the CrossListQueryCache and CrossListQueryInfo classes add on top of the base implementation is some caching support and support for audience filtering and grouping. In theory this is pretty simple, the using of the cross list classes or the SPWeb.GetSiteData method is pretty common in many cases even not related to the CQWP at all. Apart from this there are several important facts giving some further details about the exact implementation of the cross list query invocation in the CQWP:

The call of CrossListQueryCache.GetSiteData method that retrieves the list data for the CQWP takes place in the GetXPathNavigator virtual method (actually in a private method called by the latter) of the web part. Note that the GetXPathNavigator is a virtual method which means that you can override it thus it is an ideal candidate for hooking onto the base implementation of the CQWP, when it comes to modifying its data retrieval logic.
The DataTable instance returned by the CrossListQueryCache.GetSiteData method is assigned to the Data public property of the CQWP. This property has a public getter and setter and even more importantly – the CQWP issues the CrossListQueryCache.GetSiteData call only if the Data property is not initialized and has the default null value.
The CrossListQueryCache class requires an instance of the CrossListQueryInfo class in its constructor. The CrossListQueryInfo has properties like the Query, Lists, Webs, RowLimit, etc. in which you specify CAML fragments that determine what list data you want to retrieve from the target site collection. The CQWP creates and initializes a CrossListQueryInfo instance based on its filtering properties, some of which I mentioned above: FilterField1, FilterValue1, FilterType1, FilterOperator1, Filter1ChainingOperator, ListsOverride, QueryOverride, etc. The good news here is that the CQWP provides a public method which you can use to get a CrossListQueryInfo instance with the exactly same configuration as the one that the web part uses for its internal cross list query call. This method is BuildCbqQueryVersionInfo – it actually returns an instance of the CbqQueryVersionInfo class whose CbqQueryVersionInfo.VersionCrossListQueryInfo property contains the CrossListQueryInfo instance that we need.
And lastly – the call to the delegate ProcessDataDelegate property that I mentioned above also takes place in the CQWP’s GetXPathNavigator virtual method. Actually this happens at the same moment when the web part gets the list data in the DataTable instance from the CrossListQueryCache.GetSiteData call.

With all these facts it is easy now to devise a strategy for implementing the hooking logic in the inheriting class. It is obvious that the whole logic can be placed in the overridden version of the GetXPathNavigator method, just before you call the base CQWP implementation (base.GetXPathNavigator();) in the method body. For the plugin logic I thought of three alternative types of overriding the standard CQWP data retrieving and filtering logic:

The first type of override is the most radical one – you directly circumvent the standard logic for retrieving the list data in the CQWP and produce your own DataTable instance in some custom way. The only thing that you will need to do after producing the DataTable instance is to assign it to the CQWP’s “Data” property. This way you will ensure that when you call the base GetXPathNavigator method, the CQWP won’t issue its standard cross list query call, thus there won’t be any performance issues because of the double data retrieval. As of how you are going to get your DataTable instance – you have many possible choices here depending on your particular case. I can think of for instance using the SharePoint search functionality, using data in an external SQL database or even using again the CrossListQueryCache.GetSiteData method but applying some custom caching logic instead of relying on the built-in one if it doesn’t fit in your requirements.
The second type of override is to manipulate just the CrossListQueryInfo instance that will be used for the CrossListQueryCache.GetSiteData call. This instance will contain the initial configuration of the web part and you can add some additional filtering or scoping logic to it based on your requirements. Then the inheriting web part will issue the call to the CrossListQueryCache.GetSiteData method and will again set the CQWP’s Data property, so that the base implementation doesn’t make a second call. Note also that in this type of overriding the CQWP’s filtering you have to deal only with a CrossListQueryInfo instance and manipulate the CAML fragments in its properties (merge some extra filtering CAML, etc.) instead of having to modify directly the CQWP’s properties like the FilterField1, FilterField2, FilterField3, etc. which the end user may have already set values to using the SharePoint UI.
The third type of override is to simply use the standard delegate ProcessDataDelegate property override mechanism – the CQWP will make the cross list query call itself and we will only manipulate the DataTable instance (or create a new instance using the rows from the latter) that it has already retrieved.

And now, let’s move on to some code snippets which will better illustrate the implementation of the pluggable CQWP. You can download the project containing the pluggable CQWP from here. Let me first show you the code of the .NET interface that should be implemented by the custom plugin classes for the pluggable CQWP:

    public interface IContentQueryPlugin
    {
        // should return true if the pluging can generate the data for the web part - the GetDataResults method should produce a DataTable instance
        bool CanGetDataResults(PluggableContentByQuery part);
        // should return true if the plugin needs to modify the CrossListQueryInfo instance that the CQWP uses for its internal cross list query - the ProcessQueryInfo should be implemented in this case
        bool CanProcessQueryInfo(PluggableContentByQuery part);
        // should return true if the plugin needs to modify the DataTable returned by the CQWP cross list query - the ProcessData method should be implemented
        bool CanProcessData(PluggableContentByQuery part);
 
        // implement this if you want the retrieve the data without using the CQWP built-in retrieval mechanism
        DataTable GetDataResults(PluggableContentByQuery part, CrossListQueryInfo queryInfo);
        // implement this if you want to modify the CrossListQueryInfo instance before the CQWP issues its cross list query
        CrossListQueryInfo ProcessQueryInfo(PluggableContentByQuery part, CrossListQueryInfo queryInfo);
        // implement this if you want to modify the already retrieved by the CQWP DataTable with the results of the cross list query
        DataTable ProcessData(PluggableContentByQuery part, DataTable data);
    }

As you see the interface contains six methods, but actually the three types of overrides that I mentioned above are implemented by only three of the methods: GetDataResults, ProcessQueryInfo and ProcessData. The other three methods are sort of auxiliary methods in the sense that you have one auxiliary method corresponding to one of the implementation methods – the auxiliary methods all start with the verb “can” prefix. Their purpose is pretty transparent – since the types of overrides that you can implement are optional (and even mutually exclusive – the first one versus the second two) with the auxiliary methods you can simply specify whether you are implementing a particular override. For instance if you want to take care of retrieving the data results yourself you are going to implement the GetDataResults method and also you will have to implement the CanGetDataResults auxiliary method by simply returning “true” in its implementation (for the other “can” methods you will have to return “false”).

Let’s now have a closer look at the CQWP overrides implementing methods:

the GetDataResults method – as I mentioned you go for this method if you want to have your custom data retrieving logic and circumvent the CQWP’s normal data retrieving routine altogether. As you see, the method’s return type is a DataTable, meaning that you will have to gather your result items in some custom way and then produce a DataTable instance that you will pass back to the CQWP. The method accepts two parameters – a reference to the “parent” CQWP and a CrossListQueryInfo object. The latter will contain all CAML fragments for the cross list query that the web part would have issued – you can use that for the filtering against your custom data source if you use one. Further you can use the reference to the web part to inspect the properties of the latter and make use of their values in some way or another.
the ProcessQueryInfo method – you can use this method if you want to only add some additional CAML markup or make some other modifications to the CrossListQueryInfo object that the CQWP constructs for its cross list query call. This means that you are opting only for some additional filtering but the data retrieval method remains the standard one for the CQWP. The method accepts CrossListQueryInfo parameter and also returns a CrossListQueryInfo object. You can either make your modifications to the input CrossListQueryInfo instance and then return the same instance, or you can optionally create a brand new CrossListQueryInfo instance, copy some or all of the properties of the source instance and then return the new one.
the ProcessData method – if you opt for this override it will mean that you don’t want the change the CAML filtering and will only want to modify the DataTable instance that the CQWP produces with its standard cross list query call. You can perform all sorts of changes to the DataTable instance like adding new columns to it, add or remove data rows, etc. The removing of data rows will effectively be equivalent to applying some extra filtering, which in many cases will be better if implemented in CAML, meaning that probably there will be a better solution if using the ProcessQueryInfo to inject the filtering in the CrossListQueryInfo’s CAML.

Let me now show you how the IContentQueryPlugin interface fits in the overriding procedure of the CQWP. As I mentioned above, the only hooking point that we need to change the data retrieving logic of the CQWP is the virtual GetXPathNavigator method, and this is its overridden version in the pluggable CQWP:

        protected override XPathNavigator GetXPathNavigator(string viewPath)
        {
            // This method is the single point where we need to hook to get the whole plugin thing working.
            // The standard CQWP issues the cross list query in its implementation of the GetXPathNavigator method
            // setting the CQWP's Data property with a DataTable instance holding the results of the query.
            // The CQWP issues its query only if the Data property is null, so we can even set it with 
            // a customly retrieved DataTable if we want before calling base.GetXPathNavigator(viewPath).
 
            // first get the plugin instance
            IContentQueryPlugin plugin = this.GetQueryPlugin();
            if (plugin != null)
            {
                // check if the plugin wants to generate the data results first
                if (plugin.CanGetDataResults(this))
                {
                    // if so, provide it with a CrossListQueryInfo instance and set the Data property of the web part
                    CrossListQueryInfo queryInfo = this.BuildCbqQueryVersionInfo().VersionCrossListQueryInfo;
                    this.Data = plugin.GetDataResults(this, queryInfo);
                }
 
                // then check if the plugin wants to modify the original CrossListQueryInfo instance (check this only if the Data property is still null)
#if SHAREPOINT2010
                if (this.Results == null && plugin.CanProcessQueryInfo(this))
#else
                if (this.Data == null && plugin.CanProcessQueryInfo(this))
#endif
                {
                    // get the CrossListQueryInfo instance and pass it to the plugin
                    CrossListQueryInfo queryInfo = this.BuildCbqQueryVersionInfo().VersionCrossListQueryInfo;
                    queryInfo = plugin.ProcessQueryInfo(this, queryInfo);
                    // after this run the cross list query to populate the Data property
                    IssueQuery(queryInfo);
                }
 
                // set the CQWP ProcessDataDelegate property if the plugin needs to post-process the results DataTable
                if (plugin.CanProcessData(this))
                {
                    this.ProcessDataDelegate = (dt) =>
                    {
                        return plugin.ProcessData(this, dt);
                    };
                }
            }
 
            // only after we finish the plugin stuff call the base CQWP GetXPathNavigator implementation
            return base.GetXPathNavigator(viewPath);
        }

You can see that the implementation is actually pretty concise and straight-forward. Basically the implementation of the overriding GetXPathNavigator method simply dispatches the calls for modifying the data retrieving logic of the CQWP to the plugin class that implements the IContentQueryPlugin interface. The creating of the plugin class instance is also a pretty trivial thing – the pluggable CQWP has a custom string property called PluginClassName, which as its name suggests should be set to contain the fully qualified type name (plus the full assembly name if the plugin class is not in the same assembly as the assembly of the pluggable CQWP) of the plugin class. The pluggable CQWP creates the plugin class instance using reflection (there’s also the option to cache the plugin class instance – check the source code for details). Note also that you can compile the web part for SharePoint 2010 too but you will need to define an extra Debug symbol in the settings of the Visual Studio project of the web part (check the conditional compiling statements in the source code).

The Visual Studio project with the pluggable CQWP (download from here) also contains a sample plugin class implementing the IContentQueryPlugin interface. It implements only the IContentQueryPlugin.ProcessQueryInfo meaning that it only modifies the original CAML of the CQWP’s cross list query call. This is the source code of its implementation of the ProcessQueryInfo method:

        public CrossListQueryInfo ProcessQueryInfo(PluggableContentByQuery part, CrossListQueryInfo queryInfo)
        {
            // check the FilterFieldN and FilterValueN query parameters 0 through 9
            string[][] args = (new int[10]).Select((i, idx) => new string[] { HttpContext.Current.Request.QueryString["FilterField" + idx], HttpContext.Current.Request.QueryString["FilterValue" + idx] })
                .Where (arr => !string.IsNullOrEmpty(arr[0]) && !string.IsNullOrEmpty(arr[1]))
                .ToArray();
            // if none found or their values are empty just exit
            if (args.Length == 0) return queryInfo;
 
            // get the CAML query in an XElement
            XElement queryElement = XElement.Parse("<Query>" + queryInfo.Query + "</Query>");
            // retrieve the Where clause or create a new one if missing
            XElement whereEelement = queryElement.Element("Where");
            if (whereEelement == null)
            {
                whereEelement = new XElement("Where");
                queryElement.Add(whereEelement);
            }
 
            // get the existing where clause if present and append all comparison clauses from the query parameters
            string[] clauses = whereEelement.Elements().Select(el => el.ToString())
                .Union(args.Select(arg => FormatComparisonQuery(arg[0], arg[1], "Eq")))
                .ToArray();
 
            // change the contents of the Where XElement performing a CAML 'And' on the clauses array
            whereEelement.RemoveAll();
            whereEelement.Add(XElement.Parse(FormatLogicalOpQuery("And", clauses)));
 
            // set the Query property of the CrossListQueryInfo instance
            queryInfo.Query = string.Concat (queryElement.Elements().Select(el => el.ToString()).ToArray());
            // return the modified CrossListQueryInfo
            return queryInfo;
        }

Some bits of the full implementation are missing (there are also several helper private methods in the plugin class – check the source code in the project for details) but you can see that the method simply “injects” some extra CAML in the Query property of the source CrossListQueryInfo parameter (the Query property may already contain some CAML that comes from the particular query configuration of the CQWP). The additional filtering that the sample plugin class is dynamically determined by the presence of certain URL query parameters that you use to call the page containing the web part. Actually these are no other than “FilterField1” and “FilterValue1”, the same as the standard LV and XLV web parts use and also the same as Andrew Connell uses in his sample of sub-classing the CQWP that I mentioned in the beginning.

In order that you can see the plugin class working with the pluggable CQWP you will need to set its fully qualified type name in the PluginClassName property of the web part. You can easily do that from the standard SharePoint UI – you will have to put this value for the sample plugin class: “Stefan.SharePoint.WebParts.QueryStringPlugin, Stefan.SharePoint.WebParts, Version=1.0.0.0, Culture=neutral, PublicKeyToken=f9bc508a219e6843”.

New version of the Site Template Configurator utility

2011-02-01T12:37:00.001-08:00

This posting is a brief announcement of the new and enhanced version of my Site Template Configurator utility, which I first introduced several months ago in another posting of mine. You can download the source code of the utility from here.

Here is a short list of the changes and enhancements in this new version:

The tool is now capable of generating “wsp” packages and also has commands for “wsp” installing and uninstalling, both in two flavors – local (for a single machine farm) and farm (launching timer jobs on all machines in the farm). The “wsp” generation and handling functionality neatly packs the custom site templates that you create and make them easy to deploy to other SharePoint farms (see the notes about the “wsp” generation below).

The “Generate artifacts…” and the “wsp” export and install/uninstall commands are now available in the context menu of the “project/root” node of the left hand pane hierarchy. The “project” menu in the program’s toolbar now contains only the file handling commands for the “template project” file.
The “project/root” node now contains an extra property – “SiteTemplatePrefix”. This allows you to specify a common prefix for the newly generated site templates’ folders in the “TEMPLATE/SiteTemplates” system folder. If you don’t specify one the tool will use long and ugly Guid containing names as in the previous version.
The source code comes with two “sln” files – one for Visual Studio 2008 and one for Visual Studio 2010. The tool also compiles for both SharePoint 2007 and SharePoint 2010.
The tool now displays a “close warning” prompt if you have modified your “template project” and want to exit without saving the changes.

And several handy notes about the tool:

The tool uses the “makecab” Windows utility to create “wsp” packages, so you need to have it installed on the machine where you will use the utility. Alternatively on machines with SharePoint 2010, the tool may be compiled to use the SharePoint 2010 built-in CAB file support (the functionality is not publicly accessible, but the tool’s code uses a couple of reflection tricks to make use of it). In order that the tool is compiled to use this, you need to specify the “SHAREPOINT2010” compilation symbol in the project settings for the projects in the Visual Studio solution.
Specify appropriate values in the “SiteTemplatePrefix” and “StartTemplateID” properties of the “project/root” node. The latter specifies the starting ID value of the “TEMPLATE” elements in the generated WEBTEMP*.xml file – make sure, that you don’t have several projects that use one and the same starting or overlapping ID values (these should be unique in the SharePoint farm). Use the “Generate artifacts …” command of the “project/root” node and then carefully inspect the configuration files (and the subfolders that they are placed in), that the tool generates – make sure that when these get copied to the TEMPLATE/SiteTemplates and TEMPLATE/1033 system folders, they won’t overwrite existing files there.
Check the “IncludeLists” and “IncludeModules” properties of the “template” nodes (the nodes with blue icons). These have the value “false” by default, which means that the “List” and “Module” elements from the ONET.XML of the base template won’t get executed for your custom template. If you don’t have the “IncludeModules” property enabled, and you don’t have a custom feature that provisions another default/welcome page for your site template, the sites based on this custom template will end up with no default/welcome page.

Automatically publishing files provisioned with Sandboxed Solutions

2011-01-29T09:48:00.001-08:00

I couple of days ago I saw this posting on Waldek Mastykarz’s blog (a great blog by the way) and now you are reading my posting with the very same name, which treats also the very same issue, but provides a different solution to it. So, what exactly is the issue with file provisioning with sandbox solutions? In short – the problem occurs when you use “Module” elements in a feature for provisioning files to SharePoint document libraries. And not to any document library but to a library which is configured with one of the following:

Check out for files is required
Content approval is enabled
Minor versions are enabled

The files will be of course provisioned to the target library but they will be either checked out and/or in a draft or pending state depending on the enabled configurations and the actual combination of the three that is enabled on the library. And this will mean that the files will be present but they won’t be accessible for the regular users of the site. This behavior is obviously intended and by design – I suppose that the main reason for it is the ability for the site and site collection administrators to review and approve or reject the content provisioned by the sandbox solution.

Having said that, I should mention that I knew about this issue for quite some time myself and even had tackled one isolated case of it in a previous posting of mine – it is about provisioning of publishing pages with sandbox solutions. The problem with publishing pages was even more serious – not only the pages end up in a draft state and checked out, but the web parts defined in the “Module” manifest file never get provisioned (check the posting itself for more details). The solution for publishing pages was pretty neat and simple (it doesn’t need a feature receiver and code with it) but unfortunately it is applicable only for publishing pages.

As for the solution itself – it is known that the SPFile.CheckIn, SPFile.Publish and SPFile.Approve methods are available in sandbox solutions too, so with a feature receiver and several lines of code the solution would be pretty trivial. But obviously in this case the best solution would be if you have a reusable feature receiver that you write just once and then can use everywhere without changing or adjusting its code. The universal feature receiver will save you a lot of time and efforts since the usage of “Module” features is pretty frequent in SharePoint. The main technical challenge for this universal feature receiver is that the code in the receiver should be “aware” of which files exactly its feature provisions to the target library (libraries). This type of “awareness” is provided OOB by the built in SPFeatureDefinition.GetElementDefinitions method but unfortunately this method is not available in the sandbox subset of the object model. So we need a different solution and I will start with briefly explaining the solution from Waldek’s blog – it is pretty simple and requires very little additional effort – you just add an extra “Property” element to every “File” element in your “Module” manifest file like that:

<File Path="Images\someimage.png" Url="someimage.png" Type="GhostableInLibrary">
  <Property Name="FeatureId" Value="$SharePoint.Feature.Id$" Type="string"/>
</File>

The idea is as follows – since you won’t have a column name with the name “FeatureId” in your target library, the value of this property element will be saved in the property bag of the underlying SPFile instance of the provisioned file. And the value of the property element is none other but the Guid of the parent feature (note the smart usage of the Visual Studio token syntax). So basically by adding this extra “Property” element to all “File” elements in the manifest file you literally “mark” all files that your feature provisions. The logic then of the feature receiver will be to iterate all files in the target library and after finding all marked files to check them in, approve or publish them where appropriately. Additionally you will have to provide somehow the name/names of the target library/libraries to the feature receiver, which is possible by using feature properties in the feature.xml file of your feature.

And, now to the reasons as to why I started this article and thought of a different solution – the solution above is basically pretty neat and straight-forward and involves very little extra implementation effort, but what I didn’t like about it was the iteration of all files in the target library(libraries), which may happen to contain thousands of files (which in fact is very improbable but this thought always gives me the creeps when I think of some aspects of SharePoint performance). Of course there’re several ways to optimize the iteration – for instance, to iterate the SPList.Items collection instead of the SPFolder/SPFile hierarchy (the former is generally faster) and use the contents of the internal system MetaInfo list column which contains the serialized property bag of the underlying SPFile instance (unfortunately you cannot use CAML filtering on this column). Further, a very simple but effective optimization would be to use CAML filtering on the “Created”, “_Level” and “_ModerationStatus” fields so that you can fetch only items just recently created (say you can put a margin of 5 or 10 minutes in the past; note that even if the file exists already in the library, when the feature provisions it again, its “Created” date will also get updated) and are either checked out or in a draft or pending state.

Thus far so good, but still I wanted to find a solution that is as close to the “self awareness” approach with the SPFeatureDefinition.GetElementDefinitions method, so that the feature receiver knows exactly which files its feature provisions without it being necessary to traverse the target library or libraries and search for them. And what occurred to me was that if we need the contents of the manifest file in the feature receiver why can’t we simply provision it to the target site (it is available in the feature definition as an element file but there is nothing wrong if it is also referenced in a “File” element in the manifest file), then read its contents from the feature receiver, and finally when the files are published the feature receiver can safely delete it. And let me show you a sample “Module” manifest file so that you can get a better idea of the trick that I just explained:

<Elements xmlns="http://schemas.microsoft.com/sharepoint/">
  <Module Name="TestModule" Url="Style Library">
    <File Path="TestModule\Sample.txt" Url="Sample.txt" Type="GhostableInLibrary" />
    <File Path="TestModule\Sample2.txt" Url="Sample2.txt" Type="GhostableInLibrary" />
    <File Path="TestModule\Sample3.txt" Url="test/Sample3.txt" />
  </Module>
  <Module Name="TestModule2">
    <File Path="TestModule\Elements.xml" Url="Elements_$SharePoint.Feature.Id$.xml" Type="Ghostable" />
  </Module>
</Elements>

You can see that the manifest file (whose name is the commonplace “elements.xml”) contains two “Module” elements – the first one provisions several files to the standard “Style Library” library (a pretty recurring task in SharePoint development) and check carefully the second one – this is the only extra bit in the manifest file that you need (the other bit is the reusable feature receiver) – this is a “Module” element that provisions the manifest file itself. Note two things – the manifest file gets provisioned to the root folder of the target site – we don’t want to put the file in a document library because this way it will be directly visible to the site users. And secondly – check the URL of the file – it contains the Guid of the feature and the feature receiver will use exactly this so that it can locate the file when it gets to be executed. The pattern of the URL should be like this: [any number of characters that are unique among the “Module” manifest files in the parent feature]_$SharePoint.Feature.Id$.xml (the part from the underscore character on should be always the same) – basically we want to get a unique target name for every “Module” manifest file in this feature. And here is the code of the feature receiver itself:

    public class TestFeatureEventReceiver : SPFeatureReceiver
    {
        // The SharePoint elements file namespace
        private static readonly XNamespace WS = "http://schemas.microsoft.com/sharepoint/";
 
        public override void FeatureActivated(SPFeatureReceiverProperties properties)
        {
            // make it work for both 'Site' and 'Web' scoped features
            SPWeb web = properties.Feature.Parent as SPWeb;
            if (web == null && properties.Feature.Parent is SPSite) web = ((SPSite)properties.Feature.Parent).RootWeb;
            if (web != null) CheckinFiles(web, properties.Feature.DefinitionId);
        }
 
        private void CheckinFiles(SPWeb web, Guid featureID)
        {
            // create a regular expression pattern for the manifest files
            string pattern = string.Format(@"^.+_{0}.xml$", featureID);
            Regex fileNameRE = new Regex(pattern, RegexOptions.Compiled | RegexOptions.IgnoreCase);
 
            // get the manifest files from the root folder of the site
            SPFile[] manifestFiles = web.RootFolder.Files.Cast<SPFile>().Where(f => fileNameRE.IsMatch(f.Name)).ToArray();
            try
            {
                // iterate the manifest files
                foreach (SPFile manifestFile in manifestFiles)
                {
                    // load the contents of the manifest file in an XDocument
                    MemoryStream mStream = new MemoryStream(manifestFile.OpenBinary());
                    StreamReader reader = new StreamReader(mStream, true);
                    XDocument manifestDoc = XDocument.Load(reader, LoadOptions.None);
 
                    // iterate over the 'Module' and 'File' elements in the XDocument, concatenating their Url attributes in a smart way so that we grab the site relative file Url-s
                    string[] fileUrls = manifestDoc.Root.Elements(WS + "Module")
                        .SelectMany(me => me.Elements(WS + "File"), (me, fe) => string.Join("/", new XAttribute[] { me.Attribute("Url"), fe.Attribute("Url") }.Select(attr => attr != null ? attr.Value : null).Where(val => !string.IsNullOrEmpty(val)).ToArray()))
                        .ToArray();
 
                    // iterate the file url-s
                    foreach (string fileUrl in fileUrls)
                    {
                        // get the file
                        SPFile file = web.GetFile(fileUrl);
                        // depending on the settings of the parent document library we may need to check in and/or (publish or approve) the file
                        if (file.Level == SPFileLevel.Checkout) file.CheckIn("", SPCheckinType.MajorCheckIn);
                        if (file.Level == SPFileLevel.Draft)
                        {
                            if (file.DocumentLibrary.EnableModeration) file.Approve("");
                            else file.Publish("");
                        }
                    }
                }
            }
            finally
            {
                // finally delete the manifest files from the site root folder
                foreach (SPFile manifestFile in manifestFiles) manifestFile.Delete();
            }
        }
    }

As you can see, the receiver’s code is quite small and straight-forward (check also the comments inside the code). In short what it does is as follows: it first iterates the files in the site root folder, matching their names against the above mentioned name pattern, then it loads the manifest files one by one in an XDocument object and extracts the URL-s of the files in every manifest file. After that the provisioned files are being checked in, published or approved if necessary (depending on their state and the settings of the containing document library). The final step is to delete the manifest file or files in the site root folder, because they are no longer needed, of course.

Feature stapling in sandbox solutions

2010-12-30T06:20:00.001-08:00

In a recent posting of mine about the WebProvisioned event receiver in SharePoint 2010, I made a comparison between the latter and the feature stapling functionality which has been available since SharePoint 2007. In that posting I had the wrong assumption that feature stapling only works at farm level, but after a prompting comment I had to correct this. After another check of the SharePoint SDK, I saw that feature stapling can actually be scoped at three levels: farm, web application and site collection – which provides not only extra flexibility but also better isolation when necessary. When I understood that one can create a site collection scoped “feature stapling” feature, I asked myself whether it’s possible to provision such a feature in a sandbox solution (that’s the highest possible feature scope for a sandbox solution). I created a small POC project in a couple of minutes, deployed it and it turned out that feature stapling does work in sandbox solutions too.

The setup in my POC project was very simple – it contained two features – one site collection scoped (Scope="Site") and one site scoped (Scope="Web"). The latter was a dummy feature, containing no feature elements – its purpose was to be “stapled” by the stapling feature. The stapling feature was the site collection scoped one (naturally) – it contained one feature element file with two “FeatureSiteTemplateAssociation” elements – both were targeting the standard “blank site” site definition (STS#1), the first one specifying the dummy site scoped feature in the same sandbox solution and the second one – the standard “WikiPageHomePage” feature (this is the standard feature that creates the wiki page library in the standard “team site” and sets its default page to be the “home” page in the wiki page library). I used two feature template associations, because I wanted to make sure that both sandbox and farm solution features get stapled by a “feature stapling” feature in a sandbox solution – which turned out to be exactly so.

And let’s see what the possible benefits from the feature stapling (site collection scoped) in a sandbox solution may be compared to the “normal” feature stapling in farm solutions – I can see these two positive/advantageous points:

the first one stems from a general feature of the sandbox solutions – sandbox solutions can be installed and activated by site collection administrators (no need for farm administrator’s rights) and you don’t “pollute” the 14 hive with extra features.
the second one is better isolation and manageability – if you have the feature stapling features in a farm solution with the “Hidden” attribute set to FALSE, so that site collection administrators can see them and manage them (activate/deactivate) in the standard site settings pages, the stapling features will appear in the settings pages of all site collections in the SharePoint farm. If you on the other hand decide to have the stapling features with the “Hidden” attribute set to TRUE, then the site collection administrators won’t be able to manage them using the standard SharePoint UI. And on the other hand with a stapling feature in a sandbox solution you can safely leave the feature to be visible, since this visibility will be limited only to the site collection in which you have the sandbox solution activated. And even if the stapling feature is hidden, the site collection administrator will still be able to deactivate the containing sandbox solution and thus deactivate the stapling feature itself.

And now let me make another comparison of the feature stapling (site collection scoped), this time with the WebProvisioned event receiver (which was introduced in SharePoint 2010 – check this posting of mine for more details about the WebProvisioned event receiver) – here are some of the differences between the two:

With feature stapling you basically activate extra features to sites based on standard and custom site definitions and you configure the whole thing with XML in a feature element file, while with the WebProvisioned event receiver you write custom code which gets executed after the new site is created. In the WebProvisioned event receiver you can also activate features but you have to do that with code using the SharePoint object model.
When you use feature stapling you can use feature properties for the features that you staple. If you want to activate features in the WebProvisioned event receiver with code, you cannot provide feature properties – there is no public method in the SharePoint object model that allows that.
The lowest possible scope for feature stapling is the site collection level – meaning that all sub-sites in the site collection with the targeted site definitions will be affected. For the WebProvisioned event receiver you have the site collection scope but also the site scope. With the latter you target only the immediate children of the site to which you attach the WebProvisioned event receiver (this site may be the root site of the site collection but may also be a sub-site).
The time of the activation of the stapled features and the time of the execution of the code of the WebProvisioned event receiver in the timeline of the site provisioning differs and this may seriously impact your solution depending on the specifics of your case. Here is the order of activation of the standard elements in the ONET.XML of a site definition/web template:

1. site collection scoped features from the onet.xml
2. site collection scoped stapled features
3. site scoped features from the onet.xml
4. site scoped stapled features
5. “List” elements from the onet.xml
6. “Module” elements from the onet.xml

In the case of site collection scoped “feature stapling” features we can obviously staple only site scoped (Scope="Web") features, but as you see the time of execution of the site scoped stapled features is before the activation of the “List” and “Module” elements, which means that when your stapled feature gets activated your site is not fully provisioned and you may miss some of the SharePoint lists and some of the files/pages in the site depending on the site definition that is used for the site. On the other hand the custom code of the WebProvisioned event receiver gets executed after the target site is fully provisioned and you will have all artifacts in the site already created which you can further modify/adjust with the code of the receiver.

SharePoint 2007 – associated lookup columns

2010-11-30T11:25:00.001-08:00

No, this is not a mistake – we can also have associated lookup columns in SharePoint 2007 [update (Dec 13, 2010): check for two drawbacks in SharePoint 2007 below], though not with the extended UI creation capabilities available in SharePoint 2010. Let’s first have a look at this new functionality in SharePoint 2010 (and define more accurately the terminology – I don’t think that there is a specific term for this lookup field extension, but in this article I will call it as I did in the title of the posting):

So, as you can see from the screenshot (and probably tried that many times already yourself), when you create a new lookup column in a list, you have the option to select more than one column from the target lookup list (in SharePoint 2007 we had the option to select just one “show column”). Basically what SharePoint does when you create a lookup field from the UI is to create the “normal” lookup column as before and then create another read-only “associated” lookup column for every additional show field from the check-box list that you have selected. And this functions as follows – when you create a new item, or edit an existing one in the edit form you see only the “normal” lookup column there, but when you change it to point to another item in the lookup list, all “associated” lookup columns change their values so that they correspond to the related columns in the newly selected lookup item. The associated lookup columns are read-only but you can add them to the views of the list, so that you display all additional columns that you need from the lookup item. And since the main lookup and the auxiliary lookup fields are automatically synchronized (SharePoint does that for us), you don’t need any additional code in item event receivers for instance.

Let’s now have a look at the field schema of the “normal” and “associated” lookup columns:

<Field Type="Lookup" DisplayName="Department" Required="FALSE" EnforceUniqueValues="FALSE" List="{ceaf935e-b9c6-48a0-8c23-bcec58a24c91}" ShowField="Title" UnlimitedLengthInDocumentLibrary="FALSE" RelationshipDeleteBehavior="None" ID="{98948dfd-cea5-4d6c-ac47-25bafa5218de}" SourceID="{bca1cd44-3822-49b6-b68c-2ff28ced1726}" StaticName="Department" Name="Department" ColName="int1" RowOrdinal="0" Group="" />

This is the schema of the “normal” or main lookup column – as you see, there is nothing specific in its schema – it is just a regular lookup column. And this is the schema of one of the associated lookup columns that I created for the lookup column above:

<Field Type="Lookup" DisplayName="Department:Code" List="{ceaf935e-b9c6-48a0-8c23-bcec58a24c91}" WebId="4ee36ddf-5b1b-470b-9f9a-fbd970edf5aa" ShowField="Code" FieldRef="98948dfd-cea5-4d6c-ac47-25bafa5218de" ReadOnly="TRUE" UnlimitedLengthInDocumentLibrary="FALSE" ID="{cd2b77b9-6238-4e2e-99ee-826826dc09f2}" SourceID="{bca1cd44-3822-49b6-b68c-2ff28ced1726}" StaticName="Department_x003a_Code" Name="Department_x003a_Code" Version="1" />

So, as you see we have all attributes for a normal lookup column, but one extra attribute as well – “FieldRef”. The value of this attribute is a Guid and this is exactly the ID attribute of the main lookup column. So, obviously the “FieldRef” attribute is the one that defines this association between the main and the associated lookup columns. Notice also the “ReadOnly” attribite in the associated lookup field’s schema which is set to TRUE – this guarantees that the associated lookup column doesn’t appear in the new and edit forms of the SharePoint list (you don’t need it there anyway, since it is synchronized automatically).

This is indeed a very useful new functionality in SharePoint 2010, but you may already ask yourself what SharePoint 2007 has to do in the whole matter – we don’t have this functionality there. Well, it is true that SharePoint 2007 lacks the user interface for creating “associated” lookup columns but this doesn’t mean that we cannot create columns with code specifying the values for all attributes in their schema as we decide. And it turns out that when we create a lookup column in a list and set its “FieldRef” attribute to “point” to another lookup column in the same list (both columns should target one and the same lookup list) – the “associated” lookup functionality actually works – just like in SharePoint 2010 (note here – I tested this on SharePoint 2007 service pack 2). Another detail here is that you shouldn’t forget to set the “ReadOnly” attribute of the field to TRUE – otherwise the “associated” lookup column will appear in the edit form of the list and when you try to save the list item, the page will crush with a nasty error.

As to the question of how you can create a lookup column (an associated lookup column) with code (and set its FieldRef attribute) – probably the easiest way is to use the SPFieldCollection.AddFieldAsXml method, which accepts a string parameter containing the schema XML of the new field. Note that you will have to properly format the schema XML of the associated lookup field providing the correct values to its attributes (the final XML schema should look similar to the one above):

in the ID attribute – a unique Guid value
in the Name and StaticName attributes – unique (within the SharePoint list) internal name of the field
in the List attribute – the ID of the lookup list
in the WebId attribute – the ID of the web containing the lookup list (in most cases this is the web of the current list)
in the FieldRef attribute – the ID of the main lookup field to which we want to associate the new lookup column
in the ShowField attribute – the internal name of the column in the lookup list whose value should be displayed in the new lookup column
in the ReadOnly attribute – this should be set to TRUE

And the net result will look something similar to this:

In the sample screenshot, the “Department” column is the main lookup column (displaying the “Title” column from the lookup list) and the “Department country” column is the associated lookup column (displaying the custom “Country” text column from the lookup list). Note also that the value displayed in the list view in the column of the associated lookup field is not rendered as a link, unlike the value in the column of the main lookup – actually this is exactly the same behavior as we have in SharePoint 2010. And most importantly again the same as in SharePoint 2010, the values of the associated lookup column or columns get automatically synchronized with the changes of the selected lookup item in the main lookup field.

[update Dec 13, 2010]

Shortly after the publishing of this article there was a comment (by Szymon) that once you create an associated lookup field you cannot delete it – something that I have overlooked. The error that you receive is “One or more field types are not installed properly. Go to the list settings page to delete these fields.” The reason for that is some kind of restriction in SharePoint 2007 (in SharePoint 2010 it works perfectly well) and I have seen the same error when I tried to delete “computed” (not to be confused with “calculated”) fields. After some research the one common thing between the associated lookup and the computed fields is that they both lack the “ColName” attribute, which means that there is no associated database column in the SharePoint “all user data” SQL table. Which also means that these types of columns don’t take space in the content database as one can expect. The bottom line is that you have a field that doesn’t take space in the SharePoint content database, but you still can’t get rid of its schema. As a work around you can set also its “Hidden” attribute so that the field doesn’t appear in the available view fields of the list in the standard SharePoint UI. Still, it is important that you know that once created you cannot delete (at least not using standard SharePoint object model) the associated lookup fields.

The other drawback that you have with associated lookup columns in SharePoint 2007 is that if you use them in list views or as view fields in SPQuery objects you need to always specify the main lookup column as well. If you miss it, you will receive the ugly SharePoint error “Cannot complete this action”.

XsltListViewWebPart – how to display columns based on permissions

2010-11-24T06:53:00.001-08:00

The standard SharePoint security trimming in the XsltListViewWebPart works on the list item level – the items in the displayed SharePoint list may inherit the permissions from their parent list, but may also have different permissions and based on these the viewing user depending on his rights may see different result sets. In most cases this “horizontal” security trimming will be sufficient for the end users, but there may be cases when a “vertical” type of security trimming is also requested – meaning that certain columns in the list view are displayed only to users with particular rights for the target SharePoint list.

So, I created a small “proof-of-concept” type solution that demonstrates how this can be achieved. The implementation is fairly simple and involves only changing of the XSL used by the web part and modifying its “PropertyBindings” property. The modifications of the rendering XSL are obviously the core part of the implementation but you may wonder what the purpose of using the “PropertyBindings” property of the XsltListViewWebPart is (if you are not familiar with the “PropertyBindings” property and how to use it, you can check out my extensive posting on the subject here). The idea is simple – in the “PropertyBindings” property you can keep configuration information that also becomes available in the XSL of the web part. And the configuration data that is needed in our case is the permissions’ data for the list columns that the web part displays – it is about what rights exactly the user should have for the target list, so that he can see one or another column in the list view. The question here is why put this configuration in the web part’s property bindings and not in the fields’ schema itself for example (the field schema can easily be extended with custom attributes – e.g. xmlns:PermissionMask="0x7FFFFFFFFFFFFFFF"). The main reason for this is that if you use custom attributes in the field schema XML, you cannot then use them in the rendering XSL of the XLV web part, custom attributes simply don’t appear in the view schema XML that is available through the “$XmlDefinition” XSL parameter (check this MSDN article for a sample view schema XML in the XLV’s XSL – the field schema data is in the View/ViewFields/FieldRef elements). Another point here is that even if it were possible to store the field permission data in the field’s schema, this would impact all list views that use the customized XSL (and display the particular column), and with the setting of the “PropertyBindings” property only the current XLV web part will be affected. It is hard to judge whether this is of advantage or disadvantage and probably depends on the specific case that you may have.

And let me first show you the “PropertyBinding” XML that should be added to the “PropertyBindings” property (I said “added”, because the property normally already contains the XML of other property bindings):

<ParameterBinding Name="ColumnPermissions" DefaultValue="Author|9223372036854775807|Editor|756052856929" />

The “Name” attribute specifies the name of the xsl:param element that will be initialized with the value of the property binding in the web part’s XSL. Its value is in the “DefaultValue” attribute – it is a long string containing values delimited by the ‘|’ character. At odd positions you have field names (internal names actually) and at even positions you see big numbers, which are actually the permission masks that should be applied for the list columns which precede the corresponding number. The permission mask is determined by the standard SharePoint SPBasePermissions enumeration: 9223372036854775807 (hex 7FFFFFFFFFFFFFFF) corresponds to the “Full Control” permission level and 756052856929 (hex B008431061) corresponds to the “Contribute” permission level. This means that the user will see the “Author” (“Created by”) column only if he has “Full Control” rights and the “Editor” (“Modified by”) column only if he has “Contribute” rights for the SharePoint list that is displayed. Note that all fields that are not specified in the property binding will be always visible to all users.

Let’s now move to the custom XSL that should handle the rendering and display only the columns that the current user has rights to see. Before, I show you the complete source of the custom rendering XSL, I want to draw your attention to one important thing – the trick that is used in the XSL to check whether the permission masks for the fields have a match with the effective permissions of the current user for the source SharePoint list. It is very simple actually and uses … a standard SharePoint “ddwrt” XSLT extension method:

<xsl:if test="ddwrt:IfHasRights($checkResult)">

The “IfHasRights” extension method receives an integer parameter for the permission mask and returns true or false depending on whether the current user has those rights for the SharePoint list of the web part. Note that the check is made for the SharePoint list, not the items of the list and not for its parent SharePoint site.

And here is the complete source of the custom XSL (check the extensive comments inside it for more details)

<xsl:stylesheet xmlns:x="http://www.w3.org/2001/XMLSchema" xmlns:d="http://schemas.microsoft.com/sharepoint/dsp" version="1.0" exclude-result-prefixes="xsl msxsl ddwrt" xmlns:ddwrt="http://schemas.microsoft.com/WebParts/v2/DataView/runtime" xmlns:asp="http://schemas.microsoft.com/ASPNET/20" xmlns:__designer="http://schemas.microsoft.com/WebParts/v2/DataView/designer" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:msxsl="urn:schemas-microsoft-com:xslt" xmlns:SharePoint="Microsoft.SharePoint.WebControls" xmlns:ddwrt2="urn:frontpage:internal">
 
  <!-- import the standard main.xsl, so we have all standard stuff -->
  <xsl:import href="/_layouts/xsl/main.xsl"/>
  <xsl:output method="html" indent="no"/>
 
  <!-- we get here the field permissions configuration from the PropertyBinding with the same name -->
  <xsl:param name="ColumnPermissions" />
  <!-- this is the standard XmlDefinition parameter - the XLV initializes this one with the view schema -->
  <xsl:param name="XmlDefinition" />
 
  <!-- this variable contains the parsed configuration data like <token>Author</token><token>9223372036854775807</token> etc -->
  <xsl:variable name="tokens">
    <xsl:call-template name="Tokenize">
      <xsl:with-param name="string" select="$ColumnPermissions" />
      <xsl:with-param name="delimiter" select="'|'" />
    </xsl:call-template>
  </xsl:variable>
 
  <!-- here we create a copy of the original XmlDefinition removing all View/ViewFields/FieldRef elements for which the user doesn't have rights -->
  <xsl:variable name="XmlDefinition2Raw">
    <xsl:apply-templates mode="transform-schema" select="$XmlDefinition" >
      <xsl:with-param name="tokenSet" select="msxsl:node-set($tokens)" />
    </xsl:apply-templates>
  </xsl:variable>
 
  <!-- the one above is a sequence of tags, in order that it can be used exactly like the standard $XmlDefinition it should be converted to a node set -->
  <xsl:variable name="XmlDefinition2" select="msxsl:node-set($XmlDefinition2Raw)" />
 
  <!-- this one is simply a copy of the template with the same match from the standard vwstyles.xsl (thus we override it), the only difference is that it uses our trimmed $XmlDefinition2 instead of the standard $XmlDefinition -->
  <xsl:template match="/">
    <xsl:choose>
      <xsl:when test="$RenderCTXOnly='True'">
        <xsl:call-template name="CTXGeneration"/>
      </xsl:when>
      <xsl:when test="($ManualRefresh = 'True')">
        <xsl:call-template name="AjaxWrapper" />
      </xsl:when>
      <xsl:otherwise>
        <xsl:apply-templates mode="RootTemplate" select="$XmlDefinition2"/>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>
 
  <!-- the same as the template above -->
  <xsl:template name="AjaxWrapper" ddwrt:ghost="always">
    <table width="100%" border="0"  cellpadding="0" cellspacing="0">
      <tr>
        <td valign="top">
          <xsl:apply-templates mode="RootTemplate" select="$XmlDefinition2"/>
        </td>
        <td width="1%" class="ms-vb" valign="top">
          <xsl:variable name="onclick">
            javascript: <xsl:call-template name="GenFireServerEvent">
              <xsl:with-param name="param" select="'cancel'"/>
            </xsl:call-template>
          </xsl:variable>
          <xsl:variable name="alt">
            <xsl:value-of select="$Rows/@resource.wss.ManualRefreshText"/>
          </xsl:variable>
          <a href="javascript:" onclick="{$onclick};return false;">
            <img src="/_layouts/images/staticrefresh.gif" id="ManualRefresh" border="0" alt="{$alt}"/>
          </a>
        </td>
      </tr>
    </table>
  </xsl:template>
 
  <!-- this template creates the copy of the standard $XmlDefinition trimming the View/ViewFields/FieldRef elements for which the user doesn't have rights -->
  <xsl:template mode="transform-schema" match="View" >
    <xsl:param name="tokenSet" />
    <!-- copy the root View element -->
    <xsl:copy>
      <!-- copy the root View element's attributes -->
      <xsl:copy-of select="@*"/>
      <!-- copy the child elements of the root View element -->
      <xsl:for-each select="child::*">
        <xsl:choose>
          <xsl:when test="name() = 'ViewFields'">
            <!-- special handling of the ViewFields element -->
            <ViewFields>
              <!-- iterate the ViewFields/FieldRef elements here -->
              <xsl:for-each select="child::*">
 
                <!-- get the permission mask for the FieldRef element, by the Name attribute -->
                <xsl:variable name="checkResult">
                  <xsl:call-template name="GetValueFromKey">
                    <xsl:with-param name="tokenSet" select="$tokenSet" />
                    <xsl:with-param name="key" select="./@Name" />
                  </xsl:call-template>
                </xsl:variable>
 
                <xsl:choose>
                  <!-- if the permission mask is not empty and the ddwrt:IfHasRights returns true, copy the field -->
                  <xsl:when test="$checkResult != ''">
                    <!-- this is how we check whether the user has sufficient rights for the field (checking the permission mask of the field against the user's permissions for the source list) -->
                    <xsl:if test="ddwrt:IfHasRights($checkResult)">
                      <xsl:copy-of select="."/>
                    </xsl:if>
                  </xsl:when>
                  <xsl:otherwise>
                    <!-- if we don't have the field in the configuration simply copy the FieldRef element -->
                    <xsl:copy-of select="."/>
                  </xsl:otherwise>
                </xsl:choose>
 
              </xsl:for-each>
            </ViewFields>
          </xsl:when>
          <xsl:otherwise>
            <xsl:copy-of select="."/>
          </xsl:otherwise>
        </xsl:choose>
      </xsl:for-each>
    </xsl:copy>
  </xsl:template>
 
  <!-- several helper templates that parse the configuration string and return the permission mask for the field by providing the field's internal name -->
  <xsl:template name="GetValueFromKey">
    <xsl:param name="tokenSet" />
    <xsl:param name="key" />
    <xsl:apply-templates select="$tokenSet/token[text() = $key]" />
  </xsl:template>
 
  <xsl:template name="NextNode" match="token">
    <xsl:value-of select="following-sibling::*"/>
  </xsl:template>
 
  <xsl:template name="Tokenize">
    <xsl:param name="string" />
    <xsl:param name="delimiter" select="' '" />
    <xsl:choose>
      <xsl:when test="$delimiter and contains($string, $delimiter)">
        <token>
          <xsl:value-of select="substring-before($string, $delimiter)" />
        </token>
        <xsl:call-template name="Tokenize">
          <xsl:with-param name="string" select="substring-after($string, $delimiter)" />
          <xsl:with-param name="delimiter" select="$delimiter" />
        </xsl:call-template>
      </xsl:when>
      <xsl:otherwise>
        <token>
          <xsl:value-of select="$string" />
        </token>
        <xsl:text> </xsl:text>
      </xsl:otherwise>
    </xsl:choose>
  </xsl:template>
 
</xsl:stylesheet>

One note about the XSL code – if you have a look at it, you will notice that it replaces (overrides) two of the core XSL templates used in the standard vwstyles.xsl file. It then provides a modified copy of the standard view schema XSL parameter ($XmlDefinition) in these templates. And on the other hand if you check the vwstyles.xls file, you will notice that there are still other XSL templates in it that also use the standard $XmlDefinition parameter (and thus not the modified copy) – these are the templates that handle aggregations and groupings, which means that the customized XSL above won’t be able to handle properly these cases.

And finally a few words on how to use this sample: the first thing is to save the XSL to a XSL file in the TEMPLATE\LAYOUTS or TEMPLATE\LAYOUTS\XSL folder (a subfolder of these two is also possible). Then you need to select your XLV web part (it may be an XLV from a standard list view page or an XLV that you placed on a content page) and change its PropertyBindings and XslLink properties. You can do that with code or using my web part manager utility which provides an easy to use UI for that (you can download it from here). For the “PropertyBindings” property, you should append the XML of the field permissions configuration which should look like the sample above. In the “XslLink” property you should specify the link to the XSL file in which you have saved the custom XSLT above – provided you’ve saved the XSL file as TEMPLATE\LAYOUTS\columns_main.xsl, the value that you should put in the “XslLink” property should be: /_layouts/columns_main.xsl.