SharePoint 2010 - upgrading site definitions and web templates - part 1

This has been sort of a recurring issue in SharePoint since the time of MOSS 2007 (or rather SharePoint 2003). Not that you can't really upgrade an existing site definition or the new feature based web templates. All new sites that you create based on the site definitions/web templates will reflect the latest upgrades, the real problem is with the sites created before the upgrade. The problem is that those existing sites will remain exactly the same as before the upgrade of the site definition or web template that were once used to create them. In SharePoint 2010 this issue is almost half solved with the introduction of the powerful and more than useful new functionality - the possibility and support for upgrading features. Actually in many cases this can be perfectly sufficient to solve the problem with the upgrade of site definitions and web templates for existing sites. The only thing you need to do is to pack all functionality of your site template in a small set of features and when you want to add some additional functionality you simply add it to one of the existing features in this group. Then you just upgrade the feature in question and you propagate this piece of functionality to the existing sites based on this site template.

This is no doubt a working approach that can be utilized in real life scenarios. Still, it has the flaw that it defeats one of the main purposes of the SharePoint features in relation to SharePoint provisioning which is namely that features are supposed for the greater part to be reusable blocks of functionality. Of course a good share of the features that you create and use will be tailor-made for a specific site definition and won't be intended for being reused in other site definitions but the idea that you always should use tailor-made features doesn't sound good at all. For instance if you create a small custom feature that adds some web analytics support to your sites and intend for it to be reused in many site definitions, the natural approach would be to just extend these site definitions and add the feature to their CAML schema, rather than copy-pasting the feature's code to one of the existing features in these site definitions.

And just to add a small but very important detail to the idea of the SharePoint features as being small reusable blocks of functionality - this is the concept of feature properties. The feature properties allow you to customize the functionality that the feature adds to the target sites, so the outcome of the activation of a specific feature may in many cases depend on the specific context provided by the feature properties.

I have already mentioned several times in a few previous postings of mine that I look at the site definitions and web templates in SharePoint as mere groupings or sets of features. The availability of feature properties extends this idea somewhat, because the site definitions and web templates not only activate the set of features from their schema in the newly created sites but can use one and the same common features in different ways customizing the functionality that the latter add. Actually the ability to activate features with properties is legally available only declaratively in the onet.xml schema files of the site definitions and web templates, you can't do that using code and the object model (unless you use some trickery with reflection). I will mention one other way to achieve that in a little while.

Here we come close to the subject of the pros and cons of using site definitions and web templates as a provisioning approach in SharePoint. Many SharePoint professionals would even recommend against using these in favor of activating the desired features to the target sites manually. In my opinion the site definitions and web templates add a well justified level of abstraction on top of the SharePoint features, the two levels can be seen as the features being the basic blocks of functionality and the site definitions/web templates as sets of features (basic blocks) adding a level of parametrization (by means of feature properties) that creates a complete set of functionality (the sites that you create based on the site definition/web template).

Let's now go back to the main subject of this posting that I mentioned briefly in the beginning. The problem is that we have our site definitions and web templates, which are nice groupings of SharePoint features but SharePoint itself gives us no support for upgrading these "groupings", when we want that for existing sites. Of course, we can automate a lot of stuff in SharePoint 2010 (and with some limitations in SharePoint 2007) using PowerShell but still the best thing you will get out of it is to start activating the new features of your site definition to the old sites that you created based on it before you modified the site definition itself. And I wanted to find a solution to that which was neater and more self-contained. First I defined the task that I wanted to solve - and in the way that I defined it, it didn't look very hard to accomplish - I wanted to only activate the new features of the site definition/web template automatically to the sites that were created with this site definition/web template, before these new features were added to it. And what about the case when we want to make changes to an existing feature in the site definition. Well, it is simple - this is so to say not a problem of the site definition/web template, because there is a dedicated built-in mechanism for that in SharePoint 2010 - the feature upgrade capability. So if you want to modify an existing feature you should do that using the built-in approach for that, and this is basically outside the scope of the site definition/web template upgrade issue (with the concept of the site definition as a grouping, we only care of what features we have in the group, that is which new features we add, I didn't even consider the case of removing features from the site definition).

So, with the clear idea of what I wanted to do, I split the problem in several smaller tasks - the first three were these (the others will be covered in the second part of this posting):

• first we have to have a way to identify the web template which was used to create the target site - unfortunately SharePoint doesn't provide an OOB way to check this. For site templates this is an easy and trivial job with the help of the SPWeb.WebTemplate and SPWeb.Configuration properties.
• secondly, a way to get the current/latest version of the onet.xml file of the site definition or web template. We will need this to get the set of features and compare it to the set of already activated features in the target site or sites
• thirdly, a way to activate feature with feature properties. Remember that using code with the SharePoint object model we can only activate features without properties, the methods that allow activating features with properties are internal and can be accessed only with reflection (and I don't normally like to resort to reflection).

The solving of these tasks was not very difficult to start with and you can download the solution project from here. It contains three main classes, each of which solves one of the tasks mentioned above - given in the same order: WTMarkFeatureReceiver, WebTemplateHelper, FeatureHelper.
Let me now give you some quick and brief details about the way I solved these tasks.

Starting with the first one - just to mention, that it is a bit of a pity that SharePoint doesn't help us much here and we need a custom solution. The good news is that it is fairly easy to achieve that - the main idea comes from this excellent posting about web templates in SharePoint 2010 by Vesa Juvonen. The idea is to save the name of the web template in a the SPWeb.Properties property bag - whose purpose is exactly that - to store any custom data associated with the given SPWeb. Vesa achieves this by using the new PropertyBag feature element in one of the features of his sample web template (check the posting to see exactly how). I wanted to have this approach more readily reusable, so I wrapped it up in a reusable feature, which uses feature properties (no wonder). The feature has a custom feature receiver which is the aforementioned WTMarkFeatureReceiver class - you can check the code in the solution project. What it does is to save several important pieces of information related to the web template in the SPWeb.Properties properties bag property - these are the Guid of the parent feature of the web template, the name of the web template and the version of the parent feature - these are saved with the following keys in the property bag: WebTemplateFeatureId, WebTemplateName and WebTemplateFeatureVersion - basically all you need to identify the web template used to provision the target site.

And here is how you use the "marking" feature in the onet.xml file of your web template:

<Feature ID="48681fc1-97e0-4c9c-8061-c90890aeb64b">
  <Properties xmlns="http://schemas.microsoft.com/sharepoint/">
    <Property Key="WebTemplateFeatureId" Value="$SharePoint.Feature.Id$" />
    <Property Key="WebTemplateName" Value="$SharePoint.ProjectItem.Name$" />
  Properties>
Feature>
 

Note that the feature ID is of a sample feature that I used on my development server. You need to create your own web scoped feature specifying the WTMarkFeatureReceiver class as the feature receiver class for it (the feature can be in both a Farm or a Sandbox solution, the feature receiver works in both cases). Except for the feature ID, which you will have replaced, you don't need to change any other bit in this snippet. It cleverly makes use of two Visual Studio 2010 replace tokens which allow you to easily pass the feature ID of the parent feature of the web template (in whose onet.xml you have placed the snippet) and also the name of the web template itself - which is actually the name of the feature element that contains the web template's onet.xml file. So, you can easily copy-paste this snippet to all your web templates without having to manually enter the parent features' Guid-s and the web templates' names. Pretty neat, isn't it.

As regards the third task (let me jump ahead a bit, I will come back later to the second task) - if you have followed my recent postings you may have noticed that I had a dedicated posting on this subject, which was titled "SharePoint 2010–activate features with feature properties". It achieves this by ... using the SharePoint deployment API (the classes in the Microsoft.SharePoint.Deployment namespace) - maybe something that you wouldn't normally associate with feature activation. Check the posting itself for more details. The bottom line is that this gives you a valid approach for activating features with properties without having to resort to reflection.

And finally ... to the second task, which is solved with the WebTemplateHelper class in the solution project. This class provides a public method to determine which is the parent web template of a give SharePoint site (or the parent site definition if a site definition and not a web template was used to create the target site). Note that this method works only for sites which were "marked" with the custom "marking" feature mentioned above. The class also contains the implementation for retrieving and parsing the onet.xml file of the parent site definition or web template (whichever was used for the specified SharePoint site). And it also contains a public method which actually "upgrades" the target site to the latest version of its parent site definition or web template (only activating the features that are present in the latest onet.xml and not still activated in the specified site):

public static void UpdateSiteToLatestOnet (SPWeb web, bool updateIfWebTemplateOnly, bool updateWebFeaturesOnly)

You can see above the declaration of the site definition/web template "upgrading" method for an existing site. You can see that it uses only three parameters - the first one is the target SPWeb instance, the second one - specifies whether to upgrade the site only in case it is a "marked" web template based site (the "non-marked" web template based sites will have their SPWeb.WebTemplate and SPWeb.Configuration properties set, so they will be indistinguishable from sites created with the web template's base site definition). The third parameter specifies whether you want to activate only the missing web scoped features or also the site scoped features when your site is the root site of its site collection.

The code of the class is very brief and easy to follow (at least I hope so), so you can check it for further details.

In the second part of this posting I will continue with explaining how this custom web template upgrading approach can be coupled with the built-in feature upgrade capabilities in SharePoint 2010. Stay tuned.

SharePoint WCM HTML clean-up

SharePoint is a great web content management system, it is fast, scalable, reliable, comes with lots of out-of-the-box components, web parts, etc. that often make the life of the content managers much easier. There are certain aspects of the WCM capabilities of SharePoint though that sometimes need a little more time or some hacking to get them to work properly, or the way you may want them to work. One such thing is the HTML code that appears on your SharePoint pages - there are several things in the HTML generated by the SharePoint WCM system which make it look not quite neat and tidy. The SharePoint UI is built on top of the asp.net Web Forms technology, so SharePoint actually inherits some of the HTML issues directly from its asp.net foundation. The problem here is that when you use asp.net and Web Forms you don't have full control over the HTML that is going to be generated in your page. With the advent of the asp.net MVC this was one of the arguments in favor of the latter, because with asp.net MVC the developer indeed has full control over the generated HTML code. Unfortunately SharePoint doesn't utilize the MVC framework, so many of us at one or another point have had to struggle with the extra HTML bits that get injected in the SharePoint aspx page. Some examples for such bits that come directly from asp.net are the many system hidden fields that appear in the "form" element, the infamous "ViewState" field among them which can grow very big in size, the inline JavaScript blocks with "form" submit helpers, etc. Several intrinsically SharePoint items that further inflate your HTML are for instance the inclusion of the two "core" files: "core.js" and "core.css" (quite big both of them), the many nested HTML "table" elements around your web parts which are rendered by the containing WebPartZone controls, especially in cases when you want your HTML to contain only nice looking "div" elements, etc. The dilemma here is that because SharePoint utilizes in-place page content editing and it is a single aspx file that handles both the editing process and the actual displaying of the page to the end user, the items (web controls in most cases) responsible for these extra (but necessary) HTML artifacts cannot be removed directly from the page. So, we need them for the page content editing, but on the other hand we need to somehow get rid of them, or hide them, or at least suppress the extra HTML that they generate so that the page in display mode shows only the bare minimum of HTML that needs to render the page contents. In the WCM context, I assume here that the SharePoint site is publicly accessible or at least allows anonymous access within some internal network, so the hiding of the extra artifacts is necessary only when the pages are being accessed anonymously. This is a pretty broad scenario and this particular setup is quite popular in the WCM function of SharePoint. The next question is how many of the "extra" HTML SharePoint artifacts may be unwanted in your scenario. If it is about simple content pages with SharePoint field controls only or standard content editor web parts you actually won't need any of the above mentioned bits in display mode with anonymous access. This is especially true when your HTML design is very different from the standard SharePoint page design.
So, after several years and several partial solutions I decided to wrap up the whole thing in a single solution. And it turned out that the solution was pretty easy and simple to develop, and luckily - very easy to use too. It is actually a single user control that you need to place in one place only in your master page. And that's all. The control has several public properties that can be used to configure it, so that it suppresses some of the SharePoint artifacts that it can handle but not others (I will explain these in detail shortly). I chose to create the control as a user control (and there is no code behind assembly, the code is placed inline in the ascx file directly) because this way you have the two deployment options - to either place it in the TEMPLATE/CONTROLTEMPLATES folder of the SharePoint hive, or to upload it to the Master Page Gallery of your site and reference it in your master page from there (I explained this technique in this recent posting of mine). Of course, the code can be easily transferred to a simple web control and put in an assembly of yours.
You can download the user control that I named appropriately "HideSPArtifacts.ascx" from here.
Provided that you have uploaded it to the Master Page Gallery of your site collection (/_catalogs/masterpage) you will need the following lines of code to place it in your master page file:
First you need the "Register" directive at the top of your master page

<%@ Register="" TagPrefix="MyControls" TagName="HideSPArtifacts" Src="~SiteCollection/_catalogs/masterpage/HideSPArtifacts.ascx" %>

The second bit is to place the control declaration in the page mark-up:

<MyControls:HideSPArtifacts runat="server" RemoveCoreJS="false" RemoveCoreCss="false" RemoveHeadCss="false" RemoveForm="false" AddBodyOnLoadDummy="false" EnablePageViewState="true" RemoveZoneHeaders="false"/>

Two very important notes here: 1) if you upload the user control to the Master Page Gallery of your site collection you will have to make additionally certain modifications to your web.config file (check the previous posting that I mentioned above). 2) You need to place the control's declaration (MyControls:HideSPArtifacts) immediately after the opening "html" element of your page and before the "head" HTML element.
One other thing that you should check in your master page is whether you have a "head" element and whether it has the runat="server" attribute (if you have used one of the SharePoint master pages as a base for your master page you will have these). If this condition is not satisfied the control won't be able to remove some of the SharePoint artifacts from the page.
So, after you have the user control in your master page and open a page from your site anonymously (if you view the page as authenticated user the control will do nothing) and you have the values of the control's properties as they are in the snippet above you will see ... no changes in the HTML code of the page. This is because all "Remove.." properties are set to "false". I will now give you a list with the properties of the control and will briefly explain what changes in the generated HTML you will see after setting each property:

• RemoveCoreJS - as the name suggests, if this property is set, the include script declaration for the SharePoint's "core.js" file is removed from the page
• RemoveCoreCss - when set, this property causes the core.css style sheet include to be removed from the page. Note that if you use alternate style sheets, these won't be rendered either. This is because the HideSPArtifacts control will block the rendering of the standard SharePoint CssLink control (if available).
  
• RemoveHeadCss - when you use certain web controls like the TreeView control, the AspMenu control and some other controls, the asp.net page generates an inline CSS block in its "head" HTML element. If you don't want this inline style sheet to appear in the page (check if this doesn't affect any of the controls that you use on the page), set this property to true.
• RemoveZoneHeaders - the WebPartZone controls that contain your web parts have the bad habit of creating several nested "table" HTML elements. The web parts' chrome (frame) which you most often set to "none", because you don't need it in WCM public sites also renders a "table" element. If you don't want any of these "table" elements and want to have only the HTML markup directly rendered by your web parts set this property to true. Note that even if you set the "ChromeType" property of the web part, no chrome will be actually rendered (in anonymous mode only).
• EnablePageViewState - the default value of this property is true and in this case the control will change nothing on the page. If you set this property to false it will simply set the EnableViewState property of the containing page to false (only when the page is viewed anonymously). The net effect will be that ... you will still have the "ViewState" hidden field in your page, but it will contain only thirty or so bytes of data.
• RemoveForm - when set this property removes the "form" element from your page. Actually it does something much more radical - it removes also all system hidden fields including the ViewState field and all inline JavaScript blocks that were included using the methods of Page.ClientScript - e.g. RegisterClientScriptBlock, RegisterStartupScript, etc. You will get rid of a ton of HTML and JavaScript in your page, which you wouldn't need if you don't have controls and logic that need to do POST submits of the page. If your pages (at least the pages using the master page with the HideSPArtifacts control) contain only SharePoint field controls and content editor web parts this will be a perfect choice. Note however that you will need to carefully check all your pages - some controls (like Button, LinkButton, etc) directly crash if there is no "form" rendered on the page. Other controls may stop function properly because they will miss some JavaScript code that won't get rendered. Bottom line - use cautiously.
• AddBodyOnLoadDummy - the default value of this property is true. It has a visible effect only when the "RemoveForm" property is set to true. It adds a small JavaScript block with several empty JavaScript functions. One of these functions is called "_spBodyOnLoadWrapper". This function appears in the "onload" attribute of the "body" element of the default SharePoint master page. Since the "RemoveForm" property removes all inline JavaScript associated with the page's "form" element, the real definition of this JavaScript function won't be available on the page, and you will see a JavaScript error in your browser when the page loads. This is the reason why this property causes the adding of this small JavaScript block with empty definitions of this and two other system JS functions.

And now, let me briefly explain how the trick with hiding a control without hiding its contents is possible. Actually the idea is to hide the control itself (or at least parts of it) but display its child controls. This technique is used in the implementation of the "RemoveForm", "RemoveHeadCss" and "RemoveZoneHeaders" properties. The following steps are executed:

• in its OnInit override the HideSPArtifacts hooks onto the parent page's InitComplete event
• in the InitComplete event handler an empty control (class Control) is created in the "Controls" collection of the parent control of the control which we want to hide. The new control is inserted in the "Controls" collection of the parent control right after the target control.
• The SetRenderMethodDelegate method of the new empty control is called - this method receives a single delegate parameter, which you use to provide a method to be called right after the control's "Render" method exits. The idea is to use the empty control as a place-holder to inject some HTML right after the control that we want to hide.
• In the "Render" method override of the HideSPArtifacts control the "Visible" property of the control that we want to hide is set to false. Since we have placed the HideSPArtifacts right after the beginning of the master page its Render method is guaranteed to be called first in the child controls' chain. This way the control whose Visible property is set to false will not get rendered.
  
• in the render method passed as the render delegate parameter of the SetRenderMethodDelegate method, the Controls collection of the target control is iterated and all child controls are rendered using the Control.RenderControl method. This way we have the target control itself not rendered but all its children get actually rendered within the empty control that was injected right after it. This is how the goal of hiding the control itself but not its child controls is achieved.

You can check the source code of the HideSPArtifacts control for the details of the actual implementation.