How to use the lookup field across different site collections

Short Introduction

OK, so we know that the standard lookup field can be configured to have its lookup list in the same site (the web UI allows only this case) or in a site under the same site collection - this can be set using the object model. The question is - is it possible to set a lookup list which is in a site in a different site collection. The answer is - yes, I just created a small POC which demonstrates how this works - still there're some limitations - the site collection containing the lookup list should be in the same content database in which the list with the lookup field is.
First a few words of how I came up with this idea. If you have a look at the XML definition of a lookup field (checking the value of the SchemaXml property) you'll see the attributes that contain the data for the lookup relation - WebId - containing the ID of the site containing the lookup list, List - containing the ID of the lookup list and ShowField - containing the internal name of the lookup field. These three can be modified using the properties of the SPFieldLookup class - LookupWebId, LookupList and LookupField. Still, with some restrictions: the LookupWebId and LookupList can be changed only if they weren't previously set (I assume the reason for that is to protect against loss of data - imagine that someone changes the lookup list of an item in a list with hundreds of items) and the LookupWebId property setter verifies if the web ID provided is of a SPWeb under the lookup field's site collection. But still there is a work-around and the WebId and List attributes can be modified even if they were previously set - the SPField.SchemaXml property setter can be used to modify the field definition XML including any attribute in it.
And this is basically what I did - I used a small WinForm tool to modify the SchemaXml of an existing lookup field setting the WebId attribute with the ID of a site in another site collection under the same web application and the List attribute with the ID of a list that I had on that site. At first the result wasn't very promising - when I navigated to the edit form page to edit an item I just noticed that the lookup field control doesn't work at all. Then I wrote several lines of code using the object model to create a new item in the list and passed an integer value (a valid id of an item in the lookup list in the other site collection) to the recently modified lookup field. After that I browsed the all items view page only to notice that the lookup field works just fine - it showed the related item from the list in the other site collection. So, the conclusion is that internally for the lookup field it doesn't matter whether the lookup site is in the same site collection or not. It is only the field control and of course the field editor control (the one appearing in the add/edit field to list/web) that are not designed to support this scenario. And yet another problem - the display render pattern of the lookup field - it normally displays a link to the lookup item with the value of the lookup field - as I said the lookup value appears correctly, but the link to the display form of the lookup item is not the right one.
So the next question was can these problems be solved - to answer it I created a small POC project with a custom field type inheriting the standard lookup field with a custom field control and a custom field editor control - my idea was to replace the plugable items which were not working in my case with the standard lookup field - the rest I left to the lookup field which just happened to support internally cross site collection relations.
Several words about the POC project: first you can download it from here. You should keep in mind that this is a sample project with no production quality code inside, so if you want to use it in a real situation you should modify and improve it. Then there is the implication of the officially not supported by Microsoft features - bear in mind that the lookup field used as it is in the POC may cause various problems in different scenarios. Also it lacks some basic features related to code quality and best practices as well as some of the functionality expected is not fully implemented - e.g. I've hardly used proper exception handling logic, in the field editor control no validators are used, though there should have been at least several, etc. The field control uses just a simple drop-down as opposed to the text field with auto-complete logic which appears in the standard lookup control when the lookup list has more than 20 items. I didn't implement a field control for the multiple values mode of the field and this option also cannot be set from the field editor control. The latter is pretty simple, providing almost the bare minimum as user interaction and also misses some of the options that normally can be set for lookup fields. And finally I haven't modified the CAML for the field render display pattern (it inherits the one of the standard lookup), so that the broken link to the lookup list item is either fixed or at least removed (removing it is quite easy though).
OK, so I hope that I haven't scared you away with the big list of not finished stuff above - as I told from the very beginning - this was a sample project and a simple proof of concept.

The Sample Project

The sample project contains the following classes and files:

1. FieldCrossSiteCollectionLookup - the field class of the custom lookup, inherits the standard lookup field class - SPFieldLookup - adds basically only one new property LookupSiteCollectionID, which maps to a custom attribute in the field schema definition (xmlns:LookupSiteCollectionId) - this is used in the field control and the field editor user control to create the SPSite instance of the site collection containing the lookup list. I was tempted to use code like this one to get the url of a SPWeb by its ID:
   
   

   SPRequestInternalClass req = new SPRequestInternalClass();
   
   IntPtr p = IntPtr.Zero;
   req.InitHeap(ref p);
   string webAppUrl = "http://stefan.nl/";
   
   Guid g = new Guid("4bbbd404-2744-4113-ba70-adb8d4853ce4");
   string url = req.GetWebUrl(g, webAppUrl);
   req.ReleaseResources();
   

   
   This way I wouldn't have needed the extra attrubute in the custom lookup schema, since the WebId one would have been enough to first get the url of the web and with it to create a SPSite instance, and then to open the SPWeb itself. But since the stuff in the Microsoft.SharePoint.Library (an interop assembly for the COM library owssvr.dll) is not documented I don't think it would be wise to use it directly.
   One thing that would probably be good to be added as a property in the field class and as an attribute to the field schema respectively would be the server relative path to the item display form of the lookup list, so that it can be then used in the render display pattern CAML of the field type definition to fix the broken link to the lookup item.
2. CrossSiteCollectionLookup - the class of the field control of the custom lookup. A pretty simple control - creates a DropDownList control and binds it to a DataTable object containing the items of the lookup list. The DataTable is retrieved with SPList.GetItems(SPQuery).GetDataTable().
3. CrossSiteCollectionLookupFieldEditor.ascx - user control - the editor field control of the custom lookup, uses a small code behind class - CommonFieldEditor, which implements the Microsoft.SharePoint.WebControls.IFieldEditor interface and also defines several virtual methods so that control can be transfered to their overrides in the user control itself. I used inline code directly in the ascx for easy and quick testing and development, no issreset-s needed. There was a little hitch there - the SPField object that the IFieldEditor.OnSaveChange method provides to the control is in some uninitialized state and the SchemaXml property setter just crushed (it does quite a few things extra than just set the field schema), so I went directly to the forbidden fruit here - called on an internal method of the SPField object with reflection to get the schema XML changed - check the comments in the ascx file itself also.
4. fldtypes_stef.xml - the file containing the custom field type definition (should be copied to 12\template\xml) - just specifies that the field type inherits the standard lookup type and that FieldCrossSiteCollectionLookup is the field type class and CrossSiteCollectionLookupFieldEditor.ascx - the field editor control.
   

Several tips for MOSS developing & testing with less iisreset-s

THE PROBLEM

OK, so the problem in hand are the iisreset-s (or application pool's recycling if we choose the economical approach), probably one of the most frustrating things one encounters while developing stuff with SharePoint. So I complained about that with SharePoint 2003, when it used to take several seconds and I am almost used to it now, when it can take up to half a minute and more on those slow virtual PC-s (a little less for WSS).
So basically we have two major cases when recycling of the IIS worker process is needed:

1. when we change (recompile) assemblies that are in the GAC
2. when we change SharePoint artifacts - site definitions, features, custom field types' definitions, control templates, etc.

Let's start with a few words about these two. The recycling of the IIS worker process to reload an assembly in the GAC is actually an issue that stems from the design of asp.net itself. The case is this: assemblies in the "bin" subfolder of the web application root get reloaded with recycling of the application domain of the web application (remember each asp.net application runs in a separate application domain). Basically an assembly can be loaded in an application domain at any moment, but can be unloaded only when the application domain is terminated - and this is what happens when you have a web application project and recompile the assembly - asp.net just detects the change in the "bin" subfolder, copies the newly recompiled assembly to a temporary folder under the "asp.net temporary files" (the so called shadow copy) and recycles the application domain of the web application so that the updated assembly can be loaded. So far, so good - but there is a big difference with assemblies which are in the GAC (or more precisely with all strongly named assemblies) - these are loaded as "domain neutral" assemblies by asp.net - this means that they are loaded once in the process and are shared by all application domains in the process (there are separate versions of the static data in the assembly for every application domain though). And this is the core of the problem with assemblies in the GAC - the recycling of the application domain just doesn't help to get the new version of the assembly reloaded, the whole process should be restarted - and in the case of IIS this is the IIS worker process of the given application pool (in our case for a SharePoint web application).
OK, so after we know that the process recycling is unavoidable for assemblies in the GAC, we may be tempted to resolve the problem a bit more radically - i.e. by not using assemblies in the GAC at all. Unfortunately, many customizations in SharePoint should be placed just in assemblies that are in the GAC - e.g. feature receivers, list/list item receivers, classes for workflows, custom field types, etc. So it turns out that we are stuck with GAC assemblies. The other "radical" approach here is to just not use the web UI (IIS) when developing and testing these customizations - well, this is one of the first tips of course, which I think most of us utilize most of the time - it is not applicable in all cases of course, especially when we want to see how something appears or behaves on the site itself - but more on that later.

And about the SharePoint artifacts - well, it seems that it is them and their caching in the first load that causes (or at least contributes significantly for it) so much delay after iisrest. And the thing is that an application domain recycling doesn't help to refresh them (you can force it if you put something in the "bin" subfolder or change the web.config for instance) - so I guess that these definitions are cached not in the asp.net cache or in the HttpApplication instance or any other managed object but in the COM core of SharePoint somehow - owssvr.dll (which among other things is also an ISAPI extension).


THE SOLUTIONS

1. Use custom tools and the stsadm tool to debug and test if possible - the idea is simple - just stay away from the IIS worker processes and use lightweight console or WinForm tools or the nice standard stsadm tool. With console tools including stsadm you will normally set the executable as a start program for you class library assembly that you want to debug/test and with WinForm tools you can either do that or just start the tool first and then attach the Visual Studio's debugger to it. And guess what, there is a nice small bonus here - you have the "child spoiling" "edit and continue" feature here - so you can just break at the beginning of the method and then start adding code on the fly and possibly execute it many times over and over with just dragging the yellow arrow indicating the debugger current row up and down as you like. One reminder about WinForm tools - when you change something in the 12/template folder of SharePoint you should restart the tool (the same thing as with iisreset and IIS, but much much faster) so that the tool gets a fresh snapshot of the artifacts. This is not the case with console tools because they just end by themselves after completing the task they were run for.
   Here is a small example of how you can set the stsadm tool as a start program for testing and debugging a feature receiver:
   
   
   
   We assume here that you've copied the latest changes of the feature files to the 12\template folder and installed the feature before that (use the -force parameter - it just saves a lot of time).
   If you want to test the feature when it is activated on site creation (some things are just a little bit different when the site is being created) you can replace the -activatefeature command above with the -createsite one and debug the receiver while the stsadm tool is creating a new site. And basically using stsadm or some custom tool is best for testing site definitions - you just change the site definition in the template\SiteTemplates folder and run the stsadm tool or a batch file calling several different stsadm commands in a row. I can't really remember when was the last time I used the central administration site or the create web page to create a site collection or site.
   The same approach is quite suitable for testing list item event receivers too - you should use a custom tool here though, stsadm hasn't got a command for adding/updating list items (unless you've added a custom command to it for that matter). One small thing here with synchronous receivers (ItemAdding, ItemUpdating) - if you change only one or two fields in the SPListItem object before you call the Update method on it, only these fields will appear in the AfterProperties property that you'll get in the receiver's methods. This is somewhat different from the normal behaviour that you expect when you debug the receiver having attached to the IIS worker processes - then you have all relevant fields set (all fields in the item's content type) just because the edit form page just sets all these fields. So you'll need to update all fields in the list item (the generic solution) or just the few which you'll make use of in the receiver.
2. Use inline code in aspx/ascx files instead of in code-behind classes - this may not sound as the best of coding practices but can save really a lot of time. Actually I am seriously addicted to using it. And after the page or the control is implemented and tested the code can be moved to a code-behind file. This way you can create a web page (SharePoint application pages in the layouts folder are easiest to create), put the script runat="server" tag and add some code, then just browse the page and it works. Then some more changes, an F5 in the browser and that's it. You can also debug that code and even change the code while debugging, though not the same way as with the "edit and continue" - here you can change the code only when you are not on a break point or stepping over the code - so the opposite to the "edit and continue" thing.
   A little word about custom user controls - I used several of these for custom field types' editing controls - a little code behind class is also necessary in this case, since the control should implement a .NET interface as well. The next thing is to transfer control from the code behind to the ascx file - this is easy - just create several virtual methods in the code behind, which then get overridden in the ascx file and this is it.
   A little bit off-topic, but if you are interested and didn't know - how does it come that the page is recompiled every time and the application domain doesn't get recycled - the answer is simple - asp.net creates a different assembly each time the page is compiled. This also means that the class of the page is different after each change - that is - its fully qualified name - it's in different assemblies after all. You can easily check this if you define a simple class in the page code then serialize it and put it in the view state/session/http cache and then try to deserialize it after an update of the page. There is yet another small thing here - after a certain number of recompiles of this sort asp.net forces an application domain recycling - so after vigorous code changing and F5-s you'll notice that once in a while the page loads much more slowly - this is when the application domain gets recycled. The logic behind that is obviously to get all those assemblies loaded in the application domain just cleared away.
3. Tips for web parts - several suggestions here:
   - use user controls in web parts - this one is great for one other reason - you just neatly separate the presentation logic and put it in a user control where using the asp.net declarative syntax you can easily change the HTML code, add web controls, etc. Compare this with the code you would use if you need to create all your controls with code in the CreateChildControl method and|or hard-code the html in the Render method of the web part. And then remember that you can also use inline code in the ascx file and see the changes enacted after a simple refresh in the browser.
   - use XSL for rendering and place the XSLT code in a web part property - this is an alternative to the above one - it actually also nicely separates the html generation from the rest of the code - you just need to fetch some XML with the data that should be rendered (from a web service, after you convert a DataTable to XML, etc) and use an XSL transformation on it to generate the HTML output. If the XSL is stored in a web part property you can easily change it using the web UI after you open the properties pane of the web part on the page . Still this is a little bit awkward - the SharePoint UI provides you with one quite small multiline text field, which also eats all new lines in the text. Alternatively you can use a separate file to store the XSL and use a web part property just to keep its location - pretty much the same as it is in the standard content query web part (how to quickly change XSL files for the CQWP see below). One drawback here compared to the user control approach - it is somewhat more complicated to use web controls if you generate your HTML with XSLT, so the XSL approach is better suited for scenarios when you just display things and don't have much user interaction in the part.
   - place the web part assembly in the "bin" folder - this is not too much of a gain actually. You won't need to recycle the application pool after recompiling, but the delay of the next load is still considerable. Bear in mind that you should not sign the assembly in this case and you will probably need to change the trust level in your web.config - to a standard or custom level if you have prepared a custom security policy file to grant some extra permissions to your web part code. If you are used to having your web parts in GAC assemblies in which case they run with full trust, you'll see some quite different behaviour in the "bin" assembly case, especially if the trust level in your web.config is too low. Also remember that if you make calls to other assemblies of yours which reside in the GAC, then the latter should be decorated with the AllowPartiallyTrustedCallers attribute.
   
4. Tips for testing the XSLT of content query web parts - I suppose that most guys who work with the SharePoint designer should know this simple trick. The idea here is to modify the unghosted version of the ItemStyle.xsl file or the custom xsl file that you use in you CQWP-s (I normally avoid changing standard SharePoint files, so I normally set my CQWP-s to use a custom xsl file which is just a copy of the ItemStyle.xsl with some extra templates added). OK, so as I said, the trick doesn't work if the item styles xsl file is ghosted which is the case when you first create a publishing site. Then SharePoint gets the contents of the file from a file on the hard-drive (and not from the content database), in our specific case from here: C:\Program Files\Common Files\Microsoft Shared\web server extensions\12\TEMPLATE\FEATURES\PublishingResources\ItemStyle.xsl (or from the folder of the feature that provisions your custom xsl file if you use one). You can change the file on the hard drive but you'll see the changes on the site only after an iisreset. Fortunately changes to unghosted files are visible immediately. With the SharePoint designer it is quite easy to modify any file in a SharePoint site - and when you do so the file gets unghosted (if it was ghosted in the first place) with the new version stored in the content database (actually this is one of the reasons why I don't like the SharePoint designer - I don't like my aspx files to get unghosted). But what about those who don't use or don't like using the SharePoint designer - well, there is an equally simple way to edit unghosted files in SharePoint - you can use WebDav for that. Just create a network place in your windows explorer and you will be able to see and edit the folder structure and files of your SharePoint site, pretty much the same way as in the SharePoint designer.
   And a small but crucial detail just to get the whole thing working - this is how you should test the web part on the page after you make changes to the xsl file - so you need to have the page open in design view plus the property pane of the web part open - then after each change in the xsl you just hit the "apply" button (or an F5 + Retry on the prompt that appears on a page with previous post-back) and you see the changes immediately.
   
5. Tips for testing the display render pattern CAML of custom field types - my approach here is to use computed fields. The idea is simple - in a custom field type definition (in a fldtypes_nnn.xml file) you have an RenderPattern Name='DisplayPattern' element and in a computed field definition you have a DisplayPattern element - both containing the CAML for the visualization of the fields. And the fact is that the syntax in both is almost identical. Also with computed fields you can use a simple tool (a WinForm for example) to create them in a SharePoint list and modify their definitions' XML, while you keep refreshing a list view page to see the result - so without recycling the application pool you can make changes and test the CAML of the display pattern of computed fields, which then with minor modifications can be transfered to the render display pattern of a custom field type definition. You can create a custom computed field using the SPFieldCollection.AddFieldAsXml (you can "steal" a sample computed field definition from the standard SharePoint "fields" feature) and then modify its definition with setting the SPField.SchemaXml property. And finally - what needs to be changed when you copy the CAML to the custom field type definition - first you should remove the Name attribute form the Field and Column elements (in the computed field definition these are needed to specify a source column, in the case of the custom field type definition, you don't have source columns, but just the field itself), then change the FieldProperty element (if you use it) to Property and remove its Name attribute.
   
6. Tips for testing the CAML of custom list views - you can test this if you create an application page in the 12\TEMPLATE\layouts folder and use this code in it:
   

   
   <%@ Page language="C#" MasterPageFile="~/_layouts/application.master"  %>
   <%@ Import Namespace="Microsoft.SharePoint" %>
   <%@ Import Namespace="Microsoft.SharePoint.WebPartPages" %>
   <script runat="server">
   protected void Page_Load(Object sender, EventArgs e)
   {
    SPList list = SPContext.Current.Web.Lists["Announcements"];
    SPView view = list.Views["All items"];
    string htmlView = view.HtmlSchemaXml;
    // the code up to here is just an example of how you can take the CAML view schema of a
    // list view - in a real test page you will probably want to read the CAML from a file on
    // the disk so that you can edit it and check the results after the edit - the CAML in the
    // file you can get initially either with code as above, or better still - from a View
    // element in a list template schema.xml definition file
   
    ListViewWebPart part = new ListViewWebPart();
    part.ListName = list.ID.ToString("B").ToUpperInvariant();
    part.ListViewXml = htmlView;
   
    this.phHolder.Controls.Add(part);
   }
   </script>
   <asp:Content ContentPlaceHolderId="PlaceHolderMain" runat="server">
   <asp:PlaceHolder runat="server" ID="phHolder"></asp:PlaceHolder>
   </asp:Content>
   

   
   Check the comments in the code snippet first. Also note that this code will work only in an application page and not on a web part page - if there is a web part manager on the page, the ListView web part, used here as a static web control won't render its contents. Another thing which may be interesting to note is the properties that I used to initialize the ListView web part - you can see that I used only the ListName and the ListViewXml properties and not the ViewGuid one - well, this is the subtle trick in the whole sample here.
   
   

hi there

hello,
This is my first post, my name is Stefan, I'm 33, live in Amsterdam, The Netherlands and come originally from Bulgaria. I've worked with SharePoint and WSS for the last couple of years ... So that's for starting, check my home site or profile for more ...