How to use the lookup field across different site collections

Short Introduction

OK, so we know that the standard lookup field can be configured to have its lookup list in the same site (the web UI allows only this case) or in a site under the same site collection - this can be set using the object model. The question is - is it possible to set a lookup list which is in a site in a different site collection. The answer is - yes, I just created a small POC which demonstrates how this works - still there're some limitations - the site collection containing the lookup list should be in the same content database in which the list with the lookup field is.
First a few words of how I came up with this idea. If you have a look at the XML definition of a lookup field (checking the value of the SchemaXml property) you'll see the attributes that contain the data for the lookup relation - WebId - containing the ID of the site containing the lookup list, List - containing the ID of the lookup list and ShowField - containing the internal name of the lookup field. These three can be modified using the properties of the SPFieldLookup class - LookupWebId, LookupList and LookupField. Still, with some restrictions: the LookupWebId and LookupList can be changed only if they weren't previously set (I assume the reason for that is to protect against loss of data - imagine that someone changes the lookup list of an item in a list with hundreds of items) and the LookupWebId property setter verifies if the web ID provided is of a SPWeb under the lookup field's site collection. But still there is a work-around and the WebId and List attributes can be modified even if they were previously set - the SPField.SchemaXml property setter can be used to modify the field definition XML including any attribute in it.
And this is basically what I did - I used a small WinForm tool to modify the SchemaXml of an existing lookup field setting the WebId attribute with the ID of a site in another site collection under the same web application and the List attribute with the ID of a list that I had on that site. At first the result wasn't very promising - when I navigated to the edit form page to edit an item I just noticed that the lookup field control doesn't work at all. Then I wrote several lines of code using the object model to create a new item in the list and passed an integer value (a valid id of an item in the lookup list in the other site collection) to the recently modified lookup field. After that I browsed the all items view page only to notice that the lookup field works just fine - it showed the related item from the list in the other site collection. So, the conclusion is that internally for the lookup field it doesn't matter whether the lookup site is in the same site collection or not. It is only the field control and of course the field editor control (the one appearing in the add/edit field to list/web) that are not designed to support this scenario. And yet another problem - the display render pattern of the lookup field - it normally displays a link to the lookup item with the value of the lookup field - as I said the lookup value appears correctly, but the link to the display form of the lookup item is not the right one.
So the next question was can these problems be solved - to answer it I created a small POC project with a custom field type inheriting the standard lookup field with a custom field control and a custom field editor control - my idea was to replace the plugable items which were not working in my case with the standard lookup field - the rest I left to the lookup field which just happened to support internally cross site collection relations.
Several words about the POC project: first you can download it from here. You should keep in mind that this is a sample project with no production quality code inside, so if you want to use it in a real situation you should modify and improve it. Then there is the implication of the officially not supported by Microsoft features - bear in mind that the lookup field used as it is in the POC may cause various problems in different scenarios. Also it lacks some basic features related to code quality and best practices as well as some of the functionality expected is not fully implemented - e.g. I've hardly used proper exception handling logic, in the field editor control no validators are used, though there should have been at least several, etc. The field control uses just a simple drop-down as opposed to the text field with auto-complete logic which appears in the standard lookup control when the lookup list has more than 20 items. I didn't implement a field control for the multiple values mode of the field and this option also cannot be set from the field editor control. The latter is pretty simple, providing almost the bare minimum as user interaction and also misses some of the options that normally can be set for lookup fields. And finally I haven't modified the CAML for the field render display pattern (it inherits the one of the standard lookup), so that the broken link to the lookup list item is either fixed or at least removed (removing it is quite easy though).
OK, so I hope that I haven't scared you away with the big list of not finished stuff above - as I told from the very beginning - this was a sample project and a simple proof of concept.

The Sample Project

The sample project contains the following classes and files:

1. FieldCrossSiteCollectionLookup - the field class of the custom lookup, inherits the standard lookup field class - SPFieldLookup - adds basically only one new property LookupSiteCollectionID, which maps to a custom attribute in the field schema definition (xmlns:LookupSiteCollectionId) - this is used in the field control and the field editor user control to create the SPSite instance of the site collection containing the lookup list. I was tempted to use code like this one to get the url of a SPWeb by its ID:
   
   

   SPRequestInternalClass req = new SPRequestInternalClass();
   
   IntPtr p = IntPtr.Zero;
   req.InitHeap(ref p);
   string webAppUrl = "http://stefan.nl/";
   
   Guid g = new Guid("4bbbd404-2744-4113-ba70-adb8d4853ce4");
   string url = req.GetWebUrl(g, webAppUrl);
   req.ReleaseResources();
   

   
   This way I wouldn't have needed the extra attrubute in the custom lookup schema, since the WebId one would have been enough to first get the url of the web and with it to create a SPSite instance, and then to open the SPWeb itself. But since the stuff in the Microsoft.SharePoint.Library (an interop assembly for the COM library owssvr.dll) is not documented I don't think it would be wise to use it directly.
   One thing that would probably be good to be added as a property in the field class and as an attribute to the field schema respectively would be the server relative path to the item display form of the lookup list, so that it can be then used in the render display pattern CAML of the field type definition to fix the broken link to the lookup item.
2. CrossSiteCollectionLookup - the class of the field control of the custom lookup. A pretty simple control - creates a DropDownList control and binds it to a DataTable object containing the items of the lookup list. The DataTable is retrieved with SPList.GetItems(SPQuery).GetDataTable().
3. CrossSiteCollectionLookupFieldEditor.ascx - user control - the editor field control of the custom lookup, uses a small code behind class - CommonFieldEditor, which implements the Microsoft.SharePoint.WebControls.IFieldEditor interface and also defines several virtual methods so that control can be transfered to their overrides in the user control itself. I used inline code directly in the ascx for easy and quick testing and development, no issreset-s needed. There was a little hitch there - the SPField object that the IFieldEditor.OnSaveChange method provides to the control is in some uninitialized state and the SchemaXml property setter just crushed (it does quite a few things extra than just set the field schema), so I went directly to the forbidden fruit here - called on an internal method of the SPField object with reflection to get the schema XML changed - check the comments in the ascx file itself also.
4. fldtypes_stef.xml - the file containing the custom field type definition (should be copied to 12\template\xml) - just specifies that the field type inherits the standard lookup type and that FieldCrossSiteCollectionLookup is the field type class and CrossSiteCollectionLookupFieldEditor.ascx - the field editor control.