Integrating Google Cloud reCAPTCHA Enterprise in PHP

Google Cloud reCAPTCHA Enterprise is the successor (?) to reCAPTCHA v3. 

The code snippets Google supplies for integrating reCATPCHA Enterprise in PHP are frustratingly uncohesive and neither a working demo or complete code sample exists (as far as I can find at the time of writing). 

Moreover, the code snippets that are provided were incomplete for my implementation and the number of moving parts (Cloud projects, keys and credentials) is nothing short of overwhelming. 

Given the above, I struggled for some time just get everything working--which is where I'm up to now. So a disclaimer for those of you readers who are proficient PHP developers and who possess a deeper understanding of the Google Cloud Platform, please use this as a starting point only. There may be a better (more refined) approach. 

In terms of the official documentation and the code snippets I refer to above and will discuss in this post, start here: https://cloud.google.com/recaptcha-enterprise/docs/choose-key-type

I should also note I'm working in a basic web hosting environment with PHP installed. It's not a Google environment--think more budget web host ;)

Setup

Before you can get started, you'll need to login to the Google Cloud Platform, create a new project if you don't have a suitable already, and create a reCAPTCHA Enterprise key. Save the key somewhere and take note of your Cloud project ID while you're there as we'll need that later. You'll find your project ID on the relevant dashboard card. 

Incidentally, you can follow the link to reCAPTCHA Enterprise in the documentation to get to where you need to be in the Google Cloud Platform (or just navigate to Menu > Security > reCAPTCHA Enterprise). 

Once that's done, you may want to hop into Menu > APIs and services. If the reCAPTCHA Enterprise API isn't already enabled, find it in the library and enable it. While you're here, you'll also want to create credentials. This is where things are still a bit blurry for me but I created both an API key and a service account. The API key isn't used in my final code and it's possibly not necessary. I'm just used to API keys, I guess! 

The service account basically receives a name and ID and is then assigned the reCAPTCHA Agent role (locate reCAPTCAHA Enterprise in the list of products and services and then select the reCAPTCHA Agent role). 

Once the service account exists, I chose to create a key ("Manage keys"). Google will warn you about the risks of downloading service account keys and suggests using Workload Identity Federation--I don't yet know what this is so do you own homework here. I created a new key (ADD KEY > Create new key) as a .json file--you'll be prompted to download/save this file and I'd suggest storing a copy in a safe place as I have yet to find a way to download this same file again in future. 

Code

With that setup done, we now move into the code. The code runs clients side and server side and is comprised of your HTML page (including a form), a couple of scripts in the HEAD tag and some attributes on the form button. And of course the PHP code. 

Be sure to add your reCAPTCHA Enterprise key to both the script tag in the head and the data-sitekey attribute on your button. It's also required in the call to create_assessment() in the PHP code. 

I'll point out that, in my case, I want to trigger a score-based reCAPTCHA assessment when a user submits a form--no sooner and not unecessarily. I'm therefore using the relevant Google examples here, combined into a single, cohesive file. 

The HTML and script source is taken from this page: https://cloud.google.com/recaptcha-enterprise/docs/instrument-web-pages 

The PHP source is taken from this page: https://cloud.google.com/recaptcha-enterprise/docs/create-assessment

Note the PHP code is slightly modified to include the putenv call, which references the JSON key file we created earlier. You'll also want to drop in your Google Cloud Platform project ID in the call to create_assessment(). 

<html>
<head>
	<script src="https://www.google.com/recaptcha/enterprise.js?render=your_recaptcha_key"></script>
	<script>
	   function onSubmit(token) {
		 document.getElementById("demo-form").submit();
	   }
	</script>
</head>
<body>
<form id="demo-form" method="post">
<input type="text" />
<button class="g-recaptcha"
data-sitekey="your_recaptcha_key"
data-callback='onSubmit'
data-action='submit'>Submit</button>
</form>
</body>
</html>

<?php

require 'google-re/vendor/autoload.php';

use Google\Cloud\RecaptchaEnterprise\V1\RecaptchaEnterpriseServiceClient;
use Google\Cloud\RecaptchaEnterprise\V1\Event;
use Google\Cloud\RecaptchaEnterprise\V1\Assessment;
use Google\Cloud\RecaptchaEnterprise\V1\TokenProperties\InvalidReason;

/**
* Create an assessment to analyze the risk of a UI action.
* @param string $siteKey The key ID for the reCAPTCHA key (See https://cloud.google.com/recaptcha-enterprise/docs/create-key)
* @param string $token The user's response token for which you want to receive a reCAPTCHA score. (See https://cloud.google.com/recaptcha-enterprise/docs/create-assessment#retrieve_token)
* @param string $project Your Google Cloud project ID
*/
function create_assessment(
  string $siteKey,
  string $token,
  string $project
): void {
	
  // *** I added this line ***
  putenv('GOOGLE_APPLICATION_CREDENTIALS=your-service-account-key-file.json');
    
  // TODO: To avoid memory issues, move this client generation outside
  // of this example, and cache it (recommended) or call client.close()
  // before exiting this method.
  $client = new RecaptchaEnterpriseServiceClient();

  $projectName = $client->projectName($project);

  $event = (new Event())
	  ->setSiteKey($siteKey)
	  ->setToken($token);

  $assessment = (new Assessment())
	  ->setEvent($event);

  try {
	  $response = $client->createAssessment(
		  $projectName,
		  $assessment
	  );

	  // You can use the score only if the assessment is valid,
	  // In case of failures like re-submitting the same token, getValid() will return false
	  if ($response->getTokenProperties()->getValid() == false) {
		  printf('The CreateAssessment() call failed because the token was invalid for the following reason: ');
		  printf(InvalidReason::name($response->getTokenProperties()->getInvalidReason()));
	  } else {
		  printf('The score for the protection action is:');
		  printf($response->getRiskAnalysis()->getScore());

		  // Optional: You can use the following methods to get more data about the token
		  // Action name provided at token generation.
		  // printf($response->getTokenProperties()->getAction() . PHP_EOL);
		  // The timestamp corresponding to the generation of the token.
		  // printf($response->getTokenProperties()->getCreateTime()->getSeconds() . PHP_EOL);
		  // The hostname of the page on which the token was generated.
		  // printf($response->getTokenProperties()->getHostname() . PHP_EOL);
	  }
  } catch (exception $e) {
	  printf('CreateAssessment() call failed with the following error: ');
	  printf($e);
  }
}

   create_assessment(
      'YOUR_RECAPTCHA_SITE_KEY',
      $_POST['g-recaptcha-response'], // Safety first! Do you trust this code?
      'YOUR_GOOGLE_CLOUD_PROJECT_ID'
    );
?>

Drop all of the above in a file named whatever.php and upload it to your web server. Before any of this will run on your server, you'll need to attend to some dependencies. 

Upload your .json key file alongside your .php file (or put it somewhere else on the server and amend the putenv path in the PHP code). This file may warrant additional protections. 

With that sorted, you need to fetch all of the Google Cloud files the PHP above depends on. You'll likely want to do this using Composer (which needs to be installed on your desktop) and you can then run this command in a command window: 

composer require google/cloud-recaptcha-enterprise

I'll note I'm only fetching the reCAPTCHA Enterprise bits here--not the entire Google Cloud file set. You do you. 

I saved all of the Google files in a directory named google-re, which you'll see referenced in the first line our PHP. Adjust as required. 

A final note for the FileZilla users out there: when uploading the Google dependencies in particular, you may encounter a horrible runtime exception ("Fail to push limit") if you don't configure the FileZilla transfer type as Binary (Transfer > Transfer type > Binary). Refer to the answer to this question for more information: https://groups.google.com/g/protobuf/c/8_S93nJWxUE?pli=1

Run It

And now you should be able to request your page, submit the button. All being well, you'll encounter no exceptions and receive a nice meessage like "The score for the protection action is:0.89999997615814". Use that and other, related assessment information to act accordingly.  

I hope that helps someone!

My custom Visual Studio keyboard shortcuts

I do all of the builds in my position as Development Coordinator but with the Visual Studio project context menus out of control and nearly unusable with clutter from various plugins (including ReSharper, PowerCommands, and Productivity Power Tools), I've finally cracked.

Since we don't (yet) have a build server, a normal manual build workflow goes something like "do stuff, do more stuff, BUILD, do stuff, OPEN DEPLOYMENT FOLDER IN EXPLORER, copy files to drop folder, etc. Before installing ReSharper, the VS team finally had all context menu items displaying on a single screen, with no scrolling required; ReSharper has ruined all of that again and I now find myself forever scrolling the context menu from the top (Rebuild) to the bottom (Open Folder in Windows Explorer).

Because I'm forever installing Visual Studio in different environments/upgrading/etc, I'm loathe to define custom keyboard mappings (likely a hangover from my days of using the product before Import and Export Settings… was introduced). Nonetheless, these are the shortcuts I've finally created:

Project.OpenFolderinWindowsExplorer – Alt+ O

Build.RebuildSolution – Alt+ R (Why rebuild instead of build? Personal preference)

If you found this post helpful, please support my advertisers.

DateTime format safe for file system

Couldn't find anything similar when I searched so thought I'd throw this out there for use in file names in a Windows file system:

DateTime.Now.ToString ("yyyyMMddHHmmss")

Will output "201001312359"

Add a ".txt" or ".dat" if you're hard-core and you're all set.

If you found this post helpful, please support my advertisers.

How to determine if a SPPublishingPage is published

You can programmatically determine if the SPPublishingPage you're dealing with is in a published state by retrieving the SPFileLevel of the page's corresponding SPFile:

if (page.ListItem.File.Level == SPFileLevel.Published) …

You can also access this through the PublishingPage.ListItem.Level property.

The same SPFileLevel enumeration will also indicate if the page is in draft mode (checked in but not published) or checked out but in my experience this property will return SPFileLevel.Published or SPFileLevel.Draft more often than not.

To determine whether the page is checked out, use the SPFile.CheckOutStatus property in WSS 3.0 or the  SPFile.CheckOutType property in SP2010. Any value other than None means the page is checked out.

If you found this post helpful, please support my advertisers.

SPSite.AllWebs vs SPWeb.Webs

There's a subtle distinction between SPSite.AllWebs and SPWeb.Webs to be aware of—and yes, despite this being the SharePoint API we're talking about, it's more than just a naming oddity!

SPSite.AllWebs returns the immediate child webs of the site collection and all of their child webs (in other words, all webs recursively within the site collection); SPWeb.Webs has a much narrower scope and only returns the first level child webs immediately below the SPWeb object.

So what does this mean and how does it benefit you? If you're writing code that recursively walks the site hierarchy using SPWeb.Webs, you may be able to avoid the overhead by simply using SPSite.AllWebs. Recursion is fun and all but it adds complexity where it may not be warranted.

If you found this post helpful, please support my advertisers.

C# Object Initialisers and trailing commas

C# 3.0 (.NET 3.5) introduced some really handy new features and hidden gems—and I'm still coming across the latter today.

If you've been developing with C# 3.0 for a while, you'll know about object and collection initialisers: they allow you to initialise the members of an object or a collection at instantiation-time, rather than having to declare a variable and then assign values to the members of that variable. It's a convenience thing and makes life easier in a LINQ context (although the debugging experience in VS2008 sucks).

When looking back through a piece of code I written previously, I noticed I'd inadvertently left a trailing comma after the final assignment in the list. I thought this was unusual because the code compiled and the C# is normally anything but lazy when it comes to most language constructs.

StaffMember staffMember = new StaffMember()
{
    FirstName = "Michael",
    Surname = "Hanes",
};

Sure enough, the ECMA C# Language Specification actually notes the following in the section about Array Initializers: "[…] C# allows a trailing comma at the end of an array-initializer. This syntax provides flexibility in adding or deleting members from such a list […]" (page 363, section 19.7). Interestingly, object and collection initialisers simply inherit this behaviour and the ability to initialise both arrays and enums in this way is not C# 3.0-specific (the specification document actually cites the C language).

In my opinion this is a rational design decision that increases the useability of the C# language by developers. By contrast, I'm forever having to shift SQL select statement commas around when commenting out individual lines and it's a major inconvenience. Development is normally a very fluid, experimental time in the lifecycle of a code artefact and anything that simplifies that exercise and increases maintainability is a good thing.

Don't use XmlRoot with .webpart files

Just noticed using an [XmlRoot("MyRoot")] attribute on my webpart class in conjunction with a .webpart file (as opposed to a .dwp file) gives me this error when attempting to add the webpart to a page:

Incompatible Web Part markup detected. Use *.dwp Web Part XML instead of *.webpart Web Part XML.

Well I don't want to use *.dwp Web Part XML so I removed the XmlRoot attribute instead. I'm feeling wild, you see.

Beware the old doco: http://msdn.microsoft.com/en-us/library/dd584160(office.11).aspx 

Flatten XML files with the XML Flattener

Have you ever needed to “flatten” a pretty-printed XML file for pasting into a web service test page? XmlSpy and the like will happily display ugly XML in a nicely-indented format but the version I have doesn’t un-pretty-print (or flatten) XML for me.

Custom tool to the rescue! Announcing my very first codeplex project and the XML Flattener!

This first alpha release is a simple WinForms apps with zero error handling and very limited capabilities allowing you paste in an XML document, flatten it, and copy out the flattened string. The hard work is accomplished by—wait for it—leveraging the default formatting of the XmlDocument’s OuterXml property (which isn’t formatted, unlike other XML-related classes which do return formatted documents).

FYI I built this app years ago when I was doing a lot of web service integration work so it’s a fairly naive tool built primarily for testing web services. It’s original title was The XML Bastardiser but I’ve attempted to make the name a bit more relevant with this first public release ;-)

Enjoy!

Selecting a delimited list of rows with other values in SQL

There are a number of really detailed posts out there describing an easy way to concatenate the values from a specific column into a delimited list without resorting to a cursor. Using coalesce and a temporary variable, the activity is child’s play:

Assuming you have a table named MyTable with a column named FirstName:

DECLARE @List varchar(MAX)

SELECT 
        @List = COALESCE(@List + ', ', '') + CAST (FirstName as varchar (200))
FROM
        MyTable

select @List

the above will return you something like “Ben, James, Ed”.

Nice and easy, right? But how do you use this in a non-trivial situation where you need to include additional columns in the select list? You can’t do this in the above statement (as far as I’m aware); in my case today, I wanted to a) avoid declaring the temporary variable and b) join the results to another table by Id.

for xml path ('') will help in the first case:

select cast(FirstName as varchar (200)) + ', ' from MyTable
for xml path ('')

but you’ll still get stuck trying to join the results.

Thankfully Arnie Rowland has a really concise SQL wiki post describing how accomplish this feat in SQL Server 2000 and 2005/2008.

Assuming this table:

CREATE TABLE TeamInfo
   (  MemberID    int IDENTITY,
      TeamID      int, 
      FirstName   varchar(50)
   )

Arnie’s query looks like this:

SELECT
   t1.TeamID,
   MemberList = substring((SELECT ', ' + FirstName
                           FROM TeamInfo t2
                           WHERE t1.TeamID = t2.TeamID
                           ORDER BY 
                              TeamID,
                              FirstName
                           FOR XML PATH('')
                          ), 3, 1000 )FROM TeamInfo t1
GROUP BY TeamID

And returns this result:

TeamID     MemberList
1	Bob, Jim, Mary
2	Ellen, Ralph, Sue
3	Bill, Linda

Note the substring function drops the leading delimiter and limits the result to just under 1000 characters. Also TeamInfo doesn’t have to be the same table, it can be any table with same IDs used in the t2 query.

How to return an anonymous type from a method

Current .NET lore would have you believe you can’t return an anonymous type from a method but Tomas confirms it is possible by casting and cites the C# 3.0 spec:

Within the same program, two anonymous object initializers that specify a sequence of properties of the same names and types in the same order will produce instances of the same anonymous type.

Tomas also qualifies his interesting post with a request to let him know if a good reason can be found for using this technique!

In my own work I recently wanted to call a method that returns what I would ultimately implement as a data transfer object: a strongly-typed collection of structured, related data. I considered out parameters and collections but neither option really struck my fancy; a DTO would likely be the preferred option at the expense of introducing another class, maintenance overheads, etc.

In the hunt for a better, modern solution, the concept of returning an anonymous type appealed—apart from the problems anonymity introduces! While I understand how .NET constructs anonymous types and reuses those that identical, the one major problem I see with Tomas’ solution is the requirement that both “sender” and “receiver” not only know about the so-called anonymous type but actually define it in two places. This just seems like more overhead to me so I abandoned this approach before long.

Enable testing of web services

I’m bound to forever forget how to switch on (or off) the ASP.NET web service test screen and documentation options. For future reference:

http://msdn.microsoft.com/en-us/library/b2c0ew36(VS.85).aspx

or add this to the system.web section of the service’s web.config file:

<webServices>
        <protocols>
            <add name="HttpSoap12"/>
            <add name="HttpSoap"/>
            <add name="HttpGet"/>
            <add name="HttpPost"/>
        </protocols>
</webServices>

Best Practice List Development in SharePoint

I had the honour of speaking on the subject of best practice SharePoint list development at last weekend’s SharePoint Saturday event held in Perth, Western Australia but I want to take this discussion one step further. I’m therefore creating this post to point to my slide deck from the event and as an index for follow-on, detailed posts about some of the items I had to gloss over. There’s precious little out there, in a single location, on the subject of list development guidelines and I’m hoping to change that. If you want to contribute what you feel is a best practice I’ve missed or challenge one of the practices I do have listed, please comment and join the discussion.

Slides are available here: Best Practices for Developing SharePoint Lists (PowerPoint | PDF)

Here are the two key resources I cite in the presentation:

• Best Practices: Common Coding Issues When Using the SharePoint Object Model 
• Best Practices: Using Disposable Windows SharePoint Services Objects

Add and manage workflows using the API

Programmatically applying and configuring an out-of-the-box workflow to a SharePoint list isn’t the most obvious thing to do but fortunately it’s not exactly hard either. I had to dig around a bit to find the pieces I need to do this and thought I’d share…

List Settings - Versioning

In my case I’m applying an approval workflow to a Pages library created by the Publishing feature; before I wrangle the workflow I need to configure the versioning settings on my Pages list. Through the UI, I can do this by configuring these settings as follows:Essentially, content approval is required, users need to be able to create major and minor (draft) versions, all editors should be able to see all drafts, and checkout is required to edit documents.

To do this setup programmatically, I set the following properties on my Pages list:

SPList pages = web.Lists ["Pages"];

// Require content approval
pages.EnableModeration = true;
// Create major and minor versions
pages.EnableVersioning = true;
pages.EnableMinorVersions = true;
// Who should see draft items
pages.DraftVersionVisibility = DraftVisibilityType.Author;
// Require documents to be checked out before they can be edited
pages.ForceCheckout = true;

pages.Update ();

Remove Existing Workflows

In some cases, you may also want to cleanup a list’s existing workflows (i.e. remove them) and this is easily accomplished using the SPList.RemoveWorkflowAssociation() method. Note several examples on the interwebs suggest doing this within a foreach loop but doing so will invalidate the enumerator after the first item is removed; use a for loop or a while loop instead:

// Remove existing workflows
for (int associationIndex = 0; associationIndex < pages.WorkflowAssociations.Count; associationIndex++)
{
    SPWorkflowAssociation workflowAssociation = pages.WorkflowAssociations [associationIndex];
    pages.RemoveWorkflowAssociation (workflowAssociation);
}

If you need to remove individual workflows, inspect the SPWorkflowAssociation.Name property.

Workflow Setup…

With that out of the way, let’s move on to the workflow configuration proper. In short, it’s all about the collection of SPWorkflowAssociation objects exposed through the WorkflowAssociations property of the SPList class. The AssociationData property exposed by the SPWorkflowAssociation class is also crucial.

Task and History List Management

When you setup a workflow through the UI, you’ll be prompted to specify names for the task and history lists (the latter of which will be hidden); when you’re doing this programmatically, you have to manage this part yourself. It would have been nice if the API did this automatically but it doesn’t. The CreateListAssociation method we’ll examine in a moment also requires a reference to each list so you need to a) determine if each list exists or create it and b) retrieve a reference to each list. To wrap all of this logic up together, here’s my helper method (based on this by Marco):

private void EnsureWorkflowLists (SPWeb web, out SPList taskList, out SPList historyList)
{
    const string WORKFLOW_TASK_LIST_NAME = "Workflow Tasks";
    const string WORKFLOW_HISTORY_LIST_NAME = "Workflow History";

    // Task list

    try
    {
        taskList = web.Lists [WORKFLOW_TASK_LIST_NAME];

        Console.WriteLine ("Tasks list found.");
    }
    catch (System.ArgumentException)
    {
        // List does not exist so create it
        Guid taskListId = web.Lists.Add (WORKFLOW_TASK_LIST_NAME, string.Empty, SPListTemplateType.Tasks);
        taskList = web.Lists [taskListId];

        Console.WriteLine ("Created task list for web.");
    }

    // History List

    try
    {
        historyList = web.Lists [WORKFLOW_HISTORY_LIST_NAME];
        Console.WriteLine ("History list found.");
    }
    catch (ArgumentException)
    {
        Guid historyListId = web.Lists.Add (WORKFLOW_HISTORY_LIST_NAME, string.Empty, SPListTemplateType.WorkflowHistory);
        historyList = web.Lists [historyListId];
        historyList.Hidden = true;
        historyList.Update ();

        Console.WriteLine ("Created history list for web.");
    }                        
}

Call EnsureWorkflowLists method like so:

SPList taskList;
SPList historyList;
EnsureWorkflowLists (web, out taskList, out historyList);

Workflow Template

Before you can finally create the new workflow association, you’ll also need an SPWorkflowTemplate. You can do this by calling the GetTemplateByName method hanging indirectly off the SPWeb object:

SPWorkflowTemplate approvalTemplate = web.WorkflowTemplates.GetTemplateByName ("Approval", System.Globalization.CultureInfo.CurrentCulture);

If you have any issues here, ensure the Routing Workflows feature (or the relevant feature for whatever OOB workflow you’re dealing with) is enabled at the site collection level (/_layouts/ManageFeatures.aspx?Scope=Site).

Create the Workflow Association

With the leg work behind us, actually applying a new workflow association to the list and configuring it is quite easy, with one exception I’ll cover in a moment. To begin, create a new workflow association using the SPWorkflowAssociation.CreateListAssociation method and supply the workflow template and list references created previously:

SPWorkflowAssociation approvalWorkFlowAssociation = SPWorkflowAssociation.CreateListAssociation (approvalTemplate, "Page Approval", taskList, historyList);

Note the name parameter “Page Approval” is arbitrary but will appear in the UI and notification emails; I therefore aim to make it user friendly and somewhat specific.

The created object can then be configured, in particular using the AllowManual, AutoStartChange, and AutoStartCreate properties.

As mentioned, there’s one last hurdle and that’s providing the XML string to the SPWorkflowAssociation.AssociationData property. As far as I’m aware, this is the only way to specify the nominated approvers group and a few other properties such as the ability to delegate tasks and whether a change to the list item should interrupt the current workflow. I believe the easiest way to create this string is to copy an example (below) or configure the workflow as desired in the UI and use SharePoint Manager to extract the XML from the workflow association (highlighted in the image below).

<my:myFields xml:lang="en-us" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:my="http://schemas.microsoft.com/office/infopath/2003/myXSD">
  <my:Reviewers>
    <my:Person>
      <my:DisplayName>Approvers</my:DisplayName>
      <my:AccountId>Approvers</my:AccountId>
      <my:AccountType>SharePointGroup</my:AccountType>
    </my:Person>
  </my:Reviewers>
  <my:CC></my:CC>
  <my:DueDate xsi:nil="true"></my:DueDate>
  <my:Description></my:Description>
  <my:Title></my:Title>
  <my:DefaultTaskType>1</my:DefaultTaskType>
  <my:CreateTasksInSerial>false</my:CreateTasksInSerial>
  <my:AllowDelegation>true</my:AllowDelegation>
  <my:AllowChangeRequests>true</my:AllowChangeRequests>
  <my:StopOnAnyReject>true</my:StopOnAnyReject>
  <my:WantedTasks xsi:nil="true"></my:WantedTasks>
  <my:SetMetadataOnSuccess>false</my:SetMetadataOnSuccess>
  <my:MetadataSuccessField></my:MetadataSuccessField>
  <my:MetadataSuccessValue></my:MetadataSuccessValue>
  <my:ApproveWhenComplete>true</my:ApproveWhenComplete>
  <my:TimePerTaskVal xsi:nil="true"></my:TimePerTaskVal>
  <my:TimePerTaskType xsi:nil="true"></my:TimePerTaskType>
  <my:Voting>false</my:Voting>
  <my:MetadataTriggerField></my:MetadataTriggerField>
  <my:MetadataTriggerValue></my:MetadataTriggerValue>
  <my:InitLock>true</my:InitLock>
  <my:MetadataStop>false</my:MetadataStop>
  <my:ItemChangeStop>true</my:ItemChangeStop>
  <my:GroupTasks>true</my:GroupTasks>
</my:myFields>

Drop the XML into the SPWorkflowAssociation object you just created using the AssociationData property and you’re all set to add the association to your list using the SPList.AddWorkflowAssociation method:

pages.AddWorkflowAssociation (approvalWorkFlowAssociation);

Before updating the list list, you may also want to configure your new workflow as the default content approval workflow:

pages.DefaultContentApprovalWorkflowId = approvalWorkFlowAssociation.Id;

Follow that with pages.Update () and you’re done!

(!) All conflicts resolved but no files checked in due to initial conflicts

A minor annoyance with VS2008 SP1 and TFS 2008 SP1 today when attempting a check in. Here’s the stack trace:

• Delete a folder and a bunch of images from VS Solution Explorer yesterday (right-click, Delete)
• Check in the deletion and explicitly check for the delete action for the relevant files during the check in process
• Arrive this morning to do a final cleanup and make sure everything in the project was checked in before Christmas… hey, all those images are still wanting to be checked in!
• Attempt check in from Solution Explorer (specify comment, work item as per our policy configuration); Resolve Conflicts dialog pops up with the message “(!) All conflicts resolved but no files checked in due to initial conflicts”. Try again… believing, from memory, I’d seen this problem before and a second check in attempt had solved it. Same deal.
• Resume Conflict Resolution from File –> Source Control. Nothing found, “All conflicts resolved”
• Attempt check in from Source Control Explorer window in VS; same deal.
• Drop into the VS 2008 Command Prompt and navigate to the solution; tf checkin and tf resolve produce the same results as VS. tf checkin tells me “Conflict: The latest version of item $/… is deleted.
• Try tf resolve . /r /auto:AcceptTheirs but the command tells me “There are no conflicts to resolve”
• Confirm folder and images are definitely not present in either the project (via Solution Explorer) or the source control tree (via Source Control Explorer).
• Get Specific Version with ‘Overwrite all files even if the local version matches the specified version’ option checked; no change to the project or file system; same error message on check in.
• Undo Pending Changes… from Solution Explorer context menu. Success! Now I have nothing left to check in (any other files not involved in the deletion process were checked in separately) and TFS is tangle-free yet again.

Inner exception details displayed instead of wrapper exception

Wrapping an exception using the Exception (string, Exception) constructor and throwing the new exception in place of the original isn’t behaving as expected for me today: ASP.NET 3.5 displays the inner exception message in its default exception display; I expected to see the wrapper exception message. Oddly enough, details pertaining to the wrapper are logged to the Application event log.

So this:

try {
// Setup to make something go wrong
throw new Exception ("Inner exception");
}
catch (Exception ex)
{
// Wrap the inner exception and throw
// a new exception instead
throw new Exception ("wrapper exception", ex);
}

spits out this (note both the message and source error clearly pointing at the inner exception line):

Inner exception

Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.

Exception Details: System.Exception: Inner exception

Source Error:

Line 18: try
Line 19: {
Line 20: throw new Exception ("Inner exception");
Line 21: }
Line 22: catch (Exception ex)

Despite what the message and highlighted code reveal, the stack trace does imply the wrapper exception was involved along the way.

[Exception: Inner exception]
_Default.Page_Load(Object sender, EventArgs e) in Default.aspx.cs:20

[Exception: wrapper exception]
_Default.Page_Load(Object sender, EventArgs e) in Default.aspx.cs:24 …

Adding a second try/catch to catch any exception of type Exception and inspecting the results locates the inner exception property where I’d expect it to be. Alternatively, “swallowing” the inner exception in the original catch block and throwing a new exception using the Exception (string) constructor does just that—no sign of the inner exception. I’ve additionally tried catching different types of exception (I was originally throwing ApplicationExceptions and catch Exceptions) and constructing my own exception classes with the same effect.

Maybe I’m missing something but this seems like pretty basic stuff and a few other vaguely related search results hint that I’m not alone with this one. Of course, I can always handle all exceptions myself to prevent anything killing the application but the current approach is convenient at the moment for debugging outside production where it’s less convenient to attach a debugger or check the event logs.

If you’ve got any ideas or have come across this yourself, I’d love to hear your thoughts!

Retrieving a list item’s attachments

As you might expect, accessing a list item’s attachments in SharePoint isn’t straightforward. While SPListItem.Attachments returns a string-based list of attachment names, you’ll probably need to translate that into a URL at the very least, if not manipulate the attachment itself.

It helps to understand where SharePoint stores attachments after they’re uploaded. As usual, SharePoint Manager 2007 provides some insight:

Notice in the screenshot, I’m inspecting a list named “Announcements” and that list has one item (Hello Attachment—highlighted at the bottom). You’ll also notice there’s an Attachments folder sitting among the list items and it’s been expanded with to reveal another folder with the odd name of 2; below that you can actually see the attachment itself (My First Attachment.txt).

Before I get any further, I’ll point out the magical “2” folder isn’t magical at all: it relates to the list item in this list with the Id of 2—in this case, the Hello Attachment item (the first item in the list was deleted but, if it had attachments, there would be a corresponding “1” folder). It’s sensible to assume this structure exists to accommodate multiple attachments.

If you were to now try to access the Attachments folder in code using SPListItem.Folder, you’d be faced with a null reference exception. Instead, you need to ignore what SPM tells you about getting to the Attachments folder and arrive there instead using the parent list’s RootFolder property.

Assume you’re working within the scope of a foreach or have retrieved a list item via some other means:

SPList announcements = …Lists [“Announcements”]

foreach (SPListItem currentItem in announcements)
{
// we’re here
}

Before you do anything else, you’ll probably want to muck around and ensure you can actually access a list of attachments in some form. To identify the attachments by name, SPListItem.Attachment (SPAttachmentCollection) will return a collection of strings:

foreach (string currentAttachmentUrl in currentItem.Attachments)
{
}

At this point you’ve retrieved, from the example discussed earlier, a string like “My First Attachment.txt”—not of much use on its own. As mentioned, you’re probably after the full URL at the very least.

To access the actual attachment and not just it’s name, retrieve the list item’s parent list (SPListItem.ParentList) and then the parent list’s RootFolder. From there you can use the SubFolders collection to retrieve the “Attachments” SPFolder. Beware the SubFolders indexer expects the URL of the folder, not its name.

Finally, you need to retrieve the numbered folder specific to the list item. Here’s the complete mess:

SPFolder attachments = currentItem.ParentList.RootFolder.SubFolders ["Attachments"].SubFolders [currentItem.ID.ToString ()]

The SPFolder object’s URL property can now be prefixed to the attachment’s name retrieved previously or you can enumerate the SPFiles within the SPFolder.

Back in the day, Eric Shupps posted a way to do this by walking down the structure from the Lists folder. I think the above code is a slight improvement since the code’s a bit more manageable and you don’t need to retrieve the Lists folder or the parent list by name.

Ps. RootFolder is actually quite interesting as its Name property is actually the internal name of parent SPList object.

Dirty Words (Michael Hanes) - blog.mediawhole.com mailto:info@mediawole.com 

 

New Sites, New Widgets

Rottnest Island Authority Re-launch

The second-last site to be launched on Tourism WA's fully branded, MOSS-based site provisioning platform went live last Thursday; I reckon it’s one of the best looking sites they've launched to date (the Rottnest Island photography helps, of course). Check it out: http://www.rottnestisland.com

[Update 22 July: As mentioned in my profile to the right, I'd like to clarify the fact the new rottnestisland.com web site is provided by Tourism Western Australia under the Tourism eMarketplace program; although I was involved in the technical construction of the web site as a contractor working for Tourism Western Australia, Mediawhole and mediawhole.com were NOT involved in the launch of this web site. Whereas I previously used the terms "we" and "our" in this post, I was referring to Tourism WA and teams working with the Rottnest Island Authority.]

Booking Exchange

On the subject of all things new, the new online booking capability has also launched on westernaustralia.com. The system integrates with our existing search function and provides live availability information from V3’s Open Booking Exchange. The politics around this apparently simple change were massive but the technology side was relatively straightforward by comparison (web service calls from within SQL Server are as complex as this got from our end). If you’re a tourism operator, find out more.

Travel Planner

The online travel planner was also finally launched after nearly a year of work with a Sydney-based agency. Luke was our man on the ground with this one and he did a great job integrating drop after drop of this widget.

The travel planner “helps visitors collect, organise and share their WA travel itineraries.” You can sign up from the westernaustralia.com homepage and add items from across the site to your travel wallet.

A review of our team wiki seven months in

Last October I wrote about the launch of our dev wiki in the office. Today I want to report the concept, implementation, and utility of the project has been a clear winner.

First and foremost, the underlying wiki concepts make our wiki fly:

• Content is approachable by virtue of being findable through search and being categorised
• Content is maintainable and everything’s versioned just in case—no more getting stuck updating the date and version history or even having to check out a Word document.

On top of that we’ve got 100% buy-in from our ICT manager (he actually proposed the wiki) and close to the same from our CIO. What that means is our wiki is the first port of call when seeking information—big or small—on our dev work and environments. We’re still being asked to point formal paper documents to wiki articles and pointing some wiki articles out to formal documents until they need to be reviewed so we’re not quite there but we are closing the gap. Realistically the wiki won’t replace every document in the organisation but I can dream!

Perhaps more importantly, I’m also referencing wiki articles when someone asks me how to do something and this has led to genuine interest from our ICT, Service Delivery and Content Management and Digital Marketing teams. I’m openly saying to these teams: create an account, create new articles of interest to your area or your job function, and update existing articles. The entire wiki is very informal in this sense (we don’t even really have a template for articles) and the information we’re adding to the wiki is current and useful by virtue of being so accessible and approachable. It feels all wrong and a bit subversive but the wiki has proven so effective it’s hard to overlook!

The Summary Paper

I’ve got a favourite new business document: I call it the summary paper and it reminds a lot of the book report format from third grade. I’ll claim to have invented my specific version but I’m sure there’s different names for this document depending on the project management/software development/business model you’ve adopted; no matter…

So what are the summary paper’s defining features, you ask? The summary paper is:

• About something really important The summary paper exists to feed into a meeting; if someone’s called a meeting hopefully the subject matter is sufficiently important to warrant having everyone in the same room not doing other work.
• Broad Include at least a brief context or history about the topic to bring readers to the same level and quickly address any knowledge gaps.
• Specific With the context out of the way, the remainder of the paper should be exactingly specific—as always, qualify and quantify wherever possible and prefer well conceived charts, diagrams, and images that can stand on their own or at least significantly augment the text. Most importantly, the summary report should be about one single thing and that subject must inform the entire report; if something isn’t relevant to the main subject, it belongs in a different summary report. Specificity in many cases equates to authoritativeness.
• Short The whole point is to create something approachable and easy to digest between tasks or on the train ride home. No one really cares about what you do but they care event less about that fifty pager no one will ever read because they don’t have the time or interest. Shortness is goodness because a short summary paper is also easy to adapt to changing information; when it comes time to discuss the paper in a meeting it’s easy to say “let’s take five minutes to read through this document.” Simply put, a short paper is more likely to get read. Assume your readers already know something about the subject. Aim for one to four pages and print it double-sided.
• Enticing Print in colour and come up with a precise title that tells the reader exactly what the document is about.
• Textual Use bullet points and tables for short lists and data. The summary paper is meant to be read so write sentences and paragraphs grouped under headings. Avoid complex tables, tables with lots of text, and bullet points in tables… Number and call out all figures in the text.
• Fast Write it quickly before you loose focus and forget key points.
• Timely Focus on the current state of play and how things are now—not how they used to be or how they might be in the future. As a snapshot in time, ensure the data presented is current to within a few days.
• Decisive State problems clearly and write in clear terms. Don’t dick around with fancy vocabulary, wordiness, and bad writing. Do use a spell checker and proofread, proofread, proofread. Use an appendix of terms and abbreviations (and even synonyms) if required. Use an appendix for references is required. Don’t bog down the first few pages with titles and logos and document controls and sign offs and and references and disclaimers; do keep it simple: include the author’s name(s), the date, and a simple version number (eg. v1.0 where the 1 is the major version and the 0 is the internal draft version). Include page numbers and numbered headings for ease of reference during discussions. Use consistent formatting to create meaning.
• Open Encourage feedback and be willing to incorporate ideas and suggestions—v1.0 should never be the final version. Incorporate feedback as early as possible.
• Owned Limit the number of authors (preferably to a single person) to encourage ownership and clearly identify the person responsible for making updates. If the paper is eventually earmarked for distribution to senior management but was initiated by a junior, allow the junior to retain ownership—especially if continual feedback looping starts to happen. Ensure whoever initiates the paper has an existing understanding of the subject and remains engaged as discussion evolves.
• Complete and Accurate TBDs should be avoided. Exclude placeholder sections containing no content. If critical information is required for completeness, find and validate that information. If existing information is unclear, clarify it by research (speak to others, do some reading, experiment).
• Actionable List possible solutions—don’t wait for that to happen at some future meeting. Identify, by name, individuals required to provide information or complete tasks.

I love these papers. My experience with this approach so far has been to say to someone arriving at my desk “here, read this” and they actually do read it (sometimes standing right there) and it results in the flow of ideas I can use to make the paper better. I’ve easily got ten to twenty individuals feeding into a single piece of work and these briefs are the fastest, most accurate way to turn discussion into action.

Perth SharePoint User Group Webcast Q&A

Jeremy sent through a handful of questions from the February PSPUG webcast so I thought I would answer them here. Thanks to Joshua for these awesome questions!

Q. What decisions did you make early that you wish you could change now? Or what would you do differently if you could start over?

A. Where to begin?! We’ve been lucky, in one sense, having the option to rebuild the partner web sites on a separate platform. At that time, westernaustralia.com was up and running and we pretty much did start over.

On the wa.com side, we made a very conscious decision soon after launch to move from SharePoint Designer and master pages/layouts held in the content database to a solution/feature-based deployment situation. Moving away from SPD is one of the best things we’ve ever done. We’ve currently got one feature for deploying most things on the site but there are cases where it would have been useful to componentise some aspects.

The biggest issues we face currently, off the top of my head, are as follows:

• Too many page layouts I mentioned the concept of page layout sprawl in the presentation; we’ve locked ourselves into a situation where content editors love being able to easily create a new page with all the page components they need for that section of the site at the cost of flexibility from their end and maintainability from the dev perspective. We’re revisiting this now and looking at ways a content editor can create a new page from a smaller selection of layouts and have the necessary web parts, specific to that subsite, added automatically. The Andrew Connell way of doing this through the AllUsersWebPart node in manifest.xml doesn’t work very well in this case as it’s too coarse and results in duplicate web parts every time the related features are activated. I imagine we’ll probably end up using the SPLimitedWebPartManager to do this but I’m not yet sure how!
• Reliance on user controls over web parts This very much ties in with my first bullet point. User controls are easier to develop because you’ve got real markup to play with and they help construct a tight, enforceable template but you’re trading configurability and the web part storage mechanism to go down that path. The partner sites are based solely on web parts and they work quite well in that environment where different partner sites want web parts in different places. Writing web parts that are centrally configured is also an interesting twist to the web part approach.
• Separate authoring farm I’ll be speaking more about this in part two of my presentation but there was a perceived need (on recommendation from MS and for records keeping regulations) to create a separate farm for content editing activities; content would then flow happily from authoring to production using the content deployment tools. Yeah right. In reality we have no need for a separate farm and with a lot of new hardware going in shortly we’ll be looking to merge the authoring and production farms. We’re doing this successfully with the partner sites.
• Two code bases We maintain two separate Visual Studio solutions, one for wa.com and one for partners. The partner site development effort taught us how to really use the SharePoint platform as a site provisioning platform and while wa.com and the partner sites are quite different, I’d like to see the two solutions merged so wa.com becomes just another partner site, perhaps with it’s own special set of master pages/layouts.
• Lists Some things started out as lists on wa.com but we drifted away from lists for a while for performance and cacheability reasons. We’ve worked through those issues now and use lists successfully on the partner sites to simplify per-site, content editor-managed configuration and could easily apply that knowledge back to wa.com.
• Abstraction The SharePoint learning curve coupled with my predecessor’s minimalist approach to all things code mean our code base doesn’t abstract access to the platform. As a result, it’s currently up to every developer to understand SharePoint integration and things like how to correctly dispose of unmanaged objects. Good stuff to know, no doubt, but it makes life more difficult for a new developer coming on board who hasn’t worked with MOSS and means different devs end up doing things differently, which complicates maintenance.
• Variations The variations piece seemed appropriate at the time and the wa.com site structure reflects our variation structure. Variations actually work but they work at too high a level four our specific needs. Some of our pages are not to be overwritten while others should be and unless you’re using workflow to stop it happening, the variations mechanism will simply copy the entire site (AFAIK—it’s been a long time since we last looked at this one!). In reality, while our marketing division likes their language-based site structure, they’re not willing to maintain different content for each site so it’s complete overkill.

Q. Could you explain in more detail the search tool you built? Seems to have a nice combination of BDC [Business Data Catalog] and SQL features.

A. I was hoping to have our “search dude” along to speak on this subject but unfortunately he was too busy learning to fly—to give you an idea of his calibre. I’ll tell you what I know.

I may have briefly mentioned in the presentation that we started out, on Dimension Data’s recommendation, using MOSS search and the BDC. It’s worth pointing out we also used the SharePoint v2 search bits in conjunction with MCMS 2002 on the old sites so it seemed like a logical step forward. For the record, we’re indexing MOSS page content, tourism product in the ATDW database, and .pdfs.

With the weight of many different tourism operators and industry bodies seeking to influence their ranking in our search results, our search requirements are convoluted. As a rough example, we always display pages first, followed by accredited tourism operators (randomised), followed by non-accredited operators (randomised), followed, until recently, by non-members (randomised). In many cases, this resulted in the need to boost certain keywords, boost individual results in special circumstances, and “band” results. Then there’s things like B&B versus Bed and Breakfast, returning accurate results for “perth hotels five star”, and so on.

The guys working with the Beta 2 TR found it didn’t provide the customisation we needed and had some major memory management issues; despite assistance from the SharePoint Ranger Team, we were soon looking to alternatives. The Google search appliance was examined but we found even Google’s search algorithm wasn’t suited to our particular business domain. We haven’t revisited the BDC or Google since so these comments should not be considered a reflection on the current products.

With the BDC bug out of our bonnet, it was time to look at a custom build and that was accomplished by following the BDC model and adjusting it to our needs. The solution we’ve got in place now is a three-part system comprising a crawler, a SQL Server 2005 database which serves as an index, and a presentation layer built with web parts. The crawler is a console application that uses the MOSS API to iterate over the SharePoint sites it’s configured to crawl; it also has separate functionality to crawl the ATDW database. The crawler can do full or incremental crawls and the latest build has been working flawlessly.

The crawler doesn’t duplicate page and product record data but collects specific, searchable data that is then stored in the database. The database structure was originally de-normalised but performance testing during the latest rebuild found performance was better when the data was restructured according to typical relational database design principles. The database tables are also full-text indexed and a number of stored procedures expose results to the application using the business logic discussed previously.

While this solution meets our specific needs, it must be stressed the database structures reflect the things we index: pages, tourism products, and files. In other words, this solution works really well for us but it’s not a rebuild of the BDC and pointing it to a different LOB database would require a significant amount of rework.

Q. Could you profile a typical content author / publisher? What training do they do?

A. We’ve got two different types of “typical” content editor.

On the westernaustralia.com side, content editors have full control over the site (the security model is admittedly weak) and have a marketing/digital marketing background. These guys are the business owners of the site and they’ve been working with the platform and the site for a couple of years. They’re pretty comfortable creating new pages from an overly large selection of layouts, checking out an existing page, adding reusable content and rich text, checking in/approving/publishing. Basically, they’re young, web-savvy types who call themselves geeks because they know how to cut basic HTML and use Flickr ;-)

We used to have a dedicated trainer in house (no more) and while some related training materials were prepared and delivered at the time, I think the marketing team now learn it as they go and share information between themselves.

On the partners side, the experience of these content editors varies widely. Some may have worked with MCMS previously so they’ve got some preconceived ideas about how content management systems work (and how responsive they should be!). These guys all work off-site but are supported via a 1800 number and our four-person Service Delivery and Content Management team—effectively a content helpdesk. The partners range from youngish types focused solely on maintaining content to older volunteers; they all have different levels of experience using the internet. SDCM therefore train all new partners, bringing them in and going through everything they need to know to maintain the content on their sites. This includes page basics as well as managing list content, accessing stats, and so on.

Hope this was useful… any additional questions posted as comments shall be addressed here.