Best Practice

This post outlines my preferred approach to managing parallel developments on the Salesforce platform in what I refer to as the Converged Programme Model. I readily acknowledge that there’s a multitude of ways to accomplish this each with it’s own subjective merits. Before adopting a parallel work-stream model take the time to understand the technical complexity, process overhead and time investment required. Of particular concern should be the team’s readiness for such a disruptive change. In my experience it’s better to plug any skills gaps upfront, be very prescriptive with process guidance, start-small and build out incrementally – the risk otherwise is considerable. Typically resistance will come from individuals unaccustomed to a disciplined approach to software development/release process.

Objectives

1. Concurrent Development. Support parallel programme workstreams converging into a shared production Salesforce environment.
2. Automation. Deliver build automation – reducing the manual overhead required to deploy between environments.
3. Gold Standard. Deliver a best practice approach – the initial design should scale up and down in response to changing programme conditions.
4. Non-disruptive. Facilitate a staggered approach to adoption – enabling key benefits to be realised quickly without disrupting productivity.
5. Minimise Release Overhead. Project branches should be regularly and incrementally updated from the master branch – reducing the inherent risk of divergence over time.

Tools

1. GitHub

– Get started with public repositories, upgrade to a paid plan and use private repositories for any source code you don’t want to share with the world at large.

– Create an Organisation account to enable Team functionality.

– Key benefit versus Subversion (CVS etc.) is fast and efficient branch management; parallel workstreams are managed on branches with frequent merging.

– It is possible, albeit time expensive to implement a Git server within the enterprise. In my view the GitHub administration interface alone is worth the price.

2. Jenkins

– Deployed on a Windows EC2 instance with an elastic IP. A free usage tier, micro instance provides an ideal server host. Using a Linux host can be beneficial in regard to SSH authentication from Jenkins to GitHub. This is just one advantage of many. Pick the Operating System/Platform the team you’re working with are most familiar, a Linux host that only one team member can administrate makes no sense.

– On Windows the Jenkins service should be configured to run a specific user account (with least privileges assigned). This is required to generate the key files for SSH authentication.

– Enable Jenkins security. Particularly relevant if the host is open to the public web. Lock the inbound IP ranges via the EC2 security group if possible.

– Either store the Ant build files (build.xml, build.properties) in the Git repository or use an XCOPY post-build step to copy the files into the workspace from a file system location – as below. I prefer to keep the build files external to Git – there shouldn’t be any need to version manage such files – plus the build.properties file may contain passwords in plaintext.



– Install GitHub and Git Plugins
Required to build from a GitHub repository and enables build automation via Post-Receive Hooks. Under Jenkins System Configuration; configure “Manually manage hook URLs”, this requires your GitHub repository to have the hook set manually via Service Hooks under repository settings. Add a [Jenkins (GitHub plugin)] service hook like http://yourservername:8080/github-webhook/. The message sent on git-push to the remote repository will trigger any Jenkins job that builds from the branch that has been updated and has the [Build when a change is pushed to GitHub] option set to true.

– SSH Keys
In order to use SSH from Jenkins to a private GitHub repository, SSH authentication is required, which uses a generated key pair. The public key is added as a Deploy Key in GitHub under repository settings. This works well but if you want the same Jenkins user to access multiple repositories over SSH you have a problem as each Deploy Key must be globally unique across all GitHub repositories. The answer to this is to use aliasing and a SSH config file (refer: http://www.onemogin.com/blog/2011/9/1/jenkins-and-github-multiple-private-projects.html) however this won’t work with Post-Receive Hooks as the repository URL in the sent message won’t match to the aliased repository URL in the Jenkins job – typically errant behaviour below from the Jenkins log. I can’t see a way around this at the time of writing this post.

[sourcecode language=”text”]
FINE: Skipped GitHub Test – buildautomationtest repository because it doesn’t have a matching repository.
May 7, 2013 6:21:35 PM com.cloudbees.jenkins.GitHubWebHook
FINE: Considering to poke GitHub Test – buildautomationtest repository
[/sourcecode]

– Chatter Plugin
I’m a big fan of this plugin by Simon Fell. I tend to use a dedicated release manager user, e.g. release.manager@force365.com, standard user license capacity permitting, and perform all deployment tasks in this user context. This approach provides clarity on changes made by a deployment versus actual user and provides an easy way to be notified of failures etc.

Key Principles

1. Fit-for-purpose Org-set

– Org-set is the terminology I use to describe the collection of orgs, and their roles, required to deliver a project safely to production.

– One size does not fit all. Pick the minimum set of orgs roles required to deliver the project. Each org is a time expensive overhead.

– Sandbox types. In defining the org-set, factor in the availability of config-only and full-copy sandboxes. The latter must be retained for cases where infrequent refresh is required. Project-level orgs don’t need to be part of the sandbox estate, Developer Edition orgs, or perhaps Partner Developer Edition orgs can be employed. Full-copy sandboxes are incredibly expensive, valuable resources, use only when absolutely necessary for as wide a set of roles as possible.

– Connected orgs. For projects involving complex integrations, the complexity involved in creating a connected-org may influence the org-set design – there may be an argument to consolidate roles onto a single test org used for QA and UAT perhaps.

2. Continuous Integration

A best practice org-set design for non-trivial technical projects with multiple technical contributors should require isolation of developer activities into a separate developer orgs with a code-level integration org and Continuous Integration (CI) process in place.

3. Project-level sandboxes are not refreshed

Project-level orgs are all built from the Git repository. The Pre-production programme-level org must be refreshed from Production pre-deployment to ensure the deployment is verified against the current state.

4. Commit to the remote project branch is a commitment that metadata is ready for system testing

Build automation will deploy a project branch commit to the project QA org. In my experience it pays to be prescriptive in terms of development process.

5. Commit to the remote master branch is a commitment that metadata is ready for integration testing

Build automation will deploy a master branch commit to the programme INT (integration) org – this org exists to enable rigorous regression testing to be applied by all project workstream. Post-deployment suites of automated tests should be invoked and reports analysed by the test lead on each project.

6. Test Automation

It’s a significant resource overhead to execute manual test scripts for each regression test cycle, not to mention error prone. For non-trivial projects, the investment must be made at an early stage in automated-testing. Selenium is a good choice, but the tool utilised doesn’t really matter, what matters is that from the outset of the project the test team start to build-up a comprehensive suite of automated test cases with coverage of the key acceptance criteria defined for each user story. The suites then enable automation of regression testing during deployment phases – the same scripts underpin system testing and provide an often overlooked second stage to CI (unit tests + acceptance tests).

7. GitHub branch design

– A simple, clean branch design is desirable in the remote repository.

– Long-lived branches for active project workstreams. Project branches may have sub-branches for each sprint or phase.

– Long-lived branch for patches. Bug fixes are developed on local branches and committed to the remote support branch when ready for system testing.

– It can be advisable to consider how important a clean Network Graph is, this is impacted by Git merge versus rebase decisions.

8. Build automation challenges

In a perfect world, all metadata component types would be covered by the Metadata API. This isn’t the case so the nirvana of simple cloning of an org configuration is yet to exist. Instead a prescriptive process is required which spans manual configuration tasks, metadata deletion and build automation.

– Proactive management of change
A nominated release manager should proactively manage change at a programme-level, advise the project teams on release process and strive to minimise deployment conflicts through early involvement in all project developments. A change log should be maintained which lists all changes being made. This could include technical component types (ApexClass, ApexTrigger etc.) being added, modified or deleted, but as a minimum must track configuration changes requiring manual action – enablement of features, field data type changes etc. and required standing data (custom settings etc.). All changes should be mapped to a Change Type of manual or automated and a list of orgs to which the change has been deployed tracked. This is clearly an overhead to the project but without control it can be very easy to lose track of the current state of the orgs in use and face significant time expense in attempting to rationalise the situation through failing deployments. The release manager, or technical lead should apply manual tasks to target orgs pre-emptively to minimise automated build failures.

– Be prepared for build failures
Automated builds will fail; this is a fact of life where build-dependencies on manual actions exist. Proactive management will only get you so far. Attempting to minimise this is more realistic than elimination.

– Data
Automation of data setup in a target org is possible via Ant and the Data Loader CLI, or other similar means. Alternatively a data file could be deployed as a document or static resource and then loaded from an Apex script (as per the ISV approach).

– Unsupported metadata component types
Automation is possible using Selenium scripts, which execute at the UI level and can simulate, for example, a user activating a setting. Such scripts can then be integrated into an automated build. This is highly possible, but takes time and expertise with both Ant and Selenium to accomplish.

9. Programme-level Integration

The Converged Programme Model involves project workstreams building in isolated org-sets with frequent merge-from-master actions bringing across any changes to the production state. This approach should surface conflicts early, i.e. during development itself, but to be sure that shared component changes have not introduced any functional inconsistencies, regression testing must be applied by each and every project workstream on each occasion any project does a release. This is a strong argument for test automation.

10. UAT

– Project-level or programme-level?
In principle UAT should always be applied at the local project-level as the commit to the programme-level integration org is an absolute commitment that the code is production ready. In practice UAT may be two tiered; initial user acceptance of new functionality, followed by some form of secondary acceptance testing in Pre-production, in parallel to deployment verification testing.

11. Path-to-production Change Management

As with any programme of work, fit-for-purpose Change Management processes should be in place. In context this means a Change Advisory Board (CAB) should be in place to approve deployment, this must include informed and empowered representation across business and technical functions.

– A Deployment Request Form (DRF), or similar, should be produced to document the change being released, the impact, pre and post deployment tasks, GitHub commit # etc., approval date or rejection reason. The DRF could be approved by a convened board or via email response.

– The DRF process is absolutely required for the final deployment to Production, but may also be applied to the Pre-Production deployment, i.e. the commencement of the final step of the path-to-production release flow.

Experienced Salesforce technical architects will always look to declarative solution options before considering technical alternatives. This type of thinking is best practice and indicative of an architect considerate of TCO (total cost of ownership), future maintenance etc.. In my case I’ll go as far as to challenge requirements such that I can deliver a fit for purpose solution using declarative features. In the main this is exactly the right approach – but in some cases there’s more to consider. Let me explain.

Large Data Volumes (LDV) is currently a hot topic within the Salesforce technical community, there’s some great resources and prescriptive guidance available. But aside from the data aspect, what about orgs with large and complex declarative and technical customisations? In some cases such LCOs (Large Customisation Org) have grown organically, in others the org is being used as a platform to deliver non-CRM functionality such as complex portal solutions, ERP etc.. In either case what we’re talking about here is an org with high levels of custom objects, workflow rules, formula fields, Apex script and so on. In such a scenario it is highly likely that the LCO will be constrained by capacity limits (maximum number or users, custom objects, data size etc.) or execution limits (governor limits applied to Apex scripts etc.). In the organic growth case, where an org may have started life in one business division and then expanded across the enterprise, there will certainly come a point where a multi-org strategy becomes the only option, continual refactoring and streamlining to provide additional headroom will eventually cease to be viable. In light of this multiple-org architectures are becoming more commonplace with enterprises partitioning over organisation structure or business process boundaries, enabling localised innovation and growth with some data sharing and consolidation. That said, the transition from the single-org to the multiple-org model is potentially costly and disruptive, as such the design factors to consider, to optimise the longevity of a single-org implementation in the face of organic growth, are key for an architect to understand and implement from the outset. A firm understanding of the applicable limits for the org type and user licensing model is the best starting point for this, combined with the practical experience of where limits are soft, and can be increased by salesforce.com support, and where limits are hard platform constraints. This latter type of limit being most relevant to the goal of optimising org longevity. An example being the Spanning Relationships limit.

Spanning Relationships Limit
This limit constrains the total number of unique object relationships which can be referenced in declarative build elements (workflow rules, validation rules etc.) associated with a single object. This is a significant constraint on larger data models, and typically surfaces first for the central standard objects (Account, Contact, Case etc.). The soft limit here is 10, the hard limit being 15, however there are also performance degradation considerations at anything over the 10 level. When this capacity limit is reached, the only options are to refactor the declarative implementation or revert to Apex script solution options. It is therefore critical to understand that this hard limit exists when designing a data model and also when adding declarative elements which introduce a new relationship traversal. There may be an argument for some level of denormalisation in the physical data model, it’s generally unlikely that a Salesforce data model would be in 3NF anyway, unlike a traditional RDBMS a data model optimised for storage is not always the right approach.

Returning back to my original point, in considering declarative solution options versus technical alternatives, the complexity of the data model, plus capacity limits applied to the declarative build model are also factors. There’s no silver bullet answer.

A robust and intelligent data model provides the foundation upon which a custom Salesforce implementation can be built. Mistakes made in the functional or technical build are typically inexpensive to rectify (if caught quick enough), however a flawed data model can be incredibly time and cost expensive to mitigate. At the start of all projects I produce a logical data model, example provided below. this starts out as blocks and lines and improves iteratively to include physical concerns such as org-wide defaults, relationship types etc.. Only after a few revisions will I consider actually creating the model as custom objects. I use OmniGraffle for such diagrams.

As a best practice I implement a hierarchy type Custom Setting (OrgBehaviourSettings__c) which enables a highly useful, org-level selective switching-off of dynamic behaviour such as Triggers. The setting usually has independent flags for workflow rules and validation rules also, with the setting fields being referenced in rule entry criteria and formula expressions respectively. Having such a setting in place (and fully respected across the declarative build and Trigger scripts) from the outset can be really useful for diagnostics and data loading. It may be advantageous in some cases to provide a different set of values for an integration user perhaps – do this with caution..

Example object definition
[sourcecode language=”xml”]
<?xml version="1.0" encoding="UTF-8"?>
<CustomObject xmlns="http://soap.sforce.com/2006/04/metadata"&gt;
<customSettingsType>Hierarchy</customSettingsType>
<customSettingsVisibility>Protected</customSettingsVisibility>
<description>Org-level behaviour settings – enable switching-off of Apex Triggers etc.</description>
<enableFeeds>false</enableFeeds>
<fields>
<fullName>TriggersEnabled__c</fullName>
<defaultValue>true</defaultValue>
<description>When set to True, Apex Triggers (coded to respect the setting) will execute – when set to false Apex Triggers exit immediately.</description>
<externalId>false</externalId>
<inlineHelpText>When set to True, Apex Triggers (coded to respect the setting) will execute – when set to false Apex Triggers exit immediately.</inlineHelpText>
<label>TriggersEnabled</label>
<type>Checkbox</type>
</fields>
<label>OrgBehaviourSettings</label>
</CustomObject>
[/sourcecode]

Example trigger code

[sourcecode language=”java”]
trigger OnAccount on Account (after update
// after delete,
// after insert,
// after undelete,
// before delete,
// before insert,
// before update
) {
OrgBehaviourSettings__c obs = OrgBehaviourSettings__c.getInstance();
System.debug(LoggingLevel.ERROR, obs);
if (!obs.TriggersEnabled__c) return;

AccountTriggerHandler handler = new AccountTriggerHandler(Trigger.isExecuting, Trigger.size);

// if (Trigger.isInsert && Trigger.isBefore){
// handler.onBeforeInsert(Trigger.new);
// } else if (Trigger.isInsert && Trigger.isAfter){
// handler.onAfterInsert(Trigger.new, Trigger.newMap);
// } else if (Trigger.isUpdate && Trigger.isBefore){
// handler.onBeforeUpdate(Trigger.new, Trigger.newMap, Trigger.oldMap);
// } else if (Trigger.isUpdate && Trigger.isAfter){
handler.onAfterUpdate(Trigger.new, Trigger.newMap, Trigger.oldMap);
// } else if (Trigger.isDelete && Trigger.isBefore){
// handler.onBeforeDelete(Trigger.old, Trigger.oldMap);
// } else if (Trigger.isDelete && Trigger.isAfter){
// handler.onAfterDelete(Trigger.old, Trigger.oldMap);
// } else if (Trigger.isUnDelete){
// handler.onAfterUndelete(Trigger.new, Trigger.newMap);
// }
}
[/sourcecode]

Don’t forget to populate the setting in test code (example below creates default org-level values).

[sourcecode language=”java”]
OrgBehaviourSettings__c obs = OrgBehaviourSettings__c.getInstance( UserInfo.getOrganizationId() );
obs.TriggersEnabled__c = true;
insert obs;
[/sourcecode]