Agile

More often than not quality assurance is considered to be an implicit benefit of agile process with no specific techniques or practices applied. Integrating effective quality assurance can be one of the most challenging aspects of applying the Scrum framework.

Before considering how quality assurance factors can be applied to the Scrum framework, let’s start with a clear definition.

Quality assurance (QA) is a set of practices which collectively ensure that standards of quality are met (i.e. technical standards) and specified requirements are delivered (customer or user needs or expectations). Standards of quality may include for example measures of correctness, integrity, maintainability or usability. Quality assurance does not mean software testing; this is one element of a quality assurance process. Effective QA means there are limited defects in the first place. The core elements of a QA process include standards, audits, solution governance, testing, defect management (analysis, tracking and resolution) and communication.

Scrum Concepts

The following points describe how QA should be considered in relation to some of the core Scrum concepts.

Developer. In Scrum terms this means any project contributor such as Software Testers, UI Designers in addition to Software Developers. This term is obviously confusing and a source of misunderstanding.

Scrum Ceremonies. QA should attend all meetings (Sprint Planning, Product Backlog Grooming, Stand-ups, Reviews and Retrospectives). A team-wide, test-first philosophy is often the driving factor behind high performance Scrum teams. This concept is also referred to as acceptance-test driven development.

Working Software. This means fully tested quality software – not code complete.

Definition of Done. The DoD must include strict criteria for testing and quality.

Customer Satisfaction. Conditions of Satisfaction (or Acceptance Criteria) defined for each User Story drive the quality assurance process.

Scrum Artefacts

The following points describe how QA should be considered in relation to some of the core Scrum artefacts.

User Stories. The User Story is a concise description of a feature told from the perspective of the person who requires it. User Stories are best considered a reminder of a conversation and should be limited to the detail required for effective estimation and prioritisation.

The following (INVEST) factors should be considered in respect to the quality characteristics of a User Story;
Independent – can it be delivered in a single sprint, or do dependencies exist?
Negotiable – the user story is not a specification (reminder of a conversation)
Valuable – is there obvious intrinsic value to the user?
Estimable – can it be estimated (sufficient detail, available knowledge)?
Small – is it sized appropriately (to fit in a sprint)?
Testable – are there appropriate Conditions of Satisfaction – or Acceptance Criteria?

The Conditions of Satisfaction (CoS) are most significant to QA. Each condition is a specific, user-focused requirement for acceptance and should be written using the Given-When-Then format (or heuristic technique) as below.

Given [context] – e.g. I have created a Customer Record and Sales Invoice.
When [event] – e.g. I attempt to enter a Sales Payment above the Sales Invoice Total.
Then [outcome] – e.g. I am warned that the payment exceeds the outstanding amount.

The high-level CoS will drive the team discussion of the User Story (in a test-first approach), a “Design and Test” whiteboard approach can be useful in this respect where the team consider both the design and testing approach at the same time. Each high-level CoS will be translated into lower-level (detailed) test cases (technical; unit and integration or functional).

User Story Test Cases. The Conditions of Satisfaction (CoS) for a given User Story provide a high-level framework for the definition of specific test cases. Such test cases should be structured to address a small, focused end-to-end workflow (or scenario) to avoid interdependencies and should enable collaborative and automated testing. Effective test cases provide a simple set of step-by-step test instructions (or steps). Note, the CoS covers the happy path scenarios only, test cases must be developed also that cover critical path, alternate path, negative path, technical, validation and edge case scenarios.

Product Backlog. The Product Backlog is a single, ordered list of requirements (typically but not exclusively User Stories) used by Scrum teams. The list is constantly evolving, business value ordered and estimated by the team – including QA.

The following factors should be considered in respect to the quality characteristics of the Product Backlog;
Prioritisation – are the Product Backlog Items (PBI) accurately prioritised to a consistent scale based on business value, ROI, interdependencies and risk?
Estimation – are the PBI consistently estimated (and benchmarked) by the team?
Well Groomed – is the backlog frequently groomed by the team?

Product Backlog grooming entails User Story discussion by the whole team (lead by the Product Owner) with a view to revising and refining estimates and adding details such as new Conditions of Satisfaction. Regular sessions should be scheduled for this purpose; ideally 10% of the Sprint time should be dedicated to this task. In practice this is rarely the case, however given the direct relationship between the completeness of the CoS to the efficiency of the quality assurance process it is important that sufficient time is allowed for effective grooming.

Scrum Practices

The following points describe how QA should be considered in relation to some of the core Scrum practices.

In Sprint Testing. User Story testing must be completed within the sprint – full-stop. The Definition of Done must include this criterion. Testing a sprint behind the development is common bad practice and directly violates the working software principle; additionally this approach builds up technical debt and defect debt to be resolved at a later time. Instead, the sprint backlog must be shaped to enable collaborative design, development and testing of each user story within the sprint. High performing Scrum teams will also automate some of the tests within the sprint. The following simplistic process describes how collaborative sprint development can be applied.

Developers and Testers working collaboratively – Team Swarming:
Write the test case
Write the code
Test together (developers and testers)
Fix the bugs (developers) / Automate the test case (testers)
>> next user story

Test Automation. The ability to automate the execution of test cases (i.e. unit tests and acceptance tests) is critical at scale. In order to quickly identify cases where the current sprint development breaks the code or functionality developed in a previous sprint it is necessary to execute the related tests efficiently. Test automation also supports evaluation of the impact of changes (What-if analysis) that can help with design decisions. Further QA related benefits of test automation include the ability to apply load testing (e.g. multiple concurrent users) and performance testing types. Unlike manual testing which must be conducted in-sprint, test automation can be added outside of the sprint and be developed a separate team. High performance Scrum teams will automate critical tests within the sprint.

Test data quality has a significant impact upon the effectiveness of testing. Random data simply doesn’t work and prefabricated test data is often written to expectations and is not representative. Using production data (real data) for test purposes can violate data protection regulations and comes with the risk of inadvertent communication to the customer, supplier etc. or PII or financial information leakage. To mitigate this an approach that combines real data with obfuscation (scrambling or scrubbing) techniques can produce large scale, representative test data sets without the risk. This can take significant time to perfect however the return on this investment will be high.

Note, in addition to automated tests, a quality test plan still includes manual testing to cover exploratory testing and usability testing types. Automated tests are systematic and unable to gauge the quality of the user experience for example.

Regression Testing. The ability to run test cases developed in previous sprints (and perhaps by different teams) enables the impact of a current development to be understood immediately. This testing practice is referred to as regression testing. Regression testing effort gets bigger over time, as with User Story test cases, regression testing should occur in-sprint and automation should be used wherever possible. It is good practice to reserve time at the end of the sprint to allow the team to complete regression testing, this should also include performance and security test cases. At scale, it is often necessary to be selective in terms of which test cases to focus upon. An effective but contentious practice is to reserve the final sprint for end-to-end regression testing – this sprint is referred to as a hardening sprint; the key risk with this approach is that defect resolution from earlier sprints is deferred to this sprint. The hardening sprint should be driven from the regression test plan not the product backlog and be managed by the QA lead.

Post Release Support. Once the product is released – and despite the QA rigour applied – it is inevitable that production defects will arise. Such defects should be reviewed first by the Product Owner to determine priority and relevancy. Where defects are to be addressed immediately (show-stopper bugs etc.) the resolution work should replace similar sized work in the current sprint. In every respect the defect should be treated in the same manner as a User Story. Defects can also be placed on the Product Backlog for grooming. Where the Scrum team is providing defect resolution it is good practice to retain velocity for production defects; this proactive approach reduces the requirement to pull User Stories from the current sprint that may be in progress.

I’ve been fortunate to have worked as a consultant Salesforce architect for the last decade, during this time I’ve encountered an interesting variety of approaches to Agile in the Salesforce space – most of which fitting with the hybrid model unfortunately – what has been noticeably absent is the application of agile engineering practices. Test Driven Development (TDD) is one of most widely understood agile engineering practices (alongside Pair Programming) – and is perhaps the most practical to adopt, however despite the benefits, despite the level of lip-service paid to the subject in practice TDD remains a rarity. With this post I hope to provide a convincing argument to Salesforce developers and architects that TDD does not add overhead to the development lifecycle and in fact the opposite is true. The Salesforce platform in development terms is maturing, the introduction of new tools and techniques such as Salesforce DX (SFDX) are testimony to this. In this context, the idea of developers working ad-hoc and in an undisciplined manner doesn’t fit, instead the consistent application of established engineering practices will play a key role moving forward. No more low quality unit tests (written as an afterthought) and no more random coding conventions and inconsistent standards. Salesforce development teams can no longer choose to hide in the void that sits between declarative development and traditional software engineering, it’s definitely time for some new habits.

This blog post draws together a high-level conceptual view of Test Driven Development with the implementation practicalities unique to the Salesforce platform.

Let’s start with great quote to set the scene.

“Build it right to build the right thing“, Ron Jeffries. The concept is simple; build the product right, repeat until it’s the right product.

Test Driven Development

Test Driven Development is an XP agile engineering practice. eXtreme Programming (XP) is a software development methodology focused on customer value, continuous feedback, incremental feature delivery and team empowerment. It is a widely recognised fact that the earlier testing is applied in the software development lifecycle – the greater the value (shorter development timescales, more reliable code etc.). TDD takes this idea to its ultimate conclusion where testing occurs in front of actual coding. This may seem counterintuitive. In order to understand the impact of this fully it’s important to understand the high-level approach.

(1) Identify a low-level requirement (PBI, Task etc.)
(2) Describe the logical tests necessary to confirm that code satisfies the requirement
(3) Write the minimum code necessary to pass the tests
(4) Tidy-up the code to remove clutter, complexity and duplication

The steps above describe a design process where software evolves in very small increments with a focus on simplicity and testability. This step-by-step approach results in emergent design, where the code informs the design approach and guides the developer toward the best solution. This is the direct opposite of the traditional big-design-upfront (or BDUF) approach. The goals and benefits of TDD are many; “clean code that works” is a phrase that best summarises this (another golden Ron Jeffries quote). Note, TDD is often referred to as Test Driven Design in recognition of this emergent design side-effect.

TDD Cycle

It is imperative to understand the conceptual aspects of TDD in order to gain the full benefits, thankfully the practical application follows a relatively straightforward and approachable 3-stage cycle (Red-Green-Refactor).

Stage 1 – Red : Write the minimum code needed for a failing test. This may entail development of (or extension to) a test harness, plus the unit test code required to exercise the logical test case. The test code could be completely new or simply an augmentation to existing tests. In writing the test code the developer is forced to think through the requirement in detailed, logical terms – the type of thinking more typically applied at a later stage when writing the functional code.

It is imperative that test code evaluate predictable logical test conditions. The test code should fail for a specific, expected reason – not due to a test harness issue or due to the fact the test code itself is invalid.

Stage 2 – Green : Write the minimum amount of code needed to pass the test. The code written here must satisfy the test condition and nothing more – no embellishments.

Discipline is required to avoid the developers innate tendency to create.

Stage 3 – Refactor : The functional code should be reviewed and modified to remove clutter, complexity and duplication. Code should work (of course) but also (critically) be readable, updateable and testable.

Clean code is the objective here; if every tiny incremental addition of code satisfies the clean code criteria (readable, updateable and testable) the outcome will be a highly organised and easily maintainable code base that can be extended with new features at a consistent level of effort over time. Where such an approach is not applied it is often inevitable that a complete re-write becomes necessary once the steadily increasing incremental cost (time) of adding new features becomes intolerable.

The TDD cycle then repeats over and over until all requirements are addressed. The incremental design approach focuses upon each individual requirement one at a time. This minimalist, simple design lead approach can be difficult for experienced developers to adopt, however once the learning curve is addressed the focus on simpler designs allows easier modification, more efficient testing and faster production.

TDD in the Salesforce Context

The Salesforce (or Force.com) cloud platform is an ideal software development environment in which to adopt the Test Driven Development (TDD) agile engineering practice. The Apex programming language has integrated unit testing capabilities which reduce the level-of-effort required. Salesforce Developers are also accustomed to the idea that code requires a (platform enforced) test coverage level of 75% for production deployment. In the Salesforce development context therefore the tools are provided and concepts relating to unit test development well understood. The only challenge remaining is the mind-set shift from big-design-upfront and retrospective test code development to TDD and the agile benefits of emergent design, clean resilient code and reduced development effort.

Unit Testing on the Salesforce Platform

An Apex class can be easily annotated (@isTest) to indicate its role as a unit test class. The snippet below shows a very basic example.

[code language=”java”]
@isTest(SeeAllData=false)
private class FeatureTestClass {

@testSetup static void setup(){
// populate custom settings.
// create test data used by all tests.
}

/* **/
static testMethod void isAccountProcessed() {

// setup test specific test data and context
insert new Account(Name=’Acme Inc’);

Test.startTest();

// do something
AccountProcessor.process();

Test.stopTest();

// test outcome
System.assertEquals(1, [select count() from Account where IsProcessed__c=true]);
}
/* */
}
[/code]

The [SeeAllData=false] flag indicates that only test data created by the class is visible to the test method (plus some actual data; products, users etc.). There are circumstances where the flag must be set as [SeeAllData=true] flag, where test code is unable to create required test data due to object access permissions. The isTest annotation takes a second parameter [isParallel=true|false] which controls whether the test class can run in parallel with other tests (for expedient execution) or whether the behaviour requires serial execution (perhaps to avoid lock contention on shared resources such as custom settings).

Once a test class is implemented, adding individual tests becomes very straightforward. A standardised naming convention for the test cases can help the readability of the code; perhaps based on the expected outcome or the logical test case. Test code organisation at the class level should also be predictable and may follow a feature convention.

There are a number of methods available for the execution of test code on the Salesforce platform; Setup menu, Developer console and API (incl. IDE) as examples. Tests can also be executed asynchronously at a scheduled future time via Apex or the API (ApexTestQueueItem, ApexTestResult objects).

As the screenshot above shows the developer can configure a test execution to include all or selected test cases from multiple test classes. In the context of the TDD cycle the developer requires the ability to run just the one test case – not all, this would waste time.

Test Suites allow test classes to be grouped and executed together; this can be seen in screenshot below.

In the TDD context, where unit tests drive the development process it is important to have a clear naming convention defined for:

Test Case: expected outcome, or logical test case (readability is key)
Test Class: group of test cases relating to a given feature
Test Suite: group of test classes relating to a functional area

What is a Unit?

For the TDD cycle to be effective it can help to understand the level at which a unit test is applied. A unit is variably defined as the smallest testable part of an application, or the smallest change that still makes sense to be tested in the developer’s opinion. This of course is a subjective definition.

Common sense should be applied; a 200 line Apex trigger on Account will not be a single unit, a 20 field update on Contact also won’t be 20 units. Instead the feature being developed should be decomposed into a unit test per logical test case. This can take practice to get right, but striving to keep each test as simple as possible should be the primary guiding factor. Simple tests are easy to modify and provide a self-documenting benefit.

Advanced Unit Testing

The Apex language supports the advanced unit testing technique of Mocking which enables isolation of test execution from external factors such as API callouts or Salesforce configuration changes.

Mocking essentially involves the non-disruptive, dynamic substitution of a test specific class instance in the place of a functional class instance. Apex provides the HttpCalloutMock interface to allow test code to utilise pre-fabricated Http callout responses. The Stub API (StubProvider interface and Test.createStub method) supports the development of Mocking frameworks that enable any class to be substituted dynamically at runtime.

References

Apex Developer Guide > Testing Apex

Ron Jeffries Blog – All things XP and much more

Martin Fowler – All things Code Refactoring and much more

There are typically two interpretations of the term “development process” – one being the tools, practices and methods applied in software development (i.e. methodology, plus build automation, standards etc.) the other being the process applied to get from requirements to working software (i.e. iterative or waterfall, plus how the analysis-design-build-test-release disciplines are executed). This post outlines one high-level approach to the latter in the context of Salesforce developments. The intent of this isn’t to be overly prescriptive, generally speaking each project requires its own defined process that factors in resources available (and their skills and experience critically) plus the nature of the work and the timescales. That said it is a truism that failed projects fail for a variety of reasons but successful projects are typically successful for the same reasons. A fundamental success factor being the adoption of a clearly defined and simple process – others being team empowerment and shared commitment.

The process above assumes an iterative process and focuses the initial iteration on the foundation of a robust data model, set of user profiles and permission sets, role hierarchy, record access model and statement of the reporting requirements for the project. Subsequent iterations improve the quality of the foundation over time, as new functional areas are developed. The data model in this context will include a statement of the org-wide defaults for each object and the specifics of each relationship (master-detail, lookup, mandatory lookup etc.). The record access model is critical – this shows how each user population maps to a user profile and role and how they gain access to the data required, i.e. sharing rule, Apex managed sharing etc. In my experience defining an approximate access model upfront and then refining during the feature build-out helps to avoid expensive refactoring later in the process and sets out a clear understanding for all contributors to the declarative and technical build. A piecemeal approach to defining a sharing model is commonplace – this rarely provides a clear and cohesive result. For the similar reasons defining a list of permission sets upfront ensures that user profiles are kept clean and focused, avoiding proliferation of profiles down the line. It may be surprising to see analytics such as reports and dashboards being considered during the foundation stage, this however is one of the primary inputs to the definition of a fit-for-purpose data model. I’ve worked on countless projects where reporting has been overlooked until a late stage, at which point it has become apparent that standard reporting features can’t produce the reports given the structure of the data. Ideally the data model should be designed from the outset to work well for both transaction processing and analytics.

A final point for consideration is the by-exception approach to identifying technical components. When breaking out the solution components required for a certain feature – expertise must be applied to ensure that standard product functionality or declarative options (workflows, reports etc.) are considered fully before bringing expensive technical options such Visualforce or Apex to bear.