Stephan's Process Blog

Treat Automated Testing as Real Software Development

Over the last few weeks I have been working in a project that has invested heavily in automated regression testing. To that extent the project even developed a dedicated interface to allow accessing and controlling the user interface from testing programs. As a result testing is done via custom programs instead of using commercial record & playback tools. This gives the project much more flexibility and the ability to use open source testing frameworks that over the last years have matured considerably.

Back to the topic: working in this project has reminded me of what a fundamental lesson it is and how often it is ignored. All too often I see software professionals who are used to build large enterprise style applications to blithely treat automated testing as something where you just set up the tools and then testers write test scripts. Granted that the automated testing will result in much less code and even less sophisticated code than the application itself, but it is still non-trivial.

Here are some reasons why you need to address this like a real software development project:

Build a Testing Framework

I am not saying re-invent the wheel. There are by now excellent tools in the commercial and the open source area to support automated testing. My preference is for the latter as they give much more flexibility. Testing frameworks that once were used for Unit Testing have grown up and are quite capable of being used in full system level automated testing. Still you will need to develop an overall execution framework that is robust enough to withstand the demand of continuous testing over long periods of time. That means you need to select the right tools - think proof of concept evaluation of candidate tools. Then you need to tie everything together using a combination of configuration and glue code and build additional components which you may not be able to find in the commercial nor the Open Source domain.

Architect the Testing Framework

Building robust test frameworks is non-trivial. Application failures of the SUT (System Under Test) are expected so the framework needs to anticipate this and be able to recover from them in a robust fashion. Test Setups need to be robust to avoid them introducing failures that would obscure the actual test cases. Poor resource management by the testing framework may cause inexplicable failures after continuous execution over days. Integration to other tools (build environments, configuration control, reporting databases etc) means potential failure points that a right architecture can make more manageable.

Build a Test Support Library

When test scripts are being developed it quickly will become apparent that there is a lot of code repetition. Setup for tests often involves the same actions over and over again. So does validation of test results. If the test involves the UI then certain navigation paths repeat over and over again. All this needs to be made available through utility libraries to support the faster development of test scripts. Experience shows while some can be designed before regression testing starts - most will emerge during the development of test scripts. So a process and a supporting infrastructure to maintain support libraries and disseminating the knowledge of them to the test authors is critical.

Build a good Analytical Infrastructure

Automated testing is about running many tests fast. The cost of building an automated test is much higher than executing it manually – so to get you investment back you need to have a situation where you can run it often – such as right after each build, automated, regularly. That results in lots and lots of test results. When done well you can expect a few thousand test runs each day, day in day out. Once you do that it is apparent that you need to be able to slice and dice your test results. You want to be able to look for trends, look for patterns. Because tests are unattended you need to keep logs and configuration information for each test run that you may have to be able to access and scrutinize in a quick fashion.

Build a good Logging Capability

In comparison to a human doing manual testing automated tests are extremely dumb. The human tester has the ability to think "Mhh – that's strange" and then investigate. So when an automated test case fails the test log may not tell the whole story. Once you investigate a test failure there is probably a lot of valuable information that was not recorded because the test developer didn't realise it might be useful. There is the noise factor, test cases can fail not because the application has a defect but the environment does: time outs, defects in the test case script, defects in the test interface, the system it runs on – all these can cause failures and the test script may not be reporting sufficient information to discern this right away.
So you need to ensure enough information is logged – some of it automatically by the test execution infrastructure some by making it easy for the test writer to log information in a structured way. The log results need to be structured to be easily accessible – which must be integrated into the analytics infrastructure.

Build a Reporting Infrastructure

Apart from being able to analyze the test results you need to have a simple overall reporting infrastructure. After investing all that money (which often is more than anticipated) you need to demonstrate the value of the testing. You should be able to report the performance of the automated testing. Since the testing is automated, the reporting should well be automated as well. It needs to show how many test cases are being run, how many fail and trends in testing.

Build a Process to Maintain Automated Testing

Once the framework is in place you need to feed it with test scripts. Each test script is a comparatively simple program. But it still is subject to software development rules. You must have a process that is effective. You need to understand the status of each test case: is it under development, or is it in production (i.e. is it actively used in regression testing). Defects can be raised against test scripts. Test scripts must be written so that they are maintainable – that implies coding standards and code reviews.

Conclusion

I have just but scratched the surface of building an automated testing facility. But I hope it demonstrates that the project planning needs to recognize the effort and respond to it by providing it the right resources – otherwise you are setting yourself up for a waste of money. This is demonstrated in many projects where automated testing is either abandoned or remains outright anemic in nature.

How Much Is Enough?

"How much is enough?" is a common question heard by analysts, designers and testers. How much detail of requirements do you need, how much detail of design do you need, how much testing do you need?

Let me give one answer for the requirements space.  You will see that this philosophy can apply to the other areas as well.

A key question that needs to be asked first is "where in the SDLC are we talking about?". Unless the project is a classic waterfall project ,where all requirements are created in a single phase, requirements do "mature" through several stages.

To answer the question we can break down the concept of 'completeness' into three dimensions:

1. Scope: are all areas covered?
2. Detail:  are all intricacies and all aspects of the requirements sufficiently described?
3. Quality:  are these the right requirements and are they consistent?

Requirements are basically an unbounded area into which an endless effort can be poured into. The practitioner, never having unlimited resources, needs to plan on how to get the best result with what is available. So the question of how much is enough is essentially a question of how to balance effort in each of these three dimensions to deliver the best result.

To illustrate these concepts let's walk through a simplified project lifecycle:

Early Idea

At this point the project is just an idea in the minds of a person or a small group. The objective is to turn this idea into a proposal that is serious enough to get funds for developing a business case.

Scope: At this stage the most important category. The objective is to identify key features that illustrate the intent and effect of the project. It does ignore the majority of alternatives and exceptions that do not contribute to the understanding of what the project is about.

The scope needs to be outlined both via inclusion and exclusion (At this a third category called 'unknown' is also very useful).  The objective is to invite the target audience to help evolve what the project is about.

Detail:  very sketchy, mostly bullet points are used and in some instance a little of narrative such as user stories are helpful.

Quality: The requirements are trying to paint a picture for the readers to imagine.  Therefore consistency in this rather rough overview is very important to avoid confusion.

Proposal

At this point a business case is being built.  The objective is to understand the requirements well enough so resources, budgets and time lines can be estimated. This implies that size, complexity and risks are identifiable.

Depending on the engagement model (fixed cost vs. budget vs. agility) the reliability of the estimate will vary which affects the depth of requirements. Quite often a reliable forecast for the effort for the next phase and a ball park for the overall implementation is necessary.

Scope: The scope must now be well described. This is critical, because any change in scope needs to be managed from now on.

To clearly describe scope more formal approaches such as Use Cases for functional requirements, UI mockups, early business domain model and key state diagrams are useful.

All functional requirements and business domain objects should be identified (but not defined, see Detail).

Detail:  While the requirements now have structure, detail is focussed on the identification of requirements and less on the detail definition of requirements. That means that requirements elements are often identified by a name (and an ID) and an English language description to ensure common understanding in the audience.

Quality: Verification of having the right requirements is starting to become important. Reviews need to be held, however reviewers need to keep in mind the objective of the requirements at this state.

System Definition

This is the time when requirements need to be detailed and high level designs/architectures will solidify. From here on down stream consumers (designers and testers) require input to be able to do meaningful work. The end of this work provides requirements that are good enough to take design decisions, reliably estimate individual work packages and to plan and schedule the implementation.

Scope:  At this point planning of iterations often starts.  In the previous phase understanding of the system had been building to the point that it is possible to sensibly carve up the implementation into separate increments/iterations. Hence the requirements can be broken up and attacked in separate slices.

Detail: Requirements now need to be detailed to allow development of design. This is a tricky phase because diving into detail usually throws up even more detail questions - easily leading either into a morass of analysis paralysis or (usually) the subsequent abandonment of analysis and going on to implementation.  A good rule of thumb is to partition questions:

1. Show stopper questions for architects and designers: must be addressed here
2. Questions that can wait until development: mark them to be resolved when development happens (e.g. what is the default value for a parameter)
3. Changes to scope and requirement: flag them to the PM to determine if these are acceptable or need to be formally tracked as change requests

From this point on questions must be formally documented and tracked.

Quality: Requirements need to be validated to be consistent internally and with external documentation. Users need to verify that these requirements are the right requirements (with informal or formal sign off). At the end of this exercise requirements typically go into change control.

Implementation

If you think this is the end, think again. Design, implementation and testing will require more refinement of requirements. So plan for it. Also plan for changes in requirements, with each change request having a subset of requirements undergoing a similar maturation path as described above.

Conclusion

While this focussed on the requirements strand, the same strategy works for architecture, design and testing. A good SDLC recognizes this and coordinates the maturity of the deliverables across all streams, defining criteria to be able to assess what maturities deliverables have to have in various stages of the SDLC.

At the core of this approach lies the principle of not making decisions earlier than necessary. Following this principle this approach  avoids creating requirements/design or other artefact details too early which in turn reduces the risk of costly rework later down the track, making for a more adaptive approach to running a project.

(Ab)using Use Cases

I visit quite a few  companies and routinely ask if they use Use Cases. At a depressingly high rate I get an explanation along the lines of: “We tried to use it but it was too hard”, “It is actually no good because business rules can’t be documented adequately”, “They are too complicated”,  etc. etc.

The usual cause I see is that organisations try to make them the vehicle for all requirements. This leads to long, hard to read, use cases causing disenchantment and ultimately abandonment of this technique. This is a pity because there are few approaches so powerful in eliciting and structuring requirements.

So I went back and compiled a list of what I regard as the essential Must-Do’s of Use Cases.

DO #1: Employ Use Cases as an investigative tool
Use Cases are great to draw out requirements. They embody a conversational style that is easy for anyone to adopt. While working on the use case texts, it is common to have all
 No matter what your ultimate packaging of requirements is, the way use cases are patterned is the easiest to elicit requirements from users. The structure of a use case actually patterns the way we tend to think the very first time about a requirement.

When you workshop Use Cases you not only going to collect the actual scenario texts but also be able to identify user interfaces, business rules, business objects etc. Use cases are like a vacuum that suck in all the other requirements. That’s good. But be aware that all those other requirements need to move into the right place in your requirements model.
So use Use Cases to elicit the requirements and then later factor out what doesn’t really belong into the text of a Use Case (see below).

DO #2: Map use cases to business processes
Requirements creep is a significant problem, when users behave like kids in a candy store.  How do you justify the need for a use case? Mapping them to business processes is powerful, because it shows where the use case supports the business process – the ultimate justification for having the use case.

DO #3: Control the level of abstraction
Use cases can be applied at different levels of abstraction. The most common level is to describe the user interaction with the system for which requirements are being sought. It is however possible to use Use Cases to describe Business Processes (where multiple systems and even manual processes are being described) and conversely to describe processes internal to systems (which would be applicable during the design process).

To make Use Cases useful they need stick to a particular level of abstraction. If you write Use Cases to describe how a user expects to use a system for a particular objective, then the Use Case text should not describe the business process. That information would probably be better indicated in the description and then described in separate Use Cases that are at business process level.

Corollary: Group Use Cases for each level of abstraction – if you use documents, consider sticking them into separate documents.

DO #4: Refactor Use Cases
If Use Cases are the ultimate lint collector in the requirements space, then you need to clean them up. Revisit your Use Cases to see whether they represent a tight, functional, description of how a system is used. Often Use Cases have grown too complex – break them down into smaller, yet still functional, Use Cases. Remember that a Use Case still must be able to fulfil an objective that is fits into the conceptual world of your user.

If the scenario text contains other stuff like Business Rules, complex branches, UI specifics etc, then start pulling it out.  Stick them into – well here comes the next DO…

DO #5: Store different types of requirements in appropriate locations
Use Cases are really only good to describe how a system is to be used. Trying to make them a vehicle for everything is bound to fail and will give Use Cases a bad name.  Scenario text should therefore only contain the interaction between the user and the system. Anything else may be referred to but not described in the text.

Keep separate places for the following types of requirements:

• Business Rules
• User Interface Specifications
• External  Interface Specifications
• Business Objects (Class or Entity Relationship diagrams)
• A glossary of terminology

Each of these requirements needs to go into a separate template, but properly cross-referenced.

DO #6: Document questions, answers and assumptions
Sounds like a lot? Well it isn’t sufficient. Because Requirements are never definitive you need to have a space to document the questions that come up. Once the questions are answered, add them to the question. 

If you can’t get an answer (a not uncommon situation) the next best situation is to make an assumption. Assumptions equal risks. So document them well and flag them for what they are.

DO#7: Cross-reference
So you elicit your requirements with Use Cases and then extract the non-Use Case Requirements into other documents.  Now you need a good referencing system. There are some tools that do that but not many of them are actually easy to use. But help is at hand. Invent a naming scheme that clearly identifies all kinds of requirements. So when you Use Case text identifies a UI, then all you need in the text is the ID (eg UI-1020) and the title of the UI.

DO #8: Cross-validate
Once your requirements are stabilising you will need to make sure your requirements are consistent. Here are some questions:

• Do all scenarios reference User Interfaces (remember, Use Cases are typically about interactions between users and the system)?
• Alternatively are there External Interfaces referenced if the actor is another system
• Is the User Interface able to support the text in the scenario?
• Is there a Business Object that is able to carry the information displayed or entered in the user interface?
• Does the User Interface have the appropriate validations?
• Do the business objects capture the information required by the business rules?
• Do you have Use Cases that cover basic functions such as Creation, Retrieving, Viewing and Updating for each of your Business Objects?

Complexity and Enterprise Application Architectures

A client asked us to look at the complexity of their Enterprise Architecture and to come up with a way to measure it, or at least to measure the way it changes over time. Now there is no simple way of calculating a complexity number for an enterprise architecture. Complexity is a quite elusive property, with different meanings to different people. So for this purpose we looked at complexity to be a proxy for effort to maintain, build and manage the systems in the enterprise.
While this still doesn’t make it practical to arrive at an overall number there are a number of aspects that can be used to rate complexity in different categories. At the base of it is the sheer number of “things” that an architecture contains such as: 

• The amount of business processes
• The amount of business functions/products
• The amount of different kinds of data
• The amount of interfaces (to users and other systems)
  

Each of these give rise to complexity by their very nature. If, for example, the amount of business functions doubles then complexity (as we defined it above) would be expected to at least double as well. The minimum unavoidable rise in complexity is linear, directly proportional to the number business functions.
If the additional business functions introduce dependencies upon other systems, then complexity would rise in a non-linear fashion. In the extreme, an increase in size of business function could mean that all business functions have a dependency on each other. In that case the number of dependencies drives complexity up at a faster and faster rate. In mathematical terms the complexity is then proportional to the factorial of the number of business functions. This situation would represent the worst case – in reality any enterprise system would range somewhere between these two. If unmanaged, the complexity growth curve would move towards the worst case scenario. If managed through an appropriate application of enterprise architecture it can be moved closer towards the ideal.

So to rate complexity one has to look at duplication and dependency, e.g.

• How many times is the same kind of data managed?
• How many ways can data updates flow through the system?
• How many different ways is the same basic business process done (e.g. branch vs. internet self service)?
• How many updates to the system have to occur if a fundamental change (e.g. interest rates) occurs?
• How many system interconnects exist, how many of these are duplicated?
  

Enter the Enterprise Architecture
To try and rate each and every item would however be impractical – just about as impractical as it would be to measure the state of chaos in someone’s backyard shed. Because Enterprise Architecture tends to deal with big pictures and often looks at big changes it needs to be extremely practical. Enterprise Architecture initiatives sometimes get stuck very early because everything appears to be too hard, with every improvement opportunity being blocked because there is no room to move. In this case a good strategy is to start by identifying a list of mostly tactical improvement opportunities that will deliver both reduction of complexity and improvement of business function. The list’s purpose is to allow for  improvements that provide space for movement. Once this ‘space’ has been created roadblocks for larger and more strategic improvements are being removed.

Typical Case: Organisational Mergers
A merger between two organisations, if unmanaged, creates a large jump of complexity with sometimes no real improvement in business functions – if two small lenders were to merge, both organisations may have the same five basic types of home loans, but they could now be packaged as 10 different products with 10 different sets of systems to support them. If unmanaged, it puts the organisation onto a substantially faster growth curve in terms of complexity. In an ideally managed scenario, the systems would not only be merged but opportunities arising of leveraging the systems could give rise to new, previously not easy to achieve business functions.  This would move the company not only downwards in complexity but also forward in terms of its ability.

You are never done

What do Agile, XP and Process MeNtOR have in common? The idea that information get created and decisions get made when the time is right, not earlier.

The agile movement has an acronym for this: BDUF, which stands for “Big Design Up Front”.  You could extend this to BRUF (Big Requirements Up Front), BPUF etc. I think you get the drift.

Tom and Mary Poppendiek in their book “Lean Software Development” have discussed the Toyota Car Design process and why it is quicker and less expensive than that of their American counterparts. A key element in the success of Toyota is to avoid making decisions too early. While it is tempting to do make the decisions on issues as soon as the issue becomes apparent (it gives you a nice warm and fuzzy feeling), it often causes follow-on decisions to be made on the basis of the original decision. If later the original decision turns out to be wrong or no longer valid it results in having to un/re-do all the dependent decisions, an exercise that is usually quite time consuming & expensive.

Deciding not to do too much up-front too early is a good idea, but what constitutes ‘too early’ and when is the right time in a software development scenario? The answer is not straight-forward because each project has different dynamics and different dependencies.This is where good practitioners (project managers, requirement team leads, architects, test managers etc) shine by being able to make the right call.

In Process MeNtOR we decided to introduce the concepts of maturities for our deliverables (Process MeNtOR deliverables are the core management concept, it focuses on providing guidance on when and how to create these deliverables. That allows for powerful customization abilities of the project roadmaps but that is another story). On closer analysis of our deliverable we found that many deliverables have a natural maturation cycle. Let me give an example:

A common set of maturities are Initial, Core and Final.

INITIAL means the deliverable lays out the strategy of creating the deliverable, the why and the basic principles and approach to create it. If the deliverable were a software component, then Initial would be the Roles and Responsibilities and, in a test driven environment, the acceptance test case.

Initial is usually reviewed by a peer or a stakeholder representative who can provide guidance to the author for implementation. It also serves as a heads-up for others who will require it as an input for their work – allowing them to get started on their deliverables for an initial release.

CORE identifies that the deliverable, while not ready for production, is good enough to allow checking that it meets the key requirements. For most deliverables it is good enough to be used by others who require it as input for their work. It often has many questions to be answered or decisions still to be made but which are not (yet) show-stoppers for the authors of the subsequent deliverables. If the deliverable is the requirements model, then this would be the key input for the designer/developers to start moving the System Model towards a core release, without having to wait for all the little detail decisions to be completed.

FINAL is well, final. The deliverable is complete and won’t need changes (until the next iteration).

The maturities identify natural milestones in the deliverable. At these milestones it makes sense for other members of the team or the customer to get involved, either for review & guidance or as input for their own work.

Maturities help shorten the delivery time, because it allows downstream consumers of the deliverable to get started – eliminating wasted time having to wait for all little details to be completed.

Another advantage is that maturities identify outcomes not activities. In creating a software component the author is free to determine the actual selection and sequence of activities (design, code, test vs. code a bit, test, design, repeat) according to the actual circumstances.  This gives the author more freedom but also more responsibility on how to deliver what they commit themselves to.

Process MeNtOR describes maturities for each of its deliverables. A few have no real maturities because they always are only produced as final (e.g. a test case result). Some have more than three, but not too many as the maturities require to be easily grasped and understood by all involved.  Process MeNtOR provides deliverable specific descriptions of the maturity and guidance how to determine whether it has reached this maturity. That makes maturities a good tracking tool for progress which can be easily graphed and displayed to the team and the customer.

The following diagram is an example of a snapshot taken during a real project, tracking the maturities of Use Cases. Because Use Cases have more natural maturity milestones, we use the colors of the rainbow instead.

Mashing up MeNtOR

You might have seen a couple of links pop up at the side of my blog, pointing to microformats.org and  RDFa.info. I have been investigating and experimenting with the Semantic web recently. This was motivated by finding new ways of making Process MeNtOR more user friendly.

There is a big challenge for deploying a methodology. On one hand it is supposed to be a stand-alone, generic guidance on how to do something (in this case building software). On the other, when deploying it into an organisation, it needs to fit in with that organisation's way of guiding its members.

Often when I deployed Process MeNtOR I found web-sites that provide guidance on a number of existing processes, eg. how to do things like kick off projects. In other places there are tools, either in-house built ones or purchased ones, that imply processes (think governance tools, test management tools, configuration control tools etc). 

In those situation you'd then see people using the existing tool website and browsing Process MeNtOR in parallel to get the guidance of how it should be used within the corporate framework.

The usual response  to that situation is to look at modifying Process MeNtOR so that it refers to  existing  terminology or complementary practices. That's great, but modifying the Process MeNtOR website creates a maintenance problem - what do you do when you get that upgrade of Process MeNtOR ?

So while we are designing our Active MeNtOR server, I am looking at ways to add semantic guidance to our product by introducing microformats.org and RDFa.info. Then, instead of modifying the Process MeNtOR website,  our clients' tools can  refer to our website and  in an intelligent manner pull out whatever  aspects they require and  display it within their own tools .

Take the example where the client uses a resource management tools. In such a tool people can view their  task assignments using a web browser.  Using the semantic web extensions in our  server,  that browser window would display next to the task assignment all the relevant links into Process MeNtOR such as task detail descriptions, guidance and links to templates.

You have requirements

Recently we had an email discussion floating through the company. One of the questions was: “When the designer gets the requirements what questions should they ask to make sure the project is a success?”

Here are some of the comments that came back:

General

This step of any development process is all about gaining knowledge without detail level information of the solution. The key things we’re looking for are pitfalls, things that will bite us later on in the process. Every project is going to be subject to the classic forces: Scope/Requirements, Budget, Time. The information which can be gathered early in the process about the volatility of the requirements will partially dictate the choice of methodology. The budget and allowed time are usually better understood in terms of constraints.

State of the Requirements

How were the requirements gathered? Was a structured approach used? Could requirements change during the design phase?

Are the requirements reviewed and signed-off (an indication but no guarantee for the quality)

Are they clearly written, concise and consistent? Is there a structure to the requirements that allows the reader to find their way through and get an understanding for the scope and completeness? Or are they written in dense or a vague way that makes it hard to understand what is there

Check that all types of requirements included:

• Functional (maybe use cases)
• Data/object Models
• UI Models
• Business Rules
• Non Functional Requirements
• External Systems

Are the assumptions and outstanding questions documented (this is a good thing)? Are they show stoppers (this is a bad thing)?

Has an architectural been involved while the requirements were created?

Stakeholders - Who has say and sign-off, who has design input. Finding these people early is important, especially those that are of a combative nature.

Hard limits vs. Soft limits - Knowing which restrictions placed on a solution are set in stone and which are a bit flexible can dramatically alter the nature of a project

Environment - On a technological level, providing correct development and build environment can make a major difference to the rate of a project. Having a well defined and implemented development (and deployment) environment is definitely worth the investment. In political terms it is important to maintain two key aspects of the work environment: People must be able to offer up new ideas and improvements to existing practice to be socialized by the team(s); People must not be made scapegoats of – there should be an environment which allows and rewards experimentation.

Pending changes to other projects (substitution, impacts): if any of the systems you interface with has pending changes / work in progress that will impact on your project, you need to know up front so you don’t have to repeat work. The problem you then face is coordinating with all the other teams you work with, but if they don’t plan sufficiently ahead this may be impossible to avoid.

Available resources (staff, cash)

Time for research and to do POC work / spikes - We know from project experience that doing proof of concept work and trying out design ideas on a small-scale to validate them ahead of implementation can dramatically improve the quality of estimates and prevent a team from pursuing a design that later proves to be flawed.

Pre-existing commercial arrangements / alliances that will impact your project

--------

Thanks to AD, AB and JE for your contributions.

Process MeNtOR as a Prince2 Implementation

Prince2 is a generic approach to project management (PRINCE stands for Projects in Controlled Environment). Being generic it needs to use generic principles and approaches, which the practitioner then has to translate into specific activities and deliverable structures. Being more specific to Software Projects, Process MeNtOR has taken these generic principles and approaches and implemented them as concrete guidance that is specific to the problem at hand. This reduces the effort the project manager would have to spend on creating templates and procedures for a project.

Example: Setting up Project Controls
Prince 2 has a sub process called: ‘Setting up Project Controls’. Its purpose is described as:

“Establishes control points and reporting arrangements, based upon the project’s size, criticality, risks situation, the customers’ and suppliers’ control standards, and the diversity of stakeholders.”

This is a very good but also very generic principle and it requires the practitioner to develop an approach that fits this principle and is applicable for their project context. Prince2 provides some more detailed guidance:

• Agree the stage breakdown with the project board
• Agree the format of reports to the project board and stakeholders
• Agree the frequency of project board and stakeholder reports
• Establish reporting requirements from team(s) to the project manager
• Check that there are sufficient risk and business case monitoring activities in the plans

 To achieve this in a consistent manner requires standards to be defined, documented and agreed to by all parties. Its practioners need to make non-trivial judgements (what is ‘sufficient risk and business case monitoring’?) that require very good project management experience.

The effort for creating this is non-trivial and it makes sense to implement this in an organisation wide standard where the cost of creation can be spread over many projects.

So how does Process MeNtOR address this:

• It provides breakdowns in a number of roadmaps, with each roadmap addressing a typical project situation (Large Software Development, Fasttrack Software development, Software Procurement, etc.) .
• For each stage in a roadmap it defines the deliverables and deliverable content that is appropriate. This includes the reporting requirements.
• For the end of each stage it describes the process and conditions required to pass the control point.

It is important to note that Process MeNtOR uses deliverables and their maturity as its core control mechanism. If mandated deliverables do not meet their review requirements then a stage has not completed – which in turn can trigger what Prince2 calls an exception report. 

Conclusion

Process MeNtOR provides a specific implementation of many of Prince2’s generic guidance and principles, including processes, deliverable templates and quality criteria for control points. This specific implementation embodies the project experience of many practitioners that have contributed to Process MeNtOR over a time span of 14+ years. It has been put to test at Object Consulting in many projects and is being regularly reviewed and upgraded.

What to do when you don’t have funds to do your requirements up-front?

In my last blog I talked about how the Process MeNtOR requirements model provides the rigor needed to reliably determine the effort for a project. However, this approach being the big gun, it has its drawbacks. The cost for a complete requirements model consumes  a large chunk of the overall project budget, and you may not be in a position to have such funds approved before the business case.

We recently had such a situation with a client of ours, a bank. This client wanted to explore the implementation of a new e-business venture that would offer a specific part of its services to smaller banks for reselling. This would entail changes to business practices, making the service relevant for its new partners and subsequently changes to a core application in the bank. Early estimates for this project ranged in the A$ 5-10 Million.

We were asked to provide a more reliable estimate but had only limited budget available. As input for this estimate we decided to produce a Process MeNtOR Requirements Definition – which is similar to a Requirements Model in structure but reduces the depth of the information. Process MeNtOR provides a template and guidance on how to produce the template. You use the standard requirements modeling activities out of Process MeNtOR to produce the content ; the guidance in the template will tell you  to what depth you need to go.

Over four weeks we ran a total of four workshops with different participants and evaluated existing process documentation.  The document was reviewed with management twice, once about midway and once for the final document. The total effort ran to 8 days on our side and produced a document of forty pages, covering essential business processes, skinny use cases, a business domain model, user interaction models, quantitative statements, assumptions and known issues. This document was used by the architect in conjunction with existing technical documentation to validate and narrow the original development budget.

Now the client has used this to justify the budget for doing a full requirements model – answering the original question.

Requirements and your business case

 Dealing with clients I see a particular pattern in the early stages of a project. Having been stung in the past by cost blow-outs many organizations now demand a business case to justify the funding of a project. Key part of that business case is a reliable cost estimate. To provide a reliable cost estimate is dependent upon the quality of the requirements model - the more detailed the better the estimate. Or vice versa - the less detailed the requirements the less reliable the cost estimate becomes.

The project sponsors who want to start their projects find themselves in a quandary: they know that to achieve a reliable cost estimate they need the requirements - but that would cost a significant part of the project budget which they want to get approved.

This situation leads to a practice of putting up a strawman budget. Because everybody knows the estimated budget is quite arbitrary, the steering committee that approves funding for the project is often led to make equally arbitrary budget cuts to it.  In the end the project goes ahead and the project manager is faced with the need of making ends actually meet, quite often by negotiating feature reductions with the users.

At Object Consulting, where we use Process MeNtOR as our process to deliver fix-price software projects, we sell projects in two parts. Part one is the System Definition Phase where we gather the requirements. This phase is sold to our client on a time boxed/effort capped basis. Once the requirements have been produced we then quote the development part as a fix price project. The process allows us to be quite precise what activities go into this first phase. Having tuned it over the years we produce in this phase not only requirements but also a high level architecture and determine the approach that is required to identify not only effort but also be able to quantify the risks to an acceptable level.

This is all good but how can we know what it takes to define all the requirements you may ask. The short answer is we don't. We don't know how long it will take to do all of the requirements. We do know however to a quite reasonable reliability how long it takes to get enough requirements that allow us to do a reliable estimate. There is a big difference between the two. 

The former  (all requirements) is basically an open ended affair. The more detail you create the more questions appear. The more questions appear the longer it takes.  If you were to chart productivity in business analysis over time you get a sharp bulge at the start and then a steady decline over time. Take that and add to it the fact that requirements have a natural tendency to change over time; after a sufficient amount of time your speed of requirements definition has become less than the rate of change of requirements - you have entered the domain of Analysis Paralysis.

What we do instead is to focus on producing 'just enough requirements'.  In this context 'Just enough' means the amount of requirements that we need to have to determine a build approach and to be able to size it for costs. To do 'Just enough' requirements means that we need to make sure we cover the whole scope of the problem domain but dive in only so deep. The 'How deep' is part of the tuning we have applied to Process MeNtOR over time.  It gives the business analyst guidance when to stop refining use cases and to move on to the next one. It also allows us to work iteratively: Identify several use cases, then refine them later. Repeat for the next set of use cases and so on.

On top of this sits a maturity matrix. It tracks how far individual use cases have matured. So even with larger numbers of use cases in a project ( hundreds) we do not loose track of the big picture. A team of business analysts can on a regular basis review its efforts and focus it so that the overall target of 'just enough requirements' is being met.

This approach allows us to quote a reliable time box to create 'just enough' requirements. How big the time box will be is driven by a scope description and the targeted reliability of the estimate. 

For a fix cost project this phase can still be a major chunk of the overall project budget.  So what happens business is not willing to fund this without a definitive business case?

 Find out in my next blog.

Can your users read your requirements?

Organizations often struggle with getting good requirements. Some organizations invest in their business analysts  and develop standards, templates and maybe even training on how to create requirements.

With Process MeNtOR we provide a very powerful approach to developing requirements.  Users who implement the approach see much better requirements being produced. But then a new problem becomes visible: the various audiences of these documents (business users, designers and testers) start having problems making full use of these documents. When analysing this issue further we saw that it was not so much of the documentation being faulty or too complex. What was missing was that most people do not really have an understanding how a concise and consistent requirements model  can be structured and how it needs to be applied for their domain. 
For instance

• A subject matter expert needs to be able to find those parts that are relevant to them to validate the requirements represent what is needed.
• A requirements tester needs to be able to cross reference the documentation to ensure consistency and correctness is preserved
• A sponsor or major stakeholder on the business side needs to be able to see the requirements summary and be able to determine that the scope matches to what their business case says
• A project manager needs to understand size, complexity and risk expressed in the requirements model
• An architect needs to be able to find the fundamental business stories and expectations to respond with an appropriate solution model
• A designer needs to be able to understand the use cases and their operational context to develop an appropriate system design
• A tester needs to be able to identify functional areas that can be mapped into test cases

All these users will read he same material selectively or with varying focus and look at different aspects. Writing different documents for each audience is not feasible for anything but the most trivial requirements documents.

With Process MeNtOR, we developed several strategies to address this situation.  For a large project  we developed document maps ("You are here"diagrams) and improved document introductions to guide the readers to the points of their interests.

But the best response we got was to develop a series of short courses called "Reading Requirements". Customized for the different audiences these courses focus on what the reader  "needs to get out of" the requirements model. For a business user this may mean a session on effective reviewing of the requirements for sign-off. For a designer it focuses on how to translate Use Cases into system scenarios to derive component responsibilities which, in turn, result into a component design model.

The result of these courses has been positive because it breaks down barriers of having to deal with a new and not understood set of documentation. Requirements for any non-trivial system are inherently complex and the project risks are typically in those bits that are not obvious. Giving the readers of the requirements model the ability to find what they need to know improves the value of the requirements and reduces the overall risk of the project.