Journal: opinions, theories, tutorials, workshops.

Journal

Opinions, theories, tutorials, workshops. Some long reads here.

What is software production management?

Whenever I raise the subject of software production management in a peer discussion, the conversation almost invariably veers off to project management: why agile is better than waterfall, why Scrum needs XP, how to shanghai a product owner etc. Maybe one day I'll climb into that tar pit, but not today.

Googling "software production management" returns 193,000 results, while "software project management" gets 2,950,000 results. That's a 15-fold increase, just for changing two letters and dropping another three. But aren't these two terms talking about essentially the same thing? Actually, I think not.

Wikipedia says "software project management is the art and science of planning and leading software projects". But it says nothing about "software production management". It even redirects the simple term "production management" to "product management", which in my view is not at all the same thing.

Product management is about things, while production management is about a process. A product manager is concerned with the sales cycle of her products, but a production manager is concerned with the cost and efficiency of the production process. Product management is largely a marketing discipline, whereas production management is principally an engineering discipline.

Since Wikipedia has deserted me, here's my definition:

Software production management is about managing a continuous stream of software projects of various sizes for a number of clients, while balancing a changing mix of skills and resources against shifting demand.

Notice there are lots of variables and no constants in my definition. That's right, production management has no beginning and no end (other than the span of existence of the production facility). It just keeps on coming day after day. There are always:

  • Conflicting priorities to be negotiated
  • Competing schedules
  • Skills and resource bottlenecks in some areas, as well as excess capacity in other areas
  • Pressure on budgets, requirements and features, from multiple sources that often do not have a common forum or even an interest in negotiating their competing needs

A road block in one project can affect resource availability and progress in other, perhaps unrelated projects. An early finish for one task or project (admittedly rare) can require you to start other projects early, just so you can soak up newly freed resources; but this in itself creates extra scheduling work and may also increase demands on key resources that are already committed elsewhere.

Every day you have to start again from zero, checking what's due out today, what's in progress and what state it is in, and what's new in this morning: not just the feature set, but also any changes in priorities, budget, schedule, resource or equipment availability. Every change ripples through, and all those ripples must be managed, coordinated and reported.

The foregoing is a thumbnail of the software development milieu on which I cut my teeth in IT consulting, an environment which is guaranteed to develop your tolerance for failure. Over the years, this has not changed appreciably. In our target SMB market segment, this continues to be the dominant model of software production management.

You might think that the software production process I've described actually falls under the heading of Program Management, as implemented in the PRINCE2 Managing Successful Programmes discipline. This is an approach which involves a Program Manager in managing and coordinating a very large program of work that has of necessity been broken down to a number of related major projects. Again, I would disagree with this view. I think there are two major differences:

  • Even a large program of work has a specific, defined objective. That objective may be much larger than a single project team can realistically achieve, but it will still be a specified outcome. By contrast, a stream of production work has no specific objective, unless you accept as an objective an injunction to "do whatever it takes to deliver the best achievable products for these projects given the prevailing resources and constraints".
  • The team that delivers a program of work is typically controlled by a Program Manager, who manages a team of Project Managers, organised within a Project Office. However, the burden of production management typically falls on one manager. The production manager may or may not have assistants, but it is unusual for a production manager to be able to delegate accountability for the integral elements of the production activity to her assistants.

You may also think that this process is actually Continuous Delivery or Continuous Integration (CI). However, Continuous Integration is a discipline which focuses on integrating subsystems and components for production deployment on an ongoing basis. So CI concentrates on the back end of the production process, and its scope is typically limited to a specific product; whereas software production management concerns the entire development process, and it extends across a continuous flow of concurrent products.

A software production management system

So what's the solution? How do we maintain the best achievable delivery of products in priority order, that meet agreed quality standards (quality is always non-negotiable), while also remaining within tolerable reach of deadlines and budgets; and while each of these requirements (except quality) is subject to change at the drop of a hat?

Well, I have succeeded in doing this for many years. I have found that there is a system and a method to doing it, but not what you might call a methodology. I may write formally about this subject at some stage, but I doubt that I will be able to reduce it to a manifesto like Schwaber and Beedle's Agile Development with Scrum. In any case, I'm a pragmatic guy and theory interests me only to the extent that it has useful applications, so you won't get any neat prescriptions or recipes from me. Not today, anyway.

But I can describe my method in broad terms.

First, the focus of production management is work packages, not projects.

I am certified in both the PRINCE2 and Scrum project management disciplines. PRINCE2 is a comprehensive method that is concerned with how projects are conducted and controlled, but it is not at all concerned with the 'specialist' work that is done to produce the products, i.e. the actual deliverables. It describes project management activities in great detail and provides samples of a large number of forms to record and report project activities, but it says nothing about how the products of the project are built. It deals with management processes that deliver no tangible value to customers, and apart from the project impact of work packages it neglects all the productive activity that delivers tangible value to customers.

Scrum is a project and process management method that is specific to software development. Its thesis is broadly to strip management overhead away from the software development task, to recognise that we cannot accurately predict the course of a software development effort beyond the short term, to focus our efforts on dealing with work that is immediate and current, and to work in short cycles to minimise the impact of changes in requirements, resources and the technical shape of the products. This is a great improvement on PRINCE2's overarching umbrella of management, but it is still largely about the organisation and conduct of the software development work rather than the substance of it. The detail of actual product development is left to the imagination and ingenuity of the developers. To an extent this is unavoidable, but it also means that when we aspire to a consistently high standard of quality our best strategy is simply to hire really good developers.

Perhaps these criticisms seem harsh. Ask yourself, when you buy a book, a drink or a steak, what value do you receive? Is it the management input to that product or is it the product itself? When you finish the steak, do you often pause in appreciation of the management input to your experience?

And yet the management input must be there, or the product might be inferior in some significant ways, it might not have been made available to you, or it might not have been produced in the first place. Let's recognise that, but let's also keep our perspective about where the real value lies. It lies in the skills and labour of the producer. Management is only an enabler for the production of goods and services.

The valuable production activity of projects is relegated to black holes labelled 'work packages' somewhere towards the bottom of the typical PRINCE2 project process diagram. Software production management is concerned with what goes on in the work packages of software projects: the process of producing software that delivers value. Since conventional project methodology provides little guidance in this area, how are we to manage this activity, particularly when many customers, projects and resources are colliding in a constantly shifting mix?

Over the years I have evolved a system and a method to manage IT work packages in the context of formal projects (either PRINCE2 or agile). The system is named PPS (Psicom Production Stream). It provides facilities to build a model that encompasses:

  • Customers and their stakeholders, project boards and supplier teams
  • Formal descriptions of the projects and their structures, controls and organisation, including any required controls of a mandated methodology
  • The products and features to be delivered, right down to the components and elements and their configuration
  • All resources and their skills, relationships and availability. One by-product of this data collection is a skills inventory. This is a resource which is valuable in its own right, but I have found that it is rarely available in development organisations.
  • A formally defined, enforceable, work flow that can be customised and tuned to fit site or customer scale. Occasionally there may be more than one defined work flow in the model; for instance, if there is infrastructure work running in parallel with application development, then those two types of work packages may require different work flows. In the case of software development work, the work flow process will be some customised variation of an SDLC.
  • Finally, the work packages: all the jobs to be performed to deliver all the projects. There may be any number of work packages for any of the various phases or features of any project. Each work package must follow a predictable work flow process, as above.

Once enough of the model is built, it can be set in motion:

  • Jobs are created or revised as conditions change
  • Resources are assigned to jobs or moved between them
  • Jobs get done and the resources record what they did and their actual times. It is essential for resources to record their actual times accurately against jobs every day. PPS allows any form of documentation or media to be attached to any project activity, as well as most other resource or object definitions.
  • As each stage of each job is completed (i.e. the formal exit criteria for each stage are met), PPS enforces the progression of the task to the next specialist team in the work flow process, or a chosen individual in that team.
  • As each job progresses through its work flow, the stage activities float up through each resource's prioritised work load lists. No team member is ever in doubt about what to do next.
  • The PM adds or changes any of the objects in the system as conditions change, watches PPS measure and report status and progress, and she takes action when the data indicates an emerging problem.

Project managers tasked with running continuous production teams without a comprehensive production management system will usually end up gathering some of the above information on a set of spreadsheets (or worse, MS Word documents). Often customer management will impose reporting requirements, and the project manager will end up limiting her production management information to that which is required to produce the management reports. This piecemeal approach will produce mediocre production performance. The weaknesses of this ad hoc approach to the problem are:

  • Typical production management spreadsheets do not capture complete information about the production process, either in terms of the range of information content or the currency and timeliness of it
  • The spreadsheets are almost never integrated and dynamic
  • Important production documents are stored apart from the production management information and are often organised in a different set of categories so it becomes very difficult to separate the wood from the trees
  • Storing any kind of dynamic information in files in some kind of directory structure obscures the information structure and relationships and inhibits the dynamic nature of the subject matter
  • Using management reporting requirements as a yardstick to manage the production process is simply inadequate, because that does not expose production problems, it merely summarises the symptoms

By contrast, PPS is designed to be comprehensive, integrated, coherent, inclusive and dynamic. PPS surfaces all objects and activity as a multi-dimensional set of views, rather like a mutli-dimensional OLAP cube. The views implement and integrate all the model dimensions of the combined activity:

  • Customers
  • Projects
  • Jobs
  • Resources
  • Features
  • Skills
  • Priorities
  • Time periods
  • Status
  • Health

Most views show actual effort against estimates, together with a summary history of all activity on the job, and visual cues for the status and health of each job. Here is a snapshot of the dashboard for a large program of work:

PPS tasks view by project

In most cases, it takes only minutes for the PM to assess the health of the combined activity or any part of it: customer portfolio, project, resource, work package or product. Using PPS, we can usually predict that an overrun is likely to occur in a work package, project, phase or feature by about the half-way mark of the original estimate for that work unit. This is a much earlier warning than can be achieved by any other method I know of.

Once we have gathered a reasonable volume of actual performance data at a site, PPS will provide enough historical benchmark data to support project estimates with a confidence level of plus or minus 5%. It usually takes about 15 months of continuous use of PPS at a site to gather sufficient benchmark data to achieve this level of project performance, but once it is done, about 85% of projects will be delivered within 5% of the estimated effort available at the proposal stage of the project. This is a level of project estimation precision that is much better than the typical standard for acceptable project performance.

One point bears repeating: it is absolutely critical for resources to accurately record their actual times, and associate their effort with the appropriate job and activity. If actual times are not recorded every day, PPS will not produce significantly better project performance than any other ad hoc work management approach.

This is the area where we meet most resistance. Many organisations hesitate to ask their professional or back-office staff to record the use of their time, even though they may have time clocks in the factory for their production crews. It is pointless to waste energy exploring the sources of this reluctance, or even pointing out the implied double standards. It is simply human nature.

When I'm leading a program or a team and I have a mandate to use PPS for production management, I do not ask the customer's management to implement time reporting discipline for the sake of the program; I simply make time recording a non-negotiable part of the program or team charter, and I enforce it every day. After about six months to a year of experience with PPS, the team's performance speaks for itself, and there will be no further significant resistance. In fact the team usually become evangelists for the new regime, because they now have a record of achievement, and solid evidence of their work performance. This in itself is strong motivation and reinforcement for more transparent and accountable work practices.

Production management systems that provide similar services to PPS may often be found in ERP systems or MRP II systems. Such systems are oriented to manufacturing operations, but I have yet to find an equivalent production management system to control continuous software production; so over the years I wrote my own. Currently PPS is in its sixth iteration, and it is implemented as a Lotus Domino 8.5.x application that runs either in a Lotus Notes client or a web browser. My immediate plans for it are to redevelop it as a LAMP application, and to open source it if appropriate.

If you have any comments, arguments or interest in this subject or in the PPS product, please feel free to comment on this article or message me about your views.