Steps in Problem Analysis for Auditors

In audit, problems can be of two generic types: (1) problems with the adequacy of performance, and (2) problems with the adequacy of processes. Problems with the adequacy of performance are the concern in this course. Here, the root cause is often found in processes and controls, to include money, staff, equipment, methods, (policies, and/or procedures, and/or actual practices), and outputs.

Example: Problem with Performance Independent test of body armor purchased by the Army found that the armor did not meet specifications. As a consequence, it will not protect soldiers as intended.

Example: Problem with Processes Army staff, with management approval, were purchasing body armor without going through the testing required by Army policy. Testing is a universally accepted method for assuring things work as intended. As a consequence of not testing, the Army has no assurance that the body armor meets the required standards and will protect soldiers as intended.

[Note: These are created examples for illustration.]

The classic steps in problem analysis discussed here are generic. They apply, with some variation, to managers as well as auditors.

• Step 1. Identify the Problem
• Step 2. Define the Problem
• Step 3. Assess Significance
• Step 4. Identify Causes
• Step 5. Identify Potential Solutions
• Step 6. Select Best Solution
• Step 7. Develop an Implementation Plan*
• Step 8. Implement the Plan*
• Step 9. Monitor Post-Implementation Results

* Not directly applicable to audit.

Step 1: Identify the Problem

Identifying the existence of a problem with performance involves comparing actual performance with a standard or norm of desired performance. A negative difference indicates a potential problem. The magnitude of the difference indicates the severity of the problem. This step has three tasks.

1. Measure actual performance over some period. (Auditors may use entity measures or create measures.) Usually, it is important to know if performance is increasing, decreasing, or staying the same. Usually, performance will vary within some time period. Thus, we need to measure performance over a period short enough to reveal the variance and long enough to show the true long-term trend (e.g., five years). Useful tools for analyzing trends, performance variance, and the extent of a problem are the Run Chart, Control Chart, Radar Graph, and Histogram.
2. Choose or develop a norm to represent the desired performance level. This might be an entity goal, a decreed requirement, an industry standard, or a level determined by benchmarking.
3. Compare measured performance with the desired performance norm. A negative difference indicates that a problem exists.

In auditing, the condition element includes both measured performance and the variance.

Example: Agency janitorial staff were cleaning about 20,000 square feet per janitor per 8-hour shift, 10,000 less than the industrial standard of 30,000 square feet per janitor per 8-hour shift.

Step 2: Define the Problem

Defining the nature and extent of a problem involves two tasks.

Task One

Analyze the identified problem to determine if it has a history and is limited in scope (e.g., to particular customers, specific transactions or items, to specific locations, to particular time periods). For example, if the problem is that bills are being paid late, are all bills paid late? Bills of a certain value (e.g., under $100)? Bills of a particular type (e.g., phone bills)?

Task Two

Once a problem, its history, and any component parts are established, it is helpful to create a meaningful statement of the problem. The statement should not include any “implied causes” or “implied solutions.”

A description of the problem should answer five questions:

• Subject: Who and/or what is involved?
• Metric: What performance aspect(s) are involved?
• Scope: Location: Does the problem have component parts? Is it limited to particular customers, specific transactions or items, certain time periods, etc.?
• Location: Where is the problem occurring? Does the magnitude differ by location (e.g., by performing office)?
• History: How long has it been a problem? Is it getting better? Staying the same? Getting worse?

A focus on these dimensions will put boundaries on the problem. The more precision there is in the problem statement, the more precise problem-solving will be. A problem might be described as follows:

"The average score across the County School System for the 5th-grade standardized tests has decreased by 4% per year for the past three years. Scores of students in the top 10 percent have not changed; the decrease has taken place for students in the lower 90 percent. Over the past year, Agency Z paid 20% of its bills late. Of those, 80% were phone bills."

Think of the Desired Result

In creating a problem statement, it can be helpful (some say essential) to think of what the situation will look like when the problem doesn’t exist, when measured performance meets expected performance.

• What is the desired result?
• What are the minimum goals?
• What are the boundary conditions that must be satisfied?

Benchmarking with a “best of class” entity can indicate what is possible.

For our school learning and bill payment problems, the desired result might be:

The average 5th grade score on the standardized test for students in the lower 90 percentile will increase 10% by the by the end of the 2019–2020 school year.

The percent of Agency Z’s phone bills paid late will decrease to 10% by the end of fiscal year 2018.

Carefully defining a problem is a critical step in the problem-solving process. Unless we are clear on the exact nature of the problem, we may not produce an effective solution and may come up with superficial proposed solutions. For example, if we know that the number of bills paid late is increasing, a solution that the agency needs more staff is a superficial solution. 

Step 3: Assess Significance

The third step is to estimate the consequences resulting from the problem (effect in audit terms). Can the problem pass the “So what” test? In other words, does the problem warrant correction in the opinion of those who will be responsible for implementing the solution? Because of resource constraints, organizations (and auditors) generally choose only to examine problems that have substantive negative consequences, or for which a specific performance level is critical.

Step 4: Identify Causes

If our intent is to solve the problem, the next step is to identify the root cause(s). The usual tasks are:

Task One

Identify the potential cause(s). This can be an idea-generating effort or a data-gathering effort. Useful analysis tools are the Sequence Flow Chart and the Cause and Effect diagram, coupled with the brainstorming and brain-writing idea-generation tools, and possibly process benchmarking.

Task Two

Determine the most likely cause(s). These will be the causes to validate, usually, but not necessarily, with additional data. Useful analysis tools are the Sequence Flow Chart and 5-Whys Method.

Task Three

Validate the actual causes, from those selected and any others discovered in doing validation work. In some cases, it may be possible to determine the root causes using existing knowledge, like from a focus group. The Sequence Flow Chart and Scatter Diagram are useful analysis tools.

Step 5: Identify Potential Solutions

In this step, the challenge is to generate a list of possible solutions and determine the best. Here, creativity can be as important as data.3 Brainstorming and brain-writing are useful tools for generating creative alternative solutions. Benchmarking can be a useful tool for identifying potential solutions with data.

Step 6: Select Best Solution

There are several considerations in evaluating alternative solutions, ranging from ease of implementation to cost and benefit. The following are common considerations:

• Ease of implementation (e.g., how formidable)
• Resource limitations (e.g., how able—money, staff skills, management tenure, and interest)
• Time to implement (how soon will benefits be realized)
• Risk (how sure that benefits will be realized)
• Cost vs. benefit (how much return on investment)

The task is to evaluate each solution against the factors listed above and then choose the best solution appropriate to the circumstances. Auditors would recommend for implementation what they chose as the best solution. Force Field can be applied here.

Note: Auditors might not, in every case, explicitly take all the considerations into account.

Step 7: Develop an Implementation Plan

The task in this step is for management to develop a plan for implementing the solution or solutions they chose or were recommended by the auditor. No management commitment to a solution will exist until carrying it out in specific steps has become someone’s work assignment and responsibility. Until then, it is only a good idea.

Auditors might review management’s plan, particularly in follow-up on prior recommendations.

Two sub-steps for management are to divide the implementation plan for each solution into sequential tasks or activities and develop contingency plans. For each implementation task or activity, the plan should designate the:

• responsible person or group
• begin date and end date
• estimated hours
• cost

Step 8: Implement the Plan

The task for management here is to put the plan into action. This can often be the most difficult step. For auditors, there is one potential task.

That task is to track the implementation of the action plan and evaluate progress. The purpose is to determine whether or not specific tasks are being done as planned and on schedule. It gives an early alert to any implementation problems.

Step 9: Monitor Post-Implementation Results

The task for management is to create an information feedback and monitoring system using measures to provide continuous testing of actual results. It is this step of the problem-solving process that makes the structure a “closed loop.” Both management and the auditor can obtain feedback and monitoring through reports, figures, and studies. However, there is no substitute for going out and looking.

For auditors, there are two potential tasks:

Task One

Obtain feedback on actual performance relative to expectations following implementation. The purpose is to determine (measure) if the desired level of performance has been accomplished.

Task Two

Analyze the benefits attributable to the solution. First, determine/measure the impact (benefit) attributable to the solution. Second, compare the performance being achieved with what performance would likely have been without the solution. If appropriate, one might make a Cost-Benefit analysis. (Auditor may have to construct measures.)