White Paper - Mine Planners Lie With Numbers [PDF]

Zitiervorschau

MINE PLANNERS LIE WITH NUMBERS

By Dr Graham Lumley BE(Min)Hons, MBA, DBA, FAUSIMM(CP), MMICA, MAICD, RPEQ

30 November 2011

Abstract The most significant risk in developing a mine is that planner’s forecasts are not met. Cost and time allowances are rarely met and returns on investment are lower than predicted in 80-90% of developments. A primary input of this is budgeted output of the proposed equipment is not achieved. Three causes are proposed for this; technical deficiencies in the planning process; planner’s optimism and strategic misrepresentation (deliberate deception). Because the company’s balance sheet erodes every day they operate there is pressure on the highest levels of our mining companies to convert deposits to mines. Planners / consultants also have a vested interest in projects proceeding through the stages of feasibility studies. It is hardly surprising that in-house planners and consultants make forecasts which produce a result sufficient to justify the board approving the next stage of the development. This industry in Australia uses the Valmin Code (2005) and the JORC Code (2004). The ASX also has their own rules for listing. Each includes great detail on resource definition. However, they all pay scant regard (eg. The Valmin Code includes four lines in a 20 page document) to equipment performance; which is one of the primary drivers of the economics of a project, (converting a resource to a reserve). Mining companies have tight guidelines on the resource but clearly, this is not the only potential area of error? Neither The Valmin nor JORC Codes are protecting investors. They have simply shifted the source of error. The error (deliberate or not) is explained and demonstrated through six case studies. A better approach is recommended which includes better forecasting of equipment rates through benchmarking against industry standards and more accountability. Not so many mines will be developed but investors and shareholders will be better protected.

For more information; Graham Lumley +61 412787920 [email protected]

Introduction The first recorded questioning of the ethics in mining or project development was penned in 1556 by Agricola. He wrote, “A prudent owner, before he buys shares, ought to go to the mine and carefully examine the nature of the vein, for it is very important that he should be on his guard lest fraudulent sellers of shares should deceive him,” (Bullock, 2011,p78). 310 years later and it seemed nothing had changed. Mark Twain was reported (unconfirmed) to have told an assembled group of miners at Red Dog, California, in 1866, that “a mine is a hole in the ground owned by a liar,” (Bullock, 2011,p78) The alleged comments were targeted at US gold mining interests and the speculation which followed its discovery in 1849. Blainey (1963) makes it very clear that the Australian mining industry has a questionable heritage in the areas of honesty and integrity. He creates the picture of a burgeoning industry which needed an increasing number of speculators and speculative money to grow. To attract those speculative dollars directors, brokers (some of which are still listed as some of our most well-respected broking houses), bankers, politicians, etc. needed volatility in share prices. They got that volatility occasionally through the discovery of rich reefs but more often through a range of less ethical means. So in 2012, 166 years after the genesis of the Australian mining industry in the copper-rich hills out from Adelaide and the subsequent discovery of gold at Ophir in NSW in 1851, the question is posed; has anything really changed? Is the need for money still driving less than ethical approaches to mine developments? It seems that Australia is not the only country and mining not the only industry grappling with this issue. Flyvbjerg (2007) makes it clear that this problem transcends industries and countries. When referring to mega infrastructure projects he quotes Wachs (1989), "planners lie with numbers." He states that planners are busy not with getting forecasts right and following the appropriate Codes of Ethics but with getting projects funded and built. According to Wachs (1989), accurate forecasts are often not an effective means for achieving this objective. Flyvberg (2007, p.6) supports this notion when he states, “….accurate forecasts may be counterproductive, whereas biased forecasts may be effective in competing for funds and securing the go-ahead.” While agreeing with the summation of outcomes, Merrow (2011) believes that project failure is usually laid out early in a project's development through the basic business strategy and timetable for the project. Not aligning planning properly and developing the basic technical data and analytics sufficiently leads to not achieving forecast ROI. Merrow, Wachs and Flyvbjerg all point to the same need. Whether poor outcomes are caused by error, unintentional self-deception or deliberate deception, outcomes can be improved through structured definition and disclosure in each stage in the development process.

For more information; Graham Lumley +61 412787920 [email protected]

In the 21st century, like their counterparts in the second half of the 19th century, many shareholders and financiers of our mining companies are apparently not overly interested in the underlying resource nor the mine; they are speculating in the volatility. To maintain volatility these companies must keep a flow of “development” happening. From resource definition to upgrades to reserve calculations to conceptual mine studies to prefeasibility studies to detailed feasibility studies to engineering studies, the flow of progress must not be interrupted to maintain interest and volatility in the underlying company value. To maintain a constant flow of funding and go-aheads to the next stage or interest in being taken over by a big company it is essential that positive outcomes come at each stage. It is proposed that this encourages low cost inputs and high benefit outcomes in the mining industry in project development planning. Having said that, some of our miners do want openness, honestly, accountability and accuracy in their mine developments to predict outcomes. So the industry is left with two groups of miners; the profit-focussed, genuine businesses interested in creating value through extracting value from the ground and the speculative business interested in extracting value from people and companies with money. Both groups of miners need the money to keep flowing. Mining companies are producing a large number of projects which are not yielding the potential ROR predicted by feasibility studies (once changes in commodity prices are factored in). Bullock (2011, p81) summarises the reasons (provided by a range of authors) why mining developments are not reaching their projected ROR into 16 areas. This paper will provide insights on two of the areas which have arguably the most impact. 

Overoptimistic mine design or productivity.



Overoptimistic mine development schedule and start-up (learning curve) time.

This paper develops the case for a standard to be applied to the use of equipment production rates in mine plans and a system for accountability to be applied to mine developments and that this be incorporated into the Valmin and JORC Codes and should be embraced by the ASX in their disclosure rules for Mining Companies.

Mine Planning Process To take a development from a resource to a reserve and finally a mine, three levels of feasibility study (plus further engineering studies) are often required (Bullock, 2011). Each stage requires a different level of detail and each requires the mine planner to make a range of assumptions around optimal output levels and rates to achieve those levels. The average of 20 authors presented by Bullock (2011) is for the first study to have an accuracy of +/- 35%; the second study +/-22% and the third to have +/-14%. There is however, no prescribed engineering approach to move from one level to the next. Further, the author For more information; Graham Lumley +61 412787920 [email protected]

questions how a reserve can be determined without accurate inputs into determining whether any particular block is economic? When 80-90% of estimates of equipment production rates are more than 20% over what is actually achieved in the field (Lumley (2007), there is clearly no way the levels of accuracy in the overall plan can be guaranteed both in output and ROI. Consequently, 80-90% of reserve calculations are not supported by the engineering inputs being used to calculate them and should still be called a resource.

This identifies a key problem in the mine planning process. There is no prescribed process nor standard to change a resource to a reserve. The following is the definition of a Reserve (JORC Code, 2005, Clause 28)

“An ‘Ore Reserve’ is the economically mineable part of a Measured and/or Indicated Mineral Resource. It includes diluting materials and allowances for losses, which may occur when the material is mined. Appropriate assessments and studies have been carried out, and include consideration of and modification by realistically assumed mining, metallurgical, economic, marketing, legal, environmental, social and governmental factors. These assessments demonstrate at the time of reporting that extraction could reasonably be justified. Ore Reserves are sub-divided in order of increasing confidence into Probable Ore Reserves and Proved Ore Reserves”.

Any optimisation algorithm (eg. Lerchs-Grossman) will provide no help in the determination of economic reserves nor cutoff grades nor the final shape of the mine if inaccurate inputs are provided on equipment rates. For example, the economics of a block of ore might change dramatically if eight trucks are needed with a loader rather than six. Similarly, if your Liebherr 996 loader is scheduled to move 32 Mt per annum (actual best practice for this machine in 2010 – GBI, 2011) and it actually achieves 22 Mt per annum (actual average for this machine in 2010 – GBI, 2011), then the economics of this block of ore changes dramatically by virtue of the substantial increase in the time taken to mine it and the number of trucks which can be serviced by this loader in a unit of time.

The mining plan requires a range of inputs as detailed in Figure 1 (taken from Beniscelli, et al, 2000). Most of those inputs revolve around production rates and costs and can be subjected to error from incorrect inputs. Production rates and costs are the real keys to the DCF or ROR analysis but are normally done with minimal input apart from the potentially subjective opinion of the person doing the planning, (Lumley and Beckman 2009).

For more information; Graham Lumley +61 412787920 [email protected]

Figure 1.

Elements of the Mining Plan (taken from Beniscelli et al 2000)

Industry standards (actual operating results) are available in great detail from mines around the world, (GBI, 2011). The question is posed, “Why do people not use industry standards when completing mine development plans?”

Feasibility Study Data Input Geological information from exploration is utilised to a great extent in planning, (Lumley & Beckman, 2009) and this comes from the stringent requirements of the JORC Code and ASX listing rules. Mining software has clear links between geologic models and planning modules. One only needs to observe the Bowen Basin coal mines with draglines completely below the surface or the massive copper mines in Chile to see that the geometry of excavations can be incredibly complex.

Accurate inputs are recognised as paramount and the Valmin Code, JORC Code and ASX listing rules attempt to address this need. Much of the focus has been in tightening the specifications of resources and reserves, ie. the geologic inputs. However, this improvement of the geological inputs is difficult and adds significant cost to a mine development. Drilling programmes are high cost, have high lead times, and have diminishing returns. While the codes’ requirements for geological, processing and cost inputs are high, the same cannot be said for mining inputs, ie. the way a deposit will be mined and the rates predicted. The codes are very quiet on this issue.

Case Study 1. GBI (2011b) demonstrated the performance of P&H4100XPC shovels in the northern Bowen Basin. Best practice was 17.9MBCM per annum and median 14.1MBCM per annum. The project team was under pressure from Executive Management to budget 25 MBCM per annum because in their opinion, “That is what that model is capable of.” The GBI database indicates the P&H4100XPC shovel is capable of moving For more information; Graham Lumley +61 412787920 [email protected]

25MBCM per annum, however, only one machine in 40 from around the world will achieve this level and none from the northern Bowen Basin. The use of 25MBCM in their development models will make a huge difference in terms of predicted ROR but is most likely going to end in the company not meeting their forecasts for the proposed development.

Neither Valmin nor JORC Codes offer protection to financiers nor shareholders in this case.

Examples of Poor Data Input Flyvbjerg (2007) presents average errors between planned events / costs and the actual outcomes, and a study of the reasons. The findings are unequivocal and are in line with the author’s findings for the mining industry. 80-90% of projects will exceed budget cost and won’t deliver the benefits. He presents by way of example the English Channel Tunnel which cost twice the budget, delivered half the benefit and has had a negative rate of return. In other words, the British economy would have been better off if it had not been built.

Lumley and Beckman (2009) outlined a number of sources of error in mine planning inputs; both for production rates and costs. The following summarise the examples of what has led to overly ambitious production rates being used; 

Dig depths and face height impact on productivity not considered,



Variation in seam dip not considered,



Planning done in 2D and then merged to 3D,



Scheduling using maximum potential rate for KPI’s and productivity rather than what can be achieved over a longer period (sustainable rate),



Scaling performance from equipment of different capacity,



Overestimating hours of work,



Not considering fleet interactions,



Not understanding operational limitations, eg. Double side loading vs single side loading



Manipulating densities and bucket fill



Assuming every truck is full



Etc. etc

Case Study 2. A preferred contractor, quoting for overburden removal, used rates, shown by a benchmark against industry standards, to be in the 95th percentile for all equipment of the same make and model. Selection For more information; Graham Lumley +61 412787920 [email protected]

of a mining contractor will almost always be based on cost and rates achieved so the over-inflating of rates is a common occurrence (and a common source of dispute between mining company and contractor). Once the contract is underway it is difficult to terminate a contract due to dispute over causes of underperformance and consequently, mines accept claims for higher payments or less output. In this case the preferred contractor’s quote was subsequently rejected.

The Valmin Code (2005) & JORC Code (2004) The impacts of not using good data inputs are many. Most serious of these impacts are the effects on margin ranking to establish reserve estimates (transitioned from resources) and cashflow estimates to value the mine operation. (Lumley & Beckman 2009).

The Valmin Code (2005) is described as the “Code for the Assessment and Valuation of Mineral and Petroleum Assets and Securities for Independent Expert Reports”. The Securities Institute of Australia (AUSIMM, 2005) describes it as, “….indicative of best practice for independent experts preparing valuations and assessments in relation to specialist mining reports.” It is the standard for protecting investors and financiers in mining and petroleum developments. It is however, remarkably quiet on the issues of equipment performance and how this should be handled. The consequences on mine planning outcomes of not having clear standards in the critical areas under address in this paper suggest a lack of understanding within the industry of the significant implications. The code itself appears to have a heavy bias towards geology, resources and the processing of the ore itself. Multiple sections are provided in these areas plus other areas such as cost estimates and risk. The area of equipment rates and performance is covered in the broadest way under one point of the six sub-sections in the Mining and Processing Section. In Section 83, (p15), it states, “Existing and/or proposed mining and process plant practices should be reviewed to establish the technical and economic feasibility of the operation under consideration at its existing and/or proposed scales.” As a sub point of this the following are included; h) labour sources, requirements and productivity; i)

operating practices;

j)

equipment availability, utilisation and performance;

The JORC Code (2004) is described as, “The Australasian Code for Reporting of Exploration Results, Mineral Resources and Ore Reserves”. It sets out a principles-based approach to minimum standards, recommendations and guidelines for public reporting of exploration results, mineral resources and ore reserves. The JORC Code says more than the Valmin Code on mining issues because these directly impact the conversion of what is in the ground from a resource to a reserve. The JORC Code however, is again For more information; Graham Lumley +61 412787920 [email protected]

heavily biased to geologic issues. Mine planning issues are mentioned but nothing which might force a planner to be accountable for their decisions on equipment rates. The document says, “….the extraction of the ore reserve has been demonstrated to be viable under reasonable financial assumptions.” And further, “….appropriate studies will have been carried out prior to determination of the ore reserves. The studies will have determined a mine plan that is technically achievable and economically viable…….”

The focus on geological and processing is demonstrated further in Table 1 under the heading “Estimation and Reporting of Ore Reserves (pp17-18). It is the conversion of the resource to the reserve where significant input on the proposed mining operation (called the mining factors) should be provided. The guidelines are broad and demonstrate a lack of appreciation of the impact of factors such as equipment rates have on the determination of economic reserves. In leaving it broad the JORC Code follows the “principles-based” approach. There is no better demonstration of the failure of the principles-based approach than the last explanation point under the criteria “Discussion of relative accuracy / confidence,” (p. 18). That point states, “These statements of relative accuracy and confidence of the estimate should be compared with production data, where available.” Projects representing 3% of the total mining projects under development in Australia have accessed relevant operating, performance data to guide their mine plans and the determination of economic reserves. Clearly, the requirements of this clause are being ignored by the majority of “competent persons”.

This author would contend that the JORC Code, the Valmin Code and the ASX Listing Rules are failing those people it is meant to support by virtue of it being principles-based rather than prescriptive. The target audience for the competent person’s report is not well enough educated in the details of the mining industry to recognise the avoidance of some key issues and deliberate deception in other areas.

Bullock (2011, p85) supports this assessment of under-representation in mining inputs when he asks, “Why has such a tremendous effort been put forth to greatly improve the quality and standards of the resource and reserve classifications, but with little or no effort to improve the detailed definition of that which determines whether or not a resource will move from a resource to a reserve classification? Does the industry really believe that unethical practices of project feasibility studies can only come through misrepresentation of resources?”

Beckman (pers. comm. 2011) advises that in equipment selections for new or existing projects a great deal of effort is directed towards optimising the understanding of geology and costing predictions but the mine For more information; Graham Lumley +61 412787920 [email protected]

plan (in the area of equipment performance) is taken as provided. The opinion of the planning engineer, who might be an expert at mine planning but not in equipment rates is taken as the required standard of input.

GBI (2007) shows that most mine planners are producing plans which simply don’t produce the outcomes predicted and clearly under this scenario, the shortcomings of the relevant codes and rules should be addressed.

Case Study 3. The following is from a recent due diligence of a mine plan for a mine expansion. The mine plan used 18 Mt per annum as the output for Hitachi EX5500 hydraulic excavators. This assumption was not tested against industry standards. It was simply accepted as input into the mine design. The median annual output of this model was 13.5 Mt in 2010, (GBI, 2011). Consider also, that 50% of the people using this make and model will be less than 13.5 Mt per annum. In fact only 1 in 5 users of this make and model will achieve the specified rate. The implications of a significant shortfall for this and other site equipment would likely be financially devastating to a range of people investing in or financing the project.

Neither Valmin nor JORC Codes offered protection in this case.

Case Study 4. A copper mine in Zambia failed to

meet forecast performance over a number of years after

commissioning. It was described as a “ramping up in performance” but when the technical documentation is investigated the mine was planned to be running at full production rates by 2009 but continues to the present at equipment rates well below plan. To meet budget total output they have employed additional equipment. It has been reported that their difficulties are with operating hours and operating rate. It is known that rates are well below worldwide average for the equipment they are using. This is partly understandable, given the fact that the mine has used a largely unskilled workforce being trained from no experience. In this situation, a significant ramp-up period is expected. The author (who was not involved in the planning nor provided data to this development) can only surmise how such optimistic rates were used in the planning process. The real issue is that the mine will exceed the input costs that were used in the feasibility studies partly because the planning has used optimistic and probably unrealistic assumptions. The planner (a large, well known consulting company) has never been held accountable.

For more information; Graham Lumley +61 412787920 [email protected]

The Valmin Code and JORC Code (or the South African equivalents – SAMVAL/SAMREC) did not offer protection in this case despite being largely planned post-2004 when the requirements for reserves reporting were upgraded.

Why are Inputs into Mine Developments Wrong The literature provides three explanations for inaccurate inputs into mine plans; technical (inadequate or poor data and models), psychological issues (planner’s optimism) and political-economic issues (deliberate misrepresentation) (Flyvbrerg, 2007). Of these, technical reasons are most often proposed as the dominant cause, ie. errors in data and forecasting methods (Vanston and Vanston, 2004, p33).

In the case of mining data this can be seen in the following areas;

Too much data Companies are data rich and information poor. This is because the systems for storing and accessing the data have been developed by people who understand numbers but not what numbers may be useful.

Poor quality / unreliable data This is a significant inhibitor to business improvement. Data quality is the foundation for high valuegenerating data assets.

Outdated data Older data tends to have significant issues with accuracy and also relevance.

Poor quality analysis / forecasting The analysis of mining performance and costing is a specialised activity and should only be done by someone with expertise in the dual areas of data analysis and mining knowledge. This may or may not reside within the company. An example of this is with dragline data. Because of the unique arrangement of a bucket being controlled by two sets of ropes there are a range of dependencies between various kpi’s. If these dependencies are not accounted for, eg. kpi’s simply averaged in a specified time vs a different time, then the result is subject to error.

These sources of error in mining data and analysis may be reduced or eliminated by developing better forecasting models, better data and more experienced planners (Lumley & Beckman, 2009). However, if technical issues were the dominant or even a significant contributor, the distribution of errors would be For more information; Graham Lumley +61 412787920 [email protected]

normal (or near normal) and the average error would be near zero. Anecdotally this is not the case. The lack of impact of technical errors has also been proven empirically by Flyvbjerg (2008). Flyvbjerg (2008, p27) states, “ technical explanations must be rejected because they do not fit the data.”

Psychological issues account for inaccuracy in terms of optimism bias or planner’s bias (Lovallo and Kahneman, 2003). In the mining industry this is often called “planner’s optimism”. It is a cognitive predisposition (unintentional self-deception) to judge future events in a more positive light than is warranted by actual experience. It is a weakness in the way the human mind processes information and is thought to be a universal problem (Lovallo and Kahneman, 2003).

If the primary cause of failure to meet ROR on mining projects was unintentional planner’s optimism then this could be predicted and standards established for developments along those proposed by HM Treasury, (2003) for large scale infrastructure projects in Britain. They recommended that adjustments be made to a project’s cost, benefits and duration, and that the adjustments be based on data from past or similar projects elsewhere. Again, the lack of normality and consistency in the inaccuracies in project outcomes indicates that planner’s optimism is not a significant cause of the inaccuracy (Bullock, 2011).

The third explanation for inaccuracies in project outcomes is political-economic issues. They explain inaccuracy in terms of strategic misrepresentation, which Wachs (1990) puts bluntly as intentional deception. In this case, when forecasting the outcomes of projects, forecasters and planners deliberately and strategically overestimate benefits and underestimate costs in order to increase the likelihood that it is their project and not the competition’s, that gains approval and funding. Strategic misrepresentation can be traced to internal organisational pressures.

Wachs, (1990), suggests that in the early stages (project approval) of a development plan there are strong interests and incentives to emphasise benefits and de-emphasise costs and risks. Development teams and consultants need a pipeline of future work. Mining companies need future mines. Wachs, (1990), finds that this is not predominantly caused by non-intentional technical error nor planner’s optimism (which is also non-intentional). He states that it is deliberate misrepresentation which can be represented by the following Machiavellian formula.

(Underestimated costs) + (Overstated benefits) = (Project Approval)

For more information; Graham Lumley +61 412787920 [email protected]

The Machiavellian way prospers in an environment of low accountability. This environment of low accountability is supported by the principles-based JORC Code. Consequently when the industry tightened up the resource definitions and standards through the JORC in 2004, the deception didn’t stop; it just shifted.

The need for the industry to access (speculative) funds is driving this problem. Companies and mines compete for those funds. Consequently, the better a project looks the more likely will be its success in attracting those funds. The low accountability provided by the principles-based JORC Code (2004) is fuelling this reverse-Darwinism; “survival of the unfittest”, (Flyvbjerg, 2008) outcomes for our mining developments.

Case Study 5. The following is a very simple but common deception perpetrated on higher level management in Coal Mines. The output of mining equipment in the Australian coal mines is described in terms of Bank Cubic Metres. Equipment monitors measure tonnes and numbers of passes, loads, cycles, etc. Total output is tonnes per cycle multiplied by the number of cycles. To convert tonnes moved to BCM’s one must divide the tonnes by the in-situ SG. The output of a mine’s stripping fleet can and is being manipulated by the choice of in-situ S.G.’s. The use of a smaller S.G increases reported output in BCM’s. Conversely, in the development phase, the use of a higher SG means less tonnes to move and the required specification of trucks and loaders (and/or numbers of pieces of equipment) can be lower. Further to this the changing of SG over time in planning of new mines and in existing mines has been used to create an illusion of improving outcomes. An unnamed mine in Central Queensland and its Mine Manager received kudos for increasing mine overburden output. An investigation of the performance demonstrated over a 3 year period the in-situ SG had reduced from 2.40 t/CuM to 2.08 t/CuM.

Case Study 5. A new technology, implemented in the Bowen Basin, was justified on assumptions which led to an indicated 25% increase in productivity. Many millions of dollars were invested in machine modifications and hundreds of millions of dollars were lost through coal not available to be sold. The trial results were presented by the marketer of the technology (someone who had a direct financial interest in the further rollout of the technology) and the mine where the trial was held. The company which owned the mine also had a direct financial interest in the technology through a shareholding in the company which owned the IP. The industry was told productivity had improved by 28%. Verbal feedback provided by the operators suggested it was not that good. The author analysed the data and found actual productivity had For more information; Graham Lumley +61 412787920 [email protected]

dropped by 14%. Apart from filtering, clustering and normalising done in the guise of “comparing like with like” there was also a simple 6% error in the calculations which interestingly pushed the result from below to above the level used for justification. An observer would not find a conclusion of deliberate deception too hard to arrive at (but maybe difficult to prove).

It needs to be emphasised that this paper is directed at a portion of the Australian Mining Industry where deception is being perpetrated at single or multiple levels. Some Boards of Directors are complicit. Some Executive Management are guilty. Some development teams and consultants are supporting them. However, some mining houses do conduct rigorous procedures and insist on honesty to ensure their projects are appropriately ranked for development. Unfortunately, they are also subject to deception by people with interests not tied to the financial return from the resource in the ground but rather in the development process. The result is that Boards of Directors, financiers and shareholders cannot trust information provided to them about mine developments. There is a strong need to establish incentives and methods that produce more reliable information for the benefit of those providing money for developments.

A better way It is clear that many mine developments are proceeding based on doubtful engineering and reform is needed. This has been recognised by the ASX (2011) and JORC (2011). It is a macro issue but comprises a series of micro problems. In this paper the focus is on one of those micro areas; the use of accurate equipment rates in mine plans. Until the Valmin Code, JORC Code and the ASX Listing Rules address the issue of engineering inputs they will remain fatally flawed and won’t achieve their noble aims. Less error (deliberate or other) and more accountability are needed in the estimation of prospective mine returns. Two key inputs are recommended to achieve this; 1. Better forecasting of equipment rates through benchmarking against industry standards, and 2. More accountability of engineering input

The first of these two issues is described in European literature as “reference class forecasting” (Flyvbjerg, 2007 & 2008). In the mining industry it is called “benchmarking against industry standards” (Lumley, 2007). This involves taking an “outside view” (Flyvbjerg, 2008) on the particular aspect, (In this case equipment performance rates). In the mining industry a “benchmark” is based on a population of similar machines. This similar population may reflect, make and model of equipment, commodity, geographic location, diggability, pit layout, etc. This should not try to forecast what events will impact a particular

For more information; Graham Lumley +61 412787920 [email protected]

piece of equipment but should rather attempt to place the piece of equipment on a statistical distribution of current outcomes for the population.

Figure 2 and Figure 3 are two examples of how this data may be presented.

Figure 2.

Tabular presentation of benchmark data for a mining truck, in Queensland coal mines, with 242 tonne nominal payload with a flat, 13km cycle distance

For more information; Graham Lumley +61 412787920 [email protected]

Mining Truck 242 tonne Nominal Payload, 13km Cycle Distance 1,800,000 1,600,000 1,400,000

Best Practice

Annual Tonnes Carried

2,000,000

1,200,000 1,000,000 800,000 600,000 400,000

4th Quartile

200,000

3rd Quartile

2nd Quartile

1st Quartile

0 0

10

20

30

40

50

60

70

80

90

100

Percentile Figure 3.

Graphical presentation of benchmark data for a Mining Truck with 242 tonne nominal payload with a flat, 13km cycle distance

The planners, in conjunction with the owners, must develop a rationale for likely performance. It is the author’s experience that nearly all mine people believe they can perform in the 75th percentile or above and this is where forecasts are often targeted. Logically, only 25% of mines using a particular make and model of equipment achieve this rate. This should not provide comfort to financiers and shareholders.

Case Study 6. A new hydraulic excavator was being purchased for a mine. A distribution table similar to Table 3 was produced for the make and model, in the same geographic location for the same commodity. That is, the population was about as tight as could reasonably be achieved. The Mine Manager subsequently insisted rates in the 90th percentile be used in planning for the new piece of equipment (because that is what he “knew” it achieved from his experience). An analysis of the mine’s other equipment showed that they operated in the 42nd percentile on average. The difference was in excess of 10 Mt per annum. This has a substantial impact on how much of the commodity is available to be sold. The piece of equipment actually performed in the 58th percentile in the first year of operation due to additional focus on it but it never came near the 90th percentile.

For more information; Graham Lumley +61 412787920 [email protected]

If the population can be accurately defined and sufficient units are in the population to provide a statistically relevant outcome then it is strongly recommended that no forecast above the median be used.

On the issue of accountability it must be understood that planners, as a way of reinforcing their forward order book or their future job security, may not be focused on “getting it right” but rather “getting it funded”. As defined by the previous Machiavellian formula, accurate forecasts are often not an effective means of achieving the objective, (Wachs, 1990). Accountability will only take hold when decision-makers can change the power relationships that govern project development. Given the pressure on Boards of Directors to replace depleting assets this may need to be forced by shareholders, stock exchanges and/or regulators. To achieve accountability in the area of equipment production rates (and for many other areas in the development process) the following guidelines are proposed; 

Forecasts should be benchmarked against a relevant population.



Forecasts should only target median performance of the relevant population



Forecasts should be made subject to independent due diligence.



Forecasts, due diligence and benchmarking reports should be made available to all stakeholders.



Conferences should be organised where all stakeholders can scrutinise and question forecasts and planners.



Professional penalties should be enforced for planners who consistently produce deceptive forecasts.



Planners and their organisations should share financial responsibility for covering cost overruns and benefit shortfalls resulting from misrepresentation and bias in forecasting. Boards of Directors should have recourse to external consultants who provide erroneous inputs which cause financial loss. How they handle internal planners who make errors (deliberate or otherwise) is up to them.



The fiduciary responsibilities of Boards of Directors should be extended to project development.

A better future Fortunately, pockets of improvement have recently emerged in the area of inputs into equipment rates used in large developments and the subsequent accuracy of forecasts regarding mine development cost and benefits. 

A large coal mining company has over the last 18 months accessed benchmark data for use as inputs into their mine plans for two new mines and two existing mine expansions. They received data on up to 20 different makes and models for each project on own company performance,

For more information; Graham Lumley +61 412787920 [email protected]

Queensland performance and worldwide coal performance. The populations were sufficient to provide statistical significance for the results provided. They also conduct annual benchmarking on their draglines, as well as some loaders and trucks across their mines. Consequently, they have data on their performance and have knowledge of their current percentile rank for each make and model when compared to specified populations. This percentile rank can be applied to the benchmark data to provide valid inputs for mine plans. 

A large engineering company recently included a requirement in a contract with a specialist mine planner that the rates used in the mine plan be benchmarked against similar operations. The definition of the population was given great attention.



In addition, an iron ore miner accessed benchmark information for equipment they didn’t have currently operating as part of their iron ore expansion in the Pilbara. Some of this data was for internal purposes and some was provided to an external planner for use in the mine planning.



A small resource company conducting a DFS accessed actual results for a particular class of excavator for a proposed mine.

While being a significant development in this critical area, it is only the tip of the iceberg in developments in the Australian mining industry.

Conclusion The Valmin Code (2005) is, “….indicative of best practice for independent experts preparing valuations and assessments in relation to specialist mining reports.” It is the standard for protecting investors in mining and petroleum developments. The JORC Code (2004) sets out minimum standards, recommendations and guidelines for public reporting in Australasia of exploration results, mineral resources and ore reserves. The ASX Listing Rules provide a more prescriptive approach for defining resources and reserves. They have all failed to adequately protect shareholders, investors and financiers of mine developments. The source of error (deliberate or otherwise) has shifted from the definition standards of resources to the processes in shifting the mineral resources to ore reserves, ie. the engineering inputs used to define what is economic. Because of the shortcomings in the Codes, regulators have been unable to keep mining companies and planners accountable for the outcomes of their developments.

For more information; Graham Lumley +61 412787920 [email protected]

Mining industry feasibility and engineering studies must be improved. In Australia it must fall to the Valmin and JORC Codes, and the ASX regulations to offer more than a few sentences of guidance when it comes to anything apart from reserve definition, cost and processing issues. Why are there no specific guidelines nor standards for issues such as equipment rates?

Failure to achieve planned ROR in mine developments have been proposed in three areas; technical inaccuracies, planner’s optimism and strategic (deliberate) misrepresentation. The first two have been shown to be real but not significant sources of error; while deliberate deception has been shown to be rife across a range of industries.

This paper has presented a proposed two pronged approach to address this issue in the area of equipment performance inputs into feasibility studies. This approach is; 1. Better forecasting of equipment rates through benchmarking against industry standards, and 2. More accountability

They are linked and proper implementation of them will address the losses which many shareholders and financiers have incurred when investing in mining companies.

To achieve accountability in the area of equipment production rates (and for many other areas in the development process) a range of recommendations have been made to protect shareholders and financiers of mining projects

A series of statements should be required in all reports on developments outlining how reference class forecasting has been implemented in the particular study. In the area of mining inputs / equipment rates a statement along the lines of “Equipment rates used in this feasibility study conform to the XYZ standard. Median performance is used based on (number – suggested minimum 15) other machines of the same makes and models from the same commodity in the same geographic location”. And of course full details must be provided as an appendix. Processing plant performance, costs, manning levels, timing, etc., even plan outcomes (actual vs plan) should be forecast using a defined reference class and where a reference class cannot be defined then this should be stated. Only then can shareholders and financiers increase their confidence.

For more information; Graham Lumley +61 412787920 [email protected]

References ASX, 2011, ASX Listing Rules Review Issues Paper: Reserves and Resources Disclosure Rules for Mining and Oil and Gas Companies, ASX Public Consultation. Beniscelli, J, Carrasco, P, Dowd, PPA, Ferguson, G, & Tulcanaza, E 2000, Estimation of Resources and Conversion to Reserves — Protocols for the Assessment, Reduction and Management of Risk, In Proc. Mass Min 2000, Brisbane, Australia. Blainey, G, 1963, The Rush That Never Ended. A History of Australian Mining, 1st Edition, Melbourne University Press, Parkville, Victoria. Bullock, R, 2011, Accuracy of feasibility study evaluations would improve accountability, Jackling Lecture, Society of Mining Engineers, April. Flyvbjerg, B, 2007, Truth and Lies About Megaprojects. Faculty of Technology, Policy, and Management, Delft University of Technology, September. Flyvbjerg, B, 2008, Curbing Optimism Bias and Strategic Misrepresentation in Planning: Reference Class Forecasting in Practice, European Planning Studies, 16:1, pp. 3 – 21 GBI Productivity Datawarehouse, 2011a GBI, 2011b, Private report for mining company. Henning, J, Kaiser, PK, Cotesta, L & Dasys, A, 2002, Innovations in mine planning and design utilizing collaborative immersive virtual reality, 104th CIM Annual General Meeting, Vancouver, Canada HM Treasury, 2003, The Green Book: Appraisal and Evaluation in Central Government, Treasury Guidance (London: TSO) JORC, 2004, The JORC Code, Australasian Code for Reporting of Exploration Results, Mineral Resources and Ore Reserves JORC, 2011, JORC Code Review, Issues Paper Lovallo, D, & Kahneman, D, 2003, Delusions of Success: How Optimism Undermines Executives' Decisions, Harvard Business Review, July, pp 56-63 Lumley, G, 2007, Improving Dragline Operator Selection And Support Processes, University of the Sunshine Coast, Thesis submitted for the degree Doctor of Business Administration. Lumley, G, & Beckman, R, 2009, Is technology helping get mine plans right? Australian Mining Technology Conference 2009, 27 - 28 October 2009, in Brisbane, Australia Merrow, E, 2011, Industrial Megaprojects : Concepts, Strategies, and Practices for Success, John Wiley and Sons, Hoboken, New Jersey Valmin Committee, 2005, The Valmin Code, Code for the Technical Assessment and Valuation of Mineral and Petroleum Assets and Securities for Independent Expert Reports.

For more information; Graham Lumley +61 412787920 [email protected]

Vanston, J, and Vanston, K, 2004, Testing the Tea Leaves: Evaluating the Validity of Forecasts, ResearchTechnology Management, vol. 47, no. 5, pp 33-39 Wachs, M, 1989, When planners lie with numbers, Journal of the American Planning Association, 55(4), pp. 476–479. Wachs, M, 1990, Ethics and Advocacy in Forecasting for Public Policy, Business and Professional Ethics Journal, 9(1-2), pp 141-157

For more information; Graham Lumley +61 412787920 [email protected]