This paper describes a simulation study that evaluated the performance of the scientific advisory process used by ICES to recommend total allowable catches (TACs) for roundfish stocks. A "management strategy evaluation" approach is used, involving development of an operating model to represent the underlying reality, and an observation model to generate pseudo data that are then used within a management procedure. The management procedure comprises an assessment that uses data to estimate parameters of interest and a decision rule to derive TAC recommendations for the following year. There are two important results: including realistic sources and levels of uncertainty can result in far from optimal management outcomes based on the current procedures; and current ICES biomass and fishing mortality reference points are not always consistent, and several are clearly inappropriate. This is because the types of projection used by ICES do not incorporate important lags between assessing stock status and implementing management measures, and they also ignore important sources of uncertainty about the actual dynamics, as well as our ability to collect data and implement management regulations (i.e. model, measurement, and implementation error, respectively). The simulation approach also showed that better management is not necessarily going to be achieved by improving the assessment, because even with a perfect assessment (where the simulated working group knew stock status perfectly), stocks may crash at fishing levels that standard stochastic projections would suggest were safe. It is proposed that, in future, operating models that represent the best available understanding of the actual system dynamics be used to evaluate models and rules considered for application. These operating models should capture the plausible range of characteristics of the underlying dynamics, but not necessarily model their full complexity. In general, they will be more complex than those used by assessment working groups, so developing management procedures that are robust to a broad range of uncertainty. However, the models and rules used as part of the management procedure should be simpler than those used at present. Crown Copyright (c) 2005 Published by Elsevier Ltd on behalf of International Council for the Exploration of the Sea. All rights reserved.