The advance of ecosystem-based fisheries management worldwide has made scientific advice on fisheries related questions more complex. However, despite the need to take interactions between fish stocks, and between stocks and their environment into account, multispecies and ecosystem models are still hardly used as a basis for fishery advice. Although reasons are numerous, the lack of high-level guidance for target-oriented skill assessments of such models contributes to the mistrust to use such models for advice. In this study, we propose a framework of guiding questions for a pragmatic and target-oriented skill assessment. The framework is relevant for all models irrespective of their complexity and approach. It starts with general questions on the advice purpose itself, the type of model(s) and data available for performance testing. After this, the credibility of the hindcasts are evaluated. A special emphasis is finally put on testing predictive skills. The skill assessment framework proposed provides a tool to evaluate a model’s suitability for the purpose of providing specific advice and aims to avoid the bad practice of incomplete skill assessments. In the case of multiple models available, it can facilitate the evaluation or choosing of the best model(s) for a given advice product and intends to ensure a level playing field between models of different complexities. The suite of questions proposed is an important step to improve the quality of advice products for a successful implementation of ecosystem-based fisheries management.