FYI: Indicators and Benchmarking as a Support to the Decision-Making Process

[Henk Elegeert]

REPLY TO: D66 at nic.surfnet.nl

http://www.oecdbookshop.org/oecd/display.asp?TAG=X9E168XX598X5935X4MD2D&CID=&LANG=EN&SF1=DI&ST1=5LGZ28JHN6D5#OtherLanguages

http://www.oecdbookshop.org/oecd/get-it.asp?REF=3005061E.PDF&TYPE=browse
pagina's 342 - 350

"
Indicators and Benchmarking as a Support to the Decision-Making Process:
the Italian Experience in Active Employment Policies

Aviana Bulgarelli

Director General for Vocational Training and Guidance Policies, Ministry
of Labour and Social Policies, Italy

Abstract

Benchmarking can be helpful in at least two ways in the decision-making
process. It can be used for a coordinated, consistent and systemic
analysis of performance, possibly tied to a system of targets. It can
also be used to identify the most effective intervention policies,
models and instruments for achieving the targets and enabling good/best
practices to be pinpointed and transferred in the various scenarios.

The adoption of the benchmarking tool in the European experience
inevitably enhances the role of statistical information, not only for
monitoring and evaluating interventions, but also for constructing and
implementing the policy process. The European Employment Strategy, for
example, is conceived as a “guided convergence” process towards
quantitative targets and, at the same time, as a reference framework for
a complex benchmarking exercise based on the open coordination method.

The paper analyses, first of all, the impact of this approach on the
Italian experience in recent years. Reference is made not only to the
social and labour policy programming model, reorganised for the first
time into a global model, but also to the identification and the
orientation of priorities for policies. As a result, the identification
of a relevant – and feasible – common system of statistical indicators
constitutes a strategic goal that has only been partly achieved to date.

The need to dispose of a vaster and better information capital – in
terms of timeliness, comparability, pertinence, etc. – is not the only
critical point to be dealt with. How can we obtain information that is
really capable of guiding policy choices? How can we overcome the
ex-post nature of benchmarking? How can we ensure a system consistent
with the relevant level of the decision-making process? Basically, our
attempt is to open a debate on how – and to what extent – it would be
possible to raise the benchmarking system from a simple tool for
comparing and analysing policy performance (performance gap) to an input
– if not an authentic surrogate – of the evaluation process capable of
guiding the decision-making process.

Benchmarking in the public decision-making process

Benchmarking bears at least two meanings and can be helpful in two ways
in the decision-making process.

• On the one hand, it can be used for a coordinated, consistent and
systematic analysis of performance, possibly tied to a system of
targets/objectives; in this case, the choice of indicators is strictly
linked to the identification of a system of criteria/targets which are
considered relevant by public decision-makers, involving, as a first
step, the definition of a set of significant and feasible indicators.

• On the other hand, it can be used to identify the most effective
intervention policies, models and instruments for achieving the
objectives fixed and enabling good/best practices to be pinpointed and
transferred in various contexts.

Clearly, this distinction does not imply that both approaches should be
considered separately. In fact, the definition and measurement of
performance gaps in a system of targets/objectives is a necessary basis
for choosing the most suitable good/best practices for a given policy
issue. Likewise, the ex-post analysis of results achieved through the
policies adopted represents a reference for a possible redefinition of
the decision-making process.

However, our intention here is to focus mainly on the first approach,
not only because statistical information plays an essential role in this
case, but also because, as we shall see, the experience gained over
these last years within the European Employment To understand better the
difficulties encountered and to contribute to a definition of
policy-makers’ needs, it is important to remember that, in order to
provide an effective support for the decision-making process,
benchmarking should be integrated in a broader process, which could be
ideally summarised in the following subsequent and iterative stages1:


1. Identification of macro-objectives and corresponding performances to
be considered as really significant for comparing different
countries/regions in order to measure the success of implemented policies.

2. Identification of a system of elementary variables/indicators
referable to the macroobjectives and representative of the different
possible channels national/regional) through which they can be achieved
(different territories, population targets, production sectors, etc.,
should be given a different weight).

3. Comparison, on this basis, between the national/regional situation
examined and the reference benchmark. This is particularly important
since performance gaps can be measured with reference to either a system
of predefined targets or the values presented by best performers.

4. At this point, it is necessary to understand and explain the reasons
for the differences observed in performance, pinpointing those on which
it is more important to act, and identifying as well actions to be
undertaken. Here, benchmarking is used to find best/good practices with
better effectiveness/impact taking into account the current context
conditions (sociocultural, political, institutional, etc.).

5. Monitoring policy implementation, analysing benchmarking results and,
possibly, Unlike its utilisation in the business field, this approach
obviously involves different and particularly complex problems in the
case of public policies (for instance in the case of the EES). In fact:

( 1 See Tronti L., “Il benchmarking dei mercati del lavoro. Una sfida
per le regioni italiane?”; in Antonelli G. and Nosvelli M.,
“Monitoraggio e valutazione delle politiche del lavoro per una nuova
economia”, Il Mulino, 2002 )


• multiple objectives and, at the same time, specific intervention
instruments and policies make this approach complex and often difficult
to “systematise” within a single decision-making model;

• objectives sometimes can be conflicting and it is not always possible
to identify the corresponding trade-offs;

• it is often difficult to find connections between individual
policies/instruments available and the different objectives identified,
which are strongly influenced by the specific context conditions;

• the plurality of actors contributing to the identification of
objectives and to the implementation and monitoring of policies (these
latter having an impact on expenditure decisions) make this scenario
even more complex.

Therefore, the measurement of performance gaps only represents the first
stage of a decision-making process based on benchmarking, which requires
not only measuring but also understanding, planning, testing, monitoring
and, if necessary, reconsidering the policy choices made. Within this
framework, the need for good statistical information is only one of the
necessary conditions; nonetheless, as the EES experience has
demonstrated, benchmarking and its related system of statistical
indicators have often become a “surrogate” for evaluation, with an
appreciable influence on the decisionmaking process.

The European Employment Strategy and benchmarking in the Italian experience

The EES is an example of multi-level strategic programming and
represents, at the same time, the reference framework of a complex
benchmarking exercise based on the open method of coordination. It is,
in fact, a process of “guided convergence” which, starting with a
reference framework of guidelines, uses benchmarking as an instrument to
support a programming process based on objectives that are also defined
quantitatively. For non-Europeans, this brief example could be useful2:

As a matter of fact, this process is somehow similar to that initiated
with the Maastricht stability pact, but inevitably with a greater degree
of freedom:

• firstly, because of the greater complexity of social and labour
policies (in terms of objectives, targets and effectiveness of
intervention tools);

• secondly, because it is part of a particularly complex decision-making
mechanism involving a large number of actors, at different
decision-making levels with high political, managerial and instrumental
responsibility (from the EU level to the local level).

( 2 See “Council Decision of 22 July 2003 on guidelines for the
employment policies of the Member States”, in Official Journal of the
European Union, L. 197/13, 5.8.2003. )

[
Overarching objective: Full employment

Member States shall aim to achieve full employment by implementing a
comprehensive policy approach incorporating demand and supply side
measures and thus raise employment rates towards the Lisbon and
Stockholm targets.

Policies shall contribute towards achieving on average for the European
Union:

• an overall employment rate of 67% in 2005 and 70% in 2010,
• an employment rate for women of 57% in 2005 and 60% in 2010
• an employment rate of 50% for older workers (55-64) in 2010

Specific guideline: 4.
Promote development of human capital and lifelong learning

Member States will implement lifelong learning strategies, including
improving the quality and efficiency of education and training systems,
in order to equip all individuals with the skills required for a modern
workforce in a knowledge-based society, to permit their career
development and to reduce skills mismatch and bottlenecks in the labour
market.

In accordance with national priorities, policies will aim in particular
to achieve the following outcomes by 2010:
• at least 85% of 22-year olds in the European Union should have
completed upper secondary education,
• the European Union average level of participation in lifelong learning
should be at least 12.5% of the adult working-age population (25 to 64
age group).
]

However, by identifying a reference system of objectives anchored to a
set of quantified indicators, the EES has had an influence on the action
of policy-makers, at a national level in the first place, but also at a
regional and, in some cases, at a local/provincial level.

Fabrizio Barca3 has already explained in his contribution the influence
EU policies have had in Italy on the development programming and
governance model, fostering a strengthening of instruments – both
cognitive and methodological – for defining policy choices. The EU
programming model, based on the meta-objective of “economic and social
cohesion”, has triggered a profound reorganisation of structural
policies within a “global” model. This model now also constitutes a
benchmark for Italian programming at both national and regional level.

Looking specifically at active labour policies, the constant reference
to the EU framework has had a considerable influence on public
decision-makers. Some examples seem particularly significant in this
regard, but we will consider here only their institutional aspects,
leaving aside the more specific, albeit important, political aspects of
the decision-making process.

We know there are a large number of indicators proposed for monitoring
EES targets. They have been partly changed already and will undergo
further modifications over time, thanks to the identification of better
quality and more relevant indicators with respect to

( 3. Head of Department for Development Policies, Ministry of Economy
and Finance, Italy )

identified priorities. In particular, employment indicators are linked
to the different key aims defined by the Lisbon European Council
referring to strengthening employment levels, equal opportunities for
men and women and, more in general, greater participation in the labour
market of the older population and women.

A first example of the influence of the benchmark mechanism refers to
the recent indepth reform process of the labour market in Italy, and it
is quite significant since it concerns a typically national policy
sphere. Many of the targets set by the EES have already been assimilated
into the national and often regional context. This resulted in stronger
emphasis, for example, on the female component or on the older
population, even when – as in many regions in southern Italy – the high
priority of the “elderly” component of the workforce does not fully
correspond to the local labour-market conditions.

Again, at a national level, this impact has also emerged during the
implementation of the labour policy monitoring system, whose
reorganisation has been partly inspired by the European benchmark model,
with all the difficulties ensuing from the aim of coherently linking up
a system involving multiple actors at various levels and with different
degrees of involvement. As already mentioned, the Italian labour policy
programming, implementation and evaluation process is articulated at
many decision-making levels with accentuated forms of subsidiarity, both
vertical (central administrations, regions, provinces, etc.), and
horizontal (employers associations and trade unions). This complexity
also inevitably concerns, both from the producer and the user side, the
statistical information system which is necessary to monitor and
evaluate the policies adopted. The objective of compatibility with the
Lisbon benchmark has inevitably made the entire process more complex and
required the creation of a large number of “technical units”, involving
a multiplicity of actors.

Among the Lisbon benchmark indicators, that concerning lifelong learning
represents another emblematic case of how having to comply with a system
of indicators (and performance gaps) has acted as a catalyst on the
attention of policy-makers, prompting extensive corrections in targeting
and rebalancing the policy mix.

In Italy, the indicator used for lifelong learning, represented by the
participation rate of population aged between 24 and 65 years in
permanent and continuing training, shows a severe structural lag:
against the objective set to a participation rate of 12.5% by 2010, our
country only reached 4.7% in 2003, compared to an EU average of 9.7%.
This context is moreover characterized by strong disparities, not only
among generations (mainly to the detriment of older groups) but also
among regions (southern Italy is particularly affected). Over and above
the interpretative problems that this indicator also presents – and we
shall come back to this later – it is clear that public decision-makers
are well aware of this fact; however, once integrated within the Lisbon
framework, this indicator has had a significant impact on policy makers,
who only had little room for manoeuvre. In a situation where resources
are limited, redirecting them towards lifelong learning policies means
reorganising alternative intervention policies, with consequences:

• not only from the political point of view but also regarding the
management of the governance system referring to active labour policies;

• from an organisational point of view, due to the need to redirect the
“machine” towards greater needs pertaining to the programming,
management and implementation of interventions.

In this context, greater attention has therefore been devoted to adult
education and training policies, both in general and specifically
addressed to the employed population, by developing new types of
training supply. However, it was also clear that, because of the gap
with the reference benchmark and given the existing budget and time
constraints, it was going to be a “mission impossible”. Even with
greater financial efforts and making adjustments by redefining the
beneficiaries covered by the main policy tools available, the only real
“room for manoeuvre” had clearly to be found in trying to make the most
of available resources.

The existing statistical information – we refer to CVTS2, Eurostat’s
survey on continuing vocational training in enterprises – showed that,
within the broader adult population target, training activities for the
employed population had a particularly negative ranking in the European
context: in fact, only one Italian firm out of four (around 24%) carried
out any form of training activity, compared to a 62% average
participation of European enterprises. An in-depth analysis of the data
provided by this survey also showed how this could be explained – at
least partly – by the peculiarity of the Italian industrial system,
based on small and medium-sized enterprises and on socalled traditional
sectors (clothing, footwear, wood and furnishings, etc.).

In particular, the existence of smaller-sized enterprises seemed to be a
key factor explaining the low propensity of enterprises to offer
training to their employees: investing in human capital – just as in
Research & Development – does not usually produce tangible effects in
the short term and requires a strong corporate culture and strategic
vision. Factors that tend to discourage smaller firms from investing in
human capital include a greater incidence of training costs (not only
direct costs but also those deriving from missing workers in the
production process), a lesser organisational and logistical capacity,
and greater uncertainty with regards to returns of training, especially
in a context of growing labour flexibility and mobility. Within this
framework, introducing innovative forms of intervention seemed more
important than financial incentives to ensure greater effectiveness of
policies for continuing training.

To achieve such an objective, new bodies have been created
(“Interprofessional Funds for continuing vocational training” - Fondi
paritetici interprofessionali per la formazione continua) where, for the
first time, social partners would be directly involved in the
programming and management of a significant part of the resources
allocated to continuing vocational training. The aim was to encourage a
greater dissemination of the training culture, especially with regards
to smaller-sized enterprises, and contribute to steer financing towards
initiatives expected to be more in line with the real requirements of
companies.

Interprofessional Funds are now fully operational. Starting from 2004,
they have an independent financing channel as well as a significant
supply of resources in addition to those allocated in 2003 by the
Ministry of Labour and Social Policies for their start-up.
Interprofessional Funds are part of the existing system for financing
and managing continuing training initiatives (i.e. the European Social
Fund as well as the national/regional programmes and funds). The real
difference today is the fact that social partners share responsibility
for the operational management, whereas previously, the social partners’
role mainly consisted of cooperating with public administrations in
order to define intervention strategies and priorities within the
framework of concerted actions. They now have actually to manage
financial resources, to plan and direct interventions, to organise and
reconcile local and sector needs and to monitor the outcomes of activities.

The close link between the funds and the enterprises involved acts as a
facilitator, and one can reasonably expect a greater and prompter
ability to grasp the needs of enterprises.

Also in the case of lifelong learning, the construction of a system for
collecting and producing the statistical information needed for
monitoring and evaluating the policies implemented is an important
related aspect. In the light of our previous explanations, the promotion
of lifelong learning involves a set of particularly complex policies,
where not only institutional actors but also other players – such as
social partners – are directly engaged in the programming and management
of considerable financial resources (i.e. not only the Fondi
Interprofessionali but also the national funds for the vocational
training of temporary workers). As far as continuing vocational training
is concerned, the achievement of an integrated monitoring and evaluation
system in line with the European benchmarks thus requires significant
efforts from the administration.

The example of lifelong learning is undoubtedly emblematic for national
policies, but what has happened in recent years at a regional level, for
instance during the ESF programming and reprogramming, is equally
significant. In addition, it is also worth mentioning the experience
gained at a strictly local/provincial level of programming, which I will
briefly illustrate.

The Italian Provinces are required to play a leading role in active
labour policies and have also been involved, more or less directly, in
the benchmarking system proposed with the EES. A typical example is the
experience of Local Action Plans for employment, where many local
authorities have set up integrated strategic programming tools for
training, education and labour, within the framework of EES guidelines.
These local entities have directly measured themselves with the European
benchmark when defining active labour policies and, more in general,
local development policies.

All these are only brief examples and many more could be presented, but
they obviously give a positive picture of the “stimulating” role of the
benchmarking system. However, the processes involved, currently and in
the past, are very complex and require the participation of a large
number of both national and local stakeholders, as well as a firm
commitment of the administrations concerned. Therefore, the
identification of a significant and feasible common system of
statistical indicators represents a strategic goal that seems to be
however only partly achieved. In fact, the need to dispose of wider and
better information – in terms of timeliness, comparability, relevance,
etc. – is not the only crucial aspect to be dealt with.

Indicators, benchmarking and evaluation

 From the experience I have summarised previously, we can identify some
of the main difficulties, methodological and technical, linked to the
quality of the statistical information system and to the indicators used
for the construction of benchmarking. Since data not only constitute the
basis for the knowledge of different phenomena, but also play a key role
in steering policy makers’ choices, their imperfection is a potential
source of serious problems for public decision-makers and could make
their work more difficult if not “damaging”.

Of course, benchmarking makes sense insofar as the accuracy of data is
really significant in terms of coherence and above all comparability. In
particular, when the implicit goal is to evaluate the progress of
policies in different countries/regions, it is obviously better to avoid
making any comparisons at all than to make “wrong” ones. Without going
into too much technical detail, I would like to give a small example
concerning lifelong learning.

Until the recent introduction of the new continuous labour force survey,
the number of trained employees was estimated with reference to a
four-week period preceding the interviews. As it is well known to
statisticians, such choice could potentially produce biases due to the
poorer statistical visibility of short training courses. For countries
with short average duration of training courses, the shorter is the
reference time of interviews, the lower is the probability to observe
the phenomenon over a given observation period. It is of course
difficult to know how significant these biases can be in the Italian
case, but it has to be mentioned that our country has an average
duration of training courses decidedly below the European average.

The example of the Local Action Plans also enables us to make some
considerations, highlighting another important criterion, i.e. the
relevance of indicators. Relevance is understood as the ability to take
into account the cognitive needs of an often very large number of users,
with different systems of targets and context conditions. We know there
can be great differences from one country to another, not only because
of the existing disparities across countries but also because of
internal disparities within each country. In a multi-level programming
system like that of the European Union, and, to a greater extent, of
Italy, paying attention to bringing statistical information to a correct
level means, in turn, analysing the different decision-making levels. In
this sense, it is important to understand “who has to decide what” and
to provide a suitable cognitive support, especially with a view to
evaluation.

The problem of timeliness of statistical information is more general.
The acceleration of economic processes implies that equal speed is
needed to adjust decision-making processes and institutional frameworks
as well as statistical information and knowledge on the phenomena.
However, indicators inevitably reflect an “out-of-date” vision of the
economy: indicators are always ex-post and they are often available
after a long delay.

This problem is also encountered at European level – think of the
monitoring process of the EES implementation – but it is even more
relevant at a national level. The case of the Italian labour market
provides a good example of this: over the last decade, the existing
forms of employment contracts have been subject to radical changes, just
as their legal and regulatory framework, creating considerable problems
linked to the significance of statistical information, especially when
used for a diachronic analysis of the phenomena.

The issue of timeliness must be tackled more courageously, opening the
debate on the possibility to define common methodologies for
constructing scenarios/simulations, or at least common criteria and
parameters. If it is true that the acceleration economic processes
requires a switch from a reactive approach to a proactive one, also in
the field of public policies, then the construction of a system for
estimating indicators is more and more becoming as important, in the
decision-making process, as the knowledge of the context.

Finally, I would like to tackle briefly an issue which is linked to the
connection between the objectives to achieve and the policies to
undertake. Here we come to the point of the second meaning of
benchmarking, i.e. that of good/best practices, where the problem of
transferability is determinant. The existing differences between the
various national/local contexts in which policies are implemented can
have a great impact on their effectiveness. It is difficult to relate
one aspect to the other, and statistics can only help up to a certain
point. However, this is a decisive point for making the benchmarking
approach effectively useful for the decision-making process.

We can see that we have gradually come to the crucial point of the
relationship between benchmarking and policy evaluation and I have
already mentioned at the beginning of this paper that the benchmarking
approach in this case seems still very far from being fully applied
within this context. How can we obtain information that is really
capable of steering policy choices? How can we overcome the ex-post
nature of benchmarking? How can we ensure a system consistent with the
specific level of the decision-making process? Basically, our attempt is
to open a debate on how – and to what extent – it is possible to raise
the benchmarking system from a simple tool for comparing and analysing
policy performance to an input – if not an authentic surrogate – to an
evaluation process capable of guiding the decision-making process.

References
...
"


Henk Elegeert

**********
Dit bericht is verzonden via de informele D66 discussielijst (D66 at nic.surfnet.nl).
Aanmelden: stuur een email naar LISTSERV at nic.surfnet.nl met in het tekstveld alleen: SUBSCRIBE D66 uwvoornaam uwachternaam
Afmelden: stuur een email naar LISTSERV at nic.surfnet.nl met in het tekstveld alleen: SIGNOFF D66
Het on-line archief is te vinden op: http://listserv.surfnet.nl/archives/d66.html
**********