The neglected problem: How to really benchmark the effectiveness of ICT based services? A case study

Expensive, large-scale efforts are often made to develop and implement new information systems in complex areas such as support to public services (egovernment). The effectiveness of such systems is seldom - if ever - put to the test a posteriori by making an appropriate comparison with similar set-ups. Without such an ex post methodological evaluation effort, substantial investments may continue to be made to no avail, when the services could instead be improved by simply focusing on other factors. This paper outlines steps for comparing the job market place support services provided by two European regions, Lombardy (Italy) and Catalonia (Spain), and shows the objective difficulties of doing this. It compares some aspects and issues of the two different systems used. Finally, some research questions investigating the impact of the two systems on two similar socio-economical European areas are outlined.