Social media monitoring has been initially adopted by private sector firms in order to collect opinions, complaints and questions concerning their products and services, to be used for making appropriate changes and improvements of them and also for designing communication strategies. Recently government agencies have started adopting SMM, as a form of ‘passive citizen-sourcing’, in order to collect useful information from citizens concerning their needs, problems, opinions and suggestions, to be used for public policy formulation. It is therefore important to evaluate these first initiatives, so that the potential of SMM with respect to public policy making can be exploited, and at the same time appropriate adaptations and improvements of relevant ICT platforms and practices can be made, in order to reach higher levels of maturity. This paper makes a two-fold contribution in this direction. Initially it develops a framework for evaluating the use of SMM for supporting policy making, initially from the ‘classical’ ease of use perspective, and then from a public policy perspective, based on the wicked social problems theory. This framework is then used for the evaluation of three pilot applications of a novel method of SMM by government agencies and other policy stakeholders, which has been developed as part of a European research project.