Information system effectiveness is an important phenomenon for both researchers and practitioners. Despite widespread interest, and the importance of the uses, there have been no efforts to validate CompufemorM’sPremier 100 rankings of information system effectiveness. This paper uses structural equation modeling in an attempt to validate the measuring instrument usee to derive the Computerworld rankings. Alternative models for the measuring insrrument are proposed. TJsing a .reflective model, the findings raise doubts as to the reliability of the rankings, and both conteni validity 2nd construct validity are also suspect. The reliability and validity are problematic because multiple indicators of the same consvict must be homogeneous for it to make sense to combine them into a conposite index. A solution to this problem is to represent information system effectiveness as a multidimensional construct that is part of a causal model. Based on previous research in the area, suggestions are offered to improve the measuring instrument.