Measurement of information system project success (ISPS) remains an unsolved challenge. Instead of seeking a commonly accepted measurement approach, some scholars suggest to assess ISPS by applying a contingency approach. Its central idea is that no set of success dimensions fits best in all situations; instead, the dimensions’ relevance differs depending on variables like stakeholder perspective or project characteristics. We continue existing research by investigating how the relevance of success dimensions varies depending on project characteristics like complexity or size. We conduct a quantitative survey among experienced project managers in Germany. Results from 102 projects indicate that whereas certain dimensions (e.g., meeting customer requirements, customer satisfaction) are important in most projects, the relevance of many dimensions significantly varies depending on project characteristics. For researchers, we expect the more adequate assessment of ISPS to increase the validity of ISPS studies. We help practitioners to identify relevant ISPS dimensions for their projects.