Attribute-based access control (ABAC) is a very powerful and flexible security technique making it possible to overcome limitations of traditional role-based and discretionary access controls. ABAC enables the dynamic handling of vast numbers of heterogeneous and changing resources and users, a task especially relevant for E-Commerce or distributed computing. With an authentication and authorisation infrastructure (AAI) in place, service providers could benefit from synergies and outsourcing possibilities and, simultaneously, strengthening their security level. In addition, AAIs could arbitrate between users’ privacy issues and vendors’ information demands, using privacy enhancing technologies. However, implementing ABAC is not trivial; nor is the derivation of attributes or metadata. This work proposes a solution to the demands for privacy aware, usable, secure, and outsourceable E-Commerce infrastructures with an AAI / ABAC combination. We introduce relevant technologies and an implementation that is evaluated. The prototype is based on the Liberty Alliance’s ID-FF system, using XACML elements and classification tools.