Yes, exactly: you use a fast data store such as redis or memcached.
As a user performs activities, this may involve a scenario requiring escalation or revocation of authorization roles and corresponding permissions. Invalidate cache at this moment. Lazy cache updated authorization info upon next request.
