From Observation to Abstraction