The Intelligent Space (iSpace) is a space that can analyze human behavior through computer vision, present user-centric information, and autonomously control robots as users intend. This study aims to support humans and robots to enhance their performances both informatively and physically. To develop such a smart space, iSpace needs to know “when, where, and who did what”. Thus, we recognize the human gesture, facial expression, and human-object interaction and integrate them into the human state. By mapping the human state to an environmental map, iSpace can learn information like a user’s interest and utilization of space. The final goal of this research is that iSpace can infer better information presentation and robot control from the space data.