[01630755-1] Special Topics in AI and Cognition I: Cognitive Boundaries of AI
- Undergraduate course, offered by School of Psychological and Cognitive Sciences, Peking University
- Course number: 01630755
- Section: 1
- Course name: AI+心理认知专题I：人工智能的认知边界
- English name: Special Topics in AI and Cognition I: Cognitive Boundaries of AI
- Target students: Junior and senior undergraduate students
- Enrollment cap: 20 students
- Format: Small-group research projects, presentations, hands-on experiments, and writing workshops
- Description: This course examines the cognitive boundaries of current AI systems from the perspective of cognitive science. Students will work in teams of 2-4 to design and run rigorous benchmarking experiments on large language models and vision-language models.
- Location: 地学113
- Time: 2026 Fall, Weeks 1-16, Wednesdays, periods 7-8
- Credits: 2
- Language: Chinese
Contact Info
- Instructor: Yixin Zhu (朱毅鑫)
- Office: 王克桢楼1905
- Email: [email protected]
- Office Hours: Fridays, 20:00-21:00, 王克桢楼1904 (except holidays)
- TA: Hanyi Lou ([email protected])
Prerequisites
- Basic cognitive psychology
- Basic Python programming
Course Overview
Recent advances in large language models (LLMs) and vision-language models (VLMs) have demonstrated strong performance across many tasks. From the perspective of cognitive science, however, current AI systems still show systematic blind spots across many dimensions of human cognition.
This course is built around the intersection of cognitive science and artificial intelligence. Each student team selects one cognitive dimension and conducts a multi-layered evaluation of current AI models. The evaluation includes reproducing existing SOTA benchmarks, testing open-source datasets, reconstructing non-open-sourced experimental paradigms from the literature, and designing original test suites.
The final deliverable is a complete English-language paper in CogSci conference format, with 6 pages of main text plus references. The course also includes an industry visit to the NVIDIA Experience Center in Beijing, subject to scheduling.
Learning Objectives
By the end of this course, students will be able to:
- Understand core dimensions of human cognition and their theoretical foundations.
- Design cognitive benchmarking experiments for AI systems.
- Use LLMs and VLMs in systematic evaluation workflows.
- Reproduce existing benchmarks and experimental paradigms from published work.
- Build original test suites for evaluating the cognitive limits of AI.
- Analyze experimental results and write an English research paper in CogSci format.
- Understand frontier AI infrastructure and deployment practice through the NVIDIA EBC visit.
Cognitive Dimensions
Each group will focus on one cognitive dimension.
| Dimension | Example Directions |
|---|---|
| Perception | Visual illusion recognition, cross-modal perceptual consistency, scene understanding |
| Attention | Selective attention, change blindness, attention allocation strategies |
| Memory | Working memory capacity, long-term memory retrieval, false memory |
| Language | Pragmatic reasoning, metaphor understanding, ambiguity resolution, discourse coherence |
| Reasoning | Logical reasoning, inductive reasoning, spatial reasoning |
| Decision Making | Risky decision making, framing effects, probabilistic judgment biases |
| Causal Reasoning | Counterfactual reasoning, intervention inference, causal-chain understanding |
| Intuitive Physics | Object motion prediction, stability judgment, fluid intuition |
| Theory of Mind | False-belief tasks, intention inference, second-order belief reasoning |
| Abstraction and Analogy | Relational mapping, structural correspondence, conceptual combination |
Weekly Schedule
- Week 1 Course Introduction, Logistics, and Academic Integrity: cognitive science dimensions, AI benchmarking methodology, the four-layer evaluation framework, literature reading norms, presentation rubric, group formation, topic selection, and academic integrity.
- Week 2 Q&A on selected topics and assigned readings: students ask questions about the topic and paper they picked in Week 1, and the instructor answers.
- Week 3 Group Presentation Round 1, Part I: cognitive dimension definition and core literature review.
- Week 4 Group Presentation Round 1, Part II: cognitive dimension definition and core literature review.
- Week 5 Group Presentation Round 2, Part I: existing SOTA benchmarks, open-source datasets, and replication plan.
- Week 6 Group Presentation Round 2, Part II: existing SOTA benchmarks, open-source datasets, and replication plan.
- Week 7 NVIDIA EBC Beijing Visit: frontier GPU infrastructure and AI deployment practice. Exact time TBD.
- Week 8 Midterm Presentation and Defense: group presentation, paper outline submission, and instructor/TA feedback.
- Week 9 Group Presentation Round 3, Part I: novel test suite design.
- Week 10 Group Presentation Round 3, Part II: novel test suite design.
- Week 11 Experiment Execution and Preliminary Results: run experiments, collect data, discuss initial findings and issues.
- Week 12 Extended Experiments and Cross-model Comparison: test additional models, improve coverage, organize data.
- Week 13 Writing Workshop I: CogSci paper format, Introduction, Related Work, and group-level feedback.
- Week 14 Writing Workshop II: Method, Experiment, figure/table standards, and group-level feedback.
- Week 15 Writing Workshop III: Results, Discussion, full-paper revision, and peer review.
- Week 16 Final Presentation and Paper Submission: final presentation and complete CogSci-style draft.
Assessment
| Component | Weight | Description |
|---|---|---|
| Participation and attendance | 10% | Two random attendance checks and in-class discussion |
| Group presentations | 20% | Three presentation rounds |
| Midterm assessment | 20% | Midterm presentation and paper outline |
| Final paper and defense | 50% | CogSci-style paper draft, final presentation, code, and data |
Group Presentations
Three presentation rounds are graded independently and then combined.
| Round | Weight | Focus |
|---|---|---|
| Round 1, Weeks 3-4 | 30% | Cognitive dimension definition and literature review |
| Round 2, Weeks 5-6 | 30% | Existing benchmarks, open-source datasets, and replication plan |
| Round 3, Weeks 9-10 | 40% | Original test suite design |
Each presentation is 20 minutes, including 5 minutes for Q&A. Slides are recommended to be in English. The presentation grade combines instructor grading (60%) and peer review (40%).
Midterm Assessment
- Midterm Presentation, 10%: 15-minute group presentation covering the research question, preliminary evaluation plan, and literature progress.
- Paper Outline, 10%: 1-2 page English outline covering motivation, method, model choice, data sources, metrics, expected contribution, and timeline.
Final Assessment
- Final Paper, 35%: CogSci-style English paper draft, 6 pages of main text plus references.
- Final Presentation, 10%: 20-minute group presentation including 5 minutes for Q&A.
- Code and Data, 5%: Private GitHub repository containing evaluation code, data, README, and reproduction instructions.
Evaluation Framework
Each group should cover four levels of evaluation.
| Layer | Requirement | Example |
|---|---|---|
| Layer 1: SOTA benchmark reproduction | Test AI models on existing public benchmarks to obtain baseline results | Test a model on a causal reasoning subset of BIG-Bench |
| Layer 2: Open-source dataset testing | Use public datasets to design evaluation workflows and analyze model behavior | Test intuitive physics with I-PhyRe |
| Layer 3: Non-open-source paradigm replication | Reconstruct experimental paradigms reported in the literature but not publicly released | Rebuild theory-of-mind test scenarios from a paper |
| Layer 4: Original test suite design | Propose original evaluation methods or test cases | Design new visual illusion stimuli for probing AI perception |
Course Resources
- NVIDIA 3090 GPU server accounts for each group.
- Pre-deployed Qwen and Qwen-VL models through course-provided APIs.
- NVIDIA EBC Beijing visit, subject to scheduling.
- CogSci paper writing guidance.
Students are encouraged to test additional models from Week 12 onward, including GPT, Claude, and Gemini, using their own API access or free quotas.
Academic Integrity
All papers, code, presentation materials, experiments, analyses, and conclusions must be the original work of the student group. AI tools may be used for language polishing or assistance, but core research content, experimental design, data analysis, and conclusions must be completed by students and disclosed where appropriate.
Plagiarism, fabricated data, falsified results, and improper citation are strictly prohibited. Please refer to the plagiarism page for details.
References
- Binz, M. et al. (2025). A foundation model to predict and capture human cognition. Nature, 644, 1002-1009.
- Spelke, E. S. (2022). What Babies Know: Core Knowledge and Composition, Volume 1. Oxford University Press.
- Tomasello, M. (2022). The Evolution of Agency: Behavioral Organization from Lizards to Humans. MIT Press.
- Griffiths, T. L., Chater, N., & Tenenbaum, J. B. (2024). Bayesian Models of Cognition: Reverse Engineering the Mind. MIT Press.
- Griffiths, T. (2026). The Laws of Thought: The Quest for a Mathematical Theory of the Mind. Henry Holt and Co.
- Zhu, Y., Gao, T., Fan, L., et al. (2020). Dark, Beyond Deep: A Paradigm Shift to Cognitive AI with Humanlike Common Sense. Engineering, 6(3), 310-345.
Errata/Typo
Please contact the instructor or TA.