54 research outputs found

    Multimodal knowledge capture from text and diagrams

    Full text link
    Many information sources use multiple modalities, such as textbooks, which contain both text and diagrams. Each captures information that is hard to express in the other, and evidence suggests that multimodal information leads to bet-ter retention and transfer in human learners. This paper describes a system that captures textbook knowledge, using simplified English text and sketched versions of diagrams. We present experimental results showing it can use cap-tured knowledge to answer questions from the textbook’s curriculum. Categories and Subject Descriptors I.2.4 Knowledge Representation Formalisms and Method

    nuwar: A prototype sketch-based strategy game

    Get PDF
    Abstract Today's military strategy games provide unrealistic interfaces for players to interact with their units: Commanders don't use mice and menus, they sketch. Developing strategy games currently involves grafting AI capabilities on top of a separate simulation engine, with new hand-crafted strategies for each game. We are experimenting with a novel approach for solving both problems. We started with nuSketch Battlespace, a knowledge-rich sketch understanding system developed for military users, and built a game engine, nuWar, on top of it. nuWar is a prototype two-player tactical war game which can be played either hot-seat or over a network. nuWar uses sketching as the primary way for players to express their intent to their subordinate commanders. The underlying ontology used by nuSketch Battlespace is used in both the simulation engine and in the bots which serve as subordinate commanders. We describe the architecture of nuWar, focusing on how it uses sketching, how the simulation engine is built upon the rich representational facilities of nuSketch Battlespace, and how the bots work. We discuss the tradeoffs we have found in this approach so far, and describe our plans for future work

    Qualitative and quantitative reasoning about thermodynamics

    Get PDF
    Abstract: One goal of qualitative physics is to capture the mental models of engineers and scientists. This paper shows how Qualitative Process theory can be used to express concepts of engineering thermodynamics. This encoding provides the means to integrate qualitative and quantitative knowledge for solving textbook thermodynamics problems. These ideas have been implemented in a program called SCHISM, which analyzes thermodynamic cycles, such as gas turbine plants and steam power plants. We describe its analysis of a sample textbook problem and discuss our plans for future work.

    Analogical model formulation for transfer learning in AP Physics

    Get PDF
    AbstractTransfer learning is the ability to apply previously learned knowledge to new problems or domains. In qualitative reasoning, model formulation is the process of moving from the unruly, broad set of concepts used in everyday life to a concise, formal vocabulary of abstractions, assumptions, causal relationships, and models that support problem-solving. Approaching transfer learning from a model formulation perspective, we found that analogy with examples can be used to learn how to solve AP Physics style problems. We call this process analogical model formulation and implement it in the Companion cognitive architecture. A Companion begins with some basic mathematical skills, a broad common sense ontology, and some qualitative mechanics, but no equations. The Companion uses worked solutions, explanations of example problems at the level of detail appearing in textbooks, to learn what equations are relevant, how to use them, and the assumptions necessary to solve physics problems. We present an experiment, conducted by the Educational Testing Service, demonstrating that analogical model formulation enables a Companion to learn to solve AP Physics style problems. Across six different variations of relationships between base and target problems, or transfer levels, a Companion exhibited a 63% improvement in initial performance. While already a significant result, we describe an in-depth analysis of this experiment to pinpoint the causes of failures. Interestingly, the sources of failures were primarily due to errors in the externally generated problem and worked solution representations as well as some domain-specific problem-solving strategies, not analogical model formulation. To verify this, we describe a second experiment which was performed after fixing these problems. In this second experiment, a Companion achieved a 95.8% improvement in initial performance due to transfer, which is nearly perfect. We know of no other problem-solving experiments which demonstrate performance of analogical learning over systematic variations of relationships between problems at this scale
    corecore