In: Proceedings of The 3rd World Conference of the WWW, Internet, and Intranet, WebNet'98, Orlando, FL, November 7-12, 1998, AACE, pp. 92-97.
Peter Brusilovsky
Human-Computer Interaction Institute, School of Computer Science,
Carnegie Mellon University, Pittsburgh, PA 15213, USA
plb@cs.cmu.edu
John Anderson
Department of Psychology
Carnegie Mellon University, Pittsburgh, PA 15213, USA
John.Anderson@cmu.edu
Abstract: This paper presents an electronic ACT-R bookshelf, a system which supports learning ACT-R, a well-known theory in the field of cognitive psychology over the WWW. ACT-R bookshelf is a collection of electronic books on various aspects of ACT-R. The books on ACT-R bookshelf are authored and served on the WWW with InterBook system which makes most of the books on ACT-R shelf adaptive. This paper describes main functionality of the Bookshelf, provides some evaluation data, and speculates about possible extensions of Bookshelf-like systems.
This paper presents an electronic ACT-R bookshelf, a system which supports learning ACT-R, a well-known theory in the field of cognitive psychology [Anderson 1993; Anderson & Lebiere 1998], over the WWW. ACT-R bookshelf is a collection of electronic books on various aspects of ACT-R. The primary role of ACT-R bookshelf is to serve as an 24-hours available information resource for Carnegie Mellon University on-campus students taking course "Production System Models of Thought". However our intention was to build a system which can be used as a resource for distance learning of ACT-R, as well as an information resource for the international community of cognitive scientists and ACT-R researchers.
A hierarchically structured electronic book (EB) is one of the most popular metaphors for representing on-line course material. Virtually any kind of course material can be represented as an electronic book. Currently, the bookshelf contains an ACT-R tutorial, ACT-R 4.0 User Manual, a Manual for ACT-R Lisp-based environment, and an ACT-R Glossary. These books are quite different. The ACT-R tutorial is a kind of electronic textbook. It provides a step-by-step introduction into ACT-R theory and ACT-R modeling. The presentation is linearly sequenced and augmented with multiple examples, quizzes and assignments. ACT-R 4.0 User Manual is an encyclopedia-like book for reference access.
The books on ACT-R bookshelf are authored and served on the WWW with InterBook system [Brusilovsky, Eklund & Schwarz 1998]. Unlike most existing electronic textbooks which are not more than a static electronic copy of a regular textbook: chapter by chapter, page by page, picture by picture) most of the books on ACT-R shelf are adaptive. Adaptivity is especially important for educational programs on WWW which are expected to be used by very different classes of students without assistance of a real teacher (who usually can provide adaptivity in a normal classroom). An adaptive nature of ACT-R bookshelf is based on a specific concept-based approach suggested originally in [Brusilovsky 1995] and further elaborated by the ELM research group in the process of development an adaptive Web-based LISP textbook ELM-ART [Brusilovsky, Schwarz & Weber 1996]. InterBook is the first authoring system which implements this approach.
This paper uncovers concept-based knowledge representation behind adaptive electronic textbooks on the Bookshelf, describes main functionality of the system, provides some evaluation data, and speculates about possible extensions of Bookshelf-like systems.
The key to adaptivity in an adaptive textbook is knowledge about its domain (i.e., ACT-R) represented in the form of domain model and knowledge about individual students represented in the form of individual student models. The domain model serves as a basis for structuring the content of an adaptive EB. We distinguish two content parts in each adaptive EB: a glossary and a textbook . This section provides some minimal information about knowledge representation and content structuring. Some more information could be found in [Brusilovsky, Eklund & Schwarz 1998].
ACT-R bookshelf uses the simplest form of domain model: a set of domain concepts. By concepts we mean elementary pieces of knowledge for the given domain identified by a domain expert. The domain model provides a structure for an overlay student model which is a representation of the student's knowledge of the subject. For each domain model concept, an individual student's knowledge model stores some value which is an estimation of the student knowledge level of this concept. For ACT-R domain we distinguish for states of student knowledge of any concept: "unknown", "known" (learning started), "learned" and "well-learned".
An electronic textbook is a regular book represented in hypermedia form. In InterBook, each EB is hierarchically structured into units of different level: chapters, sections, and subsections. To make EB "more intelligent" and to connect it to the glossary, we have to let the system know what each unit of the textbook is about. It is done by indexing of textbook units with domain model concepts. Several books on the same subject form a bookshelf. All books from the same bookshelf are indexed with the same set of domain model concepts. Each terminal unit has an attached list of related concepts (we call this list spectrum of the unit). For each involved concept, the spectrum represents the name and the role of the concept in the unit (each concept can be either a outcome concept or a prerequisite concept). The system has an option to show all outcome and background concepts for the current section on a page border to the right of the section content (Figure 1)
The Glossary is, in fact, a visualized domain network. Each node of the domain network is represented by a node of the hyperspace, while the links between domain network nodes constitute main paths between hyperspace nodes. The links between domain model concepts constitute navigation paths between glossary entries. Thus, the structure of the glossary resembles the pedagogic structure of the domain knowledge. In addition to providing a description of a concept, each glossary entry provides links to all book sections which introduce or require the concept (Figure 2). This means that the glossary integrates traditional features of an index and a glossary.
Domain model-based indexing is a relatively simple but powerful mechanism, because it provides the system with knowledge about the content of its pages: the system knows which concepts are presented on each page and which concepts have to be learned before starting to learn each page. It opens the path for several adaptation techniques presented in this subsection.
The knowledge about the domain and about the textbook content is used to serve a well-structured hyperspace. As any well-designed EB, the system supports sequential and hierarchical links for navigation:
In addition the system uses knowledge about the concepts behind the pages to generate other types of links between the glossary and the textbook.:
All these links are not stored in an external format but generated on-the-fly by a special module which takes into account the student's current state of knowledge represented by the student model. This approach is not only reducing page design time but also provides room for adaptation. In particular, our approach supports two adaptation techniques: adaptive navigation support and prerequisite-based help.
The InterBook approach provides many more opportunities for browsing the course materials than traditional on-line textbooks. The negative side of this is that there is a higher risk for the student to get lost in this complex hyperspace. To support the student navigating through the course, the system uses adaptive annotation and direct guidance technologies. Adaptive annotation means that the system uses visual cues (icons, fonts, colors) to show the type and the educational state of each link. Direct guidance means that the system can suggest to the student the next part of the material to be learned.
The key to all adaptive functionality of ACT-R bookshelf is student modeling. The system maintains an up-to-date model of individual student knowledge on the subject. The student modeling mechanism accepts two kinds of evidences of student knowledge of a concept:
The latter evidence is stronger, so no "well-learned" grade can be given to a concept unless the student confirms his or her knowledge by answering a test.
Using the student model, it is possible to distinguish several educational states for each unit of EB: the content of a unit can be known to the student (all outcome concepts have been already learned), ready to be learned, or not ready to be learned (the latter example means that some prerequisite knowledge is not yet learned). The icon and the font of each link presented to the student are computed dynamically from the individual student model. They always inform the student about the type and the educational state of the unit behind the link. In InterBook, red means not ready to be learned, green means ready and recommended, and white means no new information. A checkmark is added for already visited units (Figures 1 and 2). The same mechanism can be used to distinguish and show several levels of students knowledge of the concepts shown on the concept bar. In InterBook, no annotation means "unknown", a small checkmark means "known" (learning started), a medium checkmark means "learned" and a big checkmark means "well-learned" (Figure 1). For many students, adaptive guidance provides enough support to make a navigation decision. Those who hesitate to make a choice could push the button "Teach me" and the system will apply several heuristics to select the most suitable node among those ready to be learned.
The system knowledge about the course material comprises knowledge about what the prerequisite concepts are for any unit of the textbook. Often, when students have problems with understanding some explanation or example or solving a problem, the reason is that some prerequisite material is not understood well. In that case they can request prerequisite-based help (using a special button) and, as an answer to help request, the system generate a list of links to all sections which present some information about background concepts of the current section. This list is adaptively sorted according to the student's knowledge represented in the student model: more "helpful" sections are listed first. Here "helpful" means how informative the section is to learn about the background concepts. For example, the section which presents information about an unknown background concept is more informative than a section presenting information about a known concept. The section which presents information about two unknown background concepts is more informative than a section presenting information about one concept.
By the Summer 1998 three groups of students (one group of 10 to 20 students per semester) learning ACT-R have been using the ACT-R Bookshelf as a learning resource. Most of the students were undergraduate or graduate Psychology students. At the first lecture of the course each group of students has 30 to 45 minutes introduction to ACT-R Bookshelf and its functionality. The Bookshelf was a primary source of learning information in addition to regular lectures. No printed handouts were provided.
It was our intention to evaluate how useful are the advanced features of the Bookshelf for Psychology students. Our standard evaluation technology in InterBook is comparing performance of InterBook users with a performance of a control group which uses a system with advanced functionality disabled [Brusilovsky & Eklund 1998]. Unfortunately, the number of students taking the ACT-R course is too small to run a control group. Instead, we have used a more subjective evaluation technology - a specially developed questionnaire. The goal of the questionnaire (41 questions in total) was to check whether the users understand the advanced functionality of the system (13 questions), whether they find it useful (24 questions), and what kind of improvements they could suggest (4 questions). We have run the questionnaire with one of the groups and collected 11 correctly filled forms. Due to the lack of space we could report only some most interesting results here.
Our major concern was that Psychology students will not be able to understand the advanced functionality of the Bookshelf even after a 30-45 minutes introduction. To check it we have developed 13 special questions to test their understanding. The results show that adaptive navigation support, the most "intelligent" functionality of the Bookshelf, is quite clear to the students. 11 or 11 students understand the role of the "green ball" and 10 of 11 understand the role of "red ball" and "checked ball". To compare with , less than half of students were able to understand the functionality of "Search" button, which provides standard search functionality (a feature of any advanced Web site). At the same time, the students have not considered adaptive navigation support as a very useful feature. Various components of it were rated 2.5 to 3.1 on a scale where 1 means totally useless and 6 means very useful. Here we should agree with the students. Adaptive navigation support is not very useful for well-organized sequentially read textbooks like the ACT-R textbook (here the best strategy is just read all units one after another). Adaptive navigation support was designed primarily for the users who need to learn selected parts of the ACT-R material and need to use "backward learning" to meet all prerequisites.
A concept-based indexing of EB on ACT-R bookshelf provide a unique opportunity of concept-based linking of various kinds of ACT-R related information. As soon as the ACT-R domain model (a consistent and comprehensive set of ACT-R concepts) is created and all book on the shelf are indexed with ACT-R concepts, these books are invisibly linked through the glossary. Each page of any book which is indexed with a concept will have a link to a glossary entry for this concept. In turn, a glossary entry for a concept will include the links to all pages in all books on the shelf which are indexed with this concept. Currently, the bookshelf contains primary the tutorial and manual-like books. However, we consider several useful extensions of the bookshelf.
There are multiple outcomes of having all ACT-R resources on a single bookshelf interconnected by a global glossary as presented above. Wherever the user starts, he or she can see the concepts behind the pages and see a glossary entry for any of these concepts. A glossary entry for a concept will provide links to all relevant information which exists on the bookshelf. For a introductory-level concept, the user will be able to see book, tutorial or manual pages explaining this concept. For more advanced concepts, the user will also see a list of papers dealing with this concept and a list of people investigating this concept deeply. We hope that such an integrated bookshelf accessible worldwide will seriously contribute for the promotion of ACT-R learning and research worldwide.
[Anderson 1993] Anderson, J.R. (1993). Rules of the Mind. Hillsdale, NJ: Lawrence Erlbaum Associates.
[Anderson & Lebiere 1998] Anderson, J.R., & Lebiere, C. (1998). Atomic Components of Thought. Hillsdale, NJ: Lawrence Erlbaum Associates.
[Brusilovsky 1995] Brusilovsky, P. (1995). Intelligent tutoring systems for World-Wide Web. Third International WWW Conference, Fraunhofer Institute for Computer Graphics. 42-45.
[Brusilovsky & Eklund 1998] Brusilovsky, P., & Eklund, J. (1998). A study of user-model based link annotation in educational hypermedia. Journal of Universal Computer Science, 4 (4), 429-448.
[Brusilovsky, Eklund & Schwarz 1998] Brusilovsky, P., Eklund, J., & Schwarz, E. (1998). Web-based education for all: A tool for developing adaptive courseware. Seventh International World Wide Web Conference. Computer Networks and ISDN Systems, 30 (1-7), 291-300.
[Brusilovsky, Schwarz & Weber 1996] Brusilovsky, P., Schwarz, E., & Weber, G. (1996). ELM-ART: An intelligent tutoring system on World Wide Web. In Frasson, C., Gauthier, G., & Lesgold, A. (Ed.), Intelligent Tutoring Systems (Lecture Notes in Computer Science, Vol. 1086). Berlin: Springer Verlag. 261-269.
The ACT-R manual and tutorial materials were converted from Word and HyperCard format into HTML format by Valeria Brusilovsky. Elmar Schwarz, the original developer of InterBook, seriously contributed to the development of the Bookshelf. Permanent support of Christian Lebiere, the author of the ACT-R manual was important for the success of the project.