VIREOS Operating System Project Lab Manual

Version 1.0 Marc L. Corliss

Hobart and William Smith Colleges corliss@hws.edu http://math.hws.edu/mcorliss

c 2010, Marc L. Corliss

This manual is licensed under a Creative Commons Attribution-NonCommercial-NoDerivs 2.5 License (http://creativecommons.org/licenses/by-nc-nd/2.5/). This license allows you to duplicate and distribute this manual in unmodied form for non-commercial purposes. See the license for more details.

About the Author

Marc L. Corliss is an Assistant Professor in the Mathematics and Computer Science Department at Hobart and William Smith Colleges. In 2006, Professor Corliss received his PhD from the University of Pennsylvania in computer science. His research interests are in system design, and in particular the design of compilers and processors. He is also interested in computer science education and building new tools for teaching computer systems courses.

Acknowledgements
This work arose out of an operating systems class taught at Hobart and William Smith Colleges by the author. First and foremost, the author wishes to thank the students in that course for their patience and useful feedback. The author also must thank Marcela Melara who ported this system to the Altera DE1 FPGA during a (productive) summer internship. This allowed the system to run on real hardware, potentially making it more appealing to future students. The author also wishes to thank Professor John Vaughn at Hobart and William Smith Colleges for his comments and feedback in various discussions on the project. Finally, the author wishes to thank the Provosts Ofce and the Department of Mathematics and Computer Science at Hobart and William Smith Colleges for their funding and support for this work.

Contents
1 Introduction 1.1 1.2 1.3 1.4 1.5 2 Project Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . About This Manual . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 3 4 4 5 6 6 7 8

VIREOS Toolset 2.1 2.2 2.3 2.4 2.5 Toolset Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Compilation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Simulation Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

File Conversion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 VIREOS on a FPGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 12 General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Registers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Traps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Memory Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Input/Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 System Management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 System Boot Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 26

Larc 3.1 3.2 3.3 3.4 3.5 3.6 3.7

VIREOS Overview 4.1 4.2 4.3

General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 System Calls . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Code Organization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 31

Project 0: Shell 5.1 5.2

Files and Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Project Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

5.3 5.4 6

Writing the Shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 Last Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 34

Project 1: Trap and I/O Handler 6.1 6.2 6.3 6.4 6.5 6.6

Files and Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Project Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41 Writing the Trap and I/O Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Testing the Trap and I/O Handler . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Last Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 46

Project 2: File System 7.1 7.2 7.3 7.4 7.5 7.6 7.7

Files and Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 Project Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Writing the File System Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 Testing the File System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59 Extra Credit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 Last Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 61

Project 3: Process Manager 8.1 8.2 8.3 8.4 8.5 8.6 8.7

Files and Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61 Project Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 Error Handling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Writing the Process Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 Testing the Process Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Extra Credit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 Last Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 78

Project 4: Memory Manager 9.1 9.2 9.3

Files and Directories . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Project Details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 79 Writing the Memory Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

vi

9.4 9.5

Testing the Memory Manager . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86 Last Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 88 89 89 92

10 Closing Remarks Appendices Appendix A: C-References

vii

Introduction

To gain a comprehensive understanding of computer systems, a computer science student must have a rm understanding of operating systems. Operating systems are responsible for managing the resources of the machine and providing abstractions (e.g., les, processes) to simplify software development [10]. Operating systems also illustrate the interaction between the hardware and software at a level much deeper than generally presented in an introductory computer systems course. To gain a rm understanding of operating systems, a computer science student must not only study the theory of operating system design but also implement one. As evidenced in many publications (too many to cite), students learn by doing. It is not enough to study how le systems work or how processes are managed. Students must actually implement a le system and process manager. This project provides a student studying operating systems with a set of assignments and resources, which guide them in the implementation of a simple (but still usable) operating system. The operating system is called VIREOS, which is short for Vanilla, Introductory, but Realistic, Educational Operating System. It is a Unix-like, operating system written in C. It is intended to run on a desktop or laptop computer (a very simple desktop/laptop) and, like other desktop/laptop operating systems, it includes support for processes, les, directories, and virtual memory. VIREOS runs on top of a Larc [3] machine, a MIPS-like education-based, 16-bit architecture. As there are no existing Larc machines, Larc is either simulated or congured on a eld programmable gate array (or FPGA). (We currently provide conguration les and support for conguring the Altera DE1 FPGA board [2] as a Larc machine instructors will need to adapt these conguration les to use other FPGA boards.) The simulator has the virtue that there is no hardware overhead. It can be advantageous when doing frequent testing of the OS. The FPGA, on the other hand, has the virtue that it is faster and more realistic. Students will certainly nd it more compelling to see (and build!) VIREOS running on real hardware. Students not only build a VIREOS kernel but also build a shell (or command-line interpreter). After all, an operating system is not very useful unless there are applications, which enable users to create processes, manipulate les and directories, and communicate with I/O devices. In most systems, there is one or more critical applications such as the Mac Finder and Windows Explorer that enable users to perform these kinds of tasks. In this manual, we call such programs core 1

applications. As the core application in this project, we use the shell as it is simple but powerful, text-based, and is used in all *nix systems. Although we mentioned above that the VIREOS kernel is programmed in C, it is actually programmed in a close variant of C called C--, which is described in Appendix 10. C-- contains a subset of the features of C although it includes most C features. We use C-- since it took less overhead to get a working compiler for the operating system and applications in VIREOS/Larc system. But in future versions of VIREOS, we plan to move to full ANSI C.

1.1

Project Goals

We had the following goals in mind when designing VIREOS. Built from scratch. We wanted students to write the OS basically from scratch with only a small amount of provided code. Provides good coverage. We wanted the operating system to contain most of the features of modern operating systems such as support for les, directories, processes, and virtual memory. Doable in one semester. We wanted to ensure that the operating system was doable in a one-semester course. For this reason, we avoided nested traps and complex I/O interfaces like those of commercial machines. Realistic. We wanted the system to be as realistic and appealing to students as possible, even given some of the simplications we have made. For this reason, we have ported VIREOS/Larc to an FPGA (Altera DE1 [2]). Integrated with other systems courses. We wanted the operating system to be integrated with our course in computer architecture. In particular, we wanted to use the same architecture, the classroom architecture Larc [4], in order to leverage the students experience and momentum. Uses a bottom-up approach. We wanted to use a bottom-up approach and have students build components on top of components they previously implemented. 2

 Exposes students to hardware. We did not want to hide the hardware details from the student. This helps give the student a more complete picture of the workings of the entire computer system. Includes the shell. We wanted students to not only write the kernel but also build a core application (e.g., Unix shell, Mac Finder, Windows Explorer) for manipulating les and directories and launching new processes. The core application is generally thought of as part of the operating system, and it is important for students to implement at least one such application. As the shell is the most basic core application, can be implemented quickly, and is critical component of a Unix-like system, we have students build a shell for VIREOS. As will be seen in later sections, these goals permeate the design of VIREOS, the assignments, and the resources.

1.2

Resources

There are several resources for helping both students and instructors install and work with this toolset. They are all freely available online at http://math.hws.edu/vireos. These include: Student manual. A comprehensive student manual (this manual), which documents all aspects of the VIREOS project (a PDF document). API. The application programming interface (API) of the operating system that students will build in this project as well as the API for user applications. Several tools. Several tools for building the operating system, running applications on this system, and debugging the inevitable errors that arise. In addition, instructors (and instructors only) can obtain solution code for the project by emailing the rst author of this manual as well as Verilog code for conguring an Altera DE1 FPGA [2] to run VIREOS.

1.3

About This Manual

This manual along with the corresponding toolset provide support code and documentation for course projects in an operating systems course. This manual does not discuss the theory or concepts of I/O handling, le systems, process management, memory management, etc. As such, it should be supplemented with a traditional operating systems textbook [8, 9, 10]. This manual also assumes that the student is following the design template laid out in the subsequent sections. It will offer limited help for those following a different design recipe. In addition, this operating systems project and this manual are intended for students who already know some C. It does not teach programming or programming in C, although it does discuss some of the features of C in Appendix 10. Students should be familiar with C or at the very least familiar with a similar language (e.g., C++). For those students who are less familiar with C, they may need to consult a C reference book [5], while reading this manual.

1.4

Future Work

We will explore several directions of future work. These include: FPGA Extensions. We will extend our FPGA implementation to include support for paging and persistent storage. We will also want to allow add support for persistently storing the FPGA processor conguration so that it can be automatically congured at start-up. Finally, we want to add support for automatically loading main memory from disk. Currently, the processor, main memory, and disk (stored in SDRAM) must be manually loaded at start-up. New I/O. We will explore new types of I/O for the VIREOS/Larc system. We want to incorporate new I/O devices such as a network card, graphical monitor, and/or mouse. These I/O devices will make the system more real to the student and potentially more appealing to work with. Interrupt-Driven I/O. We want to add one I/O feature, which was excluded for simplicity reasons: namely, interrupts. VIREOS currently has support for only one type of interrupt, a timer interrupt. This restriction avoids nested interrupts but at the expense of performance. In the future, we would like to provide support for using polling or interrupts depending on the instructors preferences and needs. 4

 Security. In addition to interrupt-driven I/O, VIREOS also does not include any kind of security. Like interrupt-driven I/O, we want to give instructors the option of adding security features (e.g., le permissions) based on their own particular classroom needs. ANSI C. We plan to migrate the VIREOS kernel and shell from C-- to C in the future. Using C-- saved us some initial overhead and could easily be customized for this project, but in the future, it would be nice to use full ANSI C. Other Bells and Whistles. Finally, we want to make the system more usable and closer to real systems. We believe the more realistic the system, the more compelling to the student. This includes adding new interesting applications, improving the look and feel of the system, and adding new functionality (e.g., up arrow in the shell). We will probably not be able to do all of these items but we hope to do many of them. In particular, we are currently working on the rst item (the FPGA extensions) and we hope to have this completed (in some form) by summer 2011. The other items represent longer term goals, which we may not get to for a while.

1.5

Outline

The remainder of this manual is organized as follows. Section 2 describes the VIREOS toolset and provides instructions for using it. Section 3 describes the Larc architecture, which VIREOS runs on top of. Section 4 gives an overview of the VIREOS operating system. Sections 5-9 contain the project assignments. Section 5 discusses the building of the shell, which is the core application in VIREOS. Sections 6-9 discuss how to implement the various components of VIREOS: a trap and IO handler, a le system, a process manager, and a memory manager.

VIREOS Toolset

This section describes the VIREOS toolset. It gives an overview of the toolset and then describes the compilation process, the simulation process, how to convert les to/from a VIREOS system, and, nally, using FPGAs to run a VIREOS/Larc system. Even if your toolset is already preinstalled, you should still read (or at least skim) these subsections, as they describe the process you will be using to build your operating system and to simulate applications on your built system, as well as how the project infrastructure is laid out.

2.1

Toolset Overview

The VIREOS toolset will compile and should run on any linux/x86 machine, although it is primarily tested on an Intel 32-bit x86 machine running Ubuntu Linux. It also may run on other *nix platforms and other architectures. It is available as a compressed tarball (vireos.tar.gz) at the website: http://math.hws.edu/vireos/. Once you have downloaded the tarball, move it to the appropriate location. To uncompress and untar the tarball, use the following:
bash$ tar xvfz vireos.tar.gz

This command will create a new directory, vireos/ (the root directory), in the current location. The vireos/ directory contains the following les and directories (note: directories end with /, les do not):
api/ bin/ lib/ man/ manual/ projects/

Each of these les and directories are described below. api. The api/ directory contains the API for the VIREOS Projects (minus project 0 for which it is unnecessary). The API is in the form of web pages, with a directory for each project (e.g., proj1). The API is generated from the source code via the application doxygen [11]. This documentation is also available online off of the VIREOS web page at http://math.hws.edu/vireos/. bin. The bin/ directory contains programs for running the reference VIREOS/Larc simulator, as well as various tools for building your own version of the operating system. lib. The lib/ directory contains library les required by the programs in bin/. man. The man/ directory contains man pages for the various tools used to build VIREOS, including

C-Program

C-Compiler

Larc Assembly Program

Larc Assembler

Larc Machine Program

Figure 1: VIREOS/Larc compilation process. the C-- compiler (C-- is described in Appendix 10), the Larc assembler, and the Larc simulator. manual. The manual/ directory contains this manual in the form of a PDF document. projects. The projects/ directory contains a directory for each project described in later sections of this manual (Sections 5-9). The contents of each project directory (e.g., proj0) are described in more detail in the corresponding project write-up section (e.g., Section 5). At the very least, each contains some C/C-- les, some of which must be implemented by the student, and a makele for building that project code. The kernel assignment projects (Sections 6-9) also contain a precompiled reference OS for testing against.

2.2

Compilation Process

The VIREOS compilation process is similar to the compilation process in commercial settings albeit with a couple of important differences. Figure 1 shows the (simplied) process when programming in the C-- language, the language that VIREOS is written in, which is discussed in Appendix 10. (Note: C-- is similar to C.) A C-- compiler converts the C-- compiler into assembly code for the target architecture, which is the educational architecture, Larc (described in detail in Section 3). Then, a Larc assembler converts the assembly code into machine code, which can be run on a Larc simulator. The compiler and assembler programs are located in the bin/ directory although each makes use of a library le within the lib/ directory. The compiler is named cmmc and the assembler is named asm. In addition there is an assembly code merge program (for separate compilation) called
merge in the bin/ directory.

The man/ directory has man pages on each tool, with details on how to

run them. Below we discuss the general usage of these tools. The C-- compiler has support for modules and some limited support for separate compilation. However, unlike in commercial systems, separately-compiled programs are merged into one program at the assembly level not at the object code level. As with standard C compilers, to compile C-- modules separately, the prototypes of all referenced functions must be dened. By default, the 7

Figure 2: VIREOS/Larc compilation process with multiple C-- les. compiler will fail if no main function is dened, however, this can be overriden via a -nomain ag. After compiling several C-- modules, the generated assembly les can be merged into a single assembly le using the assembly merger. Unlike with standard C compilers and assemblers, there can be no naming conicts, which means that the function names and global variable names in the original C-- modules must be distinct. Once the assembly les are merged into a single assembly le, the Larc assembler converts this assembly le into machine code (it can assemble a single le containing the whole program, which is why merging must be done prior to assembling). Figure 2 shows this augmented process with two source C-- les. We will use merging in the VIREOS project assignments in order to give students pre-compiled code from previous assignments. We also make use of merging when compiling user applications, as some code, such as making a system call, cannot be written at the level of C-- (or C). In this case, we put several assembly subroutines in a library le, which the C-- code can make use of. It should also be pointed out that Larc machine code, unlike in commercial systems, is stored in an ASCII le (each line contains one 16-bit word) to make it easier to work with in a class setting (Larc was designed for the classroom [4]). You can open (and even edit) Larc machine programs.

2.3

Simulation Process

As there are no existing Larc machines, Larc machine code programs are either run in a simulator (i.e., a Larc machine built in software) or on a pre-congured FPGA. We discuss the simulation process here (later on, we briey discuss FPGAs). A Larc simulator takes a Larc machine program as input and emulates the behavior (e.g., print characters to the monitor, wait for keys to be entered on the keyboard) of a real Larc machine running this particular program. The simulator is located inside the bin/ directory and is called sim. In general, it will not be run directly from bin/ but via a script in one of the project assignment working directories. These 8

Larc User Machine Program

Larc Simulator

Larc User Machine Program Larc Operating System Machine Code

Larc Simulator

(a)

(b)

Figure 3: Larc simulation process: (a) application and (b) full system mode. scripts will pass the appropriate ags to the simulator so, for example, it uses the operating system written by the student. A man page in the man/ directory describes the ags and options for running the simulator. Below we describe the general usage. The simulator has several different modes. As shown in Figure 3(a), it can be run in application mode where it runs a compiled user program and no operating system. In this case, the simulator will automatically handle the system calls, although it only supports a few of the basic calls (e.g., has no support for multiprocessing). We will rarely make use of this mode in this project. Figure 3(b) shows a second mode called full system mode where it runs a user program using a provided operating system. In this case, the simulator will transfer control to the operating system (in memory at address 0x0000) and the operating system will handle the trap. The student will be making use of this mode and they will be using a version of VIREOS, which they write part of in each project assignment. In fact, the simulator program runs in full system mode by default. Although not shown in Figure 3, the simulator can use a le to simulate a disk, which must be provided via the command-line. The le is treated as a sequence of disk blocks. On a disk operation, the simulator will access the appropriate block in the disk le. Like a real disk, the simulated disk will persist across system boots (unless the disk le is accidently removed or corrupted). If no disk is provided to the simulator, then it will fail on a disk operation. By default, the simulator will run with a disk le called disk located in the same directory as the simulator. There are two possible ways to provide the user program (the core application) and the operating system to the simulator. They can be provided via the command-line as described in the next subsection (as well as in a man page for the simulator). In this case, the simulator loads them automatically into memory prior to execution. This approach is not practical in a commercial machine but it makes testing much easier and, as a result, we will frequently make use of it. A second approach is to copy the user program and operating system from the simulated disk (in the boot sector) to memory. This approach is more realistic. If the user program and trap handler are not 9

provided to the simulator then it will it get them from the disk. If the disk is not provided nor the user program then the simulator will fail with an error message.

2.4

File Conversion

Text les, the primary type of le in a VIREOS or Unix system, are encoded differently in a VIREOS/Larc system than in a Unix sytem. Characters in VIREOS/Larc are encoded using 16 bits (like Unicode) and not 8 bits as in a Unix ASCII le. In general, this is not an issue as the VIREOS le system (stored on a disk le) and the local Unix le system are separate. But VIREOS/Larc has support for copying les to and from a remote le server when run on a simulator (not on an FPGA). The remote le server is actually the host le system where the simulator is running. The difference in le formats causes problems when performing remote le transfer. For this reason, there is a conversion program called conv for converting a VIREOS text le to a Unix text le. (The program currently does not support converting in the other direction we have not found a need for this yet.) This program runs on the Unix machine and should be used to convert a le prior to transferring it to the VIREOS/Larc system. It is located in the bin/ directory. A man page in the man/ directory describe the various options, some of which are discussed below. The output le generated by conv is supplied via a -o ag (if not, out is used) and the input le is supplied with no ag. See the man page for more details. This conversion program can be used to convert Larc machine programs as well as regular text les. On a Unix system, a Larc machine program is stored as an ASCII le with each line containing one 16-bit word. On VIREOS, it is simply a binary le. If the -m ag is used with the conversion program then it convert a Larc machine program. Otherwise, it defaults to converting a standard ASCII le.

2.5

VIREOS on a FPGA

A VIREOS/Larc system can be run on an FPGA in addition to simulated. In particular, we have congured an Altera DE1 FPGA [2] to run as a Larc processor with full architectural support for VIREOS. Although the FPGA is probably not suitable for use in assignments, it makes for a nice demonstration of a realistic VIREOS/Larc system. The hardware conguration les were written in Verilog. They can be obtained by instructors (and instructors only) by emailing the author (expect some delay as each request must be validated). 10

Using the Quartus II software, the DE1 FPGA can be congured with the specication in Verilog. The details for installing and running on an FPGA are available along with the Verilog specication les. There are some aspects of the DE1 version of Larc that remain to be implemented. In particular, the FPGA implementation does not support paging. Furthermore, the disk is currently stored on an SDRAM rather than on a persistent storage medium. In fact, both the Larc memory (stored in SRAM) and the disk must be loaded at startup time using the DE1 control panel, in addition to the processor. Moreover, many of the safety processor exceptions and checks are not currently implemented in the Verilog specication. However, the Verilog implementation does correctly run VIREOS (at least, minus paging); it has support for the Larc I/O devices, the le system, and time sharing.

11

Larc

This section describes the Larc architecture, which VIREOS runs on top of. Larc is a classroombased, simplied architecture [3]. There are no existing Larc machines; instead Larc programs are run on a software simulator or on a pre-congured FPGA. Because of its simplicity, it is an ideal platform for a novice operating system (OS) developer. As VIREOS is programmed in C, many of the details of Larc (e.g., assembly and machine instructions) can be ignored. This section will focus only on the details that are relevant to VIREOS. For more information check out the Larc lab manual [3]. The section is organized as follows. Section 3.1 describes the general characteristics of the architecture. Section 3.2 describes the registers and the conventions for using them. Section 3.3 describes how trapping and system calls work on a Larc machine. Section 3.4 describes the memory organization and layout in a Larc machine. Section 3.5 describes input/output on a Larc machine. Section 3.6 describes the hardware support available to the OS for managing user processes. Finally, Section 3.7 describes the boot process.

3.1

General

Larc is a simplied version of the MIPS [7] architecture, intended for the classroom. Like MIPS, Larc is a Reduced Instruction Set Computer (RISC). As a result, it has a simple, xed-width instruction set with a small number of addressing modes. Although the machine encoding of instructions differs between MIPS and Larc, its assembly instruction set is a subset of the MIPS assembly instruction set (at least if using pneumonic register names and not using assembler reserved registers). Although Larc is small, it is still a fully-functional and practical architecture (i.e., it is advanced enough to build an OS for!). Larc is a 16-bit wide architecture; each register and each word in memory contains 16 bits of data. For this reason, instructions are also 16 bits. Likewise, a memory address is 16 bits long, which means that the address space is 216 . Because a word in memory contains two bytes (i.e., 16 bits), the total size of the address space in bytes is 217 bytes or 128 KB. Larc is word addressable only. Data that is shorter or longer than 16 bits (bytes, double words, etc.) cannot be retrieved from memory in a single operation. Note that the width and addressability of Larc are different than in MIPS, which is 32-bit wide and byte addressable.

12

Register $0 $1 $2 $3 $4 $5 $6 $7 $8 $9 $10 $11 $12 $13 $14 $15

Name $zero $v0 $a0 $a1 $t0 $t1 $t2 $s0 $s1 $s2 $sp $ra $at0 $at1 $k0 $k1

Function always holds zero result register argument register argument register temporary register temporary register temporary register saved temporary register saved temporary register saved temporary register stack pointer register return address register assembler register assembler register OS register OS register

Table 1: Larc registers. There are three built-in data types in Larc: signed integers, bitmaps, and characters. All the built-in data types are 16-bits wide. Obviously, Larc has support for arithmetic operations for manipulating integers. These integer operations work with signed values; unsigned arithmetic must be simulated in software using signed arithmetic. An integer can also be treated as a bitmap, i.e., a sequence of bits. Multiple integers can be used to represent larger bitmaps. Bitmaps have similar uses as boolean arrays. There are bitwise logic functions (e.g., AND, OR) for manipulating a single, 16bit bitmap. Finally, Larc has support for printing characters to a (non-graphical, character-based) monitor, reading characters from a keyboard, and performing arithmetic on a character. Each character is 16 bits wide, allowing for a Unicode representation, although currently ASCII is used to represent each character (only the lowermost 8 bits are used). Strings of characters are represented using several contiguous characters in memory, which is terminated by a 0 value or null word. Because almost all values take up an entire word in memory (16-bits), endianness (or byte ordering) is irrelevant. In special cases, where endianness does arise, Larc is a big-endian architecture, meaning the most signicant byte is the leftmost byte in a word.

13

3.2

Registers

Larc has sixteen registers, shown in Table 1. Many of the registers have a specialized use: either dictated by the Larc processor or a specied software convention. The rst register is a zero register ($0 or $zero), which always holds the value 0. It is convenient to have such a register, and many architectures, like MIPS and Larc, have one. Registers $1$11

are general-purpose registers with no specialized use. There are conventions, however, that

recommend how each of these registers should be used. For instance, by convention, register $1 or $v0 is used to hold the result of a subroutine, i.e., the subroutines return value. Registers $2 ($a0) and $3 ($a1) are used to hold the rst two arguments to a subroutine. If a subroutine requires additional arguments then they should be passed via memory. (Note: some compilers may choose to use memory for all arguments.) Registers $4-$9 are temporary registers. By convention, registers $4-$6 ($t0-$t2) are not preserved across a subroutine call. Registers $7-$9 ($s0-$s2) are preserved across a subroutine call. In other words, a called subroutine must save these registers to memory before using them and restore them to their original values before returning to the caller (i.e., the subroutine that performed the call). Register $10 or $sp is used to hold the stack pointer, which points to the stack in memory. To review, the stack is a region of memory, which houses the state of each currently-executing subroutine. Because calls and returns of subroutines follow a stack pattern, a stack data structure is used to save the state of each executing subroutine (hence the name stack). Larc does not provide hardware support for managing the stack or the stack pointer. It must be managed explicitly by the program (or compiler). If a frame pointer is needed, i.e., a pointer to the start of the topmost stack frame, by convention $7 or $s0 is used for this purpose (there is no explicit frame pointer). Register $11 or $ra is used to hold the return address on a call to a subroutine. This address allows the called subroutine to link back to the call site. The caller places the address of the instruction following the call in register $11 when performing a call. When nished, the called subroutine can then transfer control to the address in register $11. Registers $12 ($at0) and $13 ($at1) do have a specialized purpose; they are reserved for use by the assembler. These registers are used to translate extended assembly instructions into several machine instructions. They essentially allow the assembler to support a larger class of assembly instructions. They are generally not used explicitly by the programmer, but will be used in the

14

Trap Identier 0 1 2 3 4 5 6 7

Trap type System initialization System call Timer interrupt Page fault Illegal memory reference error Illegal register access error Illegal instruction executed error Divide by zero error

Table 2: Larc trap types. assembled program after the assembly code is translated into machine code. Registers $14 ($k0) and $15 ($k1) are kernel registers reserved for use by the operating system. They can be used, for example, to save and restore non-kernel registers without modifying any non-kernel register. For the most part, the register conventions will not be relevant since VIREOS is programmed in C. However, the operating system will be responsible for saving and restoring a processs state, which includes registers, and in doing this, you will need to know some of these conventions. For example, only registers $1-$13 need to be saved and restored as $14 and $15 are kernel registers and $0 always holds 0.

3.3

Traps

There are many circumstances in which the CPU must transfer control to the operating system, which is called a trap. For example, if a user process attempts to access an illegal memory address then the CPU will perform a trap. On a trap the CPU postpones the currently-running process and transfers control to the to the trap handler, a component within the operating system for handling various traps. When a trap occurs, the CPU writes a trap identier into the process control register (in 7 of the 16 bits) where it can be retrieved by the trap handler (the process control register is discussed further in Section 3.6). The trap handler services the trap and then transfers control back to the user process or possibly runs a different user process. A trap can occur for a variety of reasons. Table 2 shows all the traps that are supported in a Larc machine along with the corresponding trap identier number. Trap 0 occurs when the machine is rst booted. This trap enables the operating system to set up memory and registers with the initial application (i.e., core application). Trap 1 occurs on a system call, which is discussed below. Trap 15

0x0000 Kernel space

0x9000 User space 0xFC00 Input/output space 0xFFFF

Figure 4: The memory organization of a Larc machine. 2 occurs when the machine timer goes off, which allows the operating system to multiplex the CPU between several processes and support timesharing. This trap is only relevant in the later project assignments, which include process management. Trap 3 occurs on a memory page fault, i.e., when a part of the address space currently housed on disk is accessed. This trap is only relevant in the last project assignment, which includes memory management. Traps 4 through 7 occur as the result of a programming error in the user program: (4) accessing an illegal memory address, (5) accessing a kernel register, (6) executing a system instruction (i.e., a system return instruction, which is discussed below), or (7) a divide by zero error. One important reason for trapping is so the operating system can handle a system call. System calls are essentially the interface to the operating system for the user process. They are like functions, called by the user process but executed in the operating system. They allow user processes to do things such as access I/O devices, manipulate les and directories, interact with other processes, and terminate. System calls abstract the underlying machine making it easier to write user-level programs. They also protect programs from potentially misusing resources. When the system call executes, the CPU traps to the operating system. Prior to the system call, the user process puts values in pre-dened registers to tell the trap handler what kind of system call is being executed and what the system call parameters are. In particular, the type of the system call is passed in register $1 ($v0) and the parameters (if needed) are passed in registers $2 ($a0),
$3 ($a1),

and $4 ($t0). If the system call has a return value it is returned in $1 or $v0. System calls

will be discussed further in Section 4, which gives an overview of VIREOS.

16

0x0000 Instructions (m words) m Static data (n words) m+n Heap (o words) m+n+o Unused memory (p words) m+n+o+p Stack (0x6C00-m-n-o-p words) 0x6BFF

Figure 5: The general layout of a processs address space in Larc.

3.4

Memory Organization

Figure 4 shows the memory organization in a Larc machine. The kernel is placed in memory starting at address 0. Although not shown in Figure 4, the trap handler is always the rst component of the kernel with the start of the handler residing at address 0. On a trap, the CPU transfers control to address 0. The rest of the kernel components (e.g., le system, process manager) reside after the trap handler. After the kernel is the user space. This is occupied by one or more processes. The start of user space (and the end of kernel space) is at address 0x9000. The I/O space follows the user space starting at address 0xFC00. The I/O space is used to communicate with I/O devices, which is discussed in more detail in the following subsection. Figure 5 shows the address space of a user process. Note: this gure shows a logical view of the processs memory. However, the address space need not look like this in physical memory. For instance, it might not be contiguous. In addition, parts of the address space might be stored on disk at any particular moment. Although much of the layout of the address space is not pertinent to the operating system developer some of the details are important. First, the address space goes from address 0x0000 17

to address 0x6BFF. This range differs from commercial systems, which generally use the entire addressable space (up to 0xFFFF in a 16-bit system). However, limiting the space to size 0x6C00 allows a single processs address space to t entirely in physical memory. It will also t within a single le (which is limited to size 0x7FFF as discussed in the le system project assignment), which simplies process management in VIREOS. By convention, the address space starts with the processs instructions, followed by the processs static data (data that comes within the machine code program and does not grow), and followed by the processs heap (data that is allocated dynamically), which grows towards larger addresses. This layout could vary in some processes but the rst instruction of every process must reside at address 0x0000 in the (logical) address space. On the other end of the address space is the stack, which grows towards smaller addresses. Initially, the stack pointer ($10 or $sp) starts at value 0x6C00 (the other non-kernel registers are initialized to 0). This will vary slightly if there are arguments passed to the user program. In this case, the arguments are passed on the stack and the stack pointer will be adjusted accordingly (this issue is discussed in further detail in the project assignment on process management). The stack and the heap could technically run into one another. It is left to the user program to avoid this error.

3.5

Input/Output

Larc currently has support for ve input/output (I/O) devices: a keyboard, a character-based (nongraphical) display monitor, a hard disk, a remote le transfer device (a simplied USB stick only supported when run on a simulator and not on an FPGA), and a clock. The keyboard allows for user input, while the monitor allows for program output. The hard disk provides a larger persistent storage, which a le system can be built on top of. The remote transfer device allows for les to be transferred to or from a remote device, like a USB stick or CD-ROM (only remote le transfer is supported and not remote le system mounting). The clock is required for providing timesharing (i.e., multiplexing the CPU between several processes) and to allow programs to obtain the current time. Special device registers, accessible via memory, are used for communication between the CPU and the controller of each I/O device. The keyboard and monitor are character-based devices. When a key on the keyboard is pushed, the keyboard device controller stores the character corresponding to the pushed key in a device register. When the CPU needs to send a character to the monitor (which is all that the monitor 18

supports), the CPU stores the character in a device register, which eventually gets printed to the screen. Some synchronization is required for this to work properly as the keyboard and monitor are much slower than the CPU. The hard disk is a block-based device. The CPU and disk controller can pass blocks of 256 words (1/2 KB) back and forth via a set of device registers. To write a block to the disk, the CPU stores the values to write in a set of data device registers and stores the block number in a block device register. It also sets a bit in a control device register to 1, meaning perform a write. To read a block, the CPU stores the block to read in the block device register and sets a bit in the control device register to 0, meaning perform a read. The disk controller will then copy the data into the set of data device registers, which can be read by the CPU. As with the keyboard and monitor, some syncrhronization is required for this to work properly as the disk is much slower than the CPU. The remote transfer device is similar to the hard disk. It is a block-based device where blocks of 256 words can be passed back and forth between the local disk and a le system on a remote device (e.g., USB stick). (The remote le system is actually the host le system where the simulator is running.) On top of the device, the OS can provide support for le transfer to and from the local and remote le systems. As with the hard disk, there is a set of data registers for transferring a block between the disk and remote le system and a control register for specifying whether to read from the local disk or write to the local disk. There is also a register holding the block number to read/write, however, this references a block in the remote le (it is a le block number not a disk block number). In addition, there are several other registers: one for holding a pointer to the name of the remote le to access; one for holding the maximum length to read; and one for storing the number of words that were actually read/written. The nal device, the clock, is neither a character nor a block device. This device has three device registers for nding the current time, i.e., seconds since epoch (January 1st, 1970 GMT). More than one register is required since Larc is a 16-bit architecture and the number of seconds will not t within 16 bits. The rst register holds the days since epoch, the second register holds the minutes in the current day, and the third register holds the seconds in the current minute. The clock can also act as a timer. If a non-zero value is written into a start timer device register then the timer is invoked. The amount written into the start timer is interpreted as the number of unsigned milliseconds until the timer should go off. A second timer device register contains the current timer amount at any particular instance (it cannot be written). When the timer reaches 0, the CPU traps to the operating system. If the start timer is 0, then the timer is disabled. Every time 19

// print a single character c to the screen print char(char c) { while (monitor control register is non-negative) {} set monitor data register to c // executed after loop completes }

Figure 6: Pseudocode for polling the display monitor. the start timer is written, the timer is reinitialized, even if it is set to the same value. I/O takes orders of magnitude longer to perform than executing an instruction in the processor. As a result, the processor functions at a much higher speed than the I/O devices. We say that the processor and I/O devices operate asynchronously. One challenge then is to synchronize the processor and I/O devices. Note: the clock does not require synchronization with the CPU but the other devices do. In Larc, synchronization is achieved with a simple ready bit. Each device besides the clock has a ready bit associated with it, which indicates when data can be either retrieved from the device or sent to the device. The ready bit is contained within the control register for that particular device. It is the lowest order bit in the control register. When the ready bit within the control register is set then it is safe to send or retrieve data. Otherwise, it is not safe to send or retrieve data, and so, the program must wait. For the keyboard and monitor, the ready bit is the only used bit in the control register. As mentioned above, one other bit is used in both the disk and remote transfer control registers: a bit indicating whether the CPU is reading or writing the disk. The CPU uses polling to determine when the device is ready. The CPU continuously checks if the ready bit within the corresponding control register is set. The pseudocode for polling the monitor device is shown in Figure 6. Currently, interrupts are not supported for determining when a device is ready (a few interrupts are supported). In future versions of VIREOS support for such interrupts may be added. However, because VIREOS is a single-user system and assuming there is adequate buffering in the I/O device controllers, this is not that severe a limitation. All of the device registers are listed in Table 3. Each device except the clock has a control register. All of the devices have one or more data registers. Like other Larc registers, these are 16 bits wide. Larc uses memory-mapped I/O, meaning that parts of the address space are dedicated to particular I/O device registers. To read or write these registers, the programmer can read or write memory (with a load or store instruction, respectively). Without memory-mapped I/O, special instructions 20

Name kcr kdr mcr mdr tdr1 tdr2 tdr3 dcr dbr ddrs rcr rfr rbr rlr rdrs rvr

Address 0xFC00 0xFC01 0xFC02 0xFC03 0xFC04 0xFC05 0xFC06 0xFC07 0xFC08 0xFC09-0xFD08 0xFD09 0xFD0A 0xFD0B 0xFD0C 0xFD0D-0xFE07 0xFE08

Description keyboard control register keyboard data register display control register display data register time data register 1 (days since epoch) time data register 2 (minutes in current day) time data register 3 (seconds in current minute) disk control register disk block register disk data registers (0-255) remote transfer control register remote transfer lename register remote transfer block register remote transfer length register remote transfer data registers (0-255) remote transfer value register

Table 3: Larc I/O device registers. would be required to read and write device registers. The advantage of memory-mapped I/O is that it uses pre-existing instructions to read and write device registers. The disadvantage is that parts of the address space are not available for general use by the program. Because the address space is much larger than the instruction set, most architectures, like Larc, use memory-mapped I/O. Figure 4 shows the part of the address space that is mapped to I/O. The region starts at 0xFC00 and ends at 0xFFFF (the highest address). Note that there are some parts of the I/O space, which are not mapped to a device register. It is reserved for future extensions (e.g., new devices). For safety reasons, this region of memory is not for use by user programs in general although it is up to the operating system to enforce this restriction (which is discussed in the next subsection). In general, if the user process needs to communicate with an I/O device it should do so via a system call, which will be handled by the operating system. If the OS does restrict the user process from accessing the I/O memory region, a trap will occur if the program attempts to access it.

3.6

System Management

Several memory-mapped registers in the I/O space are for managing the system as opposed for communicating with devices. These are shown in Table 4. Processor control register. There are several memory-mapped registers dedicated to system man21

Name tlb mem pfault tmr val tmr start mem base mem limit sys ra pcr

Address 0xFFC3-0xFFF8 0xFFF9 0xFFFA 0xFFFB 0xFFFC 0xFFFD 0xFFFE 0xFFFF

Description TLB entry registers (0-53) memory page fault address register timer current value register timer start value register memory base address register memory limit value register system return address register processor control register

Table 4: Larc memory-mapped system management registers. agement. The rst (starting at the bottom of the I/O space) is the processor control register or PCR. It is a multi-purpose register. Figure 7 shows contents of the PCR. The low order bit (rightmost) is a halt bit. If set by the operating system, the machine will halt. The next lowest 7 bits store the process identier (PID) of the currently executing process. Both the halt bit and PID are set by the operating system and read (but not written) by the CPU. The next 7 bits (low order 7 bits in the leftmost byte of the PCR) store the trap type. On a trap, the CPU stores the trap identier into this eld of the PCR. The operating system can read this type to determine the particular trap that occurred. Finally, the uppermost bit (leftmost bit) indicates whether the system is in user mode (1) or kernel mode (0). Like the trap type, it is set automatically by the hardware and should not be changed by the operating system. On a trap, this bit is set to 0 since the trap handler in the operating system will be executed next. On a return back into the user program (called a system return in Larc) this bit is set to 1. System return address register. The next system register (at address 0xFFFE) holds the system return address. This is the address to return to in the user program. When the trap handler completes, it executes a special Larc instruction called a system return. This instruction causes control to transfer to the address stored in the memory-mapped register holding the system return address. It also causes the user mode bit in PCR to be set to 1. Note: only a process in kernel mode can execute this instrution; a trap occurs if a user process attempts to perform a system return. On a trap in user mode, the CPU automatically writes the return address into this memory-mapped register (using the PC+1 for a system call trap, or PC for a non-system call trap). As a result, if only executing a single user process, this register need not be modied by the operating system. However, when executing several processes, the operating system will need to read and write this address in order to save the PC of the currently-executing process and to restore the PC of the next 22

user mode 1

trap identier 7

process identier 7

halt bit 1

Figure 7: Contents of PCR. process to run. Timer registers. The next two registers, the current and start timer values (at addresses 0xFFFA and
0xFFFB, respectively), enable the operating system to set and manage a timer.

This timer is used to

support timesharing. The current timer value register holds the current timer value in milliseconds. It counts down to 0 at which point the timer device triggers an interrupt. Note: the interrupt occurs only while in user mode (to avoid nested interrupts), however, the operating system can check the timer while in kernel mode. The current timer register should be read but not written. The start timer is used to initiate the timer. When set, this value is copied to the current timer value and the timer is initiated. If the start register is set while the timer is already counting down then the timer is re-initiated at the new starting value. The timer will not automatically re-initiate when 0 is reached. The start register must be set again. The start timer register can be read or written although it is primarily meant to be written. Memory base and limit registers. The next two registers, base and limit (at addresses 0xFFFC and 0xFFFD, respectively), provide a primitive way for managing memory. These registers enable the OS to sandbox a process in memory. The base register holds the base memory address for the currently running user process. If the process is in user mode then the (unsigned) address within the base register is automatically added to every memory address used by the process. Therefore, the address in the base register, is effectively the starting address of the process (called address 0 by the process). The value in the limit register denes the total size allotted to the process. The process cannot reference a memory address that is greater than or equal to the address in the base register plus the value in the limit register. On every memory reference, the processor compares the computed memory address, before adding it to base address, to the value in the limit register. If the computed memory address is greater than or equal to the value in the limit register, then the processor automatically traps to the operating system. Paging registers. The base and limit registers can be used to implement swapping, allowing several processes to simultaneously share physical memory. However, Larc also has support for paging. As you will see in the memory management project assignment, paging allows physical

23

valid bit 1

page number 7

process identier 6

mod. bit 1

read bit 1

Figure 8: Contents of a TLB entry. memory to be broken up into xed-size frames where each frame holds a section of the address space of some process called a page. Paging allows multiple processes to efciently share physical memory. The address space for any particular process is stored non-contiguously in physical memory (pages can be scattered throughout physical memory) and some of it can even be housed temporarily on disk. When an accessed page is not in physical memory, the CPU must trap to the operating system, so the operating system can retrieve the requested page from disk. This event is called a page fault. When a page fault occurs, the faulting virtual address is stored in the mem pfault memory-mapped register (address 0xFFF9). The operating system can read from this register to determine which page needs to be brought into physical memory. This register should only be read by the operating system and not written. To support paging, the CPU must have hardware support for translating a user programs logical memory address (called a virtual address) into a physical address. Translation on a Larc machine is done via a translation lookaside buffer (TLB), i.e., a hardware cache of the page table. In fact, as physical memory is small, no memory-bound (or worse, disk-bound) page table is required. Each entry in the TLB corresponds to a particular physical frame. It species the page that is occupying the current frame. Because multiple processes share physical memory it must also contain the PID of the owning process. Figure 8 shows the contents of a TLB entry. In addition to the page number and PID, the TLB houses a bit indicating whether the entry is valid (it might be invalid, for example, if the owning process has terminated), and whether it has been read or written (useful for handling page faults). The TLB is a fully-associative cache; on a memory reference the CPU searches its entire contents for a valid entry with a page number that matches the requested one and the PID that matches the current PID in the PCR. If a match occurs, the frame number of the frame that corresponds to this TLB entry replaces the page number in the memory address. Otherwise, a page fault occurs and the CPU traps to the operating system. The TLB is programmed by writing to memory-mapped addresses 0xFFC3 through 0xFFF8 (0xFFC3 refers to the rst entry in the TLB, 0xFFC4 refers to the second, and so on). Each page 24

has size 512 words (or 1024 bytes). Only the user space is paged and not the kernel or I/O space. Since the user space has length 0x6C00 and a page size is 512, the number of total paged frames is 54. It should be pointed out that 54 is a larger-sized TLB (64 is the maximum size one will see in modern machines) and that in some highly-optimized machines it might be difcult to make a TLB with this size. However, this TLB conguration makes the job of the OS developer much easier.

3.7

System Boot Process

When a Larc machine is rst booted the operating system and core application (e.g., shell process) must be loaded into memory. The CPU copies the operating system and core application from the boot blocks on the disk into memory. Unlike in commercial machines, the entire operating system and core application are stored on the boot blocks rather than a small amount of code (e.g., BIOS, LILO) responsible for loading the operating system and core application code from other parts of disk. This simplies the boot process. Moreover, because Larc is a 16-bit machine (with a 16-bit address space), the operating system and core application take up very little disk space (<= 256 blocks). A bigger problem is that an error in the operating system or core application could be a potentially semi-permanent error. For example, imagine an error in the operating system causes the system to crash shortly after booting. To x this error would require reloading the boot block code after mounting it on a separate machine. When developing and testing the operating system, we can use a second option for booting the system; we can manually load memory at start-up. This way, if a castrophic mistake is made when writing the OS or core application, it is not permanent. When running on the simulator, if a core application le and kernel le are specied on the command-line then these are automatically loaded into main memory at start-up. When running on the FPGA (the Altera DE1 [2]), we can manually load these les into memory using the conguration software. After loading (physical) memory with the OS and core application, the CPU performs a system initialization trap. The trap handler is invoked, which then can initialize the system management memory-mapped registers such as the memory base and limit registers, for example. After setting up these registers and any other state, the OS can start running the core application.

25

VIREOS Overview

This section gives an overview of the VIREOS operating system, which students will implement in the project assignments. Section 4.1 describes the general characteristics of the VIREOS operating system. Section 4.2 enumerates and discusses the supported VIREOS system calls. Section 4.3 presents the organization and layout of the VIREOS source code, which students will be completing.

4.1

General

VIREOS is a *nix-like operating system designed for the classroom. As discussed in the next subsection, it supports many of the same system calls (e.g., fork, exec). It also uses many of the same internal data structures (e.g., i-nodes). VIREOS is timeshared, meaning it can simultaneously run several processes. It uses paging to allow multiple processes to efciently share memory, and to provide each process with a logical address space. It includes a le system with support for both les and multi-level directories. It supports ve input/output devices: a keyboard, a character-based (non-graphical) monitor, a hard disk, a remote transfer device (e.g., USB stick), and a clock. But because VIREOS is a classroom OS, it has been simplied in several ways. First, it mostly does not provide security. For example, it does not have support for multiple users nor does it do any authentication. It also does not provide any support for protection domains (which would not make sense in a single user system anyway). There are only a few supported I/O devices and these are all fairly simple devices (e.g., a character-based monitor). VIREOS also uses polling to communicate with these I/O devices. Although this is highly inefcient, it greatly simplies I/O and trap handling. There is only one supported interrupt: a timer interrupt (triggered by the clock) for implementing timesharing. To further simplify VIREOS, the timer interrupt can only trigger while in user mode, which prevents the operating system from having to handle nested interrupts. Like many commercial operating systems, VIREOS is written in a C-like language. The language, called C-- is described in Appendix 10. Unlike commercial systems, VIREOS is entirely written in C (or more accurately, C--); none of it is written in assembly code. This simplies the implementation of VIREOS at the expense of some loss in performance. To make this work correctly, the compiler does have to provide some extra support. In par-

26

Class

Name halt print str* print int* read str* read int* time

ID 0 1 2 3 4 5

Description Halt system Print a string to the monitor Print an int to the monitor Read a string from the keyboard Read an int from the keyboard Get the current time (GMT) None

Arguments String to print ($2), maximum length to print ($3), Integer to print ($2) Pointer to write read string ($2), maximum length to read ($3), Nothing 3-entry int array ($2) (days since epoch in 0th entry, mins. in day in 1st, secs. in min. in 2nd) Disk block number to read ($2), buffer to write read block ($3) Disk block number to write ($2), buffer to write to disk ($3) Pointer to remote le pathname ($2), le block number to read ($3), buffer to write read words ($4) Pointer to remote le pathname ($2), le block number to write ($3), buffer of words to write ($4) None Pointer to pathname of le to create ($2) Pointer to pathname of le to open ($2) Handle of le to read from ($2), buffer to write read words ($3), maximum length to read ($4) Handle of le to write to ($2), buffer of words to write ($3), maximum length to write ($4) Handle of le to seek in ($2), seek offset ($3), seek type ($4) Handle of le to truncate ($2), new logical le size ($3) Handle of le to close ($2) Pointer to pathname of le ($2) Source le handle ($2) target le handle ($3) Pointer to buffer of two ints ($2) where handles are written (read handle in 1st entry, write in 2nd) Pointer to pathname of directory to create ($2) Pointer to pathname of directory to open ($2) Handle of directory to read from ($2) Handle of directory to close ($2) Pointer to pathname of directory to delete ($2)

Returns Doesnt return Number of printed characters if successful, <0 on error ($1) Number of printed characters if successful, <0 on error ($1) Number of read characters if successful, <0 on error ($1) Read integer ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) Number of read words (< block size if end of le reached) if successful, <0 on error ($1) Number of written words (< block size if max. size reached) if successful, <0 on error ($1) Nothing File handle if successful, <0 on error ($1) File handle if successful, <0 on error ($1) Number of read words if successful, <0 on error ($1) Number of written words if successful, <0 on error ($1) New seek pointer if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 is successful <0 on error ($1) 0 if successful, <0 on error ($1) directory handle if successful, <0 on error ($1) number of read words if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1)

I/O

dread* dwrite* rtread

6 7 8

Read a block from disk Write a block to disk Read a block from a remote le

rtwrite

Write a block in a remote le

dformat fcreate fopen fread

10 11 12 13

Disk format Create a new le Open an existing le Read from an open le

fwrite

14

Write to an open le

fseek ftrunc File System fclose fdelete fdup2 fpipe

15 16 17 18 19 20

Seek in an open le Truncate (or enlarge) an open le Close an open le Delete an existing le Duplicate a le descriptor Create a new pipe and handles (read and write) for accessing it Create a new directory Open an existing directory Read from an open directory Close an open directory Remove an existing directory and all of its subcontents

fmkdir fopendir freaddir fclosedir frmdir

21 22 23 24 25

Table 5: VIREOS system calls. * indicates system calls for instructional and testing purposes only. (Table continued on the next page.) 27

Class

Name exit getpid fork

ID 26 27 28 29 30 31 32

Description Exit process Get process ID (PID) Fork a process Execute a le Wait for a descendant process to complete Put current process to sleep Kill a descendant process None None

Arguments Status value ($1) PID ($1)

Returns Doesnt return Child process ID if parent, 0 if child ($1) doesnt return if successful, <0 on error ($1) 0 if successful, <0 on error 0 if successful, <0 on error 0 if successful, <0 on error

Process

execv waitpid sleep kill

Pointer to pathname of le to execute ($2) ID of process to wait on ($2), pointer to child status int ($3) Number of seconds to sleep ($2) ID of process to kill ($2)

Table 5: VIREOS system calls. * indicates system calls for instructional and testing purposes only. (Table continued from the previous page.) ticular, the main function is treated as the trap handler. As discussed in Section 3, this function will start at address 0x0000, and control will transfer to this function on any trap. All other kernel functions are called from this main function (and generally split into modules; one for each major kernel component). In addition, the compiler automatically saves and restores registers whenever the trap handler is invoked or returned from. It places them in a special array called reg store, which is implicitly declared. This array can be accessed like any other C array. In particular, it can be used to access the current processs register state and to set the next processs register state. See Appendix 10 for more details on reg store and the rest of the C-- language.

4.2

System Calls

From an application perspective, the most important part of any operating system is the set of system calls. These are effectively the interface to the operating system. While VIREOS is a simple operating system, it does have a fairly rich set of system calls. Figure 5 lists all of the VIREOS system calls. Students will implement each of these system calls (a few for extra credit) over the course of several project assignments. The system calls are numbered using the order that they are implemented in the project assignments. They are divided into several classes: I/O, le system, and process. When students write the trap and I/O handler in project assignment 1, they will implement the I/O system calls. When students write the le system in project assignment 2, they will implement the le management system calls. Finally, when students write the process manager in project assignment 3, they

28

will implement the process system calls. (No additional system calls are implemented in project assignment 4 on memory management.) System call parameters, if applicable, are passed in registers $2 ($a0), $3 ($a1), and $4 ($t0). A system call return value, if applicable, is returned in $1 ($v0). As in Unix/Linux, a negative return value generally means an error occurred, while a non-negative value generally means the call was successful, perhaps indicating something more (e.g., the number of words read from the keyboard). Because VIREOS is written entirely in C--, the programmer does not have direct access to the registers. Instead, the parameter values can be retrieved via the special reg store array. The system call return value is simply the result of the main function (the trap handler function). The starred system calls are system calls included for instructional and testing purposes only. For example, there is an I/O system call called print string for printing a string to the monitor. However, this can also be accomplished by calling the fwrite le management system call using the special standard output le descriptor (i.e., 1). Similarly, fread can be used in place of read string. Furthermore, VIREOS has support for directly reading and writing disk blocks rather than going through the le system. This feature makes testing the OS from the application much easier but might not be desirable in a commercial OS (not without some form of protection to prevent its misuse). These system calls would probably be disabled in a nal version of VIREOS. The system calls for each class (i.e., I/O, le system, process) is discussed further in the later sections describing each corresponding project assignment.

4.3

Code Organization

As shown in Figure 9(a), VIREOS consists of four major components: a trap handler, an I/O handler, a le system manager, a process manager, and a memory manager. Each of these components is implemented by the student in the project assignments (in separate assignments with the exception of the trap and I/O handlers, which are implemented in a single assignment). They are each implemented in a separate C module, which are also shown in Figure 9(a). The VIREOS OS makes use of some soft layering (soft in that there are a few cases where the layering abstraction is broken). Figure 9(b) shows this layering. At the bottom is the I/O handler. Above that is the le system manager, which is built on top of the I/O handler. The process and memory managers are built on top of the le system although they interact signicantly with one another so they are shown at the same level. The trap handler works with all of these components 29

Component Trap handler Input/output handler File system manager Process manager Memory manager

Assignment 1 1 2 3 4

C Module trap io le proc mem Trap Handler Process Manager Memory Manager

File System Manager Input/Output Manager

(b)

(a) Figure 9: VIREOS components: list (a) and layered architecture (b).
Utility Module util strings Functionality Functions for performing int/string conversions, working with unsigned integers, allocating memory Functions for manipulating strings

Table 6: VIREOS utility modules. so it is shown to the left. Each of these components will be discussed in greater detail in the project assignments. There are also a couple utility modules for use by the various VIREOS components. These are shown in Table 6. These will be needed when implementing the VIREOS components. For more information on all of the modules, check out the online application programming interface (API) off of the VIREOS web site at: http://math.hws.edu/vireos.

30

Project 0: Shell

In this project assignment, you will implement a simple shell or command interpreter in C. The shell program is the core application in our system, i.e., it is the program the user interacts with to launch other programs (similar to Windows Explorer or Mac Finder minus the graphics). Later, when building the operating system, we will use the shell to test our implementation. The shell program you implement in this project will allow the user to execute other command programs such as ls, mkdir, and ps for creating and managing processes as well as manipulating les, and directories. It will support both background processing and a simple version of piping. In fact, it will have many of the features, although not all, of a commercial shell such as bash.

5.1

Files and Directories

You will need to complete the code in the proj0 assignment directory (inside the projects directory). You will implement the shell in full ANSI C and run it on your local machine, which must be a *nix machine. (In future versions of this toolset, we will add support for writing the shell in C-- to be run directly on the VIREOS/Larc system.) You must write your shell in the provided le shell.c, which contains some skeleton code to get you started. A Makele is provided for building your shell. By default, it assumes the shell is written in C, so you need not modify it. To build the shell, enter make in the working directory. This will generate an executable le called shell (assuming there were no compiler errors), which you can run. To run it, enter ./shell in the working directory.

5.2

Project Details

Figure 10 shows a sample run of the working shell program (your shell program should eventually work like mine). The user input is underlined, while the programs output is not. The shell program repeatedly prompts the user for a new command using > . The user enters the command (e.g., ls or echo cool) and the shell program then runs the program, which also produces some output (e.g., foo.c goo.doc zoo.txt or cool). The shell program then reprompts the user and the process is repeated until the user enters exit, in which case the shell program prints Exiting... and terminates. In this project assignment, you will implement the following shell features: 31

Shell program by <student name> > ls foo.c goo.doc zoo.txt > echo cool cool > exit Exiting...

Figure 10: Sample run of shell program. Command execution. Obviously, your shell needs to allow the user to enter and run commands. It should continuously run commands until the user enters an exit command. If the command begins begins with a / (e.g., /bin/ls) or a . (./myprogram) then your shell program should run that program using any inputs the user enters. If the command does not begin with / or . (e.g., ls) then your shell should attach /bin/ to the front of the command (i.e., /bin/ls). Note: you can assume that all commands (including those that are piped) are less than 100 characters long. Error handling. Your shell should print out appropriate error messages for any illegal or incorrect commands. Background processing. Your shell should allow the user to put a command in the background using &. Redirection. Your shell should allow the user to redirect standard output to a le. In particular, the user should be able to use > or >> to overwrite a le or append to a le, respectively. You do not have to support redirection of standard error or input. Piping. Your shell should allow piping from one process to another via |. As with a commercial shell, several processes can be chained together via pipes. To simplify your implementation, you can assume that & can only be used at the end of a chain of pipes. Also, your implementation can handle several chained processes sequentially rather than in parallel. For example, assume process p1s output is piped to process p2 and process p2s output is piped to process p3. In a commercial shell, all three processes would begin executing as soon as the command is entered. However, you can run p1 and p2 rst (and not p3), piping
p1s

output to p2. When p1 has completed, you can then run p3, and pipe p2s output to p3. 32

In order to run other programs from within a C program you will need to use several library functions, which are wrappers for Linux system calls. fork. The fork system call allows a program to clone itself. exec. The exec system call (this is a family of system calls execv, execve, etc.) allows a process to execute a program. In this case, the original process is discarded once the exec call has been made. waitpid. The waitpid system calls allows one process to block and wait on another process. dup2. The dup2 system call duplicates a le descriptor from one handle to another. This system call can be used to implement redirection. pipe. The pipe system call creates a pipe between two handles. As the name suggests, it can be used to implement piping. You can get more information on these functions by looking at the man pages (e.g., enter man fork) into the shell or into google.

5.3

Writing the Shell

You will be writing the shell in C. Feel free to make use of any standard library code (i.e., code provided with the gcc compiler). However, you may not use (or look at) any other outside code. You must complete the le shell.c. It already contains some skeleton code. The shell program is complex enough that you should implement it in stages. Start by rst programming it to run commands. Do not worry about supporting &. >, >>, or | at rst. Then incrementally add each of these features (probably in the order that they are listed), testing your implementation after each feature is added.

5.4

Last Words

Remember to start early on this project. It will take some time and the sooner you nish the more time you can spend testing your shell. Good luck and remember to have fun. 33

Project 1: Trap and I/O Handler

In the remaining assignments, you will be building an operating system for the educational architecture Larc (see Section 3). In this assignment, you will write a trap and I/O handler for the operating system. Your trap handler will initialize the computer system and handle all the system calls performed by the user program. The I/O handler will allow the rest of the operating system, including the trap handler, to communicate with I/O devices via a simple interface. Your I/O handler will need to support ve devices: a keyboard, a text-based display monitor, a system clock, a hard disk, and a simple remote le transfer device (e.g., USB stick). Your trap handler will need to implement system calls for communicating with these devices (in later project assignments, other system calls will be added).

6.1

Files and Directories

You will need to complete the source code in the proj1 assignment directory (inside the projects directory). In particular, the two les trap.c, which contains the trap handler, and io.c, which contains the I/O handler, are incomplete. Each of these modules also has a header le (trap.h and
io.h)

containing some macro, variable, and function denitions that will be needed later on by

other parts of the operating system. You should not edit the header les. The .c les contain some skeleton code (the function signatures from the header les as well as some macros), which is there to help you get started and also so that the operating system will compile initially, although it will not work until you complete the assignment. There are also some auxiliary functions, which you may nd useful when implementing the trap and I/O handlers. strings.c (header le strings.h) contains functions for manipulating strings and util.c (header le util.h) contains functions for working with integers as unsigned values as well as for allocating memory. The header le os.h contains variable and macro denitions for use throughout the operating system. For more details, see the header les for each of these modules or check out the online application programming interface (API) for project 1 off of the VIREOS web site (http://math.hws.edu/vireos). In addition to the source code there is also a makele for building the operating system. To build the operating system, simply enter the command make from the project 1 working directory. When VIREOS is built, the machine code is written to os.out, the assembly code is written to os.s,

34

and the annotated assembly code (provided for debugging purposes) is written to os-db.s. The le
os-ref.out contains a pre-compiled version of the reference operating system, which you can use to

test against your own operating system. You should not need to edit or work with either of these les directly. The le ndsrc is a program for nding the source code line given a program memory address. This program can be used to nd where the operating system or a test program is failing. For example, assume VIREOS is run in the simulator and the simulator stops prematurely while executing due to a divide by zero error in the kernel at address 0x01C2. This error message alone is not helpful as it is unclear where in the C source code to look for the divide by zero error. However, we can use ndsrc to nd the corresponding source code statement and function. In this example, we would enter the following to nd the source code statement:
bash$ ./ndsrc -a 0x01C2 os.s

The target address is provided after the -a ag and the target assembly le is provided last. Note: ndsrc makes use of os-db.s, an annotated assembly le, to perform this reverse mapping.
ndsrc

can also be used to nd source code statements in test programs. In this case, we would

replace os.s in the command with the name of an assembly test program. Finally, there are several provided test programs in the directory tests within the project 1 working directory. These are discussed in the testing subsection below in more detail. The test programs were written in C-- (i.e., each test le has a .c extension). The test programs are pre-compiled and so you will also see les with the extension .s, -db.s, and .out and the same base name as one of the test les. The .s les contain the generated assembly code, the db.s contain annotated assembly code for help in debugging (as discussed above), and the .out le contains the generated machine code (stored in an ASCII le). There is also a makele for recompiling the test programs. To rebuild them simply enter make in the tests directory. To run a test program, you will pass the path to the machine code le to the simulator or load a binary version of the program into the main memory (at address 0x0000) on the FPGA. Feel free to edit any of the test programs although your operating system will be tested with the original programs (among other tests). Also, feel free to add your own test programs although you will want to edit the makele in order to compile them. In particular, you will want to add the base le name to the string assigned to TESTS in the makele. Once you have built the operating system for the rst time, a few other more les will appear, in addition to the pre-compiled kernel. The les sim and sim-ref are simulators for running your 35

operating system and the reference operating system, respectively. To run sim (sim-ref works similarly) enter ./sim <test> in the project 1 working directory where <test> is replaced with a Larc test program. For example, to run the test program hello.out in the tests/ directory, you would enter the following:
bash$ ./sim tests/hello.out

Finally, the le disk contains the disk image when simulating (i.e., the disk contents are stored in a large le).

6.2

Project Details

Here are some details concerning various aspects of the project assignment. Larc. We will be building our operating system for the Larc architecture, which is described in Section 3. Larc is an educational-based architecture with some simplications that benet the operating system developer. However, there are no existing Larc machines so to run Larc programs we use either a software simulator or a pre-congured FPGA. Unlike most commercial machines you have used, Larc is a 16-bit architecture (16-bit memory words, memory addresses, and register values). It has a simple ISA including 16 instructions. For the most part, we will not need to worry about the details of the Larc ISA since we will be programming in C (or more precisely, C--). However, the word size will be important; this means that all primitive types are represented using 16 bits. In addition, the layout of memory in Larc will also be important to understand in writing the OS. The trap handler and I/O handler interact with the machine more than any of the other operating system components. Therefore, you should carefully read Section 3, which covers Larc. Below we highlight some of the more prevalent issues but for more details see Section 3. Traps. The traps that your trap handler will need to handle are shown in Table 7. As shown in Table 7, two of the traps will not be supported until later assignments. For instance, since there is no timesharing and no paging, the timer interrupt trap and the memory page fault trap can be ignored in this assignment. The other traps will need to be supported in this assignment. The rst trap that needs to be supported is the system initialization trap that occurs when the machine is rst booted. On this trap, the trap handler should set up the machine so that the core application can run safely and correctly. For example, as described below, the trap handler will need to sandbox the core application in memory so that it cannot corrupt either the kernel or I/O 36

Trap Identier 0 1 2 3 4 5 6 7

Trap type System initialization System call Timer interrupt Page fault Illegal memory reference error Illegal register access error Illegal instruction executed error Divide by zero error

Assignment 1 1 2 (process manager) 3 (memory manager) 1 1 1 1

Table 7: Larc trap types. memory spaces. A second trap your trap handler will need to support occurs when the user program executes a system call. In this case, your trap handler will need to carry out the operation on behalf of the user program. For modularity purposes, this operation will not be implemented in the trap handler module (trap.c) but rather in a separate module such as the I/O handler (io.c) or le system manager (le.c). In this assignment, the system calls, which are discussed below, will all be implemented by the I/O handler. The remaining traps occur due to user program errors. There are four classes of errors: illegal memory reference (e.g., attempting to read/write I/O space), illegal register access (e.g., attempting to access a kernel register), illegal instruction executed (e.g., attempting to execute the privileged system return instruction), or a divide by zero error. In each case, your trap handler should print out an informative error message and halt the system. (Note: later on, when timesharing, we will kill the process but not the entire system.) System calls. The system calls you will need to implement in this assignment are listed in Table 8. In particular, you will need to support system calls for reading input from the keyboard, writing output to the monitor, reading the time from the system clock, reading/writing blocks to/from the disk, and reading/writing blocks to/from a remote le system. Note: some of these system calls are included only for testing purposes. For example, we might not want to provide system calls for directly accessing blocks on disk. Instead, requests should generally go through the le system (which you will implement in the next assignment). However, these system calls allow for better kernel testing from user programs. Note also: remote transfer is not supported on the FPGA implementation of VIREOS/Larc. If writing the OS on this platform you can ignore these system calls (as well as the test program for 37

Name halt print str* print int* read str* read int* time

ID 0 1 2 3 4 5

Description Halt system Print a string to the monitor Print an int to the monitor Read a string from the keyboard Read an int from the keyboard Get the current time (GMT) None

Arguments String to print ($2), maximum length to print ($3), Integer to print ($2) Pointer to write read string ($2), maximum length to read ($3), Nothing 3-entry int array ($2) (days since epoch in 0th entry, mins. in day in 1st, secs. in min. in 2nd) Disk block number to read ($2), buffer to write read block ($3) Disk block number to write ($2), buffer to write to disk ($3) Pointer to remote le pathname ($2), le block number to read ($3), buffer to write read words ($4) Pointer to remote le pathname ($2), le block number to write ($3), buffer of words to write ($4)

Returns Doesnt return Number of printed characters if successful, <0 on error ($1) Number of printed characters if successful, <0 on error ($1) Number of read characters if successful, <0 on error ($1) Read integer ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) Number of read words (< block size if end of le reached) if successful, <0 on error ($1) Number of written words (< block size if max. size reached) if successful, <0 on error ($1)

dread* dwrite* rtread

6 7 8

Read a block from disk Write a block to disk Read a block from a remote le (not supported on FPGA) Write a block in a remote le (not supported on FPGA)

rtwrite

Table 8: VIREOS project 1 (I/O) system calls. * indicates system calls for instructional and testing purposes only. testing them). Polling. In Larc, polling is used to synchronize the CPU with the I/O devices. This is inefcient (in a single-user, non-networked system like VIREOS it is not crippling) but it simplies the operating system. To determine if a device is ready, your trap handler, will need to implement the following loop (written in pseudocode):
repeat innitely check if device is ready if it is ready then break send data to/from device

Memory-mapped I/O. Obviously, your trap and I/O handlers can not make use of a system call since that is what you are implementing. Instead, it must communicate with the devices using memory-mapped I/O. Each device has a set of registers, called device registers, which the trap handler can read or write in order to communicate with the particular device. These registers are read and written by loading or storing specic memory locations. In C-- we can access these memory locations via pointers. Section 3 enumerates and describes all of the memory-mapped registers, which you will want to refer to. Several memory-mapped registers allow the OS to setup some of the state of the processor, 38

which are also described in Section 3. First, the mem base register holds a base memory address. When in user mode, the address in this register is added to any address used in the program. This allows the OS to relocate a process in memory (in this assignment, we will only have a single process at a single location). The mem limit register holds a value specifying the upper bound of the user address space. The OS will use these registers to prevent the user program from accessing memory outside of the user space (e.g., in the OS or I/O spaces). These should be set at system initialization (i.e., boot time). The pcr register holds some important system state. As described in Section 3, this state is encoded as several elds within the bits of the word. In this assignment, you will need to work with two of these elds. First, the rightmost bit (low order bit) is used to halt the machine. When halting the system, you will need to set this bit. Second, the leftmost byte excluding the leftmost bit (i.e., bits 8 through 14) contain the trap identier. The trap identiers are listed in Table 7 (e.g., 0 for system initialization). This eld is set by the CPU during a trap. In your trap handler, you will need to inspect this eld to determine how to handle the trap. The other memory-mapped system registers should be ignored in this assignment. We are not implementing paging yet (we will in assignment 4), so the tlb and mem pfault registers are not needed. We are also not implementing timesharing (we will in assignment 3) so we can ignore the timer registers (tmr val and tmr start). The sys ra register holds the return address of the user program on a trap. When the trap handler completes and returns to the user program (via a special system return instruction) the CPU transfers control to this address. Since we are not implementing timesharing, and only one user process can run, this register will always hold the correct value and will not need to be modied. Later, when timesharing, you will need to read and write this register, but for now, you can ignore it. Memory layout. At boot time, the kernel, which for now includes just a trap and I/O handler, is loaded into memory along with a core application, which in this project will be a simple test program. In general, these would be loaded off of the boot sector on the disk but we will manually load them into memory (the simulator does this automatically when the kernel and core application les are specied, on the FPGA it must be done manually). This is advantageous anyway since you are currently developing the operating system. If your operating system contains an error, it can be xed and the xed code can be reloaded into memory the next time the system is booted rather than having an error potentially persist on the disk.

39

0x0000 Kernel space

0x9000 User space 0xFC00 Input/output space 0xFFFF

Figure 11: The memory organization of a Larc machine. Figure 11 shows the memory layout of the kernel and user spaces along with memory-mapped I/O looks as follows. On a trap, such as when a system call executes, the machine transfers control to address 0x0000, which is the starting address of the kernel trap handler. When a special system return instruction is executed, control is transferred back to the core application (at the next instruction). Memory protection. With the trap handler and device registers sharing memory with the user program, there must be ways to prevent the user program from corrupting the trap handler and/or I/O devices. Larc has a very simple mechanism for doing this (as well as an additional technique, which you will implement in assignment 4). It uses a base address and limit to conne the program to one area of memory. Whenever the user program attempts to access memory, the base address is added to the programs address. This allows a user program to start at an address besides 0. If the program attempts to use an out-of-bounds address, the system is halted (in later projects when multiprocessing a trap will occur to the OS). By setting the base and limit carefully the OS can prevent the user program from writing into the kernel or memory-mapped spaces. One thing your OS must be careful of is an errant address passed via a system call. All addresses, such as on a call to print a string from memory, should be checked. In addition, your OS should make sure that it adds the base address to an address passed via a system call. This will not be automatically done by the CPU. Memory buffers. One requirement in this assignment, and later assignments, is that the other modules besides the trap handler work solely with memory buffers in kernel space. Although this requirement is not strictly necessary in the current assignment it will be in a later assignment. It is left as an exercise to the reader to imagine why it will be required. For example, when a

40

string buffer is passed to the trap handler to be printed, the trap handler should rst copy the string contents into a kernel buffer and then pass this kernel buffer to the appropriate I/O handler function (i.e., print strn) so that it is printed. Likewise, when reading a string from the user, the I/O handler should be passed a kernel buffer, which it can copy the string into. The trap handler should then copy the kernel buffer to the user buffer. It is possible to overow kernel memory if the user space buffer is too large. Therefore, you can make some assumptions about the sizes of the user data. In particular, you can assume that the maximum length of a string is 256 characters. To read or write more than 256 characters, the user program will need to use multiple system calls. We will make similar assumptions about other system calls in later assignments. Note that the time, dread, and dwrite system calls work with xed-size memory buffers so no additional limits are required in these cases. General-purpose registers. One responsibility of the trap handler is to save registers and restore them upon returning from the trap. This way the application is not corrupted. For the most part, you do not have to worry about saving and restoring registers; the compiler does it automatically for you in the generated assembly code. It saves them to an array in memory called reg store at the beginning of the trap handler and restores them at the end of it. This array is automatically dened; you do not need to declare it yourself. However, your trap handler code will need to do one thing: during system initialization, it will have to initialize the stack pointer. The stack pointer should be set to the limit of the users address space. To set the stack pointer you can assign to
reg store[9]

since the stack pointer is at index 9 in reg store (it is register 10, but register 0 is not

stored in reg store since it always holds 0).

6.3

Error Handling

Your operating system will need to do a signicant amount of error handling to ensure that an errant user program does not crash the entire system or cause damage to any persistent data (e.g., les). This will be especially important when we add support for multiprocessing. Although security is often overlooked in this operating system (after all, it is a non-networked, single user OS), error checking is also critical to prevent malicious programs from attacking the system and causing harm. The trap handler will do most of the error checking although some error checking will occur in the other modules that you write. First, the trap handler needs to respond to a programming error 41

caught by the processor. For example, if the program performs an illegal memory reference then, as discussed above, the processor will trap to the operating system. In the case of a trap due to a program error, your trap handler should print an informative error message and halt the system (when multiprocessing later on we will kill just the process). There are many other errors that are not caught by the processor. For instance, a user program could pass an illegal memory address as a parameter to a system call (e.g., 0xFFFF, which is too high). This error could be caught in either the trap handler or the I/O handler, although since the trap handler must copy buffers into kernel space anyway, it might make more sense to put these checks in the trap handler. In fact, in this project all the error checking can probably go within the trap handler. In later projects, some parameter errors will need to be caught in the module where the system call is implemented. For example, in the next project on the le system, one error that needs to be caught occurs when the user program attempts to open too many les. It makes more sense to catch this error in the le system module than the trap handler.

6.4

Writing the Trap and I/O Handler

You will write your trap and I/O handler in C-- [6], which is a variant of C. See Appendix 10 for more details on C--. Code organization. VIREOS is split into several modules. The rst module is trap.c, which will act as the trap handler. This le is unimplemented and must be completed for this assignment. It contains an unimplemented main method, which acts as the main method of the OS. On a trap, control is transferred to main, which is put at address 0x0000. When main returns, control is transferred back to the user application. The trap handler handles each trap by calling the appropriate function in the appropriate module. For example, if a system call trap occurs and the specied system call is to print a string to the screen, then the trap handler will call the print strn function in the I/O handler. The second module you will be implementing in this assignment is io.c, the I/O handler. This module is responsible for input/output and communication with devices. In the io.c le, you will nd several empty functions, which must be completed for this assignment. As mentioned above, there are several auxiliary les (os.h, strings.c, and util.c) to help you to complete your trap and I/O handlers. You should not need to add any additional modules. Instead you should complete trap.c and io.c, which are largely unimplemented. In later projects, we will 42

Type

Name halt.c hello.c print-int.c echo.c echo-int.c time.c

Description Halt program Hello world program Prints int 42 to the screen Echoes a user-entered string to the screen Echoes a user-entered int to the screen Prints current time as 3 raw ints (days since epoch, minutes in current day, and seconds in current minute) Prints current formatted date Simulates birthday probability problem Plays the 1- or 2-player game of nim Read/write disk blocks Read/write remote le blocks (not supported on FPGA) Loads from an illegal memory address Stores to an illegal memory address Transfers control to an illegal memory address Accesses a kernel register Executes a privileged system return instruction Performs a divide by zero Performs an unrecognized system call Performs a randomly-chosen system call with an illegal parameter

System calls used Halt (0) Halt (0), Print string (1) Halt (0), Print int (2) Halt (0), Print string (1), read string (3) Halt (0), Print int (2), read int (4) Halt (0), Print string (1), print int (2), time (5)

Working date.c birthday.c nim.c disk-test.c rt-test.c mem-read-err.c mem-write-err.c jump-err.c reg-err.c Broken inst-err.c div-zero-err.c syscall-id-err.c syscall-param-err.c

Halt (0), Print string (1), time (5) Halt (0), Print string (1), print int (2), time (5) Halt (0), Print string (1), print int (2), read string (3), read int (4), time (5) Halt (0), Print string (1), print int (2), read string (3), read int (4), disk read (5), disk write (6) Halt (0), Print string (1), print int (2), read string (3), read int (4), remote read (7), remote write (8) None None None None None None Illegal system call (bad identier) Time (5), other varies per run

Table 9: Assignment 1 test programs. add other modules (e.g., one for managing the le system), which you will also complete. Incremental programming. You will want to write your trap and I/O handler incrementally where possible. This way you can verify part of your solution is correct before going on to another part. It is left as an exercise to you to think about ways that you can split up the features of the trap and I/O handlers so that each part can be written and tested separately although some guidance is given in the next subsection.

6.5

Testing the Trap and I/O Handler

There are several provided test programs to help you nd errors in your trap and I/O handlers. They are listed in Table 9. There are two classes of programs: working test programs and broken test programs. The working test programs should run correctly when your trap and I/O handlers are completed. The broken system calls contain an error that should be handled by your operating system. The output generated using a working operating system is stored in a le with the extension 43

.out and the same base name as the test program. These test programs were designed to help support incremental programming, as discussed above. Each test program uses a minimal number of system calls so that most system calls can be tested in isolation. All of the tests must use the halt system call to terminate the program. Furthermore, they all must print to the screen (minus one that does only a halt) in order to be properly validated. Therefore, the halting and printing system calls should be implemented before the other system calls. After implementing those system calls, the reading input and time system calls should probably come next as several other test programs use these. Finally, the disk and remote transfer system calls should probably be implemented last as these are each used in only one test program, and the programs for testing these system calls make use of most of the other system calls. The working test program are listed in increasing order of system call usage. You might utilize this order when implementing and testing your system calls. halt.c does nothing but perform a halt. hello.c and print-int.c print a string and int, respectively, and then halt. Note that print-int.c does not use the print string system call so a newline will not be printed after the integer (42).
echo.c

and echo-int.c echo a string or int back to the screen. Note that echo-int.c does not use

the print string system call so no prompt will appear when you run it and a newline will not be printed after the echoed number. time.c and date.c make use of the time system call. time.c prints out the seconds since epoch or January 1st, 1970. Since Larc is a 16-bit architecture, this must be printed as 3 integers (days since epoch, minutes in current day, and seconds in current minute).
date.c

prints out the formatted date. Both programs use UTC. birthday.c simulates the birthday

probability problem (http://en.wikipedia.org/wiki/Birthday problem). nim.c plays a 1- or 2-player version of the game, Nim (http://en.wikipedia.org/wiki/Nim). disk-test.c is an interactive program, which allows users to read and write blocks on the disk. rt-test.c is similar to disk-test.c but for reading and writing le blocks on a remote le (which will be a local le on the system where you are running the simulator). Note: rt-test.c is not supported when running VIREOS/Larc on an FPGA. Each of the broken programs test a different type of error that can occur. mem-read-err.c, memwrite-err.c, jump-err.c, reg-err.c, inst-err.c,

and div-zero-err.c each test an error that is caught by

the processor, which traps to the operating system. Your trap handler should print an appropriate message in each case and halt. syscall-id-err.c and syscall-param-err.c test two different types of system call errors: using an unknown system call and passing a bad parameter to a system call. 44

syscall-param-err.c

randomly chooses the system call it uses although it always picks one, which

has a parameter that can be misused (e.g., a memory address). You should run it several times when testing it. While these tests are fairly comprehensive, you may nd that other tests are needed. Feel free to either extend these tests or write your own. Be aware though, that your operating system will need to work correctly with the original test programs (as well as potentially other tests). Important: take testing seriously in this project and all of the remaining projects. It is a critical part of the software design process. There is a strong correlation between students with working operating systems and students who thoroughly test their code.

6.6

Last Words

Remember to start early on this project and all the remaining projects. It will take a signicant amount of time (more than you probably think) and the sooner you nish the more time you can spend testing your trap and I/O handlers. Good luck and remember to have fun!

45

Project 2: File System

In this project, you will implement a le system for the VIREOS operating system. Your le system will allow applications to work with les (and directories if you do the extra credit) rather than working directly with disk blocks. The le system you will implement is similar to the Unix le system but with several simplications (e.g., no links, directories only if you do the extra credit).

7.1

Files and Directories

You will need to complete the source code in the proj2 assignment directory (inside the projects directory). In particular, the le le.c, which contains the le system manager is incomplete and must be nished in this assignment. This module has a header le (le.h containing some macro, variable, and function denitions that will be useful both in le.c as well as by other parts of the operating system. You should not edit the header le. le.c contains some skeleton code, i.e., the signatures of functions dened in the header le, as well as some useful macro and struct denitions. The operating system will compile initially but the le system will of course not work properly until you complete the assignment. As in the last assignment, there are also some auxiliary modules (e.g., util, strings), which you may nd useful when implementing the le system manager. Note: there are some additional functions in util.c, which you may nd helpful. You have also been provided with a working a version of the VIREOS trap and I/O handlers from the previous project. The compiled code is in the Larc assembly le lib.s, which is included in the target code when the operating system is built. You should not edit this le. Note: the trap handler has been extended to support le system calls; it calls the appropriate function in the le system module on a le system call. Although the .c les for the trap and I/O handlers are not provided, the header les (trap.h and io.h) are provided since your le system will need to call functions in these modules. For more details on any of the provided code, see the header les for each of these modules or check out the online application programming interface (API) for project 2 off of the VIREOS web site (http://math.hws.edu/vireos). As in the previous assignment, the working directory also contains a makele for building the operating system (i.e., enter make), a pre-compiled version of the operating system (os-ref.out),

46

Super block 1

Boot blocks 256

Bitmap blocks 4

I-node blocks 128

Data blocks 16384

Figure 12: File system layout. a program ndsrc for mapping a memory address back to a C source line, and a directory tests with various test programs. After building your operating system for the rst time, you will also have a simulator le (sim) for running your compiled version of VIREOS, a reference simulator le (sim-ref) for running the simulator with the reference operating system, and a disk le (disk) containing the contents of the simulated disk. See the previous assignment for more details on any of these les or directories.

7.2

Project Details

Here are some details concerning various aspects of the project assignment. General. You will be implementing a simplied Unix-like le system. The le system will not support links (hard or soft). Directory support is saved for the extra credit. Your le system will use I-nodes to keep track of the le attributes and data blocks. To keep track of the data blocks that are free we will use a bitmap stored on the disk (as opposed to a linked list). We will use the same type of disk as in the previous project. The disk controller takes care of the manipulation of the disk head with regards to the disk cylinders. The OS just sends the controller the block number, mode (read or write), and potentially the data (if writing), and the controller potentially sends the OS back the data (if reading). As discussed in Section 3, this is done via polling and memory-mapped I/O. As you are given a working I/O handler (compiled into
lib.s), you can ignore the details in this project.

One thing that is important though is the disk block

size: 256 words. Your le system will need to work with blocks of this size when it interacts with the disk. File system layout. We will be working with a single partition le system. As a result, our le system will not contain a Master Boot Record or Partition Table. Table 12 shows the organization of the le system. The rst block is the super block (at disk block 0), which describes the sizes of the rest of the sections. This block would allow for interoperability with slightly different le systems (e.g., where the I-node sections differ in length). Your le system need not support different lengths, however, it should create a correct super block when the disk is formatted. In

47

particular, the super block should contain 5 integers at the beginning of the block, which specify the length of each section. In our case, this would be: 1, 256, 4, 128, and 16384. The rest of the block should be 0. The boot blocks contain code for booting the system. Because the boot blocks come after the super block, the rst boot block is at disk block 1, the second is at disk block 2, etc.These blocks contain the operating system and core application images. These blocks are copied into physical memory at boot time. When the disk is formatted, these blocks should be unmodied. Applications may modify them if changing either the operating system or core application (e.g., wiping out the old operating system and replacing it with a new one). But this is done by writing these disk blocks (support which was added in the last assignment); no special support for modifying the boot blocks is required in the le system. The bitmap blocks contain a bitmap (which spans multiple blocks) indicating which data blocks are available and which are in use. Since the bitmap blocks come after the super block (1 block) and boot blocks (256 blocks), the rst bitmap block is at disk block 257, the second is at disk block 258, etc.Each bit in the bitmap indicates whether the corresponding data block is free: 0 means free, 1 means taken. At the outset (at the time of formatting), all data blocks should be free and so the bitmap should consist of all 0s. The I-node blocks contain the I-nodes used to keep track of le (or directory, if doing the extra credit) attributes and data blocks. Each I-node block contains 8 I-nodes per block. With 128 I-node blocks, there are a total of 1024 total I-nodes (128*8). Since the I-node blocks come after the super block (1 block), the boot blocks (256 blocks), and the bitmap blocks (4 blocks), the rst I-node block is at disk block 261, the second is at disk block 262, etc. The remaining blocks on disk are dedicated to data blocks. The data blocks contain the data of each le (and directory, if doing extra credit). They also contain any indirect blocks needed for large les. Since the data blocks come after the super block (1 block), the boot blocks (256 blocks), the bitmap blocks (4 blocks), and the I-node blocks (128 blocks), the rst data block is at disk block 389, the second data block is at 390, etc.The availability of data blocks is indicated by bits in the free bitmap blocks. I-nodes. An I-node contains several le attributes as well as pointers (possibly indirect) to the les data blocks on disk. Figure 13 shows the I-node data structure. The attributes of the le include a ag indicating whether this I-node is available; the le name; a ag indicating whether

48

Available ag File name Directory ag File size (logical) Direct data block pointers (10) 0 1 2 3 4 5 6 7 8 9

Singly indirect data block pointer (1)

Figure 13: I-node data structure. this I-node refers to a directory (only applicable if you do the extra credit); and the logical size of the le (our le system does not keep track of the actual physical size). Note: since our OS does not support multiple users many of the Unix attributes were not applicable for us (others were left out to simplify the implementation). The I-node contains 10 direct pointers and 1 indirect pointer. The 10 direct pointers map the rst 10 logical le blocks to the corresponding disk block. The indirect pointer maps the remaining logical blocks to disk blocks. If used, it refers to an indirect block of direct pointers. Since a block is 256 words, this indirect block will contain 256 pointers. The rst pointer in the indirect block maps logical block 11 to its disk block, the second maps logical block 12 to its disk block, and so on. Because there is no double or triple indirect pointers a le is limited in size to 10+256 blocks or 68096 (266*256) words. In fact, the le size is limited further to 32,768 since Larc is a 16-bit machine to 32,768 (i.e., the range of non-negative numbers in 16-bit twos complement). An I-node requires 32 words of space. Since a disk block is 256 words, 8 (256/32) I-nodes t onto a single block. With 125 I-node blocks, we have a total of 1000 I-nodes. Therefore, our le system can only support at most 1000 les since each requires its own I-node. There is no I-node pool to nd available I-nodes. Instead, the available ag is used. To nd an available I-node (e.g., when creating a new le), we can walk through the I-nodes in the I-node blocks looking for one whose available ag indicates it is free. When one is found, we initialize the attributes in the I-node and set the available ag to indicate it is in use (so it will not be reallocated). When we are done with an I-node (e.g., when removing a le), we can simply set the available ag to indicate the I-node is now free. Free space management. We use a bitmap to determine what data blocks are available and what data blocks are taken. Every time a new data block is needed (e.g., on a le write) or when one is no longer needed (e.g., on a le deletion), the free bitmap must be updated. Each bit in the

49

Bitmap block 0 0100000000000000 1010101010101010 ... 0000000000000000

Bitmap block 1 0000000000000000 0000000000000000 ... 1111000000000000

Bitmap block 2 0000000000000000 0000000000000000 ... 0000000000000000

Bitmap block 3 0000000000000000 0000000000000000 ... 0000000000000000

Figure 14: Snapshot of the data block bitmap. bitmap indicates whether the corresponding data block is free: 0 means free, 1 means taken. For example, the rst word in the bitmap indicates whether the rst 16 data blocks are free. The rst bit in the word (you can decide whether the rst bit is the leftmost bit or the rightmost bit, although make sure you are consistent) indicates whether the rst data block is free, the second bit indicates whether the second data block is free, etc.Because the bitmap is larger than 256 words, it must span several blocks. Figure 14 shows a snapshot of the data block bitmap at some particular point. The bitmap is split into 4 bitmap blocks. Each entry in the bitmap block is a word of 16 bits. Although not shown in its entirety, each bitmap block has 256 words in it. Assume that the rightmost bit is the rst bit in each word and the leftmost bit is the last bit in each word. In that case, the underlined bit in bitmap block 0 (the 1 in position 1 of the second word of the rst bitmap block) refers to data block 17 and indicates that it is taken. The 0 to the left of this bit (not underlined) indicates that data block 18 is available. Similarly, the underlined bit in bitmap block 1 (the 1 in position 12 of the last word in the second bitmap block) refers to data block 8188 (256*16+255*16+12) and indicates that it is is taken. The 0 to the right of this bit (not underlined) indicates that data block 8187 is available. The total number of bitmap blocks depends on the total number of data blocks. Because there are 16,384 data blocks, we need 4 bitmap blocks. In other words, with 4 blocks the bitmap can keep track of 16 (bits per word) * 256 (blocks per word) * 4 (blocks per bitmap) or 16,384 total data blocks. To nd a free data block (e.g., on a le write requiring a new data block), we examine each bit in the bitmap until we nd a 0. We know from this bit that the corresponding data block is free. To allocate this data block to a le, we set the bit in the bitmap to 1, meaning taken, so that it will not be reallocated. When deallocating a data block (e.g., when deleting a non-empty le), we must reset the corresponding bit in the bitmap back to 0, meaning available. After doing this, the block can be reallocated to another le. 50

Type (e.g., le, stdin, etc.) I-node number File size (logical) Seek pointer

Figure 15: File descriptor data structure. When the le system is initially formatted, the bitmap will contain all 0s as there are no existing les and thus no allocated data blocks. As les are created and written to though, this will change. File descriptors. The operating system must maintain the state of each le (or directory, if doing the extra credit) that is opened. This state is used, among other things, to keep track of the current position in the le for the next read or write. In addition, it caches some of the le attributes (e.g., le name) for fast access. This state is called a le descriptor. The operating system maintains a table of these for each user application and the user program refers to a le descriptor via its index into the table. Since our operating system does not support multiprocessing yet, your le system will need only maintain a single table for the one user program. In VIREOS, there are 20 total le descriptors for each user program, meaning that each user program can open concurrently a maximum of 20 les, directories, pipes, etc. Initially, le descriptors 0 and 1 are allocated for the standard input and output streams, respectively. Reading words from standard input results in an I/O read of characters from the keyboard. A standard input descriptor can only be read from; it is illegal to use it in any other way (e.g., to write it). Writing to standard output results in an I/O write of characters to the monitor. A standard output descriptor can only be written to; it is illegal to use it any other way (e.g., to read it). The other le descriptors (2-19) are unallocated and can be used by the user program to open les, directories, pipes, etc.However, it is not safe to assume that descriptors 0 and 1 will always contain standard input and output, or that the other descriptors will not refer to standard input and output. User programs can copy one descriptor to another so potentially any descriptor could (or could not) refer to standard input or output. Figure 15 shows the le descriptor data structure. The rst eld, a numeric value, species the type of the descriptor. There are 6 possible types: unused (-1), standard input (0), standard output (1), regular le (2), directory (3), pipe input (4), or pipe output (5). Your le system will use the type to determine whether an operation is legal (e.g., a read from a standard output descriptor

51

would be illegal) and how to satisfy that request (e.g., a write to standard output would be sent to the I/O handler). The second eld is the I-node number of the le referred to by the descriptor. This number can be used to access the I-node on disk for a particular le descriptor. This eld is not applicable for all types of descriptors. In particular, it is not valid for standard input and output streams, since these do not refer to a le. However, it is applicable for the other types of descriptors. As discussed below both directories and pipes are implemented using a le so in both of these cases there I-node number would be applicable. The le size holds the current size of the le. Note that this eld is not strictly necessary as this information is also stored in the I-node. But keeping the size in the I-node can avoid costly disk accesses in some circumstances (e.g., when seeking in a le). The nal eld is the seek pointer, which keeps track of the current location (in words) in the le. The seek pointer can go past the le size although it cannot be negative. It is not used for standard input/output descriptors. System calls. Your operating system will now support several new le system-related system calls. The trap handler you will be using has been augmented to handle these system calls. It will call the appropriate function in the le system module when one is made. Table 10 shows the system calls that will be added to the operating system and which you will need to support in your le system module. Since you are not modifying the trap handler yourself (a modied version is being given to you), you can ignore the system call identiers and parameter registers. As discussed below, you will implement each of these system calls by completing the corresponding function in the le system module. Note that the directory support is optional and can be done for extra credit. File system functions. Your le system will need to support several functions, which will be called by other parts of the operating system. For example, at boot time, the following function will be called by the trap handler: fsinit: a function for initializing the le system when booting. This function will need to initialize the le descriptor table. In addition, the trap handler will call the following functions when the corresponding system call is made: 52

Grading

Name dformat fcreate fopen fread

ID 10 11 12 13

Description Disk format Create a new le Open an existing le Read from an open le None

Arguments Nothing Pointer to pathname of le to create ($2) Pointer to pathname of le to open ($2) Handle of le to read from ($2), buffer to write read words ($3), maximum length to read ($4) Handle of le to write to ($2), buffer of words to write ($3), maximum length to write ($4) Handle of le to seek in ($2), seek offset ($3), seek type ($4) Handle of le to truncate ($2), new logical le size ($3) Handle of le to close ($2) Pointer to pathname of le ($2) Source le handle ($2) target le handle ($3) Pointer to buffer of two ints ($2) where handles are written (read handle in 1st entry, write in 2nd) Pointer to pathname of directory to create ($2) Pointer to pathname of directory to open ($2) Handle of directory to read from ($2) Handle of directory to close ($2) Pointer to pathname of directory to delete ($2)

Returns File handle if successful, <0 on error ($1) File handle if successful, <0 on error ($1) Number of read words if successful, <0 on error ($1) Number of written words if successful, <0 on error ($1) New seek pointer if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 is successful <0 on error ($1) 0 if successful, <0 on error ($1) directory handle if successful, <0 on error ($1) number of read words if successful, <0 on error ($1) 0 if successful, <0 on error ($1) 0 if successful, <0 on error ($1)

fwrite

14

Write to an open le

Required

fseek ftrunc fclose fdelete fdup2 fpipe

15 16 17 18 19 20

Seek in an open le Truncate (or enlarge) an open le Close an open le Delete an existing le Duplicate a le descriptor Create a new pipe and handles (read and write) for accessing it Create a new directory Open an existing directory Read from an open directory Close an open directory Remove an existing directory and all of its subcontents

fmkdir fopendir Extra Credit freaddir fclosedir frmdir

21 22 23 24 25

Table 10: VIREOS le system calls. fdisk format: a function for formatting the disk. This function will initialize the super block, boot blocks, bitmap blocks, and i-node blocks (the data blocks do not need to be initialized). It returns 0 when successful, <0 on an error (unless there is an internal error, this function should succeed). fcreate: a function for creating a new le. This function takes the name of the le to create as a parameter. It should nd an available I-node for this le and initialize it. It should then allocate a le descriptor for this newly-created le and return it. If an error occurs (e.g., trying to create a le that already exists) then it should return a negative value. fopen: a function for opening an existing le. This function takes as a parameter the name of the le to open. It should scan the I-nodes to nd the I-node belonging to the specied le. (Note: if doing the extra credit, then your function will open the ancestor directories, 53

starting from root until it gets to the specied le.) It should then allocate a le descriptor for this le and return it. If an error occurs (e.g., opening a non-existent le), then it should return a negative value. fread: a function for reading from an opened le. This function takes as parameters the le descriptor handle to read from, a pointer to an integer array buffer, and a length. It reads up to length words and copies them into the buffer. It reads from one of three places depending on the le descriptor handle: (1) from the keyboard using the function read strn in io.c if the descriptor is standard input; (2) from a pipe le if the descriptor is a pipe input; or (3) from a le if the descriptor is an ordinary le. If reading from a pipe le or a conventional le, your function should read until the logical le size is reached or the words read is equal to the specied length (whichever occurs rst). This function should return the number of words read if successful, or a negative value on an error (e.g., if a bad handle is used). One challenge in implementing le reading will be handling reads that do not start and/or end at block boundaries. In these cases, you will need to remove the unneeded words at the beginning or end of the read. Another challenge is dealing with holes that can occur when seeking past the end of the le and then performing a write or enlargening the le size via
ftruncate.

In these cases, some of the I-node pointers within the logical le size may refer to

0 (meaning null). If one of these le blocks is read, null words should be returned. fwrite: a function for writing to an opened le. This function takes as parameters the le descriptor handle to write to, a pointer to an integer array buffer, and a length. It writes up to length, words from the buffer. It writes to one of three places depending on the le descriptor handle: (1) to the monitor using the function print strn in io.c if the descriptor is standard output; (2) to a pipe le if the descriptor is a pipe output; or (3) to a le if the descriptor is an ordinary le. If writing to a pipe le or a conventional le, your function should write until the logical le size is reached or the words written is equal to the specied length (whichever occurs rst). This function should return the number of words written if successful, or a negative value on an error (e.g., if a bad handle is used). One challenge in implementing le writing will be handling writes that do not start and/or end at block boundaries. In these cases, you will need to make sure you do not accidently overwrite the start and/or end of a block that was not meant to be written. This will require

54

doing a read rst to get the original block data. This data can be coalesced with the userspecied data into a single block, which can be written back to the disk. Another challenge is keeping track of the logical le size. Every time a write increases the le size, the I-node should be read from disk (if not already read), adjusted, and written back to disk. fseek: a function for seeking within an opened le. This function takes as parameters the le descriptor handle of the le to seek in, a seek offset, and a seek whence (or type) value. If the whence value is 0 (SEEK SET) it sets the seek pointer in the le descriptor to the offset. If the whence value is 1 (SEEK CUR) it adds the offset (positive or negative) to the current seek pointer. Finally, if the whence value is 2 (SEEK END) it adds the offset (positive or negative) to the current logical le size. The resulting seek pointer must be non-negative (in 16 bits this would be between 0 and 32767) otherwise it should not be changed. If the seek was successful it returns the new seek pointer, otherwise it returns a negative value (e.g., if the resulting seek pointer is negative). ftruncate: a function for truncating or enlargening an opened le. This function takes as parameters the le descriptor handle of the le truncate and the new size of the le. It adjusts the le size in both the descriptor and in the I-node (which will have to be written back to disk). If shrinking the le, it sets all pointers (direct or indirect) past the end of the new length to 0. It returns 0 if successful or a negative value on an error (e.g., if a bad handle is used). fclose: a function for closing an open le. It takes the le descriptor handle of the le to close as a parameter. It closes this descriptor and returns 0 if successful or a negative value on an error (e.g., if a bad handle is used). fdelete: a function for deleting a le. It takes the name of the le to delete as a parameter. It should rst close any open le descriptors for this le. It should then scan the I-nodes to nd the I-node belonging to the specied le. (Note: if doing the extra credit, then your function will open the ancestor directories, starting from root until it gets to the parent directory where it will get the I-node for the specied le and remove the le from the directory contents.) It should then set the deleted les I-node available ag to indicate the I-node is now available for reallocation. Finally, it should deallocate any data blocks by unsetting the corresponding

55

bits in the bitmap. It returns 0 if successful or a negative value on an error (i.e., if deleting a non-existent le). fdup2: a function for duplicating a le. It works similarly to the dup2 Unix system call. It takes two le descriptor handles as parameters. It copies the rst le descriptor to the second. Before duplication, it closes the second (target) descriptor if it was previously open. Note: as our operating systems does not have support for multi-processing, descriptor duplication is not tested in any of the applications. So, make sure you use some care when implementing it as you may not be able to test it (unless you write your own test program). fpipe: a function for creating a pipe. It works similarly to the pipe Unix system call. It takes a pointer to an array of two integers as a parameter. It sets the rst entry in the array to a pipe input handle and it sets the second entry to a pipe output handle. It allocates two le descriptors for both ends of the pipe. To handle piping, your le system will create a special le to hold the piped data. It should name the le pipe-<id> (without the quotes) where <id> is replaced with a numeric identier (if implementing directories this le should go in the root directory). The numeric identier should start at 0 (i.e., for the rst pipe) and should wrap around at 32,767 (the largest 16-bit number). The output pipe descriptor will write to this le and the input pipe descriptor will read from it (but never vice versa). The le descriptors can be initialized similarly as with conventional les except the pipe type will be non-zero. For example, the le name can be set to pipe-42 assuming 42 was the next numeric identier. Note: as our operating systems does not have support for multi-processing, piping is not tested in any of the applications. So, make sure you use some care when implementing it as you may not be able to test it (unless you write your own test program). Also, because of our implementation of pipes, i.e., as conventional les, VIREOS does not correctly support overlapped reading and writing of the pipe. If these are overlapped then the process reading from the pipe might complete too early, i.e., before all of the output has been sent to the pipe. For the types of applications that we want to support in VIREOS, this is not a big limitation. There are also several functions for manipulating directories that are not required (e.g., fmkdir). As discussed below, these can be done for extra credit.

56

7.3

Error Handling

One important and challenging part of this assignment is handling errors. There are many that can arise and your le system manager should catch them all and handle them appriopriately. The provided trap handler will catch any memory errors, i.e., you can assume all memory addresses passed as parameters are valid. However, the trap handler does not catch other errors such as using a bad le descriptor handle. Many of the potential errors are described in other parts of the project writeup but others are not. When implementing any function, you will need to think carefully about the things that could potentially go wrong and prevent these pitfalls. In particular, you should think carefully about the following two classes of errors: Function parameters. These are generally coming from the user program via a system call and should not be trusted. You should check all of these carefully. For example, many functions require a le descriptor handle, which could be erroneous (e.g., a negative value). File system limits. In any le system there are limits, which the operating system must make sure are never surpassed. For example, as mentioned above, there is limit to the number of les that can exist at any point, to the length of any particular le, and to the number of simultaneously opened les. You must ensure that these limits are never reached. For function parameter errors you should not print any type of error message, just return the appropriate value. Often, a user program is not errant when passing a bad le system parameter. For example, user programs sometimes attempt to delete a le that they plan on creating without checking if the le actually exists. If the operating system prints out an error message in this case it will be very annoying to the programmer and/or user. The programmer will have to go to greater lengths to avoid any such messages printing. In some cases, the program may actually be errant but it is generally impossible to distinguish between errant and correct programs. Therefore, the appropriate way to handle such errors is to return an error value and allow the user program to react accordingly. Some system calls could result in multiple function call parameters. In a commercial system, we would want to somehow communicate the exact error that occured to the user program. To simplify VIREOS, however, we ignore this issue. 57

On the other hand, le system limit errors should be reported by the OS via a print message so the user knows what went wrong (and does not assume it was a parameter error). Unless the error is catastrophic, you should return an error value in these cases as well. For a catastrophic error (one that cannot be recovered from) you can halt the machine. Error handling is an important component of any system and this importance will be reected in the grading. So make sure you take error handling seriously.

7.4

Writing the File System Manager

You will write your le system manager in C-- [6], which is a variant of C. See Appendix 10 for more details on C--. This project will require signicantly more coding and design than the previous projects. Make sure you take some time to carefully plan out your implementation. The le le.c contains the code implementing the le system. It is largely incomplete; you will complete it in this project. At the very least, you will need to implement the functions listed above under the heading File system functions in the section titled Project Details. You will probably also want to write some additional functions to avoid code redundancy and complexity. For example, you might nd it helpful to create the functions for doing the following: Allocating le descriptors. Checking le descriptors. Finding/allocating available data blocks. Mapping a le block number to a data block number. Note: this list is by no means exhaustive. Furthermore, there are other possible ways you might organize your code. Feel free to organize your code in any way you see t so long as it is readable by others. Some of this project can be done incrementally if you plan carefully. For example, one of the rst things to implement is writing to standard output and reading from standard input. Until you get this implemented, none of your test programs will work (as they all perform I/O through the le system reads and writes). This should not be that difcult as you can use the I/O module. You can save reading and writing of les for later on. After you have standard I/O working, you should work on implementing the le system initialization and shutdown functions (e.g., disk formatting). These will be difcult to test so use care when writing them. After that you can write the le 58

Name hello.c echo.c le-test.c shell.c le-stress.c

Description Hello world program Echoes a user-entered string to the screen Interactively prompts user for a le system operation Single-process shell program (utilities are part of shell program) Non-interactively stress tests the le system

Table 11: Assignment 2 test programs. creation, open, and deletion functions. These can be tested by creating and deleting les in the test programs. Finally, when you nish these functions, you can implement the remaining ones.

7.5

Testing the File System

There are a few provided test programs in the tests directory (within your working directory) to help you nd errors in your le system manager. They are listed in Table 11. Unlike in the last assignment, there are fewer tests, but a couple of them provide signicant coverage over le system operations and errors. Two of the three are interactive and allow the user to perform particular operations. These two tests will require the user (i.e., you) to think carefully about what operations should be used in order to fully test the le system. The le hello.c prints Hello, world!\n to the screen and halts. Unlike in the previous assignment, this program uses the le system call fwrite with the standard output stream to print to the monitor. The le echo.c echoes a string entered by the user back to the screen. Like hello.c, it uses le system calls (fwrite and fread) rather than I/O system calls. The le le-test.c is an interactive program that allows the user to perform a system call and see the result. le-test.c repeatedly prints out a menu system with several options corresponding to le system calls, or disk operations for validating le system calls, and allows the user to select an option. For example, the user could rst create a le called foo.txt. The user might then see if they can write to the le. Using other operations, the user could verify that this write was successful (i.e., seek back to the beginning of the le and then perform a read). This program will likely be the most helpful test program and one that you will spend a signicant time using. The le shell.c is an single-processed shell program. All the utility programs (e.g., ls, cat) are included in the shell program since there is no multiprocessing. It also does not support background processing (&) and piping (|). To see a list of supported commands, enter help. Finally, the le le-stress.c is a stress test program. It tests the le system limits such as 59

maximum number of supported les, maximum number of open les, maximum le size, and maximum number of data blocks. If there are any bugs in the le system it will most likely fail although it may be hard to determine cause of the failure. Therefore, it is a good test to try last after you have debugged carefully with the other test programs (especially le-test.c).

7.6

Extra Credit

For extra credit, you can add support for directories. In particular, you will need to complete the following functions: fmkdir for creating a new directory. fopendir for opening an existing directory. freaddir for reading an open directory. fclosedir for closing an open directory. frmdir for removing an existing directory. You will also need to extend some of the other functions, such as those that deal with opening les. This will require a signicant amount of work and, as a result, the total extra credit points is high. An ambitious student could certainly do this and signicantly raise their grade. Important: do not start the extra credit until you nish the requirements. In addition, make a copy of your working solution before starting the extra credit.

7.7

Last Words

Remember to start early on this project. This project will take a much longer time to complete than the previous assignment. In fact, it is one of the largest projects you will work on. It will take a signicant amount of time (more than you probably think) and the sooner you nish the more time you can spend testing your le system. Good luck and remember to have fun!

60

Project 3: Process Manager

In this project, you will implement a process manager for the Larc operating system. Your process manager will allow multiple processes to run concurrently, overlapping their execution on the CPU. It will support time sharing, making sure that no one process hogs the CPU. To simplify this project, only one process will be allowed in memory at a time. This will lead to a high context switch overhead and an overall inefcient implementation. But in the following project (the last project), we will build a memory manager that will allow multiple processes to share memory and reduce the context switching overhead.

8.1

Files and Directories

You will need to complete the source code in the proj3 assignment directory (inside the projects directory). In particular, the les proc.c, which contains the process manager is incomplete and must be nished in this assignment. This module has a header le (proc.h containing some macro, variable, and function denitions that will be useful both in proc.c as well as by other parts of the operating system. You should not edit the header le. proc.c contains some skeleton code, i.e., the signatures of functions dened in the header le, as well as some useful macro and struct denitions. The operating system will compile initially but the process manager will of course not work properly until you complete the assignment. As in the previous two assignments, there are also some auxiliary modules (e.g., util, strings), which you may nd useful when implementing the process manager. You have also been provided with a working a version of the VIREOS trap handler, I/O handler, and le system manager from the previous two projects. The compiled code is in the Larc assembly le lib.s, which is included in the target code when the operating system is built. You should not edit this le. Note: the trap handler has been extended to support process-related system calls; it calls the appropriate function in the process manager module on a process system call. The trap and I/O handlers have also been extended to support a timer interrupt, which is used to implement timesharing. Finally, a few functions have been added to the le system to support multiprocessing (which are discussed below). Although the .c les for the trap handler, I/O handler, and le system are not provided, the header les (trap.h, io.h, le.h) are provided since your process manager may need to call functions in these modules.

61

For more details on any of the provided code, see the header les for each of these modules or check out the online application programming interface (API) for project 3 off of the VIREOS web site (http://math.hws.edu/vireos). As in the previous two assignments, the working directory also contains a makele for building the operating system (i.e., enter make), a pre-compiled version of the operating system (osref.out), tests

a program ndsrc for mapping a memory address back to a C source line, and a directory

with various test programs. After building your operating system for the rst time, you

will also have a simulator le (sim) for running your compiled version of VIREOS, a reference simulator le (sim-ref) for running the simulator with the reference operating system, and a disk le (disk) containing the contents of the simulated disk. See the trap and I/O handler assignment (Section 6) for more details on any of these les or directories. Unlike in previous assignments, the disk comes pre-installed with various les and directories needed for testing your process manager. If for some reason your disk becomes corrupted, remove it and enter make disk to restore it to its original contents.

8.2

Project Details

Here are some details concerning various aspects of the project assignment. General. You will be implementing a simplied process manager. Your process manager will support running 10 processes concurrently. Each process will get a quanta of 10 seconds (note: this is much higher than it should be due to the high context switch overhead). If the process uses all its quanta then a timer interrupt will occur (the only interrupt supported in this project) and the CPU will transfer control to the trap handler. The trap handler will then invoke the process manager to schedule a new process to run. Your process manager will keep track of each process in a process table (described below). All the state of the process will be kept in the process table except the processs current memory (i.e., its address space or memory image). The processs memory image is too large to store in the process table and will be stored in a le on disk. Processes will be scheduled in round robin fashion, treating the process table like a circular queue. Unlike in previous projects, the OS and CPU will support one simple interrupt, which will allow for preemptive scheduling. However, polling will still be used to interact with the devices and interrupts will only be supported when running user programs (so the OS need not worry about 62

Status (dead, running, ready, blocked) Executable lename Parent PID (<0 if none) Wait PID (<0 if not waiting) Wait status value Wait status pointer Sleep amount (0 if not sleeping) PC Registers File descriptors

Figure 16: Process table entry. being interrupted). You will be given a working version of the trap handler, I/O handler, and le system manager. You will need to use these in your implementation of the process manager, especially the le system manager. They have been augmented to deal with process system calls, the timer interrupt, and in other ways. Although you will not have source code for these modules, you can view the header (or interface) les (i.e., fs.h, io.h). Note: there is no interface le for the trap handler as you will not need access to any of its functions. As in the previous project, you be using and setting pointers from the user program. You will need to use care when manipulating these pointers as the users address space is distinct from physical memory (i.e., the user program does not start at address 0x0000 although it thinks it does). Process table. The process manager will keep track of process state in a process table, which is an array indexed by process identier (PID). Figure 16 shows the contents of each process table entry. The rst eld in each entry is a status integer indicating the state of this entry: dead (0), running (1), ready (2), or blocked (3). A dead entry is one that is not currently in use. The other three entries indicate the state of a process: (1) currently running, (2) ready to run but not currently running, or (3) blocked waiting on another process or sleeping. At the outset one entry (entry 0) is allocated to the core application and will be in the running state. The other entries are in the dead state and can be allocated to newly-created processes. The next eld is the le name of the executable process, which was passed in the most recent call to exec. A forked process inherits this name from its parent process until it calls exec. The core application, which was not executed from a le, has the value <core app> (without the quotes). Although this eld is not strictly necessary (we can manage processes without keeping

63

track of the exec le name), it allows for more informative process monitoring by the user. The third eld keeps track of the parent PID that created this process (via fork). It is -1 if no parent exists, which will usually only be true for the core application. In VIREOS, the parent PID can change over time. Assume process 0, the core application, is a parent of process 1 and process 1 is a parent of process 2. The PPID of 0 is -1 since it has no parent, the PPID of 1 is 0, and the PPID of 2 is 1. If process 1 terminates then the parent of 2 becomes 0. This update preserves the process hierarchy tree (or potentially acyclic graph should the core application exit early), which is important, as discussed below, when determining whether one process can wait on or kill another process. The next three elds are all used to implement the waitpid system call. The rst of these elds, the wait PID, contains the PID of the process that is being waited on (which we will call the wait process) by the process stored in this entry of the process table (which we will call the blocked process). If it is negative then the blocked process is not waiting on another process to complete. The wait status and wait status pointer are used to pass the exit status from the wait process (when it exits) to the blocked process. The use of these elds are described in more detail in the text following the heading process functions. The sleep eld keeps track of the number of seconds a process is sleeping due to a sleep system call. (Note: a process is blocked if and only if it is either sleeping or waiting on another process.) If the sleep eld is positive, then the process is sleeping. Otherwise, the eld will be 0. Sleeping processes are updated every time a process is scheduled. The sleep amount is decremented by the number of seconds that have elapsed, which can be found using the timer. A process with a positive sleep amount is blocked and cannot be scheduled. It is possible that all processes are sleeping, in which case, the scheduler must busy wait until one of them is nished. Support for sleeping in your process manager is saved for extra credit. The last three elds contain the state of the program minus memory. The rst of these elds holds the current PC of the process. The next is an array holding the current register values. There are 13 entries as only 13 of the 16 registers need to be saved (similar to reg store). Finally, the last eld is an array of the le descriptors. Each entry is a le descriptor structure (from the previous project). Note: the processs memory state is too large to store in the process table. It is stored in a separate le on disk. On any process operation, your process manager will be accessing and/or manipulating the process table. For example, to schedule a new process to run, your process manager will save the 64

state of the currently running process (e.g., PC, registers) in the process table entry for that process, scan the table in round robin fashion looking for a new process to run, and restore the state from the entry of the next process to run. Process image les.. The processs memory image will be stored in special les on disk. These will be stored in a directory called /proc/. Your process manager will need to create this directory, using a le system function, at boot time. The name of the le should just be the PID (e.g./proc/0 for process 0). Whenever a process is suspended, the contents of memory will be written to this le so that later on the process can be resumed. When your process manager reschedules this process, it will copy the contents of the le back into memory. Note that this backup le need not store the entire contents of memory. For example, it should not contain kernel or I/O memory. It should just contain the contents of the user space. A user program could potentially remove one of these image les, as there are no permissions in our le system. If your process manager nds that a processs memory cannot be restored from the image le then it can print an error message and exit. In a commercial system, these contents would not be stored in user accessible les (for security and efciency reasons). Round robin scheduling. Your process manager should schedule processes in a round robin fashion. It does not need a separate data structure for this, but rather can use the process table array. At any point, your process manager should keep track of the PID of the currently running process. When it needs to schedule another process to run, your process manager should start by looking at the next entry in the process table (PID+1). If the currently running process is in the last entry of the process table it should wrap around and look at the 0th position in the table. If that next entry in the process table contains a process in the READY state, it should be run. Otherwise, it should repeatedly move to the next process in the table (as described before) until it nds a process that is in the READY state. If no other processes are READY then the process manager will need to continue running the same process. When inserting processes into the process table, you can insert the new process into any free entry. Note: fairness issues can arise when a new process gets to run before an old one. But because we do not have a separate round robin circular queue there is no way to make this completely fair. However, starvation cannot occur with our approach. Each process will have to wait at most time
9*quanta

for the CPU.

65

Name tmr val tmr start

Address 0xFFFC 0xFFFD

Description timer current value register timer start value register

Table 12: Timer memory-mapped registers. Timer interrupt. Unlike in previous project, we will use a very simple interrupt to enable us to have a preemptive scheduler. The interrupt will trigger after a certain time has elapsed. However, the interrupt will only occur while in user mode. In kernel mode, the interrupt is ignored. This property means that you do not have to worry about handling multiple interrupts simultaneously, and potentially, having a stacked trap handler. The I/O system will not be altered. Our I/O module will continue to use polling to communicate with the keyboard, monitor, and disk. This would be unusual in a commercial system; a commercial system would generally use either all interrupts (more likely) or only polling (simple embedded systems). But this will work well for our needs. We can have a simple I/O system while still studying preemptive scheduling. The timer counts down from a starting value, which can be set by the operating system. When the timer value reaches 0, it triggers an interrupt. Both the starting value of the timer and the current value of the timer (which can be read but should not be written) are stored in memorymapped registers. Table 12 lists these registers along with their memory-mapped addresses. To set the starting timer value, you can write to tmr start. To nd the curent timer value, you can read tmr val. Both memory-mapped registers work in milliseconds. So storing 1000 in tmr start would set the length of the timer to 1 second. Every time tmr start is written, the timer is restarted (whether it has expired or not) and it is set to the start value. In other words, writing tmr start has both the effect of setting the length of the timer and restarting it. Your process manager should use the tmr start register to set the quanta for each process, which should be 5 seconds. In the next project, when we build a memory manager and reduce the context switch overhead, we will lower this length. You may have noticed that while our system is timeshared, it does not seem to support multiprogramming (switching to another process on an I/O block). This is because interrupts cannot occur in kernel mode and communication with devices is done via polling. If performing a costly I/O operation, the current process will not be switched until the I/O operation has been completed and control has switched back to the user program. Imagine one process is waiting for a user to

66

PID 0 1 2 3

PPID -1 0 0 0

STATUS blocked ready ready running

FILE $<$core app$>$ /tests/divisors /tests/findprimes /bin/cat

----------------------------------------------------

Figure 17: Snapshot of the process information le (/proc/info.txt). enter some text while two other processes are suspended but ready to run. Those two processes cannot run until the user is done entering text, which could be a long time. For this reason, our I/O module (which you implemented in project 1) has been augmented. When reading from the keyboard, the polling loop continuously checks the timer value in tmr val. If that value reaches 0, the I/O module, along with the trap handler, stops polling and returns back to the user program (making sure that the last system call will be repeated when this program is rescheduled). This allows the timer to go off and another process to be scheduled. Therefore, one process waiting for user input will not hog the CPU indenitely. On the other hand, because disk and monitor operations are bounded in the time that they take, this is not done for these operations. Memory protection/layout. As in the previous projects, we will continue using the base+offset technique to protect parts of memory and provide a virtual view of memory to the applications. (In the next project, we will use a more advanced and efcient technique.) Although only one process will be in memory at a time, we still need to make sure a buggy application does not write into kernel or I/O memory. We will use a base address register to specify its starting address and a limit value register to specify its address space length. These are memory-mapped registers and can be written as described in Section 3. Process status le. In Unix, one can use the ps command (or the interactive top command) to nd the status of the currently executing processes. In our implementation, we will continuously create and write the status information to a le, which can be read by user programs (e.g., via cat). This le will be stored in the /proc/ directory along with the process image les. It will be called
/proc/info.txt.

It should contain the latest information of each existing process.

Figure 17 shows a snapshot of /proc/info.txt (note: PPID is the PID of the parent). Your le should look similar. You will need to update this le whenever the status changes of any process, whenever a new process is created or terminated, or whenever exec is called and the lename for the process changes. 67

Grading

Name exit getpid fork execv waitpid

ID 26 27 28 29 30 31 32

Description Exit process Get process ID (PID) Fork a process Execute a le Wait for a descendant process to complete Put current process to sleep Kill a descendant process None None

Arguments Status value ($1) PID ($1)

Returns Doesnt return Child process ID if parent, 0 if child ($1) doesnt return if successful, <0 on error ($1) 0 if successful, <0 on error 0 if successful, <0 on error 0 if successful, <0 on error

Required

Pointer to pathname of le to execute ($2) ID of process to wait on ($2), pointer to child status int ($3) Number of seconds to sleep ($2) ID of process to kill ($2)

Extra Credit

sleep kill

Table 13: VIREOS process system calls. System calls. Your operating system will now support several new process-related system calls. The trap handler you will be using has been augmented to handle these system calls. It will call the appropriate function in the process manager module when one is made. Table 13 shows the system calls that will be added to the operating system and which you will need to support in your process manager module. Since you are not modifying the trap handler yourself (a modied version is being given to you), you can ignore the system call identiers and parameter registers. As discussed below, you will implement each of these system calls by completing the corresponding function in the le system module. Note that the last two system calls (sleep and kill) are optional and can be done for extra credit. Process functions. Your process manager will need to support several functions, which will be called by other parts of the operating system. For example, at boot time and halt time, the following two functions will be called by the trap handler: pinitsys: a function for initializing the process management system when booting. This function will need to initialize the process table entry for the core application (entry 0). It will also need to create the /proc directory for storing process image les and the status le (if it already exists due to a system crash it should remove any of its contents). pshutdown: a function for shutting down the process manager when halting. This function will need to remove the /proc directory. In addition, the trap handler will call the following functions when the corresponding system call is made:

68

 pexit: a function called when a process terminates. This function takes as a parameter the exit status and a pointer to the trap handlers return value. It returns a ag indicating whether the system should be halted as a result of the exit. If the exiting process is the last process remaining, pexit should return true, meaning halt. Otherwise, it should return false. The exit status parameter is used for communicating the exit status to any blocked processes waiting for this one to complete. pexit should scan the process table looking for waiting processes. When any are found, pexit should copy the exit status into the corresponding entry in the process table and unblock that process by changing its status from blocked to ready. When the waiting process is eventually rescheduled and its memory image is restored, the exit status is copied into memory at the address specied in the wait status pointer within the process table (this will allow the waiting process to see the exit status of the process it was waiting for).
pexit

also needs to patch the parent PIDs of any child processes of the exiting process. The

parent PIDs of any child processes should be set to the parent PID of the exiting process or -1 if one does not exist. Lastly, pexit should then schedule a new process to run. If there are no processes left, it should return true to halt the system. Otherwise, it should switch to the new process. It should also dereference the pointer to the trap handlers return value and set this to the result register of the next process to execute. Finally, it should return false meaning does not halt. pfork: a function for forking a process. This function takes no parameters and returns the PID of the child process or -1 on failure. It should create a new child process by allocating a new entry in the process table and initializing that entry. The PC, registers, and le descriptors should be copied from the current process to the new entry except for register $1 ($v0, the result register), which should be set to 0 (so that the child process can distinguish itself from the parent). The current state of memory should be copied to a process image le for the child process. Note: to obtain the registers of the current process you will need to access the reg store array (and not the parent processs entry in the table) since the registers will have changed since they were last saved. In addition, the PC will need to be saved by using the address stored in the memory-mapped system return register (the CPU automatically copies the current PC to

69

this register on a trap). Fork should continue running the parent process. This requirement allows the parent process to wait for the child process (if it wants to) before the child process exits (at least in most cases). Because pfork will continue to run the parent processs it should return the parents return value, which is the PID of the child. pexec: a function for loading a new process and executing it. This function takes the path name of the program le and a pointer to the command-line arguments as parameters. It should read from the program le and copy the contents to memory, starting from the start of the user space. If pexec is unable to open or read the program le, or if the contents of the program le are too large to t in the user space, it should return -1 . The contents of memory beyond the length of the program le can be left alone. The state of the process in the process table should also be re-initialized. First, pexec should change the name of the program le in the process table to the specied path name. This will allow it to show up in the process status le. Note: there might be less space for characters in the process table path name than in the parameter. In this case, the characters at the end of the parameter path name should be copied to the process table entry (which are probably more indicative of the exec process). In addition, the PC in the process table entry should be set to 0 along with the registers minus the stack pointer, which should be set to the end of the user space (minus the space for the command-line arguments, which is discussed below). The le descriptors should not be initialized (the shell uses this feature along with dup to implement redirection). One difcult aspect of pexec is setting up the command-line arguments for the process. Every C program has parameters to its main method, which allow it to take arguments from the command-line. The shell utilities like ls, for instance, often make use of this feature to allow users to pass arguments to the program. pexec needs to set up these parameters, which it gets via its own (second) parameter: args, an array of arguments that is terminated by a
NULL entry. pexec will

need to correctly copy these arguments into the user process.

As parameters in C are passed via the stack, pexec will need to copy these onto the end of the stack and set up the stack pointer accordingly. In particular, there are two parameters to main: the rst (argc) indicates the number of total arguments and the second (argv) is an 70

Physical Address 0xFBF2 0xFBF3 0xFBF4 0xFBF5 0xFBF6 0xFBF7 0xFBF8 0xFBF9 0xFBFA 0xFBFB 0xFBFC 0xFBFD 0xFBFE 0xFBFF 0xFC00 0x6BF5 (argv)

Value First free entry on stack (Note: 0x6BF5 is virtual address for 0xFBF5) 2 (argc) 0x6BF7 (argv[0]) (Note: 0x6BF7 is virtual address for 0xFBF7) 0x6BFC (argv[1]) (Note: 0x6BFC is virtual address for 0xFBFC) / (argv[0][0]) f (argv[0][1]) o (argv[0][2]) o (argv[0][3]) \0 (argv[0][4]) a (argv[1][0]) b (argv[1][1]) c (argv[1][2]) \0 (argv[1][3]) Start of I/O space

Figure 18: One possible layout of the initial stack on an exec call. array of strings, one for each argument. If there is at least any arguments, then the pointers to the strings and the contents of those strings will need to be copied to the stack as well. These will need to follow the parameters in memory. This is best shown by example. Assume the program /foo is being executed in pexec with arguments /foo and abc (note: in C the rst argument is always the program name). Figure 18 shows the layout of the stack when the proram begins to execute (although other variations are possible). The stack pointer initially contains the address 0xFBF3, the value right after the rst free entry on the stack. argc and argv follow the rst free entry on the stack, which is a requirement, as this is where the user program will expect them (note: parameters are always stored in reverse order on the stack). Since argv is an array the value at the argv location in memory is just an address, which points to the contents of the array. Because the user program will be accessing this array, the addresses must be virtual (i.e., physical address minus the starting physical address of the user space, which is 0x9000). That is why the value at argv is 0x6BF5 rather than 0xFBF5.

71

We must also nd some place in memory to house the contents of the argv array. It makes sense to put the them between mains parameters (argc and argv) and the end of the user space (0xFC00). At (physical) addresses 0xFBF5 and 0xFBF6 we put the contents of the array (argv[0] and argv[1]). Of course these are themselves pointers to arrays of characters. So each entry contains a (virtual) address that corresponds to a string. The characters in the strings are housed between the array and the end of the user space. Notice that you could vary this to some extent. For example, you could put the strings rst and then the argv array so long as all the pointers were adjusted accordingly. This aspect of the process manager will be challenging so you might save it for last. But it is important. None of the programs that expect command-line arguments (e.g., ls, cat, rm) will work correctly until this is implemented. pwaitpid: a function to allow a process to block (called the blocked process) and wait for another specied process (called the wait process) to complete. This function takes the PID of the wait process as a parameter as well as a pointer in memory where the exit status of the wait process can be written when it completes. A process can only wait on a descendent of itself (child process, grandchild process, etc.). If performing an illegal wait or the specied process does not exist then -1 is returned. Otherwise, the current processs status in the process table is changed from running to blocked. The wait PID of the blocked process in the process table is set to the PID parameter and the wait exit status pointer is set to the status pointer parameter. As described in pexit, when the wait process nishes, the exit status is written into the blocked processs entry in the process table, in addition, to it being unblocked. When the process is eventually rescheduled and its memory image is restored, the exit status is copied to memory at the address stored in the wait status pointer (so the blocked program can see the exit status of the wait process). pgetpid: a function for getting the PID of the current process. This function takes no parameters and simply returns the PID of the current process. In addition, whenever the timer interrupt occurs (i.e., the current process uses all of its quanta) then the trap handler will call the following function to schedule a new process to run:

72

 pschedule: a function for scheduling a new process to run. Whenever the timer interrupt occurs (i.e., when the current process uses all of its quanta) the CPU will trap to the OS and the trap handler will call pschedule to schedule a new process to run. If pschedule nds a new process to run it will switch to this process, saving the state of the current process and restoring the state of the next process. If no process is found, then pschedule will continue running the current process. pschedule returns the current value in the result register ($v0 or
$1)

of the process switched to.

To save the state of the registers, pschedule will need to access the array reg store where the trap handler saves all register values. pschedule will need to copy these values into the process table. To save the PC, pschedule will need to access the memory-mapped system return register (the CPU automatically copies the current PC to this register on a trap). Finally, the state of memory can be saved to the process image le. To restore the state of the registers and PC, pschedule just needs to get the saved values from the process table entry of the next process to run and write them into the reg store array and memory-mapped system return register, respectively. To restore memory, this next processs image le must be copied into user space memory. There are also two functions implementing the system calls sleep and kill. As discussed below, these can be done for extra credit.

8.3

Error Handling

As with the le system, one important and challenging part of this assignment is handling errors. There are many that can arise and your process manager should catch them all and handle them appriopriately. Some have already been mentioned in other parts of the project writeup but many have not. When implementing any function, you will need to think carefully about the things that could potentially go wrong and prevent these pitfalls. In particular, you should think carefully about the following two classes of errors: Function parameters. These are generally coming from the user program via a system call and should not be trusted. You should check all of these carefully. For example, many functions require a process identier (PID), which could be erroneous (e.g., a negative value). 73

 Process limits.. In any system there are limits, which the operating system must make sure are never surpassed. For example, there is limit to the number of processes that can be exist at any point. You must ensure that these limits are never reached. For function parameter errors you should not print any type of error message, just return the appropriate value. The application can decide how to report the error. Process limit errors should be reported by the OS via a print message so the user knows what went wrong (and does not assume it was a parameter error). Unless the error is catastrophic, you should return an error value in these cases as well (for a catastrophic error you can halt the machine). Error handling is an important component of any system and this importance will be reected in the grading. So make sure you take error handling seriously.

8.4

Writing the Process Manager

You will write your process manager in C-- [6], which is a variant of C. See Appendix 10 for more details on C--. This project will require a similar amount of time as the previous one. Although you will write less code, you will probably spend a signicant amount of time error testing. Make sure you give yourself plenty of time to implement and test your process manager. The le proc.c contains the code implementing the process manager. It is largely incomplete; you will complete it in this project. At the very least, you will need to implement the functions listed above under the heading Process functions in the previous subsection. You will probably also want to write some additional functions to avoid code redundancy and complexity. For example, you might nd it helpful to create the functions for doing the following: Initializing a process. Switching to another process. Terminating a process. Saving/restoring memory from a process image le. Note: this list is by no means exhaustive. Furthermore, there are other possible ways you might organize your code. Feel free to organize your code in any way you see t so long as it is readable by others. The other parts of the operating system have been modied to support multiprocessing. For example, the trap handler will call the appropriate function in the process manager on a process 74

system call or a timer interrupt. Moreover, the I/O handler will postpone reading from the keyboard if a timer interrupt occurs so that a process switch can occur. The le system has also been extended slightly to support multiprocessing. First, it now has support for multiple le descriptor tables. One will be needed for each user process. The following functions have been added to the le system manager: fdinit: intialize a set of le descriptors. This function can be used when initializing a process table entry. fset fds: set the le descriptor table. This function can be used at boot time or on a context switch to the set the le descriptor table to a particular processs table. fcopy all fds: copy a set of le descriptors. This function can be used to copy a set of le descriptors from one process to another. The le system module also has a set of system le descriptors that can be used by other parts of the operating system when working with les. (Note: it would be bad idea for the OS to use a user programs le descriptors. For example, assume the user has opened the maximum number of les already.) Each le system clal that works with a descriptor has a system ag parameter, which indicates whether the system or user descriptors should be used. Check out the project 3 API off of the VIREOS web site (http://math.hws.edu/vireos to see how to call each le system function. Although challenging, some of this project can be done incrementally if you plan carefully. There are a few strategies that you could try but here is one possibility: Initially, get your process manager working with a single process. You will need to implement pinitsys for initializing the process manager and the core application, and pexit for exiting. Do not set the timer so that no timer interrupts occur. Next, add code to set the timer so that interrupts occur and handle them appropriately (on an interrupt the trap handler will call pschedule). After that, you might implement pfork, since all the multiprocessed tests use it. Finally, you can implement the other functions such as pexec, pwaitpid, etc.Here you can use whatever order you want. As discussed below, one of the test programs will allow you to test these system calls in isolation so long as fork, exit, and scheduling are already implemented.

75

Name proc-test.c shell.c shell-nonint.c

Description Interactively prompts user for a process operation Multi-processed, interactive shell program Multi-processed, non-interactive shell program (runs a xed set of commands)

Table 14: Assignment 3 test programs.

8.5

Testing the Process Manager

Inside the directory tests are several test programs for testing your process manager. They are listed in Table 14 and include a interactive test program, a (multi-processed) interactive shell program, and a non-interactive shell program, and a stress tester. The process test program, called proctest

contains a program for testing some of the basic functionality of your process manager. It

is similar to le-test from the previous project. It has a menu system that allows the user to test various process system calls and see if they work. It will not, however, be appropriate for testing how several processes run concurrently on the system. To test your process manager when running several concurrent programs, you should use one of the shell programs. The rst is an interactive shell program called shell like the one you built in project 0. It is multi-processed, meaning the shell utilities (e.g., ls, cat) run in separate processes. The shell utilities are already on the disk that came with your code (in the le disk). Use the command ls, one of the provided utilities, to nd others. There are also two built-in shell commands:
exit

(for exiting the shell) and echo (for printing strings to standard output). These can be run There is also a non-interactive shell called shell-nonint, which runs a set of xed commands.

without having to create new processes and can be used to do some initial testing. These commands provide fairly good coverage over all of the process management functionality. The le shell-nonint-output.txt contains the correct output of the non-interactive shell. You can compare the output when run on your system to the working output. As mentioned earlier, during the course of testing your operating system, you may nd that your le system and disk get corrupted. If that occurs, then you can reload the original disk le by entering make disk in the working directory. (after all there is no journaling in our le system). If that occurs, then you can reload the original disk le by entering make disk in the working directory. Since only one process can be in memory at a time, you will nd that running a new process takes a signicant amount of time. In the next project, we will address much of this inefciency, 76

but for now, you will have to make due. In particular, each shell command and test in proc-test will take several seconds to execute. In addition, the non-interactive shell program will take a minute to execute (although since it is non-interactive, you can do other things while it is executing). So make sure you leave yourself plenty of time for testing. Feel free to edit any of tests. For example, you might edit the non-interactive shell to shorten it. In this case, you would edit the le shell-nonint.c and comment out some of the calls to exec prog (which executes a command) in the main function. But, remember, you will be graded in part on whether these test programs work correctly with your operating system. At some point, you should test your operating system using the original programs.

8.6

Extra Credit

For some extra credit (not as much as implementing directories in the previous project), you can implement two additional system calls in the process manager. psleep for sleeping a specied number of seconds. This function takes the number of seconds to sleep as an argument, which must be positive. Sleeping processes should be blocked for the specied number of seconds. If all processes are blocked (some of which must be sleeping) then the operating system should busy wait until one becomes available. The timer can be used to keep track of when a process is done sleeping. pkill for killing a process. This function takes two arguments: the process identier to kill and a status value to send to the process when terminating it. A process can kill only itself or a descendant process. The function returns 0 if successful or <0 on an error. Important: do not start the extra credit until you nish the requirements. In addition, make a copy of your working solution before starting the extra credit.

8.7

Last Words

Remember to start early on this project. This project will take as long to complete as the previous project if not longer. Even if you are able to write the code quickly, do not underestimate the amount of time you will need in testing your implementation. Good luck and remember to have fun! 77

Project 4: Memory Manager

In this (nal) project, you will implement a memory manager for the Larc operating system. Your memory manager will allow multiple processes to efciently share physical memory, which will greatly improve the overall performance of the system. Your memory manager will use paging to place processes non-contiguously in memory, as well as to provide each process with a virtual address space that is disconnected from the size of physical memory.

9.1

Files and Directories

You will need to complete the source code in the proj4 assignment directory (inside the projects directory). In particular, the les mem.c, which contains the memory manager is incomplete and must be nished in this assignment. This module has a header le (mem.h containing some macro, variable, and function denitions that will be useful both in mem.c as well as by other parts of the operating system. You should not edit the header le. mem.c contains some skeleton code, i.e., the signatures of functions dened in the header le, as well as some useful macro and struct denitions. The operating system will compile initially but the memory manager will of course not work properly until you complete the assignment. As in the previous three assignments, there are also some auxiliary modules (e.g., util, strings), which you may nd useful when implementing the memory manager. You have also been provided with a working a version of the VIREOS trap handler, I/O handler, le system manager, and process manager from the previous three projects. The compiled code is in the Larc assembly le
lib.s,

which is included in the target code when the operating system is built. You should not edit

this le. Note: the trap handler has been extended to support a new trap on a page fault. The I/O handler and le system are the same as in the previous project. Although the .c les for the trap handler, I/O handler, le system, and process manager are not provided, the header les (trap.h,
io.h, le.h, proc.h)

are provided since your memory manager may need to call functions in these

modules. For more details on any of the provided code, see the header les for each of these modules or check out the online application programming interface (API) for project 4 off of the VIREOS web site (http://math.hws.edu/vireos). As in the previous three assignments, the working directory also contains a makele for build-

78

ing the operating system (i.e., enter make), a pre-compiled version of the operating system (osref.out), tests

a program ndsrc for mapping a memory address back to a C source line, and a directory

with various test programs. After building your operating system for the rst time, you will

also have a simulator le (sim) for running your compiled version of VIREOS, a reference simulator le (sim-ref) for running the simulator with the reference operating system, and a disk le (disk) containing the contents of the simulated disk. See the trap and I/O handler assignment (Section 6) for more details on any of these les or directories. As in the previous assignment, the disk comes pre-installed with various les and directories needed for testing your memory manager. If for some reason your disk becomes corrupted, remove it and enter make disk to restore it to its original contents.

9.2

Project Details

Here are some details concerning various aspects of the project assignment. General. You will be implementing a memory manager, which will allow processes to efciently share physical memory. Your memory manager will use paging to provide non-contiguous memory storage of processes. It also provides a virtual address space, which is not tied to physical memory. The logical address space of each process will start at 0x0000 and end at 0x6C00. Obviously, since we are using paging, the address space will be broken up into several pages. To keep the page table small, we will use a page size (and frame size) of 256 words. In a commercial system, we would use the entire addressable range (216 in our case), but limiting it to 0x6C00 allows us to store one processs address entirely in physical memory and store it in a single le. To simplify our implementation, only the user space in physical memory will be paged and not the kernel or I/O spaces. Figure 19 shows the layout of memory with paging. User memory consists of 108 frames, which can hold 108 pages from various processes. We will use an inverted page table (i.e., a page table with an entry per frame rather than per page) to store the translation from virtual addresses to physical addresses. We store the entire inverted page table in a hardware cache rather than in memory (or worse, on disk). The page table will be stored in the translation lookaside buffer (TLB), which the CPU will use to translate virtual addresses used by the program into physical addresses. If the page is not in memory, then the CPU will trap to the operating system and the trap handler will invoke your memory manager to handle the page fault. The TLB is loaded via sofware, i.e., the OS updates it on a trap using memory-mapped I/O. Note: 108 entries 79

0x0000
Kernel space

0x9000 0x9100 0x9200 0x9300 0x9400

User frame 0 User frame 1 User frame 2 User frame 3

...

0xF800 0xF900 0xFA00 0xFB00 0xFC00

User frame 104 User frame 105 User frame 106 User frame 107 I/O space

Figure 19: The memory layout with paging. is probably too large to t in a (fully-associative) TLB on a commercial machine. We cheat here to keep the operating system simple. On a page fault, your memory manager will need to select a frame for holding the requested page. In this case, you may need to evict a page from some frame if none of the frames are empty. You will use the not recently used (NRU) page replacement policy to select a page to evict. Although this algorithm is not generally a good approximation of least recently used (LRU), it is simple to represent and has only two bits of state per page table entry: a bit indicating whether the page has been read and a bit indicating whether the page has been written. These bits are housed in the TLB and automatically updated by the CPU on a memory load, memory store, or control transfer. You will be given a working version of the trap handler, I/O handler, le system manager, and process manager. They have been augmented to integrate with the memory manager, to provide support for page fault traps, and in other ways. Although you will not have source code for these modules, you can view the header (or interface) les (i.e., proc.h, fs.h, io.h). Note: there is no interface le for the trap handler as you wont need access to any of its functions.

80

Page table. The page table in our memory manager is a logical data structure, meaning we dont actually keep track of all of it in hardware, memory, and/or disk. Instead, we keep track of the pages that are just in physical memory, which can be viewed as an inverted page table. Because the amount of physical memory dedicated to user programs is small, this inverted page table is small and can be housed entirely in hardware (in a TLB as discussed below). Process image les. Of course, when a page is not in the inverted page table, we need some way to nd it on disk. For this, we will use a similar approach as in the last project. Each process will have a process image le containing the logical address space of that process. When a page is on disk, your memory manager will go to this image le to nd the page. Because the image le contains the entire logical address space, your memory manager can seek to the virtual address of the start of the requested page, and then read a page into memory from the le. Likewise, when writing an evicted page from physical memory to disk, we can write the image le by seeking to the virtual address of the start of the evicted page, and writing a page from memory to the image le. The process image les will be named similarly as in the previous project: /proc/<pid> where
<pid>

is replaced with the PID of the process. These are automatically created and removed by

the process manager (similar to what you did in the previous project although a few optimizations have been added). You just need to make sure you update them on a page fault or on a page save (which occurs on a fork). TLB. The inverted page table will be stored entirely in the translation lookaside buffer (TLB) in hardware. The TLB has an entry per frame in user space and is ordered by frame number. As a result, the frame number does not have to be stored in an entry (TLB entry 0 houses the translation information for user frame 0, and so forth). Of course, because the TLB is ordered by frame number and not by page number the CPU must search all of its contents to nd a match (i.e., it is fully associative). If the requested page is found, then the translation is performed (the upper bits of the physical address are implicit in the matched TLB entry). If a page is not found, the CPU traps to the operating system, which uses your memory management module to service the miss in software. Because there is a TLB entry per physical user frame, a miss in the TLB implies a page fault. So your memory manager will need to bring the page in from disk as well as update the TLB. The TLB is read and written using memory-mapped I/O. Table 15 lists the TLB entry addresses.

81

Entry tlb[0] tlb[1] tlb[2] ... tlb[105] tlb[106] tlb[107]

Address

0xFF90 0xFF91 0xFF92

...

0xFFF6 0xFFF7 0xFFF8

Table 15: TLB memory-mapped register addresses.

V? 1 Page number 7 PID 6 W? 1 R? 1

Figure 20: TLB entry. In the operating system OS module header le (os.h), there are macros for accessing these entries. Figure 20 shows the layout of a TLB entry. On the far left, the highest order bit, is a valid ag (V?), which indicates whether this TLB entry contains a valid page table entry. After that is the page number (in 7 bits although the high order bit is always 0) of the page housed in the corresponding frame (assuming the TLB entry is valid). After that is the PID of the owner process (in 6 bits). Finally, on the far right, are two bits, which indicate whether the page referred to by this TLB entry has been read and/or written. Memory protection. Each TLB entry contains the PID of the owning process. This is to protect pages owned by one process from being read or written by a different process as all the processes are now sharing physical memory. Whenever the TLB is searched for a page, the PID of the currently running process must match the PID in the TLB entry. The CPU searches the TLB for an entry with the corresponding page number, valid bit set to 1, and PID equal to the current PID. Page faults. As mentioned above, a page fault occurs when the CPU is unable to nd a match in the TLB. In this case, the CPU performs a trap into the operating system. The CPU writes the virtual address to a memory-mapped register called mem pfault (0xFFF9) where it can be retrieved by the trap handler. The trap handler gets this address and then passes it along to the memory manager, which can service the page fault, bringing in the requested page, and, potentially, evicting another page.

82

NRU page replacement. To select a page to evict, you will use the not recently used (NRU) page replacement policy. In particular, the CPU keeps track of whether a page in physical memory has been read or written by setting some bits in the corresponding TLB entry. Your memory manager will use these bits to determine if a page was read or written between the last page fault and the current one. In picking a page to replace it will use the following priorities: 1. Unused frames. If there are any unused frames (i.e., TLB entries that are marked invalid), these should be lled before evicting a frame with a valid page in it. 2. Untouched frames. Frames that have not been read or written since the last page fault should be selected next. 3. Read but unmodied frames. Frames that have been read but not written since the last page fault should be selected next. 4. Modied but unread frames. Frames that have been modied but not read since the last page fault should be selected next. If a modied page is selected, it will need to be written back to disk. 5. Read and modied frames. Finally, a frame that is read and modied should be selected if there are no other frames in the classes above. If a modied page is selected, it will need to be written back to disk. At every page fault, your memory manager will reset the read and write bits in the page table entry in the TLB so that you can perform the same analysis at the next page fault. You will need to remember any modied frames that you reset so that you can write them back to disk if they are evicted later on. For example, a page that is written between page fault 1 and page fault 2 but then not accessed between page fault 2 and page fault 3 might be selected for replacement during page fault 3 as its read and write bits will be 0. But it must be written back to disk since it was modied between page fault 1 and page fault 2. If there are ties when performing NRU, you can arbitrarily select a frame, however, when applicable, you should select a page that will not need to be written back to disk. For example, if considering evicting page A and B, which were both read but not written between the last two page faults, you should look at whether either page has ever been written. If A was written at some point in the past but B was not, then you should select B for replacement. 83

Memory functions. Your memory manager will need to support several functions, which will be called by other parts of the operating system. These include: minit: a function for initializing the memory manager. This function is called by the trap handler at boot time. It will need to set up all of the page table entries in the TLB. Initially, each page table entry should be valid, unwritten, unread, and belong to the core application (PID 0), which starts in physical memory. Each page table entry will also need to contain the correct page number (you will need to gure out what that is). mget real addr: a function for converting virtual addresses into physical addresses. It takes the virtual address and PID of the current process as parameters and returns the corresponding physical address or -1 if the page containing the address is not in physical memory. This function is called by the trap handler so that it can pass other OS modules (e.g., the le system) physical addresses rather than virtual addresses. In addition, the trap handler can also use this function to determine when a page fault has occurred due to an address specied within a system call. (If the address wasnt actually loaded or stored, but rather provided in a system call, such as fread, it would not necessarily have caused a page fault trap.) mfault: a function for servicing page faults. This function is called by the trap handler whenever a page fault trap occurs. This function takes as a parameter the the faulting virtual address and the PID of the current process. It should bring the requested page into memory from the process image le on disk, evicting another page using the NRU page replacement policy. If the evicted page has been modied then it will need to be written back to the process image le on disk. This function should also clear the read and write bits of the page table entry in the TLB so that the NRU page replacement policy is always considering only the most recent window of memory references. However, because of this, your function will need to keep track of modied pages so that if any are later evicted they will be written back to disk. msaveall: a function for saving all of the modied pages of a process to disk. This function takes as a parameter the PID of some process and saves all the valid, modied pages of that process to the corresponding process image le on disk. 84

This function is called by the process manager on a fork to synchronize the parent processs image le with memory. This is necessary because on a fork the parents image le must be copied to the childs image le and physical memory cannot be used as in the previous project since some of the parents address space might be on disk rather than physical memory. mevictall: a function for evicting (and not saving to disk) all of the pages of a specied process. This function takes the PID of the process as a parameter and invalidates any valid page table entry in the TLB belonging to this process. It should not write back modied pages to disk. This function is called by the process manager whenever a process is terminated or an exec occurs. Because the process is either completed or will no longer need this memory any more (on an exec), the modied pages do not have to be written back to disk. mswitch pid: a function for switching the PID stored in a TLB entry. This function takes an old PID, a new PID, and a page number. It should check to see if the specied page is in the TLB, is a valid entry, and contains the old PID. If such an entry is found, then this function switches the PID to the new PID leaving all other parts of the TLB entry alone. In this case, the function returns true indicating it was successful. Otherwise, it returns false indicating failure. This function is used by the process manager to allow for page sharing between processes. Currently, this feature is only used for implementing copy on write (for optimizing fork) when enabled. Although copy on write is currently disabled we may enable it later on (you will need a lib.s patch to enable it), which will greatly improve the performance of fork. However, this function is still required even if copy on write is left disabled.

9.3

Writing the Memory Manager

You will write your memory manager in C-- [6], which is a variant of C. See Appendix 10 for more details on C--. This project will require a similar amount of time as the previous two. Like the previous project, you will probably spend a signicant amount of time error testing. Make sure you give yourself plenty of time to implement and test your memory manager. The le mem.c contains the code implementing the memory manager. It is largely incomplete; you will complete it in this project. At the very least, you will need to implement the functions 85

Name mem-stress.c proc-test.c shell.c shell-nonint.c

Description Memory system non-interactive stress tester Interactively prompts user for a process operation Multi-processed, interactive shell program Multi-processed, non-interactive shell program (runs a xed set of commands)

Table 16: Assignment 4 test programs. listed above under the heading Memory functions in the previous subsection. You will probably also want to write some additional functions to avoid code redundancy and complexity. For example, you might nd it helpful to create the functions for doing the following: Getting the page number from a virtual address. Extracting the components of a page table entry in the TLB. Seting the components of a page table entry in the TLB. Note: this list is by no means exhaustive. Furthermore, there are other possible ways you might organize your code. Feel free to organize your code in any way you see t so long as it is readable by others. One thing that makes this project more challenging than any of the prior projects is that it is difcult to do incrementally. With some cleverness, you may be able to test aspects of it incrementally, but for the most part you will nd this infeasible. So make sure you design and implement your code more carefully than in previous projects, and that you allocate even more time for testing. Be prepared to spend a signicant amount of time testing your implementation.

9.4

Testing the Memory Manager

One nice aspect of testing in this project is that the memory manager will signicantly speed up the test programs (although programs will still take a few seconds to execute). For instance, the context switch time has been reduced from 5 seconds to 1 second. So your test programs should run a lot faster. The downside is that testing the memory manager is more difcult than testing any of the other modules. It can be hard to track down an error in your memory manager. Assume for example, that one entry is in the TLB is incorrectly set so that the user program will access the wrong virtual page. When the user program is resumed it will most likely crash, but the cause of this crash will 86

not be apparent. In this case, you will need to go carefully through your memory management functions looking for possible errors. The good news is that the module is much smaller than previous modules. In other cases, with careful consideration, you might be able to isolate the error to one or two functions, but in general this is difcult. Since the system calls have not changed from the previous assignment to this one, we can mostly use the same test programs. Figure 16 shows the memory manager test programs, which are in the tests directory in the working directory. All of the test programs from the previous project are included (see Section 8 for more details) as well as one new test called mem-stress. This test program is non-interactive and runs stresses the memory management system. If this program works correctly for you, along with the others, then you are probably in good shape (although there may still be bugs!). As in previous feel free to add tests or modify tests, but make sure you also run the original tests.

9.5

Last Words

Remember to start early on this project. Like the previous project, testing will be challenging and it will take a signicant amount of time. Even if you are able to write the code quickly, do not underestimate the amount of testing time required. Good luck and remember to have fun!

87

10

Closing Remarks

Hopefully if you have reached this point then you have successfully implemented your very own operating system. If so, congratulations! We hope that you have enjoyed the VIREOS course projects and found this manual helpful and informative. Please let us know if you found any bugs or errors along the way, in either the VIREOS toolset or in this manual. In addition, please contact us if you have questions about any part of this work or if you have requests for extensions. Finally, we welcome contributions from others, so please let us know if you would like to contribute in some way. To contact us you can email the author at corliss@hws.edu.

Thanks, Marc Corliss

88

Appendix A: C-VIREOS is written in C-- a simple language based on C. This section describes the differences between C-- and C, and describes some of the special compiler features available when compiling the VIREOS kernel. It assumes that the reader is already procient in C. If this is not the case, you should review C [5]. C-- versus C. C-- was originally developed at the University of Wisconsin Madison by Jim Lenz [6], although it has been extended in several ways by the authors of this manual. Lenz originally called the language Wisc C--, but as it is has been extended in some signicant ways (at a different institution) we will call it simply C--. C-- is basically a subset of C. The C-- syntax and semantics are essentially the same as in C. C-- includes the core features of C such as basic control ow, functions, (single-dimensional) arrays, and structs, although some features were excluded. You might be wondering why C-- was used rather than C. The primary reason was that there was a simple, extendable compiler available for C--, which was easy to retarget to the Larc architecture [3]. Since C-- contained most of the features of C, we did not feel it was worthwhile to retarget a C compiler, which probably would have required more time. In addition, as discussed below, it was easy to customize the compiler for compiling a kernel. In the future though, we are considering migrating from C-- to full C. We hope to add many of the features from C that were excluded in C--. These features include: Single statement variable declaration of multiple variables. Switch statements. Loops besides while and for. Multiple variable initializations or updates in a for loop. Nested functions. Functions with variable arguments. Conditional expressions (i.e., ?, : operators). Goto statements and label denitions. Primitives besides int and char. Variable declaration modiers (e.g., unsigned, const). Multi-dimensional arrays (although you can use an array of pointers). Variable-sized, stack-allocated arrays (these must use xed sizes). Function pointers (although these can be declared and set, they can not be called). Type denitions via typedef. Nested or locally-declared (i.e., within a function) structs. 89

 Unions. Separate compilation. In addition, in C-- all non-void functions must end with a return statement (which cannot be embedded inside another statement such as if/then/else). C-- also has different library modules then C. For example, it has its own stdio (necessary since C-- does not support variable arguments in functions such as printf), among others. The API for the library modules is available off of the VIREOS web site at http://math.hws.edu/vireos. However, C-- retains most of the features of C and in practice the programming experience is much the same. Like C it is preprocessed so programmers can dene macros and include header les. It includes functions, if statements, while loops, for loops, break/continue statements, and casting (among other features). C-- includes nearly all of the C operators. The bitwise operators (e.g., , &) will be particularly useful when implementing the operating system. C-- also includes shortcut operations such as ++ and *=. Because C-- is implemented on a 16-bit Larc machine (see Section 3) one must be careful of overow when performing arithmetic. For example, an int has a signed range of -32768 (-215 ) to 32767 (215 1). Multiplying 10000 by 8 will overow (the result be 14464 instead of 80000). Larc also does not have built-in support for signed integer values. Larger ranges or unsigned integer operations must be simulated in software. For the most part, these issues will not arise, but they may in some contexts. C-- is not a strict subset of C. In particular, C-- includes a boolean type unlike C, which works like the boolean type in other languages (e.g., Java). Predicates in if statements, while loops, and for loops require a boolean type. C-- also uses some of the C99 standard [1]. In particular, it allows variables to be declared anywhere in a function rather than at the start of a block. It also allows a variable to be declared in the initialization statement of a for loop. Compiling the kernel. The C-- has one argument -k (see Section 2 for details on the C-- compiler command-line arguments), which should be used when compiling the VIREOS kernel. When in kernel mode, the compiler will treat the main function as the main trap handler function (the other functions are not treated in any special way). It will place this function at the beginning of the code section so that the (simulated) CPU will transfer control to it on a trap (the CPU automatically transfers control to the rst kernel address or 0x0000). It will also generate code for saving and restoring user registers (as discussed below) when entering and leaving this function so that 90

the user registers will not be corrupted on a trap. Because of this feature, the operating system can be written entirely in C-- (i.e., no parts of it have to be written in assembly code). As mentioned above, when compiling the kernel, the C-- compiler automatically generates code for saving and restoring user registers. To save and restore values without modifying a user register, the compiler makes use of the Larc kernel registers ($14-$15 or $k0-$k1), which are not accessible by user programs. In addition, the compiler automatically denes a special array variable called
reg store

for accessing these register values. No other variables can be declared using this name.

This array is a single-dimensional integer array with one entry for each register that must be saved and restored; it has size 13. Although the Larc architecture technically has 16 registers, three of them need not be saved and restored: $0 ($zero), which always holds 0, and the kernel registers,
$14 ($k0)

and $15 ($k1). reg store[0] holds register $1, reg store[1] holds register $2, and so on.

At the start of the main trap handler function, the compiler automatically generates code, which uses the kernel registers, to save the user registers to the reg store array. At the end of the main trap handler function, the compiler automatically generates code to restore the registers from the
reg store array.

Between these two points, the reg store array can be manipulated in whatever way

the student sees t. For example, it can be manipulated in order to support context switching. On a trap, the values in reg store, which hold the register values from the currently-running process can be saved to the process table. The saved register values in the process table for the next process to run can then be loaded into reg store. The student will be implementing this in the process manager lab. The library modules cannot be used in the operating system code. This is because much of the library functionality (e.g., printing to the screen) must be implemented within the operating system, itself. However, some auxiliary modules are provided such as a module for manipulating strings.

91

References
[1] International standard programming languages C. ISO/IEC 9899:1999. [2] Altera Coorporation. DE1 Development and Education Board User Manual, 2006. [3] Marc L. Corliss. Larc A Little Architecture for the Classroom: Lab Manual, 2009. URL: http://math.hws.edu/larc. [4] Marc L. Corliss and Robert Hendry. Larc: A little architecture for the classroom. Journal of Computing Sciences in Small Colleges, 24(6):1520, 2009. [5] Brian W. Kernighan and Dennis M. Ritchie. The C Programming Language. Prentice Hall, 1988. [6] Jim Lenz. Wisc c-- compiler. URL: http://pages.cs.wisc.edu/ lenz/compiler.html, 2003. [7] MIPS Technologies, Inc. MIPS32 Architecture For Programmers: Volume I: Introduction to the MIPS32 Architecture, 2001. [8] Abraham Silberschatz, Peter Baer Galvin, and Greg Gagne. Operating System Concepts. John Wiley & Sons, Inc., 8th edition, 2008. [9] William Stallings. Engineering a Compiler. Pearson Prentice Hall, 6th edition, 2009. [10] Andrew S. Tanenbaum. Modern Operating Systems. Pearson Prentice Hall, 3rd edition, 2008. [11] Dimitri van Heesch. Doxygen Manual, 2010. URL: http://www.doxygen.org/manual.html.

92