Previous lessons discussed programming logic and the organization of programs with subroutines. They also talked about markup languages and how they're used to add to extra information to data documents. This lesson covers how you can save and read data, and how that data is stored in memory so your program can work on it.

Different Types of Computer Memory

In the technological saga of the computer revolution, memory has always been one of the dominant factors. Early computers used some bizarre methods of data storage; for example, the first Univac stored data as vibrations propagating through thin columns of mercury. (Really! This isn't a fib! Look here). Imagine a room full of Fastrand memory drums rumbling like a power plant and holding less data than a single hard drive in a modern laptop.

As CPUs (central processing units) got faster and more compact, the limitation of the speed of light became significant and computer designers concentrated on keeping memory as close as possible to the CPU.

Now, computers remember data with a variety of technologies. As a programmer, you need to understand what these technologies imply. Table 6-1 lays out the territory for further exploration.

Table 6-1: Characteristics of various kinds of memory.

Modern CPUs have a chunk of high-speed memory, called cache memory, directly on the CPU chip. Data is moved to and from the main chip memory by CPU logic that tries to predict what the program is going to want next. This improves the overall speed by having needed data in the fastest possible memory. There's no good way for the programmer to influence this process.

Main chip memory uses an enormous number of tiny transistors to hold the bytes that your program uses for program code and data. Even the slightest loss of power can cause these transistors to forget everything. The language you use has a lot to do with how you manage this memory, as you learn later in this lesson.

Flash memory is a special kind of transistor-based memory that can hold a value after the power goes off. This is the memory that holds your favorite TV channel settings when the power goes off, or holds your digital camera pictures in a memory stick. Usually the programmer has to do something special to use flash memory unless the operating system makes it look similar to a file system.

Hard disk memory stores data as magnetic patterns that can be read or written under the control of your operating system. It's the operating system's responsibility to present this data to your program in the form of a file system.

There are several varieties of removable media in use these days, from floppy disks to DVDs. Your operating system generally handles the hardware differences and presents the programmer with a consistent file system interface. Flash memory, hard disk storage, and removable media are collectively called nonvolatile storage because continuous power is not required to hold the data.

When storing data in a database -- particularly if the database management is handled by someone other than you -- the programmer's life is greatly simplified. Communication with databases is discussed in Lesson 7.

Bits, Bytes, and Word Size

Historically speaking, computers designers have tried many different ways to organize memory. Fortunately for computer users, designs have converged on using an 8-bit byte as the basic unit of memory. One reason for this convergence is that with 8 bits, you can represent all of the numerals, punctuation, and characters in English text, with some left over for special characters.

Although the original microcomputers read and wrote to memory one byte at a time, modern CPUs read and write in 32- or 64-bit (4 or 8 byte) chunks. These details are hidden from the programmer, however, and the fundamental unit that you work with is still the byte.

When working with the simple programs you've played with so far, you haven't had to worry much about having enough memory to hold your variables. The Help system for Liberty Basic does not even mention memory; however, if you try to dimension some large array with statements such as the following, you find the program fails mysteriously.

dim verticesX(10000000)
dim verticesY(10000000)

This fails because your computer doesn't have an unlimited amount of space in main memory to store data, and you run into trouble when you request the space to store 20 million numbers. In fact, if you're running a typical Windows or Mac desktop, there are many programs of various sizes sharing memory so the amount available for any one program is greatly reduced.

The following are some of the types of programming that require lots of memory:

• Games with complex graphics
• Computer-aided design
• Web servers and other utilities that service many users and run for long periods

This brings you to the topic of memory management. You run into two types of memory management in programming languages -- automatic, and what might be termed explicit memory management. Using C, you have to explicitly request memory from the operating system for something such as the large arrays of numbers in the previous example. You must also explicitly tell the operating system when you are through with it. Too, you must make sure that you don't have two parts of your program trying to use the same memory for different purposes.

If, through bad programming practice or an error, you forget to return the memory to the control of the operating system, your program experiences what's called a memory leak . A memory leak means your program keeps requesting memory, but never gives it back to the operating system. Eventually the operating system refuses to give you any more memory and the program dies.

A memory leak problem can be one of the most frustrating things in all of computing. Frequently they only show up when an important customer is depending on your program to run correctly on Saturday night. Memory leaks can be subtle and hard to find, which is why modern object-oriented programming languages such as Java and C# provide for automatic memory management.

Automatic Memory Management

In object-oriented programming languages, requesting memory from the operating system is handled behind the scenes by the object creation mechanism. You saw the mechanism in action in the previous lesson. The following code says to create a new Date type object.

new Date()

When that code is executed, the Java program either uses some memory space it has already reserved, or requests more from the operating system and builds a Date object there. The amount of memory naturally depends on how complicated the object is.

In object-oriented programming, each object is responsible for managing the data it contains. The memory used is isolated from other parts of the program and access is controlled by the object.

When your program is no longer using the object, Java detects this fact by a process called garbage collection . Essentially, the garbage collection process looks at all the parts of your program and determines all the objects that are still in use. Every other object is considered garbage. All you have to do is stop using a variable, and the memory is reclaimed automatically.

Your operating system is responsible for managing the stored data on your computer's nonvolatile storage media so that your programs can treat the data in a consistent way. In other words, the operating system hides the hardware details that are vastly different between a floppy disk, hard disk, and CD drive, and presents a consistent interface called a file system.

A file system supports locating data stored on the hardware by a directory system and file name. It also keeps track of handy bookkeeping information such as the file size, the date the file was created, the last time it was modified, and which users have permission to read and/or write data to the file.

In turn, your programming language provides methods that bridge the gap between the operating system and the variables in your program.

Sequential File Reading

There are two fundamentally different ways of reading and writing files -- sequential access and random access. Sequential access means the opened file acts like a stream of bytes that has to be read in sequence.

Suppose you need to read the 10th line of text in a file named myfile.txt in the c:\mydocuments folder. The following is some pseudocode that's representative of all computing languages:

OPEN "c:\mydocuments\myfile.txt" AS #input
DO 
  READ #input INTO astring 
  IF EOF PRINT "hit end of file" BREAK
  INCREMENT linenumber
UNTIL linenumber EQUALS 10
IF NOT EOF use the astring variable
CLOSE #input

This pseudocode illustrates several important features that all programming languages use:

• To read from a file, you have to open it first. The operating system ensures that you're allowed to open that file and causes a fatal error if you're not.
• When a file is opened, the programmer uses a variable, #input in the example, to refer to the file. Using a special character to start the name of a variable that stands for a file is a common convention. The variable that refers to a file is sometimes called a file handle .
• The command that reads the file specifies where to put the data. This example assumes the READ command reads a single line of text into a string variable. For this to work, the file must use a convention for special character or characters signaling the end of the line.
• With sequential access you have to read and discard everything in the file up to the point where the data you want lives. In this example you read 10 lines into the astring variable with each successive line replacing the previous one.
• When reading a file you must always consider the possibility that the file is not as long as you expect. You must provide for detecting the end of the file, commonly denoted as the EOF condition. In the example, hitting EOF before the 10th line causes the program to break out of the loop.
• When you're done with the file, you have to close it. The reason being that holding a file open uses operating system resources and may prevent another program from using it.

In the pseudocode example, it's assumed that the READ command would read an entire text line into the string variable. That's actually a common convenience in programming languages because processing text files is a common problem. Opening a file and reading it in this manner is called working in the text mode . It assumes the data is text with each line terminated by one or more control characters.

Control Characters

By convention, certain characters known as control characters are used to indicate the end of the line. Early computers used a number of special control characters to control printers, ring bells on teletypes, and so on. The characters in common use for end-of-line indication are called CR (carriage return) and LF (line feed). Back in the days of teletypes, CR returned the print head to the start of the line, and LF advanced the paper one line.

Unfortunately, not every operating system uses the same combination. Windows systems use two characters -- CR and LF -- Macintosh uses just CR, and Unix and Linux systems use just LF. This difference creates an annoying requirement for a translation program when transferring files between systems. You may also see the term newline for the character or character sequence that terminates a line.

Reading in Binary Mode

Programming languages also provide a mode of file reading that doesn't try to find lines of text terminated by a newline marker. Instead, the exact contents of the file are delivered to your program without interpretation. In this binary mode, your read command has to designate an array of bytes to receive the data and explicitly say how many bytes to read.

Generally, the syntax computer languages use for writing text data to files is similar to that used for writing to the screen or printer. The language regards the screen or printer as just another place to send a stream of characters. This makes it easy to test your program with writing to the screen and then switch to writing to a file. You still have to open the file and use a special variable to refer to it as shown in the following pseudocode:

OPEN "theoutput.txt" AS #out
PRINT #out "Hello World"
CLOSE #out

In most languages, opening a file with an existing filename can cause the previous contents to be overwritten; however, there is usually a syntax for opening a file in the append mode so that new data is written at the end of the file.

You have to be careful about the conventions each particular programming language has with respect to writing complete text lines. In other words, you need to understand whether or not the language automatically writes a line terminator character sequence with every PRINT statement.

For example, if you're writing a file of addresses and want to put the entire address on one line, you don't have to write a single PRINT command to do the whole thing. But you do have to be careful when choosing the syntax for the particular language you're using. For example, in Java, the following three lines of code write a single line of text to a file named out because only the println command automatically adds a line terminator.

out.print( lastName + ", " + firstName + " - " );
out.print( streetAdr + ", " + city + " - " );
out.println( phoneNumber );

Writing and Reading Whole Objects

Some object-oriented languages, such as Java and C#, provide convenience methods that enable you to write the entire contents of a complex object to a file. Later you can read the object back in and resume working with it. This capability can be extremely handy.

For example, suppose you are writing a program that has a number of personal preference settings for each user. You want to be able to read the settings when the user starts the program and then save them whenever the user changes them.

Assuming you have designed your program so that all user settings are contained in a single object named userPref , the pseudocode for saving that object in a file named userWB.dat is as simple as the following:

OPEN "userWB.dat" FOR OBJECT WRITING AS #obj
WRITE #obj userPref
CLOSE #obj

The real code in Java or C# would be a little longer because you would have to take into account the various errors that could happen, but it would still be quite simple.

When reading and writing objects, you can't change the definition of what goes in the object and still be able to use objects saved with the old definition.

In the sequential reading example, after reading nine lines, the next read operation would start at the beginning of the 10th line. The file read point was positioned at the 10th line. That position could be described with an integer as the nth byte in the file, where the first byte in the file is at position zero, similar to the first element of an array is in position zero.

When reading and writing with random access, you must know exactly where the data starts in the file. One way to do this is to use the same record size every time. In the following pseudocode for a subroutine to read a record, the variable recordBuffer is an array of characters that has a fixed size so you can calculate where to read.

FUNCTION readCheck( chkNum, recordBuffer )
OPEN "checks.dat" FOR RANDOM AS #ranf
SEEK #ranf TO chkNum * checkRecordSize
READ #ranf recordBuffer
CLOSE #ranf
RETURN

In another part of the program, there would have to be routines that pick the characters out of the recordBuffer and then interpret them in terms of your check records.

Using random access becomes increasingly helpful as files get larger. Just think how much harder it'd be to read the last 100 bytes in a 10 megabyte file by sequential reading instead of seeking a file position equal to the size of the file minus 100.

Database technology absolutely depends on random access because the files are so large, but is limited by the fact that after you've decided on a record size for a file, you can't change it. If your record design changes, you have to write a program that can read all of the old records and write a new file with the new record sizes.

Moving On

This lesson covered the important programming concepts of data storage, file manipulation, and memory management. You should now be able to more easily solve real-world programming problems. You also learned some important data storage technology terms.

The next lesson looks at an aspect of modern computing that is related to the data storage and retrieval techniques you saw in this lesson. Instead of reading and writing data on a local computer, you talk to other computers on local networks or the Internet. Before you move on, be sure to complete the assignment and quiz for this lesson. Don't forget to drop by the Message Board to see what your fellow students have to say.