Tutorial 17: Paging and Virtual Memory

Announcements, Test requests, Job openings, Updates, or just make your web site known to everyone!

Moderator:Moderators

Post Reply
User avatar
Mike
Site Admin
Posts:465
Joined:Sat Oct 20, 2007 7:58 pm
Contact:
Tutorial 17: Paging and Virtual Memory

Post by Mike » Sun May 25, 2008 10:19 pm

Hey everyone,

Tutorial 17 is almost ready for release. It covers:

<ul><li>Physical Memory</li><li>Virtual Memory</li><li>Memory Management Unit (MMU)</li><li>PAE and PSE</li><li>Paging Methods</li><li>Pages and Page Faults</li><li>The Page Table</li><li>The Page Directory Table</li><li>Implementing Paging</li></ul>

It goes into alot of detail on both physical memory and virtual memory, virtual addressing, and paging.

It also covers different methods of paging, page table types, and more.

After this we can start getting proper memory management into the kernel, which is what we will look at in the next tutorial.

We have also updated Tutorial 7 to cover the control registers in more detail.

pathos
Moderator
Posts:97
Joined:Thu Jan 10, 2008 6:43 pm
Location:USA

Post by pathos » Mon May 26, 2008 1:36 pm

w00t

pathos
Moderator
Posts:97
Joined:Thu Jan 10, 2008 6:43 pm
Location:USA

Post by pathos » Thu Jun 05, 2008 5:12 pm

Any word on a release date?

juFo
Posts:14
Joined:Sat Feb 02, 2008 1:41 pm

Post by juFo » Fri Jun 06, 2008 5:25 am

nice! keep up the good work!

User avatar
Mike
Site Admin
Posts:465
Joined:Sat Oct 20, 2007 7:58 pm
Contact:

Post by Mike » Mon Jun 09, 2008 6:50 pm

pathos wrote:Any word on a release date?
Sorry, not yet. The tutorial itself is getting rewritten for more and better content and to make it easier to read and follow. Also, we decided to add paging support in both the bootloader and kernel.

I will have very little time to work on it the next two weeks, Sorry :(

juFo
Posts:14
Joined:Sat Feb 02, 2008 1:41 pm

Post by juFo » Tue Jun 17, 2008 8:20 am

no problem, you are doing just fine ^^

Don't have any time either this week :)

User avatar
Mike
Site Admin
Posts:465
Joined:Sat Oct 20, 2007 7:58 pm
Contact:

Post by Mike » Mon Jun 23, 2008 2:33 am

Hm... Perhaps I started this thread to soon :/

Oh well.. I am back and the tutorial text is practically completed. I am planning it for release this week for comments and suggestions on improving it.

The demo might be a little longer however. The bootloaders paging support is good. Im currently still deciding on the final interface for the kernel however. Because of this, the tutorial might be released a little sooner then the demo again.

I decided to have the bootloader execute the kernel at a 3GB virtual address similar to most higher end operating systems. It will also help demonstrate a way of doing it, and provide examples of implementing paging in assembly language as well as C for better understanding.

If anyone has any thoughts or suggestions or ideas for the upcoming tutorial or demo, feel free to let me know :D

User avatar
Mike
Site Admin
Posts:465
Joined:Sat Oct 20, 2007 7:58 pm
Contact:

Post by Mike » Tue Jun 24, 2008 4:11 am

Tutorial has been uploaded.

Please take note the demo might not be up for another week or so while I decide on the designs for the interfaces.

Please let me know if its to complex, or if there are anything you would change!

As always, any suggestions or ideas are greatly appreciated by me and I am certain our readers as well. :)

Andyhhp
Moderator
Posts:387
Joined:Tue Oct 23, 2007 10:05 am
Location:127.0.0.1
Contact:

Post by Andyhhp » Wed Jun 25, 2008 11:04 pm

Wow thats alot to take in - I shall have to read it properly when I have more time.

Sorry I have been inactive lately but I had exams then started with with Symbian - I got the job btw (Thanks).

In the meantime I offer the following as a minor correction.

In the paragraph:
Physical Memory is an abstract block of memory stored within the computers Random Access Memory (RAM). The way physical memory is "stored" within the RAM depends on the type of RAM the system uses. For example, Dynamic Random Access Memory (DRAM) stores each bit of data in its own capacitor that must be refreshed periodically. A capacitor is an electronic device that stores a current for a limited time. This allows it to either store a current (a binary 1), or no current (a binary 0). This is how DRAM chips store individual bits of data in a computer.
you say that capacitors store current.

Capacitors store charge, not current. Charge is a physical unit that depends on the number of electrons in a defined area. Current is the rate of change of charge, or the speed of movement of charge (in the same way velocity is the rate of change of displacement). Quite literally, the current is the differential of charge with respect to time. This is where some of the physics equations come from such as:
Q = IT
Charge = Current * Time
and
I = d(Q)/dt
Current = differential of Charge with respect to Time

But I think I have over-exaggerated the point so lets move away from physics onto a query:P

In the tutorial, you said that each program can have its own virtual address space. However, when we covered paging and virtual memory in OS lectures, we were told that only a single page table directory exists for all of memory, and this never gets replaces when paging needs to happen.

Is it the case that each program (i.e. process or thread depending on a windows or linux view) has its own virtual address table, which would result in wasted 4mb of memory per process, or is there a different mechanism by which a process can have its base at 0virtual yet not clash with all the other processes out there?

Thank for the great work,

Andrew
Image

User avatar
Mike
Site Admin
Posts:465
Joined:Sat Oct 20, 2007 7:58 pm
Contact:

Post by Mike » Fri Jun 27, 2008 7:31 pm

Wow thats alot to take in - I shall have to read it properly when I have more time.
Is that a good thing or bad..? Do you think it might still be too complex?
Sorry I have been inactive lately but I had exams then started with with Symbian - I got the job btw (Thanks).
No worries--Its great to hear :D
Capacitors store charge, not current.
I am going to have to admit that I always get current and charge mixed up. Thanks for the correction--I am going to fix it on the next revision (When the demo is uploaded)
In the tutorial, you said that each program can have its own virtual address space.
This is true...From that programs perspective, anyways.

The way it is normally done is indeed the way you described it. While the page tables are 4MB in size per process, keep in mind that you dont need to have them all in physical memory. ie, you can cache page tables to disk freeing up more physical memory for more active processes.

User avatar
Mike
Site Admin
Posts:465
Joined:Sat Oct 20, 2007 7:58 pm
Contact:

Post by Mike » Sat Jun 28, 2008 3:55 am

Looks like we made a slight error in the bootloaders loading code as well.

I will update all tutorials and demos with the new code.

The following code fixes a possible triple fault problem with different optimization levels that can modify the entry point location.

The following also allows us to NOT have to use Order.txt at all. ie, we can omit it entirely.

The following code also extracts the image base from the program file, so we dont need to hardcode it.

The original code did not properly extract the entry point address nor image base from the file. (Sorry, I was off by 4 bytes in the original code :oops: ) The new code does just this.

Code: Select all

mov	ebx, [0x100000+60]  ; e_lfanew is a 4 byte offset address of the PE header; it is 60th byte. Get it
	
add ebx, 0x100000         ; Add base address. EBX now points to file sig (PE00)

; Here is where the slight changes are at:

add	ebx, 24
mov	eax, [ebx]          ; _IMAGE_FILE_HEADER is 20 bytes + size of sig (4 bytes)
add	ebx, 20-4           ; address of entry point
mov	ebp, dword [ebx]    ; get entry point offset in code section	
add	ebx, 12	          ; image base is offset 12 bytes from entry point
mov	eax, dword [ebx]    ; add image base
add	ebp, eax

jmp	ebp                 ; Execute Kernel
I am going to update all the tutorials with the error fix.

User avatar
Andrew
Site Admin
Posts:7
Joined:Sat Mar 08, 2008 9:50 pm
Location:USA
Contact:

Post by Andrew » Tue Jul 01, 2008 10:33 pm

bah ima have to start with tutorial 1 =] i need to get back up to date lol how you doin mike? hey that message u emailed me i cannot find it, i saw yahoo post it on my email but i searched and its not there and i didnt delete try sending it again please.
~ Andrew
Lead Design / Lead Tester / Admin
Broken Thorn Ent.

Andyhhp
Moderator
Posts:387
Joined:Tue Oct 23, 2007 10:05 am
Location:127.0.0.1
Contact:

Post by Andyhhp » Thu Jul 24, 2008 9:00 am

Hi,

Having finally got round to continuing my OS, I have a question.

Am I correct in assuming that when you are using Virtual Addressing, you completely ignore Descriptor:Offset addressing, and therefore dont need to set up a GDT if you go straight to Virtual Addressing in the 2nd stage bootloader?

Thanks,

Andrew
Image

User avatar
Mike
Site Admin
Posts:465
Joined:Sat Oct 20, 2007 7:58 pm
Contact:

Post by Mike » Fri Jul 25, 2008 4:02 am

Virtual addresses are translated into physical addresses that follow the descriptor:offset addressing mode used by protected mode.

Because of this, a GDT is still required as pmode fundamentally requires it.

Post Reply