32 versus 64bit in Linux
Posted June 14th, 2008 at 10:14 by screwballl
I have been doing some digging lately since the issues comes up as more and more people are starting to make new builds with 4GB (and in some cases more) memory. One thing that is always mentioned relating to windows: To see the full 4GB or more, you need a 64-bit OS (usually with a mention of Vista x64).
Any mention of Linux is usually followed by the same line of reasoning, saying 32-bit is limited to 4GB no matter what OS...
Time to state that by the usual limitations, that is correct. Since the virtual address space is just 32 bits wide, each process can't grow beyond 4 GB. But the key is the wording that makes all the difference: "each process". With a CPU that can handle PAE or 64-bit registers, this means that a 32bit or i386 Linux can handle up to 64GB of RAM. PAE (also know as highmem) allows processors to access physical memory up to 64 GB (36 bits of address bus).
Since Linux can't access memory which hasn't been directly mapped into its address space, to use memory over 1 GB, the physical pages have to be mapped in the kernel virtual address space first. This means that the pages in ZONE_HIGHMEM have to be mapped in ZONE_NORMAL before they can be accessed. When going over 4GB with PAE, it continues this remap so the ZONE_HIGHMEM pages are handled it from 4GB to 64GB (in 3GB chunks).
There is also different ways to handle the linux installation on a professional level with modern day 64-bit processors. This information is pulled from osidr.com:
32bit kernel, 32bit user space: Each process gets 3GB of virtual address space. The kernel can divvy that up however it wants. You'll need to enable highmem support for that. If CPU performance isn't your bottleneck, it won't matter that you don't get to use the extra registers. You lose maybe 15% CPU performance, depending on what you're doing. (Compare SPEC scores breakdowns for AMD's submission, if you're curious.) This will be the most stable configuration, because you can just install i386, and forget all about 64bit. (Yes, Opterons support PAE and all that's needed for a 32bit kernel to use lots of RAM.) The kernel has to use bounce buffers to move data around, because it can't map all the memory. The kernel uses the remaining 1GB of virtual address space for itself, and maps all the RAM it can. What's left is highmem (1GB to 64GB in 3GB "virtual" chunks).
64bit kernel, 32bit userspace: Each process gets 4GB of virtual address space. Disadvantage: you need a module-init-tools, iptables, and so on that can talk to the kernel. All the normal system calls by 32bit programs go through a translation layer (not much overhead, don't worry) so you can boot a 64bit kernel with root=/dev/path-to-i386. You can install some 64bit libraries, or make some statically linked binaries, so you can run a few things 64bit. statically linked AMD64 iptables might be the best way to go. (you can debootstrap a 64bit chroot so you can apt-get install the stuff you need... Use dchroot to make your chroot convenient). The ia32->amd64 kernel translation layer works well, so while it might not be as stable as a fully i386 system, and you have to worry about special kernel interfaces that don't get 32bit translated, you don't have to worry about bugs in userspace programs like storing a pointer in an int variable. On an SMP (multi-core/multi-CPU) system, the kernel will know about NUMA and be able to allocate memory that's attached to the CPU the requesting process is running on.
64bit kernel, 64bit userspace: Each process gets 64bit virtual address space. Same as above, but you have to worry about user-space too. Not all packages are available in 64bit, and some of them have bugs because they truncate pointers to 32bits in some places. You can install libraries so that you can run i386 binaries if you have any binary-only programs. (32bit code can't link to 64bit code at all, ever, on amd64, not counting special translation layers like the kernel-userspace boundary). 3D acceleration is only possible with 64/64 kernel/user, or 32/32, if that matters to you.
It is very possible to run more than 4GB of RAM with a 32bit linux installation with some tweaks. Hope this may help someone down the road.
Any mention of Linux is usually followed by the same line of reasoning, saying 32-bit is limited to 4GB no matter what OS...
Time to state that by the usual limitations, that is correct. Since the virtual address space is just 32 bits wide, each process can't grow beyond 4 GB. But the key is the wording that makes all the difference: "each process". With a CPU that can handle PAE or 64-bit registers, this means that a 32bit or i386 Linux can handle up to 64GB of RAM. PAE (also know as highmem) allows processors to access physical memory up to 64 GB (36 bits of address bus).
Since Linux can't access memory which hasn't been directly mapped into its address space, to use memory over 1 GB, the physical pages have to be mapped in the kernel virtual address space first. This means that the pages in ZONE_HIGHMEM have to be mapped in ZONE_NORMAL before they can be accessed. When going over 4GB with PAE, it continues this remap so the ZONE_HIGHMEM pages are handled it from 4GB to 64GB (in 3GB chunks).
There is also different ways to handle the linux installation on a professional level with modern day 64-bit processors. This information is pulled from osidr.com:
32bit kernel, 32bit user space: Each process gets 3GB of virtual address space. The kernel can divvy that up however it wants. You'll need to enable highmem support for that. If CPU performance isn't your bottleneck, it won't matter that you don't get to use the extra registers. You lose maybe 15% CPU performance, depending on what you're doing. (Compare SPEC scores breakdowns for AMD's submission, if you're curious.) This will be the most stable configuration, because you can just install i386, and forget all about 64bit. (Yes, Opterons support PAE and all that's needed for a 32bit kernel to use lots of RAM.) The kernel has to use bounce buffers to move data around, because it can't map all the memory. The kernel uses the remaining 1GB of virtual address space for itself, and maps all the RAM it can. What's left is highmem (1GB to 64GB in 3GB "virtual" chunks).
64bit kernel, 32bit userspace: Each process gets 4GB of virtual address space. Disadvantage: you need a module-init-tools, iptables, and so on that can talk to the kernel. All the normal system calls by 32bit programs go through a translation layer (not much overhead, don't worry) so you can boot a 64bit kernel with root=/dev/path-to-i386. You can install some 64bit libraries, or make some statically linked binaries, so you can run a few things 64bit. statically linked AMD64 iptables might be the best way to go. (you can debootstrap a 64bit chroot so you can apt-get install the stuff you need... Use dchroot to make your chroot convenient). The ia32->amd64 kernel translation layer works well, so while it might not be as stable as a fully i386 system, and you have to worry about special kernel interfaces that don't get 32bit translated, you don't have to worry about bugs in userspace programs like storing a pointer in an int variable. On an SMP (multi-core/multi-CPU) system, the kernel will know about NUMA and be able to allocate memory that's attached to the CPU the requesting process is running on.
64bit kernel, 64bit userspace: Each process gets 64bit virtual address space. Same as above, but you have to worry about user-space too. Not all packages are available in 64bit, and some of them have bugs because they truncate pointers to 32bits in some places. You can install libraries so that you can run i386 binaries if you have any binary-only programs. (32bit code can't link to 64bit code at all, ever, on amd64, not counting special translation layers like the kernel-userspace boundary). 3D acceleration is only possible with 64/64 kernel/user, or 32/32, if that matters to you.
It is very possible to run more than 4GB of RAM with a 32bit linux installation with some tweaks. Hope this may help someone down the road.
Total Comments 2
Comments
 |  It's possible to run more than 4GB of RAM with Windows XP too, again with some tweaks. How simple is it, though? That's the real question. |
  |
| |  Windows XP 32bit can use the PAE extension but it has to be native in the processor such as Opterons or ia64-based Xeons and the Conroes, otherwise there is no "virtual address space" over the 3.25GB or whatever is seen by windows. Since the 32bit XP/Vista installs do not natively see or use the PAE, it is much more difficult to get anything like this working in Windows. Versus in linux, if it sees memory 4GB or higher, some can build the PAE extensions into the kernel and you never have to mess with it. It all depends on your setup. |
 |
Recent Blog Entries by screwballl
- 32 versus 64bit in Linux (June 14th, 2008)
- Ubuntu and ndiswrapper (June 4th, 2008)
- appreciating newer technology (May 31st, 2008)
- FisherPrice kids camera (March 5th, 2008)
- XP SP3 fails to impress (February 21st, 2008)
