Linux on Motorola MPC820 Bring-up. A short war story.
This document describes steps we took to bring up Linux OS on a PowerPC (MPC8260) based target board.
The board is part of the TdSoft Voice Access Gateway (TdGate) enabling connection of a V5 Class telephone switch the remote Network Access (NA). Or IAD's via ATM network (For more information see www.tdsoft.com site).
Besides a dedicated ATM (OC3), V5 (E1) and back-plane HDLC communication lines the board includes external RS232 and Ethernet (10/100 BaseT) ports used for debug purposes.
The work has been done by Ori Idan along with Codefidence Ltd. and Engineers from TdSoft.
Step 1. Getting the Toolchain
Before we could even start putting Linux on this system, there was a need to build a toolchain (cross-compiler, assembler, linker and other standard GNU tools) that could generate Linux image for the target PowerPC board on a standard x86 Host PC. In addition we also had to decide about the operating system for our Host. We chose to run the Host on Linux too as it would simplify the development process and because it has a great variety of suitable free software development tools, much more than any other operating system.
As a basis for the toolchain we decided to use the uClibc (http://www.uclibc.org). We downloaded the latest version from the project's CVS (If we hadn’t used Linux in a Host, we would have to look for a CVS client for Windows; another good reason for Linux in the Host). At first the CVS client did not login; lesson 1 to remember - have port #431 open for output on your firewall.
After opening the port all the toolchain sources have been downloaded (We used the buildroot module that contained the toolchain, root file system and optionally the Kernel). Configuration was easy, we just needed to edit the Makefile at the source chain tree root. The Makefile has many options; all of them are documented very well. We choosed to use Kernel 2.4.xx. We had an option for GCC version 3.3 or GCC version 2.95, since the Kernel documentation says that compiling under versions other then 2.95 may give unpredictable results, we chose GCC 2.95. There were also a number of options for root file system type; we stayed with the default (ext2fs).
Step 2. Compiling the Toolchain.
This step was supposed to be the easiest part... The compilation starts by issuing the make command in the toolchain directory. This downloads the sources for all system parts, that is by itself a long process as it connects by itself via the Internet to get the sources of the various programs required for compilation. After downloading the sources the compilation begins.
Everything was OK until kernel compilation began. For some reason the Kernel was not configured for PowerPC but it used a compiler and assembler for PowerPC. This resulted, of course, in errors as the PowerPC assembler doesn’t understand the i386 assembly code. Looking into the Linux Makefile revealed the problem; we had to set it manually for PowerPC.
The kernel was found under build_powerpc/linux-2.4.26-pre5-erik. Lines starting with ARCH in kernel Makefile were updated to ARCH := ppc. The kernel can be compiled alone as well my issuing the make command from kernel directory.
At this stage we started playing with Linux configuration (make xconfig) to feet what we need. However this proved to be a waste of time. It was very difficult to get the required configuration as many modules depend on each other and the kernel need to be compiled again and again... We ended up by going back to the default configuration.
Step 3. Building Root File System.
Linux must have a root file system. The uClibc toolchain builds a root file system by itself; there was only a need to compress it into a .gz file used for initrd. At least we thought so, that was proven to be a half-truth.
Now there are two options:
- Use this file as initrd
- Compile it into the Kernel using make zImage.initrd
There are other options that at this stage we did not know, more on this later.
Step 4. Downloading Kernel and Root File System to the Target.
(Thats where the real fun starts)
The target board had a resident bootloader used for booting a VxWorks application. One of its options was to download the application via the external Ethernet connecting using FTP. It was supposed to be easy, just download the Kernel and Initrd, prepare a special record with parameters required by Linux Kernel startup procedure and transfer control to the Kernel... Of course, it didn’t work; otherwise this wouldn't be an interesting story...
What actually happened was that immediately after passing control to Kernel the system got an exception and restarted. The reason was that the Kernel startup tried to find the load address from the ELF header of Kernel image. Since Linux is using the MMU, the addresses in the ELF header are supposed to be a virtual addresses and not physical memory addresses. The load address in the header was: 0xC0000000 (yes, it is not a mistake, it is 7 zeros...) obviously we did not have physical memory at this address; we only had 128Mb of physical memory mapped from 0x000000000. We tried to set the address in the ELF file to 0x00000000 (by updating the resident VxWorks bootloader before passing the control to Linux). For some reason it did not work.
Then we decided not to spend time on fixing the problem, but to use a “standard” U-boot loader that was supposed to work with Linux Kernel on another MPC8260 card. The whole download sequence should be work as follows: resident VxWorks bootloader downloads the U-Boot, U-Boot downloads the Kernel and Initrd and passes control to Kernel. Eventually, it worked; however it wasn’t that simple... (In the final configuration U-boot can reside in the onboard flash and not downloaded via FTP).
Step 5. Configuring U-boot.
U-Boot is a GPL licensed SW boot loader that was originally designed for Linux loading (and many other RTOS’s) for many different HW architectures. It even used some code from Linux (The beauty of GPL)
From reading the U-Boot Readme file we understood that it is very simple to configure U-Boot to any supported board. If the board is not supported it should not be too difficult by doing some fixes in the original code. In our case we had to do some changes in the RS232 and Ethernet U-Boot drivers. (The default 8260 settings differed from our board)
First we tried to compile the U-Boot using our original toolchain (See steps 1 and 2). As expected things didn’t go so smooth...
We still don't know what was exactly the problem was but when we removed the line compiling the test hello.c program the compilation succeeded.
Few days later we saw U-Boot booting. The next step was generation of U-Boot compatible Kernel image and loading it.
Step 6. Generation U-Boot Compatible Kernel Image
U-Boot is using it's own uImage Kernel image that is actually the original vmlinux ELF file with a special U-Boot header. The uImage is generated by using a special tool named mkimage compiled along with U-Boot.
However, before this there was a need to set MKIMAGE variable (see linux/scripts/mkuboot.sh file) according to the to the actual mkimage path.
It was also needed to set the configuration variables (CONFIG_SERVERIP, CONFIG_IPADDR, CONFIG_ETHADDR, CONFIG_BOOTDELAY, CONFIG_BOOTCOMMAND, CONFIG_BOOTARGS etc.) according to the real values used in the board and Host PC running the TFTP server.
We also prepared a U-boot compatible Initrd image (using instructions in the U-boot README file), again this file is the standard initrd image with a special U-boot header.
After that we had both Kernel and Initrd images suitable for U-Boot booting the next step was simply to download them and start the Kernel.
Step 7. Booting the Kernel
Well, do not prepare the party yet; the work is far from being done...
It seemed that U-boot was really downloading the file and uncompressing it, but after control was transferred to Kernel we were get an exception once again... Looking again at the U-boot code revealed that the IMMR register was set to a lower address; according to the Linux documentation it should be set to 0xF0000000. That wasn't so simple to change. We ended up with changing it in the first ROM resident bootloader, but anyway it did boot U-Boot again and we started again booting the kernel, but again another exception...
Ok, we now had to dive into the kernel sources. We didn’t have an ICE that can work with Linux. So how do you debug a program without a debugger? We started looking at the Kernel sources. The first step was to activate printouts, but the Kernel didn’t reach the step where it initializes the serial output so we could not use the printk () kernel function.
We decided to add code that will light some debug LEDs we had on the board, this way was not recommended by the U-Boot mailing list members but we couldn’t find a better way. We added a code that will light a LED and will stop after that (using a jump to itself). We started by putting this code in the very beginning (arch/ppc/kernel/head.S) and then moved farther inside the kernel.
We ended up finding that the exception happens when the kernel was trying to jump to a function using an uninitialized function pointer. How can this be? Diving deeper into the code showed us that this function pointer was supposed to be initialized by the bootloader, but our bootloader did not have any place to initialize this pointer. Looking again at the Linux configuration revealed that for some reason CONFIG_8260 was not set so actually we where using a Kernel image compiled for a different machine and not the 8260.
However this was not the end of our troubles. We changed the configuration and compiled the Kernel once again. It wasn't that easy since that operation broke some other parts of the configuration. After we booted the Kernel and now a different story we didn’t get an exception but we didn't see anything on the terminal.
If it’s boring to read it, just imagine how frustrating it is to do the same thing over and over again with almost no advance?
Step 8. Booting in the Dark
Before we dived again into the kernel, we wanted to make sure the kernel was actually working so we connected our ICE (remember that it didn’t support symbolic debugging with Linux. How do you use an ICE that does not support symbolic debugging to find out the exact symbol? We stopped the ICE after few second expecting it to be somewhere near the 'panic' symbol, as we did not have initrd loaded so it could not find a root file system.
We found the address it was stopped on; we added 0xC0000000, which is the virtual start address on the Kernel and looked at System.map. The address was few bytes after the panic symbol, this made us sure that the Kernel was working but for some reason couldn’t write to the serial line.
Step 9. Teaching the Kernel to Speak
First we had to find where exactly the code for serial port initialize resides, the default was to look at arch/ppc/8xx_io/uart.c but this was the code for the 860 and not 8260. It was very similar but this code was actually not linked into our image so there was no point to change it.
The actual code was residing in arch/ppc/cpm2_io/uart.c. It was very similar to the other uart.c file so similar that one may think you got the wrong file on the editor… Looking at the comment in the head of the file shows us that this code was written by Dan Malek of MontaVista software using the 8xx_io/uart.c as a base so this explained why it was so similar.
Apparently the 8260 has several communication controllers and the Kernel used the default one but not the controller we where connected to. This was a simple change in few #define statements in the file.
But this did not solve the problem... there was a point of light as it did show few random characters on the terminal. It looked like it was writing something but probably with wrong baud rate.
Ok, so the natural way to go is the rs_8xx_init() function that for some reason was close to the bottom of the file. It seemed that although the baud rate was set up correctly in the board information structure (a structure that was transferred by the boot loader to the Kernel) this function wasn't reading the right value or maybe it did not read the actual structure since for some reason it got a wrong pointer.
It wasn’t so easy to find where this pointer is read. To save time we used a brute force method here and hard coded the value into this function. And..."Let there be light".
Step 10. The Kernel is Talking
The Kernel did write some lines indicating that it is booting but it didn’t get to the stage where it really starts the init process. It seemed to be stacked in a loop after starting calibration of the Real Time Clock. It seems that it could not calculate the value of ticks_per_jiffy that is the number of clocks per kernel time unit (1/HZ which is in this case 0.01 sec. Again, this was a problem with the board info structure that was not read correctly. As before, we used a brute force method to set the clock value in m8260_calibrate_decr() function in the arch/ppc/kernel/m8260_setup.c file that did help.
After that it proceeded to the step where the Kernel was trying to load the initrd ramdisk.
It seemed that the Kernel has found the ramdisk at the right address, however it complained on a bad magic number at certain offset. A simple change in the kernel arguments, adding ramdisk_size=60000 solved this problem.
Now it seemed that the problems went from Kernel space to User space. Why?
The error now was "bummer can’t open /dev/tty1" that repeated for tty2 and tty3.
We assumed that opening /dev/ttyx is done from processing inittab, which is done by the init process.
Step 11. We are in User Space
Well, while being in user space it should be easier to solve problems (or at least that's what we thought). The file system image was under the 'root' subdirectory in the toolchain build_powerpc subdirectory. This seemed like a standard Linux filesystem tree; so we went to the etc/inittab file. According to this file it was trying to open /dev/tty1 /dev/tty2 etc.
We did not have those devices or actually we had them but the Kernel code for opening them was disabled during our tests, so we could either enable the Kernel code or delete those lines.
Since anyway we didn't have more then one serial port, we decided to delete them (comment them out). Then we built the filesystem again and rebooted the system.
The miracle happened and we have seen a login prompt. The ceiling was high enough so we did not hurt our heads jumping from joy :-)
Step 12. But We Need Ethernet…
Apparently we have a working Linux system so what next? The first step was networking activation, as the system has to communicate with the real world after all.
Running ifconfig showed us one interface (lo), which means that the system has networking, enabled but can only talk to itself... We went again to the Kernel configuration (it seems that the Kernel space problems are not over yet) and tried again to look for any hints. This was relatively easy as for some reason we did not enable 8260 networking.
Again, a new Kernel was recompiled and booted but ifconfig again showed one interface (lo). Trying to do ifconfig eth0 returned an error; the interface was unrecognized. To make the long story short (it took us few hours to find it) we had to enable networking on FCC1 in the Kernel configuration.
Booting the kernel again after adding the lines:
null::sysinit:/sbin/ifconfig eth0 172.16.1.106
null::sysinit:/sbin/route add -net 172.16.1.106 eth0
gave us a working network. We could now ping from our target to the Linux host and vice versa.
Step 13. Linux is Running. What Next?
Ok, we now have a Linux system working, meaning it displayed a prompt on the terminal and we could use the core utilities... However this system has to do a little more than displaying a prompt; we need to run some programs on it. We tried to write a simple “Hello world” program, put it in the filesystem image and reboot the system, it works.
We made a list of things we need: Telnet, debugger and it looks great to have an ftp client too.
FTP client or sort of, we actually had; busybox knows how to work as wget, it's only a matter of configuration. As for Telnet, busybox can also do it but at that time we did not know it, so we downloaded telnetd from netkit (ftp://ftp.uk.linux.org/pub/linux/Networking/netkit). We compiled it using our toolchain and downloaded it to the target. It worked, however it can’t wait for a connection by itself, it needed to be spawned from inetd or xinetd. So we downloaded inetd from netkit, compiled and configured it to run telnetd, downloaded it to the target and it works.
Encouraged by how easy it was to download software compile and run it, we continued to GDB, but this was not as easy as the other parts...
Step 14. Compiling GDB
The uClibc buildroot makefile contained an option to install GDB, so was supposed to be very easy to install it. However nothing is as easy as it looks.
We uncommented the gdb line and ran make again, it downloaded GDB and configured it, however it failed during the configuration as it could not find ncurses. At the beginning we thought it was related to the host ncurses and thus it was strange that it failed. The problem was we did not install ncurses for our target.
After installing the ncurses it turned out that it compiles gdb using the cross compile while we expected only gdbserver to be compiled using the cross compiler.
We passed over each directory in gdb and compiled it manually with the host gcc and only the gdbserver which have no dependencies was compiled with the cross compiler.
Looks a short story, but it took about 2 days to find it all out...
Step 15. Summary.
It was not a short and easy journey, but it was fun and gave us a good working system. What remains now is only to do the real work and write the application.
We hope this description will help other people to make this porting easier.
Add comment
