Installing Linux on a SolidRun Clearfog+eMMC board

A few months ago Marvell and SolidRun kindly offered me a Clearfog-A1 (now called Clearfog Pro) board. It is a nice development and/or production board adding an impressive connectivity to quite a powerful Armada388 SoC. This SoC contains two Cortex-A9 CPU cores and high performance I/O like its predecessors but provides 3 GigE ports. It makes me think of the ArmadaXP but consumes much less power and is significantly faster. I've been keeping that board in my bag with all my stuff and using it for kernel testing and various performance tests on ARM.

By wandering on the SolidRun web site recently I noticed that SolidRun had made a new board of half its size and still with all the connectivity (they removed one PCIe slot and the Ethernet switch). The board is amazingly appealing as a small network development board, being fanless, supporting wide voltage ranges, and affordable. So I ordered one. I decided to pick the eMMC version that will save me from losing the micro-SD card all the time.

When I received it I was quite disappointed. It wouldn't boot. No message, nothing. I exchanged the CPU modules between my two boards and found that the CPU module was the culprit. While leaving the board powered on and unattended, I noticed a message "Trying UART" which made me think it wouldn't boot from my micro-SD card and would be trying the UART port instead. After a few exchanges with SolidRun's support, they confirmed that the same lines are used for the eMMC and the micro-SD so the CPU cannot use the micro-SD at all when eMMC is soldered on the board. Not fun at all. I was disappointed that no bootloader was flashed on the board before it was shipped, but for their defence, the board is very new and apparently still being worked on.

But this message "Trying UART" I saw reminded me of the Mirabox. Thus I thought I would try the same procedure I used a few years ago to unbrick it. I first started by trying all possible 32 combinations of the SW1 DIP switches to know which ones allow to boot from what device. Some combinations never returned anything, but the apparently valid ones are reported below. It's a very long and tedious process because most of the time the messages appear after a failed attempt. Values are indicated with switch 1 on the left, switch 5 on the right, OFF = 0, ON = 1.

First value	Last value	Device attempted to boot from
00000	00101	SPI flash, not working
00010	-	SPI flash, working!
00110	-	MMC but unusable ("card doesn't respond to voltage select")
00111	-	MMC, working!
01001	-	UART
01010	-	NOR
01100	01101	NOR
10010	10111	NAND
11000	-	NOR
11010	11011	PEX0

Thus I've set the board to value "01001" to enable booting from the UART. And now here's how to proceed.

What you need

This howto assumes that you have a full-featured, networked Linux-based machine with superuser privileges, a properly working ARM toolchain built with soft float (ARMv5 will work fine), Git, a micro USB cable, the usb-serial driver supporting the FTDI chips, a terminal client like "screen", "minicom", "cu", a TFTP server and a network cable.

Build the U-Boot boot loader

Let's first download the Marvell-enabled U-Boot boot loader :

$ git clone https://github.com/MarvellEmbeddedProcessors/u-boot-marvell
$ cd u-boot-marvell
$ git checkout u-boot-2013.01-15t1-clearfog

Pick a soft-float toolchain. Here we use an ARMv5 toolchain. If your toolchain was built with hard-float only support, the build will fail due to some unexpected VFP registers at the end. Configure and build the boot loader for the clearfog board :

$ make CROSS_COMPILE=/toolchain_prefix armada_38x_clearfog_config
$ make CROSS_COMPILE=/toolchain_prefix -j 8 u-boot.mmc
...
 Ext. headers = 1, Header size = 79360 bytes Hdr-to-Img gap = 0 bytes
New image size = 0xd6350[877392] Source image size = 0xd634c[877388]
====>>>> u-boot.mmc was created

The image now needs to be repackaged for U-Boot for booting over UART and from the on-board SPI flash. It requires changing the first byte of the image header and recomputing the image signature. The "doimage" utility does it automatically for us :

./tools/marvell/doimage -T uart -D 0x0 -E 0x0 -G tools/marvell/bin_hdr/bin_hdr.uart.bin u-boot.bin u-boot.uart
./tools/marvell/doimage -T flash -D 0x0 -E 0x0 -G tools/marvell/bin_hdr/bin_hdr.bin u-boot.bin u-boot.flash

Configure the board to boot from the UART

As indicated in the table above, the SW1 DIP switches have to be set to 01001 or OFF,ON,OFF,OFF,ON. You can use a toothpick for this, it's better than a pen and will not leave ink on the switches. If you don't do this, it will still work but will take ages because the BootROM code will first try to boot from the configured devices. If you run a terminal emulator on the serial port, you should see the following appear after a few seconds when powering the board up :

BootROM - 1.73

Trying Uart

If it doesn't appear, it may indicate that the SW1 DIP switches are not properly set and that the board is trying to boot from another device. Don't worry, it will eventually try the UART after it times out on the other devices. It can take up to 3-4 minutes sometimes, thus why it's best to properly configure it.

Upload U-Boot to the board

Now connect the board's to your development machine via the micro-USB connector. A usb-serial device should appear :

# lsusb
Bus 001 Device 101: ID 0403:6001 Future Technology Devices International, Ltd FT232 USB-Serial (UART) IC

The "usbserial" and "ftdi_sio" modules usually need to be loaded to support this board.
In order to upload the image, it is necessary to send a "magic" header to the serial port (assumed to be /dev/ttyUSB0 here) and wait for an ACK (0x15) from the board, then start to send the data using the Xmodem protocol. Since we don't have any flow control here and we don't know when the header will be detected, we send it in loops over the serial port until we receive the ACK indicating the board is waiting for us to upload the boot loader :

# (while :; do
    printf "\xbb\x11\x22\x33\x44\x55\x66\x77"
    if read -t 0 && read -rN1 && [ -n "$REPLY" ]; then
      set -- $(echo -n "$REPLY" | od -tx1 -An);
      [ "$1" = "15" ] && break
    fi
  done) </dev/ttyUSB0 >/dev/ttyUSB0

Once this command is started, connect the power to the board. After about 5 seconds, the loop above returns to the shell. It means the board is ready to receive the boot loader.
If you make a mistake and have to try again, do not worry, just press the reset button (close to the power connector), and run the script above again.
We can then use the "sx" utility to send the UART image of the boot loader using Xmodem. It will take about 90 seconds to upload about 950 kB at 115200 bps :

# sx u-boot.uart </dev/ttyUSB0 >/dev/ttyUSB0
Sending u-boot.uart, 7464 blocks: Give your local XMODEM receive command now.
Xmodem sectors/kbytes sent: 2556/319k

After the file is transferred, the board immediately boots from this boot loader sitting in memory. Since there's nothing installed on the eMMC, the boot loader cannot boot and will stop at the prompt, waiting for your commands.

Connect to the boot loader

Use your favorite client to connect to the boot loader via the serial port, for example here with "screen". Press Enter, you should see the "Marvell>>" prompt :

$ screen /dev/ttyUSB0 115200
Marvell>>

Now the board is alive and you control it. Here there are multiple possibilities, one of which consists in loading a kernel and an initrd via TFTP, boot the machine and complete the installation. However, the risks of getting it wrong is high, and having to reboot via the serial port again is painfully slow. The board also features an SPI flash, but U-Boot doesn't seem to be able to use it at the moment, every attempts results in a system freeze. So we'll flash the boot loader to the eMMC instead.

Flash the new boot loader to eMMC

Before having to connect many cables, we'll reuse the "sx" utility to send the boot loader over the serial port again, but the MMC version this time, that can be flashed. First, let's make U-Boot wait for a file to be sent over Xmodem :

Marvell>> loadx
## Switch baudrate to 115200  115200 bps and press ENTER ...
## Ready for binary (xmodem) download to 0x02000000 at 115200 bps...

Now you need to quit the terminal client. For "screen", it will be Ctrl-A, "\" then "y". Or "killall screen" from another window will do it fine. Then from the command line we send the MMC image :

# sx u-boot.mmc </dev/ttyUSB0 >/dev/ttyUSB0

At the end of the transfer, reconnect the terminal client, and verify that the image was properly transferred. "AE 01" at the beginning indicates an MMC image :

Marvell>> md.b 0x02000000
02000000: ae 01 00 00 98 69 0d 00 01 01 00 36 00 36 01 00    .....i.....6.6..
02000010: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 a6    ................
02000020: 02 01 50 35 02 00 00 00 5b 00 00 00 00 00 00 00    ..P5....[.......
02000030: ff 5f 2d e9 c1 02 00 fa 00 00 a0 e3 ff 9f bd e8    ._-.............

The mmc command supports 512-byte blocks. Here we've uploaded almost 1 MB, or 2048 blocks :

Marvell>> mmc write 0x02000000 0 2048
MMC write: dev # 0, block # 0, count 8264 ... 8264 blocks write: OK

Verify the copy by reading to another address and dumping it again :

Marvell>> mmc read 0x03000000 0 2048
Marvell>> md.b 0x03000000
03000000: ae 01 00 00 98 69 0d 00 01 01 00 36 00 36 01 00    .....i.....6.6..
...

Configure the board to boot from eMMC

As indicated in the table above, the SW1 DIP switches have to be set to 00111 or OFF,OFF,ON,ON,ON.

Boot the board from eMMC

Start your terminal and press reset to boot the board from eMMC. It will display a lot of useful information and wait for your prompt. The board will be able to be rebooted as often as needed without all this complex procedure now :

# screen </dev/ttyUSB0 >/dev/ttyUSB0
BootROM - 1.73

Booting from MMC
BootROM: Bad header at offset 00000000


General initialization - Version: 1.0.0
Detected Device ID 6828
High speed PHY - Version: 2.0

Init Customer board board SerDes lanes topology details:
 | Lane # | Speed|    Type     |
 ------------------------------|
 |   0    |  3   |  SATA0      |
 |   1    |  0   |  SGMII1     |
 |   2    |  5   |  PCIe1      |
 |   3    |  5   |  USB3 HOST1 |
 |   4    |  5   |  PCIe2      |
 |   5    |  0   |  SGMII2     |
 -------------------------------
PCIe, Idx 1: detected no link
PCIe, Idx 2: detected no link
High speed PHY - Ended Successfully
DDR3 Training Sequence - Ver TIP-1.39.0
DDR3 Training Sequence - Switching XBAR Window to FastPath Window 
DDR3 Training Sequence - Ended Successfully
BootROM: Image checksum verification PASSED

 __   __                      _ _
|  \/  | __ _ _ ____   _____| | |
| |\/| |/ _` | '__\ \ / / _ \ | |
| |  | | (_| | |   \ V /  __/ | |
|_|  |_|\__,_|_|    \_/ \___|_|_|
         _   _     ____              _
        | | | |   | __ )  ___   ___ | |_ 
        | | | |___|  _ \ / _ \ / _ \| __| 
        | |_| |___| |_) | (_) | (_) | |_ 
         \___/    |____/ \___/ \___/ \__| 
 ** LOADER **


U-Boot 2013.01-gc1d6f3e (Sep 28 2015 - 00:17:00) Marvell version: 2015_T1.0p11

Board: A38x-Customer-Board-1
SoC:   MV88F6828 Rev A0
       running 2 CPUs
CPU:   ARM Cortex A9 MPCore (Rev 1) LE
       CPU 0
       CPU    @ 1600 [MHz]
       L2     @ 800 [MHz]
       TClock @ 250 [MHz]
       DDR3    @ 800 [MHz]
       DDR3 32 Bit Width,FastPath Memory Access, DLB Enabled, ECC Disabled
DRAM:  1 GiB
MMC:   mv_sdh: 0
sdhci_transfer_data: Error detected in status(0x408000)!
PCI-e 0: Detected No Link.
PCI-e 1: Detected No Link.
USB2.0 0: Host Mode
USB3.0 0: Host Mode
USB3.0 1: Host Mode

Map:   Code:                    0x3fed1000:0x3ff974d4
       BSS:                     0x3ffef15c
       Stack:                   0x3f9c0f20
       Heap:                    0x3f9c1000:0x3fed1000
       U-Boot Environment:      0x000f0000:0x00100000 (MMC)

Board configuration detected:
Net:   
|  port  | Interface | PHY address  |
|--------|-----------|--------------|
| egiga0 |   RGMII   |     0x00     |
| egiga1 |   SGMII   |   In-Band    |
| egiga2 |   SGMII   |   In-Band    |
egiga0 [PRIME], egiga1, egiga2
Hit any key to stop autoboot:  0 
Marvell>> 

If nothing comes, wait a few minutes and you may see it tried to boot from a different device, indicating you got the switch wrong. Just fix them and reboot.

Boot a Linux kernel

With a working boot loader, it becomes possible to boot a kernel from the network. Everything is not perfect yet but it's getting good. There are a few issues to be aware of that will save you a lot of time. First, regarding the network boot, U-Boot will default to using port 0 (the closest to the USB port). But after any network transfer error, U-Boot will automatically switch to the second port which never works. So it's important to always start a sequence by forcing the active Ethernet port to port 0. Second, there is a bug in this version of U-Boot. It can load a kernel, an initrd and a device tree. But if you pass it an initrd, it will ignore the device tree and will silently fail without any message on the serial port (since the kernel doesn't even know where to speak). The workaround against this is to always append the DTB to the zImage and never try to load the DTB from U-Boot.
Here is the boot sequence I used and which works for me :

Marvell>> setenv ethact egiga0
Marvell>> setenv bootargs 'root=/dev/ram0 rootfstype=squashfs console=ttyS0,115200n8'
Marvell>> kerneladdr=0x2000000 ; ramdiskaddr=0x6000000 ;
Marvell>> tftpboot ${kerneladdr} zImage-38x.dtb
Marvell>> tftpboot ${ramdiskaddr} uInitrd-clearfog
Marvell>> bootz ${kerneladdr} ${ramdiskaddr}

Your TFTP server will have to accept connections on IP address 10.4.50.38 from address 10.4.50.170.

Moving the boot loader

Since I wanted to partition my eMMC, it would overwrite U-Boot. I noticed in the error messages I faced initially that the BootROM code looks for the boot loader at two places on the eMMC memory :

• 0x00000000 : this is where an MBR could be installed.
• 0x00200000 : address 2 MB, there will usually be nothing there

It makes sense to properly partition the eMMC and skip the first 4 MB so that the address at 2 MB can contain the boot loader. That's what I did with a first partition starting at sector 8192 (4 MB), and I flashed the boot loader at the address above. It can also be done during the very first installation with the mmc write command.
Additionally I thought it would be a nice safety measure to install the boot loader on the SPI flash (4 MB). I did it from Linux since U-Boot cannot see it. But by default the DTS doesn't enable the SPI flash, so it needs to be modified for this to be done. I will cover this in a later post.

That's all for now

Please note that for now I'm using my own kernel and initrd which work fine on the my clearfog board. I may document later how to build a working kernel for this board but as long as you use a kernel more recent than 4.5.x, you won't even need to patch it and a working default config is present in mainline. Otherwise you can download some debian or ubuntu images from the SolidRun site, and extract the kernel, device tree and initrd from there.