Build farm, version 2 (2016-2017)

[this is a follow-up to this article on version 1 of the build farm]

Background

The first version of my build farm was described here. A significant number of shortcomings surfaced :

• poor hardware build quality resulting in instabilities
• performance limited by the single channel DDR2 memory used
• difficult cooling
• difficulties holding the cards together
• no easy access to a console port for debugging
• inefficient load balancing
• poor recovery abilities when a board crashes

For these reasons I was constantly watching the arrival of new hardware offerings.

New players

Once the first build farm was up and running, I had already identified and tested a few other solutions :

• NanoPI-Fire2A : quad-cortex-A9 at 1.4 GHz, very small form factor, easy to power, to stack and to cool, but much slower than the RK3288's quad-A17.
• Odroid-C2 :  quad-cortex-A53 at 1.5 GHz, significantly slower than the quad-A17, easy enough to cool but not very powerful.
• NanoPI-M3 / NanoPC-T3 : octa-cortex-A53 at 1.4 GHz (twice the C2's cores), but form factor not convenient to stack it nor to cool it
• UP board : this board features a quad-core Atom Z8350 at 1.92 GHz (2 cores) or 1.68 GHz (4 cores running). It doesn't heat too much but is about 50% more expensive than the CS008 for the same performance.

A very new one, called MiQi and made by mqmaker, featuring an RK3288 was announced starting at $35. While the $35 model (1GB) wasn't available by then, I ordered the $65 one (2GB) and was very pleased to discover an excellent build quality, fairly good software support with a very responsive team. I started to hack a little bit on the board, to fix the same issues plaguing almost all RK3288 boards, which are DRAM frequency and CPU frequency limitations, and the board proved to be excellent.

The board's layout made it easy to install a large heat sink reaching one side, there were 3mm holes available for screws and spacers, there was a console port, and dual-channel memory! Another interesting point is the GPIO-controlled fan connector on the edge of the board. Given that I know this CPU can draw a lot of power, it definitely makes sense to have the ability to blow a little bit of air when it's too hot to avoid throttling.

Everything was OK so I asked my employer if it was possible to order 10 of them to make a correct build farm this time, and my request was accepted :-)

Upgrade to MiQi

The boards at the office were successfully replaced with 5 MiQis for a quick test. Some packing foam was used to try to hold them in place. The vendor provided excellent quality micro-USB cables which were quite rigid and made it even more difficult to hold the cards in place. But all of them could be connected to the previous farm's USB power supply :

The board is extremely stable even when strongly abused. Mine were overclocked to 2.0 GHz. At this frequency they draw a lot of power and the power supply is not enough for 5 boards, it can only reliably power 4. Test results were published here, and looking back there, this card held the performance record for almost two years, being 40% faster than the previous farm and very stable.

I ordered some plastic M3 spacers to hold them in place (and pulled one Ethernet LED that had the bad idea of living too close to the hole) :

A test was also run at home with more power supplies and more switch port. We still see the previous farm on the photo and the good old DC-DC converter :-) The test was performed with all 10 boards and 12V fans powered under 5V so that they would be almost inaudible and just blow the hot air away from the boards.

The build time was irregular but everything was overall extremely fast. There was no significant gain between 6 and 10 boards at this point. Definitely distcc was showing some fairness limits.

Adding a build controller

The switch on top of which the 6 boards are placed above is in fact my SolidRun Clearfog Pro board :

It contains a 6-port GigE switch but most importantly it's based on a Marvell Armada 388 SoC which contains a dual-core Cortex-A9 processor. It's not usable as a build farm but it's extremely powerful on I/O intensive workloads and has no problem forwarding more than 1 Gbps of traffic. I started HAProxy on it, to fairly balance the distcc traffic to the boards. Using the leastconn algorithm, it sends new connections where there is the least number. It results in boards being used smoothly with all their memory bandwidth divided by the smallest number of CPU cores at any instant.
Doing just this improved the build time by 15-20%. With a little bit more hacking it was even possible to pre-buffer the build traffic to accelerate delivery to the boards, thus further reducing their idle time.

And now the build is much easier to deal with, I just have to know a single IP:port to aim distcc at, and the controller takes care of spreading the load to the available devices. If I need to pull a board from the farm for whatever reason, there is nothing to modify. And the haproxy stats page shows nice info about the farm's status and the build performance, which is how I saw peaks between 50 and 120 files compiled per second.

A full kernel build took 13mn on my quad-core skylake at 4.4 GHz, and it only took 4mn36 with the 10 boards there, and 4mn45 with only 6 boards.

More details on the setup and tests were shared here.

Thus I needed another build controller for the office. I didn't want to buy another Clearfog because it's quite overkill and while it's an excellent board it's too expensive for this task.

I found a very small and inexpensive gigabit router called EdgeRouter-X from Ubiquity Network, which runs Linux on a dual-core MIPS processor :

I could install HAProxy on it, and after fixing a few issues related to the early firmware image I had, I could reach about 500 Mbps of forwarded L7 traffic! That was more than I needed for 4 boards. One nice thing from this device is that it's supposed to be powered by 12V, but it boots fine starting around 5V, so actually connecting a USB-to-jack connector works pretty fine. With 4 ports available on this one for devices, using all ports from the power supply, and with 6 ports available on the Clearfog, I finally went with two build farms, a 4-nodes one at work for HAProxy builds and a 6-nodes one at home for kernel builds.

The one at work had the nice property that everything fits perfectly packed together :

Power issues

This is a recurrent problem now. When overclocking a little bit (2.0 GHz), the power losses in the USB cables and connectors become significant. The excellent quality USB cables provided with the board significantly improve the situation but there are still quite some losses measured in the micro-USB connector. I ended up soldering thick cables directly to the boards and to some USB-A male connectors. This saved up to around 700mV while overclocking, and significantly reduced the power draw and heating of the on-board DC-DC regulators. And the stability was much better after this operation.

For the home build farm I had to buy a new 5V/6 ports 60W power supply (and patch it again). The Clearfog is powered by a 5-to-12V converter stealing power from several MiQi boards. It was not a great idea though, since stopping two boards was enough to brown out the Clearfog.

Heating issues

In summer when it's above 25 degrees C in the room, the cards become extremely hot. The thresholds have been increased from 80 to 92 degrees C to prevent them from throttling too early, but the 4x4cm heat sink was definitely too small to spread all the heat, especially in the middle boards.
The one in the office had a large 12V fan connected to the 5V GPIO-controlled output, so that the system starts the fan as a cooling device when the board is above 70 degrees C. The fan is almost inaudible and never blows more than one minute.

For the 6 boards at home it was more difficult. So I ended up buying much larger heat sinks that would reach the edge of the board and connect to another much larger heat sink. I had to desolder the fan connector for the large heat sink to reach the edge. One difficult part was to assemble all this together, so after several attempts, I had quite some success with some thermal adhesive from 3M, which is 15mm wide, exactly like the CPU. It's not perfect but still good enough for the job :

The same tape was used to attach the cluster-wide heatsink to all boards at once. It's very stable since the boards are already held together via the M3 spacers, so they already constitute a solid block :

I could get these boards to work together with the 4 other ones at the office, all connected to the EdgeRouter-X using a cheap 8-port Gigabit Ethernet switch. In fact one board was not connected because I was using the 5-port DC-DC converter to power the farm (yes I know it doesn't look pretty but it works fine) :

Final assembly

This big cluster is placed on top of the Clearfog board, above an aluminum plate screwed to the Clearfog's M3 spacers to protect the Clearfog against scratches or short circuits. It aligns well with the RJ45 connectors :

All this was fixed to a plexiglass plate with rubber feet, and using some of the adhesive tape to firmly hold the power supply in place. This results in a nice, compact, fanless block delivering 3 times the build performance of my quad-core skylake 4.4 GHz!

Updates

The power issues caused to the Clearfog when stopping a board made me want to replace the power supply to have more ports. Also, this power supply is very hot during long builds, and I wanted to have a bit of headroom to add more boards. I ended up buying a cheap fanless 5V/30A framed power supply and USB-A female connectors, which I arranged to provide 12 outputs for whatever I need to plug there, including temporary boards for experimentation. This power supply is adjustable so I increased the output voltage to 5.3V to reduce heating resulting from power losses and to lower the strain on the DC/DC regulators. Overall everything works even better now.

Links

Unsurprisingly, this project was followed by a number of people, regularly asking for updates. I've put the various kernel patches here. The various performance measures of the board at different CPU and RAM frequencies are reported here. I've posted the first series of updates to the build farm on mqmaker's form here. A new presentation was made at Kernel Recipes 2017 upon various attendees' request, the slides are available here and the video is here. The MiQi code, schematics and docs are available here on GitHub.

Version 1 of the build farm is described here. Version 3 of the build farm is described here.

Build farm, version 1 (2015-2016)

Context

Being a developer and having built many thousands of Linux kernels over the last two decades, I can say that build time is something which really counts. I've always been interested in distributed build systems, I started to explore these possibilities in 1996 at the university... Nowadays hardware is much cheaper, there's a lot of choice, and lots of software possibilities as well, starting with distcc.

First serious attempt

When reading this article about a very powerful tiny quad-core device called T034 in late 2014, I figured that it was about time to give it a try. The device features 4 Cortex A17 cores which by then were the fastest ARM cores available. I could test the device in early 2015 and published the results here. By then this device was sold with an Android image, and it took me a long time to manage to get Linux to work reliably on it, and even more to recover correct performance (the RAM was running at 200 MHz stock, the CPU was limited to 1.6 GHz etc). Once reliably achieved, I ordered 4 such devices, which by then were replaced in stores by an exact copy with a new name, CS008, for the same price, about $65/piece :

The devices were assembled together, connected to an L-shaped micro-USB connector. The cables were cut to reduce losses, a DC power meter was installed on each device, glued to the unused HDMI connection, and a 4x4x1cm heat sink was installed on top of the original aluminum thermal plate:

The power was fed from a 19V power supply after conversion via a 5V/3A DC/DC converter:

First disappointment

The devices booted well, installed well, but when starting a kernel build, the power suddenly cut off. I thought that the devices consumed too much, but in reality it was the DC/DC power converters which were not precise enough. The SYR927/928 voltage regulators on the boards support an absolute maximum of 5.5V. Two boards had died in the first few seconds, indicating an unexpected over-voltage caused by the huge power variations on the boards, that the regulators do not react to fast enough, leading to much more than 5.0V when the CPU's power usage drops. I had to order two more boards to replace the defective ones, which were not fixable.

Better solution for the power supply

More serious solutions were appearing in early 2016, like these 5-port 50W USB power supply, that I ordered :

By the time this device arrived, I found quite a robust DC/DC regulator that I could salvage from a dead power supply. It took 12V in, 5.1V out under 30A, exactly what I needed. So I identified the pin-out, connected it on a piece of experimentation board with some decoupling capacitors and USB female connectors :

It's not pretty but it works fine. Since then I've added a 5th connector and the device never failed, powering up to 5 boards at full speed, spreading very little heat, indicating it's extremely efficient.
Overclocking attempts on the boards showed that at 2 GHz they could drain up to 3-4 Amps for short peaks, especially due to losses in the micro-USB connector (up to 450 mV lost there, up to 700 mV between the connector and the on-board regulation chip).

The ordered power supply arrived. It would occasionally cut off during builds. I had to patch it to disable over-current detection. It's well made, there's one such detector per port, made of a 50 mOhm resistor. But this one causes too many losses under high current, so I shorted it. Since then this power supply has always been working flawlessly, and it's much better packaged than the hack above:

Final status

The boards were stacked on top of a 5-port Gigabit Ethernet switch, itself powered by the same power supply; Everything was connected using 20cm Ethernet cables. By the way, I had to order 10 of them to find 5 working ones, the build quality was terrible. The power meters were removed as barely unreadable, and new L-shaped connectors were used :

This solution worked well enough for tests and demos, but the boards would overheat and occasionally hang. It was necessary to use a heartbeat LED trigger to know which ones were working reliably and which ones were hung. The boards were of very low build quality, some had to be fixed due to poorly soldered components, so the freezing issues were not very surprising. The DDR chips were marked with the reference of a DDR3 one while they didn't even match the form factor, they definitely were DDR2 chips. Oh and there was no easy way to keep all the boards together, I tried to make a plastic piece, then hard foam, then I soldered a thick copper plate to all HDMI connectors, nothing was very good nor a durable solution.

However, despite the stability issues, the performance was quite good thanks to the impressive power of the RK3288 SoC. A full-modules kernel build went down from 45mn to 13mn on my laptop (core i5-3320M at 3.1 GHz) when using the build farm. When factoring in the total cost of the solution, around $280, it's impressively efficient.

A solution was thus needed to improve the quality and stability, but the CPU was convincing.

More information

A presentation of this build farm was given at Kernel Recipes 2016 and was featured on LWN. The first one includes the video showing a live build of the complete kernel.

Version 2 of the build farm is described here. Version 3 of the build farm is described here.

Connecting a USB serial console to WRT1900ACS

Goal

A while ago I purchased a Linksys WRT1900ACS-v2 router to replace my aging 15-years firewall (still running kernel 2.4 on an old VIA Eden board!). This device is pretty nice, it's fully supported by Linux mainline, it offers multi-gigabit connectivity with a real CPU (Marvell Armada 385), has lots of RAM (512 MB) and storage (256 MB NAND), and dual-band WiFi. Yes I know, having NAND flash nowadays instead of eMMC is still annoying to deal with, but I won't need to write often, so I certainly can ignore wear leveling since I'll only be loading the rootfs into RAM. This will be plenty of room to run Formilux with nftables.

I don't have much time to devote to its conversion and I figured that if it's still not in production, it's because I don't want to switch to it before I'm 100% confident about the config and it doesn't have an accessible serial console to allow me to recover from my mistakes. Thus I decided to work on exposing a serial console before finishing its configuration.

Serial Connector

Linksys engineers were really cool with this device, you don't need a soldering iron as they already placed the connector inside. You have remove the screws under the rubber pads and strongly pull apart the blue from the black parts to access it.

There are still a few caveats though :

• the connector is a 2.0mm pitch connector (JST-PH-2.0 to be precise)
• the on-board marking is reversed compared to the connector's specification, which creates a lot of confusion and can easily fry an adapter since the power pins are at both ends
• there is something strange with the Tx output, the device refuses to boot if it detects too strong a pull-up (4.7kohm is too strong). I suspect the device tries to pull the line low during boot to detect such a condition in order to stop in debugging mode or something like this.

Thus it is not as trivial as it seems to simply place the connector outside.

After measuring the voltage on the pins and doing a few attempts, I finally came up with this pin-out :

That's fine, I don't need the +3.3V since I intend to connect this to a USB serial adapter which is alimented by the USB bus. So I can use a 4-pin connector which will cover the connector's pins 1 to 4 (motherboard pins 3 to 6). Such 4-pin connectors are easily found with many motherboards which provide a sound connector for the CD/DVD drive. It's the same! Yes it only has 3 wires but that's what we need : GND, Rx, Tx.

Serial adapter

I picked a CH340G-based micro-USB to TTL adapter like this one, which supports working on 3V or 5V (I've set it to 3V) :

Connecting this adapter to the connector revealed the issue with the too strong pull-UP. So I decided to try to attack it various ways. First, I cannot replace the pull-up on the adapter because it's inside the chip itself. In the end, the easiest and most reliable solutions I found were to use either a 2N4403 PNP transistor or a low voltage AO3407 P-channel MOSFET with a much weaker pull-up. These ones would amplify the WRT's Tx pin negative signal to act  stronger on the adapter's Tx input pin. I tried the two following diagrams and finally decided on the MOSFET one since it features an even weaker pull-up :

Since I had to add components to the board, I wouldn't be able to use the existing 6-pin connector, so I desoldered it to make the PCB thinner. Be careful, on mine the Tx PCB pad went off. Fortunately it's didn't serve as a via and is not used, it just made my task a bit more complicated to figure what pin the signal was going to :

Then I noted the soldered the MOSFET and the resistor on the PCB close to the IC :

And connected the extremity of the wires there. Note if you reproduce this design, I left far too long wires, you could shorten them to 15 cm I think.

Drilling a hole

Every device needs to have at least one hole for a console connector :-)

It wasn't easy to find an accessible place for this connector. I tried to place it in front, but this required disassembling the front panel which I failed to. There are screws below the label underneath but despite these the front panel doesn't come off. It looks like there are molten plastic joints at various places holding it firmly in place. In the end I found an empty place above the front right rubber pad. I drilled a large enough hole to let a micro-USB male connector enter, and placed the adapter there on top of dual-sided thick tape against the internal plastic wall. It looks as if it was made for this, there's very little margin there.

I don't want dust nor random metal stuff inadvertently entering the hole, so I wanted to fill it with hot glue. The difficulty with hot glue and USB connectors is that it tends to fill them up so that it's not possible to insert anything anymore there. So I had the idea to insert the USB connector before filling the hole with glue. But I didn't want my USB connector to be glued. Thus I reused the dual-sided tape protection. By definition it's non-adhesive, so I adapted it to the hole's form and placed it around the micro-USB connector. This way I could plug the hole while filling with the glue. The result is perfect and the adapter cannot move anymore.

Reassembly

Reassembling the device is almost as easy as opening it except when you have an excess of cable as I had. You have to find tiny places to push a few centimeters of cable in order for everything to fit. But once reassembled, it looks very clean, and the connector is very easy to access from the front.

And of course it works fine, I can now attach minicom to it at any moment to watch the boot process (yes I know there's no password now, I'll set one at the end of the installation) :

Now I can finish to install it and play with nftables without fear of being locking myself out ;-)