Redudant power supply for home servers and devices

A well known story

All those who run a small infrastructure at home know this problem. It's Monday morning, while drinking your first coffee you're checking your e-mails, you find that you received surprisingly few during the night, and that the last one was from 5 hours ago. Then this well-known feeling starts to build up: "what is it this time? FS full, unregistered from all mailing lists at once, missed a domain renewal notification, or a machine died?". At this point you finish drinking your coffee quickly because you know that it will cool down faster than you'll get the problem fixed, and you start to ping various machines and devices while chewing some biscuits, to discover (in any order of preference):

• a reverse proxy no longer responding
• many machines no longer responding (likely indicating a switch issue)
• the router not responding

Then you go to the office/lab/basement/garage/wherever the machines are located, and start debugging in underwear, thinking that it's really not a great moment because you planned to arrive early at work to prepare some stuff before a meeting...

Finally the culprit is almost always the same: a dead power brick whose LED (if any) blinks slowly indicating a dead input capacitor, or not at all anymore:

Then starts the moment of removing the dust from the sticker to find the voltage and amperage, and open all trays to find an almost equivalent one which should hopefully get the job done even if it's half the amperage because you know your devices are not pulling that much... And when you want to connect it, you notice that its connector is a 2.1mm inner while the previous one was a 2.5mm. But by forcing a lot you manage to establish a contact and consider that it will be sufficient for the time it takes to order another power block, then you can go to the shower.

There is a variant to that story: you're working in your office, notice the light flickering, and realize you had a micro power outage on the mains. Most of your small servers didn't notice, but one wasn't that lucky and experienced a brownout. You decide that enough is enough, it's really time to connect all of them to the UPS, but the UPS has too few outputs, all on C13 plugs and you have to modify a pair of power strips to install a C14 connector on them in order to connect all your small devices:

And once that's done, you discover the day your UPS fails to take over a short power cut and want to remove it, that you have no other C13 power strip to which you can connect the C14-equipped strips, something like this that I once had to build exactly because of this:

It's got only marginally better with USB power delivery because when a power brick dies, it's often easier to find another one (but rarely a high power one), or you can sometimes temporarily daisy-chain the device to another device (provided that it was not itself already daisy-chained).

All of this sounds familiar ?

Root cause 

The reason to all these problems is the multiplicity of low-power devices which all come with their own power block, each requiring a distinct mains outlet. And sometimes angled ones cannot even be placed close to each other and are masking some outlets. You quickly end up with this for 12V supplies:

And USB is not much better, all having to live together on the same power strips:

Oh by the way, for USB nowadays there is something way more appealing, the ubiquitous multi-output QC charger: 

Except you'll only try it once for servers, until you realize that it's a charger and not a power supply, and the difference is that each time you connect or disconnect something from a port, it renegotiates the voltages with all other ports, which are all cut for a second or two! That's absolutely not a problem to charge a laptop. But it is when you imagine powering multiple always-on devices from it.

The solution is in fact to set up a power distribution system which requires only one input. However if this one fails, it will be even worse, so it needs to be redundant. And if it's redundant, it can also be connected behind the UPS to get protection, as well as directly to mains (or another UPS) to survive a UPS failure.

Design of a solution

In my case I counted the number of devices I would need to connect there. It's roughly 16 in the rack, counting servers and switches. The total power is very low, around 70W, which means that I can use fanless power supplies.

Most devices take 12V on input. Some other use micro-USB and others USB-C.

I considered having multiple 12V power rails so that a short circuit would only affect a part of the components. I had found the perfect chip for that: TPS259571DSGR. It's a really nice electronic fuse, it supports programmable 0.5 to 4A and automatically re-triggers after a short. But this one comes in a tiny 2mm-wide WSON package with pins spaced by 0.25mm, and after trying for a few hours to solder one on a PCB I purposely made, I decided to postpone because my PCB quality is not good enough, at this level of thinness you definitely need a solder mask or it quickly shorts. I since ordered some PCB adapters for DIL to WSON and will try again later. I would love to spot an equivalent in SOP8 package! In the mean time I finally decided that all 12V connectors will be connected together and that this should be OK. I chose 5.5x2.1mm female jacks for which I will make male-male cables that will connect either to 2.1 or 2.5mm depending on what is needed:

For the USB outputs, we now find a number of QC-compatible multi-port USB adapter boards like this one. They are in fact 4 independent power supplies connected to the same input. They're convenient because you can also use them to extract a fixed voltage (e.g. 12V) using a tiny adapter. I decided to use one to provide 4 USB-A ports and another one for 4 USB-C ports. On another project (controlled USB outputs) I had successfully stacked two USB-C ones and that's what I initially intended, but drilling holes is a real pain and I didn't need that many ports at that moment:

For the power supplies, I thought that blocks designed for LED would be a good fit. These are not very well regulated because they focus on power and not on a perfect voltage. But nowadays their regulation is pretty good, the voltage is accurate to +/- 5% usually, which is much better than what 12V devices accept on input. Contrary to a PC power supply which must deliver a very stable voltage, here the 12V output is never used as-is but passes through other DC-DC regulators, and usually anything between 10 and 14 will be OK. The advantage is that such power supplies are simple, small and very efficient, like the one below that can run fanless up to around 500W. I found various models here and here which looked appealing:

I decided to place the power switch after the PSU, not before, in order to isolate a faulty one from the system. The idea here is probably a consequence of the trauma of replacing faulty power supplies: I want to be able to replace a dead PSU without turning everything off. The switch on the output allows to isolate a PSU from the circuit and replace it. 

Switching power supplies can be connected in parallel. But if for any reason one dies with its output short, it will bring the second one with it. Also, there's no way to know that one is dead when they're in parallel. So I decided to think about a circuit that would connect them together (just two diodes) and also report which ones are working or not (LED before the diode).

My concern was to find diodes that could stand high current but I had not much difficulty finding 20A diodes and stopped on 20SQ045. And since I didn't want to have many LEDs on the front, I had fun scratching my head a little bit to combine colors on common cathode RGB LEDs in order to report the various possible states among:

• all down (off)
• this PSU is unconnected or dead (red)
• this PSU is connected but not enabled, no output (blue)
• this PSU is connected but not enabled, output from other PSU (purple)
• this PSU is connected and enabled but diode is dead (cyan)
• this PSU is not connected and diode is short-circuited (orange) 
• this PSU is connected and delivering power (green)

All this only with passive components. The final diagram is here:

and the trivial PCB here:

Both can be downloaded in eagle format from my GitHub repository.

I just decided to place everything on the copper side so that I could leave it flat on the bottom of the enclosure.

Construction

Once I received all the components, I started assembling everything. As usual for me the most difficult is dealing with hardware (drilling, cutting etc). I think I did reasonably well overall on this one, without even scratching the front panel:

OK, the holes for the jacks could have been better centered...

It's made of two aluminum corners constituting the front and back panels, screwed to an MDF plate. The MDF is interesting for being an insulator, and also because it's easier to cut than a metal plate:

The PCB was made with my laser engraver with all components soldered on the copper side. The power diodes had their legs rolled as this slightly helps spread the heat if needed. The copper pads were large to stand high currents and permit to be generous with the solder for the large wires:

All the cabling was done using 2.5mm² wire made for house circuitry. It supports 16A under 250V, which means it will not heat enough to melt the insulator over many meters in your walls. Here on short distances like this even at 20A it will become barely warm. And I don't intend to reach 20A anyway. The advantage of using such wires is that they're rigid and make excellent contact on solder joints. Two were stripped and used as bus bars on the jack connectors. Overall I find that the result is not bad at all:

 

Tests

Tests are reasonably simple, I just operated with all 4 combinations of on/off state for the two switches, multiplied by the 4 combinations of on/off state for the mains inputs. I could confirm that the colors are as intended (not well reflected on the photo):

Installation

I initially planned on installing this horizontally in my rack, but found that it was even better vertically on its external side. It allows me to see the LEDs, it helps with cables distribution, improves ventilation and eases operation and checks if ever needed, though for now both power blocks remain cold to the touch:

I have not checked if the overall power consumption has reduced or not. It would be very possible since every power block has a minimum current leak, at least to power the oscillating circuitry. But that should be marginal. What could make a higher difference is the expected higher power conversion ratio of such high-power blocks which can reach 92-94% compared to very low-power ones which rarely aim beyond 70-80%. Anyway I'm not going to reconnect everything just to check!

Amusingly, initially I connected the two inputs on the same UPS, and forgot about it the day I decided to turn the UPS off for a repair... That's when I decided that only one input would be connected to the UPS and the other one directly to the mains. It's also convenient to use color tape on your power strips to indicates which ones are UPS-protected and which ones are not. I'm using red for UPS and blue for mains.

Now let's see how long my devices stay up.

Adding a cheap and simple RTC to Rockchip devices

Background

There are plenty of nice devices these days designed around ARMv8.2 SoCs such as RK3568 or variations around RK3588. Many of them have been using the HYM8563 I2C RTC chip for about a decade. This device is reasonably cheap, requires few components, consumes very little power and is long proven to work well. Despite its low price, entry-level devices are often lacking it and only have the pads on the board, which is understandable when every dollar counts.  Some such devices include Radxa's E20C and E52C mini routers/versatile servers, which are absolutely awesome devices, which come with either dual-1Gbps or dual-2.5Gbps, feature a USB console so as to always provide a local access, and have a metal enclosure. But they're lacking the RTC, which is quite annoying for a firewall or mini-server, as a power outage always has a painful effect on the dependency chain at home (typically the OS boots faster than the ISP's box and has to start with a bad date). At least once that issue was reported, the products' creator, Tom Cubie (aka @hipboi) acknowledged the problem and suggested that new devices should have it.

Can I do it myself ?

How to proceed with existing devices in the short term ? Isn't it possible to just order the chip and solder it on board ?

One problem with HYM8563 is that it's almost always only found in TSSOP8 format, which means it's a few millimeters wide, with a pitch of 0.65mm, which means that pins are roughly 0.35mm wide with a spacing of 0.3mm between. Actually that's not that difficult to deal with, provided that you have a fine enough soldering iron. The real problem is the components that come around are also of the same scale and very close to each other, as can be seen on the photo of the E52C below (click on the photo to zoom):

The resistors and capacitors are in 0201 format, which is 0.6mm tall by 0.3mm wide, and are very difficult to solder without causing a short circuit. To get a sense of the scale above, the chip is 3mm*3mm. The crystal oscillator is a 32.768 kHz in a flat format that's not easy to find for a hobbyist.

However, the I2C pads (SDA and SCL) are "large enough" and moderately accessible, and there's power on the other side, let's keep that in mind.

Finding a pre-made board

There are various I2C RTC boards available on the net. Most of them are DS1307, and a very nice small one is based on DS3231 with a battery like this one. However, there's no HYM8563 one, and I'd prefer to stay on the same that is referenced in the DTS so that it works out of the box. But I could find the chip alone.

Making a new board instead

Then I started to think whether I could make a board myself based on the chip. Looking at the HYM8563 datasheet above reveals that it's actually not that hard to assemble the few components around. I drew a schematic on by hand (takes one minute vs one hour in Eagle):

I found a few rare occurrences of the chip in SOP8 package, which is easy to deal with, so I could make a PCB with that chip and the 6 components around it. I ordered a pack of 10pcs of that chip, and unfortunately received the tiny TSSOP8 ones instead :-(

But that reminded me that I had some small TSSOP8/SOP8/DIL PCB adapters, which support SOP8 on one side and TSSOP8 on the other side, with the DIL pads in the holes. These are convenient for on-air cabling since pads of both sides are connected together and to the holes:

After all, there were so few components that I could probably solder them directly on that PCB on either side. So let's try.

Bill of materials

Based on the schematic, I'll need this:

• 1 such PCB
  
• 1 HYM8563TS chip
• 1 diode in 0402 format
  
• 2 4.7k resistors in 0402 format
• 1 82pF capacitor
• 1 crystal oscillator
• 1 "large enough" capacitor (I counted about 6mn per 100µF), supporting at least 3.3V
  

 

The secret to avoid shorts when soldering TSSOP chips is to use solder paste. I have some in a syringe that I keep cool in the fridge (otherwise it dries in a few months and is unusable the day you need it):

Assembly steps

As a first step, we'll need to cut a trace on the PCB so as to isolate the VCC input pin from its connection to the IC's pin 8, as we'll want to place the diode there instead. We'll need to keep the resistors connected to the external VCC so that the capacitors doesn't discharge its energy into them when off. Thus we'll cut the trace after the via that connects to the other side. What's nice with these boards is that the pads of both sides correspond to the ones of the other side at the same location, so they're easy to match. The final pinout of the board will look like this:

So let's cut the trace:

Now we're going to place the chip before the diode, because it's already painful enough to solder such small chips, we don't want to be hindered by the diode. The approach for this is to place a very little drop of solder paste over all the pads on each side of the chip. Think in terms of volume, considering that the solder paste is mostly flux that will disappear (the flux will avoid shorts by making it difficult for the solder to make bridges between pins). Count that the resulting volume will be roughly 1/3 of the disposed one. Making a small trail roughly as wide as a pad is a good estimate. Make very sure not to leave any beneath the chip, as it will never melt and will stay them forever, risking to make shorts later. Only cover the pads and areas you can clean later. Once done, just place the chip over the paste:

Now solder everything with the soldering iron, without worrying about the risk of shorts, just focus on aligning the chip as best as possible with the pads, and melt absolutely all the paste. Then check with a magnifier or better, a microscope that everything's OK. With a multi-meter you need to check there's no short between adjacent pins:

Now's time to scratch the right side of the track and to place the diode, with the positive on the right and the negative on the left:

Let's now flip the board to solder the resistors. They will attach to the two bottom left pins, and connect to the beneath pad corresponding to the Vdd pin, which is in fact connected to the positive pin at the bottom right. It has the pleasant advantage of being placed immediately next to the two pins, and close enough to have the resistors directly touching on both ends:

Now let's connect the capacitor between the chip's Vss pin and the top rightmost hole that's connected to the chip's pin 1:

On the other side, the oscillator can be soldered. The pins on the left (when reading the reference) or under the notch are the useful ones. The two other ones on the right are not connected so I could connect them to the pin1 pad, though that will depend on the model since some have integrated capacitors and might not work well when doing so, but I could verify with my multi-meter that there was no capacitor there. In order to solder these pins beneath the component, solder paste is your friend again, being cautious again not to put too much:

We're only left with the capacitor that serves as an energy storage during outages. Ideally you'd use 5.5V super capacitors of 0.1F for several days, or the new 3.8V lithium-based supercaps that have even higher capacities (typically 10F and above). But given that my goal here really to just cover occasional power cuts from the mains, when a power supply dies or when I need to move cables inside my rack, I don't need more than a few minutes. And I did happen to have a 4V/150µF capacitor that perfectly matched my needs (should support 7 to 10mn of outage, and was super flat).

Soldering it is just like for the oscillator above except that it's not easy to use solder paste. The positive terminal (the one with the colored bar) needs to be connected to pin 8 of the chip (the top leftmost here) and the negative to the topmost right  hole. Soldering that one is not too hard, just melt some solder inside the hole. Alternately, using thin wires is OK as well.

Now's time to connect wires and test:

I connected these to another board that I'm using for testing I2C, and ran i2cdetect:

Good, the device appears at address 0x51, so at least it's detected. Now we need to verify that its oscillator is properly ticking. For this we'll have to write the current time to the chip (which appears as rtc1 on this board). It will emit an "invalid argument" because it first tries to read the time which contains a bit "VL" ("Voltage Low") set in the second field indicating the the device lost power since last time it was set. But we can ignore it when writing the date, reading it next will indicate whether it works or not (the time must change):

Perfect, it works! Now's time to replace my testing wires with thin ones, to put all of that into shrink tube to protect it and solder it in the device.

Installation

Let's spot the 4 pads we'll need inside the E52C. First, at the bottom of the board, we'll find the I2C SDA and SCL pins at the bottom on these photos, where we'll solder our SDA/SCL wires (violet and green here). They'll have to pass between the board and the enclosure so they must be very thin, but where I'm passing them, there's enough room:

Next step is to install the module on the other side of the board. We're going to glue it on top of the micro SD card reader with some double-sided tape. The capacitor close to it has both positive (3.3V) and negative and is large enough to support soldering directly to it:

Conclusion

It was fun to make but took me most of the day to build; soldering small components requires delicate manipulations, and dealing with flux and solder paste requires lots of cleaning along the operations. I've made two modules so I still have one extra left. This will allow me to migrate another of my machines to a new one, but I'm impatient to see them produced with the chip already soldered so that I don't have to do this anymore!