2017-12-09

Tiny USB batteries v3

A bit of background

Since my last experiment at shrinking down USB batteries to fit in my pocket, I made a few observations resulting in new improvements :
  • it's becoming hard to find mini-USB cables these days, while there are micro-usb cables everywhere
  • the previous battery was still a bit thick due to the use of two boards, one for the charge and the other one for DC-DC conversion
  • sometimes I used the battery with a small USB-based LED to walk downstairs at the office, and I'd rather have this LED incorporated.

Migrating to micro-USB

I initially thought about just replacing the charging board by a new one. Most TP4056-equipped boards nowadays come with a micro-USB connector. But while looking at this, I started to think about the other aspects and considered some charging+DC/DC boards as well.

Reducing the thickness

I found a number of new DC/DC boards but as some readers may remember from my previous experiments, most of these are unable to deliver even 1A, let alone the 1.6-2A that I need for my bicycle's light. But recently a few very capable chips appeared : TP5400 and TP5410, equipping a few boards such as these ones.

I bought 5 of these boards and ran a few tests. They are able to deliver 5V at up to 2.8A max from 3.8V in. So there's some hope to run everything with a single board. I bought the version without the USB connector to try to optimize component placement.

 
The board is mostly flat on one side and mostly flat on the other one. Thus I decided to cut the board in two to split the charging part from the discharge part to stack all the components (the charge only needs the micro-USB connector and a few leds).



Default USB connectors are very high. Instead I picked a recessed USB connector, which perfectly fits aligned with the PCB. That makes a very thin board+connector. The coil became the tallest component so I tried a few other smaller ones I had, and ended up using a small 10 µH one.

I reinstalled the output decoupling capacitors against the USB connector and close to the output diode, which I doubled to reduce losses (there was enough room).


The charging LEDs and their 1.2k resistor were installed directly on the charging chip's leads. I don't need direct visibility, the hot glue I'll use to finish the assembly is translucent enough to make the green/red colors pretty visible.


I also noticed during a few experiments with such a board in a previous design (v2n never documented), that the TP5400/5410 stop charging at 1/4 of the programmed charging rate. While it's possible to charge small batteries in 15-30mn, it's problematic to cut off at 1/4 of the rate because they're far from being charged, in part due to the losses in the battery's protection circuit. So I had to replace the programming resistor (1.2k by default) with 4.7k and finally 10k to lower the charge current to around 200 mA (the TP5400 and 5410 have slightly different currents for the same resistor).

I kept the small part of PCB attached to the micro-USB connector as a more convenient support for this connector.


After completing this assembly, I just had to reconnect the battery to the circuit and verify that it could still deliver power and charge. The battery still needs to be a high rate one. I'm using 20C at 240mAh, that's 4.8A max output. When the battery is about to be discharged near 3V, that's still 14.4W, or 2.88A under 5V assuming 100% efficiency (which never happens), so that doesn't leave much margin in the end! Given that TP5410's internal MOSFET supports up to 4A, it's important that the battery is at least that strong. That's 25C for 160mAh, 20C for 200mAh, 16C for 250mAh, 12.5C for 320 mAh.


The resulting assembly with a 240mAh battey is very thin as can be seen below, 12mm thick, 20mm wide and 34mm long. There's very little wasted space now, which is noticeable when filling it with hot glue because very little glue is needed to keep the parts together.


Adding an integrated LED-based torch

I found that some unused space was left next to the USB connector. Just enough to place a powerful LED, a button and a resistor. So I cut two pins from a small button, soldered a 39 ohm resistor to it with the other end connected to the +5V output. The other end of the button was connected to the LED's anode, whose cathode was directly soldered to the USB connector providing the ground. This way pressing the button delivers 60mA into the LED and makes quite a bright torch.


As for v2, I simply placed two crossed layers of heat shrink tube, one in each direction, completely covering the connectors and the button, thus completely closing any hole. Then I just cut the tube around the USB connectors and the assembly is finished. The tube is soft enough to let the LED button be pressed through it.


Final thoughts

I spent a lot of time trying to figure the best way to assemble the parts together. I even hesitated to make a PCB. That may be for v4. What I figured, which will be useful for future designs (and for anyone willing to make a commercial version of this) :
  • the coil is approximately the same height as the USB connector. It must be placed behind it.
  • the IC, diode(s) and capacitors are approximately the same height. It's not really possible to stack them in a production version, but they should probably be placed in the same area.
  • the micro-USB charging plug doesn't need to be exactly on the opposite of the USB connector. I found that it could be placed on one side, possibly over the IC, diodes and capacitors if one is found with long legs or wires. It could even be glued on top of the IC.
  • all resistors and LEDs are very flat and should be placed below because they will not inflate the board's thickness by any measurable size.
  • the recessed USB connector is critical here to keep the device thin
  • the PCB could possibly be rotated 90° and be as large as the battery is long. The USB connector would then be placed on a small area, and all the components including the micro-USB, the press button and the white LED could be installed on the remaining part of the PCB
  • in any case, the battery should be placed on top of the USB connector and components. This also makes the top mostly flat and reduces the length of wires between the battery and the PCB.
An example of more efficient component placement is this one :


Feel free to copy this design and improve it. I'll be happy the day I can simply buy such a small pocket battery without having to spend a week-end building it myself :-) Do not hesitate to share suggestions and comments of course!





EDIT: 2.5 years later the new design above was finally realized: battery v4

2017-11-04

Look back to an end-of-life LTS kernel : 3.10

The end of the 3.10 branch is a good opportunity to have a look back at how that worked, and to remind some important rules regarding how to choose a kernel for your products, or the risks associated with buying products running unmaintained kernels.

Four years and a half


Linux Kernel 3.10 was released on June 30th, 2013, or 4.5 years ago. Greg KH decided that this kernel would become a long term supported one (LTS), which means that it would receive fixes for about 2 years after the regular stable cycle (ie after next the version is issued). It was expected to be dropped by September 2015, and it's now declared dead today on November 4th, 2017 after 108 maintenance releases.

At HAProxy Technologies (or HapTech for friends), we actively rely on LTS kernels for our ALOHA appliances, as the kernel is the most critical component for a networked product. Each major version or our software is maintained for 3 years and ships with a proven stable kernel. This means that the LTS cycle, despite being much longer than others, is still not enough to ensure a smooth maintenance for our products. That's why I've been taking over maintenance of some of these LTS kernels for a while now. Our version 5.5 was maintained till October 2015, explaining why I maintained kernel 2.6.32 for a while, and our version 6.5 issued in October 2014 using one-year old kernel 3.10 was maintained till October 2017, thus in theory I needed to maintain this kernel from September 2015 to October 2017. In practice I pursued 2.6.32 for 9 extra months after our product's end of life as it was also used by Debian whose kernel team helped me a lot during all this cycle, and in return Greg was kind enough to keep maintaining 3.10 till that date, saving me from having to maintain two kernels at once. Thus I inherited 3.10 on March 16th, 2016 after Greg issued 3.10.101, and I maintained it till end of October 2017.

A different experience


My experience of 3.10 was very different from the 2.6.32 one. First, as I mentioned, 2.6.32 was heavily used by Debian and Ubuntu. This usage kept some rhythm in the release cycle because we had frequent exchanges with their kernel teams regarding certain fixes and backports. For 3.10 I improved my process and tools and I thought I would release more often but the reality was a lot different. Few distros relied on it so I had to decide to work on it once in a while in order to catch up with other branches. And when you're busy working on other projects you don't always notice that a lot of time has already elapsed, so in 19 months I have only emitted 7 versions (approximately one every 3 months). Second, while 2.6.32 was mostly found in mainstream distros, 3.10 was mostly found in embedded networked products. And here it's scary to see that the vast majority of such products simply don't apply fixes at all and don't follow updates! If it wasn't at least for our products and with the faith to serve a few serious users, it would be discouraging to see that while 3.10.108 is just emitted, you still find 3.10.17, 3.10.49 and 3.10.73 on a lot of devices in the wild!

Why LTS matters


Most consumers don't realize the risks they're taking by buying products running on outdated kernels. Most people don't care, some find it convenient as it allows them to download some applications to "root" their devices by exploiting some unfixed bugs (which then basically become the only bugs the vendors care to fix when they do). Others are just used to reboot their home router in the basement once in a while because it just hangs every 3 weeks for no apparent reason (but it's a cheap one, surely it's expected). And of course everyone believes the vendors when they claim that they still backport important fixes into their kernels. This is wrong at best and in fact almost always a lie in practice.

First, there's no such notion of "important fixes". Even serious vendors employing several kernel developers got caught missing some apparently unimportant fixes and remaining vulnerable for more than two years after LTS was fixed. So you can imagine the level of quality you may expect from a $60 WiFi router vendor claiming to apply the same practices... The reality is that a bug is a bug, and until it's exploited it's not considered a vulnerability. Most vulnerabilities are first discovered as plain bugs causing a kernel panic, a data corruption, or an application to fail, and are fixed as such. And only a few of such bugs are observed with the eye of someone trying to exploit them and elevated to a vulnerability. Some vulnerabilities are found by researchers actively seeking them, but they represent a tiny part of the bugs we fix every year.

During the 3.10 life cycle, 6597 patches were backported (80% during the first ~3 years Greg maintained it). That's 4.15 per day on average or 29 per week. This simply means that in 4.5 years, we closed 6597 opportunities for malicious people to try to exploit bugs and turn them into profitable vulnerabilities. An interesting observation is that 1310 of them were discovered after the 3rd year, so the common belief of "if it's old, surely it's reliable by now" doesn't work at all there.

How do we know that these 6597 patches we merged were the right ones and that we didn't miss some ? That's simple : we don't know! We only have the biggest confidence anyone can have on the subject because LTS kernels are the de-facto reference in terms of kernel backports. First, all the patches that appear there were tagged by their authors for stable backporting when submitted for inclusion, so surely the code's author knows better than anyone else if his fix needs to be backported and how. Some developers even take the time to provide the backports themselves for various stable kernels. LTS maintainers exchange reviews, patches and suggestions for their respective branches, and have access to some unpublished reproducers needed to validate certain sensitive backports. Second, each release goes through public scrutiny and patch authors get a copy of the backports of their work to verify that it's properly done or is not missing a specific patch. Quite often we get some links to extra commits to backport, or a notice about something that will not work correctly due to a difference between mainline and the old kernel, or simply something we did wrong. Third, all stable kernels are built and booted on all supported architectures. That's 121 builds and 84 boots for every single 3.10 version before the version is released. And this process is extremely reliable, because among the 6597 patches we backported, only 9 were later reverted because they were not suited or caused trouble. That's 99.86% of success on average for each release! Who can claim to beat that in their isolated office by secretly deciding which patch is needed and which one is not, and having to perform their backport without the opportunity of the patch's author reviewing the work ? Simple : nobody. In fact it's even worse, by picking only certain fixes, these people can even damage their kernels more than by not picking such fixes, because such fixes rely on other patches to be backported. This irresponsible practice must absolutely stop and nobody should ever cherry-pick a selection of patches from stable kernels. All of them are needed.

How to use LTS for your products


LTS kernels are very convenient to use, because only what matters is updated. There's no API change, no unexpected behaviour change, no need to revalidate boot command line, userland nor scripts, no surprises. Sometimes it even causes us gray hair to backport some fixes without any user visible impact. And what's even better is that by using these kernels which experience very little changes, you can have a lot of product-specific patches that will most of the time apply well on top of the latest kernel version. It's also possible to simply merge the new kernel into yours if you're maintaining your own kernel in a Git repository. Most of the time, no human interaction will be needed at all. At HapTech, on top of 3.10 we used to have around 300 patches. We faced a patch conflict 3 times in 3 years, which each time was trivial to fix. And it's important to keep in mind that if you experience a conflict, it means that the code you used to patch (hence that you heavily rely on) used to have a bug, so actually such conflicts tend to be a good news for the stability and safety of your product.

We often hear the same comments from some users : "this kernel was issued too recently, let's wait a bit to see if anybody reports a regression". This is fine! I personally prefer users not to trust my work and to review it than them blindly deploying my occasional mistakes if it's too critical for them. As a rule of thumb, if this kernel is supposed to be easy to update (eg: used on your own machines), better deploy ASAP. But if it's going to be sent to customers where an update might involve finding a moment with the customer, or emitting another version making you look bad, better wait a week or two. What matters is that ultimately all fixes are deployed and that bugs don't stay exposed for too long. How long is too long ? It depends. At HapTech, we emit a new maintenance release every few months or immediately after a sensitive security fix gets merged in one of the components we use (ie mostly kernel, haproxy, openssl). When we emit such a release, we always upgrade the 3 of them to the latest maintenance version (in the same branch). This means that the kernels found in field on our products are in the worst case a few months old. This is orders of magnitude more responsible to customers than dropping an unfixed 3-years old kernel in field exposing them to attackers. And by doing so we've never experienced a single regression caused by a kernel upgrade within the same maintenance branch. This process is safe and proven, and should be adopted by anyone distributing kernels with products. The only thing which may vary is the frequency of updates.

If by educating users we manage to reach the point where no kernel found in field is more than 6-months old, we'll have significantly improved the stability and safety of devices connected to the internet.

What you can do to improve the situation


When you buy a product shipped with an outdated kernel, most likely it's because the device needs an update. Once updated, take a look at the version and the build date in a terminal or wherever it appears in the device's interface. For example here on my tablet :

$ uname -a
Linux localhost 3.4.39 #4 SMP PREEMPT Fri Oct 17:48:45 CST 2014 armv7l GNU/Linux

This one is based on 3.4 which is an LTS kernel. That's a good start. To know which kernels benefit from long term maintenance, please visit this kernel.org page. A non-LTS kernel is must be considered as a very bad sign, as sometimes it implies that the vendor didn't even care to port their local patches to newer kernels. Then it's important to check how old the version is :

$ git show v3.4.39
tag v3.4.39
Tagger: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date:   Fri Apr 5 10:09:01 2013 -0700

This kernel was released 4.5 years ago, while the latest version in the 3.4 branch is 3.4.113 which dates a year ago. This version missed 3357 fixes at last 3.4 release one year ago! And as can be spotted in the uname output above, last time it was built (hence had a chance to get a fix backported) was 3 years ago, which proves either a problem with the update process on this device or total a lack of care from the vendor. In short this kernel is totally unreliable and insecure. Such problems must absolutely be reported to the vendor. Some vendors are simply unaware of the problem and will be willing to improve; some have already improved a lot, at least to reduce the number of issues reported on their products. Some will explain that they just ship the kernel provided by their SoC vendor and that they have no added value on top of it, or worse that they don't even understand how it works. These ones at least need to be aware that some SoC vendors are better than others regarding mainlining, and that at least asking for a more recent kernel doesn't cost much and can result in less support calls on their side. And others absolutely don't care and must definitely be avoided in the future since there's no chance the products you buy from them will work well with thousands of unfixed bugs.

How bad can an unfixed kernel be ?


Sometimes people tell me that their kernel is not "that old". OK, let's see numbers for some 3.10 kernels still commonly encountered in field, compared to 3.10.108 (which will itself be outdated once released) :

Kernel versionage# of known unfixed bugs
3.10.108
0
0
3.10.73
17mo
2302
3.10.65
19mo
2641
3.10.49
3yr
3456
3.10.28
3.5yr
4661
3.10.17
4yr
5473

It's unknown how many exploitable vulnerabilities are present in these kernels, however it's certain that all of them are at least locally exploitable, allowing for example a browser plugin to inject malware code into the system and take full control of the device to steal data or participate to internet attacks. And if you don't care about security issues, just think about some of these bugs that I have encountered on various devices running outdated kernels, some of which disappeared after I managed to rebase and rebuild the kernel :

  • random freezes and panics of all sorts, some even causing the device to overheat
  • File-system corruption on the NAND flash bricking the device (until I could reinstall it over the serial port)
  • file-system bugs causing the NAND flash to be "tortured" on every single block and aging very fast to the point of periodically reporting bad blocks
  • eMMC bug causing I/O errors and retries to happen every few kilobytes, making the device respond very slowly
  • SD card failing to enumerate after a few insertion/removal cycles
  • Memory leaks causing progressive slowdowns and regular crashes
  • WiFi random packet truncation causing many TCP connections to freeze and DNS to fail to resolve
  • WiFi disconnections of all sorts
  • WiFi to Ethernet bridge suddenly causing network packet storms by looping on certain multicast packets
  • File-system bugs on a NAS causing the disk to be immediately remounted read-only until the next reboot
  • Ethernet port on a NAS randomly switching between 100 Mbps and 1 Gbps several times a minute
  • Ethernet port not receiving packets anymore after some time
  • webcam driver bug killing the whole USB stack

Sounds familiar ? You can be confident that none of them are considered critical by your product vendor and that the relevant patches have no chance to get backported if they don't follow the official stable kernels (at least because it's hard to spot them as it's often hard to link the cause to the impact).

Should you upgrade to 3.10.108 now ?


The response is simply "no". 3.10.108 was emitted to "flush the pipe" of known pending fixes still affecting 3.10. It's fine the day it's emitted and possibly outdated the day after. Some late upgraders may consider that it could possibly remain OK for a few weeks or months, around the same interval as between two previous subsequent 3.10 kernels. But that's just "probabilistically" true, because if a high-level vulnerability were to be revealed, a new 3.10 would have been emitted immediately after with a fix. Now it won't happen anymore, so you're playing Russian roulette by deploying it. Of course one might think that it's less critical than keeping any other 3.10. But it's far better to upgrade to other stable kernels such as 4.4 which will be maintained till 2022. We at HapTech are using 4.4 and 4.9 in our more recent versions and both of these work very well.

So it's really time to switch now! 3.10 is dead.

2017-08-23

What ESP8266 modules should look like

Context

I've been using ESP8266 modules over the last 2 years for various IoT projects. These modules come with a number of very annoying characteristics which make their adoption problematic to beginners :

  1. they don't boot by default, it is mandatory to solder a number of wires and pull-up resistors just in order to get them to boot.
  2. their pitch is not 2.54 mm, so it's not possible to plug them on a breadboard to simply connect the required resistors and wires
  3. there are two pins (RST and GPIO0) to change between operation and programming, so when you soldered to achieve point 2 above, you have to desolder and solder differently just to flash, keeping a flying wire for the reset which must not stay connected (obviously)
Some vendors understood this and started to propose very nice development boards like the NodeMCU or Wemos D1. The NodeMCU clearly is too big to serve as a production model, but it exposes all I/O, integrates a USB UART, a "flash" and a "reset" button, and connects the UART's RTS and DTR signals to the RST and GPIO0 wires so that the programming software can automatically toggle these lines to program. The Wemos D1 provides all these features except the "flash" button, and is half as small. In fact it's only twice as large as the ESP12 module. This one may be used as-is for some projects.

But both boards present a big problem : the USB UART cannot be disconnected and it draws a lot of power. So if you want to use these boards for production, you can only use them for mains-powered devices, not battery powered ones. Some people explain how to cut wires on these boards to reduce the power consumption but it's a real pain to do. Thus often you're back to using the raw ESP12 device as-is and solder the wires yourself.

But then comes another problem : most USB UART devices adopt the now ubiquitous "FTDI" pinout, which exposes GND, VCC, RXD, TXD, DTR and CTS. The problem is that while DTR is an output and may be connected to GPIO0, CTS is an input and you don't have another output to use to select between running and programming, so it's still required to plug/unplug wires during programming. Some more advanced circuits implement an automatic RST+GPIO combo signal based on DTR only. But for me all of them have proven very unreliable, even after various modification attempts. First, GPIO0 sometimes emits a strong ~20 MHz signal preventing the DTR pin from going low and triggering the RST ; this does not happen while RST is held down however. Second, most often as soon as you open a terminal , DTR is triggered and resets the device again, which is not fun. Fortunately, during this time, RTS is low so it's possible to consider the combination of RTS and DTR instead of each individual signal. So as if it was not enough, DTR and RTS must not be directly connected to GPIO0 and RST, it's required to implement an exclusion between the two so that GPIO0 is triggered only when DTR is low and RTS is high, and RST is triggered only when RTS is low and DTR is high.

So I spend quite some time scratching my head trying to find the appropriate solution. The important points are :
  1. the board must be able to boot in running mode with only the power connected. This means that the pull-ups and pull-downs must be connected
  2. the USB-UART controller must not be physically present on the board, as it sucks power, takes space and costs money. Instead only the 6 pins required to connect a serial adapter must be present
  3. the board must use 2.54mm pitch, be very narrow, not larger than the ESP12 itself so that there are still pins left around it on breadboards
  4. the serial connector must respect the FTDI-compatible pinout so that any adapter will fit at least to allow regular communication with the device, and emergency flashing by moving RST by hand if needed.
  5. the RST pin must not be connected to the serial adapter but be available for RTC-based wakeup if required. Thus the module must use the CH_PD pin instead (also called "EN" or "CHIP_EN")
  6. the serial adapter must be modified to route RTS instead of CTS on pin 2
  7. the serial adapter must provide the logic to combine RTS and DTR as described above. If small enough it can be placed on the final board, otherwise it's better to have it as an intermediary board.
By searching for existing designs I found one part of the solution : Wemos not only does nice ESP boards, they also make an FTDI compatible USB UART on which you can decide to route RTS instead of CTS to pin 2. That confirmed to me that point 6 above can be addressed and become a prerequisite. I ordered a few and decided that in the mean time I'd modify my FTDI adapters.

For point 3, I cut some experimentation board to the same dimensions as the ESP12, plus one row for an optional serial connector. It happens that the 8 pin rows on each side are the same length as the module once expanded to 2.54mm, and that exactly 6 pins fit in the module's width!
I decided to place 2.54mm male connectors in the middle, spaced by 7.62mm so that they can fit on the middle row of a breadboard and even in a DIL16 socket on any board.

This left a central area where a few wires were routed and where there's enough room to install the pull-ups and pull-downs, addressing point 1 (I moved them elsewhere on this prototype as it was a pain to solder them after the connectors were in place).

By placing the serial connector close to the antenna there's no risk of touching the SPI pins at the bottom. Also the antenna is generally supposed to be located in an accessible place so it makes sense to install this connector at the same place. It also turns out that it was convenient to route RXD and TXD.

For point 5, I noticed during tests that a weak pull-up would still be better on RST otherwise it catches RF noise around (ie if you touch it with your finger). But that's a minor detail and doesn't prevent it from working.

So let's proceed with these various steps in order of dependencies

Making a usable ESP board

I didn't want to start making my own PCBs, it was a late afternoon's project. I decided to go with experimentation board, that I cut to the appropriate dimensions to hold the ESP-12E module and a 6-pin connector for  the serial port. Then I've cut unused traces as well as a central area to have two sets of connections. For those interested in trying it, you need to keep 11 rows of 6 holes. It's mandatory that it's single-sided because the ESP module will be placed on top of it and we don't want to risk accidental contacts :


Then I prepared the ESP-12E module. I've soldered very thin wires in each hole. The wires need to be long, at least 4-5 cm, or it will be a pain to place them into the PCB holes later. For this I've cut multi-strand wires and used their thin individual wires :


Now the difficult part starts. Before proceeding, it's important to pull the wires to ensure they're firmly attached. If some wire pops on the other side of the module, it needs to be cut. While pulling the wires, try to arrange them so that each side is approximately parallel with wires approximately 2.54mm apart. These ones will be placed into the PCB. Inserting the first side is not very difficult but requires patience. What is difficult is to insert the second one without removing any wire from the first one :-)

Given that the pitch between the two series of solders is not the same, some wires could cross. Of course we don't want this. So we want to insert the wires into the second row of holes on the PCB, like this :


It should then ressemble approximately this after you pull hard enough on the wires to ensure that none is blocked by another one or touching a neighbour :



Then quickly solder them, cut the remaining wires very short, and grind the solders so that the're as flat as possible :


Then the fun can begin. Cut two 8-pin right-angled male connectors that you solder 7.62mm apart, above the 2nd and 5th columns precisely, and solder them on the outside :



The last part for this board is to install the 6-pin connector for the serial adapter, and to connect the wires and components. I've used only 10K resistors, except in series with GPIO0 where I've added a 470R to protect the serial adapter against the output signal that's sometimes present on this pin during reset. I've also installed a 10 µF decoupling capacitor between VCC and GND because there was enough room for it and it was easy. The wiring diagram looks like this (it's easier than a schematics given that these are almost only wires), followed by the final assembly :

 

 

It's worth mentionning that if you don't see the pull-up resistor for the RESET pin on the photo, it's because I omitted it (as not strictly required), but I'll change this as the reset pin is now too sensitive to my finger, and I tend to reset the device when I touch it.

Modifying a USB-TTL serial adapter to provide both DTR and RTS

I'm using different flavours of FTDI adapters, and all of them have CTS on pin 2 :


I need to have RTS here. So I expected to be able to cut the PCB trace and solder a wire but I can't follow this trace which probably is under the IC so I don't want to damage my board.

Instead I decided to proceed differently by using a 6-pin male-female connector to extend the existing connector, not connecting pin 2 to the adapter but instead cutting the other pins, bending pin #2 over the PCB and connecting it to RTS which is on the chip's pin #3 according to the datasheet. That's all! Now you have a modified FTDI adapter with RTS on pin 2! This modification is very simple to operate. However don't make the same mistake I did, you need to glue the connector once it works, otherwise it will come out of the adapter once you unplug the adapter and pull off your wire. BTW since I couldn't find a 6-pin connector, I had to cut  a 8-pin one.

In the mean time I found that Wemos proposes such an adapter on which pin #2 can be configured to be RTS or CTS, so I will probably not develop further on this adapter's mod.





Converting the DTR/RTS signals to GPIO0/EN

It's not practical to directly connect the signals to the board, it's required to implement exclusion between the signals so that when DTR is low but RTS high, the chip is forced to flash mode, and when DTR is high and RTS low, the chip is reset. Otherwise the chip will work in a single mode, or all software will have to be modified to consider your protocol.

The truth table looks like this :


So it's as simple as doing :
  • GPIO0 = DTR | !RTS
  • EN = RTS | !DTR
In practice it's often made with NPN transistors and resistors on their base, but I do have a few low-voltage dual-mosfets in SOP8 packages which are very convenient because they don't require to cross PCB traces nor to add resistors. Some of them are IRF7313, but IRF7301, IFR7303 and FDS6990 will work fine as well. I also have a few other ones which have the exact opposite pinout, but I don't remember their identifier, so I stopped at the first one I found that I knew. The IRF7313's pinout is this one :


 

 The schematics we want is trivial (GPIO0 resistor included here but it's better placed on the ESP board) :



So the wiring can even be made using flying wires :


But I decided to be reasonable and to use another piece of PCB to make this one, so that connectors are firmly attached. The purpose will be to use the same pinout as the modified FTDI adapter on one side, and as the new ESP PCB on the other side. The result comes below :




I noticed that I wrote the pinout on the least convenient side, so it's better to do it the other way around to match what is displayed on the FTDI
module and the ESP module. And it worked like a charm on first test!


Now what is needed in the end ?

After all these modifications, and after having found the Wemos serial adapter which supports RTS, I concluded that the only part which does not exist is the ESP module for end users. So in fact given that the dual mosfet above is very small and consumes zero power, it should fit on the small board I installed the ESP-12 on. This board would then be programmable and compatible with deep-sleep. Also, even smaller MOSFETs exist, I've seen some dual in SOT363 package.

So by having just a board with 2.54mm pitch, the same pinout as the FTDI board for the serial connection, the MOSFET mounted and the few pull-ups, it would be possible to have the equivalent of the current ESP-12E module, but which could easily be programmed, either for development, or just for production. I think it should be sold with the connectors unsoldered. This way everyone can use it as a simple replacement for the current ESP-12 with a different pitch, yet program it and have it run by simply sending the power. Those who want to turn it into a development model can solder the two 8-pin barrels underneath, and it becomes compatible with breadboards and DIL16 adapters.

Those who simply want the ability to reprogram it in field just have to solder the 6-pin barrel connector and they'll be able to plug their FTDI-like adaptor in situ to reprogram it.

The best thing that could happen would be that some of the vendors like AI-Thinker create a new ESP module with such characteristics. It would solve all those issues at once. Right now when you look up "ESP8266" on Google image, you find tons of connection diagrams explaining how to flash them, which proves it's all but easy, or the NodeMCU models which provide the solution for development only. Let's think about it a bit further and make it usable for everyone! I'm not an electronician, just an occasional hobbyist and I could make it. But for people like me, it takes an amazing amount of time. Having the correct boards from the factory would be so much great!

I've ordered some ESP8285 which include the flash, maybe its possible to rebuild a board from scratch featuring an appropriate pinout. The same could be said for ESP32 which doesn't seem to have improved anything in this area unfortunately :-/