Look back to an end-of-life LTS kernel : 3.10

The end of the 3.10 branch is a good opportunity to have a look back at how that worked, and to remind some important rules regarding how to choose a kernel for your products, or the risks associated with buying products running unmaintained kernels.

Four years and a half

Linux Kernel 3.10 was released on June 30th, 2013, or 4.5 years ago. Greg KH decided that this kernel would become a long term supported one (LTS), which means that it would receive fixes for about 2 years after the regular stable cycle (ie after next the version is issued). It was expected to be dropped by September 2015, and it's now declared dead today on November 4th, 2017 after 108 maintenance releases.

At HAProxy Technologies (or HapTech for friends), we actively rely on LTS kernels for our ALOHA appliances, as the kernel is the most critical component for a networked product. Each major version or our software is maintained for 3 years and ships with a proven stable kernel. This means that the LTS cycle, despite being much longer than others, is still not enough to ensure a smooth maintenance for our products. That's why I've been taking over maintenance of some of these LTS kernels for a while now. Our version 5.5 was maintained till October 2015, explaining why I maintained kernel 2.6.32 for a while, and our version 6.5 issued in October 2014 using one-year old kernel 3.10 was maintained till October 2017, thus in theory I needed to maintain this kernel from September 2015 to October 2017. In practice I pursued 2.6.32 for 9 extra months after our product's end of life as it was also used by Debian whose kernel team helped me a lot during all this cycle, and in return Greg was kind enough to keep maintaining 3.10 till that date, saving me from having to maintain two kernels at once. Thus I inherited 3.10 on March 16th, 2016 after Greg issued 3.10.101, and I maintained it till end of October 2017.

A different experience

My experience of 3.10 was very different from the 2.6.32 one. First, as I mentioned, 2.6.32 was heavily used by Debian and Ubuntu. This usage kept some rhythm in the release cycle because we had frequent exchanges with their kernel teams regarding certain fixes and backports. For 3.10 I improved my process and tools and I thought I would release more often but the reality was a lot different. Few distros relied on it so I had to decide to work on it once in a while in order to catch up with other branches. And when you're busy working on other projects you don't always notice that a lot of time has already elapsed, so in 19 months I have only emitted 7 versions (approximately one every 3 months). Second, while 2.6.32 was mostly found in mainstream distros, 3.10 was mostly found in embedded networked products. And here it's scary to see that the vast majority of such products simply don't apply fixes at all and don't follow updates! If it wasn't at least for our products and with the faith to serve a few serious users, it would be discouraging to see that while 3.10.108 is just emitted, you still find 3.10.17, 3.10.49 and 3.10.73 on a lot of devices in the wild!

Why LTS matters

Most consumers don't realize the risks they're taking by buying products running on outdated kernels. Most people don't care, some find it convenient as it allows them to download some applications to "root" their devices by exploiting some unfixed bugs (which then basically become the only bugs the vendors care to fix when they do). Others are just used to reboot their home router in the basement once in a while because it just hangs every 3 weeks for no apparent reason (but it's a cheap one, surely it's expected). And of course everyone believes the vendors when they claim that they still backport important fixes into their kernels. This is wrong at best and in fact almost always a lie in practice.

First, there's no such notion of "important fixes". Even serious vendors employing several kernel developers got caught missing some apparently unimportant fixes and remaining vulnerable for more than two years after LTS was fixed. So you can imagine the level of quality you may expect from a $60 WiFi router vendor claiming to apply the same practices... The reality is that a bug is a bug, and until it's exploited it's not considered a vulnerability. Most vulnerabilities are first discovered as plain bugs causing a kernel panic, a data corruption, or an application to fail, and are fixed as such. And only a few of such bugs are observed with the eye of someone trying to exploit them and elevated to a vulnerability. Some vulnerabilities are found by researchers actively seeking them, but they represent a tiny part of the bugs we fix every year.

During the 3.10 life cycle, 6597 patches were backported (80% during the first ~3 years Greg maintained it). That's 4.15 per day on average or 29 per week. This simply means that in 4.5 years, we closed 6597 opportunities for malicious people to try to exploit bugs and turn them into profitable vulnerabilities. An interesting observation is that 1310 of them were discovered after the 3rd year, so the common belief of "if it's old, surely it's reliable by now" doesn't work at all there.

How do we know that these 6597 patches we merged were the right ones and that we didn't miss some ? That's simple : we don't know! We only have the biggest confidence anyone can have on the subject because LTS kernels are the de-facto reference in terms of kernel backports. First, all the patches that appear there were tagged by their authors for stable backporting when submitted for inclusion, so surely the code's author knows better than anyone else if his fix needs to be backported and how. Some developers even take the time to provide the backports themselves for various stable kernels. LTS maintainers exchange reviews, patches and suggestions for their respective branches, and have access to some unpublished reproducers needed to validate certain sensitive backports. Second, each release goes through public scrutiny and patch authors get a copy of the backports of their work to verify that it's properly done or is not missing a specific patch. Quite often we get some links to extra commits to backport, or a notice about something that will not work correctly due to a difference between mainline and the old kernel, or simply something we did wrong. Third, all stable kernels are built and booted on all supported architectures. That's 121 builds and 84 boots for every single 3.10 version before the version is released. And this process is extremely reliable, because among the 6597 patches we backported, only 9 were later reverted because they were not suited or caused trouble. That's 99.86% of success on average for each release! Who can claim to beat that in their isolated office by secretly deciding which patch is needed and which one is not, and having to perform their backport without the opportunity of the patch's author reviewing the work ? Simple : nobody. In fact it's even worse, by picking only certain fixes, these people can even damage their kernels more than by not picking such fixes, because such fixes rely on other patches to be backported. This irresponsible practice must absolutely stop and nobody should ever cherry-pick a selection of patches from stable kernels. All of them are needed.

How to use LTS for your products

LTS kernels are very convenient to use, because only what matters is updated. There's no API change, no unexpected behaviour change, no need to revalidate boot command line, userland nor scripts, no surprises. Sometimes it even causes us gray hair to backport some fixes without any user visible impact. And what's even better is that by using these kernels which experience very little changes, you can have a lot of product-specific patches that will most of the time apply well on top of the latest kernel version. It's also possible to simply merge the new kernel into yours if you're maintaining your own kernel in a Git repository. Most of the time, no human interaction will be needed at all. At HapTech, on top of 3.10 we used to have around 300 patches. We faced a patch conflict 3 times in 3 years, which each time was trivial to fix. And it's important to keep in mind that if you experience a conflict, it means that the code you used to patch (hence that you heavily rely on) used to have a bug, so actually such conflicts tend to be a good news for the stability and safety of your product.

We often hear the same comments from some users : "this kernel was issued too recently, let's wait a bit to see if anybody reports a regression". This is fine! I personally prefer users not to trust my work and to review it than them blindly deploying my occasional mistakes if it's too critical for them. As a rule of thumb, if this kernel is supposed to be easy to update (eg: used on your own machines), better deploy ASAP. But if it's going to be sent to customers where an update might involve finding a moment with the customer, or emitting another version making you look bad, better wait a week or two. What matters is that ultimately all fixes are deployed and that bugs don't stay exposed for too long. How long is too long ? It depends. At HapTech, we emit a new maintenance release every few months or immediately after a sensitive security fix gets merged in one of the components we use (ie mostly kernel, haproxy, openssl). When we emit such a release, we always upgrade the 3 of them to the latest maintenance version (in the same branch). This means that the kernels found in field on our products are in the worst case a few months old. This is orders of magnitude more responsible to customers than dropping an unfixed 3-years old kernel in field exposing them to attackers. And by doing so we've never experienced a single regression caused by a kernel upgrade within the same maintenance branch. This process is safe and proven, and should be adopted by anyone distributing kernels with products. The only thing which may vary is the frequency of updates.

If by educating users we manage to reach the point where no kernel found in field is more than 6-months old, we'll have significantly improved the stability and safety of devices connected to the internet.

What you can do to improve the situation

When you buy a product shipped with an outdated kernel, most likely it's because the device needs an update. Once updated, take a look at the version and the build date in a terminal or wherever it appears in the device's interface. For example here on my tablet :

$ uname -a
Linux localhost 3.4.39 #4 SMP PREEMPT Fri Oct 17:48:45 CST 2014 armv7l GNU/Linux

This one is based on 3.4 which is an LTS kernel. That's a good start. To know which kernels benefit from long term maintenance, please visit this kernel.org page. A non-LTS kernel is must be considered as a very bad sign, as sometimes it implies that the vendor didn't even care to port their local patches to newer kernels. Then it's important to check how old the version is :

$ git show v3.4.39
tag v3.4.39
Tagger: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date:   Fri Apr 5 10:09:01 2013 -0700

This kernel was released 4.5 years ago, while the latest version in the 3.4 branch is 3.4.113 which dates a year ago. This version missed 3357 fixes at last 3.4 release one year ago! And as can be spotted in the uname output above, last time it was built (hence had a chance to get a fix backported) was 3 years ago, which proves either a problem with the update process on this device or total a lack of care from the vendor. In short this kernel is totally unreliable and insecure. Such problems must absolutely be reported to the vendor. Some vendors are simply unaware of the problem and will be willing to improve; some have already improved a lot, at least to reduce the number of issues reported on their products. Some will explain that they just ship the kernel provided by their SoC vendor and that they have no added value on top of it, or worse that they don't even understand how it works. These ones at least need to be aware that some SoC vendors are better than others regarding mainlining, and that at least asking for a more recent kernel doesn't cost much and can result in less support calls on their side. And others absolutely don't care and must definitely be avoided in the future since there's no chance the products you buy from them will work well with thousands of unfixed bugs.

How bad can an unfixed kernel be ?

Sometimes people tell me that their kernel is not "that old". OK, let's see numbers for some 3.10 kernels still commonly encountered in field, compared to 3.10.108 (which will itself be outdated once released) :

Kernel versionage# of known unfixed bugs

It's unknown how many exploitable vulnerabilities are present in these kernels, however it's certain that all of them are at least locally exploitable, allowing for example a browser plugin to inject malware code into the system and take full control of the device to steal data or participate to internet attacks. And if you don't care about security issues, just think about some of these bugs that I have encountered on various devices running outdated kernels, some of which disappeared after I managed to rebase and rebuild the kernel :

  • random freezes and panics of all sorts, some even causing the device to overheat
  • File-system corruption on the NAND flash bricking the device (until I could reinstall it over the serial port)
  • file-system bugs causing the NAND flash to be "tortured" on every single block and aging very fast to the point of periodically reporting bad blocks
  • eMMC bug causing I/O errors and retries to happen every few kilobytes, making the device respond very slowly
  • SD card failing to enumerate after a few insertion/removal cycles
  • Memory leaks causing progressive slowdowns and regular crashes
  • WiFi random packet truncation causing many TCP connections to freeze and DNS to fail to resolve
  • WiFi disconnections of all sorts
  • WiFi to Ethernet bridge suddenly causing network packet storms by looping on certain multicast packets
  • File-system bugs on a NAS causing the disk to be immediately remounted read-only until the next reboot
  • Ethernet port on a NAS randomly switching between 100 Mbps and 1 Gbps several times a minute
  • Ethernet port not receiving packets anymore after some time
  • webcam driver bug killing the whole USB stack

Sounds familiar ? You can be confident that none of them are considered critical by your product vendor and that the relevant patches have no chance to get backported if they don't follow the official stable kernels (at least because it's hard to spot them as it's often hard to link the cause to the impact).

Should you upgrade to 3.10.108 now ?

The response is simply "no". 3.10.108 was emitted to "flush the pipe" of known pending fixes still affecting 3.10. It's fine the day it's emitted and possibly outdated the day after. Some late upgraders may consider that it could possibly remain OK for a few weeks or months, around the same interval as between two previous subsequent 3.10 kernels. But that's just "probabilistically" true, because if a high-level vulnerability were to be revealed, a new 3.10 would have been emitted immediately after with a fix. Now it won't happen anymore, so you're playing Russian roulette by deploying it. Of course one might think that it's less critical than keeping any other 3.10. But it's far better to upgrade to other stable kernels such as 4.4 which will be maintained till 2022. We at HapTech are using 4.4 and 4.9 in our more recent versions and both of these work very well.

So it's really time to switch now! 3.10 is dead.


What ESP8266 modules should look like


I've been using ESP8266 modules over the last 2 years for various IoT projects. These modules come with a number of very annoying characteristics which make their adoption problematic to beginners :

  1. they don't boot by default, it is mandatory to solder a number of wires and pull-up resistors just in order to get them to boot.
  2. their pitch is not 2.54 mm, so it's not possible to plug them on a breadboard to simply connect the required resistors and wires
  3. there are two pins (RST and GPIO0) to change between operation and programming, so when you soldered to achieve point 2 above, you have to desolder and solder differently just to flash, keeping a flying wire for the reset which must not stay connected (obviously)
Some vendors understood this and started to propose very nice development boards like the NodeMCU or Wemos D1. The NodeMCU clearly is too big to serve as a production model, but it exposes all I/O, integrates a USB UART, a "flash" and a "reset" button, and connects the UART's RTS and DTR signals to the RST and GPIO0 wires so that the programming software can automatically toggle these lines to program. The Wemos D1 provides all these features except the "flash" button, and is half as small. In fact it's only twice as large as the ESP12 module. This one may be used as-is for some projects.

But both boards present a big problem : the USB UART cannot be disconnected and it draws a lot of power. So if you want to use these boards for production, you can only use them for mains-powered devices, not battery powered ones. Some people explain how to cut wires on these boards to reduce the power consumption but it's a real pain to do. Thus often you're back to using the raw ESP12 device as-is and solder the wires yourself.

But then comes another problem : most USB UART devices adopt the now ubiquitous "FTDI" pinout, which exposes GND, VCC, RXD, TXD, DTR and CTS. The problem is that while DTR is an output and may be connected to GPIO0, CTS is an input and you don't have another output to use to select between running and programming, so it's still required to plug/unplug wires during programming. Some more advanced circuits implement an automatic RST+GPIO combo signal based on DTR only. But for me all of them have proven very unreliable, even after various modification attempts. First, GPIO0 sometimes emits a strong ~20 MHz signal preventing the DTR pin from going low and triggering the RST ; this does not happen while RST is held down however. Second, most often as soon as you open a terminal , DTR is triggered and resets the device again, which is not fun. Fortunately, during this time, RTS is low so it's possible to consider the combination of RTS and DTR instead of each individual signal. So as if it was not enough, DTR and RTS must not be directly connected to GPIO0 and RST, it's required to implement an exclusion between the two so that GPIO0 is triggered only when DTR is low and RTS is high, and RST is triggered only when RTS is low and DTR is high.

So I spend quite some time scratching my head trying to find the appropriate solution. The important points are :
  1. the board must be able to boot in running mode with only the power connected. This means that the pull-ups and pull-downs must be connected
  2. the USB-UART controller must not be physically present on the board, as it sucks power, takes space and costs money. Instead only the 6 pins required to connect a serial adapter must be present
  3. the board must use 2.54mm pitch, be very narrow, not larger than the ESP12 itself so that there are still pins left around it on breadboards
  4. the serial connector must respect the FTDI-compatible pinout so that any adapter will fit at least to allow regular communication with the device, and emergency flashing by moving RST by hand if needed.
  5. the RST pin must not be connected to the serial adapter but be available for RTC-based wakeup if required. Thus the module must use the CH_PD pin instead (also called "EN" or "CHIP_EN")
  6. the serial adapter must be modified to route RTS instead of CTS on pin 2
  7. the serial adapter must provide the logic to combine RTS and DTR as described above. If small enough it can be placed on the final board, otherwise it's better to have it as an intermediary board.
By searching for existing designs I found one part of the solution : Wemos not only does nice ESP boards, they also make an FTDI compatible USB UART on which you can decide to route RTS instead of CTS to pin 2. That confirmed to me that point 6 above can be addressed and become a prerequisite. I ordered a few and decided that in the mean time I'd modify my FTDI adapters.

For point 3, I cut some experimentation board to the same dimensions as the ESP12, plus one row for an optional serial connector. It happens that the 8 pin rows on each side are the same length as the module once expanded to 2.54mm, and that exactly 6 pins fit in the module's width!
I decided to place 2.54mm male connectors in the middle, spaced by 7.62mm so that they can fit on the middle row of a breadboard and even in a DIL16 socket on any board.

This left a central area where a few wires were routed and where there's enough room to install the pull-ups and pull-downs, addressing point 1 (I moved them elsewhere on this prototype as it was a pain to solder them after the connectors were in place).

By placing the serial connector close to the antenna there's no risk of touching the SPI pins at the bottom. Also the antenna is generally supposed to be located in an accessible place so it makes sense to install this connector at the same place. It also turns out that it was convenient to route RXD and TXD.

For point 5, I noticed during tests that a weak pull-up would still be better on RST otherwise it catches RF noise around (ie if you touch it with your finger). But that's a minor detail and doesn't prevent it from working.

So let's proceed with these various steps in order of dependencies

Making a usable ESP board

I didn't want to start making my own PCBs, it was a late afternoon's project. I decided to go with experimentation board, that I cut to the appropriate dimensions to hold the ESP-12E module and a 6-pin connector for  the serial port. Then I've cut unused traces as well as a central area to have two sets of connections. For those interested in trying it, you need to keep 11 rows of 6 holes. It's mandatory that it's single-sided because the ESP module will be placed on top of it and we don't want to risk accidental contacts :

Then I prepared the ESP-12E module. I've soldered very thin wires in each hole. The wires need to be long, at least 4-5 cm, or it will be a pain to place them into the PCB holes later. For this I've cut multi-strand wires and used their thin individual wires :

Now the difficult part starts. Before proceeding, it's important to pull the wires to ensure they're firmly attached. If some wire pops on the other side of the module, it needs to be cut. While pulling the wires, try to arrange them so that each side is approximately parallel with wires approximately 2.54mm apart. These ones will be placed into the PCB. Inserting the first side is not very difficult but requires patience. What is difficult is to insert the second one without removing any wire from the first one :-)

Given that the pitch between the two series of solders is not the same, some wires could cross. Of course we don't want this. So we want to insert the wires into the second row of holes on the PCB, like this :

It should then ressemble approximately this after you pull hard enough on the wires to ensure that none is blocked by another one or touching a neighbour :

Then quickly solder them, cut the remaining wires very short, and grind the solders so that the're as flat as possible :

Then the fun can begin. Cut two 8-pin right-angled male connectors that you solder 7.62mm apart, above the 2nd and 5th columns precisely, and solder them on the outside :

The last part for this board is to install the 6-pin connector for the serial adapter, and to connect the wires and components. I've used only 10K resistors, except in series with GPIO0 where I've added a 470R to protect the serial adapter against the output signal that's sometimes present on this pin during reset. I've also installed a 10 µF decoupling capacitor between VCC and GND because there was enough room for it and it was easy. The wiring diagram looks like this (it's easier than a schematics given that these are almost only wires), followed by the final assembly :



It's worth mentionning that if you don't see the pull-up resistor for the RESET pin on the photo, it's because I omitted it (as not strictly required), but I'll change this as the reset pin is now too sensitive to my finger, and I tend to reset the device when I touch it.

Modifying a USB-TTL serial adapter to provide both DTR and RTS

I'm using different flavours of FTDI adapters, and all of them have CTS on pin 2 :

I need to have RTS here. So I expected to be able to cut the PCB trace and solder a wire but I can't follow this trace which probably is under the IC so I don't want to damage my board.

Instead I decided to proceed differently by using a 6-pin male-female connector to extend the existing connector, not connecting pin 2 to the adapter but instead cutting the other pins, bending pin #2 over the PCB and connecting it to RTS which is on the chip's pin #3 according to the datasheet. That's all! Now you have a modified FTDI adapter with RTS on pin 2! This modification is very simple to operate. However don't make the same mistake I did, you need to glue the connector once it works, otherwise it will come out of the adapter once you unplug the adapter and pull off your wire. BTW since I couldn't find a 6-pin connector, I had to cut  a 8-pin one.

In the mean time I found that Wemos proposes such an adapter on which pin #2 can be configured to be RTS or CTS, so I will probably not develop further on this adapter's mod.

Converting the DTR/RTS signals to GPIO0/EN

It's not practical to directly connect the signals to the board, it's required to implement exclusion between the signals so that when DTR is low but RTS high, the chip is forced to flash mode, and when DTR is high and RTS low, the chip is reset. Otherwise the chip will work in a single mode, or all software will have to be modified to consider your protocol.

The truth table looks like this :

So it's as simple as doing :
  • GPIO0 = DTR | !RTS
  • EN = RTS | !DTR
In practice it's often made with NPN transistors and resistors on their base, but I do have a few low-voltage dual-mosfets in SOP8 packages which are very convenient because they don't require to cross PCB traces nor to add resistors. Some of them are IRF7313, but IRF7301, IFR7303 and FDS6990 will work fine as well. I also have a few other ones which have the exact opposite pinout, but I don't remember their identifier, so I stopped at the first one I found that I knew. The IRF7313's pinout is this one :


 The schematics we want is trivial (GPIO0 resistor included here but it's better placed on the ESP board) :

So the wiring can even be made using flying wires :

But I decided to be reasonable and to use another piece of PCB to make this one, so that connectors are firmly attached. The purpose will be to use the same pinout as the modified FTDI adapter on one side, and as the new ESP PCB on the other side. The result comes below :

I noticed that I wrote the pinout on the least convenient side, so it's better to do it the other way around to match what is displayed on the FTDI
module and the ESP module. And it worked like a charm on first test!

Now what is needed in the end ?

After all these modifications, and after having found the Wemos serial adapter which supports RTS, I concluded that the only part which does not exist is the ESP module for end users. So in fact given that the dual mosfet above is very small and consumes zero power, it should fit on the small board I installed the ESP-12 on. This board would then be programmable and compatible with deep-sleep. Also, even smaller MOSFETs exist, I've seen some dual in SOT363 package.

So by having just a board with 2.54mm pitch, the same pinout as the FTDI board for the serial connection, the MOSFET mounted and the few pull-ups, it would be possible to have the equivalent of the current ESP-12E module, but which could easily be programmed, either for development, or just for production. I think it should be sold with the connectors unsoldered. This way everyone can use it as a simple replacement for the current ESP-12 with a different pitch, yet program it and have it run by simply sending the power. Those who want to turn it into a development model can solder the two 8-pin barrels underneath, and it becomes compatible with breadboards and DIL16 adapters.

Those who simply want the ability to reprogram it in field just have to solder the 6-pin barrel connector and they'll be able to plug their FTDI-like adaptor in situ to reprogram it.

The best thing that could happen would be that some of the vendors like AI-Thinker create a new ESP module with such characteristics. It would solve all those issues at once. Right now when you look up "ESP8266" on Google image, you find tons of connection diagrams explaining how to flash them, which proves it's all but easy, or the NodeMCU models which provide the solution for development only. Let's think about it a bit further and make it usable for everyone! I'm not an electronician, just an occasional hobbyist and I could make it. But for people like me, it takes an amazing amount of time. Having the correct boards from the factory would be so much great!

I've ordered some ESP8285 which include the flash, maybe its possible to rebuild a board from scratch featuring an appropriate pinout. The same could be said for ESP32 which doesn't seem to have improved anything in this area unfortunately :-/


Scaling RC servo angles / ESC speed on the fly

Initial idea

When we started to play with our recently acquired RC cars, Benoit's kids wanted to try them. Unfortunately the cars are too powerful and too fast for kids, they can really be dangerous for them, and they can break them, so they were a bit frustrated by our refusals.

This led me to realize that such devices are missing a "training mode". I thought that maybe it would be possible to implement a speed limiter using an ATTINY microcontroller. After a bit of thinking, in fact it's really easy, considering that :
  • pulses are supposed to be 500 to 2500 microseconds long
  • pulses are 20 ms apart
  • accuracy is not a big deal for forward/backward speed
  • the signals are <= 5V amplitude
  • the 0 and 5V are present on the pulse cable
I wrote a small program consisting in a loop to measure the pulse width and to reimplement such a pulse afterwards. If the pulses were more frequent it would be a problem as they could overlap, but here after a pulse ends, we have at least 17.5 milliseconds remaining, of which at most 2.5 milliseconds will be used for the next pulse. So in the end, we have 15 ms left to do whatever we want. The program will have to loop through the following iterations :
  1. wait for a pulse
  2. measure the pulse width
  3. perform whatever computation needed, for less than 15 ms
  4. emit a new pulse
I started with some example servo code made for Arduino and placed it into an ATTINY85. I noticed there was some noise in the measure, as placing a servo on the output resulted in an unstable position. I understood that interrupt management and possibly other stuff outside the arduino's loop() were making the timing unstable. Last but not least, the code was huge and couldn't possibly fit under 1 kB for the smallest devices. But I could easily replicate the input pulse to the output. So it was time to get rid of the arduino environment and reimplement it in plain C.


The first important point is to be able to measure a pulse width and to create a pulse. We don't have a very high frequency so we want to waste the least possible cycles counting. I tried several constructs for a loop and ended up with 5 cycles per loop to measure a pulse, giving a resolution of +/- 0.625 microseconds at 8 MHz  (ATTINY85) and +/- 0.52 microsecond at 9.6 MHz (ATTINY13A). The code looks like this, with width giving the pulse width in CPU cycles :

uint16_t width = 0;
while (!(PINB & (1 << PB2))); // wait for beginning of pulse on PB2
while (PINB & (1 << PB2)) // wait for end of pulse
    width += 5;

To create a pulse, you need to wait for some time. Usually this is done using a volatile integer but here I noticed that it made gcc produce very poor quality code resulting in 14 cycles being spent per loop. Instead I used a barrier made of an empty asm statement that gcc couldn't optimize away. This resulted in only 4 cycles being spent per loop, which is much better :

PORTB |= 1 << PB3; // start pulse
while (--width) asm volatile("");     // wait width*4 cycles
PORTB &= ~(1 << PB3);       // end pulse

The width can be converted to microseconds by multiplying/dividing it by the CPU's frequency. Care must be taken to avoid integer overflows while sticking to 16 bits. For integral MHz frequencies it's trivial. For non-integral MHz frequencies, the conversion can be done on 32 bit while keeping 16 bit values in and out.

With only this done, we don't need the C++ nor arduino environment anymore and we can have a very compact C code.

Scaling pulses

I did a first scaling attempt by dividing the distance to the center by two. Given that the center is 1500 microseconds, the pseudo-code looks like this :

int16t width;
width = read_pulse() - 1500;     <0 = rear; >0 = front
width /= 2;
send_pulse(width + 1500);

It worked pretty well but I realized that I couldn't brake anymore by going backwards. In fact my car brakes before trying to go backward when it was going forward, and the braking force depends on how far I pull the trigger. Not being able to brake is not acceptable, so I had to implement the whole ESC's state machine.

State machine

I ran some tests and noticed the following :

  • at rest (initialization), the car can go backward
  • at rest, the car can go forward
  • when going forward and suddenly backward, it brakes
  • the car continues to brake until the trigger is released. In order to go backward, I have to release it and pull it again. This means that after braking and releasing the trigger, it goes to rest again.
  • if I only release the trigger when it's going forward, braking is possible for the first 2.5 seconds. Waiting longer will make it go backward. This means that after 2.5 seconds not accelerating it goes back to the rest position
  • it I pull the trigger again to go forward while braking it goes forward again
  • if I pull the trigger forward when it goes backward, it goes forward immediately

I ended up implementing the following state machine :
This state machine allows a different ratio to be applied to different states. The forward and backward speeds are cut in half, but the brake force is kept at full scale.

I found that the car was lacking a bit of "punchiness" when starting up, so I modified the FWD state to allow it to reach full speed for the first 300 ms. This allows it to deploy the full force to the wheels to make it drift and perform a U-turn for example, or to accelerate very quickly from rest, without permitting it to reach a high speed. In fact it even starts to become fun :-)

I also realized that knowing the current state makes it convenient to light some LEDs to indicate what is being done. We can have a set of brake lights made of red LEDs, and a backward light made of a white LED.


I found that the internal RC oscillator is not very precise so I implemented an automatic frequency adjustment at boot so that the car would not automatically start to advance or go backward. The idea is that when booting, the only signal we're supposed to see is the center position, so we average it over a few measures and compute the offset so that this one equals 1500 microseconds. This state is implemented at boot before going to the INI state.

Also the LED on PB1 (pin 6) which is present on some ATTINY85 boards is used as a debugging indicator. It is lit when we're not at the rest position. This eases centering on the transmitter because the led must be turned off by default.

Implementing the basic ON/OFF switch

Just similar to what was implemented from a disassembled servo a few months ago, it's possible to implement an RC switch from this controller by enabling/disabling a GPIO depending on the pulse width. So I added such a very simple test to connect to an output pin. After I realized that by implementing this on a $0.30 ATTINY13A, coupled with a $0.07 cable it could be 3 times cheaper than the previous solution, I ended up writing a simplified version of the program doing just that, called onoff.c. But the same function was implemented on its own pin on the main program so that pre-programmed chips can be use for both purposes without having to be reprogrammed.

Multiplexing LEDs

ATTINY13/85 only have 5 usable GPIO (well 6 if you reprogram the RST pin but I don't want to, it's too painful for development). With one GPIO for the input pulse, one for the output pulse, one for the centering LED, one for the ON/OFF LED, it leaves only one pin for the brake/rear LEDs. However there's a solution to solve this. If we put 2 red LEDs and 1 white LED in series, their total voltage is around 6.6V so they will not turn on on 5V. However it's possible to light either the 2 red ones or the white one by connecting their middle pin to the power supply or ground. The only thing is that to turn them off we need to disconnect the pin, which is equivalent to configuring it for input. It will only keep the internal pull-up which will be too weak to turn the LEDs on. We could use a single resistor connected to the GPIO, but it's safer to use one per set of LED so that in the event an over voltage would appear beyond their turn-on voltage, the drained current remains limited.
With all these features implemented, the code is only 642 bytes long for a 8 MHz ATTINY85, and 804 bytes long for a 9.6 MHz ATTINY13A.


The device was implemented on an SOIC8 ATTINY13A soldered on an SOIC8-do-DIP8 adapter serving as a PCB. The GPIO pins were chosen on opposite sides of the device so that wires could easily be soldered without having to bend them too much :

A simple male-to-female connector was cut to provide both the connector to the RX and the connector to the ESC. An extra 5-pin connector was added to access the LED signals. There is no additional component, only the MCU and cables. All this was enclosed in heat shrink tube and could be placed inside the receiver receptacle in the car.



Some improvements could be made. First, I forgot to solder a wire to the RST pin in case I'd want to reprogram the device. Second, it would make sense to support detecting a short-circuit on one of the GPIO at boot in order to disable the throttling. It could result in sort of an adult/kid switch. For example, the debug LED GPIO could be used for this since it's only used to debug the throttling.

It would also be nice to try to make the device learn the highest possible speed from the remote, but probably that it is not very easy to implement in that small code. There are only 220 bytes left on the ATTINY13A, so that's something to keep in mind.


The source code is available here. The executables are so small that they can be dumped here. For ATTINY13A, the fuses to use are :
  • low: 0x7A
  • high: 0xFF
For ATTINY85, the fuses to use are :
  • low: 0xC1
  • high: 0xDD
  • extended: 0xFE
Hex code ready to be flashed is provided below. Just copy-paste it to a file and flash it as usual with avrdude after having set the fuses above.

Code for ATTINY13:

Code for ATTINY85:


Injecting power into mains

This title sounds scary. In fact it really is :-)

I have long been experimenting with various solar panels initially to try to make some devices more autonomous (eg: self-recharging hexapod robot, long-lasting outdoor camera, etc) and noticed that solar panels have become very cheap and that low quality batteries can also be reasonably cheap. I started to think "what if I used a lot of batteries to power some of my home appliances like servers, water heater, etc?".

Let's do some math. Looking at my electricity invoice, I'm using on average 800W between february and august, and 1400W between august and february. I don't have more details, and both periods include some cold periods where the heating is used. I suppose that taking heating out of the equation I probably need
about 400W on average (light, PCs, water heating).

In order to deliver 400W 24 hours a day with 8 hours of charging, I'd need to charge at 1200W and to store 9.6 kWh of energy. A correct LiPo cell stores 2.6Ah at 3.7V, or 9.6Wh. Thus I'd need 1000 such batteries just to store the required energy. At about $2-3 a battery it's not interesting.

But I thought, why would I need batteries ? Two thirds of my consumption are during the day and one third at night. And these are differently priced so that the day is more expensive (15cts/kWh during day, 10cts at night). So by using 400W of direct solar energy I could expect to be able to cut the day consumption in half, resulting in a 37.5% overall saving on the daily bill.

This approach sounds interesting because it uses 3 times less solar panels and no batteries. But how to power devices on irregular energy sources ?

My idea was that if it were possible to inject the solar energy directly into the mains synchronized with the alternating current, I could in fact offload the energy provider from the energy I harvest from the sun, so that I don't have to care when a cloud passes. But it's possibly very hard to keep in sync with mains...

Then I started to think about some ZVS induction heaters I've made in the past, these ones are resonant self-oscillating circuits. Wouldn't it be possible to do this with mains ? That is, monitor the mains voltage to decide what polarity to send into a transformer ? After all in order to monitor the mains, I just need another independant transformer that's not too much affected by the power I feed into the secondary coil. Thus I ran an experiment with two miniature transformers (12V/1.2VA each) installed behind a 12/230V DC/AC converter (I prefer to use this than real mains during experiments like this). Impressively this worked pretty well out of the box, I could cancel the power consumption of the primary transformer.

That immediately made me think about these old transformers I salvaged from a dead UPS, I remembered they had some additional windings. I took one of them, a 400VA model, measured the voltage across its terminals and found that it matched the markings : 8V symmetric on a 50A circuit (designed to take 12V peaks on its input to produce the 325V peaks), and 14.5V on a low power circuit. This last one would be used to monitor the mains polarity. As long as I don't feed too strong a current, mains will always win and impose its polarity, and this monitoring winding would force the mosfets to reverse the current.

So I came up with the following diagram :

The primary (left part) of the transformer is connected to a power meter and to a 8W/230V lamp. The power meter is connected to the 12/230 converter for now. The assembled circuit looks like this (the MOSFETS are IRFB4110, they are directly mounted on the transformer terminals) :

The wave across the 14V monitor winding looks like this :

It's obvious that my 12/230 converter doesn't provide a sine wave, it's almost a square one! That's optimal to drive the transistors but it doesn't reflect reality. Since I had no smoke and I could manage to cancel the power usage by adjusting the DC voltage, I restarted the experiment plugged to real mains this time after double-checking every connection. The wave across the 14V winding looks much better :

The power meter showed that the transformer alone draws 4W of power and the lamp draws another 8W, leading to 12W total :

I progressively adjusted the input DC voltage until I saw exactly 0W on the power meter. The DC input showed 8.8V and the amp meter showed 2.26A, that's 19.9W total consumed power to produce 12W of power needed to power the lamp and the transformer losses resulting in 0W on the power meter  (I'm not cheating here, except by carefully adjusting the voltage to reach exactly 0, and the white reflect on the LCD comes from the lamp, entirely powered by my circuit) :

Note that some of these 19.9W are in fact injected into the mains, this explains why almost nothing heats here and why a higher current was needed to cancel consumption when connected to mains than to the DC/AC converter. Pushing the DC voltage higher results in the power meter showing a positive power again, indicating that I'm sending even more power into the house. At this point I stopped the experiment.

So yes, it is technically possible to re-inject some power into mains with a simple enough circuit. At this point one difficulty is to find the proper input DC voltage to cancel the mains without sending too much, because I don't want to dissipate too much power nor to send power back into the grid (with even the risk that it would make the disc spin again and charge me for the power I offer). Probably that producing only the minimum of any day's power usage would be a very simple way to solve this. Otherwise maybe a current meter on the mains connection to the grid could make it possible to have some feedback and regulate the injected power.

It's worth noting that during this experiment the delivered current was not very clean. On the image below it's visible that there were short peaks at each zero-crossing, caused by the short period during which none of the MOSFET is conducting, I even had to change the scope's time base because it couldn't stabilize on it :

But these artefacts should be very easy to cancel using a small capacitor across the transformer's terminals connected to the MOSFETs and it will also increase the efficiency and limit the MOSFET heating. Note that during this experiment, the MOSFETs were barely warm. Another improvement could consist in using a voltage comparator to always saturate the MOSFETs during the switch but given that the voltage is low at this moment it's not even certain that this is needed. I should also possibly use a choke in series with the central point of the transformer to absorb current impulses during the polarity inversion. But all this is very cheap and just a matter of experimenting a little bit.

Good solar panels having about 20% efficiency are found around $1/W nowadays, so for $400 it's possible to build a 400W array. Feeding 400W of power into the mains 8 hours a day every day at peak energy cost would save 175 EUR/yr, so the panels and circuits would be amoritzed in 2.5 years approximately.

Note that I thought about charging batteries at night and re-injecting the power during the day, but the cost difference and the inevitable losses would not make this worth, and it would take decades to amortize the batteries cost.

Has anyone successfully tried such a design ? Suggestions welcome.

Because the world runs on 5V

The success of USB as a power source is amazing.

15 years ago my friend Benoit Dolez told me "why not run a whole house on low voltage and avoid the risks of electric shock for everyone ?". I remember I responded "no, that wouldn't be practical, you'd need a very high amperage for this, it would require very thick cables and would cause too many losses". "Oh too bad" he said.

Of course I was wrong. Not on the technical side of things. On the social one. I didn't count on the possible success of portable devices making almost any device vendor provide a USB connector to receive some power.  Some even just draw copper lanes imitating the USB connector on their PCB. And this success wouldn't have been that huge without all those fantastic smartphones who stopped competing on how long they last on battery, and instead compete on how large the screen is and how fast their CPU can process pixels, even if that only lasts a few minutes before having to be plugged to the mains again (no single smartphone lasts as long as our 20-years old so-called cordless phones which could run on Ni-MH batteries for 8 hours talking).

All of this madness has resulted in the need for USB everywhere, instead of its usefulness. And when there is a need somewhere, there is a market, and there are innovations, products and progress.

I remember saying around 2008 in a shop "hey look they've put a USB connector on this wall plug, that's neat!". Right now in 2016 people visiting you at home routinely ask "where is the closest USB plug?". "The closest?". Not only it became obvious you have one, but you're even supposed to have many, all easily accessible! And that's true, we see USB-equipped power plugs everywhere. We even see USB-only power plugs. There are certainly more USB-equipped homes than WiFi equipped homes now. And that's what proves Benoit was right and I was wrong. No need for thick wires, just put a power supply into each and every plug and you're done! I tried to count the number of USB connectors I have at home and it's not easy given that every simple appliance has at least one. Counting those able to deliver power immediately or by just pressing a button or putting them in a very close plug is more meaningful. That seems to be around 20-60 on average, there are already 54 around my desk under power right now due to the PC, power plugs and the build farm. These days, small 5V power supplies cost around $1 shipping included and are cheaper than the equivalent high gauge cable you'd need to connect to the central panel under 5V. And a 6-10 port 60W USB power supply costs between $10 and $20.

But there's an area where it's less convenient to have 5V. When you're moving. Then you find lithium batteries coming in 1, 2, 3 or 4 cells, selling milliamps-hours. And all are lying since they advertise the cumulated mAh at the battery voltage instead of 5V, but since everyone does it, the first one not to do it would lose. What matters are Watt-hours (Wh), a unit for energy, they are the product of the voltage and the capacity in amp-hours. Batteries are thick and heavy (though lithium batteries are much lighter than lead or NiMH batteries). You tend to have as many as the places where you need them. I have one in my bag, another one with my laptop, another one in the room where I'm doing some hacking, another one with my bike serving for the front light (which is USB-based too, strangely). But with many batteries, you often have many nearly discharged batteries.

Thus this idea : why not have a very small battery in the pocket like you have your keys or your USB stick ? After all, most often you don't need the full capacity of your battery, and if you need it it's never large enough and you'd have preferred a larger one. So let's ignore heavy usages and go back to the most common ones : lighting a portable torch for a few minutes in the garage, recharging your phone to pass an quick call, powering a small WiFi router to connect your phone to a local network, recharging a bluetooth speaker or your TV's remote, having some light on your bike to go back home at night, etc... We all know such usages. And we'd be fine with a very small battery that's constantly in our pocket.

I decided to attack this idea with pretty good results. I ordered various small cheap batteries from a few Chinese vendors. I picked some models made for small drones because these ones are very cheap and support high charge and discharge rates (up to 5C charge and 20C discharge). I bought various capacities and tried to pick models that would be resonably small compared to the power conversion module. I ordered 100, 160, 240, 300, 380, 500, 680 mAh.

The reference on the batteries often have 6 digits such as "751517" on the 100mAh battery below. The first two digits correspond to the thickness in 1/10 millimeters. The 3rd and 4th one are the width in millimeters. The last two are the length in millimeters. This 100mAh battery is 7.5mm thick, 15mm wide and 17mm long. Yes that's tiny!

Then I ordered a few power conversion modules. These ones are made for the large 18650 lithium batteries. they're only 1.5 times larger than the USB connector and are both a step-up voltage converter (3.0-4.2 to 5.0V) and an adjustable lithium battery charger (often 0.5 to 1 amp output current) :

I started to build a few small devices by connecting the batteries to the modules and protecting them with heat shrink tubes. The 160mAh battery is as small as the PCB. The assembly was complicated because in order to save space, I had to desolder the USB connector and move it to the other side, so that I'd get a flat area at least on one side. It also required to modify the battery to move one wire to the other side :

The devices looked great but there was a lot of wasted area, especially on the largest ones (300mAh and more). On both photos below, the left one is the 160mAh version and the large one is the 300mAh one. Also I noticed that the modules were unable to deliver more than 800 mA, and my bike's front light needs 1.3 to 1.6 amp peak. More importantly, it always starts at the highest level which means that if the converter is not strong enough and cuts off, it's impossible to reach the lower power level.

Then I had another idea. I realized that this 5 volt frenzy is interesting, because nobody uses the 5V power line as is anymore. It's always converted to a lower voltage, sometimes 3.3V, sometimes 1.8V, or several intermediary values. In fact the USB bus itself requires 3.3 V pull-ups. I even observed that many USB webcams and WiFi adapters work pretty fine at any voltage between 3.3 and 5.0 V, and consume the same current, hence consume less power at lower voltages thanks to the LDO regulator inside which is only here to produce heat from any extra voltage above 3.3 V . Since each and every device comes with its own regulator to reconvert the 5V to something else, we could have imagined seeing a new trend of lower voltage devices or even better, wide range inputs. But that doesn't seem to be the case yet. So I tried to design my own such battery. The smallest one (160 mAh) was directly connected to the USB connector, without any PCB at all. It is pretty small :

My bike's front light liked it pretty much (a white led is 3.2V approximatively), however I found a few devices which couldn't run off it, often those which need to boost the voltage, because generally they employ low-Rdson MOSFETs which are not saturated below 4V which which leak a lot of power. So I stopped the experimentation there, still frustrated that I didn't have the ultimate universal battery.

After searching I finally found another module, based on a TP5400 chip, which can deliver about 1.2 amps on 5V from a fresh battery. I ordered two (just in case I'd kill one).

It's pretty well designed for my use case. Very little wasted space. One side has only low-profile components and the micro-usb connector, the other side has the USB connector, the inductor and the integrated circuit. It is possible to solder the 100 mAh battery on the flat side so that it doesn't increase the module's width at all.

Since the battery is only 100 mAh, I changed the charging resistor from 1.2k to 3.3k to limit the charging current to about 330 mA (about 20 minutes). This device is capable of powering my bike's light, and even to power my Clearfog base ARM-based server via a USB-to-12V converter. The module heats, but the device runs and works fine even with the Gigabit Ethernet cable plugged. Of course it will not last long, maybe 10 minutes only, at this rate. But sometimes it's useful. Just like my bike, at half light intensity (as I normally use it), it should last about 8 minutes. That's enough to cut through the woods at night when going back home... And I think I could put the 160 mAh battery there with a bit of effort. However, I noticed that my bike is pulling hard from this module, and if it had not been charged recently, it will cut off.

Finally I think I found a very good solution a few weeks ago, eventhough from an implementation perspective it's not optimal yet. Recently some powerful 3V-to-5V DC/DC modules have started to appear on the net, like this 5V/2A step-up module : http://www.ebay.com/itm/122122888739

It's important to note that they do not include the charger, but I already have plenty of TP4056-based chargers. So I ordered two of them and started again to try to assemble them. Interestingly this module extended with one USB connector is exactly the same dimensions as the TP4056, which is also the same dimensions as a 240mAh LiPo battery, and both modules are single-sided so they could be sandwiched to take less space :

I just had to surround it with some transparent heat shrink tube and fill every hole with some hot glue and this results in a reasonably strong power cube which easily accepts being dropped on the floor :

After a few tests, it appears very powerful. I could draw slightly more than 2 Amps from it, that's 10 Watts!
In addition I forgot to reprogram the TP4056 for a lower charge current, so it charges under 1A (which the battery supports as it's supposed to support 5C hence 1.2A), and is fully loaded in 15-20 minutes. The power conversion module eats a bit of power even when idle. I measured around 100 microamps, meaning that the battery will be depleted after 3 months not being used. But given that I have it every day in my pocket now it's not a problem. And it has already served me many times since I've built it! I've used it to power some development boards while I had no more USB port available, and last night to have some light on my bike when the original 2.2Ah battery gave up mid-trip. I managed to get at least 15 minutes of light, I don't know if it will give more but that was enough. The purpose was perfectly filled since I had it available in my pocket as every day now.

It's much smaller and lighter than the original 2.2Ah battery as can be seen below :

So what's the next step now ? I'm not interested at all in trying to create a business around this, but I'm pretty sure that many people would love to have a very small power reserve in their keychains. These devices are small enough to be arranged with the connectors inline opposed to each other and the battery in the middle. Just add a small keychain ring and you're done. I'd love it if one day I can buy a small keychain looking like this. Let's hope all the makers of programmable door remotes and USB flash keychains copy this idea and allow us to bring 5V everywhere with us with real power (at least 2A). I thought about using the same micro-USB connector as an input and an output but it would cause a cable issue as micro-usb to micro-usb is very rare. And using a male micro-usb connector would make it too fragile.

For the long term, it would be nice to see an evolution of the USB norm allowing certain categories of device to *officially* run from 3 to 5V. These ones would be advertised as "green power" or something like this because they don't require a conversion module which comes with losses, and would even smaller power blocks to be made (or with higher energy by converting the PCB space to battery space).