peterbudai

Collecting broadcast UDP packets using async networking in Rust

2019-07-11T10:58:03+02:00

I’m a big fan of Rust, although I usually have little free time to fiddle with it. But recently when I had a task at my job to create a small, greenfield networking application I decided to take the chance and experiment with asynchronous programming and the Tokio framework in Rust. I didn’t just succeed but actually had so much fun, that I even managed to push it to the other extreme: the whole console application essentially became a single async event stream.

ORIGINALLY PUBLISHED ON MEDIUM ON JULY 8, 2019

The application’s task is to discover devices on the local network that broadcast over UDP: it listens on a given port for a minute and collects packets that adhere to a specific format, along with their originator IP addresses. In this article, I’ll only cover the interesting parts, but you can find the complete source on Tresorit’s GitHub.

Asynchronous programming enables you to continue working on tasks on a single thread while you’re application waits for some long-running operation to complete, resulting in a more efficient use of CPU time. When you’re doing async programming, you have an event loop at the core of your application, which dispatches execution to the processing functions once an external event arises. In Rust terms, you define a long-running, asynchronous task, called a Future, that will be repeatedly polled by the Executoruntil the external condition it’s waiting for is fulfilled and it will finally return its actual result. Rust’s nice functional-style operators also appear here that allows you to chain and combine your futures to implement a more complex, yet fully asynchronous tasks without sacrificing efficiency.

From datagram bytes to a packet stream

The UDP packets we’re looking for have a simple JSON payload, so we’re starting with declaring that as a regular Rust struct, along with annotations for the powerful Serde library that will do the heavy lifting of JSON deserialization for us.

JSON payload object as a Rust structure, to be deserialized by Serde JSON

The Tokio framework gives you all building blocks to start doing async networking, starting with UdpSocket and UdpFramed. The latter converts incoming UDP datagrams into an async Stream: a long-running async operation which, unlike to a simple Future, can yield several results of the same type before being terminated — an ideal async model for a network connection.

An UdpFramed instance can even assemble higher level protocol data structures for us out of the network byte stream, regardless of how they are aligned within the stream; they can even span over multiple datagrams. All we need to do is to create an implementation of the Decoder trait.

Continuously decoding JSON payload from UDP datagram byte stream

Here we’re doing two things:

Declaring the return type of the async stream by specifying the associated type Item. There’s a small trick here. We’re more interested in collecting as much valid packets from multiple sources as possible rather than strictly adhering to the protocol format, so we even accept ones that cannot be decoded and simply drop them. For this we specify the return type to be Option, where None will signal a malformed packet.
Implementing the actual decode routine in the decode function. Note that the return type of the function is actually Result>>. The outermost Result is for signaling network or protocol errors — cases we would like to avoid if possible to keep the packets flowing, so we always return Ok. The middle Option is for signaling UdpFramed to give more bytes for decoding, we use that when an empty buffer is encountered. The innermost Option is the actual return type carrying a (in)valid protocol data structure — the result of a (un)successful JSON deserialization of the datagram payload bytes.

Now we have everything at our hands to build a nice asynchronous Stream of incoming beacon packets out of a plain UDP socket.

Building an async stream that returns (host_id, ip_address) pairs for the received UDP packets

We open the UDP socket, wrap it in a UDPFramed instance and also set up some transformations on the stream before returning it. These transformations include dropping the invalid packets (note the inner mapthat will effectively translate inner Nones to outer Nones); extracting the source IP addresses and the host_int field from the decoded packets, because only these two pieces of information we actually need to identify the originator devices; and finally converting the result type to Event — more on this last step later.

Adding a countdown timer

The devices on the local network emit UDP broadcast packets at regular intervals of 30 seconds. Therefore the discovery needs to run for at least this long, but I doubled it to account for lost packets: our application waits and actively listens for 60 seconds and then prints the results.

Photo by Agê Barros on Unsplash

There is a Timeout future in the Tokio library that will fire and terminate another future or stream after a given period, however we cannot use that here because of two reasons:

If a valid item is returned from the stream before the timeout period elapses, it will effectively cancel the timeout: it is designed for cancelling unsuccessful futures not for general termination use-cases.
Having a console application doing seemingly nothing for a minute is not quite user friendly. It would be much nicer to have visual feedback of the remaining time.

So instead of Timeout, we’re going to use Interval that is a built-in stream that yields a result periodically — in this case once every second.

Building an async stream producing countdown timer events

This is a quite straightforward implementation, but note two peculiarities:

We include the remaining seconds in the result. However, the stream is not terminated automatically when reaching zero, so we must take care that the number can never go below zero.
The Event type appears again as the result of this stream as well. I’ll cover the reasons in the next section.

Putting it all together

Let’s start with the Event type right away. It’s a great example of using Rust’s feature-rich enum type to create type safe unions: it either can be a Host ID — IP Address pair or a number of remaining seconds.

A simple enum to carry possible stream events

As you saw earlier, we use this enum as the result type of both the UDP and the timer streams. This enables us to do the most interesting part: combining async streams together to implement application logic.

Supporting optional streams

Before we move on to that, there’s one last thing that needs to be done. As you remember, our UDP stream can handle both IP protocol versions, but only one at a time. We would like to benefit from this and build IPv4/6 support into our application: it can listen for incoming packets on either an IPv4 or IPv6 interface, or even on both of them simultaneously.

From the source code’s perspective it means that we will have one or two UDP active streams concurrently. The latter could easily be done by applying the select combinator on the stream: it will yield the results from both streams, in order they arrive. There’s a caveat however. The underlying type of a selected stream is different from a single stream: the first one is Select, …> while the second one is some opaque impl Stream, more precisely FilterMap, …> in our case.

In the Tokio library, the Either type serves for combining two futures of separate types where only one of them can be “active” at a given time — unlike to Select where both child streams are active. Unfortunately Either is not implemented for streams in the framework, only for futures. Thus, we ended up rolling our own implementation, which is actually quite straightforward.

Combinator type for selecting one of two separate underlying streams

The main function

Now that we have all building blocks available, we can finally put it together in the main function of our application. Let’s go through the steps we’d like to implement before we jump to the code.

Visualization of the event pipeline by Ábel Siptár

First, we create the UDP streams that will emit HostFound events for valid incoming packets on either the IPv4 or IPv6 interfaces, or both. We’re using our EitherStream implementation to allow any combination of the two protocol versions.
We create the other source stream that will emit Countdown events once every second and combine it to the UDP streams resulting in a stream of any Events. Due to the common result type, we could do that as easy as a simple select call.
The next transformation will print out the remaining time to the console each time a Countdown event is encountered. More importantly, the take_while operator will filter out all events with the remaining time counter being zero, effectively terminating the whole stream — along with the UDP sockets as well. A pretty elegant way to do so.
If we got this far in the stream, the countdown events are no longer used for anything, they just noise in the code, so we simply filter them out, keeping only (host_id, ip_address) pairs in the stream.
These pairs will then be fed to a HashMap to filter multiple packets from the same host device. Remember, our discovery process runs longer than the beacon packet interval, and a host may also broadcast over both IP protocol versions.
When the stream is eventually terminated, the above map will contain the actual result of our application — the list of IP addresses we were looking for — so the only thing left is to display it to the user.

The complete event processor pipeline in the main function of the application

And basically that’s it. I’ve omitted some utility functions for clarity, but otherwise our application is now complete. 🎉

Conclusion

I learned a lot while creating this application. Asynchronous programming in Rust is still quite new (so is Rust itself), and requires a somewhat different mindset than you are used to, but creating production-ready applications with it is definitely possible.

Photo by Donald Giannatti on Unsplash

The example application here is a bit extreme in the way how async streams and functional style combinators are used exclusively over the source code, making it harder to follow. Async style is not suitable for every use-case, I think that in a professional environment you should end up with a more healthier combination of async and imperative code. On the other hand, I wanted to show that once you get accustomed to this style, it can easily drag you in — and you’ll still be having fun in the meantime. 😉

ORIGINALLY PUBLISHED ON MEDIUM ON JULY 8, 2019

I got interviewed on SE-Radio

2018-05-09T15:03:19+02:00

Thanks to Kim Carter I had the honor to appear in one of SE-Radio’s regular podcasts, where we talk about the current issues and upcoming trends of cloud security and creating secure, encrypted applications. You can listen to it here.

Kim Carter of BinaryMist turned out to be a fellow Tresorit user and a SE-Radio podcast host in one person. So he decided to put together a show on cloud security for which he thought about me as a good guest speaker based on the role and experiences as a developer and cryptography engineer at Tresorit.

In this episode, Kim and I are talking about end-to-end encryption, backdoors, the scenarios where E2EE can be and should be used. IM, VoIP, email scenarios, as well as interservice communication scenarios such as securing data in use with full memory encryption, CPU-based key storage, enclaves, cryptographic protocols. We discuss encrypting communications between microservices in the cloud. Which ciphers to avoid and which to use along with how to find the right crypto library for your next software project.

Software Engineering Radio is a podcast targeted at the professional software developer. The goal is to be a lasting educational resource, not a newscast. Three to four times per month, we publish a new episode talking to experts from the software engineering world about the full range of topics that matter to professional developers.

Using #pragma once with conditional compilation

2014-10-12T14:10:17+02:00

The #pragma once directive offers some advantages over the traditional header guard idiom, but as it is not part of the C/C++ standard we have to rely on the – not too verbose – compiler documentations. In this post, I examine what happens if you mix this directive with other preprocessor directives and conditional compilation.

C/C++ programs are separated to header (.h, .hpp) and source (.c, .cc, .cpp) files and header files are usually included from one or more header and source files. Care must be taken to avoid redefinitions of types and preprocessor directives. Every programmer knows how to use header guards as the only standard way of doing this.

The #pragma once directive is a newer method for this purpose that offers superior compiler performance because the compiler won’t have to read and preprocess the header file again. Other benefits include improved code readibility, less errors introduced by moving or renaming header files, and no cluttering of preprocessor namespace. Although it’s not included in the standard, all major C/C++ compilers already support it. GCC and Clang claim that they are able to detect the header guard idiom and offer the same performance as #pragma once making it unnecessary. On the other hand, Visual Studio encourages the usage of this directive over the header guards.

However, one aspect is not covered well in any of the compilers’ documentation, namely what happens if we not follow the traditional header guard idiom and mix #pragma once with conditional compilation. So I did a little test to find out how do they work if I put the directive in the middle of a header file instead of the beginning.

I’ve created the following header file that contains alternating code depending on a value of a preprocessor define. Only one path contains the #pragma once directive. The #pragma message lines are pleaceholders for the actual source code, se will be able to clearly see which lines are processed from the compiler output.

// a.h
#ifdef HEAD
# pragma message ("Head included")
#else
# pragma once
# pragma message ("Body included")
#endif

To test it, I have prepared a source file that includes the header file multiple times with different values for the preprocessor define controlling the header file. Similarly to the other file, I’ve put #pragma message directives to have more sensible output from the compiler.

// a.c
#define HEAD
# pragma message ("Including head")
# include "a.h"
# pragma message ("Including head")
# include "a.h"
#undef HEAD

#pragma message ("Including body")
#include "a.h"
#pragma message ("Including body")
#include "a.h"

#define HEAD
# pragma message ("Including head")
# include "a.h"
#undef HEAD

I’ve compiled these files with various compilers (Clang 3.4, GCC 4.8.3 and MSVC 18) and all worked as expected. I include only the MSVC output as it’s the most readable, but all compilers behaved the same way:

a.c
Including head
Head included
Including head
Head included
Including body
Body included
Including body
Including head

If we include only the first part of the header file (HEAD defined) it is possible to include it many times, but as soon as the second part is included (and #param once is read by the preprocessor) it will never be included again (regardless of whether the HEAD macro is defined or not).

Designing the cube – part 2

2014-05-11T19:12:42+02:00

In my last article I presented an overview of the LED cube, separating the design into three parts: the cube itself and the driver and controller circuits. This time I’m going to discuss the second one and emphasize the differences between the original building instructions and my solution.

Doing the math

Before we dive into the details of the driver circuit, we need to do some math regarding the timings and currents of our LED cube configuration.

I’m dreaming of playing fancy 3D animations on the cube once I’m finished. A visually smooth animation has 25-30 frames per second (FPS). To let each frame (a complete, yet still 3D image) stabilize before advancing to the next one, the whole cube should have a refresh rate around 100 Hz, which means this many frames will be displayed per second. Because in our case, the 3D image is created by quickly multiplexing eight 2D layers onto each other, the layer refresh rate is eight times higher than the frame refresh rate, resulting in 800 Hz. Calculating with the latter value further, we get 1 s / 800 Hz = 1.25 ms active time for a single layer. Assuming the 12.5% duty cycle we have calculated in the previous article, this means that each LED will be turned on and off for 1.25 ms and 8.75 ms, respectively. The original design had roughly 15 times faster refresh rates, but I find it to be an overkill, however if I experience some flickering I can easily boost the refresh rate from software.

The datasheet of the LEDs I have ordered from Ebay says that they can withstand a current of 30 mA for continuous operation or 70 mA peak current for momentary operation. Because the brightness of a LED is proportional to the current flowing through it, I’m planning to use the highest possible current. Although in theory the 1.25 ms active time and 12.5% duty cycle would qualify as momentary, I don’t want to overstress the LEDs during building and testing when they will be switched manually, so I decided to go with 30 mA.

The datasheet also states that the forward voltage of the LEDs are around 2.2 V. From this, we can calculate the size of the current limiting resistors required to achieve 30 mA through the LEDs.d Assuming 5 V supply voltage, the voltage on the resistor is 5 – 2.2 = 2.8 V and the required resistance is 2.8 V / 0.03 A = 93 Ω. However, after trying out one of the LEDs with a 100 Ω resistor, my measurements showed that the forward voltage is a little higher, so using a 82 Ω resistor would result in exactly 30 mA current flowing through the LED.

Driving columns

We need to control all 64 column terminals simultaneously. No microcontroller has this many I/O lines, so somehow we have to reduce the number of required outputs. I choose the shift register solution suggested in the original build instructions. This version uses 8-bit serial in parallel out shift registers. A shift register has 1 data input and 1 clock input terminal and 8 data output terminals. The data is shifted in serially one bit at a time in eight shift register clock cycles and then it can be read parallel on the eight data outputs. In this configuration the data outputs directly control 8 out of the 64 LED cube columns (I like to call this a row) via current-limiting resistors, so 8 shift registers cover all 64 columns organized in 8 rows.

The clock inputs of the shift registers are connected together, thus every shift register shifts their bits synchronously. Regarding the data inputs, we have two choices: we can either daisy chain the shift registers (the bit 7 of the previous shift register goes to the data input of the next one, so setting up a layer requires 64 shift register clock cycles), or we can fed the data to the shift registers parallel (each shift registers have an independent data input, thus only 8 shift register clock cycles is required to set up a layer’s state).

In the microcontroller, several instructions are required to manipulate the shift registers: fetching the next byte to show from the memory (2-3 instructions), shifting the bits (1-2 instruction), setting the data output pin (1-2 instructions), setting the clock output pin high and back to low (2 instructions), plus branching for loops (2-3 instructions). With the daisy chain configuration, this has to be repeated 64 times, resulting in 64 × 12 = 768 instructions. Assuming 1 MHz microcontroller clock frequency and RISC architecture (1 clock cycle per instruction), setting up a layer’s state requires 768 μs, so around 0.8 ms. Now recall the 1.25 ms layer active time we have calculated in the previous section. This means that in this configuration the microcontroller would spend most of it’s CPU time controlling the LEDs, leaving no time for anything else (computing animation frames or communicating with the computer).

Opposed to this, the parallel data feeding configuration (the one that is presented in the original build instruction as well) requires 8 times less instructions (0.1 ms estimated) at the cost of using 7 more I/O lines on the microcontroller. This seemed a reasonable compromise, so I ended up using the parallel data feeding configuration. The following schematic fragment shows the two different shift register chaining configuration (I show only 3 shift registers, but the principle remains the same).

The original design suggests the 74HC164 shift register. However, the 74HC595 would be a better choice, because it has a built-in latch, so it retains its last output state until bit shifting the next state is finished, which results in reduced flickering of the outputs. This is a great explanatory video tutorial demonstrating the differences between the two components. Because of the latching functionality, the 74HC595 has an additional input pin for the latch control. With this in mind, my design requires 8+1+1 = 10 microcontroller I/O pins for controlling the columns.

So far so good, but if you read the 74HC595 datasheet carefully you can notice that although the shift register can handle 35 mA per output pin (showed as I_OUT in the datasheet) – remember, I need 30 mA for my LEDs – that may be enough, but the whole IC cannot handle more than 70 mA (I_CC) in total, so if more than 2 LEDs are active in a row at the same time, the shift register IC will most likely end in up smoke. This is unacceptable as there may be lot of 3D animations that require more than 2 LEDs per row.

To mitigate this problem one may use additional components to deal with larger currents like external transistors or the ULN2803 driver IC. While these are both valid solutions, I tried to minimalize the number of components (thus potential error sources both during calculations and assembly) in my design, so I choose the TPIC6C595 shift register instead. As its part number suggests, it has exactly the same functionality as the 74HC595, but it has open drain outputs which can handle up to 100 mA per pin even if all pins are turned on (I_D in the datasheet).

If you aren’t familiar with the terms open drain or open collector (I weren’t also before starting this project), here is a great tutorial on how they work and how they can be used. There is a caveat you have to pay attention to when using open drain outputs. As this kind of output can only sink current (the current must flow into the pin towards the ground), all LEDs in the cube must have their negative (cathode) terminal connected to the shift register (they “point” towards the shift register on the schematic). This means the LEDs has to be wired in the direction “opposite” to the one presented in the original build instructions. As I wrote in my previous article all LEDs in a layer have their anode terminal connected and all LEDs in a column have their cathode terminal soldered together.

Driving layers

After figuring out how to control the 64 columns, let’s focus on controlling the 8 layers. Because the column driver logic uses 10 microcontroller I/O lines, I decided to save on layer controller lines by using a 3 to 8 line decoder IC which was not present in the original build instructions (or at least not for this purpose). A decoder has 3 inputs representing 3 address bits and 8 output lines of which one and only one is active at a given time selected by the address on the input – if the input is 000, 010 or 110 the output line 0, 2 or 6 is active, respectively. There are two decoder ICs in the 74x family: the 74HC138 and 74HC238 with the only difference that the first one has active-low while the second one has active-high outputs.

Since it is possible that all 64 LEDs are active at the same time in a given layer, we have to switch 64 × 30mA = 1.8 A, which is quite a lot. There isn’t any decoder ICs that can handle this much current, so we will need an external transistor. Since the LEDs that form a layer in the cube are connected together by their anode (positive) terminals (this is called the common-anode configuration) we have to switch the anodes (the positive power source terminal) to turn a layer on or off. This is called high-side switching (as opposed to low-side switching when we switch the ground terminal). The difference between the two is shown on the following schematic.

The next step is to decide whether to use traditional bipolar (BJT) or field-effect (MOSFET) transistors. BJTs are much cheaper so this was the first option I looked at. Bipolar transistors are current-controlled, the amount of current flowing through between the collector and the emitter is the function of the current flowing into or out from the base terminal. To switch 1.8 A on the high side with a PNP transistor, the base current exceeds 100 mA which is too much for the decoder IC. So we would need another transistor to control the first one, plus a few additional resistors (here and here are a few example circuits, though the latter uses the terms common anode and cathode incorrectly) that overly complicates the design.

In order to keep the circuit as simple as possible I choose MOSFETs over bipolar transistors despite their somewhat higher pricetag. For high-side switching only one P-channel MOSFET is required per layer without any additional components (the exact circuit is shown on the right side of the schematic above). Because MOSFETs are voltage-controlled there are no gate current limitations to deal with. However, we still need to carefully select an appropriate variant that is able to handle at least 2 A of source-drain current. I’ve chosen the IRF9Z34N that supports I_D = 14 A maximum current as specified in the datasheet. The P-MOSFET is turned on (current flows between the source and drainterminals)if the gate terminal has negative (ground) voltage and turned off if the gate has high voltage attached to it. This implies that the MOSFET can easily be controlled with the active-low version (74HC138) of the decoder.

Getting it all together

Now that we have both parts – the column and layer driver – designed we can assemble the complete LED cube driver circuit that you can study on the following schematics.

You can see the shift registers, current limiting resistors, the decoder and the layer controller MOSFETs and the connections to other main parts of the LED cube. JROW0-JROW7 and JLAYER are the connectors going to the LED cube column and layer terminals. J0-J9 are the connectors going to the controller circuit panel. J0 to J7 are the eight shift register data inputs, J8 is the 5 V power supply input and J9 is the connector for the IC control lines: shift register clock inputs and decoder address inputs. I’ve added two input lines I haven’t discussed so far. The first is the global reset line for clearing all shift register (and resetting the microcontroller itself on the controller panel), the second is an enable line that makes it possible to completely turn off the cube from the microcontroller by disabling the outputs of the shift register and the decoder.

There are also few additional components as well. There is a 10 kΩ pull-down resistor to aid properly turning off the layers. And there are some capacitors scattered around the schematic. These are required to deal with the sudden voltage drop running through the complete circuit when a layer with many active columns (thus large current need) is turned on. The capacitors are placed near every IC and MOSFET to sustain proper supply voltage to the digital circuitry even during a short temporary voltage drop.

Now that both the cube and the driver circuit is properly designed (I hope I haven’t made any miscalculations), I can finally start building the cube.

Designing the cube – part 1

2014-04-18T23:54:08+02:00

In this series of articles, I present the overall design of my LED cube. I try to emphasize the key points I had to take into consideration and the decisions that I had to make before settling with the final version that in turn will be assembled in reality.

Since I have no prior experience with similar projects, so I based my design on these instructions that are pretty thorough and cover almost every details. However, I needed to slightly change the design at some point. I recommend studying that article, because I will only highlight the differences. Some parts were not available and I tried to keep the costs and part number at minimum to have a clean design. But don’t rush too far ahead, let’s start at the beginning.

Main parts

According to my plans, my working LED cube will consist of three distinctive parts. The first one is the cube itself without any electronics that I’m going to discuss in the next two sections. The second part in the driver electronics that is responsible to supply appropriate currents and voltages to show up the desired image on the cube. The third part is the controller which is the programmable “brain” of the cube. Its main part is a microcontroller IC that controls what image or animation to play on the cube.

Cube layout

I employ the same layout as in the original design. My cube consists of 8 layers, each layer contains 64 LEDs in 8 rows and 8 columns. All LEDs in a layer are connected together by their anode leads. The layers are stacked on each other and the LEDs directly above each other are connected together by their cathode leads forming 64 columns. This is called common-anode configuration and it differs from the original design where the cathodes were connected in a layer (common-cathode). The cause of this change will be discussed later. To light up a specific LED, we have to connect the ground to the selected column’s terminal and some positive voltage to the selected layer’s terminal. This layout results in altogether 72 terminals: 64 for the columns and 8 for the layers.

As shown in this video, at a given time only one layer is active by selecting some of the columns, then the next layer is turned on with possibly different columns selected, and so on. This is called multiplexing, and if we switch layers fast enough, the human eye can be fooled to see all layers active at the same time resulting in a 3-dimensional image. This effect is called persistence of vision or POV for short and utilized in almost every display device from LED panels to televisions.

LEDs

Since only one layer is active at a time, each LED is lit only 1/8 of the time (which gives us 1 / 8 = 0.125 = 12.5 % duty cycle). This also means that the visible brightness of the LEDs are 1/8 of their maximum brightness. Standard LEDs have brightness around 40 mcd (millicandela, thousandth of the brightness of a candle), way too dim to use in the cube. That’s why I was looking for ultra bright LEDs that start from 1000 mcd and above.

These are LEDs in various sizes and colors. The two on the left are 5 mm diameter, the others are 3 mm. The ones to the right have clear housing while the other three are diffused ones with colored housing. You can see the two leads: the long one is the anode (positive), the short one is the cathode (negative) terminal. The cathode is also marked with a flat edge on the LED casing.

However, ultra bright LEDs are slightly more expensive and since the cube contains altogether 512 LEDs (8×8×8), this little price difference begins to matter. So I looked around on Ebay, by ordering directly from China I could save up to 70% compared to buying from local suppliers here in Hungary. The original design recommended diffused LEDs for better 3D light distribution, but unfortunately diffused LEDs with this brightness levels are quite rare to find at a reasonable price. So I ended up ordering 5000mcd clear LEDs in orange colour. Almost every cubes around the internet use clear LEDs, so I guess they will be alright.

Driver circuit

The driver circuit does nothing else but selecting exactly one layer and – at the same time – selecting the appropriate columns (0-64) to light up in that layer. Besides that there are two main reasons why we need dedicated driver electronics:

According to the layout, there are 64 column wires and 8 layer wires to control which LEDs are lit. No microcontroller has this many I/O lines, so the driver circuit employs some kind of multiplexing electronics that maps the 72 cube terminals to much fewer I/O lines.
Depending on the actual 3D image shown a layer may have all its 64 LEDs turned on at the same time, which needs quite a lot of power. Microcontrollers contain fine circuitry that can’t handle high currents, so we need some electronics between the cube and the controller that is designed to deal with such high currents.

In the upcoming article I will discuss the driver circuit in more details.

Some basic enlightement

2014-03-09T18:36:21+01:00

Let’s start with some basic theory, I promise it won’t hurt. First we need to get familiar with some basic concepts like voltage, current, resistance and their relation. Please note that here and throughout this blog, I’m not seeking to provide physically perfect explanations, just a minimal description necessary for me to understand the various phenomenons.

Current

Current is the flow of electricity, which makes everything work, it makes lamps glow, motors revolve and radios play music. Current flows through the wires and electrical components that can conduct electricity, hence they called conductors. In the vast majority of cases, these conductors have subatomic particles in them called electrons, and the flow of current manifests in the movement of these particles. More electrons moving through in a specific timeframe will result in larger electric current. But don’t worry, we don’t have to deal with such microscopic details. Current (I) is measured in amperes (A). Generally we will deal with milliamperes (mA), the thousandth of an ampere (1 A = 1000 mA), which is considered small current, while ranges above 10 A are considered large currents.

Voltage

But what makes current begin to flow? I’ll jump right to the answer: voltage does. Voltage has another fancy name, it’s called potential difference. While you wouldn’t call it that way, it helps understanding how it works. Imagine a basketball, this will represent our electron. If you put it on the floor (ground potential), it doesn’t go anywhere. Now if you pick it up, hold it high, then let it go, it doesn’t keep floating int the air but falls down instead. That’s because it was placed to a higher potential, so we have created a potential difference that made the ball start moving.

The way gravitational field affects the basketball is very similar to how electric field affects the electrons. If you create protential difference between two points and connect them with some kind of conductor, current will flow from the point with higher potential to the point with lower one. But how can I create potential difference, you would ask. You have to use a battery or some kind of external power supply like an AC adapter (wall-wart). In circuit diagrams, these power sources are shown as in the following image.

Voltage (U) is measured in volts (V). Values below 50 V is considered safe low voltages, flashlight batteries supply 1.5 volts, car batteries supply 12 volts, typical wall adapters supply from 3 to 19 volts. Values above this are dangerous to deal with. Wall sockets (mains) have 110 or 230 volts, while long-distance power lines uses many thousand volts. Generally, higher voltage causes higher current to flow, that’s what makes it dangerous for humans if they touch it with hand.

Power and energy

Power is the product of voltage and current. It describes how much electric energy a device consumes (or to say it a fancy way: dissipates), and how much work can be done with that device. Power (P) is measured in watts (W), the higher the value is the higher power a device has.

The actual amount of electric energy consumed is the product of power and time. Generally, energy (E) is measured in joules (J), but standard household electricity meters show kilowatt-hours (kWh) as units.

Resistance

As it’s name suggests, resistance (R) is a property of electronic devices, it show how much a device obstructs (resists) the flow of current. It is based on the physical characteristics of the device and it is measured in ohms (Ω). A piece of wire for example has a very low resistance (nearly 0 Ω), allowing very large current to flow through it, while a standard lightbulb for example has a resistance around 660 Ω. There are specific electric circuit parts called resistors, which come in various resistance values from less than 1 ohm up to several megaohms, and they are used to limit the amount of current flowing through. The circuit diagram symbol of a resistor is shown on the following image.

Getting it all together

There is a correlation between all these properties. Let’s consider the aforementioned lightbulb as an example. It has a tungsten filament that has R = 660 Ω resistance because of it’s material and shape. It’s nominal voltage is U = 230 V, which will cause I = U / R = 230 / 660 = 0.348 A of current to flow through it. The current will make the tungsten filament to heat up and glow. Lower voltage would result in lower current that would cause the lighbulb to glow much more dimmed. This correlation between current, voltage and resistance is called Ohm’s Law (yes, it’s the same Ohm like the resistance measurement unit, named after Georg Ohm).

The power of the lightbulb can be calculated from the voltage and current values by multiplicating them: P = U × I = 230 × 0.348 = 80 W. This means that it consumes 80 J of energy every second and if we turn it on for a whole hour, it will consume E = P × t = 80 W × 1 h = 80 Wh = 0.08 kWh or E = P × t = 80 J/s × 3600 s = 288 kJ of electric energy. Diet enthusiasts can further convert it to 69 calories, but it’s not advisable to eat electricity for breakfast.

I created a circuit involving our lightbulb in an electronic circuit design software and ran the simulation. The picture above shows the results (you can click on it to enlarge). In the schematic there is an external power source of 220 V, a 80 W lightbulb, two voltmeters and two ammeters (short for ampere meter). Voltage is always measured between two points, the first voltmeter (U1) is placed between the power terminals, the other one (U2) measures the voltage between the two terminals of the lightbulb. On the contrary, to measure current, it must flow through the metering device the same way it flows through the lightbulb. Both ammeters (I1 and I2) measure the current flowing through the lightbulb, and thus the entire circuit but they are placed to the different sides of the lightbulb.

If we start the simulation, the lighbulb begins to glow and we can read the values on the meters. It’s no surprise that both voltmeters and both ammeters show the same values, and the values match our calculations above.

I also created an equivalent circuit, replacing the lightbulb with a resistor of 660 ohms. I also reversed the polarity (the + and - ends) of the U2 voltmeter and the I2 ammeter, which now show the same values but with the opposite sign. In this circuit, current flows from the higher potential (+, Mains) towards the lower one (-, GND), but from the ammeter’s perspective the current flows from its - terminal towards its + terminal, so it sees reverse current flow which is denoted by the negative sign of the value. Very similarly, the U2 voltmeter sees that it’s - terminal has higher potential that it’s + terminal, so the difference between the potentials (aka. voltage) is negative: 0 - 220 V = -220 V.

Simulation software is very useful for testing an idea of a circuit before building it. Doing so you can avoid dangerous situations which could harm to you or damage your devices or equipment. Nevertheless, having such a tool, you can bypass lots of math calculations with a trial-and-error method.

Summary

I hope I could shed a light on the basic principles of electronics. I’ve created a small cheat sheet with the aforementioned physics terms and their relations:

Current (I)          I = U / R        I = P / U
    1 A
Voltage (U)          U = I × R        U = P / I
    1 V
Resistance (R)       R = U / I
    1 Ω
Power (P)            P = U × I
    1 W = 1 J/s
Energy (E)           E = P × t
    1 J = 1 Ws
    1 kWh = 3600 kWs = 3600000 Ws = 3600000 J

Further reading

I have discovered a pretty good website, with plenty of well-written electronics tutorials covering all topics that me or you will ever need if doing a similar project. I’ve read all the relevant topics that helped me understand more complex electronics devices like transistors, LEDs and integrated circuits.

My adventures with a LED cube

2014-02-25T19:54:25+01:00

It all started a few weeks ago when I decided to build my own LED cube. The decision was easy, but the realization won’t be that trivial. After some research I had to find out that there is more that I must learn than that I already know. I plan this post to be the beginning of a series of articles in which I’m going to present the steps towards a working LED cube and the small pieces of knowledge I gained throughout the process.

What is a LED cube? Like you can see on the picture above, an 8x8x8 LED cube consists of 512 small lamps (LEDs) arranged in a 3-dimensional structure with 8 layers, 8 columns and 8 rows. There are many good things about a LED cube:

It looks cool, you can program it to show any kind of 3-dimensional animations, which is even cooler.
You cannot just buy it at any corner (however, you can buy DIY kits which contains everything needed, you just have to assemble it), which makes it a special stuff in your room.
It’s far from impossible to build by hand, so it’s perfect as a hobby project, even for a beginner like me.

If you Google it, you can find many videosof working LED cubes and some pretty good build instructions. I always wanted to build one, all I was lacking is determination, but a few weeks ago I took a deep breath and ordered a bunch of LEDs and a microcontroller from China on Ebay.

As a beginner, even if you follow the build instructions very closely, you may find yourself in trouble at some point without some basic electronics and programming knowledge. If you’re going to diverge from the blueprints in the build instruction, because either you got slightly different parts, or you think you can come up with a better solution, or you just want to educate yourself, you will need some understanding of what’s going on. All the previous statements apply to me.

Since I’m a software developer, I’m not afraid of the microcontroller programming part, but the electronics stuff. The last time I was dealing with electronics problems was in a forgettable academic course back in my BSc studies accompanied with a 10-hour seminar course where we toyed with a microcontroller and an oscilloscope. Besides that I faintly remember some guy named Ohm and his law from high school physics course. Not much.

So I started educating myself on the topic, I spent my last couple of weekends with Googling around and browsing Wikipedia. I decided that I’m not going to just let this knowledge fade away, but keep posting some articles with my findings and progress reports of my LED cube here in my blog. So stay tuned!

BitLocker on non-system partition

2013-01-19T21:36:08+01:00

I know that nowadays passwords are indispensable for keeping other people away from my precious data, but sometimes it gets really frustrating typing them again and again, so I seize every option that offers me the same level of security without having to remember a new password. My laptop has a TPM chip built into it so when I was looking for an encryption software to keep my valuable private data safe in that unfortunate case when I lose my laptop or it gets stolen, I choose a solution that can leverage the TPM’s functionality.

BitLocker is a full disk encryption software by Microsoft that is bundled with every higher-grade versions of the Windows operating systems since Vista. In encrypts a whole disk partition which then can be decrypted typing a password (called PIN in BitLocker terminology), inserting an external USB drive, using the key stored in the TPM hardware or the combinations of two or all three of them. I choose the TPM-only authentication mode, that works transparently without any user interaction as long as the hard disk remains in my laptop and it boots from it. If any part of the pre-boot environment changes BitLocker will refuse to decrypt the drive. Please note that TPM-only authentication mode is not recommended because it is vulnerable to some kind of cold-boot attacks. These attacks all have in common that they require that my laptop is still running when the attacker acquires it. I decided to take this risk for the convenience.

My laptop has two hard drives: an SSD (primary volume) on which only the operating system and program files reside for faster system and software startup times, and a standard HDD (secondary volume) for all the data, my documents, source codes, photos, etc. My original intention was to enable BitLocker only on this secondary partition, because the system partition does not contain any sensitive information and the encryption would have some minimal performance overhead.

While setting up the BitLocker drive encryption using the built-in wizard of Windows 8 Pro that is installed on my notebook, I noticed with a great surprise that no TPM-only option was present on the authentication-mode selection screen. I double checked that the TPM hardware is functioning properly, so I started digging up the internet for a solution or an explanation.

As it turned out, this is because the way TPM and BitLocker operate together by forming a chain of protection. TPM protects (detects any change to) the pre-boot environment which consist of the computer hardware, firmware (including BIOS) and operating system boot loader. The boot loader protects the essential operating system files and the system partition if BitLocker is enabled. And finally, the system partition protects any other volumes. The key for the secondary partition is not stored in the TPM chip itself but somewhere on the system partition, so for the secondary volume to be decrypted automatically the system volume must be encrypted as well.

At the end I decided to encrypt both volumes, this time the TPM-only option was present for the secondary partition, too. Now my private data is safe on my hard drive without requiring to enter another password (besides Windows logon password of course, never have your operating system without any logon password, really, don’t). It’s worth noting that Windows 8 has a new BitLocker feature (at least I don’t remember seeing this on the Windows 7 computer at my workplace) that allows the user to encrypt only the currently allocated parts of the hard disk for the first time BitLocker is enabled. This is useful for newly purchased disk drives (like my new SSD) as it greatly reduces the time needed to enable BitLocker drive encryption.

Finally I got so much into data encryption, that I also BitLocked my USB flash drive. To be honest, there is much more chance to have that tiny flash drive lost or stolen than the laptop. Flash drives can only be decrypted with a password, however, there is a convenience option that they can be automatically unlocked on computers with BitLocker-enabled system volumes. With this option it is possible that my flash drive can be used on my laptop without any further action, but it requires a password on every other machines.

Windows API at its best

2012-07-18T18:54:58+02:00

With the advent of object-oriented programming frameworks for Windows like MFC or, most recently, .NET Framework, the average programmer doesn’t have to deal with low-level Windows API programming. Nevertheless, you can still do everything with C/C++ and pure Windows API, but with a little more work, of course.

I’m not willing to start a debate whether Windows API or Hungarian notation is good or bad. Personally, I prefer the descriptive, camelcased function names (CreateFile) in Windows API to the cryptic, as short as possible UNIX/Linux standard library functions (fopen), but that’s just a matter of taste. Every long-lived software project has its legacy, and this is exceptionally true for operating systems where backward compatibility is essential. The Windows API carries a lot of old stuff, one of it is the Hungarian notation, it’s there, you can’t avoid it, but you get used to it pretty soon. But there are also the API functions from the 16-bit era, long pointer types (like LPSTR), or the infamous TCHAR type and related function macros.

The reason why all this got into my mind is that a few weeks ago I stumbled upon a question on StackOverflow which mentioned an interesting Windows API typename, PCZZTSTR. The original poster even called it scary, and to be honest, he was right to some extent. Nevertheless, with a little WinAPI knowledge and some googling around (it’s not a widely used type), one is able to “decrypt” this typename to a more digestible description:

P is for pointer,
CZ means C-style string which is zero-terminated,
the second Z signs that the string must be terminated with another zero ('\0' character), like it would be some kind of string of strings,
T means that the string consists of TCHARs, which are chars on pre Windows NT systems and wchar_ts on newer Windowses,
finally, STR simply stands for string, which may be redundant after all the previous characters, but it’s there for consistency with other string types in the API.

So PCZZTSTR essentially means just a simple C-style string of TCHARs, which is terminated with two ‘\0’ characters. It’s not so scary anymore, is it?

Base class as method argument

2012-06-09T14:00:49+02:00

In modern object-oriented programming languages, protected members had always been treated as a kind of necessary evil, as they have more disadvantage than advantage. Almost all coding style guides will discourage you from using them. In this post, I’ll show another issue related to protected members which at first may seem unexpected behaviour, but on the other hand it is perfectly reasonable.

Let’s start with a short example, consider the following C++ code excerpt:

class Base {
protected:
    int prot;

public:
    int getProt() {
        return prot;
    }
};

class Derived : public Base {
private:
    int priv;

public:
    void setThat(Derived* derived) {
        derived->prot = this->prot;
    }

    void setThis(Base* base) {
        this->prot = base->prot;
    }
};

What you see here is two classes, Base with a protected attribute prot, and Derived which inherits this protected attribute from its base class. The two methods, setThis and setThat both access the protected member prot of their argument, the only difference is the static type of the argument (Base* and Derived*, respectively). At first sight, there is no problem with this code. However, when you try to compile it, the compiler will issue an error (I used gcc in this example, but I also tried MSVC, the result was the same but the error message was not so helpful):

test.cpp: In member function ‘void Derived::setThis(Base*)’:
test.cpp:4: error: ‘int Base::prot’ is protected
test.cpp:22: error: within this context

Why is that error message, you could ask, what’s wrong? Base::prot is protected, Derived inherits from Base, so methods in Derived should be able to access protected attributes from Base. That’s right, but there is a catch. If you look at the error message, you will see that the error is caused by the setThis method, the setThat method compiles just fine.

Deriving from a base class does not mean that the derived class will be able to access the protected members of the base class. It means that the derived class will be able to access its own members that were inherited from the base class and have at least protected visibility.

This is why the setThat method compiles without any issues. It accesses the protected member from the same class (Derived) where the method resides, this would also work with private members (priv, for example). On the other hand, setThis tries to access a protected member in Base, which is not allowed by the otherwise obvious visibility rules.

I was curious and also tried the same code in C# and Java (after trimming it according to the language specifics, of course). In C# the result was the same, as I expected. But surprisingly, the Java compiler happily compiled the source code without any error. At first I thought that maybe Java treated protected members differently than the other two languages, but then I realized that the classes were in the same package. And because in Java the default (package-level) visibility is stronger than the protected visibility, Derived was able to access the prot attribute regardless whether it is protected or not. This is yet another reason to use protected members with caution: in Java, they are visible from outside of the package, and this is usually not desired.

Back to the example, all I had to do is to separate the two classes to different packages. This is the Base class:

package base;

public class Base {
    protected int prot;

    public int getProt() {
        return prot;
    }
}

And here comes the source code of the Derived class:

package derived;

import base.Base;

public class Derived extends Base {
    private int priv;

    public void setThat(Derived derived) {
        derived.prot = this.prot;
    }

    public void setThis(Base base) {
        this.prot = base.prot;
    }
}

Now that package visibility was out of scope, I was able to reproduce the expected behaviour with the Java compiler as well:

derived\Derived.java:13: prot has protected access in base.Base
            this.prot = base.prot;
                            ^
1 error

I don’t agree that protected members are evil, but there are inevitably some issues around them that have to be considered when designing classes and object-oriented programs.

Website launch

2012-05-08T07:00:37+02:00

I’m glad to announce that the peterbudai.eu website has been launched.

By this time, most of the hardware and software environment had been established and considered stable. There are still some work to do, mostly on improving and unifying the design all across the website, so short temporary outages and continous changes may still be expected for a couple of days.

I’m continuously filling up the blog and the website home page as well with contents, so it’s worth visiting the site regularly from now on. An even more convenient way to subscribe to blog updates using the RSS feed on the right.