Embedded systems debugging - overcoming inadvertent obstacles

Dr Johan Kraft
CEO and founder
De-bugging embedded systems
Debugging embedded systems running a Real-Time Operating System (RTOS) is made simpler by the use of software tracing, which allows software events to be traced in order to monitor and better understand the code’s execution. This can be useful during debugging and optimisation.

Software tracing is generally implemented in two ways; hardware-defined tracing generates data on practically every software instruction, is implemented in the processor itself, and usually requires some specialised hardware such as a trace probe or emulator to receive the data. The second method, software-defined tracing, offers greater control over which events are traced, making it easier to exclude irrelevant events, reduce the amount of data and focus on critical events.

While software defined tracing doesn’t require any additional hardware, it does introduce a small software overhead. This performance penalty can generally be accepted as the price paid for the ability to focus in on what’s really important. So, instead of representing a performance penalty, it could be seen as offering greater productivity over a hardware defined trace approach. Because of this, many RTOSs now have support for software defined tracing built into them, automatically tracing events such as task switching and kernel service calls.

Similarly, some processor architectures have support built in for hardware defined tracing of specific events commonly seen as pertinent to debugging, such as software exceptions or interrupts. But, as stated earlier, many processor types offer no control over which exceptions are traced, so excluding those of no interest isn’t possible.

This results in high amounts of data; often more than common tracing systems are able to capture. This yields an incomplete trace where the events of interest might have been missed.

Conversely, software defined tracing enables developers to capture only those exception handlers of interest. This is typically achieved through the use of functions or macros called at the start and end of the exception handler.

However, software defined tracing of exception handlers isn’t without its own challenges, particularly in the widely used ARM Cortex-M architecture, which has been designed to optimise exception processing to deliver lower interrupt latency. A prime example of how this is achieved in practice is Exception Tail-Chaining. This optimisation switches from one exception handler to another without first switching back to the primary context. This streamlines execution and is effectively invisible to the software. However, it also makes it difficult for a software defined tracing solution to correctly capture the switch between the two exception handlers, since the software does not know if Exception Tail-Chaining has been used in a specific case.

If the trace is carried out without being aware of the Exception Tail-Chaining optimisation, i.e. assuming that the exceptions always return to the previous context, the above scenario would be presented.

In order to allow accurate software-defined exception tracing also on devices with Exception Tail-Chaining, like ARM Cortex-M MCUs, a way for a software tracer to distinguish between the two cases is required. Fortunately, the benefits and penalties of Exception Tail-Chaining are understood by developers of advanced debugging tools, and some now employ methods to work around this inadvertent obstacle.

It is achieved by understanding that it takes a determinable amount of time for a processor to switch context from one exception handler to another, or to switch context. This time is, of course, definable in terms of clock cycles, so the (relatively longer) time necessary to restore a previous context can be differentiated from the (relatively shorter) time it takes to switch to another exception handler. By analysing the number of clock cycles between exception handlers it is possible to determine whether Exception Tail-Chaining has taken place, and thereby display the trace correctly.

Software-defined tracing is not difficult on a conceptual level, but the devil is in the details. Getting a correct trace, in the presence of the kind of hardware optimizations explained here, and with minimal performance footprint, requires significant tracing expertise; solutions need to be developed in collaboration with leading processor and RTOS vendors. Moreover, making the data accessible and meaningful to software developers requires tools that provide visualizations optimized for embedded software, attractive to use and easy to draw conclusions from, giving actual value during the development and debugging stages of design

Share this page

Want more like this? Register for our newsletter
The Developing Role of Electronic Component Distributors Ian Poole | Electronic Notes
The Developing Role of Electronic Component Distributors
The service that electronic component distributors has provided over the years has changed very significantly. Nowadays, distributors provide a very effective service, meeting the many needs of development, manufacturing and service organisations small and large.