You are on page 1of 10

Debugging software/firmware using trace function re-usable components

How to use functional trace logging for effective firmware/software debugging of embedded systems under dynamic conditions and how to reduce the need to fix defects or anomalies after-the-fact, using emulators/simulators.

One of the challenges in real"time systems, especially in multitasking OS based implementations, is defect fixing. To resolve the defect one has to be aware of the program flow during the defect or faulty condition. Normally, this is done by using in-circuit emulators (ICE) along with the break point feature available in the environment of the emulator. However, the ICE support may not always be available for the system under consideration. Considering the case that ICE is available for the system under consideration, and the ICE hits a breakpoint, then the dynamics of the system is lost. All the interactions the system was having with some networks or other systems/modules are stopped at once. Therefore it again may be difficult to reproduce the defect or anomaly exactly. Though some makes of ICE provides a feature of real-time trace function, but that is also limited by the ICE memory. At times it is required to stop the system in some logical trigger, so that the condition of the system can be analyzed at that point. This feature is also not common in normal debugging systems. Finally some development systems from WindRiver, tools like WindView offer powerful debugging mechanism. This mechanism allows user to track code flow and task/interrupt scheduler switching in a GUI based interface. However, the development system should have support for deploying this debugging feature. The method that we propose in this article overcomes numerous such problems with no extra investments. This requires some amount of free memory on board for both code/data and substantial amount of processing capability. The function trace re-useable component The method we have developed can be applied to debug the software/firmware of any type of real-time system and is named as "Function Trace" (FT). In this method, all the functions in the program are uniquely coded and as the function is executed the unique code is noted. These values are stored in a circular array in round-robin fashion. This is called as function code logging. The array contains the program flow information which can be used for debugging purposes. The basic requirement of employing this method is to have moderate speed processor with some free memory. Thus 32-bit processors are the best choice for applying this method of debugging. Depending on the processor speed this debugging method may be looked upon as an over head for low speed / low end systems. The tracking capability of the functions executed in the system makes it best suitable for both OS based multi-thread system and as well as OS less systems. Function Trace can also be deployed in multiple locations within a given function to figure out about the program flow within a function just as we might use it for program flow within a module / thread / task. Function Trace can also be added on with time stamp provided the system considered has a counter or a Real Time Clock on board. This enables us to figure out not only the program flow but also the time taken in various functions / modules within the system.

One of the important feature of this debugging method is that it allows the user to trace only the function(s) which they intended to trace. Therefore this mechanism eliminates tracing functions which are not required, resulting into optimal usage of memory for storage of tracing data. Therefore this helps the developer to focus on the required sections of the code. Function Trace's basic operating principles The Function Trace feature (Figure 1 below) creates an execution log of every function call defined in the application firmware. As any application-defined function is called, an entry for the current task/interrupt context (for multi-tasking system) and function called is generated in the log with the help of a unique 32bit identifier code.

Figure 1: The Concept of Function Trace

The identifier can be generated by concatenating a function code and file code, where the function is defined. To control the logging of function trace better, the functions can be divided into various levels thus allowing the control of execution flow log. The entire application might be, for example, divided into numerous small applications. The smaller applications might be divided into modules and further into units. Thus one might think of tracing at the module level functions, some might think of module level or unit level functions for zooming into the problem area. To implement this control, a function level flag is used. All the functions are categorized into some levels. Based on the flag (set/ reset) the function will be logged to the trace for a given level. The logged data shall be stored in RAM within the embedded system, therefore the user of this method has to plan a way to retrieve the function trace data through communication ports or ICE connectivity or any other method suitable for the system under debug. After the data is extracted, further analysis can be done upon the sequence of execution of the code during the faulty condition. The architecture of function tracing The basics of function trace architecture are divided into the following: encoding of functions, task/interrupt encoding, trace logging and log management. Encoding of functions. The function encoding is a very important part in this method. All the functions are to be identified by a code which should be unique. This encoding can be done in various ways. An example is discussed below:A 32-bit value is used for encoding all the functions. All the files / functions / parts within a function within the entire program are given unique code. The first 16 bits are used to encode a file, and then all the functions in the particular file are segregated by one unique 8 bit no. Combination of file identifier and function identifier will make the function code unique. The reserved section might be used to perhaps encode positions within the function or any other information that is useful for debugging. The division of 16bit for file name and 8 bits for function are just examples. One might use any other combination too as shown in Figure 2 below.

Figure 2: The Function Code For example, as shown in Table 1, below, let's suppose an embedded program contains 3 files File1, File2 and File3. File1 contains Func_a and Func_b, File2 contains Func_c, File3 contains Func_d, Func_e and Func_f.

Table 1: Coding of functions Task/interrupt encoding for rtos based systems. One might also encode the context of execution (Interrupt/Task) within the entries in the Function Trace. Tasks or threads in RTOS based system are identified with some identifier. This value (normally 32 bit) can be used to identify the task. Depending upon the RTOS, the task identifier is located in the Task Control Blocks. This value is used here to identify the tasks. Similarly interrupts may also be identified with their levels or numbers. Thus the task identifiers and the interrupt numbers can be used here to segregate the context switching. This entry can be limited to interrupts and normal context in the Non-RTOS systems since there is only one main loop in such systems. This technique also helps in identifying and debugging functions which are being used in both the contexts like interrupt and main loop, tasks etc. Logging of trace. Whenever a function is called, the function code will be stored in a circular array. The size of the array will depend on the resource availability and depth required for defect analysis. A large array will increase the time depth of the logged record. Normally in non OS based system the function code will be logged. In OS based system, along with the function code the task-id or the interrupt level from which the function is being called can also logged. The structure of the function trace variable will be as follows:

For example, a logging function (logFunctionCode) can be called at the entry point of all the functions to store the record, or a logging function might be called at a function exit or even at a critical phase or even at

an exit. Therefore depends upon where the debug is required. The pseudo code for the function definition will be as follows:

Note that the entire logging function is protected from preemption or interruption. This is noted as the "critical section" in the pseudo code. Log Management Operations and Features Log management involves xx critical operations and capabilities, indluding the indexing of the function trace, function levels, and trace start/stop. Index of function trace. The function trace is logged in a circular array with a single index. At times, it may become difficult to identify the last logged entry of the array. Therefore the index of the function trace array will have to be saved as a global entry. The index that we are discussing about has to be protected from being corrupted in case the function trace logging function is called by concurrent functions / tasks / interrupts. Therefore it is mandatory to protect the logging function using critical section. The logging function has to be compact and small. Function levels. The time depth of the function trace depends on the size of the trace log array. However, system may not require logging all the functions such as low level driver routines. To implement this selective logging, all the functions in the program can be categorized into some levels as shown in Table 2 below. The call of the logging routine can be enclosed into the level macro.

Table 2: Coding of Functions with Levels

As shown in Table 2 above all the functions in the program are associated with some level. Thus the function trace log call will be as follows:

By enabling the macro (Eg: LEVEL1 or LEVEL2) globally, the logging capability of the functions can be modified as per the requirement. Therefore, one should ideally create functions along with the function trace call along with appropriate logging level macro. Compiling the code as per the set macros would result in code generation as per logging requirements. Start/stop of function trace. While trying to fix an anomaly, one may require starting and stopping the function trace to focus onto the defect / anomaly at hand. Normally at the start of the program the function trace logging will be started. The logging can be stopped in occurrence of an event in the operation. This can be done by evaluating a Boolean variable at the start of logFunctionCode() function as shown below:

By controlling the Boolean variable "stopFunctionTrace", the logging can therefore be controlled. The function trace can be stopped in a number of different scenarios, including: * In the predefined error state, which is common in most of the embedded firmware * On occurrence of software or hardware exception * Also in some relevant logical event, a trigger can be designed to forcefully stop the logging The mechanism is however to be decided by the developers with respect to an embedded system under consideration and the hardware under consideration. Function trace enhancements Other than the features discussed above, the functionality of this component also can be enhanced in numerous ways, however not exhaustive, including time stamping, trace positioning, and function trace analysis. Time stamp functionality. In real-time systems data analysis of the timing of execution is equally important as the flow of functionality. The time stamp of the function log may also be taken along with the function code provided a real time clock is available in the system. A relative time-stamp can also be logged using a free running counter if available in the hardware. In this case the function trace array will contain one more entry for the time stamp field. While logging the function code the free running counter value would also be stored. The relative values between the logs will indicate the time required in execution of the functions traced.

Position trace. At times it is required to trace the part of the function executed especially for OS based system or in case of some functions where the entire function is divided into some cases, the actual case of execution is the point of interest. The position trace can trace different parts of the same function. The last 8 bit of the function code (See Figure 2 earlier) was left reserved. This part can be coded to identify different lines of the same function. Therefore allowing the user to understand how that particular function is executed during the defect / anomaly condition. Function trace data analysis. The data logged by function trace is stored in the Function Trace Array which is located in the system RAM. One has to design a mechanism by which the data might be extracted out of the embedded system and be viewed in a formatted form. The general options that might be considered are as follows: * Most of the embedded systems generally have a communication channel to connect to the external world. Using this communication channel, the data can be taken to the external world for analysis. * Some systems have non-volatile storage device with in the board. The function trace data can be stored into this device and can be suitably taken out at a later point in time after connecting an ICE. The data once extracted from the system can now be interpreted as per the encoding done in Table 1 earlier. The index shall provide the pointer for the last entry of data and therefore the interpretation shall be required to be done accordingly. However care should be taken about the endianness of the data since it depends on the type of the target system. (Little Endian or a Big Endian machine) Thus the developer will realize the program flow which would provide leads for resolving the anomalies.

Parser details The data available from function trace is a couple of numbers. Some procedure should be defined to parse the data into a readable form. This would provide easy interpretation of the captured data. One might develop a macro or a lookup table for interpreting the results in Microsoft Excel. Microsoft Excel based. Assuming that the data for the function trace is available in form of two columns of data, where the first column represents the task id/interrupt level and the second represents the function code. Now the parsing can be done using MSExcel as shown in Figure 3 below. Column A contains the function code and column B contains the corresponding function name. Similarly, column H contains the Interrupt level/Task ID and column I contains the corresponding Task/Interrupt name. Now the available data for function code is copied on column E and task ID/Interrupt level on column C. The table lookup formula is used to decode the function name, task name and interrupt name. Column D and F are the resultant output. Also the last index can be pointed out manually in the blue mark to segregate the start/stop of the data.

Figure 3: Excel spread sheet for interpreting the Function Trace Results C or Visual Basic Software Tool. A parser also can be written using any programming language like C or Visual Basic. A parser tool will take the input from the function trace and update the function name and task/interrupt name through table look up in a data file. The sample file in Figure 4 below shows the parsed output of the function trace.

Figure 4: The Output of Software Tool A price to pay - Function trace tradeoffs Function trace also has some points to be remembered during implementation: * Function trace logging takes some CPU time and might pose as an overhead to the system. Therefore in highly time-critical applications it may affect the timings of the application * This method might become an overhead for slow processors executing large programs * The log array may take a significant amount of memory. Thus memory critical systems may face some memory crunch to implement this debugging mechanism * The position of the logging within a function is another critical point to be taken into account. The user may be confused by the data generated in some preemption cases * This method neither locates the actual point of preemption nor does it have the facility to watch the values of the parameter passed to the function. However this limitation may be minimized using the position trace feature. A last word This method is successfully implemented in some of the live projects in the PES Industrial Automation subvertical. It was observed that the turn around for complex defects had been reduced significantly. As it is a one time implementation, it requires very little maintenance. It has also helped to measure the time required to execute a particular function using the time stamp entry and without using a DSO/Logic Analyzer. Last but not the least, the logged output of the Function Trace helps people understand the code flow, RTOS task switching and preemption and therefore improves the understanding of program flow. About the author Anindya Dutta is a Embedded Software Engineer having over 7.5 years of experience in the Industry. Currently he is working for Patni Computer Systems Ltd. His core areas of skill are design and development using C, C++ and Vx-Works in the Domain of Industrial Automation and SafetyTridib Roychowdhury is a Technical Architect with more than 11 years of total experience which spans over as a Firmware Developer, in the industry, and a Design Engineer in the engineering industry. Currently he is working for Patni Computer Systems Ltd. His experience profile includes firmware development using Assembly, C, C++. His industry experience is in designing various products for the industrial automation industry.

You might also like