TECHNICAL FIELD

The present invention relates to design-phase optimization of computational systems. More particularly, the present system and method pertains to generating performance benchmarks for custom computer software run on particular hardware systems. The present system and method also pertains to generating one or more performance benchmarks which are relevant for real-time, time-critical computational system performance.

BACKGROUND

Real-Time or Mission Critical Programming

The field of real-time computing (RTC) (also sometimes called “deterministic computing”) pertains to hardware and software systems subject to one or more “real-time constraints”, for example from event to system response. Real-time hardware and software programs must typically guarantee responses to system events within specified time constraints, which may be referred to as “deadlines.”

Systems used for many mission critical applications must be real-time, such as for control of fly-by-wire aircraft, or anti-lock brakes, both of which demand immediate and accurate mechanical and electrical response. More generally, in aviation systems or in transport systems of other types, engines and control systems may be required to respond to critical environmental events within a specified time in order to sustain safe operations of an airplane or other vehicle. Other systems which entail no mechanical components may also be mission-critical. For example, in order to maintain uninterrupted and/or high-quality communications of mission-critical data, a communications system may require mission-critical packet routing, switching, data compression/decompression, data encryption/decryption, etc.

In typical systems, time-critical responses reflect automated operations (that is, without real-time intervention of a human operator), and the deadlines for system responses to events may be on the order of milliseconds or microseconds. A system not specified as operating in real time cannot usually guarantee a response within any timeframe, although typical or expected response times may be given. Real-time processing fails if not completed within a specified deadline relative to an event; deadlines must always be met, regardless of system load, for optimal or even for safe system performance.

A real-time system can also be characterized as one which receives environmental or system data (typically from system or environmental sensors), processes the sensed data, and returns result(s) with sufficient response to affect the system operations and/or the environment at substantially the same time as the data was received, that is, without significant delay.

Software for real-time applications typically needs to be carefully coded and finely tuned for optimum performance on a specific, designated hardware microprocessor. Real-time software applications may include either or both of application-specific software and real-time operating systems.

Software Benchmarks

Software benchmarks are numeric values, determined through testing of software (either code in development or commercially released software) which indicate the performance speed of software. When a particular program or software module has a higher numeric score than other, functionally similar software modules, the high numeric rating typically indicates a faster performance speed. In the alternative, benchmarks may also be configured so that lower numbers indicate reduced performance time for tasks, and therefore better performance.

Many benchmarks exist today which measure the performance of hardware as opposed to software, typically measuring the performance of hardware microprocessors. Some well-known benchmarks include Dhrystone, whetstone, and several developed by the Embedded Microprocessor Benchmark Consortium. Typically, however, these are all generalized benchmarks principally designed to determine the performance of the underlying hardware such as microprocessors (possibly along with related hardware such as data buses). They are principally meant to characterize the relative general performance of different microprocessors, irrespective of the particular application software which is run on a given microprocessor.

The existing generalized benchmarks suffer from the similar deficiencies, which can pertain both to real-time programming and to programming for less mission-critical applications (such as general purpose business software.) The existing benchmarks measure what they are designed to measure (for example, integer performance or floating point performance, or some combination of both) on a selected hardware processor. For particular software applications running in specialized environments—and in particular for real-time systems—it is difficult to estimate or compare processing performance for a particular system and application via these generalized benchmarks. This is because particular real-time environments and real-time software applications have unique and distinctive requirements for integer commands, floating point commands, and other low-level memory commands.

What is needed then is a system and method to profile an existing application (often an in-development alpha or beta application for a real-time system), as the application is running in a representative hardware environment. What is further needed are custom benchmarks that will better estimate real-time performance of specific, in-development software for a particular target hardware system.

SUMMARY OF THE INVENTION

In at least one aspect, embodiments of the present system and method determine a unique performance benchmark for specific computer object code, as that code is actually running on a particular, designated hardware microprocessor.

Single Code Module Benchmarked on Multiple Processors: In an embodiment, the system and method may generate multiple unique benchmarks for a single, same object code module for each one of multiple processors, so that the system and method may determine which hardware processor (among the multiple potential processors or “target platforms”) is optimal for the code module.

Functionally Similar Software Modules: In another embodiment, the system and method may generate, for a single designated processor, multiple performance benchmarks, with a unique benchmark for each code module of multiple target software modules, with a goal of selecting just one of the target software modules for actual use and deployment. Each of the multiple software modules may have a same or similar functionality but have variations in detailed code or algorithms. In this way the system and method may identify those code variation(s) which is/are optimal for a particular designated target platform (for example, a particular designated hardware processor).

Functionally Different Software Modules: In another embodiment of the present system and method, multiple different target software modules have different functionalities, and the multiple modules are intended for combined (that is integrated) performance, possibly operating in parallel or in sequence, or both. Each target module may then have a very different code set. The present system and method may generate a first set of performance benchmarks for each target module when run on a first processor or first target hardware platform, then generate a second set of benchmarks for each target module when run on a second (different) target processor or different target hardware platform, etc. In this way, and guided by performance benchmark results for multiple different co-operative target modules, design engineers may select a target hardware processor which provides a desired balance of adequate performance times for most or all of the target software modules (and so provides optimum performance for the integrated software system of all the target modules).

Methods: In some embodiments, the system and method may entail a first stage for extracting selected features of object code into a code profile, and a second stage to generate the performance benchmark, based on the code profile and based on a choice of a particular designated test hardware processor. In this way, code security is achieved by fire-walling the object code from the second stage of the method. In an embodiment, the code profile reflects the distribution of microcode commands which are executed by the object code when run on a selected target processor under specific conditions.

In an embodiment, the system and method may also entail a second stage of generating a processor profile for a specific processor, and a second stage of jointly analyzing the code profile and the processor profile to generate the performance benchmark. In such an embodiment, the processor-specific performance benchmark for a particular object module may be generated either (i) without a requirement to actually run the object code on the specific processor (running the code instead on a different processor), or, in some embodiments, (ii) without a requirement to run the object code on any processor.

In some embodiments, the system and method is directed not only towards profiling the object code, but further towards identifying and isolating, for analysis, mainly those portions of the object code that are most typically or most frequently called into use as the target software module is actually being run on a processor in real-time.

In an embodiment, the system and method may also be used to generate a performance benchmark for a designated computer program or code module when the code module is run on a specific virtual processor or through an interpreter (such as for example Java).

Embodiments of the present system and method may be particularly applicable to benchmarking object code designated for use in real-time, performance critical applications. However, the present system and method may also offer advantages for benchmarking software performance for non-mission critical systems.

In some embodiments, the present system and method may entail the use of either a local or distributed computer profiling system, where the profiling system executes a method for benchmarking computer code, the method comprising:

(i) deconstructing a first object code for a first software module to create a first profile of machine-language instructions of the first object code for the first software module, which may be indicative of the machine-language instructions executed when the first object code module is run on a given target processor or target hardware system;

(ii) benchmarking a first target hardware system which comprises a first set of hardware to determine a real-time execution timing data machine-language instructions of the first target hardware system;

(iii) combining the first profile of the first software module with the timing data of the first target hardware system to generate a first performance benchmark, where the first performance benchmark is reflective of both the first object code of the first software module and the architecture of the first target hardware system.

In some embodiments, the deconstruction stage (stage (i)) may entail running the software modules on a processor which has an instruction set that is shared by other processors (for example, different processors in a manufacturers line of related processors, or different processors with a common hardware architecture, such as the well-known Intel instruction sets). In alternative embodiments, the deconstruction stage (stage (i)) may entail analyzing the object code to determine a distribution of machine-level commands, without actually running the object code on any processor.

In some embodiments of the present system and method, the stages or steps of generating the benchmark for a particular code module and a particular hardware processor may be performed by a single computer, and with the stages performed substantially consecutively in time (that is, with each successive analysis stage beginning substantially upon the completion of an immediately prior analysis stage). In alternative embodiments, stages of the analysis may be performed by separate computers of a distributed system, and may be performed with substantial time gaps (for example, minutes, hours, days or longer periods) between the analysis stages.

