Scott Craig and Justin Tanner

Test Cases

To test our RTOS we wrote 18 different test cases, broken into 3 major categories:

  1. API Correctness
  2. Performance
  3. Invalid Operations

Each test contains one source code file and a AVR Studio project file. All these test are designed to probe different sections of the RTOS for possible errors or incorrect behavior.

API Correctness Tests

API correctness test are used to verify the behavior of all the basic constructs of the API. Each function call is tested both individually and in concert with other functions.

Test 001 - Sanity Test

Objective: Can we print a basic trace with UART?

Description: Simple test to ensure that the testing software can run on the RTOS.

Code: test001_sanity.c

Expected Result: An empty trace.

T001;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0;0; ...

Comment: No problems testing code works as expected.

Test 002 - Periodic

Objective: Can the RTOS schedule PPP tasks in the correct order?

Description: Creating six PERIODIC tests in the order 1,2,4,5,6 to test the function Task_Create() and basic PERIODIC behavior.

Code: test002_periodic.c

Expected Result: 1,2,3,4,5,6 repeated.

T002;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6; ...

Comment: PERIODIC tasks are scheduled in the correct order.

Test 003 - Round Robin

Objective: Can the RTOS schedule RR tasks in the expected order?

Description: Similar to the last test case but using RR ( round robin ) tasks instead of PERIODIC.

Code: test003_roundrobin.c

Expected Result: 1,2,3,4,5,6 repeated.

T003;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6; ...

Comment: RR tasks are scheduled in the correct order.

Test 004 - SYSTEM Tasks

Objective: Can the RTOS schedule SYSTEM tasks in the expected order?

Description: Similar to the last test case but using SYSTEM tasks.

Code: test004_system.c

Expected Result: 1,2,3,4,5,6 repeated.

T004;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6;1;2;3;4;5;6; ...

Comment: SYSTEM tasks are scheduled in the correct order.

Test 005 - Interleaving Tasks

Objective: Can the RTOS interleave RR and PERIODIC tasks?

Description: Given PERIODIC and RR tasks running together, can the RTOS run the PERIODIC tasks when expected, and fill-in the blanks with the RR tasks.

Code: test005_interleave.c

Expected Result: 1 followed by many 7's then 2, repeat

T005;

1;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;
7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;

2;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;
7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;

1;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;
7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;

2;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;
7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;7;

...

Comment: After the first interval between 1 and 2, the amount of round robin tasks performed ( the number of 7's ) is the same between task 1 and task 2.

Test 006 - Event Wait

Objective: Can the RTOS wait on an event?

Description: Testing the behavior of the Event_Signal() and Signal_And_Next() functions. Does Signal_And_Next() actually signal a waiting event?

Code: test006_eventwait.c

Expected Result: 1, 3, 2 repeated

T006;1;3;2;1;3;2;1;3;2;1;3;2;1;3;2;1;3;2;1;3;2;1;3;2;1;3;2;1;3;2; ...

Comment: No problems.

Test 007 - Broadcast

Objective: Test if multiple RR tasks can wait and be signaled by a broadcast.

Description: Similar to the last test, but now testing Broadcast_And_Next() on multiple RR's waiting on the same event.

Code: test007_broadcast.c

Expected Result: 10, 20 ( create RR's ) then 3 ( broadcast ), then 11, 21 ( released from wait ), repeated

T007;10;20;3;11;21;10;20;3;11;21;10;20;3;11;21;10;20;3;11;21; ...

Comment: No problems.

Test 008 - Signal First

Objective: Multiple RR tasks wait on an event, see which one gets signaled first

Description: PERIODIC task is signaling, make sure that the waiting events get signaled in the correct order.

Code: test008_signalfirst.c

Expected Result:10, 20 ( initially wait in both RR tasks ), followed by 3 ( signal ), then 11, 10 ( release and wait again ), 3 (signal), 21, 20 ( release and wait again ), repeated. 

T008;

10;20;

3;11;10;3;21;20;3;11;10;3;21;20;3;11;10;3;21;20;3;11;10;3;21;20; ...

Comment: No problems.

Test 009 - Ping Pong

Objective: Multiple RR tasks wait on an event, see which one gets signaled first

Description: Two events with two RR tasks pinging and ponging back and forth.

Code: test009_pingpong.c

Expected Result:10, 20 ( initial wait in both ping and pong ), followed by 11,10 ( signal pong and wait in ping ), 21,20 ( signal ping and wait in pong ), repeated.

T009;

10;20;

11;10;21;20;11;10;21;20;11;10;21;20;11;10;21;20;11;10;21;20; ...

Comment: No problems.

Test 010 - Multitest

Objective: Many things at once, events, RR tasks and Periodic tasks.

Description: Trying to find problems that may crop up in more complicated scenarios where most of the API is used.

Code: test010_multitest.c

Expected Result: See code for details.

T010;

2;10;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;

3;11;201;10;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;
201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;201;

2;201;202;11;201;202;10;201;202;201;202;201;202;201;202;201;202;201;
202;201;202;201;202;201;202;201;202;201;202;201;202;201;202;201;202;
201;202;201;202;201;202;201;202;201;202;201;202;201;202;201;202;201;
202;201;

...

Comment: No problems.

Performance

The following test measured preformance on the RTOS.

Test 011 - Round Robin Latency

Objective: Measure the latency between RR tasks

Description: The full time between running a RR task, context switching and running the next RR task. Timing clock is set at 8Mhz.

Code: test011_latency_roundrobin.c

Expected Result: A short time, much less an a tick.

histogram

Comment: On average the execution time was 0.07842ms which is less than one tick.

Test 012 - Periodic Latency

Objective: Measure the execution time of a PERIODIC task.

Description: PERIODIC tasks with a 1 tick duration were measured from the start of execution, to the start of execution of the next task. Timing clock is set at 1Mhz.

Code: test012_latency_periodic.c

Expected Result: One tick exactly.

histogram

Comment: With the exception of a few measurements, the test reads a consistent 5ms exactly 1 tick.

Test 013 - Event_Signal Latency

Objective: Measure the latency of event signaling from a PERIODIC task to a RR.

Description: Time between signaling in a PERIODIC task to release from Event_Wait in a RR task. Timing clock is set at 8Mhz.

Code: test013_latency_eventsignal.c

Expected Result: Less than a tick.

histogram

Comment: Every measurement was exactly the same 0.0594ms. This was the quickest operation tested yet.

Test 014 - Task_Create Latency

Objective: Measure how long it takes to create a task

Description: Creating 3 PERIODIC tasks and 3 RR tasks, measure how long it takes to create. Clock is running at 8Mhz.

Code: test014_latency_task_create.c

Expected Result: Less that a tick for both RR and PERIODIC.

    T014;    
    628;624;699; ( RR )
    630;633;708; ( PERIODIC )

Comment: On average it took about 654 cycles @8Mhz, about 0.0817ms, much less than a tick.

Test 015 - Overloaded PERIODIC

Objective: Measure the latency of periodic tasks

Description: Maxing out what the RTOS is doing between tasks to try to overload it while it attempts to run PERIODIC tasks ontime. Timing clock is set at 8Mhz.

Code: test015_latency_maxperiodic.c

Expected Result: One tick exactly.

histogram

Comment: On average 10ms or 2 ticks! This is way off, potential bug.

Test 016 - Interrupt Latency

Objective: Measure the latency of events signaled from an interrupt.

Description: Timer starts inside an interrupt and measures the time from signal to when the RR picks up after Event_Wait(). Timing clock is set at 1Mhz.

Code: test016_latency_interrupt.c

Expected Result: Much less than a tick.

histogram

Comment: ERROR in the test, constantly reading 65ms, but this is an incorrect result, there must be a fundamental problem in the RTOS or test.

Invalid Operations

The following tests gauge how the RTOS behaves in the presence of a error.

Test 017 - Too many PERIODIC tests

Objective: Creating too many PERIODIC tests and see what happens.

Description: Creating nine PERIODIC tasks to overflow the max events by one.

Code: test017_toomany_periodic.c

Expected Result: An error code.

    ALL_LIGHTS RED RED

Comment: Looking up this error code in error_code.h revealed that ERR_RUN_2_TOO_MANY_TASKS had executed correctly.

Test 018 - Wait in a PERIODIC

Objective: Event_Wait() in a PERIODIC task and see what happens.

Code: test018_periodic_eventwait.c

Expected Result: An error code.

    ALL_LIGHTS RED RED RED RED RED RED RED

Comment: Looking up this error code in error_code.h revealed that ERR_RUN_2_TOO_MANY_TASKS had executed correctly.



For reference, Raw Test Data.