Help with parsing a data stream ... strtok?

[evsource]

I'm having a hard time coming up with a fast, reliable method to parse a data stream. It is coming in at 19.2K on the RS-232 line, using the UART, with interrupts when a byte comes in.

The data stream is variable. Here is the basic composition of the string:

B,127,148,134,H,0,0,3,E,0,0,0,0,0,0,0,P,1274,88,67,0,
(break inserted here for readability only)
B,122,144,123,H,0,0,3,E,0,1,0,0,0,0,0,P,1124,85,69,0,
T,23,5,15,1,50,
CR LF

Up to four of the lines starting with "B" can come in. The quantity of values after the "B" up to the "H" can vary from just a single value up to 6. The "H" section is really of no value, the "E" section has a fixed number of values following it up to the "P" (i.e. 7 values), and the "T" section comes after all the "B" sections arrive.

I need to convert the values to INTs and LONGs, e.g. the 127 would be stored as 0x7F.

I'm thinking using strtok would simplify, but I'm a bit confused on its usage. The other problem is my current attempt at parsing the data seems too slow, with the data streaming in at 19.2K.

I'm sure someone will say "show us your code", but I'm really hoping for a fresh perspective on how to re-write my parsing code.

Oh, all data will be stored away in arrays.

Thanks!

[dyeatman]

EVSource,

A few questions:
1. Which CPU are you using?
2. What is the CPU clock speed?
3. How much time is there between CRLF and the starting B of the next packet?

[Ken Johnson]

Not knowing the details of your application, I'd just stuff the incoming data stream into a buffer in an ISR, set a flag telling the main processing loop there's something there. Assuming there's time between bursts of data, you can then process it "leisurely".

Never used strtok, so I'm no help there.

Ken

[evsource]

[dyeatman]

You should be running the oscillator at 4xPLL to get some headroom to
work in during the data parsing.

In my applications I have two queues. I put the incoming chars in a
circular "incoming" queue at least 1.5 the size of the largest possible
incoming data packet then scan that queue in the background process.

In the first background process, when I see the "key" char (in this case
the B) I set a "key char" flag and move every character after that one, at
a time, to a processing queue until I see the termination char (in this case
the LF) at which point I set a complete packet flag. The key char flag is
reset to wait for the next incoming key char to appear.

The complete packet flag signals another background process to start at
the first location in the process queue scanning the string for each of the
delimiter characters (use a switch statement for this) and take the
appropriate action.

Notice in all this that I process each character one at a time rather than
groups. This is the safest way to avoid problems when interrupts are being
used.

I have used this process in more than 30 applications where incoming data
packets must be scanned and processed. All of them work perfectly at
even higher baud rates than you are trying to use. One note, I run all my
processors at the maximum possible speed (40MHZ or greater) to provide
plenty of processing time for the background processes.

Someone here may have other ways to approach the problem but this
works for me.

Here is another thread where I talk about how my processing is handled
that may be useful.

http://www.ccsinfo.com/forum/viewtopic.php?t=21845&highlight=queue

Dave

[Ken Johnson]

Dave's description is excellent - that's also what I do, at 115200 baud with no problems. I wanted to try to suggest this, but couldn't find the wording. Good job Dave !

Ken