Sending blocks of data over RS-232

[edhaslam]

Hi all,

I'm currently designing a simple data logger that stores ADC values in external RAM. I intend to have a switch on a port that signifies the start of the downloading of data from memory over the hardware USART (PIC16F877A) to a PC where the data will be displayed in a VB program.

Can anyone offer any tips/sample code for transmitting blocks of data over the serial port. It's something I've never done before and wondered if there were any standard ways of doing it?

Thanks in advance,
Ed

[ckielstra]

The most common protocol that I can think of is xmodem. Lots of information and program examples can be found on the internet so I won't go into that any deeper here.

In case you want to transmit the data as multiple files and/or you want to include a file name, than have a look at the ymodem protocol.

[mcafzap]

For a really simple 'protocol' just send the data when it's requested by the receiving program. If you always send the same number of bytes, there's no need to indicate the end, just guard against waiting forever.

Steve

[edhaslam]

[Neutone]

I posted an example in the code libaray that sends packets of data. I would suggest that the PC start the transfer and forget about the switch. When the PC request a range of data the PIC responds with that range of data.

This is a specification for MODBUS protocol.
http://home.houston.rr.com/neutone/Modbus%20Application%20Protocol%20v1.pdf

This is an example of some code that meets some of that specification
http://home.houston.rr.com/neutone/serial.c

You dont really need a CRC on the packets but the master/slave aproach should work for you.

[edhaslam]

Many thanks Neutone

[Bill Boucher]

oops never mind

[Bill Boucher]

I have designed, built, and programmed many systems using RS-232 and RS-485 multi-drop networks of up to 1000+ devices all connected to a single PC serial port. In short, you get what you pay for, so if you design your communication method without any error detection schemes, then you run the risk of reading corrupted data from your logger. I like to use a master/slave system that uses structured packets. The PC is the master and the logger is the slave. I create a "command set" that both devices understand and "poll" the slave at intervals. For example, the PC can ask the logger if it has data, if it is full, if certain inputs are high or low, what is the voltage of a set of inputs, and so forth. Then the logger would answer that question specifically. Someone mentioned the usefullness of transferring "blocks" of data one at a time. The PC application would ask for page 1, then page 2, and so forth until all are transferred. Someone else mentioned "little need for CRC" in the packets. While the implementation of 16-bit CRC checks may seem like a lot of work at first, it actually isn't all that hard and it goes a very long way to ensuring that your master's commands are received correctly by the slave and that the data received from the slave is accurate.

In VB6, the mscomm control operates in "text" or "binary" mode. "Text" mode uses strings. The other mode uses arrays. I have found "binary" mode much more reliable so I always have my programs converting strings to an array before Tx'ing and converting received arrays back into strings upon reception. Yes, it's a pain but runs flawlessly. Moving on, I use a structured packet for all communications. I use a "unique" value above 127 (of the 256 possible states from 0 to 255 for 8-bit data) to represent a packet begin marker (PB) and another for a packet end marker (PE). All other binary data is converted into ascii text "hex" characters. Binary data includes the "command value", any required address data, any attached data, and the 16 bit CRC checksum. Converting binary data into ascii-hex characters doubles the number of bytes required to send the data, but by using unique packet markers and a 16-bit CRC value, you can adjust the packet length to the optimum for each specific command in the set and you can rest assured that packets are "intact" before processing them.

Be aware that you can send any binary byte in "text" mode using the mscomm control in VB. Practically everyone does this for their first few projects. You just have to set "null discard" to false or the control will drop binary zero. You also have to set "parity replace" to nothing. The fact that the "mode" is called "text" means nothing really since it doesn't restrict you to the ascii set of values or anything like that. It more refers to the way input and output data is handled as strings or "text".

Here's a useful tip... make your packets short, like no more than the PC's UART hardware FIFO size. The 16550 has a 16 byte FIFO. The 16650 and 16750 have 32 byte FIFO's. If your packets are longer than the FIFO size, you run the risk of overflowing the input buffer before Windows has time to come along and empty. Setting the "receive buffer size" in VB to something like 1024 doesn't help. We're talking hardware here. It might help to actually set the size to something really small like 4 or 8 bytes because then Windows will become alerted before the buffer is actually full. I still wouldn't count on Windows being fast enough to clear the input buffer in all cases. If Windows doesn't empty the buffer in time, bytes coming in after the buffer is filled are lost. This results in lost bytes or one might say "a dropped packet". I simply use a 32-byte hardware FIFO and I make my packets no longer than 32 bytes. This way, the master clears the input FIFO, then asks for something by sending a command and the input FIFO has enough space to hold the entire response regardless of the Windows situation. Systems I've created in this manner run flawlessly and virtually never drop a packet. The CRC checks done on all packets ensure reliable execution of commands and data. Using unique values for packet markers means that data cannot be misinterpreted as a marker and vice-versa. Checking for correct structure such as packet length, marker values, valid values for commands and data (i.e. values in the command position will have a known range, and all ascii-hex characters must be '0' through '9' and 'A' through 'F'), etc. all combine to boost reliability to nearly 100% over practically any length of cable and without regard to timing issues imposed by the OS.

You should start with something very simple such as using the mscomm controls "text" mode and sending binary data literally over a short connection. Creat a very simple command set such as specific values representing "say hi" and "send data". Once that's working well, add code to produce unique packet markers and convert the data to ascii-hex. When that's working, add error checks such as CRC16. Just take it one step at a time. Later, you can copy & paste your entire comm system from one project to another so the more you put into it, the more benefit you'll get out of it in the long run.

For my systems, a packet might be structured to look like this:
PB, Command1, Command0, Data3, Data2, Data1, Data0, CRC3, CRC2, CRC1, CRC0, PE

Where PB = A0h, PE = A9h, and Command1='0' and Command0='5' and when combined represent 05h, and Data4 through Data0 characters '0123' represent a 16-bit value 0123h or two 8-bit values 01h and 23h, and CRC4 through CRC16 characters combine to represent the 16-bit CRC checksum value for the entire packet (not including the 4 CRC16 character bytes).

Such packets can be of a specific length for each unique command defined in your command set but they do not have to be because the CRC16 is a very reliable way to check it and the CRC16 calculation works on any length of packet greater than a few bytes.

Just for interest, many DMM's and other existing devices such as plotters, scopes, etc. use "structured packets" where binary data is first converted to ascii-hex characters for transmission. Some devices add CRC checks, some don't. Some simply duplicate critical data by sending it more than once within the same packet or by asking for the same data to be returned more than once and comparing them. The CRC16 method is still way better and saves a lot of coding in the long run and it verifies all data, not just some of it.

Someone suggested eliminating the "hardware switch or button" used to upload the data. I agree. Although it is doable, it's not required. Your PC application can poll the logger at intervals and purge it of its data automatically if it gets too full. Otherwise, you'll likely be sitting at the PC and mouse-clicking the "upload data" button in which case the PC master sends a command or series of commands to retrieve the logger data. It is unlikely that it would be useful to set up the PC app to be ready to receive first and then go hit the button or switch on the logger to get it to go ahead. It is possible like I said, but I think you'd find it impractical since you can control everything the slave device does from the PC master application itself.

[edhaslam]

Many thanks for that very useful post Bill - it gives me something to aim for now. You are right though, the best thing to do is to start of with some simple examples and work my way up from there.

Regards,
Ed

[edhaslam]

Edit: Double post

[ratgod]

I designed a protocol for my company which each packet is a fixed length, I have a start byte at the beginning followed by an address byte then the 3rd byte is a separator followed by a fixed length of data.
The packet is sent out using the printf command sending each byte as a character.

The receiving section of the code is the RS232 interrupt routine, it checks bytes into the buffer, but if it detects the start byte it resets the buffer back to the beginning and starts filling it with the new data.

when the buffer is full the 2nd part of the program checks that the start byte and the separator is in the correct place, if thats ok then it passes on to be processed.

This method has worked very successfully for us so far and is used in a few of our products.

[arunb]

edhaslam..Does your datalogger continue logging data even while it is downloading data to the PC ???

arunb

[Ken Johnson]

Bill's post is excellent. However, personally, I'd make the packet size fit the application, not some (likely unknown) hardware fifo size. "Dropped packets" is certainly a possibility, but I think very low probability - depends on so many things - pc horsepower, other running apps, windows update etc. But, I'd go for simplicity, meaning, put what is needed into the packet.

You have some very good guidelines, now go for it!

Good luck,
Ken

[SET]

Hi

Just a comment on Bill's post - we have succesfully captured serial data at 921k Baud (with large packets) although admittedly over USB with the PC acting as a virtual comm port. And we also regularly capture 115kBaud over a 'real' COM port. Windows comms drivers should handle the buffering, and with a modern PC it's doubtful a PIC can outrun a PC!

One thing we did notice - our mega-baud system's PC software was initially done in C#. and in that case 500kBaud was the limit, when we changed to Delphi (and real compiled code) we could use 921k without issue. So maybe VB would be limited to 500k also?

[libor]