Electrical noise causing random bit flips or function calls?

[evsource]

I'm hoping someone can shed some light on this problem.

I have several circuit boards with 16F88's being used to read battery voltages and control some hardware. A master unit polls each of these over a RS-485 line that is optically isolated from each of the PIC's. I use Modbus for the communications protocol. The whole system operates in a very noisy electrical environment, with 100's of amps flowing. Many filter caps of various uF ratings are present on the boards in an attempt to reduce some of the noise. All traces are very short, and large ground planes are used.

The system works, however it has a difficult to debug problem. Occasionally, one of the pieces of hardware that is controlled by a PIC, and a command from the master for that hardware to be operated, gets turned on without having been told so by the master. I've thought it might be all the random noise on the circuit boards eventually matching the bit pattern of the hardware control command, but after thinking about this, somewhat doubt that - it would have to match the Modbus function code, slave address, and hardware control code, not to mention getting all the CRC's right. I'm wondering if maybe the PIC is prone to jumping around randomly in the codespace in the presence of electrical noise. Is that even possible? Could it just randomly jump to a particular function without being called? Has anyone ever seen anything like this?

I know the most likely thing seems to be the Master erroneously sending the command, but multiple checks are in place so this doesn't happen (if two PIC's receive this same hardware controlling command, damage occurs in the system!). I've tried to think of ways to debug, to figure out where the spurious command is coming from on the PIC boards, but it happens so randomly, that I haven't been able to think of a good way to catch it, or figure out where the command came from.

ANY suggestions are welcome - even if you don't have a complete answer, just the ideas might help nail down the cause or some method to debug. I think I've been too close to everything to see a good solution!

[libor]

just two thoughts at first glance.

1. Do you have a metal box (EMI/RFI shielding) around your circuit ?
2. What about adding a "flight recorder" function: a device* logging all intentional activities of your circuit into a flash ram device or some other media. (possible could be connected to the an extra UART port that prints the debug information) ...so you can track down the cause, or at least exclude erroneous commands on the MODBUS as a cause. You can either log the MODBUS activity itself or log the debug port of your device (you should of course decorate your code with the debug printf() instructions in the strategic places)

*an old PC (notebook) will also do, wtih a simple serial port monitoring utility running permanently (you can download lots os free serial port logging utilities from the internet)

[Guest]

I think your initial guess to the problem is right. 100's of amps can cause volts to be induced on nearby conductors - this will indeed mess up most circuits. The other poster is also correct. Be very careful with grounds - ground is not ground in an environment like this.

Put the device in a metal box - use feedthrough capacitors in all I/O (with series resistance on the digital I/O pins to prevent ringing into the capacitors). Ground your circuit to the chassis only at the power inlet point. On the outside ground this power inlet point (negative terminal) very close to where you are getting the power for the input.

If the noise is common mode, you may need to put in common mode input filtering - like a switching power supply would have.

A good book is -

Grounding and Shielding: Circuits and Interference (Morrison, Ralph)

Analog devices and other manufacturers also have many free application notes on shielding and grounding.

HTH - Steve H.

[SherpaDoug]

A large solid ground plane may not be the best. Read up on "star ground" or "supernode". Think about how the current flows through the ground connections. You may need some slots in your ground planes. Sometimes you can do this to existing boards with an Xacto knife.
_________________
The search for better is endless. Instead simply find very good and get the job done.

[evsource]

[FvM]

You can monitor the amount of telegrams received with CRC error. This can give you a rough estimation of the likelihood to receive false correct telegrams. If the likelihood is too high, you can increase the hamming distance of the protocol. e. g. by including magic numbers with the telegram, that are checked in reception. Or introduction of timed sequences over the MODBUS to allow a critical action.

The monitoring of CRC errors could be communicated over the MODBUS. Unexpected operations of a processor can be partly detected and caught by software means and reported through the bus.