What can corrupt my Flash program code ?

[nurquhar]

I have 84 PIC16f876A-E/ML (28pin QFN) circuits running doing an LED lighting app, all the circuits are on a 485 bus. I have been soak testing the circuits which involves power them all up, run for two mins, then power off for 2 mins. I have been running them for a few days like this. When running I have a host system that sends commands over the 485 to tell the LEDs to change brightness etc.

Some of the units are starting to fail in that they no longer perform as expected. During some earlier testing I found on reading the program back with my ICD2 I found that some flash program locations had randomly changed !!! I reprogrammed them and they were fine again, ie no hardware failure evident.

I suspect this going on again now with my intermitant failures. I currently have 5 of the 84 down. Since the circuits are all dipped in silicone and wrapped in heatshrink its not easy to get to the ICD connection pads on the back of the PCBs.

Any Ideas what could be the cause ?

All code is built with CCS 3.249.

My desparate guesses are :
1) Parts have been lead free soldered (They are RoSH parts), could the higher solder temps damgae flash ?
2) The BOR is not enabled, on power down could it execute corrupt instructions which reprograms itself. Ie a jump into the middle of the bootloader code.

[PCM programmer]

[nurquhar]

I have NOLVP programming enabled. From the lst file I have :

Configuration Fuses:
Word 1: 3F32 HS NOWDT PUT NODEBUG NOPROTECT NOBROWNOUT NOLVP NOCPD NOWRT

[PCM programmer]

You will probably get better answers on Microchip forum.

You can use http://www.google.com to search the archives before you
post a question on there. Use this exact search string:

site:forum.microchip.com flash corrupt OR corrupted rom OR memory


You will get two hits at the top of the list.

[Ttelmah]

There have been cases of 'batch' faults in the past, but these are fairly rare. Some questions apply though:
How hot are the chips running?. You have them potted, what else is nearby?. Have you done any thermal testing to see what sort of temperatures the chips are likely to reach in use?.
Are you writing to the Flash memory anywhere in your code?.
The commonest failure, is caused by supply transients (this is I think what PCM is pointing at).
What process is actually being used to solder the chips?. (It is the 'product', of the temperature, and the times involved that really matters, in terms of internal problems, and the profile of the thermal curve, that matters for physical damage). Microchip, do have a number of spec sheets, for the temperature, and duration 'allowable' during soldering, and this should be provided to the assembly company for comment. There is an area in the 'complete' mid-range reference manual, and a separate application note (AN598)

Best Wishes

[rnielsen]

When the power turns on, or off, are you getting spiking on the power lines? This could cause almost any part to turn into a dead bug.

Ronald

[nurquhar]

The PCBs with the PIC QFN parts have been dipped in clear Silicone conformal coating. As each circuit is only 250 x 12mm they are then fitted inside some clear heatshrink for extra protection.

Each circuit has 10 SMT leds on it but these are not driven by the PIC directly, they are driven from two MAX6967's. Thus the current draw from the PIC is thus minimal, it just has to drive the SPI signals to the LED driver via some 5->3.3v level shifters and a Sipex 485REN for the 485 half duplex bus from the uart. Having measured with me infa red thermometer the chips are running at a pretty cool 25c. Well within spec.

5v for the circuits comes from some Trac 6A POL DC/DC converters, one converter drives 4 of the LED circuits via some 1mm cable. All 21 Trac POL's are powered from a 12v 300W Lamba Switch Mode PSU via three sets of 2.5mm cables. The PSU is switched from standby to on via its control input, no nosiy mech relays etc.

Its hard for me to say what happended in the reflow oven when the PCB's were assembled as the assebley of 90 circuits was subcontracted. However they did have problems with the Lead free soldering of the QFN's and switched to using a vapour phase process. All the same I did end up with every QFN under the microscope and touched up by hand with 0.2mm tip iron every connection that looked poor.

I have ordered some 5.6v 0603 Varistors in the hope I will be able to "pigy back" them onto one of the 0603 0.1uF decoupling caps near the PIC.

The only place I have any program flash write code is in my bootstrap loader. When I first came across this issue this was my first and only point of suspicsion. I thus embarked on a belt and braces gaurd of this code. Thus now to change a program location the host PC has to send a "wake up" signal on the 485 within the first 4s of reset to put the bootstrap active, other wise the app will start if present. The host must then send a 10 byte sequence with a 32bit ID number to request comunication with the required circuit. (Each PIC has a unique 32b serial number in ROM). The host must then send a "write enable" command to set an internal flag. Without this flag set to the correct value the procedure with the program write will return. The host must then send a small block of data with a valid checksum. The host must then send a write command to write the previous block of data. If the preceding command was not a block send with a valid checksum the write command is ignored. So its a prety torturous sequence of data the bootstrap prog must get inorder to change a location. And just to be sure the writeing procedure only allows changes to locations not with in the bootstrap code. Since the bootstrap can be instructed to read back the memory contents the host is able to verify correct programming. The boostrap seems to work fine and I have been able to program 40 pic's on same 485 bus in one session no problem.

What are your thoughts on BOR, ie I don't have this set at the moment and in most previous projects have not borthered either. When the voltage drops during power down the PIC may well keep runninng until the voltage drops well below 4v. Could this cause weird stuff to happen during this short time ?

[rnielsen]

So, let me see if I understand this correctly. You have a switching power supply powering 21 DC/DC converters.

First, I don't know if I would have a switcher powering a DC/DC.

Next, having 21 DC/DC's hooked up to the same source might cause some interesting effects. When a DC/DC's input starts to fall the converter will do whatever it can to keep it's output steady. So, as the voltage drops the current it trys to draw will go up. Now, if you have all 21 of these babys doing this you can have one dog-fight going on with all of them fighting for input current.

I've seen this sort of thing happen on an application we had come in from a vendor. The supply was powering several DC/DC's and it was fused with one fuse. It would power up and run just fine until it was powered down. The DC/DC's would try to keep their outputs steady and the fuse would pop almost every time it was powered down. The vendor couldn't understand why this was happening until I pointed this out to them.

I can't say that this is what's going on with your circuit but it's a thought.

Ronald

[nurquhar]

The 300w PSU powering all 21 of the point of load switched regulators is capable of supply over 25A in total, with all the system circuits up and running they draw about 2A in total form the PSU. I am well over rated on the supply I think. Surely once a switching reg goes below it min input voltage it shuts down. Are you saying it will try to convert an infintely small voltage at an infintly large current to achive its rated output voltage ?

POL Converter : http://uk.farnell.com/jsp/Electrical/Power+Supplies+&+Converters/TRACOPOWER/TOS+06-12SM/displayProduct.jsp?sku=1007586

Switch Mode PSU : http://rswww.com/cgi-bin/bv/rswww/searchBrowseAction.do?obs=sObs&name=SiteStandard&No=0&N=0&Ntk=I18NAll&Ntt=3381706%20&Nty=1&D=3381706%20&Ntx=mode%20matchpartial&Dx=mode%20matchpartial&callingPage=/jsp/homePage/homePage.jsp&BV_SessionID=@@@@2082883275.1178211170@@@@&BV_EngineID=ccdiaddkleifkilcefeceeldgondhgk.0&cacheID=ukie&Nr=avl:uk

[nurquhar]

I have a photo of the scope with 5v rail when the power goes off. See this thread on the Microchip forum.

http://forum.microchip.com/tm.aspx?m=251820