sending large data over tcpip

pingpong · Joined: 26 Nov 2012 Posts: 8

I am developing an application where the PIC 18F67J60 is used. I am using the 13b example for tcp server modified it to suit my requirements and have a pc application sending data to the PIC over TCPIP.
The data sent from the PC is a binary data and is as high as 10KB or more.

I read on internet that for large data... it needs to be split and then sent ( I had expected the tcpip to actually take care of that). So I split the data in 500 bytes packets and then send one packet after another. For a few packets the data transfer is good but after that the data transfer is not happening properly. I get garbled data.

I tried to study the transfer with wireshark and found that the data is getting split and once that happens, i don't get the data properly.

I saw in the stack that there are some functions written to take care of such data splitting but they don't seem to work.

I need to understand where I might be going wrong. Can someone help me.

Thanks in advance.

pingpong · Joined: 26 Nov 2012 Posts: 8

my dear friend dyeatman... it would have been better if you had given some solution instead of just locking a thread.
Both the topics are different though they refer to the same program done by me.... but I wil remember your suggestion.

At present, i am using the tcpget in a loop to get bytes of data from the receive buffer and putting them in another buffer for further processing.
I tried to do that with tcpgetarray... which seemed a more logical way of getting data.. but found that tcpgetarray was not returning the same data that I get with tcpget.
The number of bytes i have to read are fixed 500. When i try to use the tcpgetarray, i have a lot of bytes added before the data... which i have never sent.
Is there any problem reported for tcpgetarray? OR are there some preconditions for reading the data with tcpgetarray that are different that tcpget.

Look for some advice on this.

Thanks in advance.

asmallri · Joined: 12 Aug 2004 Posts: 1635 Location: Perth, Australia

I do not know the specifics for the TCP/IP stack you are using but the following is generic considerations for using TCP/IP on PIC microcontrollers with embedded Ethernet controllers or PICs using Microchip Ethernet controllers.

TCP is a stream based service, unlike UDP where you send a packet at a time, with TCP the stack receives "enough" data to fill a payload or has been waiting to send data in a buffer for some pre-defined period. The stack will have an MTU defined (maximum transmission unit) which basically means the maximum size, including all headers, of a packet. With TCP, the stack must retain a copy of any transmitted data in a retransmit buffer until the data has been acknowledged. The acknowledged data is then removed from the retransmit buffer. If the data is not acknowledged, for example a packet is lost, a timer on the transmit side will "pop" and the stack will retransmit the unacknowledged data in the retransmit buffer.

PICs have relatively limited amounts of RAM available for maintaining large retransmission buffers and large frame sizes. The Ethernet controller has some buffer memory available, not enough to be really useful but enough for most stack applications. The stack maintains a maximum window size which is used to indicate how much free space the receiver has available to receive data from the transmitter. This mechanism is used by both ends of a TCP connection.

So enough preamble, we know that the PIC and Ethernet controller have limited RAM resources. It is up to the application to ensure the stack has sufficient resources available. When writing to a TCP connection it is important to determine how much free space the stack has available for the transmit data (data to be sent). If attempt to write more to the connection (socket) than there is free space available then either the stack will only accept up to the number of available bytes and return to you the number of bytes actually written (this is how it should work) or it will overrun the transmit buffer.

If you application does not process the bytes actually written value assuming the stack has processed the entire buffer sent to it by the application then the result will be that the transmitted packet will appear corrupted because there will be data missing. The symptom of overrunning the transmit buffer in the Ethernet controller is far worse. In this case the transmit logic of the Ethernet controller will get confused and garbage traffic will be sent.

Finally, when configuring the IP stack, you carve up the Ethernet controllers memory into receive and transmit buffers. The transmit buffer may also contain the retransmit buffer. If you configure the Ethernet buffer pointers incorrectly then it is possible to corrupt the transmit and/or receive buffer contents in the Ethernet controller.
_________________
Regards, Andrew

http://www.brushelectronics.com/software
Home of Ethernet, SD card and Encrypted Serial Bootloaders for PICs!!

pingpong · Joined: 26 Nov 2012 Posts: 8

Andrew

thanks for such a detail reply.

I am using the ccs stack and my application is a server based on the 13b example provided in the stack.

How to ensure that the stack has sufficient resources? Is the transmit and recieve buffer size the only resource or any other thins that needs to be managed?
the packet size i am using is 500 bytes while the buffer values are kept at 1024. i will try keeping the buffers to maximum of 1500. can we increase the buffer more than that?

can we configure the ethernet buffer pointers? if yes then how? what is the effect of configuring these pointers?

And one thing i am unable to understand is that the application runs OK when the total packets sent are small.... the application gives problem when the the total packets are large.... which means a continuous 30-40 minutes exchange of data.

dpechman · Joined: 04 Dec 2007 Posts: 43

I am parsing some tcp data using a tcpget loop and it works fine with small amount of data.

now I have to parse a 6Kb file and its stucking the tcp for some reason.

As my target data is fix in the first 1k of data, I was thinking in get my data and discart the rest of the file setting a flag when what I am looking for is found:

benoitstjean · Posted: Fri Jan 20, 2017 6:02 am

I haven't used nor do I know the PIC device you are using...

I haven't used pre-built TCP/IP libraries either...

Is it the PIC itself doing the TCP stuff? Or is it something like a modem or a wifi module?

In my case, I've been working on continuous IP data transmission for a few years and what I am using is a DMA buffer tied to the UART's TX buffer. The UART on the PIC is tied to the modem's UART. The port speed is set to 115200bps.

I am using a PIC24EP512GP806 because it has a crap-load of program space and ROM.

Then I have a 40-deep output queue which consists of 133-byte characters per buffer: