Modifying the EtherCard Library To Handle Large TCP Packets
I tried both the UIPEthernet and EtherCard libraries for my enc28j60 with atmega328. The UIPEthernet library is implemented so that it can be used directly instead of the official ethernet library of Arduino. But it requires more space than the official library. It was also slow while sending data and it caused my atmega328 to freeze frequently for no reason. So, after many tries I gave up and kept searching for another library. EtherCard was the second library I found for enc28j60. It is faster and more stable than the other one. I am not sure if it supports all the methods implemented in UIPEthernet, but methods for receiving and sending small TCP packets worked well. I think its implementation is simpler and you don't need a long time to understand and be able to modify it.
What Exactly Is The Problem?
There is already a function that can be used to send data in multiple packets, in the library. But I couldn't make my program able to receive fragmented large TCP packets which can't fit in a small sized buffer.
Let me give an example:
#define BUFFER_SIZE 150
#include <EtherCard.h>
byte Ethernet::buffer[BUFFER_SIZE];
void setup()
{
Serial.begin(9600);
while(!Serial);
byte mac[6]={170,170,170,170,170,170};
byte ip[4]={192,168,0,253};
ether.begin(sizeof Ethernet::buffer,mac,10);
ether.staticSetup(ip);
}
const char page[] PROGMEM =
"HTTP/1.0 200 OK\r\n"
"Content-Type: text/html\r\n\r\n"
"<html>"
"<head><title>TEST</title></head>"
"<body>"
"<h3>TEST</h3>"
"<p>This is a page larger than your buffer size</p>"
"<pre>"
"01234567890123456789012345678901234567890123456789\r\n"
"01234567890123456789012345678901234567890123456789\r\n"
"01234567890123456789012345678901234567890123456789"
"</pre>"
"</body>"
"</html>";
void loop()
{
uint16_t payloadPos = ether.packetLoop(ether.packetReceive());
if (payloadPos)
{
char* incomingData = (char *) Ethernet::buffer + payloadPos;
Serial.println("INCOMING TCP PACKET");
Serial.println("-------------------");
Serial.println(incomingData);
Serial.println("-------------------");
int sz=BUFFER_SIZE-payloadPos; // max payload size (TCP header is also kept in the buffer)
if(sizeof(page)<sz)
sz=sizeof(page);
memcpy_P(Ethernet::buffer + payloadPos, page, sz); // Copy data from flash to RAM
ether.httpServerReply(sz-1);
}
}
This is an example that prints everything it receives to the serial port and send a response which can not completely fit in the buffer.
The GET request I sent to test what will be written to the serial port:
GET /?largeGETVariable=0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789 HTTP/1.1 Host: 192.168.0.253 User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3 Accept-Encoding: gzip, deflate Connection: keep-alive Cache-Control: max-age=0
The data printed to the serial port:
INCOMING TCP PACKET ------------------- GET /?largeGETVariable=012345678901234567890123456789012345678901234567890123456789012345678901 -------------------
The data transmitted to the client:
As it can be seen, the program cannot handle whole the packet and send all bytes of the page because of the size of the buffer. In fact, it is possible to send data in multiple packets using this library as I mentioned before and so the client can receive all the data. But we have to break the data into pieces that can fit into the buffer. So we still need a function which does that at background. You can think that increasing the buffer would be a solution but I especially used a small sized buffer because I were using nearly all the RAM in my project. (Or maybe I could have been used other atmel chips which have smaller RAMs.) On the other hand, the request was larger than what can fit in a single TCP packet. So, we also need a method to handle packet fragmentations.
In that case we have three questions:
- How can we handle every byte of an incoming packet without truncation?
- How can we determine whether a packet is a continuation of the previous one?
- And how can we send large responses which can not fit into buffer?
Note that, we have two buffers, the buffer which is in enc28j60 and the buffer which we allocate manually in atmega328. The first buffer is large enough to keep a large TCP packet. But the other one may not be always so large, for example if your program consumes large amount of RAM for some other tasks.
1. Handling Multiple Large Packets
- When a packet is received send an ACK, read its size and copy a part that can fit into the buffer.
- Walk the buffer to handle every byte and count each byte you handled.
- When you reach the end of the buffer, compare the number of handled bytes with the packet size which you read in the first step.
- If the number of handled bytes is smaller, then the packet hasn't been handled entirely and there are still unread bytes. Read the next part that can fit into the buffer.
- Repeat 2, 3 and 4 until the number of handled bytes became equal to the size of the TCP packet.
- Check flags of the last received packet. If PUSH is set, it means this is the last packet that the client will send until it receives a response (take a look at the screenshots below). Go to step 7. Otherwise, send an ACK and check if the next packet (which is the continuation of the packet we are currently processing) has arrived.
- If the next packet has arrived go to step 1 or else you can assume that the client is waiting for a response. Send the response and then send a packet with FIN flag to finalize the connection.
2. Determining Continuation Packets
The screenshots on the right were taken from wireshark while the request was being transmitted to test the example. The GET request was being transmitted in two packets because size of the data exceeded the maximum packet size limit which is typically 1500 (MTU). If a packet is the last continuation, then it is being sent with PUSH flag, as I observed. So this is the answer for the second question.
3. Sending Large Packets
- If the data can fit into the buffer, copy it to the buffer (starting from right after the TCP header) and do not send it until the buffer is filled up completely. If it can not fit into the buffer, copy a piece that can fit into it and send the buffer with ACK flag. Repeat this step for all your data that you want to send.
- If there is unsent data in the buffer, send it with both ACK and PUSH flags and then send a packet with FIN flag, to finalize the connection.
Implementations
Here are the implementations of those steps. Copy these codes and paste them into enc28j60.cpp right above the initialize() function.
//--------------------
// July 2014
// Huseyin Atasoy
//---------------------------------------------------------------------------
#include "EtherCard.h" // For ether.httpServerReply_with_flags() function
#include "net.h" // For TCP_FLAGS_FIN_V and TCP_FLAGS_ACK_V flags
byte* tcpPayloadPos;
byte* endOfBuffer;
int tcpMaxPayload;
byte lastReceivedPacketFlag;
int ENC28J60::packetPayloadSize;
byte* bufferPtr;
int totalBytesCopied;
byte ENC28J60::readByte()
{
if(bufferPtr==endOfBuffer)
{
int remaining=packetPayloadSize;
if(remaining>0)
{
if(remaining>tcpMaxPayload) remaining=tcpMaxPayload;
bufferPtr = tcpPayloadPos;
readBuf(remaining,bufferPtr);
}
}
packetPayloadSize--;
byte b=*(bufferPtr++);
if(packetPayloadSize==0)
{
if((lastReceivedPacketFlag & TCP_FLAGS_PUSH_V)==0)
{
uint16_t tcpPayloadOffset=ether.packetLoop(packetReceive());
if(tcpPayloadOffset>0)
setBufferPtr(tcpPayloadOffset);
else
packetPayloadSize=0; // It can be changed by packetReceive()
}
}
return b;
}
void ENC28J60::send()
{
ether.httpServerReply_with_flags(totalBytesCopied, TCP_FLAGS_ACK_V);
totalBytesCopied=0;
}
void ENC28J60::finalizeConn()
{
if(totalBytesCopied>0)
ether.httpServerReply_with_flags(totalBytesCopied,TCP_FLAGS_ACK_V|TCP_FLAGS_PUSH_V);
totalBytesCopied=0;
ether.httpServerReply_with_flags(0,TCP_FLAGS_ACK_V|TCP_FLAGS_FIN_V);
}
void ENC28J60::fillAndSend(char* charPtr,bool fromPgM)
{
bool notCompleted;
byte b;
start:
notCompleted=true;
while(totalBytesCopied<tcpMaxPayload)
{
if(fromPgM)
b=pgm_read_byte(charPtr++);
else
b=*charPtr++;
if(b==0)
{
notCompleted=false;
break;
}
*(tcpPayloadPos+(totalBytesCopied++))=b;
}
if(notCompleted) // The buffer is full, send and continue
{
send();
goto start;
}
}
void ENC28J60::fillAndSend(char* charPtr)
{
fillAndSend(charPtr,true);
}
char* chrBuff=new char[4];
void ENC28J60::fillAndSend(char chr)
{
chrBuff[0]=chr;
chrBuff[1]=0; // String terminator
fillAndSend(chrBuff,false);
}
void ENC28J60::fillAndSend(byte intgr)
{
fillAndSend((int)intgr);
}
void ENC28J60::fillAndSend(int intgr)
{
fillAndSend(itoa(intgr,chrBuff,10),false);
}
void ENC28J60::fillAndSend(byte* bytes,int s)
{
bool notCompleted;
byte b;
start:
notCompleted=true;
while(totalBytesCopied<tcpMaxPayload)
{
b=pgm_read_byte(bytes++);
if(s==0)
{
notCompleted=false;
break;
}
*(tcpPayloadPos+(totalBytesCopied++))=b;
s--;
}
if(notCompleted)
{
send();
goto start;
}
}
void ENC28J60::setBufferPtr(int tcpPayloadOffset)
{
ether.httpServerReplyAck();
tcpPayloadPos=buffer+tcpPayloadOffset;
bufferPtr=tcpPayloadPos;
packetPayloadSize-=tcpPayloadOffset;
tcpMaxPayload=bufferSize-tcpPayloadOffset-1;
}
//---------------------------------------------------------------------------
Then add the lines commented with "New" to the same file:
...
byte ENC28J60::initialize (uint16_t size, const byte* macaddr, byte csPin)
{
endOfBuffer=buffer+size-1; // New
...
uint16_t ENC28J60::packetReceive() {
...
len = header.byteCount - 4; //remove the CRC count
packetPayloadSize=len; // New
...
lastReceivedPacketFlag=buffer[TCP_FLAGS_P]; // New
return len;
}
...
And add these declarations into "enc28j60.h", above the decleration of "initialize()":
static int packetPayloadSize; static byte readByte(); static void send(); static void finalizeConn(); static void fillAndSend(char* charPtr,bool fromPgM); static void fillAndSend(char* charPtr); static void fillAndSend(char chr); static void fillAndSend(byte intgr); static void fillAndSend(int intgr); static void fillAndSend(byte* bytes,int s); static void setBufferPtr(int TCPPayloadPos);
Test
#define BUFFER_SIZE 150
#include <EtherCard.h>
byte Ethernet::buffer[BUFFER_SIZE]; // tcp ip send and receive buffer
void setup()
{
Serial.begin(9600);
while(!Serial);
byte mac[6]={170,170,170,170,170,170};
byte ip[4]={192,168,0,253};
ether.begin(sizeof Ethernet::buffer,mac,10);
ether.staticSetup(ip);
}
void loop()
{
uint16_t payloadOffset = ether.packetLoop(ether.packetReceive());
if (payloadOffset)
{
ether.setBufferPtr(payloadOffset);
Serial.println("INCOMING TCP PACKET");
Serial.println("-------------------");
byte b;
while(ether.packetPayloadSize>0)
{
b=ether.readByte();
Serial.print(char(b));
}
Serial.println("-------------------");
char* page=PSTR(
"HTTP/1.0 200 OK\r\n"
"Content-Type: text/html\r\n\r\n"
"<html>"
"<head><title>TEST</title></head>"
"<body>"
"<h3>TEST</h3>"
"<p>This is a page larger than your buffer size</p>"
"<pre>"
"01234567890123456789012345678901234567890123456789\r\n"
"01234567890123456789012345678901234567890123456789\r\n"
"01234567890123456789012345678901234567890123456789"
"</pre>"
"</body>"
"</html>");
ether.fillAndSend(page,sizeof page);
ether.finalizeConn();
}
}
I sent the same packet that I sent for the first test. These are what the program could read:
INCOMING TCP PACKET ------------------- GET /?largeGETVariable=01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789 HTTP/1.1 Host: 192.168.0.253 User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0 Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3 Accept-Encoding: gzip, deflate Connection: keep-alive Cache-Control: max-age=0 -------------------
And what it could transmit to the client:
I tried also to send 40 requests in a few seconds to test its stability. It responded to all, without any problems.
Now we can send and receive data as much as we want without thinking how many packets will be sent or received or how large each piece of data have to be...
Related Links
- EtherCard (GitHub)
- PAKEV (A project in which the library was used with the modifications explained above.)
Posted: 10/08/2014 23:03
Keywords: ethercard, arduino ethernet, enc28j60, atmega328, atmel, ip fragments, multiple
Leave Comment
Comments (25)
Hi, I have doing my first Arduino project, I have same problem too, in my project I use atmega328 to control enc28j60 & some I2C device such as light on/off PIR sensor, fan, door lock, I2C ext. I/O etc... I need to adjust my buffer till 935 just can run system was not hand & load the web page . but now have the problem is my User Interface was very simple & cannot add any image, although I increase or decrease the buffer size cause the system or web page abnormal also, so I tried to find many many web site want to solve my problem, I very happy that I can find your page & your problem same I got , I will try you way to solve my problem & I hope it work, thank you very much for your detail expand & your page .
Hi,
This is what I'm looking for -- if only will works!
I did as you explain before and came to compilation error during testing see below
In file included from C:\Program Files\Arduino\hardware\arduino\avr\cores\arduino/Arduino.h:28:0,
from C:\Program Files\Arduino\libraries\ethercard-master/EtherCard.h:28,
from enc28j60smallbuffer.ino:3:
enc28j60smallbuffer.ino: In function 'void loop()':
enc28j60smallbuffer.ino:34:16: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]
I'm using Arduino IDE 1.6.0
Will you be so kind and give a help?
In addition to my previous message ( about compilation error). I did try your modification in Arduino IDE 1.0.6 and it works.
So if you will find some time could you please update your webpage code to work with latest IDE 1.6.0 ?
If it causes an error in the new version, you can cast 'const char *' to 'char *' manually. Because PSTR() returns a const char pointer. Modify this line:
char* page=PSTR(
as follows:
char* page=(char*)PSTR(
Hi, is it a way to send varible not stored in flash? I want to read pin status and display in page.
Yes, it is possible:
ether.fillAndSend(analogRead(A0));
This function is overloaded for integers too. It writes the integer to a buffer and calls the function "fillAndSend(char* charPtr,bool fromPgM)" setting "fromPgM" to false.
Thanks for publishing this brilliant piece of code, I have also spend many days to finalize a good Ethernet library in my data logger arduino project , finally deiced to use ethercard library, but was facing some issues, but using ur suggested changes really help me reduce dynamic memory usage, but now i am facing one issue that some times my system get hangs and identified through serial monitor that junk character sending some times we request page. any idea that what could be this issue?
I haven't encountered such a problem before. I think it can be related to a hardware problem.
Hi,
First of all, thank you for your work on this issue. I try to use this method since 3 days (I do not give up).
This one works well for receiving requests (and certainly for sending web pages too :-/).
But I do not succeed on sending a page containing dynamic values (read from EEPROM) certainly due to my weak level with C language)
I tried to use multiple calls to ether.fillAndSend() as you said in comments but my web page do not receive the response from arduino, it loops forever.
Here is what I want to Send (values comes from EEPROM)
HTTP/1.0 200 OK
Content-Type: application/json;charset=utf-8
Access-Control-Allow-Origin: *
Server: Arduino
Pragma: no-cache
Connnection: close
{"num":3,"alarms":["8:15/1","8:16/0","8:20/2"]}.
Thank you again.
Did you call ether.finalizeConn(); after the last call to fillAndSend() function? It is required to finalize the connection sending a packet with FIN flag is set.
It works perfectly!
How to get a variable into the page?
Thanks for help.
char* page;
char* status;
int value = analogRead(tempPin);
if(digitalRead(5)==HIGH) status="ON"; else status="OFF";
page=(char*)PSTR(
"HTTP/1.0 200 OK\
"
"Content-Type: text/html\
\
"
"<html><body>"
"VALUE: %value "
"switch: %status "
"</body></html>"
);
ether.fillAndSend(page,sizeof page);
ether.finalizeConn();
You should call fillAndSend() function for static contents and variables separately:
ether.fillAndSend(page_part1,sizeof page_part1);
ether.fillAndSend(value);
ether.fillAndSend(page_part2,sizeof page_part2);
VERY GOOD JOB! congratulations!!! Very nice upgrade! Many problems solved!
Hi,
How to refresh website?
I tried with: "<meta http-equiv='refresh' content='5'/>"
but doesnt work...
Sorry, forget about last question. Arduino restart helped....
I am looking for example how to control LED connected to arduino via website. Do you have any? or can you tell me how to do this with yours changes?
Hello again,
I am not too good with this..
What you did is really good work. But now I dont know how to send values and put them in proper place.
Earlier I used bufFill.emit_p(PSTR($D),val); it works fine but have the same problem with website size.
How to put values in proper places? is there any way without separating commands? for example:
ether.fillAndSend(page_part1,sizeof page_part1);
ether.fillAndSend(value);
ether.fillAndSend(page_part2,sizeof page_part2);
And how to for example control led via website?
It is possible to write a new function that replaces variable names with their values before sending them to "fillAndSend()". Or we should split content as shown in the example.
You can implement a simple web server for arduino to control it from a website.
How to clear Ethernet::buffer after every communication ? My communication repeats more than once.
just tried your code, omg works like a charm, use it to display stock prices in a table with undefinite length. Thanks a lot. I must admit, I have no idea how you are able to write this code, I don't understand it even though it is plain for my eye. great stuff, congrats. The key for variables is in fillAndSend(char* charPtr,bool fromPgM) the false option.
Without I got garbage all over the page.
Greetings KLaus
the stockprice server works now for a week 7/24h without any problem. your code is obviously perfect. No hiding stack or ram overflow on the long run. absolut great. Thanks.
Greetings Klaus
Even after 6 years since this post:
Thank you for sharing. It helped me a lot.