Bir bilgisayar mühendisi için programlama dili, öğrendiklerini sınadığı, deneyler yaptığı bir laboratuardır ve mühendisler deneylerini, kestiremedikleri sonuçları gözlemlemek için değil, öngördükleri sonuçları doğrulamak için yapar...

Modifying the EtherCard Library To Handle Large TCP Packets

A few functions for the EtherCard library to make it able to handle packet fragmentations and send/receive multiple large TCP packets on Atmel MCUs.

ENC28J60 ethernet moduleI tried both the UIPEthernet and EtherCard libraries for my enc28j60 with atmega328. The UIPEthernet library is implemented so that it can be used directly instead of the official ethernet library of Arduino. But it requires more space than the official library. It was also slow while sending data and it caused my atmega328 to freeze frequently for no reason. So, after many tries I gave up and kept searching for another library. EtherCard was the second library I found for enc28j60. It is faster and more stable than the other one. I am not sure if it supports all the methods implemented in UIPEthernet, but methods for receiving and sending small TCP packets worked well.  I think its implementation is simpler and you don't need a long time to understand and be able to modify it.

What Exactly Is The Problem?

There is already a function that can be used to send data in multiple packets, in the library. But I couldn't make my program able to receive fragmented large TCP packets which can't fit in a small sized buffer.

Let me give an example:

#define BUFFER_SIZE 150

#include <EtherCard.h>
byte Ethernet::buffer[BUFFER_SIZE];

void setup()
{
  Serial.begin(9600);
  while(!Serial);

  byte mac[6]={170,170,170,170,170,170};
  byte ip[4]={192,168,0,253};
  ether.begin(sizeof Ethernet::buffer,mac,10);
  ether.staticSetup(ip);
}

const char page[] PROGMEM =
"HTTP/1.0 200 OK\r\n"
"Content-Type: text/html\r\n\r\n"
"<html>"
  "<head><title>TEST</title></head>"
  "<body>"
    "<h3>TEST</h3>"
    "<p>This is a page larger than your buffer size</p>"
    "<pre>"
    "01234567890123456789012345678901234567890123456789\r\n"
    "01234567890123456789012345678901234567890123456789\r\n"
    "01234567890123456789012345678901234567890123456789"
    "</pre>"
  "</body>"
"</html>";

void loop()
{
  uint16_t payloadPos = ether.packetLoop(ether.packetReceive());
  if (payloadPos)
  {
    char* incomingData = (char *) Ethernet::buffer + payloadPos;
    
    Serial.println("INCOMING TCP PACKET");
    Serial.println("-------------------");
    Serial.println(incomingData);
    Serial.println("-------------------");
    
    int sz=BUFFER_SIZE-payloadPos; // max payload size (TCP header is also kept in the buffer)
    if(sizeof(page)<sz)
      sz=sizeof(page);
    memcpy_P(Ethernet::buffer + payloadPos, page, sz); // Copy data from flash to RAM
    ether.httpServerReply(sz-1);
  }
}

This is an example that prints everything it receives to the serial port and send a response which can not completely fit in the buffer.

The GET request I sent to test what will be written to the serial port:

GET /?largeGETVariable=0123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789 HTTP/1.1
Host: 192.168.0.253
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cache-Control: max-age=0


The data printed to the serial port:

INCOMING TCP PACKET
-------------------
GET /?largeGETVariable=012345678901234567890123456789012345678901234567890123456789012345678901
-------------------

The data transmitted to the client:
An incomplete response received by the client.

As it can be seen, the program cannot handle whole the packet and send all bytes of the page because of the size of the buffer. In fact, it is possible to send data in multiple packets using this library as I mentioned before and so the client can receive all the data. But we have to break the data into pieces that can fit into the buffer. So we still need a function which does that at background. You can think that increasing the buffer would be a solution but I especially used a small sized buffer because I were using nearly all the RAM in my project. (Or maybe I could have been used other atmel chips which have smaller RAMs.) On the other hand, the request was larger than what can fit in a single TCP packet. So, we also need a method to handle packet fragmentations.

In that case we have three questions:

  1. How can we handle every byte of an incoming packet without truncation?
  2. How can we determine whether a packet is a continuation of the previous one?
  3. And how can we send large responses which can not fit into buffer?

Note that, we have two buffers, the buffer which is in enc28j60 and the buffer which we allocate manually in atmega328. The first buffer is large enough to keep a large TCP packet. But the other one may not be always so large, for example if your program consumes large amount of RAM for some other tasks.

1. Handling Multiple Large Packets

  1. When a packet is received send an ACK, read its size and copy a part that can fit into the buffer.
  2. Walk the buffer to handle every byte and count each byte you handled.
  3. When you reach the end of the buffer, compare the number of handled bytes with the packet size which you read in the first step.
  4. If the number of handled bytes is smaller, then the packet hasn't handled entirely and there are still unread bytes. Read the next part that can fit into the buffer.
  5. Repeat 2, 3 and 4 until the number of handled bytes became equal to the size of the TCP packet.
  6. Check flags of the last received packet. If PUSH is set, it means this is the last packet that the client will send until it receives a response (take a look at the screenshots below). Go to step 7. Otherwise, send an ACK and check if the next packet (which is the continuation of the packet we are currently processing) has arrived.
  7. If the next packet has arrived go to step 1 or else you can assume that the client is waiting for a response. Send the response and then send a packet with FIN flag to finalize the connection.

2. Determining Continuation Packets

Fragmented dataThe screenshots on the right were taken from wireshark while the request was being transmitted to test the example. The GET request was being transmitted in two packets because size of the data exceeded the maximum packet size limit which is typically 1500 (MTU). If a packet is the last continuation, then it is being sent with PUSH flag, as I observed. So this is the answer for the second question.

3. Sending Large Packets

  1. If the data can fit into the buffer, copy it to the buffer (starting from right after the TCP header) and do not send it until the buffer is filled up completely. If it can not fit into the buffer, copy a piece that can fit into it and send the buffer with ACK flag. Repeat this step for all your data that you want to send.
  2. If there is unsent data in the buffer, send it with both ACK and PUSH flags and then send a packet with FIN flag, to finalize the connection.

Implementations

Here are the implementations of those steps. Copy these codes and paste them into enc28j60.cpp right above the initialize() function.

//--------------------
// July 2014
// Huseyin Atasoy
//---------------------------------------------------------------------------
#include "EtherCard.h" // For ether.httpServerReply_with_flags() function
#include "net.h" // For TCP_FLAGS_FIN_V and TCP_FLAGS_ACK_V flags

byte* tcpPayloadPos;
byte* endOfBuffer;
int tcpMaxPayload;
byte lastReceivedPacketFlag;

int ENC28J60::packetPayloadSize;
byte* bufferPtr;
int totalBytesCopied;
byte ENC28J60::readByte()
{
    if(bufferPtr==endOfBuffer)
    {
        int remaining=packetPayloadSize;
        if(remaining>0)
        {
            if(remaining>tcpMaxPayload) remaining=tcpMaxPayload;
            bufferPtr = tcpPayloadPos;
            readBuf(remaining,bufferPtr);
        }
    }                    
    packetPayloadSize--;
    byte b=*(bufferPtr++);
    
    if(packetPayloadSize==0)
    {
        if((lastReceivedPacketFlag & TCP_FLAGS_PUSH_V)==0)
        {
            uint16_t tcpPayloadOffset=ether.packetLoop(packetReceive());
            if(tcpPayloadOffset>0)
                setBufferPtr(tcpPayloadOffset);
            else
                packetPayloadSize=0; // It can be changed by packetReceive()
        }
    }

    return b;
}

void ENC28J60::send()
{
  ether.httpServerReply_with_flags(totalBytesCopied, TCP_FLAGS_ACK_V);
  totalBytesCopied=0;
}

void ENC28J60::finalizeConn()
{
  if(totalBytesCopied>0)
    ether.httpServerReply_with_flags(totalBytesCopied,TCP_FLAGS_ACK_V|TCP_FLAGS_PUSH_V);
  totalBytesCopied=0;
  ether.httpServerReply_with_flags(0,TCP_FLAGS_ACK_V|TCP_FLAGS_FIN_V);
}

void ENC28J60::fillAndSend(char* charPtr,bool fromPgM)
{
  bool notCompleted;
  byte b;
start:
  notCompleted=true;
  while(totalBytesCopied<tcpMaxPayload)
  {
    if(fromPgM)
      b=pgm_read_byte(charPtr++);
    else
      b=*charPtr++;
    if(b==0)
    {
      notCompleted=false;
      break;
    }
    *(tcpPayloadPos+(totalBytesCopied++))=b;
  }
  if(notCompleted) // The buffer is full, send and continue
  {
    send();
    goto start;
  }
}

void ENC28J60::fillAndSend(char* charPtr)
{
  fillAndSend(charPtr,true);
}

char* chrBuff=new char[4];
void ENC28J60::fillAndSend(char chr)
{
  chrBuff[0]=chr;
  chrBuff[1]=0; // String terminator
  fillAndSend(chrBuff,false);
}

void ENC28J60::fillAndSend(byte intgr)
{
  fillAndSend((int)intgr);
}

void ENC28J60::fillAndSend(int intgr)
{
  fillAndSend(itoa(intgr,chrBuff,10),false);
}

void ENC28J60::fillAndSend(byte* bytes,int s)
{
  bool notCompleted;
  byte b;
start:
  notCompleted=true;
  while(totalBytesCopied<tcpMaxPayload)
  {
    b=pgm_read_byte(bytes++);
    if(s==0)
    {
      notCompleted=false;
      break;
    }
    *(tcpPayloadPos+(totalBytesCopied++))=b;
    s--;
  }
  if(notCompleted)
  {
    send();
    goto start;
  }
}

void ENC28J60::setBufferPtr(int tcpPayloadOffset)
{
    ether.httpServerReplyAck();
    tcpPayloadPos=buffer+tcpPayloadOffset;
    bufferPtr=tcpPayloadPos;
    packetPayloadSize-=tcpPayloadOffset;
    tcpMaxPayload=bufferSize-tcpPayloadOffset-1;
}
//---------------------------------------------------------------------------

Then add the lines commented with "New" to the same file:

...
byte ENC28J60::initialize (uint16_t size, const byte* macaddr, byte csPin)
{
    endOfBuffer=buffer+size-1; // New
...
uint16_t ENC28J60::packetReceive() {
...
    len = header.byteCount - 4; //remove the CRC count
    packetPayloadSize=len; // New
...
    lastReceivedPacketFlag=buffer[TCP_FLAGS_P]; // New
    return len;
}
...

And add these declarations into "enc28j60.h", above the decleration of "initialize()":

static int packetPayloadSize;
static byte readByte();
static void send();
static void finalizeConn();
static void fillAndSend(char* charPtr,bool fromPgM);
static void fillAndSend(char* charPtr);
static void fillAndSend(char chr);
static void fillAndSend(byte intgr);
static void fillAndSend(int intgr);
static void fillAndSend(byte* bytes,int s);
static void setBufferPtr(int TCPPayloadPos);

Test

#define BUFFER_SIZE 150

#include <EtherCard.h>
byte Ethernet::buffer[BUFFER_SIZE]; // tcp ip send and receive buffer

void setup()
{
  Serial.begin(9600);
  while(!Serial);

  byte mac[6]={170,170,170,170,170,170};
  byte ip[4]={192,168,0,253};
  ether.begin(sizeof Ethernet::buffer,mac,10);
  ether.staticSetup(ip);
}

void loop()
{
  uint16_t payloadOffset = ether.packetLoop(ether.packetReceive());
  if (payloadOffset)
  { 
    ether.setBufferPtr(payloadOffset);

    Serial.println("INCOMING TCP PACKET");
    Serial.println("-------------------");
    byte b;
    while(ether.packetPayloadSize>0)
    {
      b=ether.readByte();
      Serial.print(char(b));
    }
    Serial.println("-------------------");
    
    char* page=PSTR(
      "HTTP/1.0 200 OK\r\n"
      "Content-Type: text/html\r\n\r\n"
      "<html>"
        "<head><title>TEST</title></head>"
        "<body>"
          "<h3>TEST</h3>"
          "<p>This is a page larger than your buffer size</p>"
          "<pre>"
          "01234567890123456789012345678901234567890123456789\r\n"
          "01234567890123456789012345678901234567890123456789\r\n"
          "01234567890123456789012345678901234567890123456789"
          "</pre>"
        "</body>"
      "</html>");
    ether.fillAndSend(page,sizeof page);

    ether.finalizeConn();
  }
}

I sent the same packet that I sent for the first test. These are what the program could read:

INCOMING TCP PACKET
-------------------
GET /?largeGETVariable=01234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789 HTTP/1.1
Host: 192.168.0.253
User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:31.0) Gecko/20100101 Firefox/31.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.5,en;q=0.3
Accept-Encoding: gzip, deflate
Connection: keep-alive
Cache-Control: max-age=0

-------------------

And what it could transmit to the client:
A response that is transmitted correctly.

I tried also to send 40 requests in a few seconds to test its stability. It responded to all, without any problems.

Now we can send and receive data as much as we want without thinking how many packets will be sent or received or how large each piece of data have to be...

Related Links

Sayfayı
Yayın tarihi: 10 Ağustos 2014 Pazar, 23:03
Anahtar kelimeler: ethercard, arduino ethernet, enc28j60, atmega328, atmel, ip fragments, multiple

Yorum Gönder

 
Yorumunuzu -1. yoruma yanıt olarak gönderiyorsunuz. Yanıtlamayı iptal etmek için buraya tıklayabilirsiniz.

 

Yorumlar (18)

Man, Cheung
Yanıtla
25 Kasım 2014 Salı, 19:52
#1

Hi, I have doing my first  Arduino project, I have same problem too, in my project I use atmega328 to control enc28j60 &  some I2C device such as light on/off PIR sensor, fan, door lock, I2C ext. I/O etc... I need to adjust my buffer till 935 just can run system was not hand & load the web page . but now have the problem is my User Interface was very simple & cannot add any image, although I increase or decrease the buffer size cause the system or web page abnormal also, so I tried to find many many web site  want to solve my problem, I very happy that I can find your page & your problem same I got , I will try you way to solve my problem & I hope it work, thank you very much for your detail expand & your page .

Woj
Yanıtla
01 Mart 2015 Pazar, 11:34
#2

Hi,
This is what I'm looking for -- if only will works!

I did as you explain before and came to compilation error during testing see below

In file included from C:\Program Files\Arduino\hardware\arduino\avr\cores\arduino/Arduino.h:28:0,
                from C:\Program Files\Arduino\libraries\ethercard-master/EtherCard.h:28,
                from enc28j60smallbuffer.ino:3:
enc28j60smallbuffer.ino: In function 'void loop()':
enc28j60smallbuffer.ino:34:16: error: invalid conversion from 'const char*' to 'char*' [-fpermissive]

I'm using Arduino IDE 1.6.0
Will you be so kind and give a help?

woj
Yanıtla
01 Mart 2015 Pazar, 14:47
#3

In  addition to my  previous message ( about compilation error). I did try your modification in Arduino IDE 1.0.6 and it works.
So if you will find some time could you please update your webpage code to work with latest IDE 1.6.0 ?

Hüseyin Atasoy
Yanıtla
03 Mart 2015 Salı, 14:38
#4

If it causes an error in the new version, you can cast 'const char *' to 'char *' manually. Because PSTR() returns a const char pointer. Modify this line:
char* page=PSTR(
as follows:
char* page=(char*)PSTR(

Piter
Yanıtla
18 Mart 2015 Çarşamba, 20:17
#5

Hi, is it a way to send varible not stored in flash? I want to read pin status and display in page.

Huseyin Atasoy
Yanıtla
29 Mart 2015 Pazar, 11:29
#6

Yes, it is possible:
ether.fillAndSend(analogRead(A0));
This function is overloaded for integers too. It writes the integer to a buffer and calls the function "fillAndSend(char* charPtr,bool fromPgM)" setting "fromPgM" to false.

Rajesh
Yanıtla
13 Nisan 2015 Pazartesi, 10:15
#7

Thanks for publishing this brilliant piece of code, I have also spend many days to finalize a good Ethernet library in my data logger arduino project , finally deiced to use ethercard library, but was facing some issues, but using ur suggested changes really help me reduce dynamic memory usage, but now i am facing one issue that some times my system get hangs and identified through serial monitor that junk character sending  some times we request page. any idea that what could be this issue?

Huseyin Atasoy
Yanıtla
26 Nisan 2015 Pazar, 08:52
#8

I haven't encountered such a problem before. I think it can be related to a hardware problem.

Christophe Deroeux
Yanıtla
30 Nisan 2016 Cumartesi, 23:04
#9

Hi,

First of all, thank you for your work on this issue. I try to use this method since 3 days (I do not give up).

This one works well for receiving requests (and certainly for sending web pages too :-/).

But I do not succeed on sending a page containing dynamic values ​​(read from EEPROM) certainly due to my weak level with C language)

I tried to use multiple calls to ether.fillAndSend() as you said in comments but my web page do not receive the response from arduino, it loops forever.

Here is what I want to Send (values comes from EEPROM)

HTTP/1.0 200 OK
Content-Type: application/json;charset=utf-8
Access-Control-Allow-Origin: *
Server: Arduino
Pragma: no-cache
Connnection: close

{"num":3,"alarms":["8:15/1","8:16/0","8:20/2"]}.

Thank you again.

Hüseyin Atasoy
Yanıtla
11 Mayıs 2016 Çarşamba, 08:20
#10

Did you call ether.finalizeConn(); after the last call to fillAndSend() function? It is required to finalize the connection sending a packet with FIN flag is set.

Hawk
Yanıtla
06 Mayıs 2016 Cuma, 16:16
#11

It works perfectly!
How to get a variable into the page?
Thanks for help.

char* page;
char* status; 
int value = analogRead(tempPin);
if(digitalRead(5)==HIGH) status="ON"; else status="OFF"; 

    page=(char*)PSTR(
      "HTTP/1.0 200 OK\
"
      "Content-Type: text/html\
\
"
      "<html><body>"

            "VALUE: %value "
            "switch: %status "
             
      "</body></html>"
        );

ether.fillAndSend(page,sizeof page);
ether.finalizeConn();

Hüseyin Atasoy
Yanıtla
11 Mayıs 2016 Çarşamba, 08:27
#12

You should call fillAndSend() function for static contents and variables separately:

ether.fillAndSend(page_part1,sizeof page_part1);
ether.fillAndSend(value);
ether.fillAndSend(page_part2,sizeof page_part2);

Eduardo Santos
Yanıtla
19 Temmuz 2016 Salı, 18:21
#13

VERY GOOD JOB! congratulations!!! Very nice upgrade! Many problems solved!

artur
Yanıtla
04 Ekim 2016 Salı, 14:43
#14

Hi,
How to refresh website?
I tried with: "<meta http-equiv='refresh' content='5'/>"
but doesnt work...

artur
Yanıtla
04 Ekim 2016 Salı, 16:41
#15

Sorry, forget about last question. Arduino restart helped....
I am looking for example how to control LED connected to arduino via website. Do you have any? or can you tell me how to do this with yours changes?

artur
Yanıtla
05 Ekim 2016 Çarşamba, 12:13
#16

Hello again,
I am not too good with this..
What you did is really good work. But now I dont know how to send values and put them in proper place.
Earlier I used bufFill.emit_p(PSTR($D),val);  it works fine but have the same problem with website size.
How to put values in proper places?  is there any way without separating commands? for example:
ether.fillAndSend(page_part1,sizeof page_part1);
ether.fillAndSend(value);
ether.fillAndSend(page_part2,sizeof page_part2);

And how to for example control led via website?

Huseyin Atasoy
Yanıtla
17 Ekim 2016 Pazartesi, 08:17
#17

It is possible to write a new function that replaces variable names with their values before sending them to "fillAndSend()". Or we should split content as shown in the example.
You can implement a simple web server for arduino to control it from a website.

ronie
Yanıtla
25 Ekim 2017 Çarşamba, 13:42
#18

How to clear Ethernet::buffer after every communication ? My communication repeats more than once.

 
 
Sayfa 54 sorgu ile 0.076 saniyede oluşturuldu.
Atasoy Blog v4 © 2008-2017 Hüseyin Atasoy