Sometime last year I implemented a toy ping utility in Go, where I did most of the packet creation, and serialization manually. I got it working however, there was something off about how I was serializing the IP packet.

func (p *Packet) Serialize() ([]byte, error) {
	buf := new(bytes.Buffer)

	headerSerialized, err := p.Header.Serialize()
	if err != nil {
		return nil, errors.Wrapf(err, "error serializing IPv4 packet header")
	}
	
	// TODO: the order is wrong but it works - why?
	buf.Write(p.Payload)
	buf.Write(headerSerialized)

	return buf.Bytes(), nil
}

This function returns an entire IP packet as a byte stream. The strange part is that I’m writing the payload into the buffer first, followed by the header, which is incorrect. It should be the other way around.

But it was functional, as in, I was getting echo replies to my echo requests and I had no idea why. Whenever I tried fixing the order,

buf.Write(headerSerialized)
buf.Write(p.Payload)

Wireshark would complain, saying the packet is malformed.

wireshark-error

Upon inspecting the hexdump these were my observations.

  • The bytes seemed misaligned. The ICMP Type field aligned with 69 (45 in hex) - a value it doesn’t identify.
  • There seemed to be an extra IP header present in the hexdump. You would see some bytes appearing twice like, 45 which corresponds to VersionIHL field of an IP packet, and the bytes representing the source and destination addresses.

My initial hunch was that it could be the operating system’s network stack adding the extra IP header. However, I wasn’t very sure. I thought, if the OS was indeed adding the header and causing misalignment, the code should have failed in both scenarios, irrespective of how things are ordered. How could it be that the code was functional in one scenario and not in the other? It couldn’t be that the OS was being selective while prepending the header. Are there other factors at play here that I’m unaware of? How do I know what I do not know?

In retrospect, I was asking the right questions but I lacked clarity. I needed to dive deep.

Anyway, I ended up discounting the idea and the issue remained unsolved for a long time. I parked it and moved on with an unsettling feeling for having a functional code I couldn’t reason with. The itch persisted at the back of my mind, but ignoring it was easier.

Until I couldn’t

I’ve been thinking (I do that a lot) about blogging for a long time and while I was setting this site up, I thought I could start with my toy ping implementation. As I began writing, and explaining my code along the way, I hit a roadblock when I reached the above function. There was no way I could justify the wrong ordering of statements. I mean, if I’m writing something on the internet, I absolutely need to have a reasonable explanation for something that seems so wrong. So I went back to seeking discomfort.

I validated my previous findings and found nothing new. Dismayed, I wondered where I could have fallen short in my investigations so far. Should I’ve inspected all the octets in the hexdump and not just the ones relevant to me? The thing is, comparing hexdumps of different requests is hard with the naked eye. There are rows of bytes stacked on top of one another. I was using Wireshark to inspect the requests, so anytime I clicked on a different request, I would lose sight of the previous one and its hexdump. There was no way my human mind could remember a byte sequence accurately.

So this time, I decided to lay out the hexdump flat. I opened excalidraw, pasted the bytes from the hexdump linearly, and compared them to the bytes I was sending out programmatically. As I took time to analyze, the issue slowly started to reveal itself.

Yep! It was the OS indeed

Well, the network stack to be precise.

When I was sending bytes in the right order, final the sequence became
ip header (kernel) | ip header (mine) | icmp (mine)

while when the order was incorrect, it became
IP header (kernel) | ICMP (mine) | IP header (mine)

Basically, anything that came after the kernel’s IP header was being interpreted as the payload.

To understand how this affects packet integrity, it’s essential to know that IP header contains a Protocol field which uses pre-defined numeric values to convey the underlying protocol used in the data portion of the packet. When an IP datagram carries an ICMP payload within, this value is 1.

How the right ordering created a malformed packet

This is the ICMP datagram for Echo or Echo Reply Message message.

    0                   1                   2                   3
    0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Type      |     Code      |          Checksum             |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |           Identifier          |        Sequence Number        |
   +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   |     Data ...
   +-+-+-+-+-

The first byte is the Type field. It can accept a pre-defined set of values depending upon the message type. For Echo and Echo Reply messages the values are 8 and 0 respectively.

When the order is incorrect, the Type field got aligned with 8 which is a known value.

IP header (kernel) | ICMP (mine) | IP header (mine)
                     type

Hence, ping was “functional” and we received an echo reply even though it was all wrong.

incorrect-order

However, when the ordering was correct, Type aligned with 69 (45 in hex) which is an unknown value for the field. As a result, it deemed the packet to be malformed.

ip header (kernel) | ip header (mine) | icmp (mine)
                     type

correct-order

Final thoughts

I felt so relieved after having uncovered this. It’s better to have a non-functional thing rather than having it incorrectly functional.

What’s next is I need to tell the OS somehow not to add its IP header as I will be supplying that myself. One way of doing that is by using raw sockets. It’s essential to clarify that all this time I’ve been using conn.Write() to transmit the bytes where conn was the generic net.Conn interface which as I learned later, was not the right choice for sending raw packets.

I can probably look into RawConn or PacketConn which at a glance, seem more relatable to what I need.