1.8.Snappy wire compression

The initial Hello message is encoded as follows:

frame-data = msg-id || msg-data
frame-size = length of frame-data, encoded as a 24bit big-endian integer

where msg-id is an RLP-encoded integer identifying the message and msg-data is an RLP list containing the message data.

All messages following Hello are compressed using the Snappy algorithm.

frame-data = msg-id || snappyCompress(msg-data)
frame-size = length of frame-data encoded as a 24bit big-endian integer

Note that the frame-size of compressed messages refers to the compressed size of msg-data. Since compressed messages may inflate to a very large size after decompression, implementations should check for the uncompressed size of the data before decoding the message. This is possible because the snappy format contains a length header. Messages carrying uncompressed data larger than 16 MiB should be rejected by closing the connection.

Topics are post-fixed with an encoding. Encodings define how the payload of a gossipsub message is encoded.

• ssz_snappy - All objects are SSZ-encoded and then compressed with Snappy block compression. Example: The beacon aggregate attestation topic string is /eth2/446a7232/beacon_aggregate_and_proof/ssz_snappy, the fork digest is 446a7232 and the data field of a gossipsub message is an AggregateAndProof that has been SSZ-encoded and then compressed with Snappy.

Snappy has two formats: “block” and “frames” (streaming). Gossip messages remain relatively small (100s of bytes to 100s of kilobytes) so basic snappy block compression is used to avoid the additional overhead associated with snappy frames.

Implementations MUST use a single encoding for gossip. Changing an encoding will require coordination between participating implementations.

Size limits are placed both on the RPCMsg frame as well as the encoded payload in each Message.

Clients MUST reject and MUST NOT emit or propagate messages whose size exceed the following limits:

• The size of the encoded RPCMsg (including control messages, framing, topics, etc) must not exceed max_message_size().
• The size of the compressed payload in the Message.data field must not exceed max_compressed_len(MAX_PAYLOAD_SIZE).
• The size of the uncompressed payload must not exceed MAX_PAYLOAD_SIZE or the type-specific SSZ bound, whichever is lower.

The token of the negotiated protocol ID specifies the type of encoding to be used for the req/resp interaction. Only one value is possible at this time:

• ssz_snappy: The contents are first SSZ-encoded and then compressed with Snappy frames compression. For objects containing a single field, only the field is SSZ-encoded not a container with a single field. For example, the BeaconBlocksByRoot request is an SSZ-encoded list of Root’s. This encoding type MUST be supported by all clients.

The SimpleSerialize (SSZ) specification outlines how objects are SSZ-encoded.

To achieve snappy encoding on top of SSZ, we feed the serialized form of the object to the Snappy compressor on encoding. The inverse happens on decoding.

Snappy has two formats: “block” and “frames” (streaming). To support large requests and response chunks, snappy-framing is used.

Since snappy frame contents have a maximum size of 65536 bytes and frame headers are just identifier (1) + checksum (4) bytes, the expected buffering of a single frame is acceptable.

Encoding-dependent header: Req/Resp protocols using the ssz_snappy encoding strategy MUST encode the length of the raw SSZ bytes, encoded as an unsigned protobuf varint.

Writing: By first computing and writing the SSZ byte length, the SSZ encoder can then directly write the chunk contents to the stream. When Snappy is applied, it can be passed through a buffered Snappy writer to compress frame by frame.

Reading: After reading the expected SSZ byte length, the SSZ decoder can directly read the contents from the stream. When snappy is applied, it can be passed through a buffered Snappy reader to decompress frame by frame.

Before reading the payload, the header MUST be validated:

• The length-prefix MUST be encoded as an unsigned protobuf varint. It SHOULD be minimally encoded (i.e., without any redundant bytes) and MUST not exceed 10 bytes in length, which is sufficient to represent any Uint64 value. The length-prefix MUST be decoded into a type which supports the full range of Uint64 values.
• The length-prefix is within the expected size bounds derived from the payload SSZ type or MAX_PAYLOAD_SIZE, whichever is smaller.

After reading a valid header, the payload MAY be read, while maintaining the size constraints from the header.

A reader MUST NOT read more than max_compressed_len(n) bytes after reading the SSZ length-prefix n from the header.

A reader MUST consider the following cases as invalid input:

• Any remaining bytes, after having read the n SSZ bytes. An EOF is expected if more bytes are read than required.
• An early EOF, before fully reading the declared length-prefix worth of SSZ bytes.

In case of an invalid input (header or payload), a reader MUST:

• From requests: send back an error message, response code InvalidRequest. The request itself is ignored.
• From responses: ignore the response, the response MUST be considered bad server behavior.

All messages that contain only a single field MUST be encoded directly as the type of that field and MUST NOT be encoded as an SSZ container.

Responses that are SSZ-lists (for example List[SignedBeaconBlock, ...]) send their constituents individually as response_chunks. For example, the List[SignedBeaconBlock, ...] response type sends zero or more response_chunks. Each successful response_chunk contains a single SignedBeaconBlock payload.