This article is for open source technology enthusiasts who are interested in blockchain technologies and have a working knowledge of the Hyperledger Sawtooth project. It will give them a quick overview of the network layer in the Hyperledger Sawtooth architecture.
The network layer is responsible for communication between validators in a Sawtooth network, including performing initial connectivity, peer discovery and message handling.
- Upon start-up, validator instances begin listening on a specified interface and port for incoming connections.
- Upon connection and peering, validators exchange messages with each other based on the rules of a gossip or epidemic protocol.
A primary design goal is to keep the network layer as self-contained as possible — the application should not need to understand implementation details of the network in order to send and receive messages.
Services
Sawtooth has adopted the 0MQ (Zero MQ) asynchronous client/server pattern. This consists of a 0MQ router socket on the server side which listens on a provided endpoint, with a number of connected 0MQ dealer sockets as the connected clients.
The 0MQ guide describes the features of this pattern as follows:
- Clients connect to the server and send requests.
- For each request, the server sends 0 or more replies.
- Clients can send multiple requests without waiting for a reply.
- Servers can send multiple replies without waiting for new requests.
States
Sawtooth defines three states related to the connection between any two validator nodes.
- Unconnected
- Connected – A connection is a required prerequisite for peering.
- Peered – A bi-directional relationship that forms the base case for application-level message passing (gossip).
Message types
This protocol includes the following types of messages.
Connect is the mechanism for initiating the connection to the remote node. Connect performs a basic 0MQ dealer-to-router connection to the remote node and exchanges identity information for the purpose of supporting a two-way conversation. Connections sit atop 0MQ sockets and allow the dealer/router conversation.
Ping messages allow for keep-alive between router and dealer sockets.
Peer requests establish a bi-directional peering relationship between the two nodes. A peer request can be rejected by the remote node. If a peer request is rejected, the expectation is that a node attempts to connect with other nodes in the network via some strategy until the peering minimum connectivity threshold for that node is reached. If possible, the bi-directional relationship occurs over the already established 0MQ socket between dealer and router.
A get_peers message returns a list of peers of a given node. This can be performed in a basic connected state; it does not require peering to have occurred. The intent is to allow a node attempting to reach its minimum connectivity peering threshold to build a view of active candidate peers via a neighbour-of-neighbours approach.
An unpeer message breaks the peering relationship between nodes. This may occur in several instances, such as a node leaving the network. Nodes may also silently leave the network, in which case their departure will be detected by the failure of the ping/keep-alive message. An unpeer request does not necessarily imply a disconnect.
A disconnect message breaks the wire protocol connection to the remote node and informs the router end to clean up the connection.
Transmission methods
Transmission methods include the following.
Broadcast (msg) transmits an application message to the network following a ‘gossipy’ pattern. This does not guarantee 100 per cent delivery of the message to the whole network, but based on the gossip parameters, almost complete delivery is likely. A node only accepts messages for broadcast/forwarding from peers.
Send (node, msg) attempts to send a message to a particular node over the bi-directional 0MQ connection. Delivery is not guaranteed. If a node has reason to believe that delivery to the destination node is impossible, it can return an error response. A node only accepts a message for sending from peer nodes.
A request (msg) is a special type of broadcast message that can be examined and replied to, rather than forwarded. The intent is for the application layer to construct a message payload which can be examined by a special request handler and replied to, rather than forwarded on to connected peers. If the application layer reports that the request can’t be satisfied, the message will be forwarded to peers as per the rules of a standard broadcast message. A node only accepts request messages from peer nodes.
Peer discovery
A bi-directional peering via a neighbour-of-neighbours approach gives reliable connectivity, with messages delivered to all nodes > 99 per cent of the time, based on the random construction of the network. Peer connections are established by collecting a suitable population of candidate peers through successive connect/get_peers calls (neighbours of neighbours). The connecting validator then selects a candidate peer randomly from the list, and attempts to connect and peer with it. If this succeeds and the connecting validator has reached minimum connectivity, the process halts. If minimum connectivity has not yet been reached, the validator continues attempting to connect to new candidate peers, refreshing its view of the neighbours of neighbours if it exhausts candidates.
The network component continues to perform a peer search if its number of peers is less than the minimum connectivity. This component rejects peering attempts if its number of peers is equal to or greater than the maximum connectivity. Even if maximum peer connections are reached, a network service should still accept and respond to a reasonable number of connections for the purposes of other node topology build-outs, etc.
Message delivery
The network delivers application messages, i.e., payloads received via broadcast or send to the application layer. The network also performs a basic validation of messages prior to forwarding by calling a handler in the Message Validation component.
When the network receives a request message, it calls a provided handler to determine if the request can be satisfied. If so, the expectation is that the application layer generates a send message with a response that satisfies the request. In this condition, the network layer does not continue to propagate the request message to the network.
In cases where a node could not satisfy the request, the node stores who it received the request from and broadcasts the request on to its peers. If that node receives a send message with the response to the request, it forwards the send message back to the original requester.
The network accepts application payloads for BROADCAST, SEND, and REQUEST from the application layer.
Network layer security
0MQ includes a TLS-like certificate exchange mechanism and protocol encryption capability that is transparent to the socket implementation. Support for socket-level encryption is currently implemented with server keys, which are read from the validator.toml configuration file. For each client, ephemeral certificates are generated on connect. If the server key pair is not configured, network communications between validators will not be authenticated or encrypted.
Network permissioning
The Sawtooth permissioning design allows the validator network to limit the nodes that are able to connect to it. The permissioning rules determine the roles a connection is able to play on the network. The roles control the types of messages that can be sent and received over a given connection. Validators are able to determine whether messages delivered to them should be handled or dropped based on a set of roles and identities stored within the Identity namespace. Each requester is identified by the public key derived from its identity signing key. Permission verifiers examine incoming messages against the policy and the current configuration, and either permit, drop or respond with an error.
Authorisation types
Sawtooth implements two authorisation types: trust and challenge.
Trust is the simplest authorisation type. If trust authorisation is enabled, the validator will trust the connection and approve any roles requested that are available at that endpoint. If the requester wishes to gain access to every role it has permission to access, it can request all and the validator will respond with all available roles. However, if a role that is not available is requested, the requester is rejected and the connection will be closed. Figure 3 shows the message flow for trust authorisation.
If the connection wants to take on a role that requires a challenge to be signed, it will request the challenge by sending the empty challenge message to the validator that it wishes to connect to. The validator will send back a random payload that must be signed. The requester then signs the payload message and returns a response. The requester may also request all. The validator will respond with a status that says whether the challenge was accepted and the roles that the connection is allowed to take on. Figure 4 shows the message flow for challenge authorisation.
In the next article we will take a closer look at the Sawtooth permissioning design.
