Design Documentation for Software Failure Tolerant Highly Available Distributed Share Market System

This document describes the comp6231 final project design of the DSMS with support for tolerating a single software failure non-malicious Byzantine or achieving high availability tolerance of a single crash failure use active replication. This provide an overview of the system architecture, the roles and interactions of each module, and an outline of what needs to be implemented or modified.

1. Overview

In this project, we extend the existing DSMS from assignment 3 to incorporate active replication. The design allows for:

• Either software failure tolerance handling one non-malicious, potentially buggy replica that may produce incorrect results,or high availability that handling one replica crash failure,
• Front End: Receives client requests, sends them to the sequencer, collects responses from replicas, and returns a single correct result to the client.
• Sequencer: Receives requests from the FE, assigns unique sequence numbers, and reliably multicasts total order the requests to all server replicas.
• Replica Managers: Each replica’s RM detects failures software or crash and replaces the failed replica with a new one.
• Replicas: Three server replicas each with the DSMS logic.

2. System Architecture

Below is a diagram illustrating the system components and data flows:

• Front End (FE): For each incoming request:
  • Sends request to the Sequencer.
  • Waits for at least two matching results from the replicas.
  • If a mismatch or a timeout occurs, the FE notifies the RMs.
  • Returns the correct or majority-agreed result to the client.
• Sequencer:
  • Assigns a unique sequence number to each request.
  • Reliably multicasts UDP or another protocol with reliability measures the request + sequence number to all replicas.
• Replicas (A, B, C):
  • Each runs the DSMS logic in total order of sequence numbers.
  • Returns a response to the FE upon processing each request.
• Replica Managers (RM):
  • Each RM is associated with one replica.
  • When the FE flags a problem either an incorrect result or no response, the RMs coordinate to confirm the failure.
  • After confirmation, the RMs either restart or replace the failing/crashed replica with a fresh one.

3. Detailed Component Interaction

Below is a simplified sequence diagram showing how a typical request is processed, how failures are detected, and how RMs handle those failures.

4. Detailed Implementation Plan

Below shows how I desgined the DSMS with Front End, Sequencer, Replicas, and Replica Managers using only three physical machines on a local network. Also demonstrate how to test crash failures.

A. System Overview

We need:

• 3 Replicas + a Replica Manager attached to each replica,
• 1 Front End process, and
• 1 Sequencer process.

Because I have only 3 physical machines, one of them must host the FE + Sequencer alongside one Replica and its RM. The other two machines each host a Replica and its RM. That gives us 3 replicas total and a single FE Sequencer instance. Below is how the Distribution works:

1. Machine 1:
   • Front End
   • Sequencer
   • Replica A + RM(A)
2. Machine 2:
   • Replica B + RM(B)
3. Machine 3:
   • Replica C + RM(C)

Each block Replica + RM can be run as separate processes or combined logic, but they should communicate on the same machine localhost ports.

Machine 1 hosts:

• The Front End that clients contact.
• The Sequencer that assigns sequence numbers and multicasts requests.
• One instance each of NYK server, LON server, and TOK server, plus their respective replica managers RM_NYK, RM_LON, RM_TOK.

Key Points:

• All three city server processes run locally on Machine 1, each with its own Replica Manager.
• The Front End and Sequencer are also on Machine 1. Clients only connect to the FE, which delegates requests to the Sequencer for total ordering.

Machines 2 and 3 do not host the FE or Sequencer. They only run:

• The TOK server instance + RM_TOK
• The LON server instance + RM_LON
• The NYK server instance + RM_NYK

That way, each city code has a replica on Machine 1, 2, and 3. The Sequencer on Machine 1 sends requests to all nine processes which 3 per machine.

Key Points:

• Each machine has three DSMS processes TOK, LON, NYK plus three RMs—one per process.
• The Sequencer and Front End both reside on Machine 1, but they must send/receive data from all replicas on M1, M2, M3.

Summary

In this deployment:

• We have 9 total server processes .3 for each city code: TOK1,2,3; LON1,2,3; NYK1,2,3, across 3 physical machines.
• Each server has a local Replica Manager that monitors it.
• Machine 1 additionally runs the Front End and Sequencer, the only singletons in this system.
• Clients talk only to the FE. The FE delegates requests to the Sequencer, which ensures total ordering and multicasts to all 9 server processes.

B. Running the System

Once the system is up, point any Admin or Buyer Client to connect to the Front End on Machine 1. The FE will pass requests to the Sequencer, which multicasts them to all three Replicas in total order.

The same pseudo-code from the assignment 3. The only difference is how we assign host IP addresses and ports to each component. For example:

Machine1_IP = "192.168.0.10"
Machine2_IP = "192.168.0.11"
Machine3_IP = "192.168.0.12"

// On Machine 1
startReplica("A", "192.168.0.10", port=9000)
startRM("A", "192.168.0.10", port=9100)
startSequencer("192.168.0.10", port=7000)
startFrontEnd("192.168.0.10", port=7100, sequencerIP="192.168.0.10", seqPort=7000)

// On Machine 2
startReplica("B", "192.168.0.11", port=9000)
startRM("B", "192.168.0.11", port=9100)

// On Machine 3
startReplica("C", "192.168.0.12", port=9000)
startRM("C", "192.168.0.12", port=9100)

Each replica knows the IP address and ports of the other replicas and the Sequencer. The Front End is the only one the client ever contacts 192.168.0.10:7100 in this example.

pseudo-code for how a replica handle ordered requests:

ReplicaServer {
  sortedRequests = new PriorityQueue(... compare by seqNo ...);
  currentSeqExpected = 1;
  onReceiveRequest(seqNo, clientRequest):
      insert (seqNo, clientRequest) into sortedRequests

      while sortedRequests.peek() has seqNo == currentSeqExpected:
          nextReq = sortedRequests.poll()
          result = DSMS_logic(nextReq.clientRequest)
          sendResponseToFE(nextReq.seqNo, result)

          currentSeqExpected++
}

// DSMS_logic(request):
//   parse operation (addShare, removeShare, ....)
//   run the existing DSMS server code
//   return a string result

pseudo-code for the FE logic:

FrontEnd {
  handleClientRequest(clientRequest):
      seqNum = sendToSequencer(clientRequest)

      responses = []
      startTime = now()

      while not enoughResponses(responses):
          if responseArrivesFromReplica(rID, result):
              responses.add( (rID, result) )
              if checkMajorityOrTwoMatches(responses):
                  finalRes = getMajorityOrMatch(responses)
                  return finalRes
          if now() - startTime > TIMEOUT:
              identifyWhichReplicaDidNotRespond(responses)
              notifyAllRMs( crashedReplicaID )

      finalRes = getMajorityOrMatch(responses)
      return finalRes
}

C: RM

Each Replica Manager is bound to exactly one replica. The RM’s duties:

• Detect repeated software faults
• Detect crash failures

pseudo-code for the RM:

ReplicaManager(replicaID) {
  consecutiveFaults = 0

  onFaultSuspected(faultType):
      if faultType == "IncorrectResult":
          consecutiveFaults++
          if consecutiveFaults >= 3:
              stopReplica(replicaID)
              startNewReplica(replicaID)
              consecutiveFaults = 0
      else if faultType == "CrashSuspected":
          if confirmCrashWithPeers(replicaID):
              stopReplica(replicaID)
              startNewReplica(replicaID)
              consecutiveFaults = 0

  stopReplica(replicaID):
      // forcibly kill the process or call a cleanup method

  startNewReplica(replicaID):
      // spawn a fresh DSMS replica process
      // rejoin the group for receiving requests
}

D. Sequencer

The Sequencer enforces total order on all client requests. It:

• Maintains a global counter nextSeqNo.
• On receiving a request from the FE, increments nextSeqNo and reliably multicasts (nextSeqNo, request) to all replicas.
• Uses a simple reliability mechanism on UDP or a library for group communication to ensure no requests are lost.
• In this project we must assume that Sequencer never fail.

pseudo-code :

Sequencer {
  nextSeqNo = 1

  onReceiveRequestFromFE(clientRequest):
      seqNo = nextSeqNo
      nextSeqNo++
      for each replica in replicaList:
          sendUDPWithAck(replica.address, (seqNo, clientRequest))

  sendUDPWithAck(address, message):
      do {
        sendUDP(address, message)
        wait for ack or timeout
      } while (no ack received && retryCount < MAX_RETRIES)
}

5. Test Crash Failures

Here is a list of comprehensive test cases to verify the Distributed Share Market System (DSMS) functionality, covering both normal operations and crash scenarios process crash. The DSMS includes:

• Three replicated servers Replicas + Replica Managers
• One Sequencer
• One Front End (FE)

A. Normal Operation Test Cases

Below is a listing of core admin and buyer operations, covering typical edge cases. These assume all replicas are running and no process crashes occur.

B. Crash Failure Test Cases

In the following scenarios, at least one replica is forcibly terminated during or right after a request is broadcast by the Sequencer. We assume DSMS is in crash-failure tolerance mode not the software-fault mode. The system should continue to operate with the remaining replicas.

In each crash case, once the Replica Manager detects or is notified of the crash, it spins up a new instance or restarts so the system returns to having 3 replicas. The FE can respond to requests as soon as it has at least 2 matching responses from the healthy replicas.

C. Crash and Recovery

Below are two sample sequence diagrams illustrating crash scenarios and the subsequent recovery. The first focuses on a crash during an admin operation addShare, the second focuses on a buyer operation purchaseShare.

Conclusion

The test cases above normal operations + crash scenarios will verify correctness, robustness, and recovery in the actively replicated DSMS:

1. Normal Tests: Check that basic admin and buyer operations behave correctly under no failures.
2. Crash Tests: Forcibly terminate one replica during or just after receiving a request. Confirm the FE obtains matching results from the remaining replicas, and the Replica Manager replaces or restarts the crashed replica.

Together, these tests ensure the DSMS achieves the high availability goal under a single crash failure.

6. Verifying All Functionalities

After coding all the above Functionalities, need to verify:

• Admin functions: Log into the system with a valid Admin ID in the FE. Perform:
  • addShare
  • removeShare
  • listShareAvailability
• Buyer functions: Perform:
  • purchaseShare
  • getShares
  • sellShare
  • swapShares

Confirm each replica’s data capacities and buyer records remain consistent. If forcibly stop any single replica process on Machine 2 or Machine 3, the FE should continue returning correct results the other two healthy replicas.

7. Final Conclusion

This document outlines the architectural structure and key components that must be implemented or extended in the existing DSMS code to achieve software failure tolerance or high availability using active replication. These design ensure correctness, fault tolerance, and ease of debugging.