Skip to main content

Transceiving Debug Data

The ASE framework allows for Over-the-Air (OtA) debugging and tuning due to its integrated capabilities to transceive (send and receive) data to and from the Rover. In this tutorial, we will describe the architecture behind the transceiver service and its dependencies. Finally, we will describe how to write a custom transceiver for your own data needs.

Schematic overview with just the transceiver process

Elias Groot

Elias Groot

Founding Member, Ex-Software Lead and Ex-Project Administrator

Besides capturing logs from stdout and stderr, the ASE framework provides advanced capabilities to capture, plot and debug inter-service communication. The first step to achieve this, is to agree on a language-agnostic representation for communication between these services. We have found this in the protobuf standard.

Compiling Protobuf​

From defining protobuf messages, the protobuf compiler can automatically generate code in virtually any language to handle serialization and deserialization for us. For example, we can define what RPM data looks like in an rpm.proto file:

rpm.proto
syntax = "proto3";
package protobuf_msgs;
option go_package = "ase/pb_outputs";

//
// This is the message format that a RPM sensor service can send out.
//
message RpmSensorOutput {
    MotorInformation leftMotor = 1; 
    MotorInformation rightMotor = 2; 
}

message MotorInformation {
    float rpm = 1; // RPM
    float speed = 2; // Speed in m/s, as computed from the RPM
}

Then, using protoc (the protobuf compiler), we can generate Go code to serialize a struct and send it to another service:

package main

import (
  // Import the Go code generated by protoc
  pb_outputs "github.com/VU-ASE/rovercom/v2/packages/go/outputs"
)

...

// Create an RPM message, as defined in rpm.proto
rpmMsg := pb_outputs.SensorOutput{
    Timestamp: uint64(time.Now().UnixMilli()),
    Status:    0,
    SensorId:  1,
    SensorOutput: &pb_outputs.SensorOutput_RpmOutput{
      RpmOutput: &pb_outputs.RpmSensorOutput{
        LeftMotor:  &pb_outputs.MotorInformation{
          Rpm: 100.0,
          Speed: 10.4,
        },
        RightMotor:  &pb_outputs.MotorInformation{
          Rpm: 200.0,
          Speed: 20.8,
        },
      },
    },
  }

// Send the message off to another service
// NB: this will call the .serialize() equivalent in Go
err = writeStream.Write(&rpmMsg)

Let's assume that the service that will receive this RPM data is not written in Go, but in Python instead. We turn to protoc again and create the according deserialization code.

# Import the Python code generated by protoc
import roverlib.rovercom as rovercom

...

# Generic way to write data (details omitted)
# NBL this will call the .deserialize() equivalent in Python
data = read_stream.Read()
if data is None:
  raise ValueError("Failed to read data")

if data.rpm_output:
  print(f"Got left rpm: {data.rpm_output.left_motor.rpm}") # "100.0"
  print(f"Got right speed: {data.rpm_output.right_motor.speed}") # "20.8"

Using protobuf, we hence have to define our communication definitions ("what can be sent") only once, in a set of .proto files. From this set, we can then generate the serialization and deserialization logic. This eliminates the need to write error-prone JSON parsers or custom structs for every language we want to support.

You can find the complete set of messages that a service can send in our rovercom repository. We automatically bundle these definitions with each roverlib, so you will never need to use protoc manually. By statically defining these messages using protobuf definitions, we enforce that if a message is sent, its type should be one of these definitions. While this limits your flexibility as a service author somewhat, it enables fast communication with convenient debugging.

Might the rovercom definitions be too limited for your needs, you can use generic debugging outputs and still enjoy our provided debugging tools. If that is still too limited, you can use the roverlib's .WriteBytes() and .ReadBytes() methods to just send raw bytes (for example if you want to communicate using JSON). It is then up to you to handle serialization and deserialization, and you will lose debugging functionality.

// Create JSON data instead of using rovercom
// This can **not** be debugged!
jsonData := map[string]interface{}{
  "foo": 42,
  "bar": "Hello, World!",
  "pi": 3.14,
}

// Serialize
jsonBytes, err := json.Marshal(jsonData)
if err != nil {
  log.Warn().Err(err).Msg("Could not serialize JSON data")
  return
}

// Send the message
err = writeStream.WriteBytes(jsonBytes)
if err != nil {
  log.Warn().Err(err).Msg("Could not write to actuator")
}

The transceiver Is a Special Service​

When the communication format (also often called the "over the wire" format) is established, we can start snooping on communication between any two services. The ASE framework allows for this because it is built on the flexible zeroMQ pub/sub pattern: when a service writes data, it acts as a publisher. Another service, that reads data, registers as a subscriber. Unlike more traditional "request response" communication, adding a subscriber is very efficient, and does not impose a noticeable latency on communication. It is designed for one-to-many communication.

This is great, because if we know how services communicate (protobuf), and where they do so (because roverd tells us), we can just create a new lightweight subscriber and start snooping! This part of the transceiver service is hence very straightforward. It just opens a read stream for every service it gets passed through roverd and then starts listening to all data that is published. It does so in a non-blocking way, to keep iterating over all available streams. The relevant source code can be found here.

Diagram of a transceiver service listening to all other services

convention

Keep in mind that the only reason that roverd passes all available inputs to the transceiver service, is because of its name, which is checked with case-sensitivity. If you want to create your own transceiver, put as: transceiver in your service.yaml definition. (See this example)

You can find the relevant source code in roverd here.

This works well for receiving data from other services, but that is only half the work carried out by the transceiver service. When roverd detects a transceiver in its pipeline, it also enables the tuning section for all other services in the pipeline. This allows bi-directional communication, where the transceiver can not only listen, but also talk to services. The roverlibs of these services will regularly check the tuning communication channel for updates (find the roverlib-go source code here).

These updates come in the form of a "tuning message", which is also defined in our rovercom repository here. Each roverlib maintains its own set of configuration variables. Once it receives a tuning message with a new structure, it will update its internal variables. Then, the next time the user code of a service calls .getFloat() or .getString(), it will receive the new version of this variable.

The tuning stream that all services read from, is redirected to the "transceiver" output stream as defined in its service.yaml. This is a convention, and you should follow it when building your own transceiver service. How the transceiver sends tuning data is shown here. How a roverlib receives and handles tuning data is highlighted here.

Sending Data to Your Device​

When data is "snooped" on by the transceiver, it still needs to be transported to your laptop somehow in order for you to be able to inspect and plot it. This is where the passthrough server and roverctl-web come in. These are automatically configured for you when you start roverctl in debug mode.

The idea is that the passthrough server acts as a proxy between one Rover and one or more laptops running roverctl-web. With this setup, multiple users can view camera streams and data plots from a Rover, without the need for the Rover to set up multiple connections or to do extra processing.

The transceiver flow to your laptop, in theory

While the passthrough server can be hosted on dedicated hardware in theory, it is often hosted on the same laptop where roverctl-web is launched in practice.

The transceiver flow to your laptop, in practice

Both the connections between the transceiver and the passthrough server, and the connection between the passthrough server and one or more roverctl-web instances are configured to run over webRTC. This is a common standard for receiving and sending realtime data in the browser (since raw UDP ports are inaccessible in the browser's sandboxed model). The messages that can be sent on these connections are also modelled in our rovercom library.

The passthrough server is integrated in the roverctl binary. The roverctl-web Svelte app is published as a Docker image (find the latest release here). When you launch roverctl in debug mode, it will resolve your local IP and inject it into the roverctl-web Docker environment. You can find the relevant source code here.

If you take a look at the transceiver service.yaml file. You will see that it has a configuration property with the name passthrough-address. This is the server address that the transceiver uses to open a connection to the passthrough server. When you click "enable debug mode" in roverctl-web, this will automatically overwrites the passthrough-address to your local IP, as found by roverctl. You can find the relevant source code here.

Plotting Transceiver Data​

Once roverctl-web receives the debug data from the passthrough server, it's tasked with plotting it accordingly. Because the possible message definitions are known from the rovercom library, we can decide what data to plot, and how. The relevant source code can be found here. At the moment of writing, we only extract numeric values to plot in a chart, but this could be extended by plotting a map of LiDAR data, or creating a heatmap of data.

Creating a Custom Transceiver​

Sometimes, you might want to add custom capabilities to the transceiver service. For example:

  • To capture data and write it to a CSV or Excel file
  • To debug data that communicates with custom message formats, beyond our provided rovercom definitions
  • To capture data and send it to cloud storage

Fortunately, because the transceiver is "just another ASE service", creating your own transceiver and debugging logic is really easy. In this short tutorial, we will describe how to write your own, and how to run it on the Rover. We will use Go, but you can use any supported roverlib for this.

  1. Initialize a new service using roverctl
roverctl service init go --name my-transceiver --source <YOUR_REPO_URL>
cd my-transceiver
tip

Give your transceiver a unique name using --name, so that you can distinguish it from the official ASE transceiver.

  1. Edit your main.go file use service.Inputs to open read streams for every service
main.go
func run(service roverlib.Service, configuration *roverlib.ServiceConfiguration) error {
  if configuration == nil {
    return fmt.Errorf("No configuration was provided. Do not know how to proceed")
  }

  // These are all services, passed by roverd
  // ! this only works if you call your service "transceiver" or use "as: transceiver" in your service.yaml!
  readStreams := make([]*roverlib.ReadStream, 0)
  log.Info().Msg("Starting transceiver with inputs:")
  for _, input := range service.Inputs {
    log.Info().Msgf("  - Service %s", *input.Service)
    for _, stream := range input.Streams {
      log.Info().Msgf("         - Stream %s at %s", *stream.Name, *stream.Address)
      stream := service.GetReadStream(*input.Service, *stream.Name)
      readStreams = append(readStreams, stream)
    }
  }
  log.Info().Msgf("There are %d inputs in total for this service", len(service.Inputs))
  ...
}

  1. Now all read streams in readStreams are ready to be read from. You can then write this data to a file, or upload it to cloud storage.
main.go
func run(service roverlib.Service, configuration *roverlib.ServiceConfiguration) error {
  if configuration == nil {
    return fmt.Errorf("No configuration was provided. Do not know how to proceed")
  }

  // These are all services, passed by roverd
  // ! this only works if you call your service "transceiver" or use "as: transceiver" in your service.yaml!
  readStreams := make([]*roverlib.ReadStream, 0)
  log.Info().Msg("Starting transceiver with inputs:")
  for _, input := range service.Inputs {
    log.Info().Msgf("  - Service %s", *input.Service)
    for _, stream := range input.Streams {
      log.Info().Msgf("         - Stream %s at %s", *stream.Name, *stream.Address)
      stream := service.GetReadStream(*input.Service, *stream.Name)
      readStreams = append(readStreams, stream)
    }
  }
  log.Info().Msgf("There are %d inputs in total for this service", len(service.Inputs))

  for i := range readStreams {
    stream := readStreams[i] // need to use index-based access to avoid copying the struct with lock

    // Blocking receive
    res, err := stream.Read()
    if err != nil {
      log.Err(err).Msg("Error reading from stream")
      continue
    }
    
    // Data received
    log.Info().Msgf("Received data from %v", res)
    // todo: write to file, or upload to cloud storage
  }

  return nil
}
Blocking calls

A call to stream.Read() is blocking by default. This means that if a stream has no data available, the debugger will wait until it has data available, before continuing to other streams. If you do not have realtime data requirements, this is probably fine. Yet, if you want to make sure that you capture all data as soon at is available, you should read from the service outputs in a non-blocking manner.

See this example to understand how to do this.

bring your own serialization

If you use an over-the-wire format that is different than our rovercom messages, you can use stream.ReadBytes() instead of stream.Read(). It is then up to you to do deserialization (such as JSON parsing).

  1. Update your service.yaml file to the example below. Make sure to follow the inputs, outputs and configuration properties exactly, even if you do not use them.
service.yaml
name: my-custom-transceiver # this can be anything, useful to distinguish your transceiver 
as: transceiver # this *must* be set to "transceiver" (case-sensitive)
author: just-a-student # can be anything
source: https://github.com/student/transceiver # can be anything
version: 1.0.0

# Update these commands if you have a different build/run process
commands:
  build: make build 
  run: ./bin/transceiver

# No 'official' dependencies, but this will be passed by roverd
inputs: []
outputs:
  - transceiver # leave this here, even if not used

# Leave these configuration values here, even if not used
configuration:
  - name: passthrough-address # address of the passthrough server to connect to
    value: http://192.168.0.115:7500
    type: string
  - name: connection-identifier # this is how we will identify ourselves to the passthrough server
    value: car
    type: string
  - name: data-channel-label # label for the data channel (passthrough server should use the same label)
    value: data
    type: string
  - name: control-channel-label # label for the control channel (passthrough server should use the same label)
    value: control
    type: string
  - name: use-wan # whether to use the WAN or not. Enabling this when passthrough server is accessed on LAN will break the connection
    value: 0
    type: number
  1. Upload your custom transceiver to your Rover
roverctl upload . -r <ROVER_NUMBER>
  1. Start roverctl-web
roverctl -r <ROVER_NUMBER>
  1. Enable your custom transceiver service. Do not click on "enable debug mode". This will enable the official ASE transceiver service, instead of yours!

The custom transceiver enabled in roverctl-web

  1. If your custom transceiver writes files to the Rover, you can SSH into it to retrieve them. Note that all files are written relative to the service installation directory
roverctl ssh -r <ROVER_NUMBER>
# In your SSH shell, navigate to your transceiver's service installation directory, in the .rover folder
cd .rover/just-a-student/my-transceiver/1.0.0
# Open your written files
ls
cat my_data.csv

This should give you an intuition for what you can do with the transceiver and how easy it is to replace the official transceiver service with your own implementation.