Cloud

In this hands-on example we send end-to-end encrypted messages through Instaclustr.

Ockam encrypts messages from a Producer all-of-the-way to a specific Consumer. Only that specific Consumer can decrypt these messages. This guarantees that your data cannot be observed or tampered with as it passes through Instaclustr or the network where it is hosted. The operators of Instaclustr can only see encrypted data in the network and in service that they operate. Thus, a compromise of the operator's infrastructure will not compromise the data stream's security, privacy, or integrity.

To learn how end-to-end trust is established, please read: “How does Ockam work?”

Run

This example requires Bash, Git, jq, Curl, Docker, and Docker Compose. Please set up these tools for your operating system, then run the following commands:

# Clone the Ockam repo from Github.
git clone --depth 1 https://github.com/build-trust/ockam && cd ockam

# Navigate to this example’s directory.
cd examples/command/portals/kafka/instaclustr/docker/

# Run the example, use Ctrl-C to exit at any point.
./run.sh

If everything runs as expected, you'll see the message: The example run was successful 🥳

Walkthrough

The run.sh script, that you ran above, and its accompanying files are full of comments and meant to be read. The example setup is only a few simple steps, so please take some time to read and explore.

This example requires Instaclustr Username and API key to create a kafka cluster to use for the example. You can create a trial account at https://www.instaclustr.com/platform/managed-apache-kafka/

Administrator

• The run.sh script calls the run function which invokes the enroll command to create an new identity, sign into Ockam Orchestrator, set up a new Ockam project, make you the administrator of this project, and get a project membership credential.

• The run function then generates three new enrollment tickets, each valid for 10 minutes, and can be redeemed only once. The first ticket is meant for the Ockam node that will run in Instaclustr Operator’s network. The second and third tickets are meant for the Consumer and Producer, in the Ockam node that will run in Application Team’s network.

• Run function authorizes to instaclustr using Username and API Key and setup a free trial Instaclustr Kafka Cluster to create and configure a kafka cluster

  • Upon logged in to Instaclustr console, Account API keys can be created from the console by going to gear icon to the top right > Account Settings > API Keys. Create a Provisioning API key and note it down.

  • Alternative to entering the username and API key, you can export them as environment variables INSTACLUSTR_USER_NAME and INSTACLUSTR_API_KEY

• cluster_manager.sh gets invoked which:

  • Creates a trial cluster.

  • Creates a user for kafka consumer and producer to use.

  • Setup firewall rules to access the cluster from the machine running the script.

  • Obtains the bootstrap server public address.

• In a typical production setup, an administrator or provisioning pipeline generates enrollment tickets and gives them to nodes that are being provisioned. In our example, the run function is acting on your behalf as the administrator of the Ockam project. It provisions Ockam nodes in Instaclustr Operator’s network and Application Team’s network, passing them their tickets using environment variables.

• The run function takes the enrollment tickets, sets them as the value of an environment variable, and invokes docker-compose to create Instaclustr Operator’s and Application Teams’s networks.

Instaclustr Operator

# Create a dedicated and isolated virtual network for instaclustr_operator.
networks:
  instaclustr_operator:
    driver: bridge

• Instaclustr Operator’s docker-compose configuration is used when run.sh invokes docker-compose. It creates an isolated virtual network for Instaclustr Operator.

• In the same network, docker compose starts a Kafka UI , connecting directly to ${BOOTSTRAPSERVER}:9092. The console will be reachable throughout the example at http://127.0.0.1:8080.

• Docker compose starts an Ockam node in a container described by ockam.yaml, embedded in the script. The node will automatically create an identity, enroll with your project using the ticket passed to the container, and set up Kafka outlet with the bootstrap server details passed to the container

• The Ockam node then uses this identity and membership credential to authenticate and create a relay in the project, back to the node, at relay: instaclustr. The run function gave the enrollment ticket permission to use this relay address.

Application Teams

# Create a dedicated and isolated virtual network for application_team.
networks:
  application_team:
      driver: bridge

• Application Teams’s docker-compose configuration is used when run.sh invokes docker-compose. It creates an isolated virtual network for Application Teams. In this network, docker compose starts a Kafka Consumer container and a Kafka Producer container.

• The Kafka consumer node container is created using this dockerfile and this entrypoint script. The consumer enrollment ticket from run.sh is passed to the container via environment variable.

• When the Kafka consumer node container starts in the Application Teams network, it runs its entrypoint. The entrypoint creates the Ockam node described by ockam.yaml, embedded in the script. The node will automatically create an identity, enroll with your project, and setup Kafka inlet.

• Next, the entrypoint at the end executes the consumer commands, which launches a Kafka consumer waiting for messages in the demo topic. Once the messages are received, they are printed out.

• In the producer container, the process is analogous, once the Ockam node is set up the command within docker-compose configuration launches a Kafka producer that sends messages.

• Both consumer and producer uses kafka.config that has credentials of the kafka user created when setting up the cluster

• You can view the Kafak UI available at http://127.0.0.1:8080 to see the encrypted messages

Recap

We sent end-to-end encrypted messages through Instaclustr.

Messages are encrypted with strong forward secrecy as soon as they leave a Producer, and only the intended Consumer can decrypt those messages. Instaclustr and other Consumers can only see encrypted messages.

All communication is mutually authenticated and authorized. Keys and credentials are automatically rotated. Access can be easily revoked.

Cleanup

To delete all containers, images and instaclustr cluster:

./run.sh cleanup