Back to tutorials

Devloping Locally

How to efficiently develop a swarm locally before deploying to Bytewax.

Developing Locally

In this example, we are going to leverage the bytewax DevSwarm capabilities to develop and test our swarm locally. This example will not deploy a swarm to Bytewax.

What is a DevSwarm?

DevSwarm is a class available in the Bytewax python package that can be used to test the functionality of bees as we develop and also for writing tests. When our swarm is running on bytewax, a swarm object is passed to it and the bee can use the publish() or respond() methods. This is similar to the methods available to the DevSwarm. The difference is that the publish() and respond() methods assign values to a DevSwarm object that you can retrieve in your local dev environment.

Importing the DevSwarm Class The DevSwarm is part of the bytewax SDK. We can import it at the top of our file.

Using the DevSwarm In order to simulate how a swarm object is passed to bees when they run remotely on bytewax, We instantiate a DevSwarm object that we can then pass to our bees we will be developing.

Running a DevSwarm In our swarm we have two Bees, verify and create-record-airtable. In verify we will be publishing data to the next bee with the publish() method if the email is verified and if not we will respond with the respond() method. If the user is verified we will then use the original data in the next bee to create a record in our Airtable base. Now that we have our DevSwarm object that we assigned to swarm we can then call our bees like functions (which they are) and pass the swarm to them. In the instance in which our email address is verified, our first bee calls the publish method on our swarm object. Our DevSwarm object has an attribute called published that will contain the result from the publish call we made in our bee that we can use as input for the next Bee.

A note on published and responded published and responded are lists and for each time a Bee publishes, the output will be appended. This will continue to add objects until it is cleared from memory, so when you are running a DevSwarm, you need to be explicit in calling the index of the output if you are passing it to the next bee. In our example we use -1 to always get the last element of the list.

Serialization in a DevSwarm The Gateway only accepts JSON serializeable objects, so you will have to keep that in mind when writing and running your Swarm locally.

Writing Bees

Like the python Bees we have written in other tutorials we decorate the Bee with register_bee, which takes an argument called name. It’s important that this name argument matches the one that we defined in the bee field in the swarm definition for this step. After successful verification we call swarm.publish() otherwise we use swarm.respond() to exit the swarm and return the results. Note that there is no return here and thats why we used the published object earlier with the DevSwarm.

You will notice there are environment variables used in these bees. For more information on using these, checkout the complete swarm tutorial or the detailed documentation

Running a Swarm

Once we have finished writing our bees we can import them to dev_swarm.py and run them as regular python functions. The output in the published or responded attributes can be printed out or passed to the next Bee. We are recreating the flow of a Swarm that would be defined in a swarm.yaml file. Running python dev_swarm.py will execute our bees and we can test them easily before deploying to Bytewax.

Success! 🐝 Congratulations! 🐝 We’ve successfully run a swarm locally.