As trivial as this may sound, all software is about processing data. Yet, when you look at code written in a “traditional” programming language, the actual data flow is not readily visible. Instead, what you mainly see are just the control structures. The actual data flow only happens to occur at runtime, as a consequence from the control structures.
Flow-Based Programming (FBP) turns the view on code and data upside down. Here, the data flow is the first thing you look at; it is the main principle that defines the structure of your application. Processing of data happens within many small nodes that sit between the endpoints of data pipelines.
At this level, the processing nodes are just black boxes in a graphic flow diagram. The actual code hides within these boxes.
Flow-based programming and concurrency
Looking at an FBP diagram immediately raises two thoughts.
First, the data flow model is inherently concurrent. Data streams are independent of each other, and so are the nodes. Looks like optimal separation of concerns.
Second, a data flow looks darned close to channels and goroutines!
Do we have a natural match here? It seems tempting to build an FBP model directly on Go’s built-in concurrency concepts.
In fact, this has been done already.
Go FBP libraries
A quick search on GitHub reveals a handful of Go-based FBP projects, which I list here together with their own descriptions.
“This is quite a minimalistic implementation of Flow-based programming and several other concurrent models in Go programming language that aims at designing applications as graphs of components which react to data that flows through the graph.”
“SciPipe is an experimental library for writing scientific Workflows in vanilla Go(lang). The architecture of SciPipe is based on an flow-based programming like pattern in pure Go (…)
“A Flow-based Programming (FBP) micro-framework for Go (Golang).”
“A LabVIEW and TensorFlow Inspired Graph-Based Programming Environment for AI handled within the Go Programming Language.”
“A cancellable concurrent pattern for Go programming language”
“Language-Agnostic Programming Framework for Data-Driven Applications”
“Go implementation of Flow-based programming.”
(The last one actually relies on input from a graphical FBP editor (DrawFBP) that it turns into code stubs.)
To be fair, some of these libs seem not actively maintained anymore. I included them anyway as there is no single true approach to this, and each of these libs shows a different approach and focuses on different aspects.
I also most certainly left out a few FBP libs that I failed to find in the short time of researching this topic, so feel free to do some more research on your own.
A simple FBP flow
For today’s code, I picked the first of the libraries above,
trustmaster/goflow. It provides a quite readable syntax and comes with detailed documentation. (On the flipside,
goflow uses quite some reflection inside, which some of you might frown upon.)
Our sample code is an incarnation of the schematic FBP graph in the initial animation. Let’s turn the abstract nodes and data items into someting more tangible. For example, we could feed the network with sentences and let one node count the words in each sentence and the other all letters. The final node then prints the results.
First, we define the nodes. Each node is a struct with an embedded
flow.Component and input and output channels (at least one of each kind, except for a sink node that only has input channels).
Nodes can act on input by functions that are named after the input channels. For example, if an input channel is named “Strings”, the function that triggers on new input is called “OnStrings” by convention.
We define these nodes:
- A splitter that takes the input and copies it to two outputs.
- A word counter that counts the words (i.e., non-whitespace content) of a sentence.
- A letter counter that counts the letters (a-z and A-Z) of a sentence.
- A printer that prints its input.
None of these nodes knows about any of the other nodes, and does not need to.
How to get and run the code
go get the code. Note the
-d flag that prevents auto-installing the binary into
go get -d github.com/appliedgo/flow
cd to the source code directory.
Step 3. Run the binary.
go run ./flow
The output should look like:
Letters: 45 Words: 17 Words: 13 Letters: 36 Words: 8 Letters: 70
The unordered output shows that the nodes are indeed running asynchronously. Homework assignment: Add more info to the
count struct to allow the
printer node grouping the output by input sentence.
Still, although we were able to nicely describe our nodes and the connections between them, the resulting code is far from representing an intuitive view on the flow of data within the program. This should not be surprising. A textual representation rarely matches up with the intuitiveness of a graphic representation.
So where is the visual flow diagram editor, you ask?
There are indeed some options.
Just recently, an experimental visual Go environment has been presented to the public – Shenzhen Go. (Careful though – “experimental” means exactly this.)
Some nodes contain configurable standard actions, others contain Go code that reads from input channels and writes to output channels (unless the node is a sink).
go-fbp and DrawFBP
If you want a graphic editor now and don’t want to wait until Shenzhen Go is production ready, have a look at themalkolm/go-fbp. This project generates Go code from the output of a graphical FBP editor called DrawFBP (a Java app). (Disclaimer: I have tested neither
go-fbp nor DrawFBP.)