Cloudify properties and intrinsic functions

Passing data between nodes in Cloudify blueprints is one of the basic things blueprint author might want to do. In this post, we will have a look at how can we exchange and manipulate data in Cloudify blueprints, with emphasis on using intrinsic functions and their limitations.

Seting up our sandbox

If you would like to play along with our examples, you will need Cloudify’s command line tool cfy installed. In order to keep our system clean, we will install cfy in a virtual environment by running

$ cd /tmp
$ virtualenv -p python2 venv
(venv) $ . venv/bin/activate
(venv) $ pip install 'cloudify<4'

Make sure you create a virtual environment for python 2, since Cloudify does not support python 3. Also, we will be using latest Cloudify in 3.x release. All of this can be done using Cloudify 4, but commands need to be changed a bit. Those modifications are left to the reader as an exercise.

Another thing that we will need is blueprint, that is available in this repo. We will clone it into /tmp/blueprint by running

(venv) $ git clone --branch init \
  https://github.com/xlab-si/cloudify-intrinsic-functions blueprint
(venv) $ cd blueprint

And with this being done, we are ready to start running some cfy commands.

Initial blueprint incarnation

At the end of this post, we would like to have a blueprint that deploys node_a and node_b components, where node_b needs node_a to be running. Additionally, node_b also needs some information about node_a at installation time. But for starters, we will write down the blueprint that simply creates both components.

tosca_definitions_version: cloudify_dsl_1_3

imports:
  - http://www.getcloudify.org/spec/cloudify/3.4.2/types.yaml


node_types:

  type_a:
    derived_from: cloudify.nodes.Root
    properties:
      property_a: { default: property_a_value }
    interfaces:
      cloudify.interfaces.lifecycle:
        create:
          implementation: scripts/create_a.py
          executor: central_deployment_agent

  type_b:
    derived_from: cloudify.nodes.Root
    properties:
      property_b1: { required: true }
      property_b2: { required: true }
    interfaces:
      cloudify.interfaces.lifecycle:
        create:
          implementation: scripts/create_b.py
          executor: central_deployment_agent


node_templates:

  node_a:
    type: type_a

  node_b:
    type: type_b
    properties:
      property_b1: some data from node_a
      property_b2: more data from node_a

This is a fairly simple blueprint that we can install locally by running

(venv) $ cfy local install -p blueprint.yaml
... Starting 'install' workflow execution
... [node_b_foxvkq] Creating node
... [node_a_ctosv3] Creating node
... [node_b_foxvkq.create] Sending task 'script_runner.tasks.run'
... [node_a_ctosv3.create] Sending task 'script_runner.tasks.run'
... [node_b_foxvkq.create] Task started 'script_runner.tasks.run'
... [node_b_foxvkq.create] INFO: Creating node of type B
... [node_b_foxvkq.create] INFO: property_b1 = some data from node_a
... [node_b_foxvkq.create] INFO: property_b2 = more data from node_a
... [node_b_foxvkq.create] Task succeeded 'script_runner.tasks.run'
... [node_a_ctosv3.create] Task started 'script_runner.tasks.run'
... [node_a_ctosv3.create] INFO: Creating node of type A
... [node_a_ctosv3.create] Task succeeded 'script_runner.tasks.run'
... [node_a_ctosv3] Configuring node
... [node_b_foxvkq] Configuring node
... [node_b_foxvkq] Starting node
... [node_a_ctosv3] Starting node
... 'install' workflow execution succeeded

Output of the command clearly indicates that we have some work left to do, since node_b has been created before node_a.

Adding order to execution

Official documentation states that execution order can be controlled by specifying relationships between nodes. So let us do this by adding relationships key to our node_b definition. This will take care of order issue that we had before.

What we also need to do is retrieve some information from node_a and use it while creating node_b. To do this, we will use get_property and get_attribute intrinsic functions. And if you are wondering where is attribute_a coming from, it is set by the scripts/create_a.py. Unfortunately, Cloudify’ blueprint DSL does not allow us to declare attributes, so we need to grep around a bit to find the information we need.

After our modifications, node_b template looks like this:

node_b:
  type: type_b
  properties:
    property_b1: { get_property:  [ node_a, property_a  ] }
    property_b2: { get_attribute: [ node_a, attribute_a ] }
  relationships:
    - type: cloudify.relationships.connected_to
      target: node_a

If we try installing updated blueprint again, we get

(venv) $ git checkout order
(venv) $ cfy local install -p blueprint.yaml
... Starting 'install' workflow execution
... [node_a_aky88e] Creating node
... [node_a_aky88e.create] Sending task 'script_runner.tasks.run'
... [node_a_aky88e.create] Task started 'script_runner.tasks.run'
... [node_a_aky88e.create] INFO: Creating node of type A
... [node_a_aky88e.create] Task succeeded 'script_runner.tasks.run'
... [node_a_aky88e] Configuring node
... [node_a_aky88e] Starting node
... [node_b_fe0sl4] Creating node
... [node_b_fe0sl4.create] Sending task 'script_runner.tasks.run'
... [node_b_fe0sl4.create] Task started 'script_runner.tasks.run'
... [node_b_fe0sl4.create] INFO: Creating node of type B
... [node_b_fe0sl4.create] INFO: property_b1 = property_a_value
... [node_b_fe0sl4.create] INFO: property_b2 = {u'get_attribute': [u'node_a', u'attribute_a']}
... [node_b_fe0sl4.create] Task succeeded 'script_runner.tasks.run'
... [node_b_fe0sl4] Configuring node
... [node_b_fe0sl4] Starting node
... 'install' workflow execution succeeded

Node creation order is OK now, and get_property function behaves as expected, but get_attribute call was not evaluated and made it through verbatim. Well, let us sort this out.

But wait, before we tackle this problem, let us have a quick look at the creation scripts to see what they are actually doing. Creating node_a is as simple as it gets, since all we do is log (un)informative message and set the attribute_a.

from cloudify import ctx

ctx.logger.info("Creating node of type A")
ctx.instance.runtime_properties["attribute_a"] = "attribute_a_value"

Creating node_b is a bit more interesting, since we also access two properties of the node.

from cloudify import ctx

ctx.logger.info("Creating node of type B")

props = ctx.node.properties
ctx.logger.info("property_b1 = {}".format(props["property_b1"]))
ctx.logger.info("property_b2 = {}".format(props["property_b2"]))

Having seen these scripts, we can resume our quest to get the get_attribute function call sorted out.

Down the rabbit hole

Since the intrinsic functions’ documentation does not list our use case among the examples, we must assume that direct property access does not evaluate the get_attribute function.

From the examples in the documentation we can see that we can use it when adding inputs to lifecycle operation. So let us try this. We modify our node_b template to this:

node_b:
  type: type_b
  properties:
    property_b1: { get_property:  [ node_a, property_a  ] }
    property_b2: { get_attribute: [ node_a, attribute_a ] }
  relationships:
    - type: cloudify.relationships.connected_to
      target: node_a
  interfaces:
    cloudify.interfaces.lifecycle:
      create:
        implementation: scripts/create_b.py
        executor: central_deployment_agent
        inputs:
          input_b2: { get_attribute: [ node_a, attribute_a ] }

This time, we also need to modify the creation script, since we need to handle the input:

from cloudify import ctx
from cloudify.state import ctx_parameters as inputs

ctx.logger.info("Creating node of type B")

props = ctx.node.properties
ctx.logger.info("property_b1 = {}".format(props["property_b1"]))
ctx.logger.info("input_b2 = {}".format(inputs["input_b2"]))

If we now run the installation again, we get what we were after:

(venv) $ git checkout inputs
(venv) $ cfy local install -p blueprint.yaml
...
... [node_b_t0mdbd] Creating node
... [node_b_t0mdbd.create] Sending task 'script_runner.tasks.run'
... [node_b_t0mdbd.create] Task started 'script_runner.tasks.run'
... [node_b_t0mdbd.create] INFO: Creating node of type B
... [node_b_t0mdbd.create] INFO: property_b1 = property_a_value
... [node_b_t0mdbd.create] INFO: input_b2 = attribute_a_value
... [node_b_t0mdbd.create] Task succeeded 'script_runner.tasks.run'
... [node_b_t0mdbd] Configuring node
... [node_b_t0mdbd] Starting node
... 'install' workflow execution succeeded

But at what cost! We now need to configure operation inputs on each node template instead of hiding this mess inside the node type. Sure we can do better, right?

Digging a hole inside the rabbit hole

By the fundamental theorem of software engineering, there must be another level of indirection that will resolve the conundrum that we got ourselves into. And indeed there is.

Our initial output contained a line:

INFO: property_b2 = {u'get_attribute': [u'node_a', u'attribute_a']}

So the function call is still there, we just need to trick Cloudify into evaluating this for us without resorting to duplication. And we can do this by resetting node_b template back to previous state:

node_b:
  type: type_b
  properties:
    property_b1: { get_property:  [ node_a, property_a  ] }
    property_b2: { get_attribute: [ node_a, attribute_a ] }
  relationships:
    - type: cloudify.relationships.connected_to
      target: node_a

Next, we employ another level of indirection inside the type_b definition:

type_b:
  derived_from: cloudify.nodes.Root
  properties:
    property_b1: { required: true }
    property_b2: { required: true }
  interfaces:
    cloudify.interfaces.lifecycle:
      create:
        implementation: scripts/create_b.py
        executor: central_deployment_agent
        inputs:
          input_b2: { default: { get_property: [ SELF, property_b2 ] } }

The trick is in the last line. Since we know that get_attribute function call is evaluated before the operation is executed, we use the get_property function to retrieve the “stored” get_attributes call and evaluate it.

And just to verify our hypothesis, here is the installation log:

(venv) $ git checkout master
(venv) $ cfy local install -p blueprint.yaml
...
... [node_b_h8yzi9] Creating node
... [node_b_h8yzi9.create] Sending task 'script_runner.tasks.run'
... [node_b_h8yzi9.create] Task started 'script_runner.tasks.run'
... [node_b_h8yzi9.create] INFO: Creating node of type B
... [node_b_h8yzi9.create] INFO: property_b1 = property_a_value
... [node_b_h8yzi9.create] INFO: input_b2 = attribute_a_value
... [node_b_h8yzi9.create] Task succeeded 'script_runner.tasks.run'
... [node_b_h8yzi9] Configuring node
... [node_b_h8yzi9] Starting node
... 'install' workflow execution succeeded

Final thoughts

As we learned today, intrinsic functions can be a bit counter-intuitive to work with, but if we stick to the following rule, most of the problem will resolve itself: life-cycle operation implementation should get all of its data as inputs and should not access properties and/or attributes directly. And by following this rule, we will also gain ability to use concat intrinsic function in properties for free;)

There are other, more advanced and flexible, ways of passing data around the blueprint (like creating custom relationship type that copies attributes between node instances, etc.), but sometimes this approach can simplify our implementation quite a bit.

Cheers!