RedisGears Functions

A RedisGears function is a formal description of the processing steps in the data flow.

                                      +------------+
                                      | Function   |
                    +-------------+   | +--------+ |
                    | Input data  +-->+ | Reader | |
                    +-------------+   | +---+----+ |
                                      |     v      |
                                      | +---+----+ |
                                      | | Step 1 | |
                                      | +---+----+ |
                                      |     |      |
                                      |    ...     |
                                      |     v      |
                                      | +---+----+ |
                                      | | Step n | |
                                      | +---+----+ |
                                      |     v      |
                    +-------------+   | +---+----+ |
                    | Results     +<--+ | Action | |
                    +-------------+   | +--------+ |
                                      +------------+

A function always:

  1. Starts with a reader
  2. Operates on zero or more records
  3. Consists of zero or more operations (a.k.a steps)
  4. Ends with an action
  5. Returns zero or more results
  6. May generate zero or more errors

Execution

A function is executed by the RedisGears engine in one of two ways:

  • Batch : execution is immediate and on existing data
  • Event : execution is triggered by new events and on their data

The function's mode of execution is determined by its action. There are two types of actions:

  • Run : runs the function in batch
  • Register : registers the function to be triggered by events

When executed, whether as batch or event, the function's context is managed by the engine. Besides the function's logic, the context also includes its breakdown to internal execution steps, status, statistics, results and any errors encountered among other things.

Related commands

The following RedisGears commands are related to executing functions:

Execution ID

The execution of every function is internally assigned with a unique value called Execution ID .

The ID is a string value made of two parts that are delimited by a hyphen ('-'), as follows:

  1. Shard ID : the 40-bytes-long identifier of a shard in a cluster
  2. Sequence : an ever-increasing counter

Example: Execution IDs

When used in stand-alone mode, the Shard ID is set to zeros ('0') so the first execution ID would be:

0000000000000000000000000000000000000000-1

Whereas in cluster mode it could be:

a007297351b007297351c007297351d007297351-1

Execution Plan

Before executing the function the engine generates an Execution Plan . The plan consists of the basic steps that the engine will take to execute the function.

Execution Parallelization

When executed in a cluster, the execution plan is generated by the initiator. It is then shared and executed in parallel across all shards by default. The initiator's coordinator orchestrates the distributed operation.

Execution Status

The Execution Status describes the function's current execution status. It can be one of the following:

  • created : the execution has been created
  • running : the execution is running
  • done : the execution is done
  • aborted : the execution has been aborted
  • pending_cluster : initiator is waiting for all workers to finish
  • pending_run : worker is pending ok from initiator to execute
  • pending_receive : initiator is pending acknowledgement from workers on receiving execution
  • pending_termination : worker is pending termination messaging from initiator

The following diagram illustrates the status transitions:

              Initiator                                  Worker
       +---------------------+  execution plan   +---------------------+
       |             created +------------------>+ created             |
       +----------+----------+                   +----------+----------+
                  v                                         v
       +----------+----------+  acknowledgement  +----------+----------+
       |     pending_receive +<------------------+ pending_run         |
       +----------+----------+                   +---------------------+
                  v
       +----------+----------+  start execution  +---------------------+
       |             running +------------------>+ running             |
       +----------+----------+                   +----------+----------+
                  v                                         v
       +----------+----------+      results      +----------+----------+
       |     pending_cluster +<------------------+ pending_termination |
       +----------+----------+                   +---------------------+
                  v
       +----------+----------+     terminate     +---------------------+
       |                done +------------------>+ done                |
       +---------------------+                   +---------------------+

Registration

The representation of an event-driven function is called a registration.

Registrations are persisted in Redis' snapshots, a.k.a RDB files. This allows recovering both data and event handlers in the database in the event of failure.

Related commands

The following RedisGears commands are related to registering functions:

Registration ID

Every registration has a unique internal identifier that's referred to as its Registration ID . It is generated in the same manner as the Execution ID and despite appearing the same the two should not be confused.

Context Builder

RedisGears functions in Python always begin with a context builder - the class GearsBuilder .

Tip

GB() is an alias for GearsBuilder() .

It is intended to be used for brevity, increased productivity and the reduction of finger strain due to repetitive typing.

Python API

class GearsBuilder(reader='KeysReader', defaultArg='*', desc=None)
Arguments

  • reader : the function's reader
  • defaultArg : An optional argument that the reader may need. These are usually a key's name, prefix, glob-like or a regular expression. Its use depends on the function's reader type and action.
  • desc : an optional description

Examples

# Here's the default context builder being run
GearsBuilder().run()

# You can also do this
gb = GB()
gb.register()

Actions

An action is a special type of operation. It is always the function's final step.

Run

The Run action runs a function as a batch. The function is executed once and exits once the data is exhausted by its reader.

Trying to run more than one function in the same execution will fail with an error.

Example: multiple executions error

127.0.0.1:30001> RG.PYEXECUTE "GB().run()\nGB().run()"
(error) [... 'spam.error: Can not run more then 1 executions in a single script']

Execution is always asynchronous

Batch functions are always executed asynchronously by the RedisGears engine. That means that they are run in a background thread rather than by the main process of the Redis server.

Python API

class GearsBuilder.run(arg=None, convertToStr=True, collect=True)

Arguments

  • arg : An optional argument that's passed to the reader as its defaultArg . It means the following:
  • convertToStr : when True adds a map operation to the flow's end that stringifies records
  • collect : when True adds a collect operation to flow's end

Examples

# Runs the function
gb = GB()
gb.run()

Register

The Register action registers a function as an event handler. The function is executed each time an event arrives. Each time it is executed, the function operates on the event's data and once done is suspended until its future invocations by new events.

Python API

class GearsBuilder.register(convertToStr=True, collect=True, mode='async', onRegistered=None)

Arguments

  • convertToStr : when True adds a map operation to the flow's end that stringifies records
  • collect : when True adds a collect operation to flow's end

  • mode : the execution mode of the triggered function. Can be one of:

    • 'async' : execution will be asynchronous across the entire cluster
    • 'async_local' : execution will be asynchronous and restricted to the handling shard
    • 'sync' : execution will be synchronous and local
  • onRegistered : A function callback that's called on each shard upon function registration. It is a good place to initialize non-serializable objects such as network connections.

Notice that more argumets can be passed to the register function, those arguments are depends on the reader and specified for each reader on the readers page.

Examples

# Registers the function
gb = GB()
gb.register()

Results

The execution of a function yields zero or more result records. The result is made up of any records output by the function's last operation and just before its final action.

Results are stored within the function's execution context.

Related commands

The following RedisGears commands are related to getting results: