idci/task-bundle

Symfony task bundle

IDCITaskBundle

The purpose of this bundle is to add an easy way to asynchronously process CPU-intensive scripts. You might be used to cron jobs to solve this kind of problems. This bundle use RabbitMq which solve those issues more efficiently and more effectively. A CPU-intensive script is called an action. To keep traces on actions, this bundle uses monolog to store the action logs in MongoDB. If you want to understand how this bundle works, you need the basics about RabbitMQ., _(*1)

• Introduction
  • Glossary
  • Simple schema
  • Lifecycle of a task with RabbitMq
• Installation
• Run the tests
• How to create an extract rule service
• How to create an action service
• How to create a task configuration object
• Persist a task configuration
• Use the task configuration form type
• How to run tasks
• How to scale your application to run tasks concurrently
• How to separate actions in different applications
• Example
• UML

Introduction

Glossary

• An extract_rule refers to a symfony service that will retrieve an array of data. A task will be created for each item of this array.
• An action is a service doing any work you want. It can be triggered by other previous actions in a definable and predictable order (composing what we call a workflow).
• A workflow refers to the way actions are linked together. You can use conditions depending of the results of previous actions to trigger an action or another.
• A task refers to multiple actions linked together for one extracted data. It's a mongo document, and can be used to resume actions if they failed0
• With all these, we can compose a task configuration to define how tasks are created and processed.

Simple schema

Here is a simple schema that will help get a picture of how tasks are created and processed. Each arrow can represent a RabbitMq message that is sent and will be consumed., _(*2)

, _(*3)

Lifecycle of a task with rabbitmq

A task can be created in 2 different ways. The processor service is the entrypoint of this bundle. See the "How to run tasks" part of the documentation for more details., _(*4)

A single action with some (or no) data to process, _(*5)

, _(*6)

• 1 - The task producer send a message with the name of an action and some data (or no data).
• 2 - The task consumer create a single task from these data and use the action producer to send the created task.
• 3 - The action consumer retrieve the task to run the single action.

A complex configuration with a workflow and multiple actions, _(*7)

, _(*8)

• 1 - The extract_rule producer send a message to RabbitMQ with the name of the extract rule service.
• 2 - The extract rule consumer extract the data and use the task producer to send a message with the data array.
• 3 - The task consumer create the tasks for each item in the extracted data array, and use the action producer to send the created task.
• 4 - The action consumer read the configuration to run the action depending on the workflow. An action can lead to another one or just end the process.

Installation

Add dependencies in your composer.json file:, _(*9)

"require": {
    ...,
    "idci/task-bundle": "dev-master",
}

Enable bundles in your application kernel:, _(*10)

<?php
// app/AppKernel.php

public function registerBundles()
{
    $bundles = array(
        // ...
        new Symfony\Bundle\TwigBundle\TwigBundle(),
        new Doctrine\Bundle\MongoDBBundle\DoctrineMongoDBBundle(),
        new OldSound\RabbitMqBundle\OldSoundRabbitMqBundle(),
        new IDCI\Bundle\TaskBundle\IDCITaskBundle(),
    );
}

Import the bundle configuration:, _(*11)

# app/config/config.yml

imports:
    - { resource: @IDCITaskBundle/Resources/config/config.yml }

Update your parameters.yml file, _(*12)

application_name: 'Task Runner'
task_database_host: localhost
task_database_port: ~
task_database_name: task_manager
task_database_user: dbuser
task_database_password: dbpwd
task_rabbitmq_host: localhost
task_rabbitmq_port: 5672
task_rabbitmq_user: user
task_rabbitmq_password: password
task_rabbitmq_lazy: false
task_rabbitmq_vhost: /
task_mongo_database_name: task
task_mongo_database_host: 'mongodb://127.0.0.1:27017'

Run the tests

Install bundle dependencies:, _(*13)

$ make composer-update

To execute unit tests:, _(*14)

$ make phpunit