tarantool/jobserver

A skeleton application for creating and processing background jobs.

JobServer

JobServer is a skeleton repository used for creating and processing background jobs backed by Tarantool. It contains configuration files and folders you will need for quick start from scratch., _(*1)

Installation

The recommended way to create a new application is through Composer:, _(*2)

composer create-project tarantool/jobserver -s dev

Quick start

First, create your own docker-compose.override.yml file by copying docker-compose.override.yml.dist (or docker-compose.override.yml.full.dist if you want to test the full setup including a Tarantool cluster with automatic failover and monitoring tools) and customize to your needs. Do the same for .env.dist and all *.dist files located in app/config and res., _(*3)

Then, browse to the project directory and execute this command:, _(*4)

docker-compose up -d

After the command has completed successfully, you'll have a running server ready to execute jobs. If you open a log file in follow mode (tail -f var/log/workers.log), you'll see something like the following:, _(*5)

[2017-11-19 00:00:23] default:worker.DEBUG: Idling... [] []
[2017-11-19 00:00:24] default:worker.DEBUG: Idling... [] []
[2017-11-19 00:00:25] default:worker.DEBUG: Idling... [] []

Let's now try to add a task to the queue. This repository comes with a demo job that writes a greeting to the log. By running the following command:, _(*6)

docker-compose exec worker ./jobserver queue:put default -H tarantool \
    '{"payload": {"service": "greet", "args": {"name": "foobar"}}}'

we add a task to the default queue with a job payload, where greet is a job name and foobar is an argument passing to a job callable., _(*7)

Now in the log you will see that the job is executed:, _(*8)

[2017-11-19 00:00:32] jobserver.INFO: HELLO FOOBAR [] []
[2017-11-19 00:00:33] default:worker.DEBUG: Idling... [] []
[2017-11-19 00:00:34] default:worker.INFO: Task #0 was successfully processed. {"payload":{"args":{"name":"foobar"},"service":"greet"}} []

Also, you can run the job directly in the console, bypassing the queue:, _(*9)

docker-compose exec worker ./jobserver -vvv handler:greet foobar

To be able to run a job from the console, you need to write an adapter for the symfony command and register it in app/config/commands.php. This is how the adapter looks like for GreetHandler: GreetCommand., _(*10)

To see a list of all registered commands, run:, _(*11)

docker-compose exec worker ./jobserver

Tarantool

To use a web interface for Tarantool, open your browser and access the http://localhost:8001 address (make sure that Docker containers are running)., _(*12)

To get into Tarantool console as admin on a running Docker container, execute:, _(*13)

docker-compose exec tarantool tarantoolctl connect /var/run/tarantool/tarantool.sock

On a server:, _(*14)

sudo tarantoolctl enter jobserver_instance

On the server as a job queue user:, _(*15)

sudo tarantoolctl connect $TNT_JOBQUEUE_USER:$TNT_JOBQUEUE_PASSWORD@$TNT_JOBQUEUE_HOST:3301

Monitoring

Open your browser and access:, _(*16)

• Prometheus
• Alert manager
• Grafana

, _(*17)

Make sure to use a copy of docker-compose.override.yml.full.dist to have all monitoring containers running., _(*18)

Testing

docker-compose exec worker vendor/bin/phpunit

Debugging

To debug a job runner, first, stop the worker container, _(*19)

docker-compose stop worker

Then, start listening for php debug connections and then execute:, _(*20)

LOCAL_IP=<your-local-ip> docker-compose run --rm worker bash -c ' \
    TNT_JOBQUEUE_PASSWORD=jobserver \
    vendor/bin/jobqueue run default \
    --config app/config/jobqueue.php \
    --executors-config app/config/executors.php \
    --user jobserver \
    --host tarantool \
'

Check this manual to learn how to debug multiple processes (for example, the runner and background jobs) simultaneously in PhpStorm., _(*21)

License

The library is released under the MIT License. See the bundled LICENSE file for details., _(*22)