2017 © Pedro Peláez
 

silverstripe-module silverstripe-sqs-jobqueue

A module for interacting with AWS's Simple Queue Service (SQS)

image

silverstripe-australia/silverstripe-sqs-jobqueue

A module for interacting with AWS's Simple Queue Service (SQS)

  • Thursday, July 19, 2018
  • by nyeholt
  • Repository
  • 2 Watchers
  • 1 Stars
  • 3,081 Installations
  • PHP
  • 0 Dependents
  • 2 Suggesters
  • 4 Forks
  • 1 Open issues
  • 10 Versions
  • 2 % Grown

The README.md

SilverStripe SQS job queue

A module for sending and consuming SQS tasks. Can be configured to work as the trigger for queuejobs., (*1)

When used as the queuedjobs handler, there's a few slight changes to how queuedjobs are run - aside from not needing Cron jobs anymore., (*2)

  • On calling QueuedJobService->queueJob, a message is sent to SQS
  • A consumer picks up that message, then processes that job ID
  • All "type 1" aka immediate jobs are subsequently processed in that execution thread
  • If the added job is meant to be run in the future, the handler sets the job type to 'Scheduled'.
  • A separate SQS task runs that looks for any jobs of type "Scheduled" and executes those
  • That task re-queues itself for to run again in 30 seconds time
  • The "Scheduled" task runner will also look for any job currently sitting in the "wait" status; this is how paused jobs get picked up for further execution without needing to trigger another SQS message

Configuration for use as the queuejobs handler

---
Name: jobrunner
After: queuedjobs
---
SilverStripe\Core\Injector\Injector:
  QueueHandler:
    class: Symbiote\SqsJobQueue\Service\SqsQueueHandler
    properties:
      sqsService: '%$SqsService'
  SqsClient:
    class: Aws\Sqs\SqsClient
    constructor:
      connection_details: 
        region: ap-southeast-2
        version: latest
        credentials: 
          key: YourKey
          secret: YourSecret

That expects a queue to exist with the name 'jobqueue' - if the queue name is different,, (*3)

SilverStripe\Core\Injector\Injector:
  SqsService:
    properties:
      queueName: your-queue-name

Writing a task

Define a class with a method in it. This is the task runner; no need for any specific implementations., (*4)

eg, (*5)


namespace \Whatever\Class\Implements; class Work { public function doStuff() { } }

Add configuration to your project that binds the method name to the SqsService, (*6)

SilverStripe\Core\Injector\Injector:
  MyJobName: 
    class: \Whatever\Class\Implements\Work
  SqsService:
    properties:
      handlers: 
        doStuff: %$MyJobName

Triggering tasks

To trigger a task, the call, (*7)

$sqsService->sendSqsMessage(['args' => ['param1', 'param2']], 'taskName');, (*8)

for convenience, SqsService implements a __call() method that remaps a call like, (*9)

$sqsService->taskName($arg1, $arg2), (*10)

into, (*11)

$sqsService->sendSqsMessage(['args' => [$arg1, $arg2']], 'taskName');, (*12)

sendSqsMessage in turn converts this into a message structure such as, (*13)

$message = [
    'message' => [
        'args' => [$arg1, $arg2'],
        'handler' => 'taskName',
    ]
];

$sqsMessage = [
    'QueueUrl' => 'sqs://in.amazon'
    'MessageBody' => json_encode($message)
];

So, to manually trigger the messages, create the $sqsMessage structure from your own code, and send using $this->client->sendMessage($sqsMessage);, (*14)

Running

php vendor/symbiote/silverstripe-sqs-jobqueue/sqs-worker.php

Development environments

If you don't have SQS available, you can run a file-based queue system by swapping out the AWS queue for the file based SQS queue., (*15)

---
Name: local_sqs_config
After: '#sqs_config'
---
SilverStripe\Core\Injector\Injector:
  SqsService:
    properties:
      client: '%$FileSqsClient'
  FileSqsClient:
    class: Symbiote\SqsJobQueue\Service\FileBasedSqsQueue

By default, this will create serialised data in sqs-jobqueue/code/service/.queueus (configurable on the FileBasedSqsQueue class)., (*16)

Run sqs-worker as before., (*17)

Docker

If you're using https://github.com/symbiote/docker-runtime, this will spin up an sqsrunner container for you that runs the sqs-worker automatically., (*18)

Troubleshooting

So for a project, your config may look something like this where it defaults to the file-based queue system only for test environments:, (*19)

---
Name: prod_sqs
After: '#sqs_config'
---
SilverStripe\Core\Injector\Injector:
  Symbiote\SqsJobQueue\Service\SqsService:
    properties:
      client: '%$SqsClient'
  SqsClient:
    class: Aws\Sqs\SqsClient
    constructor:
      connection_details:
        region: ap-southeast-2
        version: latest
---
Name: dev_sqs
After: '#sqs_config'
Only:
  environment: test
---
SilverStripe\Core\Injector\Injector:
  Symbiote\SqsJobQueue\Service\SqsService:
    properties:
      client: '%$Symbiote\SqsJobQueue\Service\FileBasedSqsQueue'

In the production environment you will still need to provide the credentials using local config., (*20)

To get the file-based queue system working, your local config will need something like this:, (*21)

---
Name: jobrunner
After: 
  - queuedjobs
---
SilverStripe\Core\Injector\Injector:
  QueueHandler:
    class: Symbiote\SqsJobQueue\Service\SqsQueueHandler
    properties:
      sqsService: '%$Symbiote\SqsJobQueue\Service\SqsService'
  Symbiote\SqsJobQueue\Service\SqsService:
    properties:
      queueName: '%sqs_jobqueue_name%'
---
Name: sqs_location
After: '#sqs_config'
---
SilverStripe\Core\Injector\Injector:
  Symbiote\SqsJobQueue\Service\FileBasedSqsQueue:
    properties:
      queuePath: /var/www/html/mysite/fake-sqs-queues

The important part is making sure /var/www/html/mysite/fake-sqs-queues is writable since that's where your queued jobs will be written to., (*22)

Docker Testing

The simplest way to test this once all your configs are in place is to:, (*23)

  • Queue up a job (e.g. using the https://github.com/symbiote/silverstripe-queuedjobs/ module)
  • Ensure a file is written to /var/www/html/mysite/fake-sqs-queues
  • Run docker logs sqsrunner to see whether it picked up the job
  • View the queued jobs admin to ensure the job was completed

The Versions

19/07 2018

dev-master

9999999-dev

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs thrive symbiote

06/02 2018

2.2.2

2.2.2.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs thrive symbiote

06/02 2018

dev-fix-editing-descriptors

dev-fix-editing-descriptors

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs thrive symbiote

30/11 2017

2.2.1

2.2.1.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs thrive symbiote

15/09 2017

2.2.0

2.2.0.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs thrive symbiote

14/09 2017

2.1.0

2.1.0.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs thrive symbiote

01/07 2017

2.0.0

2.0.0.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs thrive symbiote

17/10 2016

1.0.2

1.0.2.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs

17/10 2016

1.0.1

1.0.1.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs

02/09 2016

1.0.0

1.0.0.0

A module for interacting with AWS's Simple Queue Service (SQS)

  Sources   Download

BSD-3-Clause

The Requires

 

aws silverstripe sqs queuedjobs