Production-Ready Beanstalkd with Laravel 4 Queues

Production-Ready Beanstalkd with Laravel 4 Queues

原文地址:http://fideloper.com/ubuntu-beanstalkd-and-laravel4

Note: TL;DR version at the bottom!

Queues are a great way to take some task out of the user-flow and put them in the background. Allowing a user to skip waiting for these tasks makes our applications appear faster, and gives us another opportunity to segment our application and business logic out further.

For example, sending emails, deleting accounts and processing images are all potentially long-running or memory-intensive tasks; They make great candidates for work which we can off-load to a queue.

Laravel can accomplish this with its Queue package. Specifically, I use the Beanstalkd work queue with Laravel.

Here’s how I set that up to be just about production-ready.

Note: I use Ubuntu for development and often in production. The following is accomplishsed in Ubuntu 12.04 Server LTS. Some instructions may differ for you depending on your OS

Here’s what we’ll cover:

  1. Laravel and Queues
  2. Installing Beanstalkd
  3. Churning through the queue with Supervisor

Laravel and Queues

Laravel makes using queues very easy. Our application, the "producer", can simply run something likeQueue::push('SendEmail', array('message' => $message)); too add a "job" to the queue.

On the other end of the queue is the code listening for new jobs and a script to process the job (collectively, the "workers"). This means that in addition to adding jobs to the queue, we need to set up a worker to pull from the stack of available jobs.

Here’s how that looks in Laravel. In this example, we’ll create an image-processing queue.

Install dependencies

As noted in the docs, Laravel requires the Pheanstalk package for using Beanstalkd. We can install this using Composer:

$ composer require pda/pheanstalk:dev-master

Create a script to process it

Once our PHP dependency in installed, we can begin to write some code. In this example, we’ll create aPhotoService class to handle the processing. If no method is specified, laravel assumes the class will have a fire() method. This is half of a worker – the code which does some processing.

<?php namespaceMyapp\Queue;classPhotoService{
    publicfunction fire($job, $data){
    // Minify, crop, shrink, apply filters or otherwise manipulate the image}}

Push a job to a Queue

When a user uploads an image, we’ll add a job to the queue so our worker can process it.

In Laravel, we’ll create a job by telling the Queue library what code will handle the job (in this case thefire() method inside of Myapp\Queue\PhotoService as defined above) and give it some data to work with. In our example, we simply pass it a path to an image file.

Queue::push('Myapp\Queue\PhotoService', array('image_path'=>'/path/to/image/file.ext'));

Process the jobs

At this point, we have code to process an image (most of a worker), and we’ve added a job to the queue. The last step is to have code pull a job from the queue.

This is the other half of a worker. The worker needs to both pull a job from the queue and do the processing. In Laravel, that’s split into 2 functionalities – Laravel’s queue listener, and the code we write ourselves – in this case, the PhotoService.

Laravel has some CLI tools to help with queues:

// Fire the latest job in the queue $ php artisan queue:work // Listen for new jobs in the queue// and fire them off one at a time// as they are created $ php artisan queue:listen

When not working with the “sync” driver, these tools are what you need to use in order to process the jobs in your queue. We run the queue:listen command to have laravel listen to the queue and pull jobs as they become available.

Let’s install Beanstalkd to see how that works.

By default, laravel will run queue jobs synchronously – that is, it runs the job at the time of creation. This means the image will be processed in the same request that the user created when uploading an image. That’s useful for testing, but not for production. We’ll make this asynchronous by introducing Beanstalkd.

Beanstalkd

Let’s install Beanstalkd:

# Debian / Ubuntu: $ sudo apt-get update $ sudo apt-get install beanstalkd

Note: You may be able to get a newer version of Beanstalkd by adding this PPA. Ubuntu 12.04 installs an older version of Beanstalkd.

Next, some quick configuration. The first thing we need to do is tell Beanstalkd to start when the system starts up or reboots. Edit /etc/default/beanstalkd and set START to “yes”.

$ sudo vim /etc/default/beanstalkd > START yes # uncomment

Then we can start Beanstalkd:

$ sudo service beanstalkd start # Alternatively: /etc/init.d/beanstalkd start

Now we can setup Laravel. In your app/config/queue.php file, set the default queue to ‘beanstalkd’:

'default'=>'beanstalkd',

Then edit any connection information you need to change. I left my configuration with the defaults as I installed it on the same server as the application.

'connections'=> array('beanstalkd'=> array('driver'=>'beanstalkd','host'=>'localhost','queue'=>'default',),),

Now when we push a job to the queue in Laravel, we’ll be pushing to Beanstalkd!

Installing Beanstalkd on a remote server

You may (read: should) want to consider installing Beanstalkd on another server, rather than your application server. Since Beantalkd is an in-memory service, it can eat up your servers resources under load.

To do this, you can install Beanstalkd on another server, and simply point your “host” to the proper server address, rather than localhost.

This leaves the final detail – what server runs the job? If you follow all other steps here, Supervisord will still be watching Laravel’s listener on your application server. You may want to consider running your job script (or even a copy of your application which has a job script) on yet another server whose job is purely to churn through Beanstalkd queue jobs. This means having a listener and working listener/job code on yet another server.

In fact, in a basic distributed setup, we’d probably have an application server (or 2, plus a load-balancer), a database server, a queue server and a job server!

Supervisord

Let’s say you pushed a job to Beanstalkd:

Queue::push('Myapp\Queue\PhotoService', array('image_path'=>'/path/to/image/file.ext'));

Now what? You might notice that it goes to Beanstalkd, but Myapp\Queue\PhotoService@fire() doesn’t seem to be getting called. You’ve checked your error logs, you see if the image was edited, and found that the the job is just “sitting there” in your Beanstalkd queue.

Beanstalkd doesn’t actually PUSH jobs to a script – instead, we need a worker to check if there are jobs available and ask for them.

This is what $ php artisan queue:listen does – It listens for jobs and runs them as they become available.

If you run that command, you’ll see your job being sent to code. If all goes well, your image will beproperly manipulated.

The question then becomes: How do I make php listen at all times? We need to avoid having to “supervise” that process manually. This is where Supervisord comes in.

Supervisord will watch our queue:listen command and restart it if it fails. Let’s see how to set that up.

First, we’ll install it:

# Debian / Ubuntu: $ sudo apt-get install supervisor

Next, we’ll configure it. We need to define a process to listen to.

$ sudo vim /etc/supervisor/conf.d/myqueue.conf

Add this to your new conf file, changing file paths and your environment as necessary:

[program:myqueue] command=php artisan queue:listen --env=your_environment directory=/path/to/laravel stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log redirect_stderr=true

We now have a process called “myqueue” which we can tell Supervisord to start and monitor.

Let’s do that:

$ sudo supervisorctl > reread # Tell supervisord to check for new items in /etc/supervisor/conf.d/> add myqueue # Add this process to Supervisord> start myqueue # May say "already started"

Now the “myqueue” process is on and being monitored. If our queue listener fails, Supervisord will restart the php artisan queue:listen --env=your_environment process.

You can check that it is indeed running that process with this command:

$ ps aux | grep php # You should see some output like this: php artisan queue:listen --env=your_environment sh -c php artisan queue:work --queue="default"--delay=0--memory=128--sleep --env=your_environment php artisan queue:work --queue=default--delay=0--memory=128--sleep --env=your_environment

Wrapping up

Now we have a full end-to-end queue working and in place!

  1. We create a script to process a queued job
  2. We installed Beanstalkd to act as the work queue
  3. We use Laravel to push jobs to our queue
  4. We use Laravel queue:listen to act as a worker and pull jobs from the queue
  5. We wrote some code to process a job from the queue
  6. We use Supervisord to ensure queue:listen is always listening for new jobs

Notes

  1. You might want to consider setting up log rotation on the Laravel and Supervisord logs
  2. You can read here for more information on setting up Supervisord on Ubuntu.
  3. Read the Laravel docs on queues to learn how and when to release or delete jobs.

TL;DR

For reference, just copy and paste the whole process from here:

$ sudo apt-get update $ sudo apt-get install -y beanstalkd supervisor $ sudo vim /etc/default/beanstalkd > START yes # uncomment this line $ sudo service beanstalkd start $ sudo vim /etc/supervisor/conf.d/myqueue.conf

Enter this, changing as needed:

[program:myqueue] command=php artisan queue:listen --env=your_environment directory=/path/to/laravel stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log redirect_stderr=true

Start Supervisord:

$ sudo supervisorctl > reread # Get available jobs> add myqueue > start myqueue

Read more on Supervisord here for info on supervisorctl.

 

另外还有一篇相关的不错的文章:http://www.lornajane.net/posts/2014/working-with-php-and-beanstalkd

转载于:https://www.cnblogs.com/argb/p/3729084.html

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/109830.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)


相关推荐

  • jdk8 hashmap线程安全吗_Python中的线程

    jdk8 hashmap线程安全吗_Python中的线程前言只要是对于集合有一定了解的一定都知道HashMap是线程不安全的,我们应该使用ConcurrentHashMap。但是为什么HashMap是线程不安全的呢,之前面试的时候也遇到到这样的问题,但是当时只停留在***知道是***的层面上,并没有深入理解***为什么是***。于是今天重温一个HashMap线程不安全的这个问题。首先需要强调一点,HashMap的线程不安全体现在会造成死循环、数据丢…

    2022年10月11日
  • 深度学习之pix2pix原理解析「建议收藏」

    今天我来给大家介绍一下基于CGAN的pix2pix模型,给大家简单讲解一下pix2pix的原理。这里我简单给大家CGAN,我用这张图给大家介绍CGAN的原理:CGAN和传统的GAN不同,传统GAN的生成器Generator只是单纯地传入一个随机噪声从而生成一张图片,那么CGAN和传统的GAN有所不同。CGAN的生成器不再只是一个随机噪声,而是传入一张图片z和一个控制条件y去生成一张假的图片…

  • 基因突变的实例:加拿大无毛猫

    基因突变的实例:加拿大无毛猫

    2021年11月15日
  • android广播注册方式_安卓广播接收器

    android广播注册方式_安卓广播接收器前面分析了Android系统的广播机制,从本质来说,它是一种消息订阅/发布机制。因此,使用这种消息驱动模型的第一步便是订阅消息;而对Android应用程序来说,订阅消息其实就是注册广播接收器。       接下来,我们继续分析Android应用程序是如何注册广播接收器的,以及把广播接收器注册到哪里去的。       在Android的广播机制中,ActivityManagerServi

  • #利用DialogResult属性实现主程序的打开当前窗口的关闭

    #利用DialogResult属性实现主程序的打开当前窗口的关闭利用DialogResult属性实现主程序的打开当前窗口的关闭首先介绍一下非模式化窗体show()和模式化窗体showdialog()的概念:两种方法都能打开显示窗体,1.非模式化窗体show()建立新窗口后仍能对原窗口进行操作,比如点击frm窗口上的“登录”按钮,会弹出FrmMain窗口,弹出后仍能对原窗口frm进行操作(移动,点击登录按钮等操作)2.模式化窗体showdialog()建…

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号