Production-Ready Beanstalkd with Laravel 4 Queues

Production-Ready Beanstalkd with Laravel 4 Queues

原文地址:http://fideloper.com/ubuntu-beanstalkd-and-laravel4

Note: TL;DR version at the bottom!

Queues are a great way to take some task out of the user-flow and put them in the background. Allowing a user to skip waiting for these tasks makes our applications appear faster, and gives us another opportunity to segment our application and business logic out further.

For example, sending emails, deleting accounts and processing images are all potentially long-running or memory-intensive tasks; They make great candidates for work which we can off-load to a queue.

Laravel can accomplish this with its Queue package. Specifically, I use the Beanstalkd work queue with Laravel.

Here’s how I set that up to be just about production-ready.

Note: I use Ubuntu for development and often in production. The following is accomplishsed in Ubuntu 12.04 Server LTS. Some instructions may differ for you depending on your OS

Here’s what we’ll cover:

  1. Laravel and Queues
  2. Installing Beanstalkd
  3. Churning through the queue with Supervisor

Laravel and Queues

Laravel makes using queues very easy. Our application, the "producer", can simply run something likeQueue::push('SendEmail', array('message' => $message)); too add a "job" to the queue.

On the other end of the queue is the code listening for new jobs and a script to process the job (collectively, the "workers"). This means that in addition to adding jobs to the queue, we need to set up a worker to pull from the stack of available jobs.

Here’s how that looks in Laravel. In this example, we’ll create an image-processing queue.

Install dependencies

As noted in the docs, Laravel requires the Pheanstalk package for using Beanstalkd. We can install this using Composer:

$ composer require pda/pheanstalk:dev-master

Create a script to process it

Once our PHP dependency in installed, we can begin to write some code. In this example, we’ll create aPhotoService class to handle the processing. If no method is specified, laravel assumes the class will have a fire() method. This is half of a worker – the code which does some processing.

<?php namespaceMyapp\Queue;classPhotoService{
    publicfunction fire($job, $data){
    // Minify, crop, shrink, apply filters or otherwise manipulate the image}}

Push a job to a Queue

When a user uploads an image, we’ll add a job to the queue so our worker can process it.

In Laravel, we’ll create a job by telling the Queue library what code will handle the job (in this case thefire() method inside of Myapp\Queue\PhotoService as defined above) and give it some data to work with. In our example, we simply pass it a path to an image file.

Queue::push('Myapp\Queue\PhotoService', array('image_path'=>'/path/to/image/file.ext'));

Process the jobs

At this point, we have code to process an image (most of a worker), and we’ve added a job to the queue. The last step is to have code pull a job from the queue.

This is the other half of a worker. The worker needs to both pull a job from the queue and do the processing. In Laravel, that’s split into 2 functionalities – Laravel’s queue listener, and the code we write ourselves – in this case, the PhotoService.

Laravel has some CLI tools to help with queues:

// Fire the latest job in the queue $ php artisan queue:work // Listen for new jobs in the queue// and fire them off one at a time// as they are created $ php artisan queue:listen

When not working with the “sync” driver, these tools are what you need to use in order to process the jobs in your queue. We run the queue:listen command to have laravel listen to the queue and pull jobs as they become available.

Let’s install Beanstalkd to see how that works.

By default, laravel will run queue jobs synchronously – that is, it runs the job at the time of creation. This means the image will be processed in the same request that the user created when uploading an image. That’s useful for testing, but not for production. We’ll make this asynchronous by introducing Beanstalkd.

Beanstalkd

Let’s install Beanstalkd:

# Debian / Ubuntu: $ sudo apt-get update $ sudo apt-get install beanstalkd

Note: You may be able to get a newer version of Beanstalkd by adding this PPA. Ubuntu 12.04 installs an older version of Beanstalkd.

Next, some quick configuration. The first thing we need to do is tell Beanstalkd to start when the system starts up or reboots. Edit /etc/default/beanstalkd and set START to “yes”.

$ sudo vim /etc/default/beanstalkd > START yes # uncomment

Then we can start Beanstalkd:

$ sudo service beanstalkd start # Alternatively: /etc/init.d/beanstalkd start

Now we can setup Laravel. In your app/config/queue.php file, set the default queue to ‘beanstalkd’:

'default'=>'beanstalkd',

Then edit any connection information you need to change. I left my configuration with the defaults as I installed it on the same server as the application.

'connections'=> array('beanstalkd'=> array('driver'=>'beanstalkd','host'=>'localhost','queue'=>'default',),),

Now when we push a job to the queue in Laravel, we’ll be pushing to Beanstalkd!

Installing Beanstalkd on a remote server

You may (read: should) want to consider installing Beanstalkd on another server, rather than your application server. Since Beantalkd is an in-memory service, it can eat up your servers resources under load.

To do this, you can install Beanstalkd on another server, and simply point your “host” to the proper server address, rather than localhost.

This leaves the final detail – what server runs the job? If you follow all other steps here, Supervisord will still be watching Laravel’s listener on your application server. You may want to consider running your job script (or even a copy of your application which has a job script) on yet another server whose job is purely to churn through Beanstalkd queue jobs. This means having a listener and working listener/job code on yet another server.

In fact, in a basic distributed setup, we’d probably have an application server (or 2, plus a load-balancer), a database server, a queue server and a job server!

Supervisord

Let’s say you pushed a job to Beanstalkd:

Queue::push('Myapp\Queue\PhotoService', array('image_path'=>'/path/to/image/file.ext'));

Now what? You might notice that it goes to Beanstalkd, but Myapp\Queue\PhotoService@fire() doesn’t seem to be getting called. You’ve checked your error logs, you see if the image was edited, and found that the the job is just “sitting there” in your Beanstalkd queue.

Beanstalkd doesn’t actually PUSH jobs to a script – instead, we need a worker to check if there are jobs available and ask for them.

This is what $ php artisan queue:listen does – It listens for jobs and runs them as they become available.

If you run that command, you’ll see your job being sent to code. If all goes well, your image will beproperly manipulated.

The question then becomes: How do I make php listen at all times? We need to avoid having to “supervise” that process manually. This is where Supervisord comes in.

Supervisord will watch our queue:listen command and restart it if it fails. Let’s see how to set that up.

First, we’ll install it:

# Debian / Ubuntu: $ sudo apt-get install supervisor

Next, we’ll configure it. We need to define a process to listen to.

$ sudo vim /etc/supervisor/conf.d/myqueue.conf

Add this to your new conf file, changing file paths and your environment as necessary:

[program:myqueue] command=php artisan queue:listen --env=your_environment directory=/path/to/laravel stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log redirect_stderr=true

We now have a process called “myqueue” which we can tell Supervisord to start and monitor.

Let’s do that:

$ sudo supervisorctl > reread # Tell supervisord to check for new items in /etc/supervisor/conf.d/> add myqueue # Add this process to Supervisord> start myqueue # May say "already started"

Now the “myqueue” process is on and being monitored. If our queue listener fails, Supervisord will restart the php artisan queue:listen --env=your_environment process.

You can check that it is indeed running that process with this command:

$ ps aux | grep php # You should see some output like this: php artisan queue:listen --env=your_environment sh -c php artisan queue:work --queue="default"--delay=0--memory=128--sleep --env=your_environment php artisan queue:work --queue=default--delay=0--memory=128--sleep --env=your_environment

Wrapping up

Now we have a full end-to-end queue working and in place!

  1. We create a script to process a queued job
  2. We installed Beanstalkd to act as the work queue
  3. We use Laravel to push jobs to our queue
  4. We use Laravel queue:listen to act as a worker and pull jobs from the queue
  5. We wrote some code to process a job from the queue
  6. We use Supervisord to ensure queue:listen is always listening for new jobs

Notes

  1. You might want to consider setting up log rotation on the Laravel and Supervisord logs
  2. You can read here for more information on setting up Supervisord on Ubuntu.
  3. Read the Laravel docs on queues to learn how and when to release or delete jobs.

TL;DR

For reference, just copy and paste the whole process from here:

$ sudo apt-get update $ sudo apt-get install -y beanstalkd supervisor $ sudo vim /etc/default/beanstalkd > START yes # uncomment this line $ sudo service beanstalkd start $ sudo vim /etc/supervisor/conf.d/myqueue.conf

Enter this, changing as needed:

[program:myqueue] command=php artisan queue:listen --env=your_environment directory=/path/to/laravel stdout_logfile=/path/to/laravel/app/storage/logs/myqueue_supervisord.log redirect_stderr=true

Start Supervisord:

$ sudo supervisorctl > reread # Get available jobs> add myqueue > start myqueue

Read more on Supervisord here for info on supervisorctl.

 

另外还有一篇相关的不错的文章:http://www.lornajane.net/posts/2014/working-with-php-and-beanstalkd

转载于:https://www.cnblogs.com/argb/p/3729084.html

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/109830.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)


相关推荐

  • TCP协议和UDP协议

    TCP协议和UDP协议1.传输控制协议TCP1.1TCP的主要特点:1.1.1面向连接的运输层协议socket部分讲述tcp连接的建立tcp连接的释放tcp的有限状态机1.1.2每一条TCP连接只能有两个端点,每一条TCP链接只能是点对点的(一对一)1.1.3TCP提供可靠交付的服务可靠传输的工作原理可靠传输的实现流量控制拥塞控制1.1.4TCP提供全双工通信1.1.5面向字节流流式服务的特点1.2与TCP有关的面试问题2.用户数据报协议UDP2.1UDP协

  • 关于HeartBleed漏洞的总结「建议收藏」

    关于HeartBleed漏洞的总结「建议收藏」一:前言HeartBleed漏洞又称为心脏出血漏洞,编号(CVE-2014-0160),产生原因:由于未能在memcpy()调用受害用户输入的内容作为长度参数之前正确进行边界检查。攻击者可以追踪OpenSSl所分配的64KB内存,将超出必要范围的字节信息复制到缓存当中,再返回缓存内容,这样一来,受害者的内存内容就会每次泄露64KB.简单来说,这就是OpenSSL缺陷造成的漏洞二:环境…

  • 简述py2exe的使用方法[通俗易懂]

    简述py2exe的使用方法[通俗易懂]下载py2exe编写mysetup.py

  • python json 编码_python乱码转中文

    python json 编码_python乱码转中文python2.x版本的字符编码有时让人很头疼,遇到问题,网上方法可以解决错误,但对原理还是一知半解,本文主要介绍python中字符串处理的原理,附带解决json文件输出时,显示中文而非un

  • 矩阵范数推导_向量范数怎么求

    矩阵范数推导_向量范数怎么求参考自:1999年08月清华大学出版社出版的施妙根的《科学和工程计算基础》。

  • linux重启网卡命令失败_centos7网卡重启

    linux重启网卡命令失败_centos7网卡重启  首先,命令ifconfig查看网卡状态,发现eth0网卡未启动。  然后,用命令ifupeth0启动网卡。   用命令ifdowneth0则是卸载网卡。转载于:https://www.cnblogs.com/dayboy2416/p/11231314.html…

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号