Gearman architecture is equally simple: it consists of job servers, that accept task requests from clients and forward them to workers, and send results back to clients. Each worker can be connected to many job servers, and a client can choose which job server to use - this way there is no single point of failure that could break down the whole cluster. Job servers have their own queues and in case of worker failure they can reassign tasks to other workers. According to High Scalability Gearman has been successfully used by LiveJournal, Yahoo!, and Digg (which claims to run 300000 jobs a day through Gearman without any issues).
I decided to try out Gearman at home, and I must say that it was a really pleasant experience. I wrote a simple C++ worker and even simpler Python client. The worker recursively finds Fibonacci number for given n:
#include <cstring>
#include <cstdlib>
#include <iostream>
#include <sstream>
#include <libgearman/gearman.h>
#include <libgearman/worker.h>
using namespace std;
void *fib_worker(gearman_job_st *job, void *cb_arg, size_t *result_size, gearman_return_t *ret_ptr);
long fib(long n);
static void usage(char *name);
int main(int argc, char *argv[])
{
int c;
char *host = "127.0.0.1";
in_port_t port = 0;
gearman_worker_st worker;
while ((c = getopt(argc, argv, "h:p:")) != -1) {
switch(c) {
case 'h':
host = optarg;
break;
case 'p':
port = (in_port_t) atoi(optarg);
break;
default:
usage(argv[0]);
exit(1);
}
}
if (argc != optind) {
usage(argv[0]);
exit(1);
}
gearman_worker_create(&worker);
gearman_worker_add_server(&worker, host, port);
gearman_worker_add_function(&worker, "fib", 10, fib_worker, NULL);
while (1) {
gearman_worker_work(&worker);
}
return 0;
}
void *fib_worker(gearman_job_st *job, void *cb_arg, size_t *result_size, gearman_return_t *ret_ptr) {
char ch[256];
ostringstream os;
int size = gearman_job_workload_size(job);
strncpy(ch, (char *) gearman_job_workload(job), size);
ch[size] = 0;
long n = atol(ch);
os << fib(n);
string s = os.str();
*result_size = s.size();
*ret_ptr = GEARMAN_SUCCESS;
return strdup(s.c_str());
}
long fib(long n) {
if (n < 2) {
return 1;
} else {
return fib(n - 2) + fib(n - 1);
}
}
static void usage(char *name) {
cout << "Usage: " << name << " [-h <host>] [-p <port>] <string>" << endl;
cout << "\t-h <host> - job server host" << endl;
cout << "\t-p <port> - job server port" << endl;
}
Python client prepares a set of jobs for a sequence of n numbers, runs them simultaneously through a job server and sums up the result:
import optparse
from gearman import *
parser = optparse.OptionParser()
parser.add_option('--host', help = "Specifies gearman job server")
parser.add_option('-n', '--num', help = "Amount of Fibonacci numbers to compute")
(opts, args) = parser.parse_args()
client = GearmanClient([opts.host])
ts = Taskset()
for i in range(1, int(opts.num)):
t = Task(func = "fib", arg = i)
ts.add(t)
client.do_taskset(ts)
sum = 0
for task in ts.values():
sum += int(task.result)
print sum
You can download the source code of both worker and client here. After you compile and install Gearman with traditional:
./configure make sudo make install sudo ldconfigInstall Python extension with:
easy_install gearmanAnd compile the C++ example with:
makeThen you can run a job server as a daemon:
gearmand -d -L 127.0.0.1or in debug mode:
gearmand -vv -L 127.0.0.1Next, run a couple of Gearman workers:
./GearmanWorker -h 127.0.0.1And the Python client:
python GearmanClient.py --host 127.0.0.1 -n 45For a single machine, it makes sense to run at most as many workers as there are CPUs (or CPU cores) available. For a network cluster, you can run more job servers and workers (and clients) respectively.
I've made some tests with the client and worker above. using my home laptop and an Intel Atom based net-top running together in a local network. For only one laptop worker, computing the sum of 45 Fibonacci numbers took 66.955 seconds, for two laptop workers it took 35.702 seconds, and adding a remote worker reduced the total time to 25.593 seconds. Adding more workers didn't reduce computation time, it even slightly slowed the cluster down - which is quite understandable, as the number of workers exceeded the number of free CPUs (Intel Atom in fact has only one physical core, although applications see it as dual-core CPU).
