Go Worker Pools

 Worker pools is a design in which a fixed number of m workers (Go goroutines) works on n tasks in a work queue (Go channel). The work resides in a queue until a worker finish its current task and pull a new one.

Let's See it by an example

1. package main  
2.   
3. import (  
4. "fmt"  
5. "time"  
6. )  
7.   
8. func worker(id int, jobs <-chan int, results chan<- int) {  
9.     for j := range jobs {  
10.         fmt.Println("worker", id, "processing job", j)  
11.         time.Sleep(time.Second)  
12.         results <- j * 2  
13.     }  
14. }  
15.   
16. func main() {  
17.     job := make(chan int, 10)  
18.     result := make(chan int, 10)  
19.     for w := 1; w <= 2; w++ {  
20.         go worker(w, job, result)  
21.     }  
22.     for j := 1; j <= 9; j++ {  
23.         job <- j  
24.     }  
25.     close(job)  
26.     for a := 1; a <= 9; a++ {  
27.         <-result  
28.     }  
29. }  

Output:

worker 2 processing job 1
worker 1 processing job 2
worker 2 processing job 3
worker 1 processing job 4
worker 1 processing job 5
worker 2 processing job 6
worker 1 processing job 7
worker 2 processing job 8

In this example, 2 workers are started and 9 work items are in put onto a job channel. Workers have a work loop with a time.Sleep so that each ends up working 2 jobs. close is used on the channel after all the work's been put onto it, which signals to all 2 workers that they can exit their work loop by dropping them out of their loop on range.