top

Package mr provides a local concurrent computing model(MapReduce) using Sophie serialization.

A simple word count example is like this:

job := MrJob {
	Source: []Input{...},

	NewMapperF: func(src, part int) Mapper {
		return &MapperStruct {
			NewKeyF: sophie.NewString,
			NewValF: sophie.ReturnNULL,
			MapperF: func(key, val sophie.SophieWriter, c PartCollector) error {
				line := key.(*RawString).String()
				words := strings.Split(line, " ")
				for _, word: range words {
					c.CollectTo(0, sophie.RawString(word), sophie.VInt(1))
				}
			},
		}
	},

	NewReducerF: func(part int) Reducer {
		return &ReducerStruct {
			NewKeyF: sophie.NewRawString,
			NewValF: sophie.NewVInt,
			ReducerF: func((key sophie.SophieWriter, nextVal SophierIterator,
				c []sophie.Collector) error {
				var count sophie.VInt
				for {
					val, err := nextVal()
					if err == io.EOF {
						break
					}
					if err != nil {
						return err
					}
					count += val.(*sophie.VInt).Val()
				}
				return c[0].Collect(key, count)
			},
		}
	},

	Dest: []Output{...},
}

if err := job.Run(); err != nil {
	log.Fatalf("job.Run failed: %v", err)
}

One can also use MapOnlyJob for simple jobs.

Imported by 31 package(s)

  1. github.com/c4e8ece0/gcse
  2. github.com/c4e8ece0/gcse/crawler
  3. github.com/c4e8ece0/gcse/mergedocs
  4. github.com/danielqsj/gcse
  5. github.com/danielqsj/gcse/crawler
  6. github.com/danielqsj/gcse/mergedocs
  7. github.com/daviddengcn/gcse
  8. github.com/lanastasov/gcse
  9. github.com/lanastasov/gcse/crawler
  10. github.com/lanastasov/gcse/mergedocs
  11. github.com/luosangnanka/gcse
  12. github.com/luosangnanka/gcse/crawler
  13. github.com/luosangnanka/gcse/mergedocs
  14. github.com/mewbak/gcse
  15. github.com/mewbak/gcse/crawler
  16. github.com/mewbak/gcse/mergedocs
  17. github.com/mhutter/gcse
  18. github.com/mhutter/gcse/pipelines/crawler
  19. github.com/mhutter/gcse/pipelines/mergedocs
  20. github.com/mipearson/gcse
  21. github.com/mipearson/gcse/crawler
  22. github.com/mipearson/gcse/mergedocs
  23. github.com/skisulli/gcse
  24. github.com/skisulli/gcse/crawler
  25. github.com/skisulli/gcse/mergedocs
  26. github.com/subosito/gcse
  27. github.com/subosito/gcse/crawler
  28. github.com/subosito/gcse/mergedocs
  29. github.com/xladykiller/gcse
  30. github.com/xladykiller/gcse/crawler
  31. github.com/xladykiller/gcse/mergedocs

Imports 5 package(s)

  1. github.com/golangplus/errors
  2. github.com/golangplus/bytes
  3. github.com/daviddengcn/sophie
  4. github.com/daviddengcn/sophie/kv
  5. github.com/daviddengcn/go-villa

Test imports 1 package(s)

  1. github.com/golangplus/testing/assert