top
(README.md)
# Kinsumer

Native Go consumer for AWS Kinesis streams.

[![Build Status](https://travis-ci.org/TwitchScience/kinsumer.svg?branch=master)](https://travis-ci.org/TwitchScience/kinsumer) [![Go Report Card](https://goreportcard.com/badge/github.com/twitchscience/kinsumer)](https://goreportcard.com/report/github.com/twitchscience/kinsumer)

## Rationale
There are several very good ways to consume Kinesis streams, primarily [The Amazon Kinesis Client Library](http://docs.aws.amazon.com/kinesis/latest/dev/developing-consumers-with-kcl.html), and it is recommended that be investigated as an option.

Kinsumer is designed for a cluster of Go clients that want each client to consume from multiple shards. Kinsumer is designed to be at-least-once with a strong effort to be exactly-once. Kinsumer by design does not attempt to keep shards on a specific client and will shuffle them around as needed.

## Development state
Currently Kinsumer is in alpha state. We are putting it into production slowly, but it's probable that issues will arise.

## Behavior
Kinsumer is designed to suit a specific use case of kinesis consuming, specifically when you need to have multiple clients each handling multiple shards and you do not care which shard is being consumed by which client.

Kinsumer will rebalance shards to each client whenever it detects the list of shards or list of clients has changed, and does not attempt to keep shards on the same client.

## Testing

### Testing with local test servers
By default the tests look for a dynamodb server at `localhost:4567` and kinesis server at `localhost:4568`


For example using [kinesalite](https://github.com/mhart/kinesalite) and [dynalite](https://github.com/mhart/dynalite)
```
kinesalite --port 4568 --createStreamMs 1 --deleteStreamMs 1 --updateStreamMs 1 --shardLimit 1000 &
dynalite --port 4567 --createTableMs 1 --deleteTableMs 1 --updateTableMs 1 &
```
Then `go test ./...`

### Testing with real aws resources
It's possible to run the test against real AWS resources, but the tests create and destroy resources, which can be finicky, and potentially expensive.

Make sure you have your credentials setup in a way that [aws-sdk-go](https://github.com/aws/aws-sdk-go) is happy with, or be running on an EC2 instance.

Then `go test . -dynamo_endpoint= -kinesis_endpoint=  -resource_change_timeout=30s`

Imports 9 package(s)

  1. github.com/aws/aws-sdk-go/service/kinesis
  2. github.com/aws/aws-sdk-go/aws/awserr
  3. github.com/aws/aws-sdk-go/service/dynamodb
  4. github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute
  5. github.com/twinj/uuid
  6. github.com/aws/aws-sdk-go/aws/session
  7. github.com/aws/aws-sdk-go/aws
  8. github.com/aws/aws-sdk-go/service/kinesis/kinesisiface
  9. github.com/aws/aws-sdk-go/service/dynamodb/dynamodbiface

Test imports 3 package(s)

  1. github.com/twitchscience/kinsumer/mocks
  2. github.com/stretchr/testify/assert
  3. github.com/stretchr/testify/require