luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
288 commits 6 branches 9 tags 7.5 MiB
Commit graph

288 commits

This branch
This branch
All branches
Author SHA1 Message Date
gram-signal
6720a01733
Maintain parent/child shard ordering across shard splits/merges. (#155) 
Kinesis allows clients to rely on an invariant that, for a given partition key, the order of records added to the stream will be maintained.  IE: given an input `pkey=x,val=1  pkey=x,val=2  pkey=x,val=3`, the values `1,2,3` will be seen in that order when processed by clients, so long as clients are careful.  It does so by putting all records for a single partition key into a single shard, then maintaining ordering within that shard.

However, shards can be split and merge, to distribute load better and handle per-shard throughput limits.  Kinesis does this currently by (one or many times) splitting a single shard into two or by merging two adjacent shards into one.  When this occurs, Kinesis still allows for ordering consistency by detailing shard parent/child relationships within its `listShards` outputs.  A split shard A will create children B and C, both with `ParentShardId=A`.  A merging of shards A and B into C will create a new shard C with `ParentShardId=A,AdjacentParentShardId=B`.  So long as clients fully process all records in parents (including adjacent parents) before processing the new shard, ordering will be maintained.

`kinesis-consumer` currently doesn't do this.  Instead, upon the initial (and subsequent) `listShards` call, all visible shards immediately begin processing.  Considering this case, where shards split, then merge, and each shard `X` contains a single record `rX`:

```
time ->
  B
 / \
A   D
 \ /
  C
```

record `rD` should be processed after both `rB` and `rC` are processed, and both `rB` and `rC` should wait for `rA` to be processed.  By starting goroutines immediately, any ordering of `{rA,rB,rC,rD}` might occur within the original code.

This PR utilizes the `AllGroup` as a book-keeper of fully processed shards, with the `Consumer` calling `CloseShard` once it has finished a shard.  `AllGroup` doesn't release a shard for processing until its parents have fully been processed, and the consumer just processes the shards it receives as it used to.

This PR created a new `CloseableGroup` interface rather than append to the existing `Group` interface to maintain backwards compatibility in existing code that may already implement the `Group` interface elsewhere.  Different `Group` implementations don't get the ordering described above, but the default `Consumer` does.
 2024-06-06 08:37:42 -07:00
dependabot[bot]
b7242de6cf
Bump github.com/prometheus/procfs from 0.12.0 to 0.15.1 
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs) from 0.12.0 to 0.15.1.
- [Release notes](https://github.com/prometheus/procfs/releases)
- [Commits](https://github.com/prometheus/procfs/compare/v0.12.0...v0.15.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
  dependency-type: indirect
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-06-03 12:42:15 +00:00
dependabot[bot]
caa313cf95
Bump github.com/prometheus/client_model from 0.5.0 to 0.6.1 
Bumps [github.com/prometheus/client_model](https://github.com/prometheus/client_model) from 0.5.0 to 0.6.1.
- [Release notes](https://github.com/prometheus/client_model/releases)
- [Commits](https://github.com/prometheus/client_model/compare/v0.5.0...v0.6.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_model
  dependency-type: indirect
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 12:27:59 +00:00
Alex
9e7f1fd270
Merge pull request #48 from alexgridx/47-add-prometheus-metrics 
#47 adds prometheus metrics
 2024-05-28 12:10:26 +02:00
Alex Senger
c9c13a9914
#47 adds prometheus metrics 2024-05-28 12:09:19 +02:00
Alex
14ab603b79
Merge pull request #46 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue-1.13.19 
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue from 1.13.14 to 1.13.19
 2024-05-28 10:08:43 +02:00
dependabot[bot]
41539557e7
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue 
Bumps [github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue](https://github.com/aws/aws-sdk-go-v2) from 1.13.14 to 1.13.19.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.13.14...credentials/v1.13.19)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:08:34 +00:00
Alex
02c0ada62c
Merge pull request #36 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/dynamodbstreams-1.20.8 
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams from 1.20.4 to 1.20.8
 2024-05-28 10:07:47 +02:00
dependabot[bot]
92a8648de7
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams 
Bumps [github.com/aws/aws-sdk-go-v2/service/dynamodbstreams](https://github.com/aws/aws-sdk-go-v2) from 1.20.4 to 1.20.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.20.4...service/ivs/v1.20.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/dynamodbstreams
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:07:39 +00:00
Alex
ef891eb523
Merge pull request #42 from alexgridx/dependabot/go_modules/github.com/yuin/gopher-lua-1.1.1 
Bump github.com/yuin/gopher-lua from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1
 2024-05-28 10:07:23 +02:00
Alex
3bf0572500
Merge pull request #35 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/kinesis-1.27.8 
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8
 2024-05-28 10:06:59 +02:00
dependabot[bot]
10dc9d7031
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8 
Bumps [github.com/aws/aws-sdk-go-v2/service/kinesis](https://github.com/aws/aws-sdk-go-v2) from 1.27.4 to 1.27.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.4...config/v1.27.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/kinesis
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:06:07 +00:00
dependabot[bot]
c288cb88ef
Bump github.com/yuin/gopher-lua 
Bumps [github.com/yuin/gopher-lua](https://github.com/yuin/gopher-lua) from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1.
- [Release notes](https://github.com/yuin/gopher-lua/releases)
- [Commits](https://github.com/yuin/gopher-lua/commits/v1.1.1)

---
updated-dependencies:
- dependency-name: github.com/yuin/gopher-lua
  dependency-type: indirect
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:01:17 +00:00
Alex
fa5681358b
Merge pull request #32 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/config-1.27.16 
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16
 2024-05-28 09:54:09 +02:00
dependabot[bot]
c8a75ba8b5
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16 
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.11 to 1.27.16.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.11...config/v1.27.16)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 07:54:01 +00:00
Alex
9c040ee238
Merge pull request #31 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery-1.9.8 
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery from 1.9.6 to 1.9.8
 2024-05-28 09:53:05 +02:00
dependabot[bot]
cad20b6cc8
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery 
Bumps [github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery](https://github.com/aws/aws-sdk-go-v2) from 1.9.6 to 1.9.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pipes/v1.9.6...service/grafana/v1.9.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 07:52:17 +00:00
Alex
39d374f4fa
Merge pull request #29 from alexgridx/27-enable-dependabot 
#27 enables dependabot
 2024-05-07 11:04:55 +02:00
Alex Senger
ac83ac27e7
#27 enables dependabot 2024-05-07 11:03:38 +02:00
Alex
37b05ea426
Merge pull request #28 from alexgridx/26-upgrade-dependencies 
#26 upgrades dependencies
 2024-05-07 11:00:10 +02:00
Alex Senger
e73405c9cd
#26 upgrades dependencies 2024-05-07 10:58:18 +02:00
Alex
bbe5ed2a22
Merge pull request #22 from alexgridx/21-allow-overwriting-postgres-connection 
#21 allows to overwrite postgres connection
 2024-04-19 17:11:50 +02:00
Alex Senger
1e621394d5
#21 allows to overwrite postgres connection 2024-04-19 17:11:10 +02:00
Alex
c372167032
Merge pull request #20 from alexgridx/19-fix-retry-on-kinesis-error 
#19 fixes retry on kinesis error
 2024-04-19 13:33:41 +02:00
Alex Senger
ca7a109d1b
#19 fixes retry on kinesis error 2024-04-19 13:32:53 +02:00
Alex
2d05606a38
Merge pull request #18 from alexgridx/5-move-to-slog 
#5 moves to slog
 2024-04-19 13:31:21 +02:00
Alex Senger
95a08a91e8
#5 moves to slog 2024-04-19 13:29:57 +02:00
Alex
36e7fc8b89
Merge pull request #17 from alexgridx/16-update-examples 
#16 updates examples
 2024-04-12 10:02:14 +02:00
Alex Senger
3c37f72e41
#16 updates examples 2024-04-12 09:44:32 +02:00
Alex
8219b385d3
Merge pull request #15 from alexgridx/12-rename-package 
#12 rename package
 2024-04-10 17:20:26 +02:00
Alex Senger
7e0806126c
#12 rename package 2024-04-10 17:19:07 +02:00
Alex
c169ad4789
Merge pull request #13 from alexgridx/11-check-in-vendor-files 
#11 checks in vendor files
 2024-04-10 16:55:34 +02:00
Alex Senger
262dce8ba6
#11 checks in vendor files 2024-04-10 16:50:33 +02:00
Alex
b71c1964bc
Merge pull request #10 from alexgridx/9-introduce-linter 
#9 introduce linter
 2024-04-10 16:47:43 +02:00
Alex Senger
5100c7ce77
#9 introduces linter 2024-04-10 16:45:57 +02:00
Alex Senger
f0acb329f7
#9 fixes linting issues 2024-04-10 16:45:34 +02:00
Alex
fd263c3d60
Merge pull request #8 from alexgridx/7-use-awslabs-package 
#7 imports awslabs aggregator in favor of local one
 2024-04-10 16:43:52 +02:00
Alex Senger
eaf4defe57
#7 imports awslabs aggregator in favor of local one 2024-04-10 16:42:31 +02:00
Alex
1918e43f3e
Merge pull request #6 from alexgridx/4-minor-cleanups 
#4 minor cleanups
 2024-04-10 15:17:01 +02:00
Alex Senger
040fa06efa
#4 minor cleanups 2024-04-10 15:16:07 +02:00
Alex
a5089538fd
Merge pull request #3 from alexgridx/1-upgrade-dependencies 
#1 Removes Import Renames
 2024-04-10 15:01:24 +02:00
Alex Senger
a38bb99afe
#1 removes import renames 2024-04-10 15:00:24 +02:00
Alex
392532b6e8
Merge pull request #2 from alexgridx/1-upgrade-dependencies 
#1 Upgrades Dependencies
 2024-04-10 14:59:42 +02:00
Alex Senger
c7f3c79ff7
#1 upgrades dependencies 2024-04-10 14:58:37 +02:00
guangwu
188bdff278
fix: typo (#156) 2023-12-14 16:36:20 -08:00
Dario Emerson
af2db0d43f
update go-redis to v9 (#150) 2023-06-16 11:20:07 -07:00
Harlow Ward
c2b9f79d7a Put aws-skd-v2 note in installation section 2021-12-04 13:43:18 -08:00
Harlow Ward
6cbda0f706
Update ddb checkpoint item to use dynamodbav (#144) 
Update the DynamoDB checkpoint item to use the `dynamodbav` value for marshaling. 

Fixes: https://github.com/harlow/kinesis-consumer/issues/142

Minor changes:
* Update the DDB example consumer to support a local version of DDB for streamlined testing
 2021-12-04 13:40:26 -08:00
Harlow Ward
3b3b252fa5 Add all examples to example directory 2021-12-04 10:42:17 -08:00
Harlow Ward
61fa84eca6
Update to use aws-sdk-go-v2 (#141) 2021-09-21 22:00:14 -07:00