luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
241 commits 6 branches 9 tags 7.5 MiB
Commit graph

241 commits

This branch
This branch
All branches
Author SHA1 Message Date
Alex
b72a9328da
Merge pull request #72 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/sts-1.28.12 
Bump github.com/aws/aws-sdk-go-v2/service/sts from 1.28.10 to 1.28.12
 2024-06-11 11:15:50 +02:00
Alex Senger
0ee8cc4623
Merge branch 'harlow-master' 2024-06-11 11:15:01 +02:00
Alex Senger
a8e4ff111c
Merge branch 'master' of github.com:harlow/kinesis-consumer into harlow-master 2024-06-11 11:10:16 +02:00
dependabot[bot]
5e312aa870
Bump github.com/aws/aws-sdk-go-v2/service/sts from 1.28.10 to 1.28.12 
Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2) from 1.28.10 to 1.28.12.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/efs/v1.28.10...service/fsx/v1.28.12)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-06-10 13:12:41 +00:00
Jarrad
553e2392fd
fix nil pointer dereference on AWS errors (#148) 
* fix nil pointer dereference on AWS errors

* return Start errors to Scan consumer

before the previous commit e465b09, client errors panicked the
reader, so consumers would pick up sharditerator errors by virtue of
their server crashing and burning.

Now that client errors are properly returned, the behaviour of
listShards is problematic because it absorbs any client errors it gets.

The result of these two things now is that if you hit an aws error, your server will go into an
endless scan loop you can't detect and can't easily recover from.

To avoid that, listShards will now stop if it hits a client error.

---------

Co-authored-by: Jarrad Whitaker <jwhitaker 📧 swift-nav.com>
 2024-06-06 08:38:16 -07:00
gram-signal
6720a01733
Maintain parent/child shard ordering across shard splits/merges. (#155) 
Kinesis allows clients to rely on an invariant that, for a given partition key, the order of records added to the stream will be maintained.  IE: given an input `pkey=x,val=1  pkey=x,val=2  pkey=x,val=3`, the values `1,2,3` will be seen in that order when processed by clients, so long as clients are careful.  It does so by putting all records for a single partition key into a single shard, then maintaining ordering within that shard.

However, shards can be split and merge, to distribute load better and handle per-shard throughput limits.  Kinesis does this currently by (one or many times) splitting a single shard into two or by merging two adjacent shards into one.  When this occurs, Kinesis still allows for ordering consistency by detailing shard parent/child relationships within its `listShards` outputs.  A split shard A will create children B and C, both with `ParentShardId=A`.  A merging of shards A and B into C will create a new shard C with `ParentShardId=A,AdjacentParentShardId=B`.  So long as clients fully process all records in parents (including adjacent parents) before processing the new shard, ordering will be maintained.

`kinesis-consumer` currently doesn't do this.  Instead, upon the initial (and subsequent) `listShards` call, all visible shards immediately begin processing.  Considering this case, where shards split, then merge, and each shard `X` contains a single record `rX`:

```
time ->
  B
 / \
A   D
 \ /
  C
```

record `rD` should be processed after both `rB` and `rC` are processed, and both `rB` and `rC` should wait for `rA` to be processed.  By starting goroutines immediately, any ordering of `{rA,rB,rC,rD}` might occur within the original code.

This PR utilizes the `AllGroup` as a book-keeper of fully processed shards, with the `Consumer` calling `CloseShard` once it has finished a shard.  `AllGroup` doesn't release a shard for processing until its parents have fully been processed, and the consumer just processes the shards it receives as it used to.

This PR created a new `CloseableGroup` interface rather than append to the existing `Group` interface to maintain backwards compatibility in existing code that may already implement the `Group` interface elsewhere.  Different `Group` implementations don't get the ordering described above, but the default `Consumer` does.
 2024-06-06 08:37:42 -07:00
Alex
9e7f1fd270
Merge pull request #48 from alexgridx/47-add-prometheus-metrics 
#47 adds prometheus metrics
 2024-05-28 12:10:26 +02:00
Alex Senger
c9c13a9914
#47 adds prometheus metrics 2024-05-28 12:09:19 +02:00
Alex
14ab603b79
Merge pull request #46 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue-1.13.19 
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue from 1.13.14 to 1.13.19
 2024-05-28 10:08:43 +02:00
dependabot[bot]
41539557e7
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue 
Bumps [github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue](https://github.com/aws/aws-sdk-go-v2) from 1.13.14 to 1.13.19.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.13.14...credentials/v1.13.19)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:08:34 +00:00
Alex
02c0ada62c
Merge pull request #36 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/dynamodbstreams-1.20.8 
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams from 1.20.4 to 1.20.8
 2024-05-28 10:07:47 +02:00
dependabot[bot]
92a8648de7
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams 
Bumps [github.com/aws/aws-sdk-go-v2/service/dynamodbstreams](https://github.com/aws/aws-sdk-go-v2) from 1.20.4 to 1.20.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.20.4...service/ivs/v1.20.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/dynamodbstreams
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:07:39 +00:00
Alex
ef891eb523
Merge pull request #42 from alexgridx/dependabot/go_modules/github.com/yuin/gopher-lua-1.1.1 
Bump github.com/yuin/gopher-lua from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1
 2024-05-28 10:07:23 +02:00
Alex
3bf0572500
Merge pull request #35 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/kinesis-1.27.8 
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8
 2024-05-28 10:06:59 +02:00
dependabot[bot]
10dc9d7031
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8 
Bumps [github.com/aws/aws-sdk-go-v2/service/kinesis](https://github.com/aws/aws-sdk-go-v2) from 1.27.4 to 1.27.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.4...config/v1.27.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/kinesis
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:06:07 +00:00
dependabot[bot]
c288cb88ef
Bump github.com/yuin/gopher-lua 
Bumps [github.com/yuin/gopher-lua](https://github.com/yuin/gopher-lua) from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1.
- [Release notes](https://github.com/yuin/gopher-lua/releases)
- [Commits](https://github.com/yuin/gopher-lua/commits/v1.1.1)

---
updated-dependencies:
- dependency-name: github.com/yuin/gopher-lua
  dependency-type: indirect
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 08:01:17 +00:00
Alex
fa5681358b
Merge pull request #32 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/config-1.27.16 
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16
 2024-05-28 09:54:09 +02:00
dependabot[bot]
c8a75ba8b5
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16 
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) from 1.27.11 to 1.27.16.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.11...config/v1.27.16)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 07:54:01 +00:00
Alex
9c040ee238
Merge pull request #31 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery-1.9.8 
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery from 1.9.6 to 1.9.8
 2024-05-28 09:53:05 +02:00
dependabot[bot]
cad20b6cc8
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery 
Bumps [github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery](https://github.com/aws/aws-sdk-go-v2) from 1.9.6 to 1.9.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pipes/v1.9.6...service/grafana/v1.9.8)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
  dependency-type: indirect
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2024-05-28 07:52:17 +00:00
Alex
39d374f4fa
Merge pull request #29 from alexgridx/27-enable-dependabot 
#27 enables dependabot
 2024-05-07 11:04:55 +02:00
Alex Senger
ac83ac27e7
#27 enables dependabot 2024-05-07 11:03:38 +02:00
Alex
37b05ea426
Merge pull request #28 from alexgridx/26-upgrade-dependencies 
#26 upgrades dependencies
 2024-05-07 11:00:10 +02:00
Alex Senger
e73405c9cd
#26 upgrades dependencies 2024-05-07 10:58:18 +02:00
Alex
bbe5ed2a22
Merge pull request #22 from alexgridx/21-allow-overwriting-postgres-connection 
#21 allows to overwrite postgres connection
 2024-04-19 17:11:50 +02:00
Alex Senger
1e621394d5
#21 allows to overwrite postgres connection 2024-04-19 17:11:10 +02:00
Alex
c372167032
Merge pull request #20 from alexgridx/19-fix-retry-on-kinesis-error 
#19 fixes retry on kinesis error
 2024-04-19 13:33:41 +02:00
Alex Senger
ca7a109d1b
#19 fixes retry on kinesis error 2024-04-19 13:32:53 +02:00
Alex
2d05606a38
Merge pull request #18 from alexgridx/5-move-to-slog 
#5 moves to slog
 2024-04-19 13:31:21 +02:00
Alex Senger
95a08a91e8
#5 moves to slog 2024-04-19 13:29:57 +02:00
Alex
36e7fc8b89
Merge pull request #17 from alexgridx/16-update-examples 
#16 updates examples
 2024-04-12 10:02:14 +02:00
Alex Senger
3c37f72e41
#16 updates examples 2024-04-12 09:44:32 +02:00
Alex
8219b385d3
Merge pull request #15 from alexgridx/12-rename-package 
#12 rename package
 2024-04-10 17:20:26 +02:00
Alex Senger
7e0806126c
#12 rename package 2024-04-10 17:19:07 +02:00
Alex
c169ad4789
Merge pull request #13 from alexgridx/11-check-in-vendor-files 
#11 checks in vendor files
 2024-04-10 16:55:34 +02:00
Alex Senger
262dce8ba6
#11 checks in vendor files 2024-04-10 16:50:33 +02:00
Alex
b71c1964bc
Merge pull request #10 from alexgridx/9-introduce-linter 
#9 introduce linter
 2024-04-10 16:47:43 +02:00
Alex Senger
5100c7ce77
#9 introduces linter 2024-04-10 16:45:57 +02:00
Alex Senger
f0acb329f7
#9 fixes linting issues 2024-04-10 16:45:34 +02:00
Alex
fd263c3d60
Merge pull request #8 from alexgridx/7-use-awslabs-package 
#7 imports awslabs aggregator in favor of local one
 2024-04-10 16:43:52 +02:00
Alex Senger
eaf4defe57
#7 imports awslabs aggregator in favor of local one 2024-04-10 16:42:31 +02:00
Alex
1918e43f3e
Merge pull request #6 from alexgridx/4-minor-cleanups 
#4 minor cleanups
 2024-04-10 15:17:01 +02:00
Alex Senger
040fa06efa
#4 minor cleanups 2024-04-10 15:16:07 +02:00
Alex
a5089538fd
Merge pull request #3 from alexgridx/1-upgrade-dependencies 
#1 Removes Import Renames
 2024-04-10 15:01:24 +02:00
Alex Senger
a38bb99afe
#1 removes import renames 2024-04-10 15:00:24 +02:00
Alex
392532b6e8
Merge pull request #2 from alexgridx/1-upgrade-dependencies 
#1 Upgrades Dependencies
 2024-04-10 14:59:42 +02:00
Alex Senger
c7f3c79ff7
#1 upgrades dependencies 2024-04-10 14:58:37 +02:00
guangwu
188bdff278
fix: typo (#156) 2023-12-14 16:36:20 -08:00
Dario Emerson
af2db0d43f
update go-redis to v9 (#150) 2023-06-16 11:20:07 -07:00
Harlow Ward
c2b9f79d7a Put aws-skd-v2 note in installation section 2021-12-04 13:43:18 -08:00