dependabot[bot]
4f1f924151
Bump golang.org/x/sys from 0.17.0 to 0.21.0
...
Bumps [golang.org/x/sys](https://github.com/golang/sys ) from 0.17.0 to 0.21.0.
- [Commits](https://github.com/golang/sys/compare/v0.17.0...v0.21.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/sys
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:17:04 +00:00
Alex
bf51213698
Merge pull request #62 from alexgridx/dependabot/go_modules/github.com/prometheus/common-0.54.0
...
Bump github.com/prometheus/common from 0.48.0 to 0.54.0
2024-06-11 11:17:02 +02:00
dependabot[bot]
839e981436
Bump github.com/aws/aws-sdk-go-v2/service/dynamodb from 1.32.5 to 1.32.8
...
Bumps [github.com/aws/aws-sdk-go-v2/service/dynamodb](https://github.com/aws/aws-sdk-go-v2 ) from 1.32.5 to 1.32.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/fsx/v1.32.5...service/dynamodb/v1.32.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/dynamodb
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:16:53 +00:00
dependabot[bot]
935d8cb78e
Bump github.com/prometheus/common from 0.48.0 to 0.54.0
...
Bumps [github.com/prometheus/common](https://github.com/prometheus/common ) from 0.48.0 to 0.54.0.
- [Release notes](https://github.com/prometheus/common/releases )
- [Changelog](https://github.com/prometheus/common/blob/main/RELEASE.md )
- [Commits](https://github.com/prometheus/common/compare/v0.48.0...v0.54.0 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/common
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:16:52 +00:00
Alex
ff592832c2
Merge pull request #58 from alexgridx/dependabot/go_modules/github.com/prometheus/procfs-0.15.1
...
Bump github.com/prometheus/procfs from 0.12.0 to 0.15.1
2024-06-11 11:16:27 +02:00
Alex
496e9243c6
Merge pull request #51 from alexgridx/dependabot/go_modules/github.com/prometheus/client_model-0.6.1
...
Bump github.com/prometheus/client_model from 0.5.0 to 0.6.1
2024-06-11 11:16:11 +02:00
Alex
b72a9328da
Merge pull request #72 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/sts-1.28.12
...
Bump github.com/aws/aws-sdk-go-v2/service/sts from 1.28.10 to 1.28.12
2024-06-11 11:15:50 +02:00
Alex Senger
0ee8cc4623
Merge branch 'harlow-master'
2024-06-11 11:15:01 +02:00
Alex Senger
a8e4ff111c
Merge branch 'master' of github.com:harlow/kinesis-consumer into harlow-master
2024-06-11 11:10:16 +02:00
dependabot[bot]
5e312aa870
Bump github.com/aws/aws-sdk-go-v2/service/sts from 1.28.10 to 1.28.12
...
Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2 ) from 1.28.10 to 1.28.12.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/efs/v1.28.10...service/fsx/v1.28.12 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
dependency-type: indirect
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-10 13:12:41 +00:00
Jarrad
553e2392fd
fix nil pointer dereference on AWS errors ( #148 )
...
* fix nil pointer dereference on AWS errors
* return Start errors to Scan consumer
before the previous commit e465b09📧 swift-nav.com>
2024-06-06 08:38:16 -07:00
gram-signal
6720a01733
Maintain parent/child shard ordering across shard splits/merges. ( #155 )
...
Kinesis allows clients to rely on an invariant that, for a given partition key, the order of records added to the stream will be maintained. IE: given an input `pkey=x,val=1 pkey=x,val=2 pkey=x,val=3`, the values `1,2,3` will be seen in that order when processed by clients, so long as clients are careful. It does so by putting all records for a single partition key into a single shard, then maintaining ordering within that shard.
However, shards can be split and merge, to distribute load better and handle per-shard throughput limits. Kinesis does this currently by (one or many times) splitting a single shard into two or by merging two adjacent shards into one. When this occurs, Kinesis still allows for ordering consistency by detailing shard parent/child relationships within its `listShards` outputs. A split shard A will create children B and C, both with `ParentShardId=A`. A merging of shards A and B into C will create a new shard C with `ParentShardId=A,AdjacentParentShardId=B`. So long as clients fully process all records in parents (including adjacent parents) before processing the new shard, ordering will be maintained.
`kinesis-consumer` currently doesn't do this. Instead, upon the initial (and subsequent) `listShards` call, all visible shards immediately begin processing. Considering this case, where shards split, then merge, and each shard `X` contains a single record `rX`:
```
time ->
B
/ \
A D
\ /
C
```
record `rD` should be processed after both `rB` and `rC` are processed, and both `rB` and `rC` should wait for `rA` to be processed. By starting goroutines immediately, any ordering of `{rA,rB,rC,rD}` might occur within the original code.
This PR utilizes the `AllGroup` as a book-keeper of fully processed shards, with the `Consumer` calling `CloseShard` once it has finished a shard. `AllGroup` doesn't release a shard for processing until its parents have fully been processed, and the consumer just processes the shards it receives as it used to.
This PR created a new `CloseableGroup` interface rather than append to the existing `Group` interface to maintain backwards compatibility in existing code that may already implement the `Group` interface elsewhere. Different `Group` implementations don't get the ordering described above, but the default `Consumer` does.
2024-06-06 08:37:42 -07:00
dependabot[bot]
b7242de6cf
Bump github.com/prometheus/procfs from 0.12.0 to 0.15.1
...
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs ) from 0.12.0 to 0.15.1.
- [Release notes](https://github.com/prometheus/procfs/releases )
- [Commits](https://github.com/prometheus/procfs/compare/v0.12.0...v0.15.1 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-03 12:42:15 +00:00
dependabot[bot]
caa313cf95
Bump github.com/prometheus/client_model from 0.5.0 to 0.6.1
...
Bumps [github.com/prometheus/client_model](https://github.com/prometheus/client_model ) from 0.5.0 to 0.6.1.
- [Release notes](https://github.com/prometheus/client_model/releases )
- [Commits](https://github.com/prometheus/client_model/compare/v0.5.0...v0.6.1 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/client_model
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 12:27:59 +00:00
Alex
9e7f1fd270
Merge pull request #48 from alexgridx/47-add-prometheus-metrics
...
#47 adds prometheus metrics
2024-05-28 12:10:26 +02:00
Alex Senger
c9c13a9914
#47 adds prometheus metrics
2024-05-28 12:09:19 +02:00
Alex
14ab603b79
Merge pull request #46 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue-1.13.19
...
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue from 1.13.14 to 1.13.19
2024-05-28 10:08:43 +02:00
dependabot[bot]
41539557e7
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
...
Bumps [github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue](https://github.com/aws/aws-sdk-go-v2 ) from 1.13.14 to 1.13.19.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.13.14...credentials/v1.13.19 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:08:34 +00:00
Alex
02c0ada62c
Merge pull request #36 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/dynamodbstreams-1.20.8
...
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams from 1.20.4 to 1.20.8
2024-05-28 10:07:47 +02:00
dependabot[bot]
92a8648de7
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams
...
Bumps [github.com/aws/aws-sdk-go-v2/service/dynamodbstreams](https://github.com/aws/aws-sdk-go-v2 ) from 1.20.4 to 1.20.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.20.4...service/ivs/v1.20.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/dynamodbstreams
dependency-type: indirect
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:07:39 +00:00
Alex
ef891eb523
Merge pull request #42 from alexgridx/dependabot/go_modules/github.com/yuin/gopher-lua-1.1.1
...
Bump github.com/yuin/gopher-lua from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1
2024-05-28 10:07:23 +02:00
Alex
3bf0572500
Merge pull request #35 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/kinesis-1.27.8
...
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8
2024-05-28 10:06:59 +02:00
dependabot[bot]
10dc9d7031
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8
...
Bumps [github.com/aws/aws-sdk-go-v2/service/kinesis](https://github.com/aws/aws-sdk-go-v2 ) from 1.27.4 to 1.27.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.4...config/v1.27.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/kinesis
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:06:07 +00:00
dependabot[bot]
c288cb88ef
Bump github.com/yuin/gopher-lua
...
Bumps [github.com/yuin/gopher-lua](https://github.com/yuin/gopher-lua ) from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1.
- [Release notes](https://github.com/yuin/gopher-lua/releases )
- [Commits](https://github.com/yuin/gopher-lua/commits/v1.1.1 )
---
updated-dependencies:
- dependency-name: github.com/yuin/gopher-lua
dependency-type: indirect
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:01:17 +00:00
Alex
fa5681358b
Merge pull request #32 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/config-1.27.16
...
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16
2024-05-28 09:54:09 +02:00
dependabot[bot]
c8a75ba8b5
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16
...
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2 ) from 1.27.11 to 1.27.16.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.11...config/v1.27.16 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 07:54:01 +00:00
Alex
9c040ee238
Merge pull request #31 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery-1.9.8
...
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery from 1.9.6 to 1.9.8
2024-05-28 09:53:05 +02:00
dependabot[bot]
cad20b6cc8
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
...
Bumps [github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery](https://github.com/aws/aws-sdk-go-v2 ) from 1.9.6 to 1.9.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pipes/v1.9.6...service/grafana/v1.9.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
dependency-type: indirect
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 07:52:17 +00:00
Alex
39d374f4fa
Merge pull request #29 from alexgridx/27-enable-dependabot
...
#27 enables dependabot
2024-05-07 11:04:55 +02:00
Alex Senger
ac83ac27e7
#27 enables dependabot
2024-05-07 11:03:38 +02:00
Alex
37b05ea426
Merge pull request #28 from alexgridx/26-upgrade-dependencies
...
#26 upgrades dependencies
2024-05-07 11:00:10 +02:00
Alex Senger
e73405c9cd
#26 upgrades dependencies
2024-05-07 10:58:18 +02:00
Alex
bbe5ed2a22
Merge pull request #22 from alexgridx/21-allow-overwriting-postgres-connection
...
#21 allows to overwrite postgres connection
2024-04-19 17:11:50 +02:00
Alex Senger
1e621394d5
#21 allows to overwrite postgres connection
2024-04-19 17:11:10 +02:00
Alex
c372167032
Merge pull request #20 from alexgridx/19-fix-retry-on-kinesis-error
...
#19 fixes retry on kinesis error
2024-04-19 13:33:41 +02:00
Alex Senger
ca7a109d1b
#19 fixes retry on kinesis error
2024-04-19 13:32:53 +02:00
Alex
2d05606a38
Merge pull request #18 from alexgridx/5-move-to-slog
...
#5 moves to slog
2024-04-19 13:31:21 +02:00
Alex Senger
95a08a91e8
#5 moves to slog
2024-04-19 13:29:57 +02:00
Alex
36e7fc8b89
Merge pull request #17 from alexgridx/16-update-examples
...
#16 updates examples
2024-04-12 10:02:14 +02:00
Alex Senger
3c37f72e41
#16 updates examples
2024-04-12 09:44:32 +02:00
Alex
8219b385d3
Merge pull request #15 from alexgridx/12-rename-package
...
#12 rename package
2024-04-10 17:20:26 +02:00
Alex Senger
7e0806126c
#12 rename package
2024-04-10 17:19:07 +02:00
Alex
c169ad4789
Merge pull request #13 from alexgridx/11-check-in-vendor-files
...
#11 checks in vendor files
2024-04-10 16:55:34 +02:00
Alex Senger
262dce8ba6
#11 checks in vendor files
2024-04-10 16:50:33 +02:00
Alex
b71c1964bc
Merge pull request #10 from alexgridx/9-introduce-linter
...
#9 introduce linter
2024-04-10 16:47:43 +02:00
Alex Senger
5100c7ce77
#9 introduces linter
2024-04-10 16:45:57 +02:00
Alex Senger
f0acb329f7
#9 fixes linting issues
2024-04-10 16:45:34 +02:00
Alex
fd263c3d60
Merge pull request #8 from alexgridx/7-use-awslabs-package
...
#7 imports awslabs aggregator in favor of local one
2024-04-10 16:43:52 +02:00
Alex Senger
eaf4defe57
#7 imports awslabs aggregator in favor of local one
2024-04-10 16:42:31 +02:00
Alex
1918e43f3e
Merge pull request #6 from alexgridx/4-minor-cleanups
...
#4 minor cleanups
2024-04-10 15:17:01 +02:00
