dependabot[bot]
d4c2493caf
Bump github.com/aws/aws-sdk-go-v2/service/dynamodb from 1.32.8 to 1.34.1
...
Bumps [github.com/aws/aws-sdk-go-v2/service/dynamodb](https://github.com/aws/aws-sdk-go-v2 ) from 1.32.8 to 1.34.1.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/service/s3/v1.34.1/CHANGELOG.md )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/dynamodb/v1.32.8...service/s3/v1.34.1 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/dynamodb
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-07-09 08:29:28 +00:00
Alex
061285078c
Merge pull request #119 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/kinesis-1.29.1
...
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.8 to 1.29.1
2024-07-09 10:28:41 +02:00
dependabot[bot]
caeb4b4907
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.8 to 1.29.1
...
Bumps [github.com/aws/aws-sdk-go-v2/service/kinesis](https://github.com/aws/aws-sdk-go-v2 ) from 1.27.8 to 1.29.1.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.8...service/s3/v1.29.1 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/kinesis
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-07-09 07:24:54 +00:00
Alex
e5f5c42a16
Merge pull request #126 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/config-1.27.24
...
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.16 to 1.27.24
2024-07-09 09:23:40 +02:00
dependabot[bot]
8f2e30a83e
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.16 to 1.27.24
...
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2 ) from 1.27.16 to 1.27.24.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.16...config/v1.27.24 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-07-09 07:16:38 +00:00
Alex
1edf359f0d
Merge pull request #127 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/credentials-1.17.24
...
Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.16 to 1.17.24
2024-07-09 09:15:38 +02:00
dependabot[bot]
12b2c0f310
Bump github.com/aws/aws-sdk-go-v2/credentials from 1.17.16 to 1.17.24
...
Bumps [github.com/aws/aws-sdk-go-v2/credentials](https://github.com/aws/aws-sdk-go-v2 ) from 1.17.16 to 1.17.24.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.17.16...credentials/v1.17.24 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/credentials
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-07-04 13:01:23 +00:00
Alex
a46a055ee0
Merge pull request #67 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue-1.14.1
...
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue from 1.13.19 to 1.14.1
2024-06-11 11:18:06 +02:00
dependabot[bot]
e9784dbcd8
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
...
Bumps [github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue](https://github.com/aws/aws-sdk-go-v2 ) from 1.13.19 to 1.14.1.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/credentials/v1.13.19...service/pi/v1.14.1 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:17:59 +00:00
Alex
146ee3041e
Merge pull request #68 from alexgridx/dependabot/go_modules/github.com/redis/go-redis/v9-9.5.3
...
Bump github.com/redis/go-redis/v9 from 9.5.1 to 9.5.3
2024-06-11 11:17:54 +02:00
dependabot[bot]
4606e103b2
Bump github.com/redis/go-redis/v9 from 9.5.1 to 9.5.3
...
Bumps [github.com/redis/go-redis/v9](https://github.com/redis/go-redis ) from 9.5.1 to 9.5.3.
- [Release notes](https://github.com/redis/go-redis/releases )
- [Changelog](https://github.com/redis/go-redis/blob/master/CHANGELOG.md )
- [Commits](https://github.com/redis/go-redis/compare/v9.5.1...v9.5.3 )
---
updated-dependencies:
- dependency-name: github.com/redis/go-redis/v9
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:17:43 +00:00
Alex
3599a58a6e
Merge pull request #69 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery-1.9.10
...
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery from 1.9.8 to 1.9.10
2024-06-11 11:17:41 +02:00
Alex
ffc4d56786
Merge pull request #71 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/dynamodb-1.32.8
...
Bump github.com/aws/aws-sdk-go-v2/service/dynamodb from 1.32.5 to 1.32.8
2024-06-11 11:17:31 +02:00
Alex
7d87dc30e4
Merge pull request #65 from alexgridx/dependabot/go_modules/golang.org/x/sys-0.21.0
...
Bump golang.org/x/sys from 0.17.0 to 0.21.0
2024-06-11 11:17:19 +02:00
dependabot[bot]
3c62315bd8
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
...
Bumps [github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery](https://github.com/aws/aws-sdk-go-v2 ) from 1.9.8 to 1.9.10.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/grafana/v1.9.8...service/grafana/v1.9.10 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
dependency-type: indirect
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:17:09 +00:00
dependabot[bot]
4f1f924151
Bump golang.org/x/sys from 0.17.0 to 0.21.0
...
Bumps [golang.org/x/sys](https://github.com/golang/sys ) from 0.17.0 to 0.21.0.
- [Commits](https://github.com/golang/sys/compare/v0.17.0...v0.21.0 )
---
updated-dependencies:
- dependency-name: golang.org/x/sys
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:17:04 +00:00
Alex
bf51213698
Merge pull request #62 from alexgridx/dependabot/go_modules/github.com/prometheus/common-0.54.0
...
Bump github.com/prometheus/common from 0.48.0 to 0.54.0
2024-06-11 11:17:02 +02:00
dependabot[bot]
839e981436
Bump github.com/aws/aws-sdk-go-v2/service/dynamodb from 1.32.5 to 1.32.8
...
Bumps [github.com/aws/aws-sdk-go-v2/service/dynamodb](https://github.com/aws/aws-sdk-go-v2 ) from 1.32.5 to 1.32.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/fsx/v1.32.5...service/dynamodb/v1.32.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/dynamodb
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:16:53 +00:00
dependabot[bot]
935d8cb78e
Bump github.com/prometheus/common from 0.48.0 to 0.54.0
...
Bumps [github.com/prometheus/common](https://github.com/prometheus/common ) from 0.48.0 to 0.54.0.
- [Release notes](https://github.com/prometheus/common/releases )
- [Changelog](https://github.com/prometheus/common/blob/main/RELEASE.md )
- [Commits](https://github.com/prometheus/common/compare/v0.48.0...v0.54.0 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/common
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-11 09:16:52 +00:00
Alex
ff592832c2
Merge pull request #58 from alexgridx/dependabot/go_modules/github.com/prometheus/procfs-0.15.1
...
Bump github.com/prometheus/procfs from 0.12.0 to 0.15.1
2024-06-11 11:16:27 +02:00
Alex
496e9243c6
Merge pull request #51 from alexgridx/dependabot/go_modules/github.com/prometheus/client_model-0.6.1
...
Bump github.com/prometheus/client_model from 0.5.0 to 0.6.1
2024-06-11 11:16:11 +02:00
Alex
b72a9328da
Merge pull request #72 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/sts-1.28.12
...
Bump github.com/aws/aws-sdk-go-v2/service/sts from 1.28.10 to 1.28.12
2024-06-11 11:15:50 +02:00
Alex Senger
0ee8cc4623
Merge branch 'harlow-master'
2024-06-11 11:15:01 +02:00
Alex Senger
a8e4ff111c
Merge branch 'master' of github.com:harlow/kinesis-consumer into harlow-master
2024-06-11 11:10:16 +02:00
dependabot[bot]
5e312aa870
Bump github.com/aws/aws-sdk-go-v2/service/sts from 1.28.10 to 1.28.12
...
Bumps [github.com/aws/aws-sdk-go-v2/service/sts](https://github.com/aws/aws-sdk-go-v2 ) from 1.28.10 to 1.28.12.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/efs/v1.28.10...service/fsx/v1.28.12 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/sts
dependency-type: indirect
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-10 13:12:41 +00:00
Jarrad
553e2392fd
fix nil pointer dereference on AWS errors ( #148 )
...
* fix nil pointer dereference on AWS errors
* return Start errors to Scan consumer
before the previous commit e465b09📧 swift-nav.com>
2024-06-06 08:38:16 -07:00
gram-signal
6720a01733
Maintain parent/child shard ordering across shard splits/merges. ( #155 )
...
Kinesis allows clients to rely on an invariant that, for a given partition key, the order of records added to the stream will be maintained. IE: given an input `pkey=x,val=1 pkey=x,val=2 pkey=x,val=3`, the values `1,2,3` will be seen in that order when processed by clients, so long as clients are careful. It does so by putting all records for a single partition key into a single shard, then maintaining ordering within that shard.
However, shards can be split and merge, to distribute load better and handle per-shard throughput limits. Kinesis does this currently by (one or many times) splitting a single shard into two or by merging two adjacent shards into one. When this occurs, Kinesis still allows for ordering consistency by detailing shard parent/child relationships within its `listShards` outputs. A split shard A will create children B and C, both with `ParentShardId=A`. A merging of shards A and B into C will create a new shard C with `ParentShardId=A,AdjacentParentShardId=B`. So long as clients fully process all records in parents (including adjacent parents) before processing the new shard, ordering will be maintained.
`kinesis-consumer` currently doesn't do this. Instead, upon the initial (and subsequent) `listShards` call, all visible shards immediately begin processing. Considering this case, where shards split, then merge, and each shard `X` contains a single record `rX`:
```
time ->
B
/ \
A D
\ /
C
```
record `rD` should be processed after both `rB` and `rC` are processed, and both `rB` and `rC` should wait for `rA` to be processed. By starting goroutines immediately, any ordering of `{rA,rB,rC,rD}` might occur within the original code.
This PR utilizes the `AllGroup` as a book-keeper of fully processed shards, with the `Consumer` calling `CloseShard` once it has finished a shard. `AllGroup` doesn't release a shard for processing until its parents have fully been processed, and the consumer just processes the shards it receives as it used to.
This PR created a new `CloseableGroup` interface rather than append to the existing `Group` interface to maintain backwards compatibility in existing code that may already implement the `Group` interface elsewhere. Different `Group` implementations don't get the ordering described above, but the default `Consumer` does.
2024-06-06 08:37:42 -07:00
dependabot[bot]
b7242de6cf
Bump github.com/prometheus/procfs from 0.12.0 to 0.15.1
...
Bumps [github.com/prometheus/procfs](https://github.com/prometheus/procfs ) from 0.12.0 to 0.15.1.
- [Release notes](https://github.com/prometheus/procfs/releases )
- [Commits](https://github.com/prometheus/procfs/compare/v0.12.0...v0.15.1 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/procfs
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-06-03 12:42:15 +00:00
dependabot[bot]
caa313cf95
Bump github.com/prometheus/client_model from 0.5.0 to 0.6.1
...
Bumps [github.com/prometheus/client_model](https://github.com/prometheus/client_model ) from 0.5.0 to 0.6.1.
- [Release notes](https://github.com/prometheus/client_model/releases )
- [Commits](https://github.com/prometheus/client_model/compare/v0.5.0...v0.6.1 )
---
updated-dependencies:
- dependency-name: github.com/prometheus/client_model
dependency-type: indirect
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 12:27:59 +00:00
Alex
9e7f1fd270
Merge pull request #48 from alexgridx/47-add-prometheus-metrics
...
#47 adds prometheus metrics
2024-05-28 12:10:26 +02:00
Alex Senger
c9c13a9914
#47 adds prometheus metrics
2024-05-28 12:09:19 +02:00
Alex
14ab603b79
Merge pull request #46 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue-1.13.19
...
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue from 1.13.14 to 1.13.19
2024-05-28 10:08:43 +02:00
dependabot[bot]
41539557e7
Bump github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
...
Bumps [github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue](https://github.com/aws/aws-sdk-go-v2 ) from 1.13.14 to 1.13.19.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.13.14...credentials/v1.13.19 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:08:34 +00:00
Alex
02c0ada62c
Merge pull request #36 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/dynamodbstreams-1.20.8
...
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams from 1.20.4 to 1.20.8
2024-05-28 10:07:47 +02:00
dependabot[bot]
92a8648de7
Bump github.com/aws/aws-sdk-go-v2/service/dynamodbstreams
...
Bumps [github.com/aws/aws-sdk-go-v2/service/dynamodbstreams](https://github.com/aws/aws-sdk-go-v2 ) from 1.20.4 to 1.20.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/mq/v1.20.4...service/ivs/v1.20.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/dynamodbstreams
dependency-type: indirect
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:07:39 +00:00
Alex
ef891eb523
Merge pull request #42 from alexgridx/dependabot/go_modules/github.com/yuin/gopher-lua-1.1.1
...
Bump github.com/yuin/gopher-lua from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1
2024-05-28 10:07:23 +02:00
Alex
3bf0572500
Merge pull request #35 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/kinesis-1.27.8
...
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8
2024-05-28 10:06:59 +02:00
dependabot[bot]
10dc9d7031
Bump github.com/aws/aws-sdk-go-v2/service/kinesis from 1.27.4 to 1.27.8
...
Bumps [github.com/aws/aws-sdk-go-v2/service/kinesis](https://github.com/aws/aws-sdk-go-v2 ) from 1.27.4 to 1.27.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.4...config/v1.27.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/kinesis
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:06:07 +00:00
dependabot[bot]
c288cb88ef
Bump github.com/yuin/gopher-lua
...
Bumps [github.com/yuin/gopher-lua](https://github.com/yuin/gopher-lua ) from 0.0.0-20200603152657-dc2b0ca8b37e to 1.1.1.
- [Release notes](https://github.com/yuin/gopher-lua/releases )
- [Commits](https://github.com/yuin/gopher-lua/commits/v1.1.1 )
---
updated-dependencies:
- dependency-name: github.com/yuin/gopher-lua
dependency-type: indirect
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 08:01:17 +00:00
Alex
fa5681358b
Merge pull request #32 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/config-1.27.16
...
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16
2024-05-28 09:54:09 +02:00
dependabot[bot]
c8a75ba8b5
Bump github.com/aws/aws-sdk-go-v2/config from 1.27.11 to 1.27.16
...
Bumps [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2 ) from 1.27.11 to 1.27.16.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/config/v1.27.11...config/v1.27.16 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/config
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 07:54:01 +00:00
Alex
9c040ee238
Merge pull request #31 from alexgridx/dependabot/go_modules/github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery-1.9.8
...
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery from 1.9.6 to 1.9.8
2024-05-28 09:53:05 +02:00
dependabot[bot]
cad20b6cc8
Bump github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
...
Bumps [github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery](https://github.com/aws/aws-sdk-go-v2 ) from 1.9.6 to 1.9.8.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases )
- [Commits](https://github.com/aws/aws-sdk-go-v2/compare/service/pipes/v1.9.6...service/grafana/v1.9.8 )
---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/service/internal/endpoint-discovery
dependency-type: indirect
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
2024-05-28 07:52:17 +00:00
Alex
39d374f4fa
Merge pull request #29 from alexgridx/27-enable-dependabot
...
#27 enables dependabot
2024-05-07 11:04:55 +02:00
Alex Senger
ac83ac27e7
#27 enables dependabot
2024-05-07 11:03:38 +02:00
Alex
37b05ea426
Merge pull request #28 from alexgridx/26-upgrade-dependencies
...
#26 upgrades dependencies
2024-05-07 11:00:10 +02:00
Alex Senger
e73405c9cd
#26 upgrades dependencies
2024-05-07 10:58:18 +02:00
Alex
bbe5ed2a22
Merge pull request #22 from alexgridx/21-allow-overwriting-postgres-connection
...
#21 allows to overwrite postgres connection
2024-04-19 17:11:50 +02:00
Alex Senger
1e621394d5
#21 allows to overwrite postgres connection
2024-04-19 17:11:10 +02:00
Alex
c372167032
Merge pull request #20 from alexgridx/19-fix-retry-on-kinesis-error
...
#19 fixes retry on kinesis error
2024-04-19 13:33:41 +02:00
