luciano/kinesis-consumer
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
271 commits 6 branches 9 tags 7.5 MiB
Commit graph

271 commits

This branch
This branch
All branches
Author SHA1 Message Date
Alex
39d374f4fa
Merge pull request #29 from alexgridx/27-enable-dependabot 
#27 enables dependabot
 2024-05-07 11:04:55 +02:00
Alex Senger
ac83ac27e7
#27 enables dependabot 2024-05-07 11:03:38 +02:00
Alex
37b05ea426
Merge pull request #28 from alexgridx/26-upgrade-dependencies 
#26 upgrades dependencies
 2024-05-07 11:00:10 +02:00
Alex Senger
e73405c9cd
#26 upgrades dependencies 2024-05-07 10:58:18 +02:00
Alex
bbe5ed2a22
Merge pull request #22 from alexgridx/21-allow-overwriting-postgres-connection 
#21 allows to overwrite postgres connection
 2024-04-19 17:11:50 +02:00
Alex Senger
1e621394d5
#21 allows to overwrite postgres connection 2024-04-19 17:11:10 +02:00
Alex
c372167032
Merge pull request #20 from alexgridx/19-fix-retry-on-kinesis-error 
#19 fixes retry on kinesis error
 2024-04-19 13:33:41 +02:00
Alex Senger
ca7a109d1b
#19 fixes retry on kinesis error 2024-04-19 13:32:53 +02:00
Alex
2d05606a38
Merge pull request #18 from alexgridx/5-move-to-slog 
#5 moves to slog
 2024-04-19 13:31:21 +02:00
Alex Senger
95a08a91e8
#5 moves to slog 2024-04-19 13:29:57 +02:00
Alex
36e7fc8b89
Merge pull request #17 from alexgridx/16-update-examples 
#16 updates examples
 2024-04-12 10:02:14 +02:00
Alex Senger
3c37f72e41
#16 updates examples 2024-04-12 09:44:32 +02:00
Alex
8219b385d3
Merge pull request #15 from alexgridx/12-rename-package 
#12 rename package
 2024-04-10 17:20:26 +02:00
Alex Senger
7e0806126c
#12 rename package 2024-04-10 17:19:07 +02:00
Alex
c169ad4789
Merge pull request #13 from alexgridx/11-check-in-vendor-files 
#11 checks in vendor files
 2024-04-10 16:55:34 +02:00
Alex Senger
262dce8ba6
#11 checks in vendor files 2024-04-10 16:50:33 +02:00
Alex
b71c1964bc
Merge pull request #10 from alexgridx/9-introduce-linter 
#9 introduce linter
 2024-04-10 16:47:43 +02:00
Alex Senger
5100c7ce77
#9 introduces linter 2024-04-10 16:45:57 +02:00
Alex Senger
f0acb329f7
#9 fixes linting issues 2024-04-10 16:45:34 +02:00
Alex
fd263c3d60
Merge pull request #8 from alexgridx/7-use-awslabs-package 
#7 imports awslabs aggregator in favor of local one
 2024-04-10 16:43:52 +02:00
Alex Senger
eaf4defe57
#7 imports awslabs aggregator in favor of local one 2024-04-10 16:42:31 +02:00
Alex
1918e43f3e
Merge pull request #6 from alexgridx/4-minor-cleanups 
#4 minor cleanups
 2024-04-10 15:17:01 +02:00
Alex Senger
040fa06efa
#4 minor cleanups 2024-04-10 15:16:07 +02:00
Alex
a5089538fd
Merge pull request #3 from alexgridx/1-upgrade-dependencies 
#1 Removes Import Renames
 2024-04-10 15:01:24 +02:00
Alex Senger
a38bb99afe
#1 removes import renames 2024-04-10 15:00:24 +02:00
Alex
392532b6e8
Merge pull request #2 from alexgridx/1-upgrade-dependencies 
#1 Upgrades Dependencies
 2024-04-10 14:59:42 +02:00
Alex Senger
c7f3c79ff7
#1 upgrades dependencies 2024-04-10 14:58:37 +02:00
guangwu
188bdff278
fix: typo (#156) 2023-12-14 16:36:20 -08:00
Dario Emerson
af2db0d43f
update go-redis to v9 (#150) 2023-06-16 11:20:07 -07:00
Harlow Ward
c2b9f79d7a Put aws-skd-v2 note in installation section 2021-12-04 13:43:18 -08:00
Harlow Ward
6cbda0f706
Update ddb checkpoint item to use dynamodbav (#144) 
Update the DynamoDB checkpoint item to use the `dynamodbav` value for marshaling. 

Fixes: https://github.com/harlow/kinesis-consumer/issues/142

Minor changes:
* Update the DDB example consumer to support a local version of DDB for streamlined testing
 2021-12-04 13:40:26 -08:00
Harlow Ward
3b3b252fa5 Add all examples to example directory 2021-12-04 10:42:17 -08:00
Harlow Ward
61fa84eca6
Update to use aws-sdk-go-v2 (#141) 2021-09-21 22:00:14 -07:00
Harlow Ward
8257129066
Remove note about consumer groups (#139) 2021-09-02 11:20:14 -07:00
jonandrewj
a334486111
Implement a WithShardClosedHandler option for the consumer (#135) 2021-07-30 14:16:15 -07:00
jonandrewj
c75a9237b6
Add InternalFailureException to the list of retriable errors (#132) 
* Add InternalFailureException to the list of retriable errors
 2021-02-25 14:54:48 -08:00
Frank Meyer
27055f2ace
Wrap underlying errors via %w verb (#130) 
As introduced in Go 1.13. This enables user of this library
to check for an underlying wrapped error type via errors.Is and
errors.As functions.
 2021-02-05 09:10:34 -08:00
James Regovich
799ccf2d40
Add support for aggregated records (#127) 
Add config option for aggregated records and deaggregation on records in ScanShard

This PR adds an option to consume aggregated records.
 2020-10-13 20:41:18 -07:00
Jason Tackaberry
e60d217333
Include MillisBehindLatest in Record for ScanFunc (#124) 2020-08-01 22:05:17 -07:00
Jason Tackaberry
3f2519e51c
Run initial scan immediately (#123) 
Rather than starting the shard scan loop and waiting for next scan tick before
fetching any data, do the first poll immediately, and then wait.
 2020-08-01 15:45:37 -07:00
Jason Tackaberry
97ffabeaa5
Include ShardID in Record passed to ScanFunc (#121) 
* Include ShardID in Record passed to ScanFunc
* Update mock to explicitly use kinesis.Record

Supports change wherein consumer.Record is changed from an alias of
kinesis.Record to a composition containing it.
 2020-07-30 14:18:38 -07:00
Harlow Ward
bae065cf53 Make sure gofmt run on all files 2020-07-21 20:31:38 -07:00
Harlow Ward
3b95644d77 Move license to file without prefix 2020-07-21 20:31:04 -07:00
Nicolas Augusto Sassi
ef5ce02f91
Update README.md (#119) 
fix typo
 2020-07-21 20:27:21 -07:00
Harlow Ward
89db667ce5
Bump AWS SDK to v1.33.3 (#118) 
* Bump AWS SDK to v1.33.3
* Bump redis version
 2020-07-21 20:27:03 -07:00
chumbert2
400ef07463
Allow to override Limit parameter in GetRecords (#113) 
Add option maxRecords to set the maximum number of records that can
be returned by GetRecords. Default value: 10,000. Use WithMaxRecords
to change the default value.

See

https://docs.aws.amazon.com/sdk-for-go/api/service/kinesis/#GetRecordsInput
https://docs.aws.amazon.com/streams/latest/dev/service-sizes-and-limits.html
 2020-04-27 21:12:20 -07:00
Jason Pepper
dcd9d048fb
Don't send StreamName when calling ListShards with NextToken. (#110) 
Co-authored-by: Jason Pepper <jason.pepper@crypsisgroup.com>
 2020-03-19 21:45:53 -07:00
0livd
bc5c012fd3 Add scanInterval option (#105) 
By default a consumer will scan for records every 250ms.
This interval can be configured with WithScanInterval.
 2020-01-17 10:22:10 -08:00
Patrik Karlström
217999854b Only initialize ddb client if none is provided (#106) 2020-01-17 10:02:37 -08:00
Andrew Shannon Brown
f85f25c15e Add an in-memory checkpoint to the API (#103) 
* Add an in-memory checkpoint to the API

* Rename memory to store

* Rename test package to store
 2019-09-08 13:13:04 -07:00