Commit Graph
53 Commits
Author SHA1 Message Date
Urban Ishimwe 6d812ceb7f changes plugins reader and writer method
// PluginReader is an interface for input plugins
type PluginReader interface {
	PluginRead() (msg *Message, err error)
}

// PluginWriter is an interface for output plugins
type PluginWriter interface {
	PluginWrite(msg *Message) (n int, err error)
}
2020-11-02 06:15:10 +02:00
Urban IshimweandGitHub fdc8b094f0 Benchmarking, packaging, and fix issues, tests and perfomance (#797)
### performance
- handling of the very big packet(any size that can be buffered)
- speeding up TCP sessions by using message hints: Added **proto.HasFullPayload** that helps to validate the entire HTTP request, it supports `Chunked` encoding too! Added **proto.HasRequestTitle** and **proto.HasResponseTitle** for validating the beginning of HTTP request. Those methods are used `input_raw.go` with `TCP`.
- supports Keep-Alive: the above functions helps to support keep-alive

### Packaging
- **capture:** engines(capture/doc.go)
- **tcp:** tcp message parser (tcp/doc.go)

### benchmarking
- **capture.BenchmarkPcapDump:** the benchmarks regarding dumping packets in a pcap file
- **capture.BenchmarkPcapFile:** the benchmarks of reading packets from a pcap file
- **capture.BenchmarkPcap:** the benchmarks of parsing packets from the loopback interface with pcap handles
- **proto.BenchmarkHasFullPayload:**: benchmarking this function which validates the HTTP payload
- **tcp.BenchmarkPacketParseAndSort:** benchmarks of parsing and sorting packets
- **tcp.BenchmarkMessageParserWithoutHint:** benchmarks of message reasembling by using `SYN` and `FIN` flag
- **tcp.BenchmarkMessageParserWithHint:** benchmarks of message reasembling by using `proto.HasRequestTitle` and `proto.HasFullPayload` flag

### issues
see linked issues

###  tests
- fixed input raw and engine tests

**Most of the changed of the files, was about using functionalities of** `tcp` **and** `capture` **in existing functionalities**
2020-08-11 12:44:53 +03:00
arijitad 01b6f1129c Restructre AppSetting and address comments. 2020-07-23 10:00:16 +05:30
arijitad 80b19f918b Map Appsetting to flag names. 2020-07-15 00:14:38 +05:30
Romanenko AlexeyandGitHub ea822d83da Correct size for gz file (#777)
Fix #775 

When we are using .gz for output we are counting incorrect file size. Now it should be ok
2020-06-23 18:11:45 +03:00
Leonid Bugaev d67bc13db3 Merging PRO functionality back to OSS repo 2020-06-08 14:29:32 +03:00
Alexey Romanenko 261c592b83 fix parse settings; fix typo 2020-06-04 23:20:23 +07:00
Arijit DasandGitHub df14b29e74 Fix race when writing to the file. (#756)
Reproduce: Run TestInputFileWithGET with -race flag
#755 needs to go before this is merged.
2020-05-27 16:52:48 +03:00
Urban IshimweandGitHub 7e90a73b81 parse buffer from different bases and data unit: (#754)
All flags that expect buffer as input i.e. `--output-file-size-limit`, `--output-file-max-size-limit`, `--copy-buffer-size` and `input-raw-buffer-size` can now parse  inputs from differents bases and data units like: `10mb`, `10kb`, `100gb`,   `18tb`, `11839023`.... 
data units and bases are case insensitive, the parser accepts only the format of [Go integer literals](https://golang.org/ref/spec#Integer_literals)
2020-05-26 18:05:59 +03:00
Leonid Bugaev 7300b83894 Merge remote-tracking branch 'upstream/master' 2019-03-29 22:50:04 +07:00
Leonid Bugaev c48d07d2c6 Merge remote-tracking branch 'upstream/master' 2019-03-29 22:45:27 +07:00
Leonid Bugaev a8cfaa7581 Add --output-http-compatibility-mode
Add support for using default Go HTTP client
2019-03-10 10:29:28 +01:00
Leonid Bugaev 264d8cf00e Add more error handling 2019-02-16 09:37:00 +01:00
Leonid Bugaev 85c6070775 File output now case limit by size
Added new `output-file-max-size-limit` option
2019-02-16 08:49:31 +01:00
Leonid Bugaev d0d1527d9b Merge remote-tracking branch 'upstream/master' 2018-08-08 22:00:20 +05:00
Leonid Bugaev eeb09a1494 Merge 2018-08-02 21:42:45 +05:00
Leonid Bugaev 0097c599df Fix response latency calculation
Previously latency calcualted as Response.End - Request.Start
Where both End and Start is a last and first packets

This calcualtion is wrong, because it is total roundtrip

Correct server latency will be Response.End - Request.End

In addition added new `--input-raw-timestamp-type` option
which allows choose more precise packet timestamp source (if available).
2018-02-05 21:47:48 +02:00
Leonid Bugaev 0f8e2f2d52 Merge branch 'master' of https://github.com/buger/gor 2017-08-22 20:43:22 +05:00
SoulZxzandLeonid Bugaev eb8eb1db52 fix file rotate issues 2017-08-22 19:12:51 +05:00
Leonid Bugaev 80777061a7 Merge remote-tracking branch 'upstream/master' 2017-05-01 17:22:45 +02:00
Leonid Bugaev 2104bec5be Fix S3 2017-04-29 11:31:30 +02:00
Leonid Bugaev dcc7af4c09 Fix output-file sizeLimit 2017-04-07 15:09:16 +02:00
Leonid Bugaev 9173379cc8 Merge branch 'master' of https://github.com/buger/gor 2017-02-05 17:26:44 +03:00
Leonid Bugaev 6a48882125 Fix infinite lock at file output 2017-02-05 17:26:22 +03:00
Leonid Bugaev 1a32de3dc6 Merge branch 'master' of https://github.com/buger/gor 2017-01-23 20:04:33 +03:00
Leonid BugaevandGitHub ecb16664db Allow to use payload type for file names (#417) 2017-01-23 16:40:14 +04:00
Leonid BugaevandGitHub 8c16220532 Track http responses (#416) 2017-01-23 16:20:27 +04:00
Leonid Bugaev 590304d687 Ensure that flush do not get called on closed file 2017-01-23 14:43:43 +03:00
Leonid Bugaev 973b028517 Merge branch 'master' into s3 2017-01-23 13:27:56 +03:00
Sean Lewis b0b697cba6 flush the output buffer when it should be flushed
fixes #409
2017-01-04 14:59:39 -07:00
Sergey Mostovoy 5909dd71c5 Support %r - request id - output file pattern #381 2016-12-27 12:18:49 +02:00
Leonid Bugaev cd51e09fcd Merge branch 'master' into s3 2016-12-19 20:25:06 +03:00
Leonid Bugaev 544d9402f0 Disable forced flush 2016-08-08 18:09:14 +03:00
Leonid Bugaev e38f6a5801 Do not quite on flush error 2016-08-02 12:55:02 +03:00
Leonid Bugaev 55fe5b9c11 s3-output 2016-07-08 13:20:33 +03:00
Marc LallaouretandLeonid Bugaev 2c49556b2e Clean current output file name (#304) 2016-06-16 22:33:55 +06:00
Marc LallaouretandLeonid Bugaev d0e1858eb3 Fix Close implementation of plugins (#305)
Indeed they do not declare error as return type and so, were not called at the end of the program
2016-06-16 22:33:07 +06:00
Leonid Bugaev 95eecb6b1c Output file fixes
* Fixed flush/Write race
* Fixed names with underscores
2016-06-14 20:27:52 +05:00
Leonid Bugaev 06d129d20d Add support for injecting real ip (#296) 2016-06-09 19:06:59 +06:00
Leonid Bugaev cefa2b5cc2 Write files in chunks (#293)
* Chunked output strategy
* Fix sorting with indexes >= 10
2016-06-07 18:46:13 +06:00
Leonid Bugaev b6f7de00c5 Add compression support 2016-05-31 16:41:01 +05:00
Leonid Bugaev ba98e6e18a Allow date templates for file names 2016-05-31 15:52:35 +05:00
Leonid Bugaev b53fba0415 Fix formatting issues 2015-08-18 17:28:09 +03:00
Leonid Bugaev 7790b33cea Refactor protocol scanner 2015-08-18 15:52:23 +03:00
Leonid Bugaev a971d8a843 Change file format to be text based 2015-08-18 15:48:42 +03:00
Leonid Bugaev 3e763c43cf Unify all input/output plugins 2015-08-18 14:32:52 +03:00
Leonid Bugaev 5cc4228dcd Merge branch 'master' into input-modifier
Conflicts:
	Dockerfile
	emitter.go
	http_client.go
	plugins.go
	proto/proto.go
2015-07-21 09:49:37 +05:00
Leonid Bugaev 125d9db834 Next batch 2015-07-17 11:30:43 +05:00
Leonid Bugaev 67cfca0ede Make limiter work for all inputs/outputs 2014-10-26 01:02:49 +04:00
Leonid Bugaev 2d66ccc62d Add limiter support to output-file 2014-10-13 19:37:43 +04:00