Commit Graph
1163 Commits
Author SHA1 Message Date
Urban Ishimwe 6d812ceb7f changes plugins reader and writer method
// PluginReader is an interface for input plugins
type PluginReader interface {
	PluginRead() (msg *Message, err error)
}

// PluginWriter is an interface for output plugins
type PluginWriter interface {
	PluginWrite(msg *Message) (n int, err error)
}
2020-11-02 06:15:10 +02:00
Urban IshimweandGitHub 20435af11b Merge pull request #839 from davidFR/master
Fix #838 : Kafka in TLS (not only mutual TLS)
2020-11-02 06:14:19 +02:00
David FradinandGitHub f719071358 Update kafka.go
Correct invalid character
2020-10-20 16:16:06 +02:00
716fb72e27 Update kafka.go
Co-authored-by: Urban Ishimwe <urbainishimwe@gmail.com>
2020-10-20 16:11:26 +02:00
3fcdb6f1bc Update kafka.go
Co-authored-by: Urban Ishimwe <urbainishimwe@gmail.com>
2020-10-20 16:11:19 +02:00
715ce88a63 Update kafka.go
Co-authored-by: Urban Ishimwe <urbainishimwe@gmail.com>
2020-10-20 16:11:07 +02:00
David FRADIN bcf81c6410 Kafka TLS 2020-10-19 13:57:34 +02:00
David FradinandGitHub 358cb78a53 Fix #835: kafka bug (#836)
Correction of error introduce in PR #800
2020-10-19 10:56:07 +03:00
Urban IshimweandGitHub 48aa326ba6 engine benchmarks (#837) 2020-10-19 10:55:47 +03:00
Urban IshimweandGitHub 5d8ca525a4 deprecate output-http non-compatible clients (#833)
for easy readability check changes by their underlying commits!

these benchmarks address the whole operation of the request cycle in goreplay.

_**goos: linux
goarch: amd64**_

**Using Compatible client:**

```
BenchmarkHTTPOutput-4      	   10417	    118969 ns/op	   12172 B/op	      93 allocs/op
BenchmarkHTTPOutputTLS-4   	    9136	    132929 ns/op	   12448 B/op	      97 allocs/op
```

**Using non-compatible client**
```
BenchmarkHTTPOutput-4      	     859	   1175040 ns/op	   15598 B/op	      46 allocs/op
BenchmarkHTTPOutputTLS-4   	     880	   1189643 ns/op	   15544 B/op	      52 allocs/op

```
Binary size reduced: **7%**

from these benchmarks, we may trade allocations with performance and memory!
2020-10-13 08:36:16 +03:00
Urban IshimweandGitHub 3635d66a76 Merge pull request #818 from DimaGolomozy/increase-latency-delta
increase-latency-delta-tests
2020-10-11 21:33:37 +02:00
Urban IshimweandGitHub a4e78cf22c Merge pull request #830 from buger/tcp-output-distribute-workers
Tcp output distribute workers fairly in non-sticky env
2020-10-07 09:38:54 +02:00
Urban IshimweandGitHub 8aedc15027 Merge pull request #827 from tomerf/syscall_error_check
Fixed check of syscall errors
2020-10-05 21:59:52 +02:00
Urban Ishimwe 30d66d31de distribute tasks in workers 2020-09-30 14:38:21 +02:00
Urban Ishimwe b1d7c4134b debug on stderr 2020-09-30 14:01:05 +02:00
Tomer Froumin 66e680a56f Fixed check of syscall errors 2020-09-29 10:50:22 +03:00
Urban IshimweandGitHub a4ed67acd6 Merge pull request #826 from buger/fix-buf-has-http-title
fix #825
2020-09-25 09:21:34 +02:00
Urban Ishimwe ee9ff8163b fix #825 2020-09-24 15:39:20 +02:00
Urban IshimweandGitHub 9519b9a9f5 Reduce allocation and unnecessary layers (#822)
The focus here was to **reduce allocation in TCP parser** but speed may have hopeful improved too!
pool no longer use map's key of **string** it uses **uint64**
**Benchmarks** was revamped to be more clear
if you want to compare these results copy the benchmark in tcp/bench_test.go@reduce-allocation to tcp/bench_test.go@master:

**before(master)**:
```
BenchmarkPacketParseAndSort-4         	 1000000	      1006 ns/op	      64 B/op	       2 allocs/op
BenchmarkMessageParserWithoutHint-4   	     625	   1772309 ns/op	      1000 packets/op	  419096 B/op	   10045 allocs/op
BenchmarkMessageParserWithHint-4      	      74	  14969926 ns/op	      1000 chunks/op	      1002 packets/op	  450992 B/op	   10126 allocs/op
```

**After(this branch)**:
```
BenchmarkPacketParseAndSort-4         	 1267662	       941 ns/op	      64 B/op	       2 allocs/op
BenchmarkMessageParserWithoutHint-4   	    2256	    523474 ns/op	      1000 packets/op	  243530 B/op	    1037 allocs/op
BenchmarkMessageParserWithHint-4      	      80	  13990955 ns/op	      1000 chunks/op	      1002 packets/op	  268609 B/op	    1099 allocs/op

```
2020-09-22 21:14:31 +03:00
Tomer FrouminandGitHub 3401dfbf2d Make the number of TCP connections configurable (#819) 2020-09-22 21:13:52 +03:00
Urban IshimweandGitHub cfa14b0a32 Merge pull request #824 from swills/size_int64
Use int64 for Size
2020-09-21 19:53:29 +02:00
Steve Wills 85065cb41a Use int64 for Size 2020-09-21 13:43:02 -04:00
Urban IshimweandGitHub 7100ac3593 fix add header out of bounds (#820)
this fixes the issue that arises in adding an HTTP header in an invalid HTTP request(request without CRLF)
2020-09-17 15:16:48 +03:00
Dima Golomozy d5184c6cbd faster latency is not a problem 2020-09-17 10:22:25 +03:00
4d7d4671c4 Update Dockerfile to work with v1.2.0 (#803)
* Update Dockerfile to work with v1.1.0

* Update Dockerfile

Co-authored-by: Leonid Bugaev <leonsbox@gmail.com>
2020-09-17 09:56:05 +03:00
Tomer FrouminandGitHub 8c668d252c Added flag to skip hostname verification (#814)
Allow to configure the output to ignore TLS certificates that doesn't match the hostname.
This is necessary when connecting using an IP address instead of a domain or using a self signed certificate.
2020-09-17 09:54:22 +03:00
Dima Golomozy 887293230b increase latency delta 2020-09-16 19:31:58 +03:00
Tomer FrouminandGitHub 2b73ea1f0c Make makefile release commands reentrant and the move the binary name… (#815)
Improved the Makefile:
* Fixed release-* commands fail when running the second time
* Added command to create only the binary executable on mac
* Moved binary name to a variable
* Fixed hardcoded container name
v1.2.0
2020-09-16 18:34:58 +03:00
Urban IshimweandGitHub 66c333656c added current version and updated travis (#816)
made version to be `1.2.0`  and modified travis-ci to make test results more predictable
2020-09-16 18:34:27 +03:00
Urban IshimweandGitHub f11c65889e create interface for socket (#812)
This is to make socket implementations independent of the host OS.
this is done by creating an interface that define all required behaviors of the socket.
2020-09-11 09:16:21 +03:00
Urban IshimweandGitHub 5de9830f9d Merge pull request #810 from buger/afpacket_linux_only
fix #809 and minor bug in pool
2020-08-30 13:02:52 +02:00
Urban Ishimwe 48150e53ba make syn work with hint 2020-08-30 12:16:24 +02:00
Urban Ishimwe f9b0ae6d55 fix #809 and minor bug in pool 2020-08-30 12:07:00 +02:00
Urban IshimweandGitHub df6c4e8d02 Merge pull request #807 from buger/raw_socket_impls
implements af_packet socket
2020-08-28 08:35:47 +02:00
Urban Ishimwe 571487b135 dispatch on rst flag 2020-08-26 17:48:17 +02:00
Urban Ishimwe 83174c0fd4 fix duplicate packets raw_sockets 2020-08-24 18:36:22 +02:00
Urban Ishimwe dcef7a6eb7 Merge branch 'master' into raw_socket_impls 2020-08-23 22:46:31 +02:00
Urban Ishimwe a03eb92c1e implements af_packet socket 2020-08-23 22:42:18 +02:00
Urban IshimweandGitHub 9a2b763e31 Merge pull request #800 from davidFR/feature/kafka_tls
Add Mutual TLS config for Kafka
2020-08-18 14:21:11 +02:00
Urban IshimweandGitHub 4e1da0dc98 Merge pull request #802 from luweimy/fix-libpcap-spelling
Fix `EngineType` libpcap spelling
2020-08-18 14:17:34 +02:00
welu c0ee094317 Fix libpcap spelling 2020-08-18 19:58:28 +08:00
David FradinandGitHub a2bbbb18ef Merge branch 'master' into feature/kafka_tls 2020-08-12 14:44:51 +02:00
Urban IshimweandGitHub fdc8b094f0 Benchmarking, packaging, and fix issues, tests and perfomance (#797)
### performance
- handling of the very big packet(any size that can be buffered)
- speeding up TCP sessions by using message hints: Added **proto.HasFullPayload** that helps to validate the entire HTTP request, it supports `Chunked` encoding too! Added **proto.HasRequestTitle** and **proto.HasResponseTitle** for validating the beginning of HTTP request. Those methods are used `input_raw.go` with `TCP`.
- supports Keep-Alive: the above functions helps to support keep-alive

### Packaging
- **capture:** engines(capture/doc.go)
- **tcp:** tcp message parser (tcp/doc.go)

### benchmarking
- **capture.BenchmarkPcapDump:** the benchmarks regarding dumping packets in a pcap file
- **capture.BenchmarkPcapFile:** the benchmarks of reading packets from a pcap file
- **capture.BenchmarkPcap:** the benchmarks of parsing packets from the loopback interface with pcap handles
- **proto.BenchmarkHasFullPayload:**: benchmarking this function which validates the HTTP payload
- **tcp.BenchmarkPacketParseAndSort:** benchmarks of parsing and sorting packets
- **tcp.BenchmarkMessageParserWithoutHint:** benchmarks of message reasembling by using `SYN` and `FIN` flag
- **tcp.BenchmarkMessageParserWithHint:** benchmarks of message reasembling by using `proto.HasRequestTitle` and `proto.HasFullPayload` flag

### issues
see linked issues

###  tests
- fixed input raw and engine tests

**Most of the changed of the files, was about using functionalities of** `tcp` **and** `capture` **in existing functionalities**
2020-08-11 12:44:53 +03:00
David FRADIN f97f2ba5f3 Add Mutual TLS config for Kafka input and ouput 2020-08-06 13:26:36 +02:00
Arijit DasandGitHub 2f81fba370 Merge pull request #792 from buger/flag-mapping
Map Appsetting to flag names.
2020-07-25 00:52:04 +05:30
arijitad 8e7cf658b2 Merge remote-tracking branch 'origin/master' into flag-mapping 2020-07-23 10:06:23 +05:30
arijitad 01b6f1129c Restructre AppSetting and address comments. 2020-07-23 10:00:16 +05:30
Urban IshimweandGitHub 3da60ed4b6 Merge pull request #796 from sonmezonur/prettify_http_usage_fix
Fix typo in usage text of --prettify-http flag
2020-07-17 17:31:05 +02:00
Onur Sonmez 8593af3e8a Fix typo in usage text of --prettify-http flag 2020-07-17 17:45:06 +03:00
Urban IshimweandGitHub 99f44a6718 Merge pull request #791 from suiwenfeng/master
fix wrong param value updated while --http-set-param
2020-07-15 15:26:34 +02:00