Commit Graph
1155 Commits
Author SHA1 Message Date
Urban Ishimwe f7ad4b0745 engine benchmarks 2020-10-15 21:56:15 +02:00
Urban IshimweandGitHub 5d8ca525a4 deprecate output-http non-compatible clients (#833)
for easy readability check changes by their underlying commits!

these benchmarks address the whole operation of the request cycle in goreplay.

_**goos: linux
goarch: amd64**_

**Using Compatible client:**

```
BenchmarkHTTPOutput-4      	   10417	    118969 ns/op	   12172 B/op	      93 allocs/op
BenchmarkHTTPOutputTLS-4   	    9136	    132929 ns/op	   12448 B/op	      97 allocs/op
```

**Using non-compatible client**
```
BenchmarkHTTPOutput-4      	     859	   1175040 ns/op	   15598 B/op	      46 allocs/op
BenchmarkHTTPOutputTLS-4   	     880	   1189643 ns/op	   15544 B/op	      52 allocs/op

```
Binary size reduced: **7%**

from these benchmarks, we may trade allocations with performance and memory!
2020-10-13 08:36:16 +03:00
Urban IshimweandGitHub 3635d66a76 Merge pull request #818 from DimaGolomozy/increase-latency-delta
increase-latency-delta-tests
2020-10-11 21:33:37 +02:00
Urban IshimweandGitHub a4e78cf22c Merge pull request #830 from buger/tcp-output-distribute-workers
Tcp output distribute workers fairly in non-sticky env
2020-10-07 09:38:54 +02:00
Urban IshimweandGitHub 8aedc15027 Merge pull request #827 from tomerf/syscall_error_check
Fixed check of syscall errors
2020-10-05 21:59:52 +02:00
Urban Ishimwe 30d66d31de distribute tasks in workers 2020-09-30 14:38:21 +02:00
Urban Ishimwe b1d7c4134b debug on stderr 2020-09-30 14:01:05 +02:00
Tomer Froumin 66e680a56f Fixed check of syscall errors 2020-09-29 10:50:22 +03:00
Urban IshimweandGitHub a4ed67acd6 Merge pull request #826 from buger/fix-buf-has-http-title
fix #825
2020-09-25 09:21:34 +02:00
Urban Ishimwe ee9ff8163b fix #825 2020-09-24 15:39:20 +02:00
Urban IshimweandGitHub 9519b9a9f5 Reduce allocation and unnecessary layers (#822)
The focus here was to **reduce allocation in TCP parser** but speed may have hopeful improved too!
pool no longer use map's key of **string** it uses **uint64**
**Benchmarks** was revamped to be more clear
if you want to compare these results copy the benchmark in tcp/bench_test.go@reduce-allocation to tcp/bench_test.go@master:

**before(master)**:
```
BenchmarkPacketParseAndSort-4         	 1000000	      1006 ns/op	      64 B/op	       2 allocs/op
BenchmarkMessageParserWithoutHint-4   	     625	   1772309 ns/op	      1000 packets/op	  419096 B/op	   10045 allocs/op
BenchmarkMessageParserWithHint-4      	      74	  14969926 ns/op	      1000 chunks/op	      1002 packets/op	  450992 B/op	   10126 allocs/op
```

**After(this branch)**:
```
BenchmarkPacketParseAndSort-4         	 1267662	       941 ns/op	      64 B/op	       2 allocs/op
BenchmarkMessageParserWithoutHint-4   	    2256	    523474 ns/op	      1000 packets/op	  243530 B/op	    1037 allocs/op
BenchmarkMessageParserWithHint-4      	      80	  13990955 ns/op	      1000 chunks/op	      1002 packets/op	  268609 B/op	    1099 allocs/op

```
2020-09-22 21:14:31 +03:00
Tomer FrouminandGitHub 3401dfbf2d Make the number of TCP connections configurable (#819) 2020-09-22 21:13:52 +03:00
Urban IshimweandGitHub cfa14b0a32 Merge pull request #824 from swills/size_int64
Use int64 for Size
2020-09-21 19:53:29 +02:00
Steve Wills 85065cb41a Use int64 for Size 2020-09-21 13:43:02 -04:00
Urban IshimweandGitHub 7100ac3593 fix add header out of bounds (#820)
this fixes the issue that arises in adding an HTTP header in an invalid HTTP request(request without CRLF)
2020-09-17 15:16:48 +03:00
Dima Golomozy d5184c6cbd faster latency is not a problem 2020-09-17 10:22:25 +03:00
4d7d4671c4 Update Dockerfile to work with v1.2.0 (#803)
* Update Dockerfile to work with v1.1.0

* Update Dockerfile

Co-authored-by: Leonid Bugaev <leonsbox@gmail.com>
2020-09-17 09:56:05 +03:00
Tomer FrouminandGitHub 8c668d252c Added flag to skip hostname verification (#814)
Allow to configure the output to ignore TLS certificates that doesn't match the hostname.
This is necessary when connecting using an IP address instead of a domain or using a self signed certificate.
2020-09-17 09:54:22 +03:00
Dima Golomozy 887293230b increase latency delta 2020-09-16 19:31:58 +03:00
Tomer FrouminandGitHub 2b73ea1f0c Make makefile release commands reentrant and the move the binary name… (#815)
Improved the Makefile:
* Fixed release-* commands fail when running the second time
* Added command to create only the binary executable on mac
* Moved binary name to a variable
* Fixed hardcoded container name
v1.2.0
2020-09-16 18:34:58 +03:00
Urban IshimweandGitHub 66c333656c added current version and updated travis (#816)
made version to be `1.2.0`  and modified travis-ci to make test results more predictable
2020-09-16 18:34:27 +03:00
Urban IshimweandGitHub f11c65889e create interface for socket (#812)
This is to make socket implementations independent of the host OS.
this is done by creating an interface that define all required behaviors of the socket.
2020-09-11 09:16:21 +03:00
Urban IshimweandGitHub 5de9830f9d Merge pull request #810 from buger/afpacket_linux_only
fix #809 and minor bug in pool
2020-08-30 13:02:52 +02:00
Urban Ishimwe 48150e53ba make syn work with hint 2020-08-30 12:16:24 +02:00
Urban Ishimwe f9b0ae6d55 fix #809 and minor bug in pool 2020-08-30 12:07:00 +02:00
Urban IshimweandGitHub df6c4e8d02 Merge pull request #807 from buger/raw_socket_impls
implements af_packet socket
2020-08-28 08:35:47 +02:00
Urban Ishimwe 571487b135 dispatch on rst flag 2020-08-26 17:48:17 +02:00
Urban Ishimwe 83174c0fd4 fix duplicate packets raw_sockets 2020-08-24 18:36:22 +02:00
Urban Ishimwe dcef7a6eb7 Merge branch 'master' into raw_socket_impls 2020-08-23 22:46:31 +02:00
Urban Ishimwe a03eb92c1e implements af_packet socket 2020-08-23 22:42:18 +02:00
Urban IshimweandGitHub 9a2b763e31 Merge pull request #800 from davidFR/feature/kafka_tls
Add Mutual TLS config for Kafka
2020-08-18 14:21:11 +02:00
Urban IshimweandGitHub 4e1da0dc98 Merge pull request #802 from luweimy/fix-libpcap-spelling
Fix `EngineType` libpcap spelling
2020-08-18 14:17:34 +02:00
welu c0ee094317 Fix libpcap spelling 2020-08-18 19:58:28 +08:00
David FradinandGitHub a2bbbb18ef Merge branch 'master' into feature/kafka_tls 2020-08-12 14:44:51 +02:00
Urban IshimweandGitHub fdc8b094f0 Benchmarking, packaging, and fix issues, tests and perfomance (#797)
### performance
- handling of the very big packet(any size that can be buffered)
- speeding up TCP sessions by using message hints: Added **proto.HasFullPayload** that helps to validate the entire HTTP request, it supports `Chunked` encoding too! Added **proto.HasRequestTitle** and **proto.HasResponseTitle** for validating the beginning of HTTP request. Those methods are used `input_raw.go` with `TCP`.
- supports Keep-Alive: the above functions helps to support keep-alive

### Packaging
- **capture:** engines(capture/doc.go)
- **tcp:** tcp message parser (tcp/doc.go)

### benchmarking
- **capture.BenchmarkPcapDump:** the benchmarks regarding dumping packets in a pcap file
- **capture.BenchmarkPcapFile:** the benchmarks of reading packets from a pcap file
- **capture.BenchmarkPcap:** the benchmarks of parsing packets from the loopback interface with pcap handles
- **proto.BenchmarkHasFullPayload:**: benchmarking this function which validates the HTTP payload
- **tcp.BenchmarkPacketParseAndSort:** benchmarks of parsing and sorting packets
- **tcp.BenchmarkMessageParserWithoutHint:** benchmarks of message reasembling by using `SYN` and `FIN` flag
- **tcp.BenchmarkMessageParserWithHint:** benchmarks of message reasembling by using `proto.HasRequestTitle` and `proto.HasFullPayload` flag

### issues
see linked issues

###  tests
- fixed input raw and engine tests

**Most of the changed of the files, was about using functionalities of** `tcp` **and** `capture` **in existing functionalities**
2020-08-11 12:44:53 +03:00
David FRADIN f97f2ba5f3 Add Mutual TLS config for Kafka input and ouput 2020-08-06 13:26:36 +02:00
Arijit DasandGitHub 2f81fba370 Merge pull request #792 from buger/flag-mapping
Map Appsetting to flag names.
2020-07-25 00:52:04 +05:30
arijitad 8e7cf658b2 Merge remote-tracking branch 'origin/master' into flag-mapping 2020-07-23 10:06:23 +05:30
arijitad 01b6f1129c Restructre AppSetting and address comments. 2020-07-23 10:00:16 +05:30
Urban IshimweandGitHub 3da60ed4b6 Merge pull request #796 from sonmezonur/prettify_http_usage_fix
Fix typo in usage text of --prettify-http flag
2020-07-17 17:31:05 +02:00
Onur Sonmez 8593af3e8a Fix typo in usage text of --prettify-http flag 2020-07-17 17:45:06 +03:00
Urban IshimweandGitHub 99f44a6718 Merge pull request #791 from suiwenfeng/master
fix wrong param value updated while --http-set-param
2020-07-15 15:26:34 +02:00
arijitad 80b19f918b Map Appsetting to flag names. 2020-07-15 00:14:38 +05:30
suiwenfeng aaf8c81c91 fix wrong param value updated while --http-set-param 2020-07-14 22:37:04 +08:00
Leonid Bugaev 839b41f91c Fix scripts to build packages, and MacOS pkg, and remove java dependency
Plus updated version.go
2020-07-06 14:51:54 +03:00
Urban IshimweandGitHub 5cbb5ea85f updating modules (#778)
the major purpose of this PR is to update modules to their recent **minor version**.
this is helpful in terms of having update functionalities and fixing some issues caused by non-up-to-date modules.
v1.1.0
2020-06-24 17:19:19 +03:00
Romanenko AlexeyandGitHub ea822d83da Correct size for gz file (#777)
Fix #775 

When we are using .gz for output we are counting incorrect file size. Now it should be ok
2020-06-23 18:11:45 +03:00
gavinsharkandGitHub 3dff07107a bugfix:fix the bug that middleware will return illegal http body when http body is empty (#737)
If the http body is empty. The httpBody function will return HTTP header content from index 3. This is unexpected.
Function caller also cannot tell whether the httpbody is empty or not.
2020-06-22 14:53:49 +03:00
Urban IshimweandGitHub c20b2734dc Merge pull request #771 from buger/pro-merge
Pro merge
2020-06-16 22:53:43 +02:00
Urban Ishimwe 39b1f80457 Merge branch 'master' into pro-merge 2020-06-16 22:43:08 +02:00