127 Commits
Author SHA1 Message Date
Dima GolomozyandGitHub 40946831b9 goreplay-cli package (#1148)
change package from `main -> goreplay`
this will allow importing `goreplay` as a package
2023-01-12 18:24:45 +03:00
Dima GolomozyandGitHub 23825d82f5 release-tars (#1149)
- removed deb/rpm/pkg packages
- archiving with `.tar.gz` and `.zip`
- change archive naming
2023-01-10 09:09:13 +03:00
Dima GolomozyandGitHub f53d6c4466 go-bump-1.19 (#1083) 2022-11-09 22:20:32 +03:00
Leonid BugaevandGitHub d440b3dc8f K8s native support (#1052)
Now you are be able to capture traffic inside k8s like this:

```
gor --input-raw k8s://namespace/deployment/app:80 --output-http http://replay.com
```

Supported format for filtering required pods:

```
k8s://[namespace/]pod/[pod_name] - k8s://default/pod/nginx-7848d4b86f-5nxz8
k8s://[namespace/]deployment/[deployment_name] - k8s://default/deployment/nginx
k8s://[namespace/]daemonset/[daemonset_name] - k8s://default/daemonset/nginx
k8s://[namespace/]labelSelector/[selector] - k8s://default/labelSelector/app=nginx
k8s://[namespace/]fieldSelector/[selector] - k8s://default/fieldSelector/metadata.name=nginx-7848d4b86f-5nxz8
```

`namespace` is optional, omit to use all namespaces: `k8s://labelSelector/app=replay`

GoReplay designed to be running running as a daemonset (e.g. on each physical k8s node). 

See the full guide in here: https://github.com/buger/goreplay/blob/ca8205a5c5d2a1facb00214c78e4120aae6d772d/k8s/README.md
2022-02-12 21:01:03 +03:00
Leonid Bugaev 9c5f011bb7 Fix builds on m1 mac
Ensure that docker images gets build in x64 arch, and use nfpm instead of fpm
2022-01-12 13:00:23 +03:00
Dima GolomozyandGitHub 1bd550ac03 Makefile fixes2 (#1013)
* add phony
fix target x86

* delete container

* remove dup

* try test
2021-12-21 12:43:01 +03:00
Dima GolomozyandGitHub 4a1c08859b add more targets (#1003)
created dep. between targets
using --rm flag on docker
2021-09-06 15:24:31 +03:00
Leonid Bugaev 59b12abbe1 Fix linux compilation issue 2021-07-07 19:33:49 +03:00
Leonid Bugaev f701dace50 Fix MacOS builds 2021-06-29 23:10:39 +03:00
Leonid BugaevandGitHub 58ff8865df Fix windows packet capture (#943)
Issues is that Go built-in net.Interfaces function in newer Windows versions return wrong interface names, which libpcap can't consume.
Now we use pcap.FindDevices instead of net.Interfaces.
See this Article for deep understanding of the issue https://haydz.github.io/2020/07/06/Go-Windows-NIC.html

Additionally, found a bug causing big memory allocations, for large requests, when we perform check if messages finished or not.
Because of this bug chunked body encoding check was not working properly.
Was not caught in tests, because test was working on packet array level, and this issue happens when dealing with TCP message object.

Additionally, added a small fix for windows Makefile task, it now generates proper file name.
2021-06-10 14:46:17 +03:00
Leonid BugaevandGitHub 07a4ef626c Add windows support (#940)
For a long time there were no official binaries for the windows platform.
One of the reasons is the complexities of the build toolchain. Not only CGO is required, but also installing the needed libraries and header files, not talking about mingw and etc.

2 weeks ago [golang-crossbuild](https://github.com/elastic/golang-crossbuild) project added native support for Libpcap based applications.
Windows support is based on the WinPcap which is a bit (a lot) outdated, BUT, since we depend only on its interface, it is still possible to use projects like npcap https://nmap.org/npcap/#download.

Npcap needs be installed with WinPcap compatibe mode (checkbox during installation)
It is also possibe install it in silent mode like this: `npcap-0.86.exe /S /winpcap_mode=yes`

After PR merge, will be updated related documentation.

Additionally fix dependency on "unix" package (apparently it can be totally replaced using universal syscall package)
2021-06-09 09:46:41 +03:00
Urban IshimweandGitHub e74e945e7f Fixing performance issues and out-of-order packets (#916)
### Reducing CPU context switching and number of goroutines. 
Packet capture and packet processing now use only two goroutines which helps to minimize CPU context switches. Spawning too many goroutines is harmful here. 

### Optimized packet capture - allocated memory only when required, and only for data which is used
Using ZeroCopy methods from libpcap library to avoid unnecessary allocations. Now memory gets allocated ONLY for the valid packets, and only for the packets which have the data. E.g. no SYN/FIN packets are used now. Additionally we now use `sync.Pool` for re-using packet objects, which helps to re-use already allocated memory. 

### Simplification and optimization of request/response detection
There is no SYN/FIN packets anymore etc. Now only packet payload is used to detect start and end of the packet. More over payload detection now does not require generating a total “message” buffer, and works with individual packet payloads. 

Message payloads now concatenated from packets only in the end when message is dispatched. Also, before checking if message is complete, added additional check if all received packets in the valid order, e.g. if their SEQ is valid, and no packets are missing. 

Reworked chunked encoding validation, and now it does not need expensive operation of re-calculating all the chunks. Now it “trust” that client gives valid chunk body, check if packets are in the right order (e.g. SEQ match), and checks if message ends with the right suffix. All is done with 0 allocations. 

Parsing all Headers using `proto.GetHeaders` was proved to be very slow. Now we only parse the headers we need(and do it only once).

Packets gets matched together using ACK, which on high RPS removed chances of duplicating IDs. Additionally, even if packets are received out of order, now it will properly sort them, before dispatching the message.

### Changes in ID generation algorithm
Message ID generation and relations between request and response IDs is fully rewritten. Responses now do not have to lookup for request data in order to get the same ID. ID no rely on the fact that SEQ of the first packet of the response should be the same as ACK of the request. If previously Message ID contained random values, like current timestamp, now it has a consistent algorithm which is based on TCP stream id (SrcPort + DstPort + SrcIP/DstIP) and current ACK/SEQ number (to distinguish multiple messages within the same stream).

### BPF filter optimizations
When tracking response it now uses a more accurate BPF rule to filter only needed traffic. 

### Misc
The packet code is now fully moved to tcp/Packet, so packet processing done only once in one place.

TCP output now has a 5 second timeout, and has a proper Close method.

Fully switching to go modules and removing vendoring.
2021-05-19 20:11:25 +03:00
Leonid Bugaev 40a3c51c9c Fix Makefile 2020-11-22 20:10:00 +03:00
Tomer FrouminandGitHub 2b73ea1f0c Make makefile release commands reentrant and the move the binary name… (#815)
Improved the Makefile:
* Fixed release-* commands fail when running the second time
* Added command to create only the binary executable on mac
* Moved binary name to a variable
* Fixed hardcoded container name
2020-09-16 18:34:58 +03:00
Leonid Bugaev 839b41f91c Fix scripts to build packages, and MacOS pkg, and remove java dependency
Plus updated version.go
2020-07-06 14:51:54 +03:00
Urban Ishimwe a7c5788052 Merge branch 'master' into pro-merge 2020-06-13 11:39:51 +02:00
Leonid Bugaev 15bd2608f7 Fix Makefile 2020-06-08 14:34:19 +03:00
Leonid Bugaev d67bc13db3 Merging PRO functionality back to OSS repo 2020-06-08 14:29:32 +03:00
Leonid Bugaev ab2ba8400d Improve S3 debugging 2020-06-06 17:07:51 +03:00
Urban Ishimwe 0b437716be make raw socket usable, minor optimazation and fixed tests.
- ListenPacket can now capture request(non-multicast)
2020-06-06 09:54:25 +02:00
Leonid Bugaev df143cdeec Update to latest upstream master 2019-05-18 10:36:46 +03:00
Leonid Bugaev 81e8f98ab6 Merge remote-tracking branch 'upstream/master' 2019-05-13 20:20:06 +03:00
Leonid Bugaev 36ca809e4a Add commands to generate rpm/deb 2019-04-02 10:21:59 +07:00
Leonid Bugaev 6eea02ee86 Add Demo mode 2019-01-24 23:37:53 +01:00
Leonid Bugaev 927215bd9b Merge branch 'master' of https://github.com/buger/gor 2017-09-14 22:38:51 +05:00
Leonid Bugaev e310fc8369 Fix zero-length 100-continue packets
Should fix https://github.com/buger/goreplay/issues/496
2017-09-03 22:11:42 +05:00
Leonid Bugaev 0f8e2f2d52 Merge branch 'master' of https://github.com/buger/gor 2017-08-22 20:43:22 +05:00
Leonid Bugaev b8befff286 Sync makefile 2017-08-22 17:07:37 +05:00
Leonid Bugaev 39ebff56b4 Update to use latest docker.dev 2017-08-22 17:07:37 +05:00
Leonid Bugaev 06ec570286 Merge remote-tracking branch 'upstream/master' 2017-07-20 21:09:04 +03:00
Leonid Bugaev 74225ebb22 Rename all package imports from gor to goreplay 2017-05-01 17:56:50 +02:00
Leonid Bugaev 9173379cc8 Merge branch 'master' of https://github.com/buger/gor 2017-02-05 17:26:44 +03:00
Leonid Bugaev bf1c67c5a1 Add install task 2017-02-05 17:26:04 +03:00
Leonid Bugaev 973b028517 Merge branch 'master' into s3 2017-01-23 13:27:56 +03:00
Leonid Bugaev b48aadc9ea Merge branch 'master' of https://github.com/buger/gor 2017-01-20 22:37:11 +03:00
Leonid Bugaev be49c7e7ac Run GC on each 1000th request 2016-12-23 20:25:09 +03:00
Sergey Mostovoy 5b5c3f33f4 Filter response if request is filtered #388 2016-12-22 11:25:29 +02:00
Leonid Bugaev cd51e09fcd Merge branch 'master' into s3 2016-12-19 20:25:06 +03:00
Leonid Bugaev ddf2dc2876 fixes 2016-12-19 20:16:11 +03:00
Leonid Bugaev 75b31fa701 Fix make file 2016-08-10 15:01:06 +03:00
Leonid Bugaev a12357b497 Fix Makefile 2016-08-10 14:08:59 +03:00
Leonid Bugaev dac3b171a3 Fix test 2016-08-10 13:35:19 +03:00
Leonid Bugaev 4268dcf7d1 Merge branch 'master' of https://github.com/buger/gor 2016-08-08 18:28:32 +03:00
Leonid Bugaev 25b998fa3f Fix run command 2016-08-01 19:23:14 +03:00
Leonid Bugaev 07fa6d9a7c Force Go DNS resolver 2016-08-01 18:05:52 +03:00
Leonid Bugaev 2e768f776d Fix versioning 2016-07-13 20:45:44 +03:00
Leonid Bugaev a6d263df1c Add support for binary input/output 2016-07-13 16:01:25 +03:00
Leonid Bugaev 55fe5b9c11 s3-output 2016-07-08 13:20:33 +03:00
Leonid Bugaev f17f51390a Fix 100-continue with multi-packet headers
Also adds —output-null for testing purpose
2016-07-01 20:01:25 +03:00
Leonid Bugaev 5a0c03ec33 Fix BodySize for multi-packet headers 2016-06-30 14:13:27 +03:00