Commit Graph
1181 Commits
Author SHA1 Message Date
HarshavardhanaandNitish Tiwari 8e0910ab3e Fix build issues on BSDs in pkg/cpu (#7116)
Also add a cross compile script to test always cross
compilation for some well known platforms and architectures
, we support out of box compilation of these platforms even
if we don't make an official release build.

This script is to avoid regressions in this area when we
add platform dependent code.
2019-01-22 09:27:23 +05:30
HarshavardhanaandGitHub 5353edcc38 Support policy variable replacement (#7085)
This PR supports iam and bucket policies to have
policy variable replacements in resource and
condition key values.

For example
- ${aws:username}
- ${aws:userid}
2019-01-21 10:27:14 +05:30
Harshavardhanaandkannappanr 74c2048ea9 Add proper contexts with timeouts for etcd operations (#7097)
This fixes an issue of perceived hang when incorrect
unreachable URLs are specified in MINIO_ETCD_ENDPOINTS
variable.

Fixes #7096
2019-01-18 09:36:45 -08:00
Harshavardhanaandkannappanr 1a7e6d4768 Handle multiple conditions in policies (#7079)
Fixes #7078
2019-01-17 10:28:24 -08:00
Bala FAandNitish Tiwari e23a42305c Rebase minio/parquet-go and fix null handling. (#7067) 2019-01-16 21:52:04 +05:30
HarshavardhanaandNitish Tiwari e8791ae274 Remove Minio server arch, version from Server: header (#7074) 2019-01-15 13:16:11 +05:30
Scott Dunlopandkannappanr 309975d477 Add missing time import to counter_darwin.go (#7081) 2019-01-14 17:21:27 -08:00
Praveen raj ManiandHarshavardhana 6571641735 Persist offline mqtt events in the queueDir and replay (#7037) 2019-01-14 12:39:00 +05:30
Sidhartha Maniandkannappanr f3f47d8cd3 Add ServerCPULoadInfo() and ServerMemUsageInfo() admin API (#7038) 2019-01-09 19:04:19 -08:00
Bala FAandkannappanr b0deea27df Refactor s3select to support parquet. (#7023)
Also handle pretty formatted JSON documents.
2019-01-08 16:53:04 -08:00
Harshavardhanaandkannappanr 2d19011a1d Add support for AssumeRoleWithWebIdentity (#6985) 2019-01-04 13:48:12 -08:00
Nitish Tiwariandkannappanr fcb56d864c Add ServerDrivesPerfInfo() admin API (#6969)
This is part of implementation for mc admin health command. The
ServerDrivesPerfInfo() admin API returns read and write speed
information for all the drives (local and remote) in a given Minio
server deployment.

Part of minio/mc#2606
2018-12-31 09:46:44 -08:00
HarshavardhanaandGitHub 2a0e4b6f58 Add boolean function condition support (#7027) 2018-12-28 12:18:58 -08:00
HarshavardhanaandGitHub 4e4f855b30 Add support for new policy conditions (#7024)
This PR implements following condition types

- StringEqualsIgnoreCase and StringNotEqualsIgnoreCase
- BinaryEquals
2018-12-26 17:39:30 -08:00
HarshavardhanaandGitHub 2db22deb93 Fix policy bugs Null conditions and canonical names (#7021)
This PR fixes two different issues

- Null condition implementation
- HTTP Canonical request value names

This PR fixes handling of null conditions and
handle HTTP canonical names in request values.

This PR was tested with policies mentioned in the following blog
https://aws.amazon.com/blogs/security/how-to-prevent-uploads-of-unencrypted-objects-to-amazon-s3/

Fixes #6955
2018-12-26 02:03:28 -08:00
Aditya ManthramurthyandNitish Tiwari 2aeb3fbe86 Fix csv output delimiter bug (#6994) 2018-12-19 11:49:06 +05:30
HarshavardhanaandNitish Tiwari bebaff269c Support IPv6 in minio command line (#6947)
Fixes #6946
2018-12-14 13:07:46 +05:30
Harshavardhanaandkannappanr 3498f5b0ec List exact DNS entries for a requested bucketName (#6936)
Currently we would end up considering common prefix
buckets to be part of the same DNS service record,
which leads to Minio server wrongly forwarding the
records to incorrect IPs.
2018-12-12 10:47:03 -08:00
Harshavardhanaandkannappanr 4c7c571875 Support JSON to CSV and CSV to JSON output format conversion (#6910)
This PR implements one of the pending items in issue #6286
in S3 API a user can request CSV output for a JSON document
and a JSON output for a CSV document. This PR refactors
the code a little bit to bring this feature.
2018-12-07 14:55:32 -08:00
Praveen raj ManiandNitish Tiwari e7af31c2ff Removed clientID from NATS-Streaming Config (#6391)
clientID must be a unique `UUID` for each connections. Now, the
server generates it, rather considering the config.

Removing it as it is non-beneficial right now.

Fixes #6364
2018-11-30 10:46:17 +05:30
HarshavardhanaandNitish Tiwari d4265f9a13 Simplify OPA to use rootCAs custom transport (#6843)
Also close the connections properly to use the
connection pooling properly for HTTP clients.
2018-11-22 08:31:05 +05:30
HarshavardhanaandNitish Tiwari eddf468aef Lock the targetList properly in go-routines (#6838)
Fixes #6483
2018-11-21 21:25:54 +05:30
Ashish Kumar SinhaandNitish Tiwari b0d04b9a81 Retry Connection for RabbitMQ (#6837)
Add retries to connect to RabbitMQ 5 times 
with 2s interval

Fixes #6807
2018-11-21 08:37:29 +05:30
Harshavardhanaandkannappanr 272b8003d6 Honor header only when requested for use (#6815) 2018-11-16 10:27:48 -08:00
Harshavardhanaandkannappanr 7e1661f4fa Performance improvements to SELECT API on certain query operations (#6752)
This improves the performance of certain queries dramatically,
such as 'count(*)' etc.

Without this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m42.464s
user	0m0.071s
sys	0m0.010s
```

With this PR
```
~ time mc select --query "select count(*) from S3Object" myminio/sjm-airlines/star2000.csv.gz
2173762

real	0m17.603s
user	0m0.093s
sys	0m0.008s
```

Almost a 250% improvement in performance. This PR avoids a lot of type
conversions and instead relies on raw sequences of data and interprets
them lazily.

```
benchcmp old new
benchmark                        old ns/op       new ns/op       delta
BenchmarkSQLAggregate_100K-4     551213          259782          -52.87%
BenchmarkSQLAggregate_1M-4       6981901985      2432413729      -65.16%
BenchmarkSQLAggregate_2M-4       13511978488     4536903552      -66.42%
BenchmarkSQLAggregate_10M-4      68427084908     23266283336     -66.00%

benchmark                        old allocs     new allocs     delta
BenchmarkSQLAggregate_100K-4     2366           485            -79.50%
BenchmarkSQLAggregate_1M-4       47455492       21462860       -54.77%
BenchmarkSQLAggregate_2M-4       95163637       43110771       -54.70%
BenchmarkSQLAggregate_10M-4      476959550      216906510      -54.52%

benchmark                        old bytes       new bytes      delta
BenchmarkSQLAggregate_100K-4     1233079         1086024        -11.93%
BenchmarkSQLAggregate_1M-4       2607984120      557038536      -78.64%
BenchmarkSQLAggregate_2M-4       5254103616      1128149168     -78.53%
BenchmarkSQLAggregate_10M-4      26443524872     5722715992     -78.36%
```
2018-11-14 15:55:10 -08:00
Pontus Leitzlerandkannappanr f9779b24ad Enable default vet flags (#6810)
Enable default vet flags except experimental
2018-11-14 10:23:44 -08:00
Chester Liandkannappanr aa2d8583ad Check key length before adding a new user. (#6790)
User's key should satisfy the requirement of `mc config host add`.
Check access key and secret key length before adding a new user,
avoid creating a useless user which cannot be added into config
host or log into the browser.
2018-11-09 15:48:24 -08:00
Harshavardhanaandkannappanr b24b320807 Set notification namespace for NotificationConfiguration (#6789) 2018-11-09 10:40:03 -08:00
Harshavardhanaandkannappanr a40610d331 Re-populate public key if JWT fails to parse (#6786)
This is done such that if WSO2 was re-configured
with new TLS certs, and newer tokens are signed
with a newer public key. Once populated parse the JWT
again
2018-11-08 17:01:20 -08:00
Harshavardhanaandkannappanr d58fc68137 Fix shadowing issue in elasticsearch target (#6774) 2018-11-07 12:09:03 -08:00
Matthias Schneiderandkannappanr 71c66464c1 feature: added nsq as broker for events (#6740) 2018-11-07 10:23:13 -08:00
Ashish Kumar Sinhaandkannappanr 572719872d Event Notification for ElasticSearch (#6764)
Using access format for Event Notification for Elastic Search
2018-11-06 11:38:54 -08:00
Harshavardhanaandkannappanr a9cda850ca Add forceStop flag to provide facility to stop healing (#6718)
This PR also makes sure that we deal with HTTP request
count by ignoring the on-going heal operation, i.e
do not wait on itself.
2018-11-04 19:24:16 -08:00
Aarushi Aryaandkannappanr 89b14639a9 avoid using URL encoding to generate keys (#6731) 2018-10-31 15:07:20 -07:00
Harshavardhanaandkannappanr 3f744c0361 Fix mimedb update files (#6744) 2018-10-31 14:15:27 -07:00
HarshavardhanaandNitish Tiwari f162d7bd97 Performance improvements by re-using record buffer (#6622)
Avoid unnecessary pointer reference allocations
when not needed, for example

- *SelectFuncs{}
- *Row{}
2018-10-31 08:48:01 +05:30
HarshavardhanaandNitish Tiwari bab4c90c45 Fix broken links in docs (#6700) 2018-10-25 11:39:31 +05:30
Harshavardhanaandkannappanr fde8c38638 Add default canned policies (#6690) 2018-10-24 17:14:27 -07:00
Anis Elleuchandkannappanr e29009d347 Register postgre driver in pkg/event/target (#6689)
Commit 5c13765168 removed postgre registration triggerd
by the automatic gofmt command but it was the only where pg is registered. This commit
fixes behavior and adds unit tests to check whether postgre & sql are registered or not.
2018-10-23 11:44:46 -07:00
Pontus Leitzlerandkannappanr 9631d65552 Fix goroutine test fatalf (#6682)
Use t.Error/t.ErrorF instead if t.Fatal/t.Fatalf

Add returns to achieve same behaviour as earlier
2018-10-23 09:44:20 -07:00
HarshavardhanaandNitish Tiwari b99aaab42e Sid value can be any unicode character support it (#6676)
Fixes #6476
2018-10-23 16:11:06 +05:30
Ashish Kumar Sinhaandkannappanr c0b4bf0a3e SQL select query for CSV/JSON (#6648)
select * , select column names have been implemented for CSV.
select * is implemented for JSON.
2018-10-22 12:12:22 -07:00
Praveen raj ManiandDee Koder cef044178c Treat columns with spaces inbetween [s3Select] (#6597)
replace the double/single quotes with backticks for the xwb1989/sqlparser
to recognise such queries.

Fixes #6589
2018-10-17 11:01:26 -07:00
HarshavardhanaandDee Koder b251454dd6 Fix toggling users status (#6640) 2018-10-16 14:55:23 -07:00
HarshavardhanaandGitHub 1e7e5e297c Add canned policy support (#6637)
This PR adds an additional API where we can create
a new set of canned policies which can be used with one
or many users.
2018-10-16 12:48:19 -07:00
HarshavardhanaandNitish Tiwari 3ef3fefd54 Add ListUsers API to list all configured users in IAM (#6619) 2018-10-13 12:48:43 +05:30
Aditya Manthramurthyandkannappanr e3eec89d24 Optimize string processing in select (#6593)
Reduce allocations during string concatenation and simplify some
processing code.
2018-10-09 14:02:19 -07:00
Harshavardhanaandkannappanr 54ae364def Introduce STS client grants API and OPA policy integration (#6168)
This PR introduces two new features

- AWS STS compatible STS API named AssumeRoleWithClientGrants

```
POST /?Action=AssumeRoleWithClientGrants&Token=<jwt>
```

This API endpoint returns temporary access credentials, access
tokens signature types supported by this API

  - RSA keys
  - ECDSA keys

Fetches the required public key from the JWKS endpoints, provides
them as rsa or ecdsa public keys.

- External policy engine support, in this case OPA policy engine

- Credentials are stored on disks
2018-10-09 14:00:01 -07:00
Aditya Manthramurthyandkannappanr 16a100b597 Fix out-of-bound array access crash in select processing (#6594)
Fix test case.
2018-10-09 09:45:56 -07:00
Ashish Kumar Sinhaandkannappanr 670f9788e3 Count(*) to give integer value (#6564)
The Max, Min functions were giving float value even when they were integers.  
Resolved max and Min to return integers in that scenario.

Fixes #6472
2018-10-04 17:33:53 -07:00