Commit Graph
1433 Commits
Author SHA1 Message Date
ebozdumanandHarshavardhana 882a1a1ccc Stops listing objects for write-only access (#6396) 2018-08-31 13:20:27 -07:00
Harshavardhanaandkannappanr 8690d62146 Allow fallback listen if first listener fails (#6380)
On linux listen() uses kernel features TCP_FASTOPEN, DEFER_ACCEPT

Fixes #6379
2018-08-31 13:17:05 -07:00
Anis Elleuchandkannappanr 85117d554f xl: Avoid removing a directory if it is not an object dir (#6395)
DeleteObject should not remove any directory unless the latter
is an empty directory.

Fixes #6394
2018-08-31 13:16:35 -07:00
Harshavardhanaandkannappanr 4487f70f08 Revert all GetObjectNInfo related PRs (#6398)
* Revert "Encrypted reader wrapped in NewGetObjectReader should be closed (#6383)"

This reverts commit 53a0bbeb5b.

* Revert "Change SelectAPI to use new GetObjectNInfo API (#6373)"

This reverts commit 5b05df215a.

* Revert "Implement GetObjectNInfo object layer call (#6290)"

This reverts commit e6d740ce09.
2018-08-31 13:10:12 -07:00
HarshavardhanaandNitish Tiwari fb27388101 HTTP headers are case insensitive handle them appropriately (#6390)
An issue was reproduced when minio-js client functional
tests are setting lower case http headers, in our current
master branch we specifically look for canonical host header
which may be not necessarily true for all http clients.
This leads to a perpetual hang on the *net.Conn*.

This PR fixes regression caused by #6206 by handling the
case insensitivity.
2018-08-31 17:00:32 +05:30
Bala FAandNitish Tiwari 72fa2b4537 Add RPC counters for HTTP stats. (#6206)
This patch introduces separate counters for HTTP stats for minio
reserved bucket.

Fixes #6158
2018-08-30 14:17:58 +05:30
HarshavardhanaandGitHub 53a0bbeb5b Encrypted reader wrapped in NewGetObjectReader should be closed (#6383) 2018-08-29 19:18:00 -07:00
HarshavardhanaandGitHub 384a862940 Return quorum error based on disks in abortMultipartUpload (#6362)
Fixes #4980
2018-08-29 13:36:19 -07:00
HarshavardhanaandGitHub 029f52880b With no read quorum config should be treated as notFound (#6374)
This will allow the config subsystem to initialize properly
in situations where many servers are coming up in a rolling
fashion.
2018-08-28 14:23:22 -07:00
HarshavardhanaandGitHub 5b05df215a Change SelectAPI to use new GetObjectNInfo API (#6373)
This PR also removes some double checks
2018-08-28 13:08:30 -07:00
Aditya ManthramurthyandNitish Tiwari e6d740ce09 Implement GetObjectNInfo object layer call (#6290)
This combines calling GetObjectInfo and GetObject while returning a
io.ReadCloser for the object's body. This allows the two operations to
be under a single lock, fixing a race between getting object info and
reading the object body.
2018-08-27 15:28:23 +05:30
Anis Elleuchandkannappanr 06d2dfa31c Fix WORM and BROWSER status calculation (#6360)
One typo introduced in a recent commit miscalculates if worm and browser
are enabled or not. A simple test is also added to detect this issue
in the future if it ever happens again.
2018-08-24 14:36:14 -07:00
poornasandkannappanr d547873b17 webhandler - display encryption errors properly (#6339)
For encrypted objects, download errors need to be
displayed in web response format instead of xml format.

Fixes #6327
2018-08-24 07:56:24 -07:00
Krishna Srinivasandkannappanr 52f6d5aafc Rename of structs and methods (#6230)
Rename of ErasureStorage to Erasure (and rename of related variables and methods)
2018-08-23 23:35:37 -07:00
Harshavardhanaandkannappanr 2211a5f1b8 Avoid ListenBucket targets to be listed in ServerInfo (#6340)
In current master when you do `mc watch` you can see a
dynamic ARN being listed which exposes the remote IP as well

```
mc watch play/airlines
```

On another terminal
```
mc admin info play
●  play.minio.io:9000
   Uptime : online since 11 hours ago
  Version : 2018-08-22T07:50:45Z
   Region :
 SQS ARNs : arn:minio:sqs::httpclient+51c39c3f-131d-42d9-b212-c5eb1450b9ee+73.222.245.195:33408
    Stats : Incoming 30GiB, Outgoing 7.6GiB
  Storage : Used 7.7GiB
```

SQS ARNs listed as part of ServerInfo should be only external targets,
since listing an ARN here is not useful and it cannot be re-purposed in
any manner.

This PR fixes this issue by filtering out httpclient from the ARN list.

This is a regression introduced in #5294 0e4431725c
2018-08-23 23:31:14 -07:00
HarshavardhanaandNitish Tiwari 1ffa6adcd4 Ignore io.EOF returned by ReadFrom for zero byte fs.json (#6346)
Fixes #6256
2018-08-24 11:34:21 +05:30
kannappanrandGitHub add57a6938 Add content-length as part of event notification structure (#6341)
Fixes #6321
2018-08-23 14:40:54 -07:00
Praveen raj ManiandNitish Tiwari 65e05a06fb Remove notifications Fix (#6082)
Remove all the notifications for an empty rulesMap

Fixes #6053
2018-08-23 22:53:18 +05:30
HarshavardhanaandGitHub b01e69e08f Initialize global object layer after all subsystems have initialized (#6333)
This is to ensure that object API operations are not performed
on a server on which subsystems are yet to be initialized.
2018-08-22 23:11:17 -07:00
HarshavardhanaandGitHub beb6d40ce6 Avoid crash when policy subsystem is not initialized (#6326)
Fixes #6324
2018-08-21 15:38:51 -07:00
poornasandkannappanr 19db921555 CopyObject: fix regression in key rotation (#6331)
After key rotation, metadata was not being replaced with new sealed key.
Regression introduced in commit e71ef905f9
2018-08-21 15:12:00 -07:00
kannappanrandDee Koder 2d84b02bc4 Check for absence of checksum field and attributes. (#6298)
Fixes #6295
2018-08-20 16:58:47 -07:00
Harshavardhanaandkannappanr 7d7e21aebb Merge initConfig logic to ConfigSys (#6312) 2018-08-19 13:57:18 -07:00
Andreas AuernhammerandHarshavardhana d531080b7e add SSE-KMS not-implemented error handling (#6234)
This commit adds error handling for SSE-KMS requests to
HEAD, GET, PUT and COPY operations. The server responds
with `not implemented` if a client sends a SSE-KMS
request.
2018-08-17 21:07:19 -07:00
Harshavardhanaandkannappanr a6b8a5487a Fail gateway properly with an error on port conflicts (#6303) 2018-08-17 21:06:36 -07:00
Harshavardhanaandkannappanr 9f14433cbd Ensure that setConfig uses latest functionality (#6302) 2018-08-17 18:51:34 -07:00
Harshavardhanaandkannappanr 50a817e3d3 Use new listener which implements enhanced tcp features (#6289)
This package provide customizable TCP net.Listener with various
performance-related options:

 * SO_REUSEPORT. This option allows linear scaling server performance
   on multi-CPU servers.
   See https://www.nginx.com/blog/socket-sharding-nginx-release-1-9-1/ for details.
 * TCP_DEFER_ACCEPT. This option expects the server reads from the accepted
   connection before writing to them.
 * TCP_FASTOPEN. See https://lwn.net/Articles/508865/ for details.
2018-08-17 18:44:02 -07:00
Harshavardhanaandkannappanr 5a4a57700b Add select docs and fix return values for Select API (#6300) 2018-08-17 17:11:39 -07:00
Harshavardhanaandkannappanr 3de5a3157f Enhance picking valid xlMeta based on quorum (#6297)
This PR borrows the idea from getFormatXLQuorum()
2018-08-17 14:42:04 -07:00
poornasandkannappanr e71ef905f9 Add support for SSE-S3 server side encryption with vault (#6192)
Add support for sse-s3 encryption with vault as KMS.

Also refactoring code to make use of headers and functions defined in
crypto package and clean up duplicated code.
2018-08-17 12:52:14 -07:00
Harshavardhanaandkannappanr 65de2d68c0 Allow for proper garbage collection pooling bytes.Buffer (#6266) 2018-08-16 18:37:43 -07:00
HarshavardhanaandDee Koder eab947cf42 Make sure to update modTime in erasure metadata (#6296)
This is to ensure that when we update xl.json with new
parts have the latest modtime, in-turn avoids consistency
issues when the disk is offline.
2018-08-16 17:55:01 -07:00
Harshavardhanaandkannappanr 0fe9e95250 Validate prefixes on all sets (#6294)
This PR fixes a regression introduced in 8eb838bf91
where hashing technique was used on prefixes to get the right set
to perform the operation, this is not correct since prefixes and
their corresponding keys might hash to a different value depending
on the key length.

For prefixes/directories we should look everywhere to support proper
quorum based listing.

Fixes #6293
2018-08-16 16:49:38 -07:00
kannappanrandGitHub c7946ab9ab Remove unnecessary error log messages (#6186) 2018-08-16 12:57:49 -07:00
Harshavardhanaandkannappanr f26325c988 Support supplying custom drives per set count (#6261) 2018-08-15 16:35:21 -07:00
Arjun Mishraandkannappanr 7c14cdb60e S3 Select API Support for CSV (#6127)
Add support for trivial where clause cases
2018-08-15 03:30:19 -07:00
HarshavardhanaandNitish Tiwari 0e02328c98 Migrate config.json from config-dir to backend (#6195)
This PR is the first set of changes to move the config
to the backend, the changes use the existing `config.json`
allows it to be migrated such that we can save it in on
backend disks.

In future releases, we will slowly migrate out of the
current architecture.

Fixes #6182
2018-08-15 10:11:47 +05:30
HarshavardhanaandGitHub 380524ae27 Unlock read lock on uploadID upon errors (#6283) 2018-08-14 18:35:30 -07:00
kannappanrandHarshavardhana 0286e61aee Log disk not found error just once (#6059)
Modified the LogIf function to log only if the error passed
is not on the ignored errors list.

Currently, only disk not found error is added to the list.
Added a new function in logger package called LogAlwaysIf, 
which will print on any error.

Fixes #5997
2018-08-14 13:58:48 -07:00
wd256andkannappanr ff29aed05d gcs: Translate S3 user-defined metadata prefix to/from GCS custom metadata prefix (#6270) 2018-08-14 11:53:39 -07:00
Harshavardhanaandkannappanr 64f2c61813 Implement memory efficient readdir for windows (#6247)
Fixes #6164
2018-08-09 14:52:29 -07:00
Andreas Auernhammerandkannappanr 525c04fd07 crypto: add SSE-KMS HTTP header detection (#6228)
This commit adds support for detecting SSE-KMS headers.
The server should be able to detect SSE-KMS headers to
at least fail such S3 requests with not implemented.
2018-08-09 13:02:57 -07:00
Anis ElleuchandHarshavardhana 5a1ae862a7 Avoid sending an error after 206 HTTP code (#6264)
When a S3 client sends a GET Object with a range header, 206 http
code is returned indicating success, however the call of the object
layer's GetObject() inside the handler can return an error and will lead
to writing an XML error message, which is obviously wrong since
we already sent 206 http code. So in the case, we just stop sending
data to the S3 client, this latter can still detect if there is no
error when comparing received data with Content-Length header
in the Get Object response.
2018-08-08 15:39:47 -07:00
HarshavardhanaandNitish Tiwari 2f1756489e Add tracing capabilities for internode rpc Servers (#6254) 2018-08-07 15:21:30 +05:30
Harshavardhanaandkannappanr 9719640e34 Use sha256-simd instead of crypto/sha256 (#6252) 2018-08-06 18:00:10 -07:00
Krishna Srinivasandkannappanr ce02ab613d Simplify erasure code by separating bitrot from erasure code (#5959) 2018-08-06 15:14:08 -07:00
Oleg Kovalovandkannappanr 37de2dbd3b simplifying if-else chains to switches (#6208) 2018-08-06 10:26:40 -07:00
HarshavardhanaandNitish Tiwari a82500f162 Support dumb terminals by turning off color (#6246)
ANSI colors do not work on dumb terminals, in situations
when minio is running as a service under systemd.

This PR ensures we turn off color in those situations.
2018-08-06 18:16:49 +05:30
HarshavardhanaandNitish Tiwari 2dede2fdc2 Add reliable RemoveAll to handle racy situations (#6227) 2018-08-06 09:45:28 +05:30
HarshavardhanaandNitish Tiwari 13fbb96736 Hold locks granularly in nslockMap (#6242)
With benchmarks increases the performance for small files
by almost 4x times the previous releases.
2018-08-06 08:55:25 +05:30