Commit Graph
3125 Commits
Author SHA1 Message Date
Anis ElleuchandGitHub d90044b847 federation: Redirect Lifecycle PUT request by bucket name (#11062)
The bucket forwarder handler considers MakeBucket to be always local but
it mistakenly thinks that PUT bucket lifecycle to be a MakeBucket call.

Fix the check of the MakeBucket call by ensuring that the query is empty
in the PUT url.
2020-12-09 07:25:26 -08:00
HarshavardhanaandGitHub d8c1f93de6 reject mixed drive situations with drives on root disks (#11057)
till now we used to match the inode number of the root
drive and the drive path minio would use, if they match
we knew that its a root disk.

this may not be true in all situations such as running
inside a container environment where the container might
be mounted from a different partition altogether, root
disk detection might fail.
2020-12-09 00:27:02 -08:00
Anis ElleuchandGitHub a51488cbaa s3: Fix reading GET with partNumber specified (#11032)
partNumber was miscalculting the start and end of parts when partNumber
query is specified in the GET request. This commit fixes it and also
fixes the ContentRange header in that case.
2020-12-08 13:12:42 -08:00
Harshavardhana dc819afa44 fix: auto update crawler meta version
PR 038bcd9079 introduced
version '3', we need to make sure that we do not
print an unexpected error instead log a message to
indicate we will auto update the version.
2020-12-08 10:40:51 -08:00
Harshavardhana 4a564336fe Revert "Add metrics for nodes online and offline (#11050)"
This reverts commit f60bbdf86b.
2020-12-08 09:23:35 -08:00
Ritesh H ShuklaandGitHub f60bbdf86b Add metrics for nodes online and offline (#11050) 2020-12-08 01:06:27 -08:00
Poorna KrishnamoorthyandGitHub f3beb1236a Add cache usage, total capacity to prometheus metrics (#11026) 2020-12-07 16:35:11 -08:00
Poorna KrishnamoorthyandGitHub 934bed47fa Add transition event notification (#11047)
This is a MinIO specific extension to allow monitoring of transition events.
2020-12-07 13:53:28 -08:00
Ritesh H ShuklaandGitHub 038bcd9079 Add replication capacity metrics support in crawler (#10786) 2020-12-07 13:47:48 -08:00
HarshavardhanaandGitHub ce93b2681b fix: re-use er.getDisks() properly in certain calls (#11043) 2020-12-07 10:04:07 -08:00
HarshavardhanaandGitHub 8d036ed6d8 fix: allow sub-admin to modify password for other users (#11039)
fixes #11037
2020-12-06 20:36:34 -08:00
HarshavardhanaandGitHub 9c53cc1b83 fix: heal multiple buckets in bulk (#11029)
makes server startup, orders of magnitude
faster with large number of buckets
2020-12-05 13:00:44 -08:00
HarshavardhanaandGitHub 3514e89eb3 support envs as well for new crawler sub-system (#11033) 2020-12-04 21:54:24 -08:00
Klaus PostandGitHub a896125490 Add crawler delay config + dynamic config values (#11018) 2020-12-04 09:32:35 -08:00
HarshavardhanaandGitHub e083471ec4 use argon2 with sync.Pool for better memory management (#11019) 2020-12-03 19:23:19 -08:00
HarshavardhanaandGitHub 80d31113e5 fix: etcd import paths again depend on v3.4.14 release (#11020)
Due to botched upstream renames of project repositories
and incomplete migration to go.mod support, our current
dependency version of `go.mod` had bugs i.e it was
using commits from master branch which didn't have
the required fixes present in release-3.4 branches

which leads to some rare bugs

https://github.com/etcd-io/etcd/pull/11477 provides
a workaround for now and we should migrate to this.

release-3.5 eventually claims to fix all of this
properly until then we cannot use /v3 import right now
2020-12-03 11:35:18 -08:00
Ritesh H ShuklaandGitHub 7e2b79984e Stream bucket bandwidth measurements (#11014) 2020-12-03 11:34:42 -08:00
Harshavardhana 951b6b203b skip metacache entries healing to speed up startup 2020-12-02 21:30:54 -08:00
Harshavardhana 44e23b7f4f fix: startup being slow - wait only if IOCount > 0 2020-12-02 21:06:17 -08:00
HarshavardhanaandGitHub 96c0ce1f0c add support for tuning healing to make healing more aggressive (#11003)
supports `mc admin config set <alias> heal sleep=100ms` to
enable more aggressive healing under certain times.

also optimize some areas that were doing extra checks than
necessary when bitrotscan was enabled, avoid double sleeps
make healing more predictable.

fixes #10497
2020-12-02 11:12:00 -08:00
ebozdumanandGitHub 303be1866d Adds "x-amz-usr-agent" and "x-id" params to be used in authentication of presignedURL (#10792) 2020-12-02 02:02:49 -08:00
Harshavardhana 4ec45753e6 rename server sets to server pools 2020-12-01 13:50:33 -08:00
Klaus PostandGitHub e6ea5c2703 crawler: Missing folder heal check per set (#10876) 2020-12-01 12:07:39 -08:00
Harshavardhana 790833f3b2 Revert "Support variable server sets (#10314)"
This reverts commit aabf053d2f.
2020-12-01 12:02:29 -08:00
HarshavardhanaandGitHub 7cbca43eb1 fix: allow admins to create users (#11005)
PR #10978 introduced a regression, root
credential should be allowed to create users
2020-11-30 21:53:23 -08:00
Poorna KrishnamoorthyandGitHub 2f564437ae Disallow writeback caching with cache_after (#11002)
fixes #10974
2020-11-30 20:53:27 -08:00
HarshavardhanaandGitHub bdd094bc39 fix: avoid sending errors on missing objects on locked buckets (#10994)
make sure multi-object delete returned errors that are AWS S3 compatible
2020-11-28 21:15:45 -08:00
HarshavardhanaandGitHub e6fa410778 fix: allow accountInfo, addUser and getUserInfo implicit (#10978)
- accountInfo API that returns information about
  user, access to buckets and the size per bucket
- addUser - user is allowed to change their secretKey
- getUserInfo - returns user info if the incoming
  is the same user requesting their information
2020-11-27 17:23:57 -08:00
HarshavardhanaandGitHub aabf053d2f Support variable server sets (#10314) 2020-11-25 16:28:47 -08:00
Anis ElleuchandGitHub 91130e884b Avoid sending errors in gob in storage requests (#10977) 2020-11-25 12:42:48 -08:00
Poorna KrishnamoorthyandGitHub 2ff655a745 Refactor replication, ILM handling in DELETE API (#10945) 2020-11-25 11:24:50 -08:00
Klaus PostandGitHub 0422eda6a2 metacache: Always close block writer (#10973)
In some cases a writer could be left behind unclosed, leaking compression blocks.

Always close and set compression concurrency to 2 which should be fine to keep up.
2020-11-25 09:37:30 -08:00
HarshavardhanaandGitHub 31e6f60847 fix: improve error handling in metacache (#10965) 2020-11-25 01:11:22 -08:00
Poorna KrishnamoorthyandGitHub 3ad41fe89d Add admin API to edit remote bucket target credentials (#10848) 2020-11-24 19:09:05 -08:00
Klaus PostandGitHub a75fafdbe2 Remove msgp workaround (#10964)
The error in `github.com/philhofer/fwd` was quickly fixed through 
https://github.com/philhofer/fwd/pull/22 - update the dependency 
and remove the workaround.
2020-11-24 11:58:10 -08:00
Klaus PostandGitHub a58b7874ef Temporary workaround for msgp skipping (#10960)
Due to https://github.com/philhofer/fwd/issues/20 when skipping a metadata entry that is >2048 bytes and the buffer is full (2048 bytes) the skip will fail with `io.ErrNoProgress`.

Enlarge the buffer so we temporarily make this much more unlikely.

If it still happens we will have to rewrite the skips to reads.

Fixes #10959
2020-11-23 18:51:59 -08:00
HarshavardhanaandGitHub 6990de9c94 fix: dangling object delete shall return object doesn't exist (#10961)
dangling object when deleted means object doesn't exist
anymore, so we should return appropriate errors, this
allows crawler heal to ensure that it removes the tracker
for dangling objects.
2020-11-23 18:50:53 -08:00
75a8e81f8f azure: Specify different Azure storage in the shell env (#10943)
AZURE_STORAGE_ACCOUNT and AZURE_STORAGE_KEY are used in 
azure CLI to specify the azure blob storage access & secret keys. With this commit, 
it is possible to set them if you want the gateway's own credentials to be
different from the Azure blob credentials.

Co-authored-by: Harshavardhana <harsha@minio.io>
2020-11-23 16:45:56 -08:00
HarshavardhanaandGitHub 519c0077a9 fix: do not return an error for successfully deleted dangling objects (#10938)
dangling objects when removed `mc admin heal -r` or crawler
auto heal would incorrectly return error - this can interfere
with usage calculation as the entry size for this would be
returned as `0`, instead upon success use the resultant
object size to calculate the final size for the object
and avoid reporting this in the log messages

Also do not set ObjectSize in healResultItem to be '-1'
this has an effect on crawler metrics calculating 1 byte
less for objects which seem to be missing their `xl.meta`
2020-11-23 09:12:17 -08:00
HarshavardhanaandGitHub 734d07a532 fix: all hosts local and port same should be local erasure setup (#10951)
this is needed to avoid initializing notification peers
that can lead to races in many sub-systems

fixes #10950
2020-11-23 09:07:50 -08:00
HarshavardhanaandGitHub df93102235 fix: unwrapping issues with os.Is* functions (#10949)
reduces  3 stat calls, reducing the
overall startup time significantly.
2020-11-23 08:36:49 -08:00
Poorna KrishnamoorthyandGitHub 39f3d5493b Show Delete replication status header (#10946)
X-Minio-Replication-Delete-Status header shows the
status of the replication of a permanent delete of a version.

All GETs are disallowed and return 405 on this object version.
In the case of replicating delete markers.

X-Minio-Replication-DeleteMarker-Status shows the status 
of replication, and would similarly return 405.

Additionally, this PR adds reporting of delete marker event completion
and updates documentation
2020-11-21 23:48:50 -08:00
Klaus PostandGitHub 692ff41ef7 Unwrap network errors (#10934)
Alternative to #10927

Instead of having an upstream fix, do unwrap when checking network errors.

'As' will also work when destination is an interface as checked by the tests.
2020-11-20 22:55:35 -08:00
HarshavardhanaandGitHub 86409fa93d add audit/admin trace support for browser requests (#10947)
To support this functionality we had to fork
the gorilla/rpc package with relevant changes
2020-11-20 22:52:17 -08:00
Shireesh AnjalandGitHub 7bc47a14cc Rename OBD to Health (#10842)
Also, Remove thread stats and openfds from the health report 
as we already have process stats and numfds
2020-11-20 12:52:53 -08:00
HarshavardhanaandGitHub 73e308079a fix: handle errors appropriately as they are wrapped (#10917) 2020-11-20 10:43:07 -08:00
Poorna KrishnamoorthyandHarshavardhana 08b24620c0 Display storage-class of transitioned object in HEAD 2020-11-20 09:17:31 -08:00
HarshavardhanaandGitHub 95675b0c9a fix: do not crash PutObjectTags when node is down (#10940)
fixes #10939
2020-11-20 09:10:48 -08:00
Poorna KrishnamoorthyandHarshavardhana 251c1ef6da Add support for replication of object tags, retention metadata (#10880) 2020-11-19 18:56:09 -08:00
Poorna KrishnamoorthyandHarshavardhana 0fa430c1da validate service type of target in replication/ilm transition config (#10928) 2020-11-19 18:47:33 -08:00