Compare commits

...
17 Commits
Author SHA1 Message Date
chaunceyandGitHub 959e1f2576 fixed: webdav 上传接受图片时不生成缩略图的问题 (#779)
* fix:修复webdav上传图片时无缩略图问题

* 修改 Trim 为 TrimRight

* 1.增加图片生成缩略图时自动识别EXIF信息并存储到数据库

2.修改迁移配置、manage 接口返回文件增加 exif 信息字段

3.计划任务增加自动识别将经纬度转换为文本地址任务

* 变更assets

* fix:修复前端页面合并遗留bug

* 1. 将获取更新EXIF地址的文件列表函数更名为 GetEmptyLocationFilesByPage

2.页面API展示列表接口增加过滤默认空时间

* update assets
2021-03-10 22:44:29 -08:00
AaronLiuandGitHub e44ec0e6bf Update issue templates 2021-03-05 15:44:33 +08:00
HFO4 a93b964d8b Modify: OneDrive file URL cache will refreshed after file is updated 2021-03-03 17:07:26 +08:00
HFO4 d9cff24c75 Modify: disable association_autoupdate in model.File.UpdateSourceName 2021-03-03 14:10:08 +08:00
HFO4 e2488841b4 Test: #765 2021-03-02 12:45:54 +08:00
日下部 詩andGitHub a276be4098 注册帐号时,如果尚未验证,再发一次验证信 (#765)
* 注册帐号时,如果尚未验证,再发一次验证信

* 修正2个bug。 1:未验证显示密码错误 2:未验证无法重发email

* 小修正,如果已存在user,拿已有user资讯取代掉新user资讯来寄送激活码

* 激活码改成激活邮件

* 忘记密码以后,重设二步验证设定

* Revert "忘记密码以后,重设二步验证设定"

This reverts commit c5ac10b11c.

* 實作 https://github.com/cloudreve/Cloudreve/pull/765#discussion_r584313520
2021-03-02 12:43:14 +08:00
HFO4 4cf6c81534 Fix: failed unit test 2021-03-02 12:32:34 +08:00
HFO4 5a66af3105 Fix: failed unit test 2021-03-02 12:21:43 +08:00
HFO4 fc5c67cc20 Feat: disable overwrite for OneDrive policy 2021-03-01 13:27:18 +08:00
HFO4 5e226efea1 Feat: disable overwrite for non-updating put request, only works under local,slave,OneDrive,OSS policy. (#764) 2021-03-01 13:03:49 +08:00
HFO4 c949d47161 Update submodule version 2021-02-28 16:51:27 +08:00
HFO4 e699287ffd Modify: mark as success when deleting a file that does not exist;
Fix: minio is not usable in S3 policy
Modify: use batch request to delete S3 files
2021-02-28 16:48:51 +08:00
Cinhi YoungandGitHub 9c78515c72 Fix: email address should be lowercase for requesting Gravatar (#758) 2021-02-08 19:33:09 +08:00
HFO4 3b22b4fd25 Update version number 2021-01-06 18:18:24 +08:00
HFO4 08d998b41e Update submodule 2021-01-06 17:38:20 +08:00
Breeze ChenandGitHub 488e62f762 Fix qiniu last modify time. (#691) 2021-01-06 17:01:24 +08:00
HFO4 f35ad3fe0a Fix: #663 2021-01-06 16:35:31 +08:00
41 changed files with 486 additions and 115 deletions
+38
View File
@@ -0,0 +1,38 @@
---
name: Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''
---
**Describe the bug**
A clear and concise description of what the bug is.
**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
**Expected behavior**
A clear and concise description of what you expected to happen.
**Screenshots**
If applicable, add screenshots to help explain your problem.
**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]
**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
**Additional context**
Add any other context about the problem here.
+20
View File
@@ -0,0 +1,20 @@
---
name: Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''
---
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
+1 -1
Submodule assets updated: d882649a86...1c827ee20a
+1
View File
@@ -31,6 +31,7 @@ require (
github.com/rafaeljusto/redigomock v0.0.0-20191117212112-00b2509252a1
github.com/rakyll/statik v0.1.7
github.com/robfig/cron/v3 v3.0.1
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd // indirect
github.com/smartystreets/goconvey v1.6.4 // indirect
github.com/speps/go-hashids v2.0.0+incompatible
github.com/stretchr/testify v1.5.1
+2
View File
@@ -208,6 +208,8 @@ github.com/rakyll/statik v0.1.7/go.mod h1:AlZONWzMtEnMs7W4e/1LURLiI49pIMmp6V9Ung
github.com/rcrowley/go-metrics v0.0.0-20181016184325-3113b8401b8a/go.mod h1:bCqnVzQkZxMG4s8nGwiZ5l3QUCyqpo9Y+/ZMZ9VjZe4=
github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd h1:CmH9+J6ZSsIjUK3dcGsnCnO41eRBOnY12zwkn5qVwgc=
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=
github.com/satori/go.uuid v1.2.0 h1:0uYX9dsZ2yD7q2RtLRtPSdGDWzjeM3TbMJP9utgA0ww=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
+1 -1
View File
@@ -90,7 +90,7 @@ func WebDAVAuth() gin.HandlerFunc {
return
}
expectedUser, err := model.GetUserByEmail(username)
expectedUser, err := model.GetActiveUserByEmail(username)
if err != nil {
c.Status(http.StatusUnauthorized)
c.Abort()
+34 -1
View File
@@ -26,6 +26,12 @@ type File struct {
// 数据库忽略字段
Position string `gorm:"-"`
// exif 信息
ExifModel string
ExifDateTime time.Time
ExifLatLong string
ExifAddress string
}
func init() {
@@ -79,6 +85,16 @@ func GetFilesByIDs(ids []uint, uid uint) ([]File, error) {
return files, result.Error
}
// GetEmptyLocationFilesByPage 分页获取所有经纬度未更新文件
func GetEmptyLocationFilesByPage(page uint, pageSize uint) ([]File, error) {
var files []File
var result *gorm.DB
var offset = int(page) * int(pageSize) - int(pageSize)
result = DB.Where("exif_lat_long != '' AND (exif_address is null or exif_address = '')").Limit(pageSize).
Offset(offset).Find(&files)
return files, result.Error
}
// GetFilesByKeywords 根据关键字搜索文件,
// UID为0表示忽略用户,只根据文件ID检索
func GetFilesByKeywords(uid uint, keywords ...interface{}) ([]File, error) {
@@ -196,7 +212,24 @@ func (file *File) UpdateSize(value uint64) error {
// UpdateSourceName 更新文件的源文件名
func (file *File) UpdateSourceName(value string) error {
return DB.Model(&file).Update("source_name", value).Error
return DB.Model(&file).Set("gorm:association_autoupdate", false).Update("source_name", value).Error
}
// UpdatePicExifModel 更新文件的设备信息
func (file *File) UpdatePicExifModel(value string) error {
return DB.Model(&file).Update("exif_model", value).Error
}
// UpdatePicExifDateTime 更新图片EXIF时间
func (file *File) UpdatePicExifDateTime(value time.Time) error {
return DB.Model(&file).Update("exif_date_time", value).Error
}
// UpdatePicExifLatLong 更新图片EXIF坐标
func (file *File) UpdatePicExifLatLong(value string) error {
return DB.Model(&file).Update("exif_lat_long", value).Error
}
// UpdatePicExifAddress 更新图片EXIF位置信息
func (file *File) UpdatePicExifAddress(value string) error {
return DB.Model(&file).Update("exif_address", value).Error
}
/*
+1
View File
@@ -147,6 +147,7 @@ Neue',Helvetica,Arial,sans-serif; box-sizing: border-box; font-size: 14px; verti
{Name: "home_view_method", Value: "icon", Type: "view"},
{Name: "share_view_method", Value: "list", Type: "view"},
{Name: "cron_garbage_collect", Value: "@hourly", Type: "cron"},
{Name: "cron_sync_photo_lat_long_to_address", Value: "@hourly", Type: "cron"},
{Name: "authn_enabled", Value: "0", Type: "authn"},
{Name: "captcha_height", Value: "60", Type: "captcha"},
{Name: "captcha_width", Value: "240", Type: "captcha"},
+7
View File
@@ -139,6 +139,13 @@ func GetActiveUserByOpenID(openid string) (User, error) {
// GetUserByEmail 用Email获取用户
func GetUserByEmail(email string) (User, error) {
var user User
result := DB.Set("gorm:auto_preload", true).Where("email = ?", email).First(&user)
return user, result.Error
}
// GetActiveUserByEmail 用Email获取可登录用户
func GetActiveUserByEmail(email string) (User, error) {
var user User
result := DB.Set("gorm:auto_preload", true).Where("status = ? and email = ?", Active, email).First(&user)
return user, result.Error
+11 -1
View File
@@ -352,10 +352,20 @@ func TestUser_IncreaseStorageWithoutCheck(t *testing.T) {
}
}
func TestGetUserByEmail(t *testing.T) {
func TestGetActiveUserByEmail(t *testing.T) {
asserts := assert.New(t)
mock.ExpectQuery("SELECT(.+)").WithArgs(Active, "abslant@foxmail.com").WillReturnRows(sqlmock.NewRows([]string{"id", "email"}))
_, err := GetActiveUserByEmail("abslant@foxmail.com")
asserts.Error(err)
asserts.NoError(mock.ExpectationsWereMet())
}
func TestGetUserByEmail(t *testing.T) {
asserts := assert.New(t)
mock.ExpectQuery("SELECT(.+)").WithArgs("abslant@foxmail.com").WillReturnRows(sqlmock.NewRows([]string{"id", "email"}))
_, err := GetUserByEmail("abslant@foxmail.com")
asserts.Error(err)
+3 -3
View File
@@ -1,13 +1,13 @@
package conf
// BackendVersion 当前后端版本号
var BackendVersion = "3.2.0"
var BackendVersion = "3.2.1"
// RequiredDBVersion 与当前版本匹配的数据库版本
var RequiredDBVersion = "3.2.0"
var RequiredDBVersion = "3.2.1"
// RequiredStaticVersion 与当前版本匹配的静态资源版本
var RequiredStaticVersion = "3.2.0"
var RequiredStaticVersion = "3.2.1"
// IsPro 是否为Pro版本
var IsPro = "false"
+108
View File
@@ -0,0 +1,108 @@
package crontab
import (
"net/http"
"encoding/json"
model "github.com/cloudreve/Cloudreve/v3/models"
"github.com/cloudreve/Cloudreve/v3/pkg/util"
"github.com/cloudreve/Cloudreve/v3/pkg/request"
)
type AddressJson struct {
Status int `json:"status"`
Message string `json:"message"`
RequestID string `json:"request_id"`
Result struct {
Location struct {
Lat float64 `json:"lat"`
Lng float64 `json:"lng"`
} `json:"location"`
Address string `json:"address"`
FormattedAddresses struct {
Recommend string `json:"recommend"`
Rough string `json:"rough"`
} `json:"formatted_addresses"`
AddressComponent struct {
Nation string `json:"nation"`
Province string `json:"province"`
City string `json:"city"`
District string `json:"district"`
Street string `json:"street"`
StreetNumber string `json:"street_number"`
} `json:"address_component"`
} `json:"result"`
}
type AddressExif struct{
Address string `json:"address"`
Nation string `json:"nation"`
Province string `json:"province"`
City string `json:"city"`
District string `json:"district"`
Street string `json:"street"`
StreetNumber string `json:"street_number"`
}
func syncPhotoAddress() {
// 同步照片的经纬度为文本地址
syncPhotoLatLongToAddress()
util.Log().Info("定时任务 [cron_sync_photo_lat_long_to_address] 执行完毕")
}
func syncPhotoLatLongToAddress() {
page := 1
pageSize := 10
for true{
files , _ := model.GetEmptyLocationFilesByPage(uint(page), uint(pageSize))
if len(files) <= 0 {
break
}
for i := 0; i < len(files); i++{
file := files[i]
util.Log().Debug("file name: %s",file.Name)
util.Log().Debug("file ExifLatLong: %s",file.ExifLatLong)
util.Log().Debug("file ExifAddress: %s",file.ExifAddress)
// 获取文件数据流
url := "https://apis.map.qq.com/ws/geocoder/v1/?location="+ file.ExifLatLong +"&get_poi=1&key=OB4BZ-D4W3U-B7VVO-4PJWW-6TKDJ-WPB77"
client := request.HTTPClient{}
resp := client.Request(
"GET",
url,
nil,
request.WithHeader(
http.Header{"Referer": {"https://lbs.qq.com/"}},
),
)
respString, err := resp.GetResponse()
if err != nil{
util.Log().Warning("response error: %s",err)
}
var addressJson AddressJson
err = json.Unmarshal([]byte(respString), &addressJson)
if err != nil {
util.Log().Warning("解析经纬度结果错误原始文本: %s",respString)
util.Log().Warning("解析经纬度结果错误: %s",err)
continue
}
var addressInfo AddressExif
addressInfo.Address = addressJson.Result.Address
addressInfo.StreetNumber = addressJson.Result.AddressComponent.StreetNumber
addressInfo.District = addressJson.Result.AddressComponent.District
addressInfo.City = addressJson.Result.AddressComponent.City
addressInfo.Province = addressJson.Result.AddressComponent.Province
addressInfo.Nation = addressJson.Result.AddressComponent.Nation
if addressInfoStr, err := json.Marshal(addressInfo); err == nil {
file.UpdatePicExifAddress(string(addressInfoStr))
util.Log().Debug("解析经纬度结果重组: %s",string(addressInfoStr))
}
}
page++
}
}
+3 -1
View File
@@ -21,13 +21,15 @@ func Reload() {
func Init() {
util.Log().Info("初始化定时任务...")
// 读取cron日程设置
options := model.GetSettingByNames("cron_garbage_collect")
options := model.GetSettingByNames("cron_garbage_collect","cron_sync_photo_lat_long_to_address")
Cron := cron.New()
for k, v := range options {
var handler func()
switch k {
case "cron_garbage_collect":
handler = garbageCollect
case "cron_sync_photo_lat_long_to_address":
handler = syncPhotoAddress
default:
util.Log().Warning("未知定时任务类型 [%s],跳过", k)
continue
+16 -5
View File
@@ -100,6 +100,14 @@ func (handler Driver) Put(ctx context.Context, file io.ReadCloser, dst string, s
defer file.Close()
dst = util.RelativePath(filepath.FromSlash(dst))
// 如果禁止了 Overwrite,则检查是否有重名冲突
if ctx.Value(fsctx.DisableOverwrite) != nil {
if util.Exists(dst) {
util.Log().Warning("物理同名文件已存在或不可用: %s", dst)
return errors.New("物理同名文件已存在或不可用")
}
}
// 如果目标目录不存在,创建
basePath := filepath.Dir(dst)
if !util.Exists(basePath) {
@@ -130,11 +138,14 @@ func (handler Driver) Delete(ctx context.Context, files []string) ([]string, err
var retErr error
for _, value := range files {
err := os.Remove(util.RelativePath(filepath.FromSlash(value)))
if err != nil {
util.Log().Warning("无法删除文件,%s", err)
retErr = err
deleteFailed = append(deleteFailed, value)
filePath := util.RelativePath(filepath.FromSlash(value))
if util.Exists(filePath) {
err := os.Remove(filePath)
if err != nil {
util.Log().Warning("无法删除文件,%s", err)
retErr = err
deleteFailed = append(deleteFailed, value)
}
}
// 尝试删除文件的缩略图(如果有)
+30 -15
View File
@@ -2,13 +2,6 @@ package local
import (
"context"
"io"
"io/ioutil"
"net/url"
"os"
"strings"
"testing"
model "github.com/cloudreve/Cloudreve/v3/models"
"github.com/cloudreve/Cloudreve/v3/pkg/auth"
"github.com/cloudreve/Cloudreve/v3/pkg/conf"
@@ -16,12 +9,19 @@ import (
"github.com/cloudreve/Cloudreve/v3/pkg/util"
"github.com/jinzhu/gorm"
"github.com/stretchr/testify/assert"
"io"
"io/ioutil"
"net/url"
"os"
"strings"
"testing"
)
func TestHandler_Put(t *testing.T) {
asserts := assert.New(t)
handler := Driver{}
ctx := context.Background()
ctx := context.WithValue(context.Background(), fsctx.DisableOverwrite, true)
os.Remove(util.RelativePath("test/test/txt"))
testCases := []struct {
file io.ReadCloser
@@ -33,6 +33,11 @@ func TestHandler_Put(t *testing.T) {
dst: "test/test/txt",
err: false,
},
{
file: ioutil.NopCloser(strings.NewReader("test input file")),
dst: "test/test/txt",
err: true,
},
{
file: ioutil.NopCloser(strings.NewReader("test input file")),
dst: "/notexist:/S.TXT",
@@ -55,24 +60,34 @@ func TestHandler_Delete(t *testing.T) {
asserts := assert.New(t)
handler := Driver{}
ctx := context.Background()
filePath := util.RelativePath("test.file")
file, err := os.Create(util.RelativePath("test.file"))
file, err := os.Create(filePath)
asserts.NoError(err)
_ = file.Close()
list, err := handler.Delete(ctx, []string{"test.file"})
asserts.Equal([]string{}, list)
asserts.NoError(err)
file, err = os.Create(util.RelativePath("test.file"))
asserts.NoError(err)
file, err = os.Create(filePath)
_ = file.Close()
file, _ = os.OpenFile(filePath, os.O_RDWR, os.FileMode(0))
asserts.NoError(err)
list, err = handler.Delete(ctx, []string{"test.file", "test.notexist"})
asserts.Equal([]string{"test.notexist"}, list)
asserts.Error(err)
file.Close()
asserts.Equal([]string{}, list)
asserts.NoError(err)
list, err = handler.Delete(ctx, []string{"test.notexist"})
asserts.Equal([]string{"test.notexist"}, list)
asserts.Error(err)
asserts.Equal([]string{}, list)
asserts.NoError(err)
file, err = os.Create(filePath)
asserts.NoError(err)
list, err = handler.Delete(ctx, []string{"test.file"})
_ = file.Close()
asserts.Equal([]string{}, list)
asserts.NoError(err)
}
func TestHandler_Get(t *testing.T) {
+19 -7
View File
@@ -220,15 +220,21 @@ func (client *Client) UploadChunk(ctx context.Context, uploadURL string, chunk *
// Upload 上传文件
func (client *Client) Upload(ctx context.Context, dst string, size int, file io.Reader) error {
// 决定是否覆盖文件
overwrite := "replace"
if ctx.Value(fsctx.DisableOverwrite) != nil {
overwrite = "fail"
}
// 小文件,使用简单上传接口上传
if size <= int(SmallFileSize) {
_, err := client.SimpleUpload(ctx, dst, file, int64(size))
_, err := client.SimpleUpload(ctx, dst, file, int64(size), WithConflictBehavior(overwrite))
return err
}
// 大文件,进行分片
// 创建上传会话
uploadURL, err := client.CreateUploadSession(ctx, dst, WithConflictBehavior("replace"))
uploadURL, err := client.CreateUploadSession(ctx, dst, WithConflictBehavior(overwrite))
if err != nil {
return err
}
@@ -287,9 +293,15 @@ func (client *Client) DeleteUploadSession(ctx context.Context, uploadURL string)
}
// SimpleUpload 上传小文件到dst
func (client *Client) SimpleUpload(ctx context.Context, dst string, body io.Reader, size int64) (*UploadResult, error) {
func (client *Client) SimpleUpload(ctx context.Context, dst string, body io.Reader, size int64, opts ...Option) (*UploadResult, error) {
options := newDefaultOption()
for _, o := range opts {
o.apply(options)
}
dst = strings.TrimPrefix(dst, "/")
requestURL := client.getRequestURL("me/drive/root:/" + dst + ":/content")
requestURL += ("?@microsoft.graph.conflictBehavior=" + options.conflictBehavior)
res, err := client.request(ctx, "PUT", requestURL, body, request.WithContentLength(int64(size)),
request.WithTimeout(time.Duration(150)*time.Second),
@@ -303,7 +315,7 @@ func (client *Client) SimpleUpload(ctx context.Context, dst string, body io.Read
retried++
util.Log().Debug("文件[%s]上传失败[%s]5秒钟后重试", dst, err)
time.Sleep(time.Duration(5) * time.Second)
return client.SimpleUpload(context.WithValue(ctx, fsctx.RetryCtx, retried), dst, body, size)
return client.SimpleUpload(context.WithValue(ctx, fsctx.RetryCtx, retried), dst, body, size, opts...)
}
return nil, err
}
@@ -370,7 +382,7 @@ func (client *Client) Delete(ctx context.Context, dst []string) ([]string, error
func getDeleteFailed(res *BatchResponses) []string {
var failed = make([]string, 0, len(res.Responses))
for _, v := range res.Responses {
if v.Status != 204 {
if v.Status != 204 && v.Status != 404 {
failed = append(failed, v.ID)
}
}
@@ -456,7 +468,7 @@ func (client *Client) MonitorUpload(uploadURL, callbackKey, path string, size ui
case <-time.After(time.Duration(ttl) * time.Second):
// 上传会话到期,仍未完成上传,创建占位符
client.DeleteUploadSession(context.Background(), uploadURL)
_, err := client.SimpleUpload(context.Background(), path, strings.NewReader(""), 0)
_, err := client.SimpleUpload(context.Background(), path, strings.NewReader(""), 0, WithConflictBehavior("replace"))
if err != nil {
util.Log().Debug("无法创建占位文件,%s", err)
}
@@ -504,7 +516,7 @@ func (client *Client) MonitorUpload(uploadURL, callbackKey, path string, size ui
// 取消上传会话,实测OneDrive取消上传会话后,客户端还是可以上传,
// 所以上传一个空文件占位,阻止客户端上传
client.DeleteUploadSession(context.Background(), uploadURL)
_, err := client.SimpleUpload(context.Background(), path, strings.NewReader(""), 0)
_, err := client.SimpleUpload(context.Background(), path, strings.NewReader(""), 0, WithConflictBehavior("replace"))
if err != nil {
util.Log().Debug("无法创建占位文件,%s", err)
}
+3 -1
View File
@@ -12,6 +12,7 @@ import (
model "github.com/cloudreve/Cloudreve/v3/models"
"github.com/cloudreve/Cloudreve/v3/pkg/cache"
"github.com/cloudreve/Cloudreve/v3/pkg/filesystem/fsctx"
"github.com/cloudreve/Cloudreve/v3/pkg/request"
"github.com/stretchr/testify/assert"
testMock "github.com/stretchr/testify/mock"
@@ -499,11 +500,12 @@ func TestClient_Upload(t *testing.T) {
client, _ := NewClient(&model.Policy{})
client.Credential.AccessToken = "AccessToken"
client.Credential.ExpiresIn = time.Now().Add(time.Duration(100) * time.Hour).Unix()
ctx := context.WithValue(context.Background(), fsctx.DisableOverwrite, true)
// 小文件,简单上传,失败
{
client.Credential.ExpiresIn = 0
err := client.Upload(context.Background(), "123.jpg", 3, strings.NewReader("123"))
err := client.Upload(ctx, "123.jpg", 3, strings.NewReader("123"))
asserts.Error(err)
}
+7 -2
View File
@@ -152,8 +152,13 @@ func (handler Driver) Source(
isDownload bool,
speed int,
) (string, error) {
cacheKey := fmt.Sprintf("onedrive_source_%d_%s", handler.Policy.ID, path)
if file, ok := ctx.Value(fsctx.FileModelCtx).(model.File); ok {
cacheKey = fmt.Sprintf("onedrive_source_file_%d_%d", file.UpdatedAt.Unix(), file.ID)
}
// 尝试从缓存中查找
if cachedURL, ok := cache.Get(fmt.Sprintf("onedrive_source_%d_%s", handler.Policy.ID, path)); ok {
if cachedURL, ok := cache.Get(cacheKey); ok {
return handler.replaceSourceHost(cachedURL.(string))
}
@@ -162,7 +167,7 @@ func (handler Driver) Source(
if err == nil {
// 写入新的缓存
cache.Set(
fmt.Sprintf("onedrive_source_%d_%s", handler.Policy.ID, path),
cacheKey,
res.DownloadURL,
model.GetIntSetting("onedrive_source_timeout", 1800),
)
@@ -2,6 +2,7 @@ package onedrive
import (
"context"
"fmt"
"io"
"io/ioutil"
"net/http"
@@ -150,6 +151,21 @@ func TestDriver_Source(t *testing.T) {
asserts.Equal("res", res)
}
// 命中缓存 上下文存在文件 成功
{
file := model.File{}
file.ID = 1
file.UpdatedAt = time.Now()
ctx := context.WithValue(context.Background(), fsctx.FileModelCtx, file)
handler.Client.Credential.ExpiresIn = time.Now().Add(time.Duration(100) * time.Hour).Unix()
handler.Client.Credential.AccessToken = "1"
cache.Set(fmt.Sprintf("onedrive_source_file_%d_1", file.UpdatedAt.Unix()), "res", 0)
res, err := handler.Source(ctx, "123.jpg", url.URL{}, 0, true, 0)
cache.Deletes([]string{"0_123.jpg"}, "onedrive_source_")
asserts.NoError(err)
asserts.Equal("res", res)
}
// 成功
{
handler.Client.Credential.ExpiresIn = time.Now().Add(time.Duration(100) * time.Hour).Unix()
+7
View File
@@ -235,8 +235,15 @@ func (handler Driver) Put(ctx context.Context, file io.ReadCloser, dst string, s
// 凭证有效期
credentialTTL := model.GetIntSetting("upload_credential_timeout", 3600)
// 是否允许覆盖
overwrite := true
if ctx.Value(fsctx.DisableOverwrite) != nil {
overwrite = false
}
options := []oss.Option{
oss.Expires(time.Now().Add(time.Duration(credentialTTL) * time.Second)),
oss.ForbidOverWrite(!overwrite),
}
// 上传文件
+2 -1
View File
@@ -265,10 +265,11 @@ func TestDriver_Put(t *testing.T) {
},
}
cache.Set("setting_upload_credential_timeout", "3600", 0)
ctx := context.WithValue(context.Background(), fsctx.DisableOverwrite, true)
// 失败
{
err := handler.Put(context.Background(), ioutil.NopCloser(strings.NewReader("123")), "/123.txt", 3)
err := handler.Put(ctx, ioutil.NopCloser(strings.NewReader("123")), "/123.txt", 3)
asserts.Error(err)
}
}
+2 -2
View File
@@ -92,7 +92,7 @@ func (handler Driver) List(ctx context.Context, base string, recursive bool) ([]
RelativePath: filepath.ToSlash(rel),
Size: uint64(object.Fsize),
IsDir: false,
LastModify: time.Now(),
LastModify: time.Unix(object.PutTime/10000000, 0),
})
}
@@ -206,7 +206,7 @@ func (handler Driver) Delete(ctx context.Context, files []string) ([]string, err
if err != nil {
failed := make([]string, 0, len(rets))
for k, ret := range rets {
if ret.Code != 200 {
if ret.Code != 200 && ret.Code != 612 {
failed = append(failed, files[k])
}
}
+10 -1
View File
@@ -155,6 +155,13 @@ func (handler Driver) Put(ctx context.Context, file io.ReadCloser, dst string, s
if err != nil {
return err
}
// 决定是否要禁用文件覆盖
overwrite := "true"
if ctx.Value(fsctx.DisableOverwrite) != nil {
overwrite = "false"
}
// 上传文件
resp, err := handler.Client.Request(
"POST",
@@ -164,6 +171,7 @@ func (handler Driver) Put(ctx context.Context, file io.ReadCloser, dst string, s
"Authorization": {credential.Token},
"X-Policy": {credential.Policy},
"X-FileName": {fileName},
"X-Overwrite": {overwrite},
}),
request.WithContentLength(int64(size)),
request.WithTimeout(time.Duration(0)),
@@ -321,7 +329,8 @@ func (handler Driver) getUploadCredential(ctx context.Context, policy serializer
// 签名上传策略
uploadRequest, _ := http.NewRequest("POST", "/api/v3/slave/upload", nil)
uploadRequest.Header = map[string][]string{
"X-Policy": {policyEncoded},
"X-Policy": {policyEncoded},
"X-Overwrite": {"false"},
}
auth.SignRequest(handler.AuthInstance, uploadRequest, TTL)
+2 -1
View File
@@ -34,7 +34,7 @@ func TestHandler_Token(t *testing.T) {
},
AuthInstance: auth.HMACAuth{},
}
ctx := context.Background()
ctx := context.WithValue(context.Background(), fsctx.DisableOverwrite, true)
auth.General = auth.HMACAuth{SecretKey: []byte("test")}
// 成功
@@ -49,6 +49,7 @@ func TestHandler_Token(t *testing.T) {
asserts.Equal(true, policy.AutoRename)
asserts.Equal("dir", policy.SavePath)
asserts.Equal("file", policy.FileName)
asserts.Equal("file", policy.FileName)
asserts.Equal([]string{"txt"}, policy.AllowedExtension)
}
+28 -46
View File
@@ -8,13 +8,13 @@ import (
"encoding/hex"
"encoding/json"
"errors"
"github.com/cloudreve/Cloudreve/v3/pkg/util"
"io"
"net/http"
"net/url"
"path"
"path/filepath"
"strings"
"sync"
"time"
"github.com/aws/aws-sdk-go/aws"
@@ -60,7 +60,7 @@ func (handler *Driver) InitS3Client() error {
Credentials: credentials.NewStaticCredentials(handler.Policy.AccessKey, handler.Policy.SecretKey, ""),
Endpoint: &handler.Policy.Server,
Region: &handler.Policy.OptionsSerialized.Region,
S3ForcePathStyle: aws.Bool(false),
S3ForcePathStyle: aws.Bool(true),
})
if err != nil {
@@ -229,53 +229,35 @@ func (handler Driver) Delete(ctx context.Context, files []string) ([]string, err
return files, err
}
var (
failed = make([]string, 0, len(files))
lastErr error
currentIndex = 0
indexLock sync.Mutex
failedLock sync.Mutex
wg sync.WaitGroup
routineNum = 4
)
wg.Add(routineNum)
failed := make([]string, 0, len(files))
deleted := make([]string, 0, len(files))
// S3不支持批量操作,这里开四个协程并行操作
for i := 0; i < routineNum; i++ {
go func() {
for {
// 取得待删除文件
indexLock.Lock()
if currentIndex >= len(files) {
// 所有文件处理完成
wg.Done()
indexLock.Unlock()
return
}
path := files[currentIndex]
currentIndex++
indexLock.Unlock()
// 发送异步删除请求
_, err := handler.svc.DeleteObject(
&s3.DeleteObjectInput{
Bucket: &handler.Policy.BucketName,
Key: &path,
})
// 处理错误
if err != nil {
failedLock.Lock()
lastErr = err
failed = append(failed, path)
failedLock.Unlock()
}
}
}()
keys := make([]*s3.ObjectIdentifier, 0, len(files))
for _, file := range files {
filePath := file
keys = append(keys, &s3.ObjectIdentifier{Key: &filePath})
}
wg.Wait()
return failed, lastErr
// 发送异步删除请求
res, err := handler.svc.DeleteObjects(
&s3.DeleteObjectsInput{
Bucket: &handler.Policy.BucketName,
Delete: &s3.Delete{
Objects: keys,
},
})
if err != nil {
return files, err
}
// 统计未删除的文件
for _, deleteRes := range res.Deleted {
deleted = append(deleted, *deleteRes.Key)
}
failed = util.SliceDifference(failed, deleted)
return failed, nil
}
+4 -2
View File
@@ -29,8 +29,8 @@ const (
ShareKeyCtx
// LimitParentCtx 限制父目录
LimitParentCtx
// IgnoreConflictCtx 忽略重名冲突
IgnoreConflictCtx
// IgnoreDirectoryConflictCtx 忽略目录重名冲突
IgnoreDirectoryConflictCtx
// RetryCtx 失败重试次数
RetryCtx
// ForceUsePublicEndpointCtx 强制使用公网 Endpoint
@@ -39,4 +39,6 @@ const (
CancelFuncCtx
// ValidateCapacityOnceCtx 限定归还容量的操作只執行一次
ValidateCapacityOnceCtx
// 禁止上传时同名覆盖操作
DisableOverwrite
)
+1
View File
@@ -321,6 +321,7 @@ func GenericAfterUpload(ctx context.Context, fs *FileSystem) error {
fs.recycleLock.Lock()
go func() {
defer fs.recycleLock.Unlock()
file.Name = strings.TrimRight(file.Name, ".tacitpart")
fs.GenerateThumbnail(ctx, file)
}()
}
+36 -4
View File
@@ -11,6 +11,8 @@ import (
"github.com/cloudreve/Cloudreve/v3/pkg/filesystem/response"
"github.com/cloudreve/Cloudreve/v3/pkg/thumb"
"github.com/cloudreve/Cloudreve/v3/pkg/util"
"github.com/rwcarlsen/goexif/exif"
)
/* ================
@@ -54,7 +56,6 @@ func (fs *FileSystem) GenerateThumbnail(ctx context.Context, file *model.File) {
if !IsInExtensionList(HandledExtension, file.Name) {
return
}
// 新建上下文
newCtx, cancel := context.WithCancel(context.Background())
defer cancel()
@@ -65,7 +66,6 @@ func (fs *FileSystem) GenerateThumbnail(ctx context.Context, file *model.File) {
return
}
defer source.Close()
image, err := thumb.NewThumbFromFile(source, file.Name)
if err != nil {
util.Log().Warning("生成缩略图时无法解析 [%s] 图像数据:%s", file.SourceName, err)
@@ -74,7 +74,6 @@ func (fs *FileSystem) GenerateThumbnail(ctx context.Context, file *model.File) {
// 获取原始图像尺寸
w, h := image.GetSize()
// 生成缩略图
image.GetThumb(fs.GenerateThumbnailSize(w, h))
// 保存到文件
@@ -83,7 +82,6 @@ func (fs *FileSystem) GenerateThumbnail(ctx context.Context, file *model.File) {
util.Log().Warning("无法保存缩略图:%s", err)
return
}
// 更新文件的图像信息
if file.Model.ID > 0 {
err = file.UpdatePicInfo(fmt.Sprintf("%d,%d", w, h))
@@ -91,6 +89,40 @@ func (fs *FileSystem) GenerateThumbnail(ctx context.Context, file *model.File) {
file.PicInfo = fmt.Sprintf("%d,%d", w, h)
}
// 更新文件的图像信息
// 记录文件句柄位置并还原, 获取 exif 信息
currentPosition, err := source.Seek(0, 1)
source.Seek(0,0)
x, err := exif.Decode(source)
source.Seek(currentPosition, 0)
if err != nil {
util.Log().Warning("照片解析EXIF失败:%s", err)
}else{
ExifCamModel, _ := x.Get(exif.Model)
file.ExifModel,_ = ExifCamModel.StringVal()
ExifDateTime, _ := x.DateTime()
if !ExifDateTime.IsZero() {
file.ExifDateTime = ExifDateTime
}
lat, long, _ := x.LatLong()
if lat > 0 && long > 0 {
file.ExifLatLong = fmt.Sprintf("%f,%f", lat, long)
}
util.Log().Debug("照片的经纬度:%f,%f", lat,long)
if file.Model.ID > 0 {
file.UpdatePicExifModel(file.ExifModel)
if !ExifDateTime.IsZero() {
file.UpdatePicExifDateTime(file.ExifDateTime)
}
if lat > 0 && long > 0 {
file.UpdatePicExifLatLong(file.ExifLatLong)
}
}
}
// 失败时删除缩略图文件
if err != nil {
_, _ = fs.Handler.Delete(newCtx, []string{file.SourceName + conf.ThumbConfig.FileSuffix})
+19 -4
View File
@@ -28,6 +28,10 @@ type Object struct {
Type string `json:"type"`
Date string `json:"date"`
Key string `json:"key,omitempty"`
ExifModel string `json:"exif_model"`
ExifDateTime string `json:"exif_date_time"`
ExifLatLong string `json:"exif_lat_long"`
ExifAddress string `json:"exit_address"`
}
// Rename 重命名对象
@@ -350,6 +354,10 @@ func (fs *FileSystem) listObjects(ctx context.Context, parent string, files []mo
Size: 0,
Type: "dir",
Date: subFolder.CreatedAt.Format("2006-01-02 15:04:05"),
ExifModel: "",
ExifDateTime: "",
ExifLatLong: "",
ExifAddress: "",
})
}
@@ -361,7 +369,10 @@ func (fs *FileSystem) listObjects(ctx context.Context, parent string, files []mo
processedPath = parent
}
}
ExifDateTimeText := ""
if !file.ExifDateTime.IsZero(){
ExifDateTimeText = file.ExifDateTime.Format("2006-01-02 15:04:05")
}
newFile := Object{
ID: hashid.HashID(file.ID, hashid.FileID),
Name: file.Name,
@@ -370,6 +381,10 @@ func (fs *FileSystem) listObjects(ctx context.Context, parent string, files []mo
Size: file.Size,
Type: "file",
Date: file.CreatedAt.Format("2006-01-02 15:04:05"),
ExifModel: file.ExifModel,
ExifDateTime: ExifDateTimeText,
ExifLatLong: file.ExifLatLong,
ExifAddress: file.ExifAddress,
}
if shareKey != "" {
newFile.Key = shareKey
@@ -403,8 +418,8 @@ func (fs *FileSystem) CreateDirectory(ctx context.Context, fullPath string) (*mo
isExist, parent := fs.IsPathExist(base)
if !isExist {
// 递归创建父目录
if _, ok := ctx.Value(fsctx.IgnoreConflictCtx).(bool); !ok {
ctx = context.WithValue(ctx, fsctx.IgnoreConflictCtx, true)
if _, ok := ctx.Value(fsctx.IgnoreDirectoryConflictCtx).(bool); !ok {
ctx = context.WithValue(ctx, fsctx.IgnoreDirectoryConflictCtx, true)
}
newParent, err := fs.CreateDirectory(ctx, base)
if err != nil {
@@ -427,7 +442,7 @@ func (fs *FileSystem) CreateDirectory(ctx context.Context, fullPath string) (*mo
_, err := newFolder.Create()
if err != nil {
if _, ok := ctx.Value(fsctx.IgnoreConflictCtx).(bool); !ok {
if _, ok := ctx.Value(fsctx.IgnoreDirectoryConflictCtx).(bool); !ok {
return nil, ErrFolderExisted
}
+7 -1
View File
@@ -6,6 +6,7 @@ import (
model "github.com/cloudreve/Cloudreve/v3/models"
"github.com/cloudreve/Cloudreve/v3/pkg/filesystem"
"github.com/cloudreve/Cloudreve/v3/pkg/filesystem/fsctx"
)
// DecompressTask 文件压缩任务
@@ -81,7 +82,12 @@ func (job *DecompressTask) Do() {
}
job.TaskModel.SetProgress(DecompressingProgress)
err = fs.Decompress(context.Background(), job.TaskProps.Src, job.TaskProps.Dst)
// 禁止重名覆盖
ctx := context.Background()
ctx = context.WithValue(ctx, fsctx.DisableOverwrite, true)
err = fs.Decompress(ctx, job.TaskProps.Src, job.TaskProps.Dst)
if err != nil {
job.SetErrorMsg("解压缩失败", err)
return
+1 -1
View File
@@ -102,7 +102,7 @@ func (job *ImportTask) Do() {
// 列取目录、对象
job.TaskModel.SetProgress(ListingProgress)
coxIgnoreConflict := context.WithValue(context.Background(), fsctx.IgnoreConflictCtx,
coxIgnoreConflict := context.WithValue(context.Background(), fsctx.IgnoreDirectoryConflictCtx,
true)
objects, err := fs.Handler.List(ctx, job.TaskProps.Src, job.TaskProps.Recursive)
if err != nil {
+3 -1
View File
@@ -10,6 +10,7 @@ import (
model "github.com/cloudreve/Cloudreve/v3/models"
"github.com/cloudreve/Cloudreve/v3/pkg/filesystem"
"github.com/cloudreve/Cloudreve/v3/pkg/filesystem/fsctx"
"github.com/cloudreve/Cloudreve/v3/pkg/util"
)
@@ -102,7 +103,8 @@ func (job *TransferTask) Do() {
dst = path.Join(job.TaskProps.Dst, strings.TrimPrefix(src, trim))
}
err = fs.UploadFromPath(context.Background(), file, dst)
ctx := context.WithValue(context.Background(), fsctx.DisableOverwrite, true)
err = fs.UploadFromPath(ctx, file, dst)
if err != nil {
job.SetErrorMsg("文件转存失败", err)
}
+5 -2
View File
@@ -373,6 +373,9 @@ func (h *Handler) handlePut(w http.ResponseWriter, r *http.Request, fs *filesyst
fs.Use("AfterUploadFailed", filesystem.HookGiveBackCapacity)
}
// 禁止覆盖
ctx = context.WithValue(ctx, fsctx.DisableOverwrite, true)
// 执行上传
err = fs.Upload(ctx, fileData)
if err != nil {
@@ -407,8 +410,8 @@ func (h *Handler) handleMkcol(w http.ResponseWriter, r *http.Request, fs *filesy
return http.StatusUnsupportedMediaType, nil
}
if strings.Contains(r.UserAgent(), "rclone") {
if _, ok := ctx.Value(fsctx.IgnoreConflictCtx).(bool); !ok {
ctx = context.WithValue(ctx, fsctx.IgnoreConflictCtx, true)
if _, ok := ctx.Value(fsctx.IgnoreDirectoryConflictCtx).(bool); !ok {
ctx = context.WithValue(ctx, fsctx.IgnoreDirectoryConflictCtx, true)
}
}
if _, err := fs.CreateDirectory(ctx, reqPath); err != nil {
+5 -3
View File
@@ -1,7 +1,8 @@
package controllers
import (
"net/url"
model "github.com/cloudreve/Cloudreve/v3/models"
"path"
"strconv"
"github.com/cloudreve/Cloudreve/v3/pkg/serializer"
@@ -85,13 +86,14 @@ func OneDriveOAuth(c *gin.Context) {
var callbackBody callback.OneDriveOauthService
if err := c.ShouldBindQuery(&callbackBody); err == nil {
res := callbackBody.Auth(c)
redirect, _ := url.Parse("/admin/policy")
redirect := model.GetSiteURL()
redirect.Path = path.Join(redirect.Path, "/admin/policy")
queries := redirect.Query()
queries.Add("code", strconv.Itoa(res.Code))
queries.Add("msg", res.Msg)
queries.Add("err", res.Error)
redirect.RawQuery = queries.Encode()
c.Redirect(301, "/"+redirect.String())
c.Redirect(303, redirect.String())
} else {
c.JSON(200, ErrorResponse(err))
}
+1
View File
@@ -319,6 +319,7 @@ func FileUploadStream(c *gin.Context) {
// 执行上传
ctx = context.WithValue(ctx, fsctx.ValidateCapacityOnceCtx, &sync.Once{})
ctx = context.WithValue(ctx, fsctx.DisableOverwrite, true)
uploadCtx := context.WithValue(ctx, fsctx.GinCtx, c)
err = fs.Upload(uploadCtx, fileData)
if err != nil {
+5
View File
@@ -71,6 +71,11 @@ func SlaveUpload(c *gin.Context) {
fs.Use("AfterUpload", filesystem.SlaveAfterUpload)
fs.Use("AfterValidateFailed", filesystem.HookDeleteTempFile)
// 是否允许覆盖
if c.Request.Header.Get("X-Overwrite") == "false" {
ctx = context.WithValue(ctx, fsctx.DisableOverwrite, true)
}
// 执行上传
err = fs.Upload(ctx, fileData)
if err != nil {
+2 -2
View File
@@ -18,7 +18,7 @@ import (
// StartLoginAuthn 开始注册WebAuthn登录
func StartLoginAuthn(c *gin.Context) {
userName := c.Param("username")
expectedUser, err := model.GetUserByEmail(userName)
expectedUser, err := model.GetActiveUserByEmail(userName)
if err != nil {
c.JSON(200, serializer.Err(serializer.CodeNotFound, "用户不存在", err))
return
@@ -52,7 +52,7 @@ func StartLoginAuthn(c *gin.Context) {
// FinishLoginAuthn 完成注册WebAuthn登录
func FinishLoginAuthn(c *gin.Context) {
userName := c.Param("username")
expectedUser, err := model.GetUserByEmail(userName)
expectedUser, err := model.GetActiveUserByEmail(userName)
if err != nil {
c.JSON(200, serializer.Err(serializer.CodeCredentialInvalid, "用户邮箱或密码错误", err))
return
+1
View File
@@ -78,6 +78,7 @@ func (service *SingleFileService) Create(c *gin.Context) serializer.Response {
// 上下文
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
ctx = context.WithValue(ctx, fsctx.DisableOverwrite, true)
// 给文件系统分配钩子
fs.Use("BeforeUpload", filesystem.HookValidateFile)
+6
View File
@@ -94,6 +94,12 @@ func (service *UserResetEmailService) Reset(c *gin.Context) serializer.Response
// 查找用户
if user, err := model.GetUserByEmail(service.UserName); err == nil {
if user.Status == model.Baned || user.Status == model.OveruseBaned {
return serializer.Err(403, "该账号已被封禁", nil)
}
if user.Status == model.NotActivicated {
return serializer.Err(403, "该账号未激活", nil)
}
// 创建密码重设会话
secret := util.RandStringRunes(32)
cache.Set(fmt.Sprintf("user_reset_%d", user.ID), secret, 3600)
+15 -4
View File
@@ -64,10 +64,17 @@ func (service *UserRegisterService) Register(c *gin.Context) serializer.Response
user.Status = model.NotActivicated
}
user.GroupID = uint(defaultGroup)
userNotActivated := false
// 创建用户
if err := model.DB.Create(&user).Error; err != nil {
return serializer.DBErr("此邮箱已被使用", err)
//检查已存在使用者是否尚未激活
expectedUser, err := model.GetUserByEmail(service.UserName)
if expectedUser.Status == model.NotActivicated {
userNotActivated = true
user = expectedUser
} else {
return serializer.DBErr("此邮箱已被使用", err)
}
}
// 发送激活邮件
@@ -100,8 +107,12 @@ func (service *UserRegisterService) Register(c *gin.Context) serializer.Response
if err := email.Send(user.Email, title, body); err != nil {
return serializer.Err(serializer.CodeInternalSetting, "无法发送激活邮件", err)
}
return serializer.Response{Code: 203}
if userNotActivated == true {
//原本在上面要抛出的DBErr,放来这边抛出
return serializer.DBErr("用户未激活,已重新发送激活邮件", nil)
} else {
return serializer.Response{Code: 203}
}
}
return serializer.Response{}
+3 -2
View File
@@ -7,6 +7,7 @@ import (
"net/url"
"os"
"path/filepath"
"strings"
model "github.com/cloudreve/Cloudreve/v3/models"
"github.com/cloudreve/Cloudreve/v3/pkg/serializer"
@@ -200,8 +201,8 @@ func (service *AvatarService) Get(c *gin.Context) serializer.Response {
if err != nil {
return serializer.Err(serializer.CodeInternalSetting, "无法解析 Gravatar 服务器地址", err)
}
has := md5.Sum([]byte(user.Email))
email_lowered := strings.ToLower(user.Email)
has := md5.Sum([]byte(email_lowered))
avatar, _ := url.Parse(fmt.Sprintf("/avatar/%x?d=mm&s=%s", has, sizes[service.Size]))
return serializer.Response{