使用通配符匹配删除 s3 中的对象

Question

我有以下工作代码可以从 Amazon s3 中删除一个对象

params := &s3.DeleteObjectInput{

        Bucket: aws.String("Bucketname"),

        Key : aws.String("ObjectKey"),

    }

s3Conn.DeleteObjects(params)

但我想要做的是使用通配符**删除文件夹下的所有文件。我知道亚马逊 s3 不会将“x/y/file.jpg”视为 x 中的文件夹 y，但我想要实现的是通过提及“x/y*”删除所有具有相同前缀的后续对象。尝试亚马逊多对象删除

params := &s3.DeleteObjectsInput{

        Bucket: aws.String("BucketName"),

        Delete: &s3.Delete{

            Objects: []*s3.ObjectIdentifier {

                {

                    Key : aws.String("x/y/.*"), 

                },

            },

        },

    }

    result , err := s3Conn.DeleteObjects(params)

我知道在 php 中它可以通过s3->delete_all_objects根据这个答案轻松完成。在 GOlang 中是否可以执行相同的操作。

Answer 1

不幸的是 goamz 包没有类似于 PHP 库的delete_all_objects.但是，delete_all_objects此处提供了 PHP 的源代码（切换源视图）：http : //docs.aws.amazon.com/AWSSDKforPHP/latest/#m=AmazonS3/delete_all_objects以下是重要的代码行：public function delete_all_objects($bucket, $pcre = self::PCRE_ALL){// Collect all matches    $list = $this->get_object_list($bucket, array('pcre' => $pcre));    // As long as we have at least one match...    if (count($list) > 0)    {        $objects = array();        foreach ($list as $object)        {            $objects[] = array('key' => $object);        }        $batch = new CFBatchRequest();        $batch->use_credentials($this->credentials);        foreach (array_chunk($objects, 1000) as $object_set)        {            $this->batch($batch)->delete_objects($bucket, array(                'objects' => $object_set            ));        }        $responses = $this->batch($batch)->send();如您所见，PHP 代码实际上会在存储桶上发出 HTTP 请求以首先获取所有匹配的文件PCRE_ALL，该文件在别处定义为const PCRE_ALL = '/.*/i';.您一次只能删除 1000 个文件，因此delete_all_objects创建了一次删除 1000 个文件的批处理功能。您必须在 go 程序中创建相同的功能，因为 goamz 包尚不支持此功能。幸运的是，它应该只有几行代码，而且您有来自 PHP 库的指南。完成后，可能值得提交 goamz 包的拉取请求！

Answer 2

使用mc工具，您可以：mc rm -r --force https://BucketName.s3.amazonaws.com/x/y它将删除所有前缀为“x/y”的对象您可以像这样使用minio-go来实现与 Go 相同的效果：package mainimport (    "log"    "github.com/minio/minio-go")func main() {    config := minio.Config{        AccessKeyID:     "YOUR-ACCESS-KEY-HERE",        SecretAccessKey: "YOUR-PASSWORD-HERE",        Endpoint:        "https://s3.amazonaws.com",    }    // find Your S3 endpoint here http://docs.aws.amazon.com/general/latest/gr/rande.html    s3Client, err := minio.New(config)    if err != nil {        log.Fatalln(err)    }    isRecursive := true    for object := range s3Client.ListObjects("BucketName", "x/y", isRecursive) {        if object.Err != nil {            log.Fatalln(object.Err)        }        err := s3Client.RemoveObject("BucketName", object.Key)        if err != nil {            log.Fatalln(err)            continue        }        log.Println("Removed : " + object.Key)    }}

Answer 3

在游戏中有点晚，但由于我遇到了同样的问题，我创建了一个小包，您可以将其复制到您的代码库并根据需要导入。func ListKeysInPrefix(s s3iface.S3API, bucket, prefix string) ([]string, error) {&nbsp; &nbsp; res, err := s.Client.ListObjectsV2(&s3.ListObjectsV2Input{&nbsp; &nbsp; &nbsp; &nbsp; Bucket: aws.String(bucket),&nbsp; &nbsp; &nbsp; &nbsp; Prefix: aws.String(prefix),&nbsp; &nbsp; })&nbsp; &nbsp; if err != nil {&nbsp; &nbsp; &nbsp; &nbsp; return []string{}, err&nbsp; &nbsp; }&nbsp; &nbsp; var keys []string&nbsp; &nbsp; for _, key := range res.Contents {&nbsp; &nbsp; &nbsp; &nbsp; keys = append(keys, *key.Key)&nbsp; &nbsp; }&nbsp; &nbsp; return keys, nil}func createDeleteObjectsInput(keys []string) *s3.Delete {&nbsp; &nbsp; rm := []*s3.ObjectIdentifier{}&nbsp; &nbsp; for _, key := range keys {&nbsp; &nbsp; &nbsp; &nbsp; rm = append(rm, &s3.ObjectIdentifier{Key: aws.String(key)})&nbsp; &nbsp; }&nbsp; &nbsp; return &s3.Delete{Objects: rm, Quiet: aws.Bool(false)}}func DeletePrefix(s s3iface.S3API, bucket, prefix string) error {&nbsp; &nbsp; keys, err := s.ListKeysInPrefix(bucket, prefix)&nbsp; &nbsp; if err != nil {&nbsp; &nbsp; &nbsp; &nbsp; panic(err)&nbsp; &nbsp; }&nbsp; &nbsp; _, err = s.Client.DeleteObjects(&s3.DeleteObjectsInput{&nbsp; &nbsp; &nbsp; &nbsp; Bucket: aws.String(bucket),&nbsp; &nbsp; &nbsp; &nbsp; Delete: s.createDeleteObjectsInput(keys),&nbsp; &nbsp; })&nbsp; &nbsp; if err != nil {&nbsp; &nbsp; &nbsp; &nbsp; return err&nbsp; &nbsp; }&nbsp; &nbsp; return nil}因此，如果您有一个名为“somebucket”的存储桶，其结构如下：s3://somebucket/foo/some-prefixed-folder/bar/test.txt并且想从some-prefixed-folder以后删除，用法将是：func main() {&nbsp; &nbsp; // create your s3 client here&nbsp; &nbsp; // client := ....&nbsp; &nbsp; err := DeletePrefix(client, "somebucket", "some-prefixed-folder")&nbsp; &nbsp; if err != nil {&nbsp; &nbsp; &nbsp; &nbsp; panic(err)&nbsp; &nbsp; }}此实现仅允许从给定的前缀中删除最多 1000 个条目，ListObjectsV2但它是分页的，因此需要添加功能以保持刷新结果，直到结果 < 1000。