如何将 exec 函数的标准输出通过管道传输到另一个函数的读取器中？

Question

我正在尝试将标准输出从 mongodump 流式传输到 s3。我已经正确掌握了 S3 任意长度流的语法，但我不明白如何耦合这两个函数。我不想在开始上传到 S3 之前处理整个 mongodump 命令。这是我到目前为止：

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")

dumpCmd.Stdout = os.Stdout

uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))

result, err := uploader.Upload(&s3manager.UploadInput{

    Body:   dumpCmd.Stdout,

    Bucket: aws.String("myBucket"),

    Key:    aws.String("myKey"),

})

if err != nil {

    log.Fatalln("Failed to upload", err)

}

log.Println("Successfully uploaded to", result.Location)

不幸的是 dumpCmd.Stdout 是作者，而不是读者，我不知道如何将作者的输出通过管道传输到读者。

使用管道后，我现在收到一个新错误，这可能属于一个新问题：

dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")

body, err := dumpCmd.StdoutPipe()

if err != nil {

    // handle error

}

if err := dumpCmd.Start(); err != nil {

    // handle error

}

uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))

result, err := uploader.Upload(&s3manager.UploadInput{

    Body:   body,

    Bucket: aws.String("myBucket"),

    Key:    aws.String("myKey"),

})

if err != nil {

    log.Fatalln("Failed to upload", err)

}

if err := dumpCmd.Wait(); err != nil {

    // handle error

}

log.Println("Successfully uploaded to", result.Location)

错误：

2016/03/10 12:39:18 Failed to upload MultipartUpload: upload multipart failed

    upload id: QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1

caused by: RequestError: send request failed

caused by: Put https://myBucket/myKey?partNumber=1&uploadId=QOWW4jBHH4PKjs1Tloc8dlCTtFN94vDHIJIWJChsrjxLZggScZbRUhM4FU9V.xOnIg9uYnBWqOA1x1xqStfA1p8vdAOHNyUp4gOO5b1gbuXvUitQyLdfFhKg9MnyxsV1: read |0: illegal seek

exit status 1

Answer 1

使用管道将命令的输出连接到上传的输入。这里的棘手问题是上传者在 body上查找，但管道不支持查找。为了避免这种情况，在管道周围创建了一个包装器，以对上传者隐藏 Seek 方法。dumpCmd := exec.Command("mongodump", "--host", "<host>", "--port", "<port>", "--archive")body, err := dumpCmd.StdoutPipe()if err != nil {&nbsp; &nbsp; // handle error}if err := dumpCmd.Start(); err != nil {&nbsp; &nbsp; // handle error}// Wrap the pipe to hide the seek methods from the uploaderbodyWrap := struct {&nbsp; &nbsp; io.Reader}{body}uploader := s3manager.NewUploader(session.New(&aws.Config{Region: aws.String("us-east-1")}))result, err := uploader.Upload(&s3manager.UploadInput{&nbsp; &nbsp; Body:&nbsp; &nbsp;bodyWrap,&nbsp; &nbsp; Bucket: aws.String("net-openwhere-mongodb-snapshots-dev"),&nbsp; &nbsp; Key:&nbsp; &nbsp; aws.String("myKey"),})if err != nil {&nbsp; &nbsp; log.Fatalln("Failed to upload", err)}if err := dumpCmd.Wait(); err != nil {&nbsp; &nbsp;// handle error}log.Println("Successfully uploaded to", result.Location)

Answer 2

您可以使用io.Copy方法 (&nbsp;godoc&nbsp;)，其定义如下：功能复制func Copy(dst Writer, src Reader)（写入int64，err错误）将副本从 src 复制到 dst，直到在 src 上达到 EOF 或发生错误。它返回复制的字节数和复制时遇到的第一个错误（如果有）。成功的复制返回 err == nil，而不是 err == EOF。因为 Copy 被定义为从 src 读取直到 EOF，所以它不会将读取的 EOF 视为要报告的错误。如果 src 实现了 WriterTo 接口，则通过调用 src.WriteTo(dst) 来实现副本。否则，如果 dst 实现了 ReaderFrom 接口，则通过调用 dst.ReadFrom(src) 来实现副本。