AWS S3 学习小结

AWS S3 学习小结1.首先,这个是AWS的开发资源使用文档:AWS开发文档,AWS官网-S3教程​​​​​​​2.我们可以通过AWSCli和JavaApi来操作AWS的S3,AWSCli安装教程:AWSCli安装3.Linux下连接S3前,需要先获取到AWS的IAM的accessKey和secretKey,那么获取方式是:服务->安全、身份与合规分组下的IAM->用户…

大家好,又见面了,我是你们的朋友全栈君。如果您正在找激活码,请点击查看最新教程,关注关注公众号 “全栈程序员社区” 获取激活教程,可能之前旧版本教程已经失效.最新Idea2022.1教程亲测有效,一键激活。

Jetbrains全系列IDE稳定放心使用

1.首先,这个是AWS的开发资源使用文档:AWS开发文档AWS官网 – S3教程

2.我们可以通过AWS Cli和Java Api来操作AWS 的 S3,AWS Cli安装教程:AWS Cli安装

3.Linux下连接S3前,需要先获取到AWS的IAM的accessKey 和secretKey,那么获取方式是:

服务->安全、身份与合规 分组下的 IAM->用户->安全证书->创建访问密钥

AWS S3 学习小结

然后,

AWS S3 学习小结

4.获取到了key之后,以下通过AmazonS3来操作S3:

1) 上传文件到S3

public static String uploadToS3(AmazonS3 s3, File tempFile, String remoteFileName, String bucketName) throws IOException {
        try {
            //上传文件
            s3.putObject(new PutObjectRequest(bucketName, remoteFileName, tempFile).withCannedAcl(CannedAccessControlList.PublicRead));
            //获取一个request
            GeneratePresignedUrlRequest urlRequest = new GeneratePresignedUrlRequest(
                    bucketName, remoteFileName);
            //生成公用的url
            URL url = s3.generatePresignedUrl(urlRequest);
            System.out.println("=========URL=================\n" + url + "\n============URL=============");
            return url.toString();
        } catch (AmazonServiceException ase) {
            ase.printStackTrace();
        } catch (AmazonClientException ace) {
            ace.printStackTrace();
        }
        return null;
    }

2)下载文件到本地(两种方法自己权衡吧)


import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.auth.profile.ProfileCredentialsProvider;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.GetObjectRequest;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;

import java.io.File;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import java.io.IOException;
import java.text.SimpleDateFormat;
import java.util.Date;
import com.amazonaws.ClientConfiguration;

/**
 * @description:
 * @create: 2019/4/26 16:24
 **/
public class DownloadFile {

    /**
     * @Title: downFromS3
     * @Description: 将文件下载到本地路径
     * @param @param remoteFileName 文件名
     * @param @param path 下载的路径
     * @param @throws IOException    设定文件
     * @return void    返回类型
     * @throws
     */
    public static void downFromS3(AmazonS3 s3, String remoteFileName, String path, String bucketName) throws IOException {
        try {
            GetObjectRequest request  = new GetObjectRequest(bucketName,remoteFileName);
            s3.getObject(request,new File(path));
        } catch (Exception e) {
            e.printStackTrace();
        }
    }

    public static String millToDate(long mills) {
        SimpleDateFormat sdf =new SimpleDateFormat("yyyy-MM-dd HH:mm:ss");
        return sdf.format(new Date(Long.valueOf(mills+"")));
    }

    public static void main(String[] args)
    {
        ClientConfiguration clientConfig = new ClientConfiguration();
        clientConfig.setConnectionTimeout(60*60*1000);
        clientConfig.setSocketTimeout(60*60*1000);

        // (AWS_ACCESS_KEY_ID (or AWS_ACCESS_KEY) and AWS_SECRET_KEY (or AWS_SECRET_ACCESS_KEY)
        System.setProperty("aws.accessKeyId","asd");
        System.setProperty("aws.secretKey","tTc");

        String bucket_name = "buk";
        String key_name = "testDir/files.zip";

        System.out.format("Downloading %s from S3 bucket %s... on %s\n", key_name, bucket_name, millToDate(System.currentTimeMillis()));
        final AmazonS3 s3 = AmazonS3ClientBuilder.standard().withCredentials(DefaultAWSCredentialsProviderChain.getInstance())
//.withClientConfiguration(clientConfig).withPathStyleAccessEnabled(true)
.withRegion(Regions.US_EAST_2).build();
        try {
            S3Object o = s3.getObject(bucket_name, key_name);
            S3ObjectInputStream s3is = o.getObjectContent();
            FileOutputStream fos = new FileOutputStream(new File("C:\\Users\\Desktop\\rawfiles.Zip"));
            byte[] read_buf = new byte[1024];
            int read_len = 0;
            while ((read_len = s3is.read(read_buf)) > 0) {
                fos.write(read_buf, 0, read_len);
            }
            s3is.close();
            fos.close();
        } catch (AmazonServiceException e) {
            System.err.println(e.getErrorMessage());
            System.exit(1);
        } catch (FileNotFoundException e) {
            System.err.println(e.getMessage());
            System.exit(1);
        } catch (IOException e) {
            System.err.println(e.getMessage());
            System.exit(1);
        }
        System.out.println("Done!"+ millToDate(System.currentTimeMillis()));
    }
}

3) 测试

import com.amazonaws.AmazonServiceException;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.s3.AmazonS3;
import com.amazonaws.services.s3.AmazonS3ClientBuilder;
import com.amazonaws.services.s3.model.S3Object;
import com.amazonaws.services.s3.model.S3ObjectInputStream;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Region;
import com.amazonaws.services.s3.model.Bucket;
import com.amazonaws.services.s3.transfer.TransferManager;
import utils.DownloadFile;
import utils.UploadFile;

import java.io.File;
import java.io.IOException;
import java.util.List;


public class testConnection {

    static AmazonS3         s3;
    static TransferManager tx;
    private static String AWS_ACCESS_KEY = "abc";
    private static String AWS_SECRET_KEY = "efg";
    static final String bucketName = "hij";

    static {
        ClientConfiguration config = new ClientConfiguration();
        config.setConnectionTimeout(10000);
        config.setSocketTimeout(300000);
        s3 = new AmazonS3Client(new BasicAWSCredentials(AWS_ACCESS_KEY, AWS_SECRET_KEY), config);
        Region usWest2 = Region.getRegion(Regions.US_EAST_1);
        s3.setRegion(usWest2);
    }

    public static void main(String[] args) {
        
        //枚举bucket
        List<Bucket> buckets = s3.listBuckets();
        for (Bucket bucket : buckets) {
            System.out.println("Bucket: " + bucket.getName() + bucket.getOwner());
        }

        // 上传
        try {
            UploadFile.uploadToS3(s3, new File("D:\\workplace\\xml_\\i2.xml"), "testFile", bucketName);
        } catch (IOException e) {
            e.printStackTrace();
        }

        // 下载
        try {
            DownloadFile.downFromS3(s3, "test", "C:\\Users\\Desktop\\in.xml", bucketName);
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

    // 按行读取S3的GZip
    public void test13() {
        System.setProperty("aws.accessKeyId", "abc");
        System.setProperty("aws.secretKey", "asd");
        String bucket_name = "test";
        String key_name = "data_date=20190814/inst.txt.gz";

        System.out.format("Downloading %s from S3 bucket %s...\n", key_name, bucket_name);
        final AmazonS3 s3 = AmazonS3ClientBuilder.standard().withCredentials(DefaultAWSCredentialsProviderChain.getInstance()).withRegion(Regions.US_EAST_2).build();
        try {
            S3Object o = s3.getObject(bucket_name, key_name);
            S3ObjectInputStream s3is = o.getObjectContent();
            InputStream in = new GZIPInputStream(s3is);
            BufferedReader in2=new BufferedReader(new InputStreamReader(in));
            String y="";
            while((y=in2.readLine())!=null){//一行一行读

                System.out.println(y);

            }
            s3is.close();
        } catch (AmazonServiceException e) {
            System.err.println(e.getErrorMessage());
            System.exit(1);
        } catch (FileNotFoundException e) {
            System.err.println(e.getMessage());
            System.exit(1);
        } catch (IOException e) {
            System.err.println(e.getMessage());
            System.exit(1);
        }

    }
}

5. 以下通过S3Client来操作S3:

1)  小文件流上传到S3

    public void putObject(String bucket, String key, String filePath) throws IOException {
        s3.putObject(PutObjectRequest.builder().bucket(bucket).key(key).build(),
                RequestBody.fromByteBuffer(ByteBuffer.wrap(inputStream2ByteArray(filePath))));
    }

    public void putObject(String bucket, String key, byte[] bytes) {
        s3.putObject(PutObjectRequest.builder().bucket(bucket).key(key).build(),
                RequestBody.fromByteBuffer(ByteBuffer.wrap(bytes)));
    }

    public static byte[] inputStream2ByteArray(String filePath) throws IOException {
        try (InputStream in = new FileInputStream(filePath)) {
            byte[] data = toByteArray(in);
            return data;
        }
    }

2) 分部上传文件流到S3

    /**
     * Get upload id from s3
     */
    public static String getUploadId(String bucketName, String key, Region region){
        s3 = S3Client.builder().region(region).build();
        // First create a multipart upload and get upload id
        CreateMultipartUploadRequest createMultipartUploadRequest = CreateMultipartUploadRequest.builder()
                .bucket(bucketName).key(key)
                .build();
        CreateMultipartUploadResponse response = s3.createMultipartUpload(createMultipartUploadRequest);
        String uploadId = response.uploadId();
        return uploadId;
    }
    /**
     * Multipart upload
     */
    public static void multipartUpload(String bucketName, String key,
                                       List<String> etags, List<CompletedPart> completedParts,
                                       String uploadId, byte[] cbuf, int off) {
        Region region = Region.US_EAST_2;
        s3 = S3Client.builder().region(region).build();
        //record upload end tag
        ByteBuffer bf = ByteBuffer.wrap(cbuf);
        UploadPartRequest uploadPartRequest = UploadPartRequest.builder().bucket(bucketName).key(key)
                .uploadId(uploadId)
                .partNumber(off).build();
        String etag = s3.uploadPart(uploadPartRequest, RequestBody.fromByteBuffer(bf)).eTag();
        CompletedPart part = CompletedPart.builder().partNumber(off).eTag(etag).build();
        etags.add(etag);
        completedParts.add(part);
    }

    /**
     * Complete multipart upload
     */
    public static void completeMultipartUpload(String bucketName, String key, List<CompletedPart> completedParts, String uploadId) {
        // Finally call completeMultipartUpload operation to tell S3 to merge all uploaded
        // parts and finish the multipart operation.
        CompletedMultipartUpload completedMultipartUpload = CompletedMultipartUpload.builder().parts(completedParts).build();
        CompleteMultipartUploadRequest completeMultipartUploadRequest =
                CompleteMultipartUploadRequest.builder().bucket(bucketName).key(key).uploadId(uploadId)
                        .multipartUpload(completedMultipartUpload).build();
        s3.completeMultipartUpload(completeMultipartUploadRequest);
    }

其中的先后顺序是:getUploadId -> multipartUpload … -> completeMultipartUpload

 

1. 遇到:AWS error downloading object from S3, “profile file cannot be null”,参考:https://stackoverflow.com/questions/41796355/aws-error-downloading-object-from-s3-profile-file-cannot-be-null

2. 遇到:SocketTimeoutException: Read timed out,参考:http://www.unixresources.net/faq/28195217.shtml

3. 遇到:S3的Status Code: 404 指的是 bucket 名字写错了

4. 遇到:S3的Status Code: 301,那么检查一下Region对不对。

AWS S3 学习小结

5. 遇到:Unable to unmarshall response (null). Response Code: 200, Response Text: OK… ,问题在于调用s3.getObject()的时候,本地已存在相同名字的文件了。

6. 遇到:Your socket connection to the server was not read from or written to within the timeout period… ,解决这个问题可能需要设置S3 configuration的retry policy,参考:Recurrent “Idle connection..” exception in S3Client.putObject

AWS S3 学习小结

官网的解释是:How can I troubleshoot the error

 

参考

1.AWS S3使用demo

2.S3 java SDK连接

3.命令行管理aws s3

1. https://blog.csdn.net/harryhare/article/details/80710279

2. API 示例查看文档

版权声明:本文内容由互联网用户自发贡献,该文观点仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 举报,一经查实,本站将立刻删除。

发布者:全栈程序员-用户IM,转载请注明出处:https://javaforall.cn/180099.html原文链接:https://javaforall.cn

【正版授权,激活自己账号】: Jetbrains全家桶Ide使用,1年售后保障,每天仅需1毛

【官方授权 正版激活】: 官方授权 正版激活 支持Jetbrains家族下所有IDE 使用个人JB账号...

(0)


相关推荐

  • c语言createthread函数用法,CreateThread函数「建议收藏」

    c语言createthread函数用法,CreateThread函数「建议收藏」当使用CreateProcess调用时,系统将创建一个进程和一个主线程。CreateThread将在主线程的基础上创建一个新线程,大致做例如以下步骤:1在内核对象中分配一个线程标识/句柄,可供管理,由CreateThread返回2把线程退出码置为STILL_ACTIVE。把线程挂起计数置13分配context结构4分配两页的物理存储以准备栈。保护页设置为PAGE_READWRITE。第2页设为PA…

  • mac 激活码GoLand 2021.2.3 破解(JetBrains全家桶)

    (mac 激活码GoLand 2021.2.3 破解)这是一篇idea技术相关文章,由全栈君为大家提供,主要知识点是关于2021JetBrains全家桶永久激活码的内容IntelliJ2021最新激活注册码,破解教程可免费永久激活,亲测有效,下面是详细链接哦~https://javaforall.cn/100143.html1STL5S9V8F-eyJsa…

  • Python 实现毫秒级淘宝抢购脚本

    Python 实现毫秒级淘宝抢购脚本本篇文章主要介绍了Python通过selenium实现毫秒级自动抢购的示例代码,通过扫码登录即可自动完成一系列操作,抢购时间精确至毫秒,可抢加购物车等待时间结算的,也可以抢聚划算的商品。博主不提供任何服务器端程序,也不提供任何收费抢购软件。该文章仅作为学习selenium框架的一个示例代码。该思路可运用到其他任何网站,京东,天猫,淘宝均可使用,且不属于外挂或者软件之类,只属于一个自动化点击工…

  • RSA算法简述

    RSA算法简述52tangzongb+TR/9sbreGJhbKT5U5rQCTUebfRngB0uhNMnvMClf0f/IpPTsM5+7zWJyT9drzVKzV4oR0J8lyMSWepKvv3BR/3Ab6vC8dmo7NDbzuDtLaDLYhYG+bggQNVvuA5C3TolntxdL4+mGZwfd86WoznJM+Y5TO/0C5MSxvaAMTMZuga7yyBKTH4Wl+7GFHDDZqAXmvPHW/Dz0i45vlToz/+E/RnznY5dBhkw3nnNoNsJIutAUDm4T18J

  • RemObjects的特征

    RemObjects的特征RemObjectsSDK‘Vinci’是成功的跨平台远程框架的第五个版本,它允许用户方便地创建能够在面向对象模式中的跨网络通信的客户端以及服务器应用程序。使用强大且灵活的组件库、以及用于定义以及测试服务的高级工具,RemObjectsSDK提供了使用RAD进行分布式开发的方法。RemObjectsSDK的特征以下列表概述了RemObjectsSDK的核心特征,这些特征

  • java中使用uuid函数_java生成随机数

    java中使用uuid函数_java生成随机数什么是UUID?UUID是UniversallyUniqueIdentifier的缩写,它是在一定的范围内(从特定的名字空间到全球)唯一的机器生成的标识符。UUID具有以下涵义:经由一定的算法机器生成为了保证UUID的唯一性,规范定义了包括网卡MAC地址、时间戳、名字空间(Namespace)、随机或伪随机数、时序等元素,以及从这些元素生成UUID的算法。UUID的复杂特性在保证了其唯一性的同…

发表回复

您的电子邮箱地址不会被公开。

关注全栈程序员社区公众号