Blog

AWS S3 security best practices - Part 1

Jan 22, 2019

Security camera with colouring

AWS S3 has been the subject of some of the largest data breaches in recent history. From the leak of nearly 200m US voters' voting preferences to the exposure of 48m personal data records from private social media accounts to the 100GB treasure trove of classified information left exposed by the Pentagon, there's been some massive AWS breaches recently.

You may be thinking of a sophisticated Mission Impossible-esque break in, involving advanced state actors and clandestine cyber attacks. In actual fact, the front door was left wide open. In technical speak this is a “Security Misconfiguration”, and the prevalence of these types of issues has seen OWASP list it as #6 in their most recent top 10 application security risks.

 

S3 itself isn’t insecure, it just depends how you use it.

AWS responsibility diagram

Diagram outlining the AWS shared responsibility model. 

To help mitigate against breaches like the ones we've already mentioned, cloud service providers offer a “Shared Responsibility Model” to manage cloud security. They are responsible for security of the cloud, whereas cloud customers are responsible for security in the cloud. This includes configuring cloud services correctly to avoid security disasters.

The diagram above outlines the general responsibilities of AWS and the customer when it comes to security and securing your data. It's useful to familiarise yourself with these responsibilities so that you know which areas are within your control and which ones aren't.

This blog post is part of a new two-parter from the SysOps team that will explore techniques for securing your data on AWS, including tips and best practice. In this blog post we will:

  • Threat model common security risks for S3

  • Cover core security practices for S3

  • Discuss proactive measures

In a follow up post, we’ll investigate how to securely provide access to S3 for applications and give examples of the infrastructure setup using CloudFormation.

 

S3 refresher.

 AWS S3 is a fantastically versatile data storage service, offering world class scalability, data availability, and performance. It is built to store and retrieve any amount of data from anywhere, for any purpose. Every time you settle in to stream your favourite Netflix series, S3 is the underlying service responsible for shuttling the video across to your devices. Storing logs, IoT data, backups & uploads are just a few examples of how S3 can be used.

A quick note on terminology:

  • Object - A file and optionally any metadata & permissions that describes that file. All data in S3 is stored as objects. Objects are referenced by their key, which is unique per bucket.

  • Bucket - The containers for objects. Buckets sit in specific geographical regions, and you can have one or more buckets. Bucket names are unique globally.

 

Security risks.

 Aside from the risk of exposing access to private data, there are a few other risks to consider. What if an attacker is able to delete objects in the bucket? If the proper backup procedures and disaster recovery solutions are not in place, this can leave companies in an irrecoverable position. Unauthorised writes are another concern. This can lead to a new type of cloud specific attacks - Denial of Wallet. There is no limit to how much data you can store, and you pay per GB. Attackers can flood your bucket, and you have to pick up the tab.

Craftier yet, attackers could upload illegal content which you may be liable for. Or they could encrypt the files stored and hold you for ransom if you didn’t have a backup of the data. S3 can even be used as an attack vector for injection attacks. If an attacker had write access to a bucket hosting a website, they could easily upload malicious JavaScript to attack users. Or if objects in the bucket were later processed by a Lambda function, they could intentionally craft a malicious object.

 

Basic threat modelling.

Before using S3, it is important to map out what you are intending on storing in S3 and who needs to access it. This will allow you to consider the different threats and their severity, and guide you towards more effective defensive measures.

  1. Write a list of the different types of data you intend to store, or are already storing in S3.

  2. Alongside each type, grade the sensitivity of the data from 1 to 5. 1 meaning the data can be publicly available without any negative consequences, and 5 meaning the data is very sensitive and could have catastrophic consequences if exposed or tampered with. The CIA (Confidentiality, Integrity & Availability) triad is useful to consider when grading.

  3. Additionally note down who needs access to the data, whether it is people or programs / systems. Do they need create, read, update or delete access?

When you have a list, remember that everything doesn’t need to exist in 1 bucket. You can segregate your data by storing them in separate buckets, with different security profiles. You wouldn’t store backups of your web server logs alongside public photo uploads for example.

Also consider whether the data always has to exist in S3 - if the data doesn’t exist it can’t be compromised. Set up lifecycle rules for your data to automatically delete data that is no longer needed.

Follow the principle of least privilege when considering who needs access to the data. Only give the bare minimum permissions necessary. This will map nicely to ACLs & IAM policies.

 

Core security practices.

 

Bucket policies, ACLs & IAM

By default, buckets and objects are private when created. You can use a combination of bucket policies, ACLs and IAM policies to give access to the right entities. However, this can get complex as you then need to consider how all of these permissions interact and how Amazon evaluates them all at once.


A better approach is to keep it simple. Rather than mixing public & private objects in the same bucket, and having to carefully manage the associated object ACLs, create one public bucket that has a bucket policy giving any principal (*) GET access to objects in that bucket.

 


{
  "Effect": "Allow",
  "Principal": "*",
  "Action": "s3:GetObject",
  "Resource": "arn:aws:s3:::publicbucketnamegoeshere/*"
}

 

You can then create one or more private buckets to store your private objects. You can then use the “Block Public Access” option mentioned later to prevent all public access. If you need to give specific users or applications access, you can use IAM policies to give a specific user or role the relevant permissions. Once again, follow the principle of least privilege. Refer to the AWS documentation explaining the different S3 actions and what they mean. Generally you only want to fallback to per-object ACLs if the situation truly requires it.

 

Encryption.

Enable encryption at rest and during transit to protect your data. You can easily set up AWS to encrypt objects server-side before they get stored in S3, either using default AWS-managed S3 keys, or keys you have created in the Key Management Service. To use the default keys, you can add the BucketEncryption property to the CloudFormation definition of your bucket as so:


  BucketEncryption:
    ServerSideEncryptionConfiguration:
      - ServerSideEncryptionByDefault:
          SSEAlgorithm: AES256

You can also enforce encryption during transit by mandating that HTTPS be used for all bucket operations. Add the following statement to your bucket policy:

 


{
  "Action": "s3:*",
  "Effect": "Deny",
  "Principal": "*",
  "Resource": "arn:aws:s3:::bucketname/*",
  "Condition": {
    "Bool": { "aws:SecureTransport": false }
  }
}

 

Logging & alerts.

Logging is a key area of all security best practices. Not only is it a requirement for auditors, but logs can also be integrated with alerting systems to immediately highlight suspicious activity.

 

CloudTrail

CloudTrail is a service that maintains an audit trail of events occurring across AWS services within your AWS account. There are broadly 2 types of events: management events, and data events. Data events are much more granular and are disabled by default.

In the context of S3, management events cover events such as creating, deleting and updating S3 buckets, whereas data events include API calls made on objects within the buckets, such as GetObject, PutObject and DeleteObject. Data events include a cost of $0.10 per 100,000 events.

You can create specific trails to monitor resources in a target region, or globally. These trails will write logs to an S3 bucket. If you are running multiple AWS accounts, you may want to deliver these logs to a bucket in your management account.

This follows best practice to have a separate high level management account, with environments being sub accounts of this. This allows you to leverage service control policies to restrict the ability of sub-accounts to disable CloudTrail logs, which is often the first step an attacker will perform in order to cover their tracks.

 

CloudWatch and alerting

Having CloudTrail set up to log the S3 events to a logging bucket is great, and often this is all that is needed by 3rd party monitoring solutions such as Splunk or Alert Logic. However, depending on the monitoring solution or if you want to have control over alerting and self-healing, you need to be using CloudWatch. CloudWatch also offers immediate logging of events, compared to the delayed delivery of log files to S3.

You can set up CloudTrail to create log streams within a CloudWatch log group, which CloudTrail will log events to. Having CloudTrail events in CloudWatch is very powerful, as this then allows you to set up metric filters to enable alerting based on suspicious activity and even run Lambdas as a result of alarms triggering to customise your response actions.

You can create metric filters based upon the S3 API event names to target specific activities you deem of interest to your organisation. One example would be to run a Lambda function to make a bucket private if a PutBucketPolicy event was detected that made it public - although AWS now offers this functionality which is discussed later.

 

Lifecycle policies

Once you no longer need your data, stop storing it. This ensures that the data can no longer be accessed by attackers and results in cost savings. To learn more about lifecycle policies, head to Stephen Jefferson’s blog post about S3 lifecycle policies.

As a brief CloudFormation example, the following LifecycleConfiguration property could be added to a S3 bucket resource to automatically delete files in the “temp” folder after 7 days:


  LifecycleConfiguration:
    Rules:
      - Status: Enabled
        Id: DeleteTempAfter7Days
        ExpirationInDays: 7
        Prefix: temp/

 

 

Proactive security.

Preventative measures are important, but you can’t prevent every threat. This is where proactive monitoring comes into play to close the gaps. AWS provides a few tools to help you do this effectively, we've covered the most important ones below.

 

S3 Block Public Access

AWS have taken the step to automate functionality which previously required a combination of CloudTrail, CloudWatch and Lambda. The new block public access settings allows the proactive blocking of any attempts to make a bucket public, or specifying a public ACL for objects in the bucket. This prevents those fat-finger moments where developers accidentally make buckets or objects public. This setting can be easily enabled in the AWS console, or in CloudFormation by specifying the PublicAccessBlockConfiguration property on your S3 bucket resource:


  PublicAccessBlockConfiguration:
    BlockPublicAcls: true
    BlockPublicPolicy: true
    IgnorePublicAcls: true
    RestrictPublicBuckets: true

 

 

AWS Trusted Advisor

Trusted Advisor is a built in AWS service that analyses your AWS resources and gives recommendations relating to 5 categories, one of which being security. Since February 2018, AWS alerts you for free if your S3 buckets are publicly accessible.

 

AWS CloudFormation Drift Detection

If you have followed an infrastructure as code approach and defined your infrastructure, along with the security best practices, using CloudFormation then Drift Detection can be used to detect if any of your resources have been manually modified from what your templates define. This then allows you to either manually adjust the resources to restore them to the configuration defined in your template, or to re-run the CloudFormation stack.

 

Third-party AWS security tools

It's not just Amazon who provide crucial tools to keep your data safe. AWS has become so ubiquitous in recent years that a whole cottage industry of tools has popped up to support your cloud security efforts. They can save you enormous amounts of time and keep your data secure. Some of our favourites include:

 

Security Monkey

Security Monkey is a tool developed by Netflix that monitors AWS for policy changes and alerts on insecure configurations. Security Monkey has a set of audits for S3 to ensure certain best practices are in place. A great benefit of Security Monkey is that it also supports Google Cloud Platform, so if you’re operating a hybrid cloud you’re in luck.

 

Cloud Custodian

Cloud Custodian is a tool developed by Capital One to manage cloud resources according to defined policies. In other words, once you have settled on a S3 security strategy and have identified best practices, you can use Cloud Custodian to scan your resources and ensure that these practices are being met. If they aren’t, you have a wealth of options to send alerts or even automatically enforce those policies.

 

Cloud Mapper

Cloud Mapper is a cloud visualisation and audit tool created by Duo Security. Similar to Security Monkey, it can perform audits on your AWS resources, including S3 buckets to check for common misconfigurations. Cloud Mapper shines when you want a visual representation of your infrastructure, which can often lead to the identification of further issues.

 

Providing Application Access to S3.

Now that your S3 buckets are locked down like Fort Knox, what happens when we want to give applications selective access to S3?

That will be the topic of a follow up post detailing the different methods of achieving this, along with pros & cons, based upon your usage requirements. Keep your eyes peeled on the blog for part two.

 


 

Have we missed anything? Have your own tips for Amazon AWS S3 security best practices? Let us know in the comments below or tweet us at @hedgehoglab.

 


Want to save money on your cloud storage costs? Take a look at our blog post on optimising storage costs in AWS S3.

Macbook with code

Optimise your cloud storage spend

 

Want to work with us?

Get in touch