OIDC Forever, IAM Credentials Never!

Understanding why OIDC is superior to IAM credentials

So the standard process has typically been to: create an IAM user with programmatic keys, give that user permissions of least privilege, and then upload the IAM credential keys to the service provider that will be making changes to your AWS infrastructure (again in my case Terraform Cloud and CircleCI) on your behalf.

When you really think about it, storing your IAM credentials in any service provider has its limitations. I like to think of providing IAM credentials to a service similar to giving my credit card details to a service (e.g. CircleCI). I trust CircleCI to not do anything nefarious with my details, but if my card details are stored in the service and that service for whatever reason becomes compromised, that credit card can be used by anyone (or any service) that has the details. This is no different from IAM credentials and provides us with a good reason as to why you always want to pair down the permissions.

side note: please, please don’t create an IAM user with programmatic keys that has AdministratorAccess permissions to your AWS Account that you then upload those IAM credentials to these third-party services. You are basically giving anyone or any service the ability to create anything (well…within service quota limits) within your AWS account by doing so. You personally don’t want that or want to deal with an unexpected large AWS bill shortly after! 😅

Continuing the credit card analogy; so instead of giving our credit card details to a service, what if there is a way to give only card authorization to that one service? Well, that sounds more like OIDC! The benefit is that you are only authorizing/allowing the service to use your credit card, meaning that there are no credit card details to be leaked, only card authorization details. If the card authorization details were somehow leaked, these are actually kind of worthless, as only that particular service is authorized to use them, not some other random third party! 🎉

I’m confident there will come a day soon when some hacker will figure out how to spoof the service that has the OIDC Identity provider authorization permission (e.g. a hacker spoofing CircleCI can then access your AWS infrastructure based on the policies attached to the OIDC provider), but for now this setup seems a whole lot more secure than storing your IAM credentials directly at the service and having to remember to rotate your access keys every so often. Further, if the IAM credentials are leaked, you really have to rotate them quickly or risk your AWS account/services being compromised.

How to setup an OIDC Identity Provider with CircleCI in AWS

1. Start by logging into CircleCI and then go to Organization Settings > then copy your organization ID
2. Log into the AWS console, and head to IAM.
3. Go to Identity Providers (under Access Management) and click “Add Provider”
4. Choose “OpenID Connect”
5. In Provider URL enter https://oidc.circleci.com/org/<INSERT ORGANIZATION ID HERE> (e.g. https://oidc.circleci.com/org/th1s-i5-n0t-a-0rg1d) and then click “Get Thumbprint”
6. In Audience, enter the same Organization ID
7. Click “Add Provider” and the provider should be created!
8. Next, go to Roles (under Access Management)
9. Click “Create Role” and select “Web Identity”
10. In Identity Provider select the CircleCI identity provider that was just created.
11. In Audience, select the Organization ID and click Next to add permissions
12. In Permissions, you will be selecting or creating policies based on what your CircleCI pipelines should have permission to do. This will vary case by case. For example, my CICD pipelines need only access to S3 and CloudFront services to publish my blog posts.
13. Click Create Role and when complete, copy this Role’s ARN, you are good from the AWS perspective!

Using OIDC as your AWS credentials in CircleCI

Before making any changes to your CircleCI pipeline, as per the, OpenID Connect Tokens doc, you will need to
create a context for your job, which can be done using these steps.

Once you have a context created
and added to the workflow section of your pipeline job, you can now edit your
CircleCI pipeline to include the OpenID Connect token to utilize our newly
created IAM role. For example, using these directions, my .circleci/config.yml file
was changed from this:

version: 2.1
jobs:
  www:
    docker:
      - image: paulmarsicloud/hexo-aws:2.1
    working_directory: ~/hexo-cloudonmymindblog
    steps:
      - checkout
      - run:
          name: Generate static website
          command: hexo generate
      - run:
          name: Push to S3 bucket
          command: cd public/ && aws s3 sync . s3://www.thecloudonmymind.com
      - run:
          name: Invalidate Cloudfront
          command: aws cloudfront create-invalidation --distribution-id <REDACTED> --paths "/*" --no-cli-pager
workflows:
    version: 2
    build_and_deploy:
      jobs:
        - www:
            context:
              - cloudonmymind-web

to this:

version: 2.1
jobs:
  www:
    docker:
      - image: paulmarsicloud/hexo-aws:2.1
    environment:
      AWS_DEFAULT_REGION: <ENTER YOUR REGION HERE>
      AWS_ROLE_ARN: <ENTER YOUR IAM ROLE ARN HERE>
    working_directory: ~/hexo-cloudonmymindblog
    steps:
      - checkout
      - run:
          name: Generate static website
          command: hexo generate
      - run:
          name: authenticate-and-interact, Push to S3 bucket, Invalidate CloudFront
          command: |
            # use the OpenID Connect token to obtain AWS credentials
            read -r AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN <<< 
              $(aws sts assume-role-with-web-identity 
               --role-arn ${AWS_ROLE_ARN} 
               --role-session-name "CircleCI-${CIRCLE_WORKFLOW_ID}-${CIRCLE_JOB}" 
               --web-identity-token $CIRCLE_OIDC_TOKEN 
               --duration-seconds 3600 
               --query 'Credentials.[AccessKeyId,SecretAccessKey,SessionToken]' 
               --output text)
            export AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_SESSION_TOKEN
            # interact with AWS
            aws sts get-caller-identity --no-cli-pager
            cd public/ && aws s3 sync . s3://www.thecloudonmymind.com
            aws cloudfront create-invalidation --distribution-id <REDACTED> --paths "/*" --no-cli-pager
workflows:
    version: 2
    build_and_deploy:
      jobs:
        - www:
            context:
              - cloudonmymind-web

So for the eagle-eyed reader, you’ll notice a few interesting pieces. Firstly, the addition of --no-cli-pager is necessary probably only for the docker image used paulmarsicloud/hexo-aws:2.1. This is a custom docker image I created that has both hexo and the aws-cli installed, as this blog/site uses the Hexo platform. By adding the --no-cli-pager option, I no longer receive a Unable to redirect output to pager. error 👍

Secondly, I had to combine the Push to S3 bucket and Invalidate Cloudfront steps into one during testing, as it seems that the read and export commands need to be run per step. I tried having just a singular authenticate-and-interact step, but subsequent steps that ran aws cli commands would fail with a Unable to locate credentials. You can configure credentials by running "aws configure" error. I will need to sit down and spend some further time testing out and potentially refactoring my CircleCI steps and pipeline as I am sure there is a way to somehow cache or pass the credentials from one step to another, rather than having to write and execute those read and export commands each time you want to run an aws-cli command. When I have the answer, ya’ll will be the first to know!

In a way, it makes more sense to have these S3 and CloudFront AWS commands run in one step, even though for readability purposes, I personally like to have a “one command per step” layout for my CI/CD pipelines. It’s truly a “6 in one, half a dozen in the other” kinda deal though.

Here’s a link to the gist of my final updated .circleci/config.yml if you need it.