Connecting Spring Boot to AWS DocumentDB with Secure Credentials

Most of the dozens of code snippets in Spring Boot books and articles showing how to connect Spring Boot to MongoDB aren't secure, embedding usernames and passwords directly into the configuration. Many times those explanations will indicate that "you shouldn't do it this way", and imply (hands waving) that it's easy to do it the correct way. But it's not straightforward to do it right, especially when using Amazon Web Services DocumentDB which supposedly is compatible with MongoDB (but is slightly incompatible, in tricky ways). I'll show you how to configure Spring Boot using ECS Fargate, connected to DocumentDB, with username and password securely stored in AWS Secrets Manager.

In this tutorial I'll assume you already know how to deploy a Spring Boot application in AWS ECS Fargate using a Docker image. You should already have a vague idea that you can set environment variables in the task definition to be passed to the container. I'll show you exactly what those settings should be, defining everything declaratively using CloudFormation, which I also assume you're familiar with. Your Spring Boot project should include spring-boot-starter-data-mongodb.

Username and Password Management

Let's jump straight into the important part: management of the database credentials. You shouldn't put those in a configuration. Just don't. You want to manage them somewhere—in a secure "vault" that not only protects values but only allows access only to the programs that need them. AWS comes with Secrets Manager which can serve this purpose. With CloudFormation we'll define a "secret" using AWS::SecretsManager::Secret, which will contain our database username and password:

  DbCredentials:
    Type: AWS::SecretsManager::Secret
    Properties:
      Name: db-creds
      GenerateSecretString:
        SecretStringTemplate: '{"username": "somebody"}'
        GenerateStringKey: "password"
        PasswordLength: 99
        ExcludeCharacters: "/\"@%:?#[]"

This tells Secrets Manager to generate a password with a secret internal lookup key of password for us, alongside your designated username with a secret internal lookup key of username. You'll be able to see the generated password in the console after deplying the CloudFormation stack, but you won't need to. You'll never even need to know what the password is. We might as well make a password as long as possible.

Already there are lots of gotchas that you would never know until you start trying to make work.

Database and Credentials Attachment

When you define the DocumentDB cluster using AWS::DocDB::DBCluster, you'll need to reference the username and password sub-secrets you created above, using a special resolve:secretsmanager string that tells AWS to look up the secrets from Secrets Manager:

  DbCluster:
    Type: AWS::DocDB::DBCluster
    DependsOn: DbCredentials
    Properties: 
      DBClusterIdentifier: "db-cluster"
      DBSubnetGroupName: …
      EngineVersion: 4.0.0 # current latest
      …
      MasterUsername: !Sub "{{resolve:secretsmanager:${DbCredentials}:SecretString:username}}"
      MasterUserPassword: !Sub "{{resolve:secretsmanager:${DbCredentials}:SecretString:password}}"

Database Parameter Group to Disable TLS

You can't use the default.docdb4.0 DocumentDB parameter group, which is present automatically, because it has TLS enabled and Spring Boot Data MongoDB does not support SSL/TLS by default. The best thing to do would be to enable SSL/TLS in Spring Boot, but I don't immediately know how to do that in a fully automated way in AWS using CloudFormation. So for the meantime here is how you can turn off SSL/TLS on your DocumentDB cluster.

Create a custom AWS::DocDB::DBClusterParameterGroup.

  DbClusterNoTls:
    Type: AWS::DocDB::DBClusterParameterGroup
    Properties: 
      Name: docdb-4.0-no-tls
      Family: docdb4.0
      Parameters:
        # whatever other parameters you want to set here
        tls: disabled

Then reference that in your AWS::DocDB::DBCluster

  DbCluster:
    Type: AWS::DocDB::DBCluster
    Properties: 
      DBClusterIdentifier: db-cluster
      EngineVersion: 4.0.0
      DBClusterParameterGroupName: !Ref DbClusterNoTls
      …

See my Stack Overflow question and answer Spring Boot ECS service cannot connect to DocumentDB cluster for more details.

Custom Excution Role for ECS

Ah, not so fast. You're not yet ready to set up the actual ECS Fargate container yet, because ECS needs permissions to access Secrets Manager at runtime. You might have set up an ecsTaskExecutionRole (or AWS might have set up one for you automatically) containing the AWS managed AmazonECSTaskExecutionRolePolicy, which allows these actions:

Notice that the secretsmanager:GetSecretValue action is not among them. You'll need to define a separate role that includes secretsmanager:GetSecretValue as explained at IAM policy examples for secrets in AWS Secrets Manager. You could go ahead and add secretsmanager:GetSecretValue to some global custom role, but you probably don't want to let all services have access to the secrets meant for only the other services (i.e. PoLP). It's probably best to define a separate role for each service needing secrets injected, using AWS::IAM::Role. The following example assumes that you're going to be injecting the DbCredentials referenced above.

  ServiceTaskExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      RoleName: !Sub "my-service-${AWS::Region}-taskExecRole"
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement: 
          - Effect: Allow
            Principal: 
              Service: 
                - ecs-tasks.amazonaws.com
            Action: 
              - sts:AssumeRole
      Policies:
        - PolicyName: !Sub "my-service-${AWS::Region}-secret-access-policy"
          PolicyDocument:
            Version: 2012-10-17
            Statement: 
            - Effect: Allow
              Action:
                - secretsmanager:GetSecretValue
              Resource:
                - !Ref DbCredentials
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy

This sets an ECS task execution role shown at the beginning of this answer, with permissions to access only the DbCredentials secret, in addition to executing ECS tasks. You can reference this in your ECS Fargate task definition AWS::ECS::TaskDefinition:

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub "my-service-task"
      ExecutionRoleArn: !GetAtt ServiceTaskExecutionRole.Arn
      …

See my Stack Overflow question and answer AWS region considerations for creating ECS ecsTaskExecutionRole via CloudFormation for more background and discussion on creating task execution roles.

Injecting Credentials into ECS Fargate

You're now ready to inject the credentials into the environment variables of the ECS Fargate task definition's container definition using AWS::ECS::TaskDefinition, but that gets even tricker. The "isn't this easy?" books and articles will tell you just to set the SPRING_DATA_MONGODB_HOST and SPRING_DATA_MONGODB_USERNAME environment variables. You can inject the DB cluster in the task container definition Environment section, along with the username and password from Secrets Manager in the Secrets section. (The Secrets section creates environment variables, it's just that the values are retrieved from Secrets Manager.)

  TaskDefinition:
    Type: AWS::ECS::TaskDefinition
    Properties:
      Family: !Sub "my-service-task"
      ExecutionRoleArn: !GetAtt ServiceTaskExecutionRole.Arn
      RequiresCompatibilities:
        - FARGATE
      NetworkMode: awsvpc
      ContainerDefinitions:
          Environment:
            - Name: SPRING_DATA_MONGODB_HOST
              Value: !GetAtt DbCluster.Endpoint
            - Name: SPRING_DATA_MONGODB_PORT
              Value: !GetAtt DbCluster.Port
          Secrets:
            - Name: SPRING_DATA_MONGODB_USERNAME
              ValueFrom: !Sub "${DbCredentials}:username::"
            - Name: SPRING_DATA_MONGODB_PASSWORD
              ValueFrom: !Sub "${DbCredentials}:password::"
      …

And here are where the problems start rolling in. It turns out that DocumentDB doesn't support retryable writes. So if you spin up an ECS task with just the configuration above, it will say something like:

com.mongodb.MongoCommandException: Command failed with error 301: 'Retryable writes are not supported' on server my-db-cluster.cluster-xxxxxxxxxxxx.us-east-1.docdb.amazonaws.com:27017. The full response is {"ok": 0.0, "code": 301, "errmsg": "Retryable writes are not supported", "operationTime": {"$timestamp": {"t": xxxxxxxxxx, "i": 1}}}

The AWS documentation and various Stack Overflow answers say to set retryWrites=False by switching to the full SPRING_DATA_MONGODB_URI environment variable with the connection URI format like this:

mongodb://<username>:<password>@my-db-cluster.cluster-xxxxxxxxxxxx.us-east-1.docdb.amazonaws.com:27017/database?retryWrites=False

That would be something like this in my task container definition in CloudFormation:

            - Name: SPRING_DATA_MONGODB_URI
              Value: !Sub "mongodb://${DbCluster.Endpoint}:${DbCluster.Port}/?retryWrites=False"

Unfortunateyl defining the connection URI overrides the username and password environment variables. Whenever you specify the individual Spring Boot Data MongoDB environment variables such as SPRING_DATA_MONGODB_PASSWORD (which will be accessible internally via configuration properties such as spring.data.mongodb.password), Spring will construct the connection URL dynamically. But if you specify the connection URI environment variable SPRING_DATA_MONGODB_URI, Spring will ignore the individual components, including the username and password ECS has injected from Secrets Manager. CloudFormation has no way to inject the secrets directly into a URI pattern.

But there is a workaround to force Spring to pull in the secret credentials you've injected even via the connection URI. Spring Boot provides a facility for interpolating configuration properties dynamically using property placeholders, similar to CloudFormation references except that these references are replaced at runtime. So it's possible to go ahead and inject the username and password, and then use placeholder in the actual connection URI, like this:

          Environment:
            - Name: SPRING_DATA_MONGODB_HOST
              Value: !GetAtt DbCluster.Endpoint
            - Name: SPRING_DATA_MONGODB_PORT
              Value: !GetAtt DbCluster.Port
            - Name: SPRING_DATA_MONGODB_URI
              Value: mongodb://${spring.data.mongodb.username}:${spring.data.mongodb.passsword}@${spring.data.mongodb.host}:${spring.data.mongodb.port}/?retryWrites=False
          Secrets:
            - Name: SPRING_DATA_MONGODB_USERNAME
              ValueFrom: !Sub "${DbCredentials}:username::"
            - Name: SPRING_DATA_MONGODB_PASSWORD
              ValueFrom: !Sub "${DbCredentials}:password::"

Based on the recommendations the AWS DocumentDB console provides, you might even want to use this extended connection URI:

mongodb://${spring.data.mongodb.username}:${spring.data.mongodb.password}@${spring.data.mongodb.host}:${spring.data.mongodb.port}/?replicaSet=rs0&readPreference=secondaryPreferred&retryWrites=false

Now username and password are injected into environment variables at deploy time, and Spring Boot itself will plug those values into the connection URI for you at runtime. Finally you understand why we needed to restrict the password characters at the beginning of this exercise. You can read more about this issue at my Stack Overflow question and answer Disable Spring Boot Data MongoDB retryable writes in AWS ECS Fargate with CloudFormation.