Tukukkk

I am having trouble deploying a fargate cluster, and it is failing on the docker pull image with error "CannotPullContainerError". I am creating the stack with cloudformation, which is not optional, and it creates the full stack, but fails when trying to start the task based on the above error.

I have attached the cloudformation stack file which might highlight the problem, and I have doubled checked that the subnet has a route to nat(below). I also ssh'ed into an instance in the same subnet which was able to route externally. I am wondering if i have not correctly placed the pieces required i.e the service + loadbalancer are in the private subnet, or should i not be placing the internal lb in the same subnet???

This subnet is the one that currently has the placement but all 3 in the file have the same nat settings.

subnet routable (subnet-34b92250)
* 0.0.0.0/0 -> nat-05a00385366da527a

cheers in advance.

yaml cloudformaition script:

AWSTemplateFormatVersion: 2010-09-09

Description: Cloudformation stack for the new GRPC endpoints within existing vpc/subnets and using fargate

Parameters:

  StackName:

    Type: String

    Default: cf-core-ci-grpc

    Description: The name of the parent Fargate networking stack that you created. Necessary

  vpcId:

    Type: String

    Default: vpc-0d499a68

    Description: The name of the parent Fargate networking stack that you created. Necessary

Resources:

  CoreGrcpInstanceSecurityGroupOpenWeb:

    Type: 'AWS::EC2::SecurityGroup'

    Properties:

      GroupName: sgg-core-ci-grpc-ingress

      GroupDescription: Allow http to client host

      VpcId: !Ref vpcId

      SecurityGroupIngress:

        - IpProtocol: tcp

          FromPort: '80'

          ToPort: '80'

          CidrIp: 0.0.0.0/0

      SecurityGroupEgress:

        - IpProtocol: tcp

          FromPort: '80'

          ToPort: '80'

          CidrIp: 0.0.0.0/0

  LoadBalancer:

    Type: 'AWS::ElasticLoadBalancingV2::LoadBalancer'

    DependsOn:

      - CoreGrcpInstanceSecurityGroupOpenWeb

    Properties:

      Name: lb-core-ci-int-grpc

      Scheme: internal

      Subnets:

      # # pub

      #   - subnet-f13995a8

      #   - subnet-f13995a8

      #   - subnet-f13995a8

      # pri

        - subnet-34b92250

        - subnet-82d85af4

        - subnet-ca379b93

      LoadBalancerAttributes:

        - Key: idle_timeout.timeout_seconds

          Value: '50'

      SecurityGroups:

        - !Ref CoreGrcpInstanceSecurityGroupOpenWeb

  TargetGroup:

    Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'

    DependsOn:

      - LoadBalancer

    Properties:

      Name: tg-core-ci-grpc

      Port: 3000

      TargetType: ip

      Protocol: HTTP

      HealthCheckIntervalSeconds: 30

      HealthCheckProtocol: HTTP

      HealthCheckTimeoutSeconds: 10

      HealthyThresholdCount: 4

      Matcher:

        HttpCode: '200'

      TargetGroupAttributes:

        - Key: deregistration_delay.timeout_seconds

          Value: '20'

      UnhealthyThresholdCount: 3

      VpcId: !Ref vpcId

  LoadBalancerListener:

    Type: 'AWS::ElasticLoadBalancingV2::Listener'

    DependsOn:

      - TargetGroup

    Properties:

      DefaultActions:

        - Type: forward

          TargetGroupArn: !Ref TargetGroup

      LoadBalancerArn: !Ref LoadBalancer

      Port: 80

      Protocol: HTTP

  EcsCluster:

    Type: 'AWS::ECS::Cluster'

    DependsOn:

      - LoadBalancerListener

    Properties:

      ClusterName: ecs-core-ci-grpc

  EcsTaskRole:

    Type: 'AWS::IAM::Role'

    Properties:

      AssumeRolePolicyDocument:

        Statement:

          - Effect: Allow

            Principal:

              Service:

                # - ecs.amazonaws.com

                - ecs-tasks.amazonaws.com

            Action:

              - 'sts:AssumeRole'

      Path: /

      Policies:

        - PolicyName: iam-policy-ecs-task-core-ci-grpc

          PolicyDocument:

            Statement:

              - Effect: Allow

                Action:

                  - 'ecr:**'

                Resource: '*'

  CoreGrcpTaskDefinition:

    Type: 'AWS::ECS::TaskDefinition'

    DependsOn:

      - EcsCluster

      - EcsTaskRole

    Properties:

      NetworkMode: awsvpc

      RequiresCompatibilities:

        - FARGATE

      ExecutionRoleArn: !Ref EcsTaskRole

      Cpu: '1024'

      Memory: '2048'

      ContainerDefinitions:

        - Name: container-core-ci-grpc

          Image: 'nginx:latest'

          Cpu: '256'

          Memory: '1024'

          PortMappings:

            - ContainerPort: '80'

              HostPort: '80'

          Essential: 'true'

  EcsService:

    Type: 'AWS::ECS::Service'

    DependsOn:

      - CoreGrcpTaskDefinition

    Properties:

      Cluster: !Ref EcsCluster

      LaunchType: FARGATE

      DesiredCount: '1'

      DeploymentConfiguration:

        MaximumPercent: 150

        MinimumHealthyPercent: 0

      LoadBalancers:

        - ContainerName: container-core-ci-grpc

          ContainerPort: '80'

          TargetGroupArn: !Ref TargetGroup

      NetworkConfiguration:

        AwsvpcConfiguration:

          AssignPublicIp: DISABLED

          SecurityGroups:

            - !Ref CoreGrcpInstanceSecurityGroupOpenWeb

          Subnets:

            - subnet-34b92250

            - subnet-82d85af4

            - subnet-ca379b93

      TaskDefinition: !Ref CoreGrcpTaskDefinition

Unfortunately AWS Fargate only supports images hosted in ECR or public repositories in Docker Hub and does not support private repositories which are hosted in Docker Hub.
For more info - https://forums.aws.amazon.com/thread.jspa?threadID=268415

Even we faced the same problem using AWS Fargate couple of months back. You have only two options right now:

1. Migrate your images to Amazon ECR.




2. Use AWS Batch with custom AMI, where the custom AMI is built with Docker Hub credentials in ECS config (which we are using right now).

Do define this policy in your ECR registry and attach the IAM role with your task.

{

    "Version": "2008-10-17",

    "Statement": [

        {

            "Sid": "new statement",

            "Effect": "Allow",

            "Principal": {

                "AWS": "arn:aws:iam::99999999999:role/ecsEventsRole"

            },

            "Action": [

                "ecr:GetDownloadUrlForLayer",

                "ecr:BatchGetImage",

                "ecr:BatchCheckLayerAvailability",

                "ecr:PutImage",

                "ecr:InitiateLayerUpload",

                "ecr:UploadLayerPart",

                "ecr:CompleteLayerUpload"

            ]

        }

    ]

}