aws alb ingress controller

ingress►k8s , , Comments

AWS-alb-ingress-controller

项目：https://github.com/kubernetes-sigs/aws-load-balancer-controller

aws-alb-ingress-controller

  • 步骤1: controller 监听api-services的event事件,当发现 Ingress 资源满足要求，则将开始创建 AWS 资源。
  • 步骤2: 为 Ingress 资源创建 ALB
  • 步骤3: 为 Ingress 资源中指定的每个后端创建目标组
  • 步骤4: 为 Ingress 资源注释中指定的每个端口创建侦听器
  • 步骤5: 为 Ingress 资源中指定的每个路径创建规则

注意

  • ingress-controller 需要有权限访问aws创建资源，具体权限可参考：iam-policy.json。本例子中直接对eks worknode role进行授权。理论上也可以进行OIDC接入aws iam针对于pod级别进行权限管理（未测试）。
  • 确保ingress annotation的资源存在，否则创建会失败，具体可看pod日志。
  • service 的type为 nodeport。

Install controller

rbac+sa+ns.yaml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
apiVersion: v1
kind: Namespace
metadata:
  name: alb-ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
rules:
  - apiGroups:
      - ""
      - extensions
    resources:
      - configmaps
      - endpoints
      - events
      - ingresses
      - ingresses/status
      - services
    verbs:
      - create
      - get
      - list
      - update
      - watch
      - patch
  - apiGroups:
      - ""
      - extensions
    resources:
      - nodes
      - pods
      - secrets
      - services
      - namespaces
    verbs:
      - get
      - list
      - watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: alb-ingress-controller
subjects:
  - kind: ServiceAccount
    name: alb-ingress
    namespace: alb-ingress-controller
---
apiVersion: v1
kind: ServiceAccount
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress
  namespace: alb-ingress-controller
controller.yaml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: alb-ingress-controller
  name: alb-ingress-controller
  # Namespace the ALB Ingress Controller should run in. Does not impact which
  # namespaces it's able to resolve ingress resource for. For limiting ingress
  # namespace scope, see --watch-namespace.
  namespace: alb-ingress-controller
spec:
  replicas: 1
  selector:
    matchLabels:
      app: alb-ingress-controller
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: alb-ingress-controller
    spec:
      containers:
        - args:
            # Limit the namespace where this ALB Ingress Controller deployment will
            # resolve ingress resources. If left commented, all namespaces are used.
            # - --watch-namespace=uat

            # Setting the ingress-class flag below ensures that only ingress resources with the
            # annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
            # choose any class you'd like for this controller to respect.
            - --ingress-class=alb

            # Name of your cluster. Used when naming resources created
            # by the ALB Ingress Controller, providing distinction between
            # clusters.
            - --cluster-name=aux-eks

            # AWS VPC ID this ingress controller will use to create AWS resources.
            # If unspecified, it will be discovered from ec2metadata.
            # - --aws-vpc-id=vpc-xxxxxx

            # AWS region this ingress controller will operate in.
            # If unspecified, it will be discovered from ec2metadata.
            # List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
            # - --aws-region=us-west-1

            # Enables logging on all outbound requests sent to the AWS API.
            # If logging is desired, set to true.
            # - ---aws-api-debug
            # Maximum number of times to retry the aws calls.
            # defaults to 10.
            # - --aws-max-retries=10
          env:
            # AWS key id for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_ACCESS_KEY_ID
            #  value: KEYVALUE

            # AWS key secret for authenticating with the AWS API.
            # This is only here for examples. It's recommended you instead use
            # a project like kube2iam for granting access.
            #- name: AWS_SECRET_ACCESS_KEY
            #  value: SECRETVALUE
          # Repository location of the ALB Ingress Controller.
          image: 894847497797.dkr.ecr.us-west-2.amazonaws.com/aws-alb-ingress-controller:v1.0.0
          imagePullPolicy: Always
          name: server
          resources: {}
          terminationMessagePath: /dev/termination-log
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      serviceAccountName: alb-ingress
      serviceAccount: alb-ingress
iam-policy.json 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "acm:DescribeCertificate",
        "acm:ListCertificates",
        "acm:GetCertificate"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "ec2:AuthorizeSecurityGroupIngress",
        "ec2:CreateSecurityGroup",
        "ec2:CreateTags",
        "ec2:DeleteTags",
        "ec2:DeleteSecurityGroup",
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeSecurityGroups",
        "ec2:DescribeSubnets",
        "ec2:DescribeTags",
        "ec2:DescribeVpcs",
        "ec2:ModifyInstanceAttribute",
        "ec2:ModifyNetworkInterfaceAttribute",
        "ec2:RevokeSecurityGroupIngress"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "elasticloadbalancing:AddTags",
        "elasticloadbalancing:CreateListener",
        "elasticloadbalancing:CreateLoadBalancer",
        "elasticloadbalancing:CreateRule",
        "elasticloadbalancing:CreateTargetGroup",
        "elasticloadbalancing:DeleteListener",
        "elasticloadbalancing:DeleteLoadBalancer",
        "elasticloadbalancing:DeleteRule",
        "elasticloadbalancing:DeleteTargetGroup",
        "elasticloadbalancing:DeregisterTargets",
        "elasticloadbalancing:DescribeListeners",
        "elasticloadbalancing:DescribeLoadBalancers",
        "elasticloadbalancing:DescribeLoadBalancerAttributes",
        "elasticloadbalancing:DescribeRules",
        "elasticloadbalancing:DescribeSSLPolicies",
        "elasticloadbalancing:DescribeTags",
        "elasticloadbalancing:DescribeTargetGroups",
        "elasticloadbalancing:DescribeTargetGroupAttributes",
        "elasticloadbalancing:DescribeTargetHealth",
        "elasticloadbalancing:ModifyListener",
        "elasticloadbalancing:ModifyLoadBalancerAttributes",
        "elasticloadbalancing:ModifyRule",
        "elasticloadbalancing:ModifyTargetGroup",
        "elasticloadbalancing:ModifyTargetGroupAttributes",
        "elasticloadbalancing:RegisterTargets",
        "elasticloadbalancing:RemoveTags",
        "elasticloadbalancing:SetIpAddressType",
        "elasticloadbalancing:SetSecurityGroups",
        "elasticloadbalancing:SetSubnets",
        "elasticloadbalancing:SetWebACL"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "iam:GetServerCertificate",
        "iam:ListServerCertificates"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "waf-regional:GetWebACLForResource",
        "waf-regional:GetWebACL",
        "waf-regional:AssociateWebACL",
        "waf-regional:DisassociateWebACL"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "tag:GetResources",
        "tag:TagResources"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "waf:GetWebACL"
      ],
      "Resource": "*"
    }
  ]
}
ingress.yaml 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
      { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-1:xxxxxxxxx:certificate/281bee29-717c-4b50-bf8e-41a39748c548
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/subnets: subnet-32ce5544, subnet-41cf6c19
    alb.ingress.kubernetes.io/security-groups: sg-018347dc7a3ade5a2
    kubernetes.io/ingress.class: alb
  name: xiemx-web-ingress
spec:
  rules:
  - host: www.xiemx.com
    http:
      paths:
      - backend:
          serviceName: echo-v1
          servicePort: 80
  - host: '*.xiemx.com'
    http:
      paths:
      - backend:
          serviceName: ssl-redirect
          servicePort: use-annotation
        path: /*
      - backend:
          serviceName: echo-v1
          servicePort: 80
        path: /*

测试：

  • 创建alb-ingress-controller
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
➜  aws-alb-ingress-controller git:(master) ✗ k get all
NAME                                         READY   STATUS    RESTARTS   AGE
pod/alb-ingress-controller-9596b67b9-d7p69   1/1     Running   0          80d

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/alb-ingress-controller   1/1     1            1           361d

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/alb-ingress-controller-9596b67b9   1         1         1       361d



➜  aws-alb-ingress-controller git:(master) ✗ k logs -f pod/alb-ingress-controller-9596b67b9-d7p69
-------------------------------------------------------------------------------
AWS ALB Ingress controller
  Release:    v1.0.0
  Build:      git-c25bc6c5
  Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
-------------------------------------------------------------------------------

W0917 09:50:21.934737       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0917 09:50:21.990251       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0917 09:50:21.990434       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
I0917 09:50:21.990555       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null}}}
I0917 09:50:21.990841       1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource"  "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
I0917 09:50:21.993054       1 leaderelection.go:185] attempting to acquire leader lease  alb-ingress-controller/ingress-controller-leader-alb...
I0917 09:50:38.515944       1 leaderelection.go:194] successfully acquired lease alb-ingress-controller/ingress-controller-leader-alb
I0917 09:50:38.716164       1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller"  "Controller"="alb-ingress-controller"
I0917 09:50:38.816322       1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers"  "Controller"="alb-ingress-controller" "WorkerCount"=1
  • 创建ingress测试
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
➜  aws-alb-ingress-controller git:(master) ✗ k get ingress
NAME                HOSTS                       ADDRESS                                                                      PORTS   AGE
xiemx-web-ingress   www.xiemx.com,*.xiemx.com   71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com   80      14m
➜  aws-alb-ingress-controller git:(master) ✗ curl 71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com -H host:www.xiemx.com -I
HTTP/1.1 200 OK
Date: Mon, 07 Dec 2020 06:39:29 GMT
Content-Type: text/plain
Connection: keep-alive
Server: echoserver

➜  aws-alb-ingress-controller git:(master) ✗ curl 71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com -H host:www.xiemx.com -i
HTTP/1.1 200 OK
Date: Mon, 07 Dec 2020 06:39:37 GMT
Content-Type: text/plain
Transfer-Encoding: chunked
Connection: keep-alive
Server: echoserver



Hostname: echo-85fb7989cc-d556c

Pod Information:
	node name:	ip-10-200-2-113.ap-northeast-1.compute.internal
	pod name:	echo-85fb7989cc-d556c
	pod namespace:	alb-ingress-controller
	pod IP:	10.200.2.197

Server values:
	server_version=nginx: 1.12.2 - lua: 10010

Request Information:
	client_address=10.200.2.21
	method=GET
	real path=/
	query=
	request_version=1.1
	request_scheme=http
	request_uri=http://www.xiemx.com:8080/

Request Headers:
	accept=*/*
	host=www.xiemx.com
	user-agent=curl/7.54.0
	x-amzn-trace-id=Root=1-5fcdce29-473e8ac375ce203f4961bec3
	x-forwarded-for=101.231.43.114
	x-forwarded-port=80
	x-forwarded-proto=http

Request Body:
	-no body in request-

➜  aws-alb-ingress-controller git:(master) ✗ curl 71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com -H host:blog.xiemx.com -I
HTTP/1.1 301 Moved Permanently
Server: awselb/2.0
Date: Mon, 07 Dec 2020 06:39:44 GMT
Content-Type: text/html
Content-Length: 134
Connection: keep-alive
Location: https://blog.xiemx.com:443/

  • aws alb 规则

aws-alb-ingress-ex1
aws-alb-ingress-ex2
aws-alb-ingress-ex3
aws-alb-ingress-ex4

mingxu.xiedevops

进城打工农民工👨‍🌾