LoadBalancer Type Service¶
Kube-OVN supports the implementation of VPC and VPC gateway. For specific configurations, please refer to the VPC configuration.
Due to the complexity of using VPC gateways, the implementation based on VPC gateways has been simplified. It supports creating LoadBalancer type Services in the default VPC, allowing access to Services in the default VPC through LoadBalancerIP.
First, make sure the following conditions are met in the environment:
- Install
multus-cniandmacvlan cni。 - LoadBalancer Service support relies on simplified implementation of VPC gateway code, still utilizing the vpc-nat-gw image and depending on macvlan for multi-interface functionality support.
- Currently, it only supports configuration in the default VPC. Support for LoadBalancers in custom VPCs can be referred to in the VPC configuration.
Steps to Configure Default VPC LoadBalancer Service¶
Enable Feature Flag¶
Modify the deployment kube-ovn-controller under the kube-system namespace and add the parameter --enable-lb-svc=true to the args section to enable the feature (by default it's set to false).
containers:
- args:
- /kube-ovn/start-controller.sh
- --default-cidr=10.16.0.0/16
- --default-gateway=10.16.0.1
- --default-gateway-check=true
- --enable-lb-svc=true // parameter is set to true
Create NetworkAttachmentDefinition CRD Resource¶
Refer to the following YAML and create the net-attach-def resource:
apiVersion: "k8s.cni.cncf.io/v1"
kind: NetworkAttachmentDefinition
metadata:
name: lb-svc-attachment
namespace: kube-system
spec:
config: '{
"cniVersion": "0.3.0",
"type": "macvlan",
"master": "eth0", //Physical network card, configure according to the actual situation
"mode": "bridge"
}'
By default, the physical NIC eth0 is used to implement the multi-interface functionality. If another physical NIC is needed, modify the master value to specify the name of the desired physical NIC.
Create Subnet¶
The created Subnet is used to allocate LoadBalancerIP for the LoadBalancer Service, which should normally be accessible from outside the cluster. An Underlay Subnet can be configured for address allocation.
Refer to the following YAML to create a new subnet:
apiVersion: kubeovn.io/v1
kind: Subnet
metadata:
name: attach-subnet
spec:
protocol: IPv4
provider: lb-svc-attachment.kube-system //The provider format is fixed and consists of the Name.Namespace of the net-attach-def resource created in the previous step
cidrBlock: 172.18.0.0/16
gateway: 172.18.0.1
excludeIps:
- 172.18.0.0..172.18.0.10
In the provider parameter of the Subnet, ovn or .ovn suffix is used to indicate that the subnet is managed by Kube-OVN and requires corresponding logical switch records to be created.
If provider is neither ovn nor ends with .ovn, Kube-OVN only provides the IPAM functionality to record IP address allocation without handling business logic for the subnet.
Create LoadBalancer Service¶
Refer to the following YAML to create a LoadBalancer Service:
apiVersion: v1
kind: Service
metadata:
annotations:
lb-svc-attachment.kube-system.kubernetes.io/logical_switch: attach-subnet #Optional
ovn.kubernetes.io/attachmentprovider: lb-svc-attachment.kube-system #Required
labels:
app: dynamic
name: test-service
namespace: default
spec:
loadBalancerIP: 172.18.0.18 #Optional
ports:
- name: test
protocol: TCP
port: 80
targetPort: 80
selector:
app: dynamic
sessionAffinity: None
type: LoadBalancer
In the yaml, the annotation ovn.kubernetes.io/attachmentprovider is required, and its value is composed of the Name.Namespace of the net-attach-def resource created in the first step. This annotation is used to find the net-attach-def resources when creating Pods.
The subnet used for multi-interface address allocation can be specified through an annotation. The annotation key format is net-attach-def resource's Name.Namespace.kubernetes.io/logical_switch. This configuration is optional and if LoadBalancerIP address is not specified, addresses will be dynamically allocated from this subnet and filled into the LoadBalancerIP field.
If a static LoadBalancerIP address is required, the spec.loadBalancerIP field can be configured. The address must be within the specified subnet's address range.
After creating the Service using the YAML, you can see the Pod startup information in the same namespace as the Service:
# kubectl get pod
NAME READY STATUS RESTARTS AGE
lb-svc-test-service-6869d98dd8-cjvll 1/1 Running 0 107m
# kubectl get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
test-service LoadBalancer 10.109.201.193 172.18.0.18 80:30056/TCP 107m
When specifying the service.spec.loadBalancerIP parameter, it will be assigned to the service's external IP field. If not specified, the parameter will be assigned a random value.
View the YAML output of the test Pod to see the assigned multi-interface addresses:
# kubectl get pod -o yaml lb-svc-test-service-6869d98dd8-cjvll
apiVersion: v1
kind: Pod
metadata:
annotations:
k8s.v1.cni.cncf.io/network-status: |-
[{
"name": "kube-ovn",
"ips": [
"10.16.0.2"
],
"default": true,
"dns": {}
},{
"name": "default/test-service",
"interface": "net1",
"mac": "ba:85:f7:02:9f:42",
"dns": {}
}]
k8s.v1.cni.cncf.io/networks: default/test-service
k8s.v1.cni.cncf.io/networks-status: |-
[{
"name": "kube-ovn",
"ips": [
"10.16.0.2"
],
"default": true,
"dns": {}
},{
"name": "default/test-service",
"interface": "net1",
"mac": "ba:85:f7:02:9f:42",
"dns": {}
}]
ovn.kubernetes.io/allocated: "true"
ovn.kubernetes.io/cidr: 10.16.0.0/16
ovn.kubernetes.io/gateway: 10.16.0.1
ovn.kubernetes.io/ip_address: 10.16.0.2
ovn.kubernetes.io/logical_router: ovn-cluster
ovn.kubernetes.io/logical_switch: ovn-default
ovn.kubernetes.io/mac_address: 00:00:00:45:F4:29
ovn.kubernetes.io/pod_nic_type: veth-pair
ovn.kubernetes.io/routed: "true"
test-service.default.kubernetes.io/allocated: "true"
test-service.default.kubernetes.io/cidr: 172.18.0.0/16
test-service.default.kubernetes.io/gateway: 172.18.0.1
test-service.default.kubernetes.io/ip_address: 172.18.0.18
test-service.default.kubernetes.io/logical_switch: attach-subnet
test-service.default.kubernetes.io/mac_address: 00:00:00:AF:AA:BF
test-service.default.kubernetes.io/pod_nic_type: veth-pair
Check the service information:
# kubectl get svc -o yaml test-service
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"test-service.default.kubernetes.io/logical_switch":"attach-subnet"},"labels ":{"app":"dynamic"},"name":"test-service","namespace":"default"},"spec":{"ports":[{"name":"test", "port":80,"protocol":"TCP","targetPort":80}],"selector":{"app":"dynamic"},"sessionAffinity":"None","type":"LoadBalancer "}}
ovn.kubernetes.io/vpc:ovn-cluster
test-service.default.kubernetes.io/logical_switch: attach-subnet
creationTimestamp: "2022-06-15T09:01:58Z"
labels:
app: dynamic
name: test-service
namespace: default
resourceVersion: "38485"
uid: 161edee1-7f6e-40f5-9e09-5a52c44267d0
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 10.109.201.193
clusterIPs:
- 10.109.201.193
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: test
nodePort: 30056
port: 80
protocol: TCP
targetPort: 80
selector:
app: dynamic
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- ip: 172.18.0.18
Testing LoadBalancerIP access¶
Refer to the following YAML to create a test Pod that serves as the Endpoints for the Service:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: dynamic
name: dynamic
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: dynamic
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: dynamic
spec:
containers:
- image: docker.io/library/nginx:alpine
imagePullPolicy: IfNotPresent
name: nginx
dnsPolicy: ClusterFirst
restartPolicy: Always
Under normal circumstances, the provided subnet addresses should be accessible from outside the cluster. To verify, access the Service's LoadBalancerIP:Port from within the cluster and check if the access is successful.
# curl 172.18.0.11:80
<html>
<head>
<title>Hello World!</title>
<link href='//fonts.googleapis.com/css?family=Open+Sans:400,700' rel='stylesheet' type='text/css'>
<style>
body {
background-color: white;
text-align: center;
padding: 50px;
font-family: "Open Sans","Helvetica Neue",Helvetica,Arial,sans-serif;
}
#logo {
margin-bottom: 40px;
}
</style>
</head>
<body>
<h1>Hello World!</h1>
<h3>Links found</h3>
<h3>I am on dynamic-7d8d7874f5-hsgc4</h3>
<h3>Cookie =</h3>
<b>KUBERNETES</b> listening in 443 available at tcp://10.96.0.1:443<br />
<h3>my name is hanhouchao!</h3>
<h3> RequestURI='/'</h3>
</body>
</html>
Enter the Pod created by the Service and check the network information:
# ip a
4: net1@if62: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether ba:85:f7:02:9f:42 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 172.18.0.18/16 scope global net1
valid_lft forever preferred_lft forever
inet6 fe80::b885:f7ff:fe02:9f42/64 scope link
valid_lft forever preferred_lft forever
36: eth0@if37: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1400 qdisc noqueue state UP group default
link/ether 00:00:00:45:f4:29 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 10.16.0.2/16 brd 10.16.255.255 scope global eth0
valid_lft forever preferred_lft forever
inet6 fe80::200:ff:fe45:f429/64 scope link
valid_lft forever preferred_lft forever
# ip rule
0: from all lookup local
32764: from all iif eth0 lookup 100
32765: from all iif net1 lookup 100
32766: from all lookup main
32767: from all lookup default
# ip route show table 100
default via 172.18.0.1 dev net1
10.109.201.193 via 10.16.0.1 dev eth0
172.18.0.0/16 dev net1 scope link
# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 172.18.0.18 tcp dpt:80 to:10.109.201.193:80
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- * * 0.0.0.0/0 10.109.201.193