From 2289586a6ebaee2370cd7d81eb15c59081f59c5a Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Mon, 6 Mar 2017 20:37:34 +0800 Subject: [PATCH 1/3] init kubernetes service tutorials --- README.md | 2 +- kubernetes/networking/service-network.md | 181 ++++++++++++++++++++++ kubernetes/networking/service-network.png | Bin 0 -> 27791 bytes 3 files changed, 182 insertions(+), 1 deletion(-) create mode 100644 kubernetes/networking/service-network.md create mode 100644 kubernetes/networking/service-network.png diff --git a/README.md b/README.md index 6a0584a..11cb4f5 100644 --- a/README.md +++ b/README.md @@ -1 +1 @@ -# tutorials \ No newline at end of file +# tutorials diff --git a/kubernetes/networking/service-network.md b/kubernetes/networking/service-network.md new file mode 100644 index 0000000..b02c2de --- /dev/null +++ b/kubernetes/networking/service-network.md @@ -0,0 +1,181 @@ +## Service +Pod的IP是动态分配在**docker0**所在网段的,当发生重启，扩容等操作时，IP地址会随之变化。当某个Pod(frontend)需要去访问其依赖的另外一组Pod(backend)时，如果backend的IP发生变化时，如何保证fronted到backend的正常通信变的非常重要。为了解决此类问题,Kubernetes设计了Service的概念。 + +这里docker0是一个网桥，docker daemon启动container时会根据docker0的网段来划粉container的IP地址[Docker网络](https://docs.docker.com/engine/userguide/networking/dockernetworks/) + + +在实际生产环境中，对Service的访问可能会有两种来源：Kubernetes集群内部的程序（Pod）和Kubernetes集群外部，为了满足上述的场景，Kubernetes service有以下三种类型： + +* **ClusterIP**:提供一个集群内部的虚拟IP以供Pod访问。 +* **NodePort**:在每个Node上打开一个端口以供外部访问。 +* **LoadBalancer**:通过外部的负载均衡器来访问。 + +### ClusterIP && NodePort && LoadBalancer +#### 1. ClusterIP +此模式会提供一个集群内部的虚拟IP（与Pod不在同一网段)，以供集群内部的pod之间通信使用。 +ClusterIP也是Kubernetes service的默认类型。 + +![Pod-Service](service-network.png) + +为了实现图上的功能主要需要以下几个组件的协同工作 + +* **apiserver** 用户通过kubectl命令向apiserver发送创建service的命令，apiserver接收到请求以后将数据存储到etcd中。 +* **kube-proxy** kubernetes的每个节点中都有一个叫做kube-proxy的进程，这个进程负责感知service，pod的变化，并将变化的信息写入本地的iptables中。 +* **iptables** 使用NAT等技术将virtualIP的流量转至endpoint中。 + +下面我们实际发布一个Service，能够更清晰的了解到Service是如何工作的。 + +##### 1.1 发布一个rc，并指定replices count为3. + +``` +yancey@ yancey-macbook kubernetes-1$kubectl create -f rc_nginx.yaml +yancey@ yancey-macbook kubernetes-1$kubectl get pods +NAME READY STATUS RESTARTS AGE +k8s-master-core-v2-01 4/4 Running 0 8m +k8s-proxy-core-v2-01 1/1 Running 0 8m +nginx-controller-6bovu 1/1 Running 0 4m +nginx-controller-iowux 1/1 Running 0 4m +nginx-controller-o7m6u 1/1 Running 0 4m +yancey@ yancey-macbook kubernetes-1$kubectl describe pod nginx-controller-6bovu +Name: nginx-controller-6bovu +Namespace: default +Node: core-v2-01/172.17.8.101 +Start Time: Fri, 17 Jun 2016 15:22:20 +0800 +Labels: app=nginx +Status: Running +IP: 10.1.13.3 +Controllers: ReplicationController/nginx-controller +``` + +##### 1.2. 发布一个service，并指定步骤1中的label。 + +``` +yancey@ yancey-macbook kubernetes-1$kubectl create -f service_nginx.yaml +yancey@ yancey-macbook kubernetes-1$kubectl get svc +NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE +kubernetes 10.0.0.1 443/TCP 9m +nginx-service 10.0.0.252 8000/TCP 4m +``` +kubernetes为nginx-server这个service创建了一个10.0.0.252这个ClusterIP，也可以称为VirtualIP. + +``` +yancey@ yancey-macbook kubernetes-1$kubectl get ep +NAME ENDPOINTS AGE +kubernetes 172.17.8.101:6443 9m +nginx-service 10.1.13.2:80,10.1.13.3:80,10.1.13.4:80 4m +``` +查看endpoint信息，发现nginx-service一共有三个endpoint地址，分别对应nginx的三个pod。 + +##### 1.3. 查看iptables，观察其NAT表中的信息（只截取了部分和这个service有关的信息） +查看iptables中NAT表的命令:```iptables -L -v -n -t nat``` + +``` +Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) + pkts bytes target prot opt in out source destination + 37 2766 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service portals */ + 33 2112 DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL +``` +在PREROUTING链中会先匹配到KUBE-SERVICES这个Chain。 + +``` +Chain KUBE-SERVICES (2 references) + pkts bytes target prot opt in out source destination + 0 0 KUBE-SVC-GKN7Y2BSGW4NJTYL tcp -- * * 0.0.0.0/0 10.0.0.252 /* default/nginx-service: cluster IP */ tcp dpt:8000 + 18 1080 KUBE-NODEPORTS all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes service nodeports; NOTE: this must be the last rule in this chain */ ADDRTYPE match dst-type LOCAL + +``` + +所有destinationIP为10.0.0.252的包都转到**KUBE-SVC-GKN7Y2BSGW4NJTYL**这个Chain + +``` +Chain KUBE-SVC-GKN7Y2BSGW4NJTYL (1 references) + pkts bytes target prot opt in out source destination + 0 0 KUBE-SEP-7ROBBXFV7SD4AIRW all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-service: */ statistic mode random probability 0.33332999982 + 0 0 KUBE-SEP-XY3F6VJIZ7ELIF4Z all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-service: */ statistic mode random probability 0.50000000000 + 0 0 KUBE-SEP-JIDZHFC4A3T535AK all -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-service: */ +``` +这里能看到请求会平均执行**KUBE-SEP-7ROBBXFV7SD4AIRW**,**KUBE-SEP-XY3F6VJIZ7ELIF4Z**,**KUBE-SEP-XY3F6VJIZ7ELIF4Z**这三个Chain.最后我们看下**KUBE-SEP-7ROBBXFV7SD4AIRW**这个Chain做了什么 + +``` +Chain KUBE-SEP-7ROBBXFV7SD4AIRW (1 references) + pkts bytes target prot opt in out source destination + 0 0 KUBE-MARK-MASQ all -- * * 10.1.13.2 0.0.0.0/0 /* default/nginx-service: */ + 0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-service: */ tcp to:10.1.13.2:80 + +``` +这个Chain使用了DNAT规则将流量转发到10.1.13.2:80这个地址。至此从Pod发出来的流量通过本地的iptables将流量转至了service背后的pod上。 +#### 2. NodePort +Kubernetes将会在每个Node上打开一个端口并且**每个Node的端口都是一样的**，通过\:NodePort的方式Kubernetes集群外部的程序可以访问Service。 + +修改kubernetes/service_nginx.yaml，将Service的type改为NodePort类型。 + +``` +apiVersion: v1 +kind: Service +metadata: + name: nginx-service +spec: + type: NodePort + ports: + - port: 8000 + targetPort: 80 + protocol: TCP + # just like the selector in the replication controller, + # but this time it identifies the set of pods to load balance + # traffic to. + selector: + app: nginx +``` +发布这个service，可以看到已经随机分配了一个NodePort端口。 + +``` +yancey@ yancey-macbook kubernetes-1$kubectl get svc nginx-service -o yaml +apiVersion: v1 +kind: Service +metadata: + creationTimestamp: 2016-06-18T03:29:42Z + name: nginx-service + namespace: default + resourceVersion: "1405" + selfLink: /api/v1/namespaces/default/services/nginx-service + uid: e50ba23a-3504-11e6-a94f-080027a75e9e +spec: + clusterIP: 10.0.0.229 + ports: + - nodePort: 30802 + port: 8000 + protocol: TCP + targetPort: 80 + selector: + app: nginx + sessionAffinity: None + type: NodePort +status: + loadBalancer: {} +``` +观察Iptables中的变化，KUBE-NODEPORTS这个Chain中增加了如下内容 + +``` +Chain KUBE-NODEPORTS (1 references) + pkts bytes target prot opt in out source destination + 0 0 KUBE-MARK-MASQ tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-service: */ tcp dpt:30802 + 0 0 KUBE-SVC-GKN7Y2BSGW4NJTYL tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* default/nginx-service: */ tcp dpt:30802 +``` + +可以看到流量会转至KUBE-SVC-GKN7Y2BSGW4NJTYL这个Chain中处理一次，也就是ClusterIP中提到的通过负载均衡将流量平均分配到3个endpoint上。 + +#### 3. LoadBalancer +待补充。 + +### 服务发现 +当发布一个服务之后，我们要使用这个服务，第一个问题就是要拿到这些服务的IP和PORT，kubernetes提供两种方式以便在程序中去动态的获取这些信息。 + +1. ENV环境变量 +在Pod其中之后，kubernetes会将现有服务的IP，PORT以环境变量的方式写入pod中，程序只要读取这些环境变量即可。 +2. DNS，程序中可以使用server的名称对服务进行访问，在程序启时候，不必预先读取环境变量中的内容。 + +这是两种方式的详细说明：[http://kubernetes.io/docs/user-guide/services/#discovering-services](http://kubernetes.io/docs/user-guide/services/#discovering-services) + +### 附录 +1. [iptables文档](https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/4/html/Security_Guide/s1-fireall-ipt-act.html#s2-firewall-policies) +2. [Debuging Service](http://kubernetes.io/docs/user-guide/debugging-services/) diff --git a/kubernetes/networking/service-network.png b/kubernetes/networking/service-network.png new file mode 100644 index 0000000000000000000000000000000000000000..ddaee8d7a11732148595106b63702b7e65ac4b99 GIT binary patch literal 27791 zcmeFZWmMH&_xB6ijev--NkzK5rDIEXDV>6JNk~hBbax{m(%qrbDWEin(%qeB?dyNt z&vT#W&3SWPoEK*dhGWR)w|*<;Tx-tH{H`6Uq9l#^i0Ba#5)!7YjD#8z5+nx+3E2Vd zA^1&cJ<&K45;>Br#LL%i2D_Q4zwjhy>*Xc7p(!x^?hO2H-w^*W{+6{sV*(Cir5nbu za_cDaurQd@h_O;p%7dO4Z${osB;Mo5G&|1iZ*8uO*RyqR&~RtUc^ zh1+tRI*%Me`CmWJU@2k{ZmMyQNVE{&|Mf#dj$9o4Uw=gz!+?xFs^G9~`rpsu4&0w?r(KPKCnn&}4kqDwGoPh;hL(IjquKQ^q(!h(sL?UJe)(LaE`nOLw~s zi#t;`*Jr!9g}>%|8G2v8QZy}0F%DZZqh7b4g2o*`lH>3cy}zm1-6j!mmN{gR6L<@n z7%yAM_P**@5WPLwY^U!2b$^_FpCWp9(W$^S%+1hFBPyFL=1c0ak-J8Af6@E)SG3?B zpVP(PkG*LMvST$ey+mg{KC)8(j!Q2Wi{E5C*7|3i~IS4 zt8M#wmZzQ5;)iqh?T>jqNM>fmV( zJG7rn2wf}=_*+GiCq_^O88Ps&^NIP;{Vc?b=?sy2Uqj&?trG1vXwHt#t2 zY|Z$tEmnp9oHtIY%-pDMh;uOIbC$=z!TH`oOFHjwx6UE%p9&&|@q@K;6mZu$Ti%-A z9^2YgAr?1l>GqHB>qi8(Q(h?x*X@P5-`}1O)27=P8@T;NVH#&`J5f9S?YW=294-2L z6b{Sb{-kZ5IZ=+gVqNI8K4KQdzgBcjQYJb7t`l>2R(#+umBe7cDz4P^en-dct6TxU zY6bG*`+F*1Nz=`O{B(dmVgx!E_E@!;8UE8}m%m zyA|I~*yDtqF5{j5Z9M;ds(Dt`_jAMhtK`eM#XEwQSaP%mFy^EtQ5Q&=5d1Dl&r!C7 zq))Kf|2)%ifZ;!LW62kQip9fJS}7(^Gw(8II|&&eoTbb5G6n;E>NI1TJj2)d5v#$F z0?JGxxKkN$E4UQTR+VnwK67)i99{qIIPt{~q2uY1i56G;xvxya>&{sZp74Ej-kqHo zP0jK;l0632k*4o5)7^SDYi;(`7kO_&KQNc_nGfd+WisBtM$7Pn z;dIO4p`13YhdpPD7tM3_9k->I-s_pJ^-F0dvd5poS*_gbHQcd(p>5mN&-AojZIuKW zs>Nv7GYvxH2s%C$gqz}o)7}0;m$;fnA=ERvyF2fg6NwB;@Ls;+{G9}*X(ozfO3x|s za;v0bXg%AH%VaGAPJWoSF!bA6guLV#yF~4womevjTjbO7e5T^SeKcitPtqh zym|P9*kZ47`H7(6c=kg~qPOdcdV7QhZn^Har&GhgzD&zPPEvL4+a~U0|8&NlkCSbR z4o&4tlXY{3PbAz)y8)lHy(9IxaS*(J@|e+K_-s zFhCO4&u6XK<}Rv?D9J5Ptjn7U@jINQ#wmAM8s^6%o^pI|W>1+0{z`VeH_S6P#4|h0 zyEy+RYJaYw!hIw6qYc&SIf?81n2XMUI2|=h7xU)NM%a(KfdHiL+0K=Vjl5_LeI;yX z<)WH*rm66)E|?ehmBanrEoKritq3z8Q+cCtIR5HSWzjmyg!Uon#nD`GTT$YLg zN22$a%3O0cHL1XM7E0@9tz4G!e8!bw)nwimxx48KW+#?C50fwtl|)Ek^;(9P=bcXk zhYE*#*97-lO@6A=Kk=)MAXD-cIh!^)Gl}p%nbfsUozOH~Ao56&PQ@)5_Tyb}j?e66 zue>?v#40rG@Sc3y*(z;Fr6}@H%9StxIgPm{8 z?7kvnkOP||*eG`OcI6uEO}4O{O5@+Xf4cT5KB}RqJlu8~;Ww}Lkf0tvD9mt_W5RJL zsA-QIkVYC;y?FGD%Pb;S{LiN~>nJv=b{(x`v2_P)+DxR#_CO{7ApC8V{db0Z-18RA zN*9OkrRn&ucWQJeQ;$g71KrpnV?)vZ4iQe*{#3Fq%kx@LB^cnho9DY`HD?G-JKq#S ztk-G?Hf2Xx!ugW?NO^>$E)+Oc5#dY+Qebc*bfBZ3KADLCU7QT1od45>MbFdn09ns(O z;`O{8w*ir$@nHy6RAmQ0p;ujZ=Z)0I+_CBa9!5s@$@fp&ABToB`MocI3NXdvVsbPm z3DKz02)o>X}*?VN@L3F!>^4ZBhWzsds?9$dcxul^GDA$C`vgW zbyKJ$VQKO1AB^=hOReeaEoh0Lmks+dzX@ec4eyck+5@)0gBc-%rcs(PG^HhVh{i`K zG`KLYCZJRazc~EclR={2zBX>W*M7a%vU_9D#7jMk3dc^MTW0^C-oTJKQ zj_1DmL}0v==)2A<|84e=S>8pj8p(GWxH-Aoq{{bZT374U|fCNzU^3#uD@f--=KY z>atal(INkK*`tG*Vwj%Ip-PgJh#DA8q)hUdm`cZN84oD%P&p3X1^U` zGRv8LGGbY4SoEH&e)y|t8AfHR4bRii5Ww6#>h2L>Z^ucsz>aCW_!O^EfRYd_?;g^7 z4XZBIelz^o4t^w;HUXt4nA2I!eXN95!YO+yk+ZRNAe*NKk$?+i}Y=CpqoH!vG7R$>8eCZ;_HAug8mT zT<7<_u(y{&jO(hvQDuuh3TEmG?YTdQ#D3Q zj5)zKYtR%+iV!66EO*rc-~*6eZeO zTOLN}GQx!>>koo7Qyc2Xi!?|DIG%s5EidJ11Ho}*?fPhTZpGj4B`!Z7-G=F-4q;C;C|8UM^921!w?v$Uyeclgb?jSa;&~ zdQjz{cRiP*p@tMt#Nd$E!{Sh5>`^GrgQQPkzGw|$1*NG9R00l_$Uy?xC5DRZfU!4PUr@4w~hxJQz3kZWnH^Urd%mYL+tb#ruW``I@A!9-;C72ntKpizwN%Cw&vo!_@ zC2OEr+b6DVbvaPy80VKvN;3q0#E4h@-+GG}HXJHOn}anSW36c#E)R!rvvPEU#2ouS zKQa>FcQ6rQJsyZ-Zem%Fx&iD!8qP~`Y4Rl@Oc@O9;(Za$TN&_2;?o|9Xi%^;(nk2j zOVpGqPm!-cV4Zfp=(Bk6*&mHN{HtlQz8S4z+FO_3?(1fzFZc10Vf6IKkz^=kjlMMt zO-|-N%XE1=v$a2dM;kp{?oC(ClS&1l9EY`(M>bk5_8+Pc_(yd_*ZlavK)Z4a8nv-E zez&!3E_QVM{ZSeKwV3QWZ5P7@^Rcn@YE3fd3-~i`{0;+yAYZ5j$*>sfg#bB&+!l02 zOrE65fE@A-tP#v+f}q~W@f%_NR>EO5Y?Ri&dZKZF333M(_n=pe0Y4l5M(?0ox(p`} zvAAe;MVujUC-9*KnBN!Kz-W%O*#;8-p;B-os57qG|BN441X?yJ!G9(abUWnV`DJ^7 z+%zhuf6Lg4MooE;eSg!NMK$04;aXyI;|&bW4|EmrlDAS|Qn`M4#G*l^!KA)ZFNV;N zgGQ0ENLk2`70FjioD#L5Ua--Nzv21g=>PQ?u8djHDJ2Xo3b@>#;@2p0aL^L6kg$#j zBn>{Gxz1RHc0rkQ*hwf`fC~XujcuiB+85zV8#AP`ElG(@3|8cu`j_8HD9$TCo9Tb9-L-Fg=K+H4HlhoYFa0zlgV3__TYWVcXph@V#NzuQY z?w7pL#%S&t@n~MONb_%LJY}dq0Cz+-6U86}`VVr7Q3efOmJ+Tqeg=y(XxUA#OR0hc zX-5Bol$g(K_U}}vqJh~KFOmcy8*r*+X{W~RvP)_(CkQullmn}y#3VI?;|IYWy-@X) zQzPJ_0v|rJYNLmOk&FF*=E48*I|x-ydJLpQ1)mNEW=!JZr1F1}O|%fW9+(h^PzLWW z=zT#NTU(lcCCOpP3xNs1rk+HAg+UluKEj~%+vW5TK72^EGOIqb1n6d@yuzWpBAAUY zC5AzSeiFb5sK0;y_y9}>kycPK-zM0H;#ds}SRoX+!2TU~i)|dhuGvFVcu_zLcA;Mr z72m_*lwdt^{{HOrgT)|WmWruoID<_{Hn~EL&@TpTP-smq>h+CaJ(h7^-KZg4?mwN3 zBnv&*{QDJbPY*S68I_uW_2ds~ehG*7fsO>~wJ_@bJGS6NszC9#5;t*-Pqa z^FVnU215%6W7InRGXB4LFMJPla;-fq;hPXF27^VpF1IxVUWIl8AQ(^Q?ra^CMvrd) z?$IC{nSyqhN{qvD4?acu`&Cs)EvrHpHRN6z93(6+!A7qNp@<^G$%(8F?#YTrvW-a3 z!twi#i{=1f@G2-mqEx{~e6HjhOdWAFogucujA22ln|73-3!zXVZwe}l2nB8~Ml33Z zC*^MALgAQ%G;AXOD$7xxybGPWp)!#Jh86)2S(Xm@P9HtYDlvkS<8EU)@X8K7kk4<0 zpq`T!;s-->xIt5Wx>_|8)Ta&Mj%HDGA7Jsrri+>XuF)ew&W`8+#iKkSo)&C=dEd3F z0tV$K8tf%#obL2M3fN@HyAG)&-X!#z5mK6?j|$auugd;&h5;v}_#KG%iG9GN6C<_; z`cKEH|8+7H@|f380m`s5n^4z$aeIB

{*z@JZV&)pX~5!2{GMwfAR$Hl#=r>i-hD z+Q>8e8$?(xXZk%5-3)-v7bd+?W$vT*omiqaS7*D{2407fH7DcppQX$dl)BWA` zKfFgn+Z3OPcCNwRKI_Aob`4-3DVhfE-3o%cwd&RITX({pehrvhW@t1GDvg*PKs zE>33+K`m>#33O{i28GY(f1Yej3_j{YfR7=N9BKS&0}OF(r{)Kbq4#A0ehgYazJ}H} zJ&(--vrw8Ob}jgnmI-DJFjg((Mg$U#rIJo-T1y+$*pepu@K(dX-PCD0k~i(el-|!m zUVv0)7K1ULn|a?|nmcu3@}^7~cn<41_S2XFJjc^n))#V}Fn%0qW$X)&MO% zTIU41P-A=qMqjsg`7+?4_TP`btLK`AA0~8Et7=6MH2dPUrKmiG|1Xw6Fx{>!kDBj!R-HZ&;pQfO!w}ACXj}wNAk?}-2m8M|3;C2HK9=a z;}e$1odKqbT5D|@xbD6yII^B5CxrWBBlZq757SrBN6P=K0#bn*%qC>OZlO8#hv@x0 z@%vw@Yrr@g%8!6u{n+WaztdoDKmYBi&P!4C8;62uAuw6hq+EE#@Nd#GyK9V7Dpqkc304DX^C={)+on`=K@PK98_#S|#{dW3Fti--Qw-BXoNL zVy;ZL#??z)COZFUl3gu<&}i$-#lkcz#$&(0y1>g$YV8SV_8Ved;jkK6%W@yweXB42vNK}H9SwAadv$j(q?oh5!Ac{D8zbUB}Bbz z2`rqYBqhNQ1o~{AtIDd*N>F^c+#B6sEv-+k$Xr^wE`-Rz1m6wUJVWR3SW7jZeeqdK zRhs~AkmY@4*)p~Xz;a!Bo)t>dpY82eI60>dYRs@og!jCz15wBfSizeE3V5y$V616a zA{SrvKAcXlth-S%H_Td<^h<0GvUS*?P?mc@_Rc??&Gfe3p3Q0aQpz}_)|h0j!%}wA z*3syg^H|9}G(m-?Md>(Jnv}!P=eDkCvoL+o$#a@lztoD8}ZD<$AnObV002< zYBXFt>0AH1f*M})%=LuepRn664w0AK5g|>g=xyF;KD)pq{LqHna0Qblev_nQ#jri# z`aBH#imZmqCKU?l%MC+8tXQ)M%mAE_Xe?m?_e{6V;gBcxahv@md6H5Q7 zVNL(6Dsxr>4UyI|f9GSal#2GNfgzCkDzaolEnXdNuTqYBK`5F>)0EZuvHcve(KczR zUA{PJ&}!(HMmF|@6TSQCMBk?CNyLJhp24Rlvej-c#ZLoIf2qHZ=SK{}gke=vdL~L0 zaC}9O+29#Ru7dS?gF@enS%r)S`y4|G!km+7MpGC))DN3BM$A`(R+1bJkt7v;*H9Br z{1;INn{yH%)OS|(8{ck z*P71UIqEs@$dJHx=~?+HChz>{9{akG(%pjepLzSR_R` zl^@~oZ|Hcy)A<}t4Wa%NDz%3X&abAcOoqV;VsbAL;fN%d?$Q{a**W#$Q6AXNM}Kp7 zecRvLQ65=!)-uH8<_wtQu z`SHs1=M7br#K?*)w~2w+TG;hU1~*%K))cFoA$6h>0~($!gh8^PCK@Lg=CHM$#I-HAC85-2xt%8D#jb6IJ5lW`Z9hXFt?gr6AVubwNkOyG9vRjc>IPEF%s)AZDVt=maYSJmTk z1W(|)wzR}mzu?wJS8-``>bO>wCV8RlQmKX5JB98pmd_Z6b#irto}TP4E+R@TYoKVz z1&*QC4_=6vsVfA=Vt3)OEbH5|ErFAB=*HmEkG?V!q;XWNdMHVL4n2B@8z-pI-!$i- zT~m(hdR>@qXXSCWWBKE>l=m$Kyq~LOPdi)oPVXy8$5I$mTCVs<9SmAAoGE_6r5Ty` zKNWdr^3vW_Oob8F6-GVd>M^2uZw=pXJ2x0CJPI4}Snh`~Vpz1@${b4MD1scvmQoE0 z5_%$Nny~jDgY`bjhLNAUTRlsg^ScgAY?#P8P3WPSt7v`Tpe|jVraR)dnPzwDf=MeR znU>~jIWbsAlM3%dT_8&BX001`?kia8Rw z(z(u6?R%B~rPiNrbO)Wv@xd_e%lDVQQz?N&1q?CH<~JL?*c2bj;p$7G(GYb$rl_v zKVw**2ZVuuWK{zs9WNl-QDl1rRO@zhhKUVNFN?~sgJQ8*+kc|P)-dO^>xPbFq`{ML zWk$)>`C#-Ee#|{sQR+(sR8QjUw+X1o)UiLYAARd~B|_vIrweik2BXmG6uAGxqVJ*b zQ1pC$??&AS?932lo~7FN;jz)hAicQ!vm_XaNFc84Xyn)|AeKtxF9Tlv1p`pfS(eU2 zbRzD5cLG|pukq>Re^OJ4oJYuuycB=!)`^uT-K)pw_gJJv7#SqqtOSJbbAyx!d8;Ln zUie=Nm5GI)5Kwi|0as89^8_zv4yKRrdUu#96_P;%|ImO_K*;S^t7q|2{GXRI`{e_E z)ZX1EUB~xkg^HzMj0ZVt{1J_eZ?IZCqcknZDY?FtCWpXdpm7momh;9d52~R>SYfB1 zDUfwUnY!`(a?q$#;r@`^?;ttsq8;tKh+Sims=~-YmH0}9tPL0b{#xxXuEkpjdUlu` zJ%myP%a3nanxNt7M7()7%&!)4qDX3NK9nUNf#Ogj;md*Lc>_6(BGXNG*HT42?Hi7TH3Yr>4`T|* z+KnM~J<(PTPZ38#JV^;RIA*E6J8-c5K+1|V{K}S|t%(jGiEBo+hmzzC(Beq4Iy;V8 zpovi2|J^oRS~HS1cSLVO850N+790-`U^xFEx-Dw#HuNt?1XOp~RFOQ>Mh|{g7>Xib z*-wCt1YT)y!{`x*>(WH3BlADmo{9gGfos9}XR5k}cDIfYkmp!0>^`DY;Q{kx zTH=t%`@bkhLfBVubX)X9liBYzt&=)|V?P~{64^1%g4_W!J%Rv40AftJH@_;o= zkV=q01h0EqiKh;gll!`09JtZC93@zawmV)BzByf$r)}hiGBjhJ&Eo1QI~N>wD+=Ce zg-MAW0DAjaD~u6BL#+b5h&k(r@Oeh6_toZb428IyjFcpSUq65f#gZZL$KVkTBZgwM zgb<`&`pDu(3TuScWnr*3<~$Q{E=uLLjDuQ)&IX0)q$B9_M?jweRzZMi zI69hyJ;(QjyPoIv(~6SYr8QCbSs^N};jul@INWIXRd(PN@AK3oVC(|HvXKbn9E2n| z+Z#nK@)~p!UF8KS?1Bs#U8U~fzYZ4i5{%h-kfk97=vWpj=i0J#BR=s` zbh~AOx5j`O3Mkb;STShwD=+eb@k)BpI1?9`G0v&>0n&6|J&UT?O~6`(kSE=j?0RSW z1(bQ7ziYT2O3oQcv%;eWPS8=kHCCMWSCz-I#$u>hikaF2$mSB+ETpi`5S%c(<;;5l`= z5eU*y@lyc@5UkwK=Hdu~r4v~!^aVk%7!7DxA&L|#KUk!8=NgnVcma5VG5Yx;z|-qg zM!72lc>1AbkL_2A*3|^In=Eeci<}~EF+kOcp-^+8R; z)2f!XNwC{-<{x&VN*2H#foW8+Qd|8WB5n~62fae5u$Y9j*ZYO{MCNxVI6tMw%$AZ$ z(jzP!N4*}{8*nRAnjGDT0i9z;R|5pMsxwQd(8B@TDtY3QLMU7jGGWKD^t1qh2y}!! z&vs@Gs^lE1wrTSlyYNKcf(4+MDj!(O1UBZHFQ)=kqe2+)x?Q&CUeha2*Sf$L-fzQn z*#`N+D>F1>s*~9O;XG0&8;C@EO{L-<4odwW-r934v0qBra~!?YS^kI>CP#$_+Rgoy zKM;$Kz_vz{@)9`|h>c?DYUurx{*?3nZ(%Oi7UmpG^;}ZM|BNOr19*{Ru-r@NIJs}4 z)6R6&T9)Ttl@AU$Ek*1&j$+Lt$wcTGBUQk}|GR^8`@QwoW`p@Pqk~fkA>cvy`18pR zj)sC7KfG_RQh~yVz0M?YYh=ZT%IYQr084RNuqp@>`3*4RVbp8%Ft|9BPh%6HpUO|T zhy9EEoj=QtyB0?N9=Sgv5-N`v%d| zU$0UThetsV78zcm6g{v-@h`H8;s`&1(2Wl1MtJ2+D9ZeI*FPeNBvcN`_4?v)xvU#E zSrD9Q>W(dQ$pF{PH_bgLdxS3!ZH1wU`BwkYqY#8F_e2ivneuza>sf57hg>;$*rcaA z#vM|6gpUD1kFkvPq_(;305GSg!EU&5S7p&jsy;-rSFdjE`)LcI7CadN9`Z6k{n3rU zuNppHif>PA7}bWPdc#9;>CymT8qyKg-Vep+_9#`G1P8Ve>C#26_jaD?uWYHj+b?&# zZwa~Rk%`p0-%dfdeBz&53=Hviq37Qb#Mmj+RXAUoWqRbDZrX7;_A*jP9KQ>X#qtJd zF>Aj~#A`zyp;`p8xZJXGxUm9kw7m zVri}jAXkTH`S^yrOf1#IvJG`t%u8T4>4W*aix#gep4oneAqZ&wwm!!l;IB`x8yv;| z?=LC4Ds>4TOS3Mw&5%uN85m=+ORgQ;B;mKJYA9|@PoB>RBWvv*?<$Y{VIUJ@2(0A^9=8_ zM+Z7?Whi=c5Wo*O#!%&(P8|b9vIQV}7{+_0@CM{gyfr{7mj);#h0E#*fe_vvXe8X#WcoKc^meJBmPue?O9?A(-RMuDwqULnO!?l!&o} z4&cK|Umd8FMK1F*oq-N%>!DN3tqH=K@Wh>SyG>&>Zn$TsKz3t8M_g)+HVT4(g;pQ| zGh0p6GL!Xa|lS4XibQW1qDzzOYOISP%MdJaLUrb*gE z`LgkkJAj7qYHdVtpL@Hk!5|!|jLtxKzhkF*a0rwl-wbAI78aw$(Y2Bob8A5SrjAP#ok}>Q08MEhe zu9rq71Q#iDAmkYiz!K4_V;iq{`1q-jR3wq<6aE!!r>!raJ#48?w7WRS^yHbV0-Nrx zwuk%avgfU`mfw;`9&(Ia{|sS5o+VeOg9|!u7=eI|)fy!g;0`*V?4j+Vk0)+$`Qz;l zhOv{p05GP#lHe})F_7e?nndyUfX#kE>7jG0&)sqMKh(5kr#d{7{v2ozj{&Qkd(yY~ ztTi1bwu9urFZa*x1ZJ8r{wp9RXUVd@SY1B1+lG^m16pshmhei60|bv*w{Bf> zn|IlFu7CVcXZtZ=o?s<(kedKTo9T}YwFT+tnMuNlf#>dvYkQ!x79ASm`e9QUp$F)2 zeTNS)k2OR+p)oTP1@Dcl0EwWB4RWx068i3+0IVzp-+F?5gkpC&*i#gj{8G}(MA9Gi z{4-TMI#`^lI-&v?f|=`1Zhp=d2?2Wx`y+=NnOS?o|Iq#u)!7df25qHq1i%~iIvzF$ zE65C>?0yPZn7MieDD?*z(=u>TD-<%wyT})l?4CPSA2UrZN1yRK=4WQ36Q6c_c>uUS zTcRf$Eux&`b5P|0nErJ8vJkZwd@Q;u353yq>)8&4mKr%n&k^{M%VL!L8aGMrY9Az| zf4YCBCe%cqZ3|+ek3I|)aD9sD60gX^0nY2%8Q|z(?QViET0x!Waf9dR2Jfr|mDA~j zX8LN9?uZaL((1X!WME0aue0&9x%KoeZj{8$DCWrByTgmA*sz=v!Sj27lt1AoBecY)cB50k@U5+LU$z>5kq~l0 z6b}SwJ3x9rYhKwmnhmJ5{9H$Bmxi8$tTVYt6xd9*G?P2 znIwD7LgvaW+sPy|C!uh34#N8pgEl#`6_ep-76%riSQnM~_8j(EcL7RUy8s zuCt=HPvp0ce~8)#W0Bd-QGS?ix~&>t_WL6q+pc`yg)?{)PLtj43gE+Rv;>s(wnjdg zwe0M<$XjL6TW4*!ZZ0+Q?4QwX+6P1@pkQP!IDimvzY`zbNXggg2;%Fe6#Ek%i5|qX zmIQ>@1jt)@4yCB?S%rDgt6g@B@}f8uqDpoDT32dQY{Gw*dR=?jhv0@Ni^z`4<$b7X zk0@XTNtI_h1kaT{>FLyYWl`h3`FO66iR#<9&a}R_&)$DK{??x#vSIQaRDFI7QHC=c z(@FQBOH}G+Yi#;g=sl1#t9pGjHuL4xX9DC|9idW5X5{bB1I|}JXM1ykL3fbYr+6I} z^I0>JeM{)Q^@;N+Ib}8=DU}N`xDx@1XZ~TDSi!TUi`d#lKn{pQpXS?-?Ny15XH~T9 z32d|EV2v7ZtYCj!tg@-j_j^C&B{SBZD!-}g^_bGeeGefKme=j0OY&(Y@WLL~%;Yo8 zFj8`{R;nuId_ryv>pHQWWp_7~eY}j`lMhnAg?iD}Y&UUz@UYG71KBGWUbb zV5miaL?%*!ABV#j`OJ>*pDuh?kc4TuA`##cu_B)n8cG74TXOR@+L&xK8%G2A^C{%a z!c+=ylqZ{mI)UCp%Wmbo4-|i}O&f*pYjv5b3X-}+61x9dmZ87Qh#6-kWe_j0P`t%IbZs-esh1%<8Q!%d-WA_ESmff9v zSzinj`7{W3GpRjI$&| zqzM2wAd(_Zffa z4*$z^@n?sLt$jQe$u{QgYu|Bp2Xl_pG52ZM;9Fmm1k6VYDt4p2!Ws41uch+cq3j;&;9a zPd}0iAN5g{O~fs-lmA_F{Vg|@tmQetfCYI+=tdIjuoD7>w4#Sqn;x&!fDIcXjLBexZw{V!fUWskXx$#7bw935cGTLk_c8 zb8Uciy0#UnX15=wy!QZmpvcyYHq)jqn#>dywiCN};1Q`1;fVT3FRm?V)qq~Vj@kTi z?e%|ZU`M%qy;OASqdt1%iaie}oyIFJeb#J~BFUwIoKaz}VN?FZ=jb~v)!GV(fX>+l z|GJUFhG*>~RNN%AA9ZRinJflljc7NV*Z-zNo+Zd7>d?f^*-Jt7HaYsX`cX2lf{FHF zl+{uR+PQza%*MM!0+l#>eAf*&5fM zqN@TDL;Hh3ssH2)LYlAiPC=rSj;hij*kn8bVm7JXDYUq_DX?@ z-K9+4?XeIfUjI1px!Z!2aZ>KVn5^NNN&Huw=U2QRqP+b>-ZEhMVQD{BO!SN9Pg<3D z;2yFjEUSCm%`Z#pCCY@nm2N>T+@8Z1)R+K4$@puW|MM8h$w!ob(vfZh z`=Qp^-|t~Yc@O06yejAh{VRIHndy1&NhnTqMUE=n9Pcc+55Dz(Bwjyvl+fV<$~vyi zhNC?p&BrZgTGWv*P|h`D%YTCk>dQ98Y*a!wZEO(O7Fb)3P!zuTqT&T1Vhc0; zc$z>S=pCM(RW~}ZX1-=WTOIvAu>9?&-Xj)bO8{HxvqQ zH)9ygiiz0nbnJmgY&FcBD1!`*uh&~X1RwXg*h8$o0<#@Shz*a4YmL7lgrf2228Y4L z3h{u?zYqMxey{X}C1<|!D+kNo*#nn>>g9y5dnzOQ%NuwjyDOyHF3Hu;d>6{*Bfi2g z6TfAWOzQ}_EIZtYmn}FD&wPKP2N7K2508k%trIilpT_>i|8Y-AC#uBgoW@#fewbw? zng)4?uTY6*atcWpP5k!(e)mJ5ek}j zD~}JSU(W0B^LdBrAMQ|=6B{;0#k}H0%Ru`YEUN^upuQ;nA@w_R${eU#yr!H?@4Kyr z4lRcInJrEJQ|B~SDxZjEoP!G=N`ZU?D|vk-e4`sb(Xx)9qVAGcw&i4)Ulx?Obmfo- zp72YMgh%mz|AwwI>Gv+1*;;s-WspOL81q{d^SoKt*kj_3ue9h-awUf(yPsnuZBmzY zJ2^C}m8K!BE>7onX?qjjejO1ydbP{Ct+NV$9cLcMY@B*ixqKSIHBza0?4b0Fgb+47 z;$@c-Vn9WUtkdEd#OHTGq!3rQY~pJ=#U~aYkhd`7w|6BJ_3Ez{SCmff!ZF2D;a5|x zLT>EO9P)|G!OVsvZze-MAz5w%P*;wiI2bc9w0JOx|ij5%MAb+xp8(8pY|N8 zX1pAj)qXO+-YO#Lgo#bH!#P*$Lk0m%!J?+6|O&1&IbJN~3 zL$cFn=b7%&WI!`flT~tRs8sNd7X{RAFC*R4LFx8h=azo9L3i7pH7ngS@3vOQ(~8NU z>~cyUd(7hbX@Aq0(1mxoGvt>dXQ}eiJDv8ulFX8Rem^Z0iwi@q!=N@6zy9!^unD|# zoco?s6V`X@HlXr4ELAsxN)-um>z`RHGGC3ImO|<3^^RA6Pw4Pp_%G|cbiKjF#f!&@ zC%J#qRIp5$!=IfX`z32w6_E#%F#a1A(AgT=^)fyXrc*lj8yD1>wO+$Ah+#nmcqL5njOG-aB2|Uad8ro{rl{=e1iUzL zljc1<1n@t2jk?J1}T1?*9#S`mzTEKQ{OEU0Hj z)~gz3JpbTB^dnKp7ZUq2xyKy{g{GHd8Go!wKtUc-*X^%{o@HI{alUs;h+(07*ccVv zY2^^@#P{AB^%o}vEE;(mREsMc*j@+M54`lE)qT=qHtwkLI!evhxC;_dhi0uJ3?fPu z<)%D`|1hZNv(+n!lRu{$b;)*W#X>2WoR?A3Y4gGUA>vwgpqe}Bsb*_zBDPA@Q3SG2 zYmm0O5F05hF{-vcMO-thE0hXi*#Ev2{d`;5*^3t4uk@YHmwUY&CO|lA^A838j6` zbVvE1aUk?!-{)>DL^Hw|_1N1~dqKuBZdr>W&8pI%|*2fdD3?lN3d;2wG{kX6RH z+>vs3Be7*{u`(rw%+CIAu>s-5u5-xHf3yIuA};!dWvw5&^>4$waibrxzQ21rEMRwR zwJPb$2Kbl84yBs*gRs?v`=<{ozwfo3|DG&}M(S1@=yjy>yL$Wh`&Gi^oAbz(lGGmv zanMdfxn1$8au);X9=#J+~{rEV^wxZBC~dre<98y|rR6>(~Uj_%6- zWH}#(OkG$7ExgKcDawRo1Cb8$8oYj}N2(>%$kmV*RLAKZLh@BpT1}uI{ldl&Op+&DxIeLi0n5FY1}9;`L@;VUxgbhJhid8JgFhC3`U z!nV}OJQYUP(6!nuRtg`X^6w2^m7Vylt882I7%7?NIS+QdgI(blx?9SgF-4iv37r?n zVJ}166Y6!`Y0ZUiLwGfNoLon1HVG>rgN`7F;c0<|imZsW9qQe9tSw&2+^{A6ION!C z`5DHBtV01`5MbZ)UNh1m%qZ%zRbDT`x9T=o#!7!wDYB$PMgFKtO=7Bew4eRnZQdAnFAk#SGabuPeUFonoD51VC@T5_V)KgCvs8@Z( z26U|zt6bS>f^;dwlwK8iOVA)e?*#^Z5j@EbmPp-Y4}8vFRv-Mv&)-N32QTm zuL`SX7s1}xZbX|ok?@!p2=~l%&L~Kvk+G384n^wSWQVm-^&E^=_Oy+)l)4j=jmWb$ z<%2ub`la28rqHorikUjrHU<`m$aUGE37DXcKiS z9TgC@ojmCDE*ZA9^jwxwpR`qr_VO{&!B@8OObQ$ zrcJ$pM0Tcvru{&B)R4vC_1o%8RM%(mI6q=U?6_SDxh(%I^^){bhrW*+#wf+i!)BNC5}3rj+EfmIJ?zA#wX1zNp2L9?o6_P+;8%=ETFfcEN|mfr|V5b7Wj+w+Q5{GX!7ilFc zzF$cCJwR=>^5IV(JWHN-?nh<6v|Lg0gwm7EvzzT0N4%|;zBzI*lxh$H_3Sc|!+4E* zIVSz=+xjG=Zm#|wz1HZl$d0m8Dy1}sV;YScom&enAi^=jVEc^(OXla>#dhF%J2XmbN@tYl-3jZtChG(;Nu@71j(C8w97oc*3` zF|*jdNc(3>lYr*E=kWf@25ScIL>QYTI_H6+$|Do}qXBwc5fob);S{K%!n ztd^Y{8-W(YgkYKca}YO9kuq~_#;dT6btzoRd_*%+M5=%~Tsa>zq?EQm2bES9ME>BUmCHOpKSGx7j8#=wyc|Cka(5_+q^i zDRMYiRtob@y92>i4aw)bOX=OLgN1}uymEB{F^9NoFx*>tmW&L zh;&?SF?qhmlSiG730+O9D{5TRjt-OOe%;^&V9b>*&n*+8N;OlPN&XuL(G`yWqbp@s zD=J>|M|v0oSFSF7HL{^h(hh?SbtgIe1#)MT*uBy9Tl~*Bgm=*p#jDr5Ep`wPG&N(~ zFNXXd4P{+*vHalg&S3o>wT$>01B3Bz!7MyZl^o%U+PQ(CA&)Kogez+^euVsGc8EZ| z(iKq!(|GwWrP92X3%79^k9?I)EFPqrg)7sD5;d%up;NiwGB;yV)tJiA|72+m`(j&H zoLF?}my^L(ZpbAN|3EEZy2kNG#Kx!=IC5?EkC5vsD$^vXPW!)|5?l5H%in17h^0#B zo8u0gO8~cvv}}b2$GOL|tA`A!_T!Lj19fDxLGdb~meEPurmjLyL%uUrPA%I;sTuLL z41Rxb>6B_4UTpFd1eZ`wpvM1lDt?B+e2C4DN&|mjZQsQ5#C0b5Mc{_mpuB!FG7sCO zs>lttg>&mymLIk%#3Cc#epAdB;vFf*>S|}wzOi6mrgRiCh3T8#_&uZ z2gN437R4``6$WOYz&dvCGPBzA<8X6cZZ_vtUiiKoB(V7-W3*@E2eVNT-J(Lj+WPm;T3K6P2g%oy*bHtza zIhd$Taw3#F{Elsv;2w(~Uej5^IX3#QX2npG^hU){CXr*u^thgAY~7rJU-Y0TFV(-< zqy}UPY6Di9xft-ABOaP4+wUwtqzYQ+3-b^2z_%-D4(kn|M7hbU+MZgmbEMD-v@KMd?z?p0lL1m zN%_LCO#HkYmji&6c*CD1LKg}>P1bO40^{G(_6F2A-^uDl*+ zJ*+Ss3iemta*STvCI>{Kn#r*sK$Y-PkPVF-4u)vULLa0%a$6JD!SnKEI)#~JQCwTK zsqqU#s!ic(F@$^4RWNycO8~A<rnmz`(<)hdWqHr~*+`c8VP5 zYCtZ0!c{-wYMT>cfdy}h^Afn%CK0q_io)5jhF!kfs(t)bH2hZ)F2G)GQ zw_i3jULzfRK|yYd6Euy#M9Xd46pn`c3$8m?-TMJ;w1IC?G8{iPDOEmlF{q;F+;F6y zM&zWLH;y@`MvN8Z;h3>%f>;b@a03WsIIFod5k^7cB@NKu9Uouczky(wMdj@#?oiZJ zvoh0k+m_RAY82>fnVkTHOl4#J!sH{+?9b2htP$U|+t+00^_W%5J)dQ65Gd=gbxc`{ ztVeSwR{Yfq9BX>2q(QTcjyasR>}7d2(2_rmqVMOH|std{aAMKaF+C~5SlrdjBQ&KowyLZ^%@w86mwTxbss4c zY44=K$uQ7Ajz7&Ve;%>S;itm@bn+cmKO$C>l(_#HfH9^iLI4cfH&(ERa#w}@%GU&7 zB4?p;y8Bk&@s)l-kKKMMgH@5E#X10e&Pqo`Jj)W$C;xzC0m(0wK{<-~7Vua&rXGOY zZE$+w8#k40a<@g#K*6$D&RQ4WY6g7Ox5|u{(*)NcGad{#u+p#k<3#|ziN z4fP%j0H54Opq;ukf7)%c$g^ z4*m8DFofJBFs2s}X9i%BOn$pM2T_Q6a@9MmtrC;sgtnW3@m?0~PrR%Pzbpv7zY{P+ z?nAwe10a^M1~jUyZu**}?JA9`H}%TTNm7NhOmN)(E?XTex6!IYA#Y}YTtK49W7izn zRrNGz!LwXzvjBxa4nn#V*(B06;K})!HdiS`E%vK%i^0`7h@64APPeqbyoL2ou3$!# z(3b6}CB+95>#(RLjQcXsk|ajOa2DiUpZ~7=unO5RwpuPzjQs<|XAY$4uuSJoy(!&D zMZua5MoTK~vUz2IkU1u1ccNH?a*ZPo_NFCtk9ZqqLclM`q1wc5f=}7534#Eb5%xhg3CsJ1r_R8KDV*uwBvT-nvS z>&nD{`$>KAd%h-t7S`EyNZ!q-jkW_if&Mwgzy*ybx&(c!V)xktZ!&4qzW8JGc=Zk`iZDzD<#U432Z_Ty6` zJk3I)n>>o)(QP`dX9!R@i5uptnnZL5KwPf#k$87EGt6ti*92BBm207X89eAqiC>1( z1_-g3fdl}_o4Kqz05PiSX}#uaGQ~Ai^%V=eqQtyi%IdVAh^FF*Lz-+3kZ2h-gmFn7 z7#2=%56PEUA5=8)dGLbios3^hlVE;DC}o8oP|B>yh{xV44 zR77P0LRCnl6yT5-wrmKU!Xe*P;WC2>*}t&4k=Z5+xR)~V*T$d2e2z4DK%2?~qD-v} zpde{T$f^i{_O{&5b*6wL=H5t>Kr2H;U8GkbBy)h{0k=(fCYi)mx)JA4MRz8>T%GgS zJ*^GW0H~16#Ps&x&^t}@FVXC0agidNgQ5wGLf)^=@wrgnd2#O%q`KCfTMmGP12yP9 z{b9BLh*xfwU?Xm!pCt@#wYLCLxf=ma)tnkIWeD^dJdkoj=)QrTrE@tf0#a=3wslD6 zZBG6tRLMt^+Os^#)HXp;>Z7OM(&w2E^1l5Y2IH4qU97zqju2A=JMC{=mtN^{CNK%% zQ0wdD=^iUnhSE9sJP)49id{JS%XhC*^V_2s69m~B!4`pH?<%_gF44Z-2teXDk)NBY z&{>!ulqI~f?yk_yds&8Q^xWjDNQlL~xT3RkLU&W;?q@w(aHQPna-5R^4lEq_05#FS z^CxZh$Xs1C12eg<;Nf^O_?yddWp~MgZ&#HyiEK;;uFBO?^q40h-{U@r$eykkE36e7@Ler356(SanoeD;|l{{@h#yFrEQ` z(w5uqXy33$^j{Mw2|-N7&UWCXcB4`4Q*e{N8|yVFt$8UIg|FIqJsDJr?)BJeyyRL@ z{ep}&{DV06mG*ceA_B(H>Zdbo`$iEx_Ha~;KAbFr6+GXx^J#Y6p1nmz_V}wQ&0~NptKeG0SHH>66 zFO$lAmXHo^`O{&2B^`L7h)n5*nIlsMJo^*rysZm->lwB0cVes(LD#+@B!g3Hnc?!F&MxRGI3fiR{-U-7txf|A7ieWg@msK zeIj1bpzK9qNny{5JwVBF^igQ6KRdedFka9cpu-U}ItAQ#hMJ$=2)m}@33rQVl%iHX z=0w`&qRi^{Lf`nls@5D_aZa4$nZ4^z|1+AV5BAK8z zPjkbc)PYcDkg~f4pO4|CsT0_X*H2~T0lG@DR|xV5 zzLM?!EWAI90wLFzx+y4luegT=pI^wroc%tVvtBZ`ltt0rFR-{`4;u_>!AB;~2^KBF zw$803gnkag=tX3$X_Om(b=nS~wl6=qaHXOM?|DbQi7f6CAzYo^q)<&c&LYTCfk8|< zr(&}89$l75@pu=yIWv>ivIhwoggZJaq!9X#gWy5rZs#W74Q~TxBr;G3Cj)0bDO@8HQUB&@W1E`F=xp%p!T|lQ$@im!m zkk?n9C)038pTWF-3m~v3^HzNxNT;b~NCNu_Z`r~PLnLBNZ?haR_5esXOb*dwv?qr) z9K25y__el7NAvXkZ7D4a_r-eky^_1#-~UZBj=MV98f^80TS zQ#-Z}ja|=b@n18^A<5hV#oAXdwGW(4A8nU|HLRs*tD!XFi;LmvROvV@B10R&7^^bb z+ZYCTM2^+%zI`PVtpg2RrWr59W%sYQs-Rj)#MO4$XGNt+J=DQ{Y7~%~yqVgD&a6f< zZb%*NP6w8BaZiTz?lQBmz4af=6f^Li4z7tMGpWQRh>cORI~7(TswAriRxd$^X+A1I zmUPX!Ge?zluTssZ4Lfosu=A3SPL0VZh*y*?JNX&5)d^R-!`N9s({pu!e~x-F=YlM>m1(#c~ z>)(4mJY8C&U~Ck&H(_%8fGzF;&{Zs!!qO}!-+Qq&EdQt7XalIVNG;q5sFs$kZ{4>`*zACNS4~DVJAD6b|cP{+Y3EMxhV3&9D z&OkmY@3i(iZX(j&n=Y%Posozc)3xy8MxZ?OQrOJDm`z15M6^RsD#1dj$GH3by+f$z z`oM6b)zTbLA%HqwtCg-kuVqF(#ng{#vqS2KZ2+9j1_(P!h_V}yF2EE6^DU~j&N49J zResKjJ1N8^kOy(ZV-}KW+fbo;^iezae76Yj5n7aY!7c(PGd*Cn?kxMRg7WX=Tc-Dz zp-7V_w6Oh_knEIjkkRZIP5yvKdM~b$6c79F@4I&_X;zS>A4&NXbc=g*SNqqrId;bY zy>?1Z2kHVd!Z@hSuiBVVy|12MhJsJ$G2@d=vKA7RdESu#L+*quX!x8RJGWIMlBL9J zdtOo{&V@QpsBLo zX-E&GsD6quktM?G*w)$6Cl`P{9`EUNWRjU6U8DyEkbDccb|T^a-W*xz6t|;9?>Orx zxK`(6O<|*2L^j1HW*3myS|ZANC^%`P2If*+u<3_Znv!#hf)+dA>p)WKITS(#QH7Gc zINxPjm#9CD<)_!W|4-Wf$Efa*RMqo?tb7g>;2b_v$ryhSloO9k>H{e{rr;~8Y=yvg zf*(g0dOc4`AEeNdV_4T0X%L`-YO{>Uj(}!)5z-_@uq2i1o|EQkM8pWhq8c4FKH}nQ zX)yWJ2XGq@If*}z9Vz6W>bh8j2S7|L$dv|`9*~PzIe9Q7PHW*$iQOaKBh@2MjL9nk zStf49fX#bvyernGmdIWLMsAwEp_EmyfCJI^T|)n^782zOemgcWNs8xJKN@&$In;Qj zjlp5uh1Er@4{sX=!&i*HBOt6~`{LVcur!C~Z{Wj<3 zYn+6@5Pd$_F*EZ(>B-yxy5d$8m{{Ws;&~b|W{aX3ufhM>?Q5`D4#c0$USw=p#?A&y zBwg47nP479WPd{;;PV$hOgYXRx}TRCF~>MXR7d0=d>*)479CHdXIc{3r4Yv!ovC&; zh2SFVcX~f+CpP2IHyup5l@S2VYmrdkC(#taz;4CCiH_nZ#EgvKEJTkLvK4Q#i6Li%?^i9kRw~c?J0eoH>U@M>7U?C zeTC>WE0rDEdkoBF@3nJ8aYULdR%$`0o-IBlf`Jtr2hfhwEAdSFM1zE`MR<~dq%?+Y zO&IqiX5(U+WDNGji}B}SnU3MPulYRi`Xq;?!QmLZrF3tWZoJT15jF--6f0-ng`b)C zNJJcw1>Xrlrj6Li_Y-jX(!Db`e|o{^VM~j?+ukgLC&vq+23R+MR3f=Sh~Jm2j>YSH z?!&`#;e7-K2KSZNo4f1i8`XCSWLi7bEQHAVekq@by2fwa#5~j+bJMEHfU$?bXpq8H zYfV_4=HhW)qC*q8Z_DkIH0q(mCsz525PjA5XNgD*zK_b{anfkRx;!oXq2)TEYwS*+ zrd07o;@x`NinoTM@bE29vyyhhHK$+ zhHIt3LWlD7TcNW}dYCY>cnlNAwiI!iq`6gCdD_xE`A~;*xI)+%yGNVo zbnI%RL-zbxbGVu`+(;``<2UI$Mq%-2)`2JO(fejE_aDcEQg)2#`azrL3-*@AOCL&z z08yOr16Dzeh2d|{8)0-$Qg$VTV(=rT@au=q())?cYS`;>S`QZnZ%l=|y{;_8FnKwAb;3r6eKS5bn6Y)6`@_hQ({%aVA~{B0~mx z8`E$~39m(N7=NxSXBj>h@k1vt^VqvJOoGWEnFOl2TTWTdr2w&+1GY2hH5SK75}hW8 zcp`K?Pk7kfR|Eo)+QDV|_nQ};mrTe|X!!o~gW^;Amxsx4&n&3}W3Ye92AmJEG034>3$BZT-cdD<$ z>CMIm`ErTnRZ)gHx8@Ul{cQ4~@S%t>%$azjnbcAmpGmD=$gHSS+cQvxTQBS9 z#}S*k@K~!!*@bL9@I7tc*vSbF^nipFSq?`ckzg5}Kx1Qa{A#ewKi)b|H+S54Iqn#> zVcg=?8<|lOgbf8JyB1IGU4)&FJNhOefkF0#O!N|!uh$|m%=`0KHw#0^(XsPGHw~X! zix)Cw4QJi$FdUHivBgX9UmwWHu^C;^c{F|mLwYn@`NsbQuC#(^P3e>OCy1qj$ukS% zZW%&2atQIgvw6dS0Px~-gW-a<_nlSHk156c{--P*{2u$E^DF3r_;xB7@*i7>*Z<$u zIcTv8VKRTC365P315vhV`o|F_r2Y9W@=D0lT)3)wZF3)yrjA`vsrXkn4n@ z#X#5jx8FSlkLJKAMk0|LA&I|x%G@1NLG_`w+h5+pgbz6w!G&{whhhJl=$~skHw6xaL&s Date: Wed, 8 Mar 2017 09:57:03 +0800 Subject: [PATCH 2/3] update --- kubernetes/networking/README.md | 0 kubernetes/networking/service-network.md | 2 +- 2 files changed, 1 insertion(+), 1 deletion(-) create mode 100644 kubernetes/networking/README.md diff --git a/kubernetes/networking/README.md b/kubernetes/networking/README.md new file mode 100644 index 0000000..e69de29 diff --git a/kubernetes/networking/service-network.md b/kubernetes/networking/service-network.md index b02c2de..888c583 100644 --- a/kubernetes/networking/service-network.md +++ b/kubernetes/networking/service-network.md @@ -8,7 +8,7 @@ Pod的IP是动态分配在**docker0**所在网段的,当发生重启，扩容等 * **ClusterIP**:提供一个集群内部的虚拟IP以供Pod访问。 * **NodePort**:在每个Node上打开一个端口以供外部访问。 -* **LoadBalancer**:通过外部的负载均衡器来访问。 +* **LoadBalancer**:通过负载均衡器来访问。 ### ClusterIP && NodePort && LoadBalancer #### 1. ClusterIP From 2e8d2379b316288f12e37e312443465d9fe3c6d1 Mon Sep 17 00:00:00 2001 From: Yancey1989 Date: Wed, 8 Mar 2017 14:58:07 +0800 Subject: [PATCH 3/3] update --- README.md | 4 ---- kubernetes/networking/service-network.md | 13 ++++++++++--- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 5fab223..a9a2836 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,3 @@ -<<<<<<< HEAD -# tutorials -======= # Kubernetes实战教程 * 一个运行Paddle Job的例子 @@ -64,4 +61,3 @@ * AI开发平台 * ServerLess服务 * 大数据和AI ->>>>>>> 9b4d60bfd4a264633cc3edaaa8b26df9827e3a83 diff --git a/kubernetes/networking/service-network.md b/kubernetes/networking/service-network.md index 610c894..9bd77a0 100644 --- a/kubernetes/networking/service-network.md +++ b/kubernetes/networking/service-network.md @@ -1,7 +1,10 @@ -# Service的网络模型 +# Service网络模型 + +## Service简述 +### 什么是Service? + 在Kubernetes中Servie是一种抽象资源类型，在逻辑上通过[Label selectors](https://kubernetes.io/docs/user-guide/labels/#label-selectors)的机制定义了一组Pod以及对这一组Pod的访问方式。 -## 为什么要有Service 在Kubernetes的Node上，Pod的IP地址是被动态分配在**docker0**所在网段的,当发生重启，扩容等操作时，IP地址会随之变化。当某个Pod(frontend)需要去访问其依赖的另外一组Pod(backend)时，如果backend的IP发生变化时，如何保证fronted到backend的正常通信变的非常重要。为了解决此类问题,Kubernetes设计了Service的概念。 在实际生产环境中，对Service的访问可能会有两种来源： @@ -14,6 +17,10 @@ * **NodePort**:在每个Node上打开一个端口以供外部访问。 * **LoadBalancer**:在某几个node上开放端口，并通过负载均衡器来供外部访问。 +## 什么是Endpoint? +上面讲到每个Service定义了的一组逻辑上的Pod,这些Pod的IP地址被称为endpoint，通常情况下endpoint是通过[label-selectors](https://kubernetes.io/docs/user-guide/labels/#label-selectors))自动选择的，但在一些特殊的场景下，也可以人工制定一组endpoint。 + +## Service的几种类型 ### ClusterIP 此模式会提供一个集群内部的虚拟IP（与Pod不在同一网段)，以供集群内部的pod之间通信使用。 @@ -48,7 +55,7 @@ ClusterIP也是Kubernetes service的默认类型。 ``` 能够看到Kubernetes为名尾nginx-service的Service创建了一个clusterIP,地址为10.0.0.252. - 在本文开头处提到每个Service定义了的一组逻辑上的Pod,他们的IP地址称为**endpoint**,通过`get endpoint`命令能够看到这一组endpoint的IP地址,分别对应了nginx的三个Pod的IP。 + 通过`get endpoint`命令能够看到这一组endpoint的IP地址,分别对应了nginx的三个Pod的IP。 ```bash yancey@ yancey-macbook kubernetes-1$kubectl get endpoint NAME ENDPOINTS AGE