-
Notifications
You must be signed in to change notification settings - Fork 40
/
flink_eks.html
34 lines (34 loc) · 14.4 KB
/
flink_eks.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"/><meta name="viewport" content="width=device-width, initial-scale=0.8"/><title>Flink on Amazon EKS</title><link rel="stylesheet" type="text/css" href="scribble.css" title="default"/><link rel="stylesheet" type="text/css" href="racket.css" title="default"/><link rel="stylesheet" type="text/css" href="manual-style.css" title="default"/><link rel="stylesheet" type="text/css" href="manual-racket.css" title="default"/><link rel="stylesheet" type="text/css" href="extra.css" title="default"/><link rel="stylesheet" type="text/css" href="fancyverb.css" title="default"/><script type="text/javascript" src="scribble-common.js"></script><script type="text/javascript" src="manual-racket.js"></script><!--[if IE 6]><style type="text/css">.SIEHidden { overflow: hidden; }</style><![endif]--></head><body id="scribble-racket-lang-org"><div class="tocset"><div class="tocview"><div class="tocviewlist tocviewlisttopspace"><div class="tocviewtitle"><table cellspacing="0" cellpadding="0"><tr><td style="width: 1em;"><a href="javascript:void(0);" title="Expand/Collapse" class="tocviewtoggle" onclick="TocviewToggle(this,"tocview_0");">►</a></td><td></td><td><a href="index.html" class="tocviewlink" data-pltdoc="x"><span style="font-weight: bold">Flock:<span class="mywbr"> </span> A Serverless Streaming SQL Engine</span></a></td></tr></table></div><div class="tocviewsublisttop" style="display: none;" id="tocview_0"><table cellspacing="0" cellpadding="0"><tr><td align="right"></td><td><a href="Introduction.html" class="tocviewlink" data-pltdoc="x">Introduction</a></td></tr><tr><td align="right"></td><td><a href="Benchmarks.html" class="tocviewselflink" data-pltdoc="x">Benchmarks</a></td></tr></table></div></div><div class="tocviewlist"><table cellspacing="0" cellpadding="0"><tr><td style="width: 1em;"><a href="javascript:void(0);" title="Expand/Collapse" class="tocviewtoggle" onclick="TocviewToggle(this,"tocview_1");">▼</a></td><td></td><td><a href="Benchmarks.html" class="tocviewlink" data-pltdoc="x">Benchmarks</a></td></tr></table><div class="tocviewsublist" style="display: block;" id="tocview_1"><table cellspacing="0" cellpadding="0"><tr><td align="right"></td><td><a href="flink_intro.html" class="tocviewlink" data-pltdoc="x">Introduction to Apache Flink</a></td></tr><tr><td align="right"></td><td><a href="flink_ec2.html" class="tocviewlink" data-pltdoc="x">Flink on EC2</a></td></tr><tr><td align="right"></td><td><a href="" class="tocviewselflink" data-pltdoc="x">Flink on Amazon EKS</a></td></tr><tr><td align="right"></td><td><a href="flink_kinesis.html" class="tocviewlink" data-pltdoc="x">Amazon Kinesis Data Analytics for Apache Flink</a></td></tr><tr><td align="right"></td><td><a href="flink_nexmark.html" class="tocviewlink" data-pltdoc="x">Nexmark Benchmark for Apache Flink</a></td></tr></table></div></div><div class="tocviewlist"><table cellspacing="0" cellpadding="0"><tr><td style="width: 1em;"><a href="javascript:void(0);" title="Expand/Collapse" class="tocviewtoggle" onclick="TocviewToggle(this,"tocview_2");">►</a></td><td></td><td><a href="" class="tocviewselflink" data-pltdoc="x">Flink on Amazon EKS</a></td></tr></table><div class="tocviewsublistbottom" style="display: none;" id="tocview_2"><table cellspacing="0" cellpadding="0"><tr><td align="right">1 </td><td><a href="#%28part._.Overview%29" class="tocviewlink" data-pltdoc="x">Overview</a></td></tr><tr><td align="right">2 </td><td><a href="#%28part._.Amazon_.E.K.S%29" class="tocviewlink" data-pltdoc="x">Amazon EKS</a></td></tr><tr><td align="right">3 </td><td><a href="#%28part._.A.W.S_.Spot_.Instances%29" class="tocviewlink" data-pltdoc="x">AWS Spot Instances</a></td></tr><tr><td align="right">4 </td><td><a href="#%28part._.Flink_.Deployment%29" class="tocviewlink" data-pltdoc="x">Flink Deployment</a></td></tr><tr><td align="right">5 </td><td><a href="#%28part._.Conclusion%29" class="tocviewlink" data-pltdoc="x">Conclusion</a></td></tr><tr><td align="right"></td><td><a href="#%28part._.Reference%29" class="tocviewlink" data-pltdoc="x">Reference</a></td></tr></table></div></div></div><div class="tocsub"><div class="tocsubtitle">On this page:</div><table class="tocsublist" cellspacing="0"><tr><td><span class="tocsublinknumber">1<tt> </tt></span><a href="#%28part._.Overview%29" class="tocsubseclink" data-pltdoc="x">Overview</a></td></tr><tr><td><span class="tocsublinknumber">2<tt> </tt></span><a href="#%28part._.Amazon_.E.K.S%29" class="tocsubseclink" data-pltdoc="x">Amazon EKS</a></td></tr><tr><td><span class="tocsublinknumber">3<tt> </tt></span><a href="#%28part._.A.W.S_.Spot_.Instances%29" class="tocsubseclink" data-pltdoc="x">AWS Spot Instances</a></td></tr><tr><td><span class="tocsublinknumber">4<tt> </tt></span><a href="#%28part._.Flink_.Deployment%29" class="tocsubseclink" data-pltdoc="x">Flink Deployment</a></td></tr><tr><td><span class="tocsublinknumber">5<tt> </tt></span><a href="#%28part._.Conclusion%29" class="tocsubseclink" data-pltdoc="x">Conclusion</a></td></tr><tr><td><span class="tocsublinknumber"></span><a href="#%28part._.Reference%29" class="tocsubseclink" data-pltdoc="x">Reference</a></td></tr></table></div></div><div class="maincolumn"><div class="main"><div class="versionbox"><span class="version">7.9</span></div><div class="navsettop"><span class="navleft"><div class="nosearchform"></div> </span><span class="navright"> <a href="flink_ec2.html" title="backward to "Flink on EC2"" data-pltdoc="x">← prev</a> <a href="Benchmarks.html" title="up to "Benchmarks"" data-pltdoc="x">up</a> <a href="flink_kinesis.html" title="forward to "Amazon Kinesis Data Analytics for Apache Flink"" data-pltdoc="x">next →</a></span> </div><h4><a name="(part._flink_eks)"></a>Flink on Amazon EKS</h4><table cellspacing="0" cellpadding="0"><tr><td><p><span class="hspace"> </span><a href="#%28part._.Overview%29" class="toclink" data-pltdoc="x">1<span class="hspace"> </span>Overview</a></p></td></tr><tr><td><p><span class="hspace"> </span><a href="#%28part._.Amazon_.E.K.S%29" class="toclink" data-pltdoc="x">2<span class="hspace"> </span>Amazon EKS</a></p></td></tr><tr><td><p><span class="hspace"> </span><a href="#%28part._.A.W.S_.Spot_.Instances%29" class="toclink" data-pltdoc="x">3<span class="hspace"> </span>AWS Spot Instances</a></p></td></tr><tr><td><p><span class="hspace"> </span><a href="#%28part._.Flink_.Deployment%29" class="toclink" data-pltdoc="x">4<span class="hspace"> </span>Flink Deployment</a></p></td></tr><tr><td><p><span class="hspace"> </span><a href="#%28part._.Conclusion%29" class="toclink" data-pltdoc="x">5<span class="hspace"> </span>Conclusion</a></p></td></tr><tr><td><p><span class="hspace"> </span><a href="#%28part._.Reference%29" class="toclink" data-pltdoc="x">Reference</a></p></td></tr></table><h5>1<tt> </tt><a name="(part._.Overview)"></a>Overview</h5><p>Flink supports different deployment modes when running on <a href="https://kubernetes.io/">Kubernetes</a>. We
will show you how to deploy Flink on Kubernetes using the
<a href="https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/deployment/resource-providers/native_kubernetes/">Native Kubernetes Deployment</a>.</p><h5>2<tt> </tt><a name="(part._.Amazon_.E.K.S)"></a>Amazon EKS</h5><p><a href="https://aws.amazon.com/eks/">Amazon EKS</a> is a fully managed Kubernetes service. EKS supports creating and managing spot instances using Amazon EKS
managed node groups following Spot best practices. This enables you to take advantage of the steep savings and scale
that Spot Instances provide for interruptible workloads. EKS-managed node groups require less operational effort
compared to using self-managed nodes.</p><p>Flink can run jobs on Kubernetes via Application and Session Modes only.</p><h5>3<tt> </tt><a name="(part._.A.W.S_.Spot_.Instances)"></a>AWS Spot Instances</h5><p>Flink is distributed to manage and process high volumes of data. Designed for failure, they can run on machines with
different configurations, inherently resilient and flexible. Spot Instances can optimize runtimes by increasing
throughput, while spending the same (or less). Flink can tolerate interruptions using restart and failover strategies.</p><p>Job Manager and Task Manager are key building blocks of Flink. The Task Manager is the compute intensive part and
Job Manager is the orchestrator. We would be running Task Manager on Spot Instances and Job Manager on On Demand
Instances.</p><p>Flink supports elastic scaling via <a href="https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/elastic_scaling/">Reactive Mode</a>.
This is ideal with Spot Instances as it implements elastic scaling with higher throughput in a cost optimized way.</p><h5>4<tt> </tt><a name="(part._.Flink_.Deployment)"></a>Flink Deployment</h5><p>For production use, we recommend deploying Flink Applications in the <span style="font-weight: bold">Application Mode</span>, as these modes provide a
better isolation for the Applications. We will be bundling the user code in the Flink image for that purpose and
upload in <a href="https://aws.amazon.com/ecr/">Amazon ECR</a>. Amazon ECR is a fully managed container registry that
makes it easy to store, manage, share, and deploy your container images and artifacts anywhere.</p><ol><li><p><div class="SIntrapara">Build the Amazon ECR Image
</div><div class="SIntrapara"><ul><li><p>Login using the following cmd and don’t forget to replace the AWS_REGION and AWS_ACCOUNT_ID with your details.</p></li><li><p>Create a repository.</p></li><li><p>Build the Docker image.</p></li><li><p>Tag and Push your image to Amazon ECR.</p></li></ul></div></p></li><li><p><div class="SIntrapara">Create Amazon S3/Amazon Kinesis Access Policy
</div><div class="SIntrapara"><ul><li><p>We must create an access policy to allow the Flink application to read/write from Amazon S3 and read Kinesis data streams.
Run the following to create the policy. Note the ARN.</p></li></ul></div></p></li><li><p><div class="SIntrapara">Cluster and node groups deployment
</div><div class="SIntrapara"><ul><li><p>Create an EKS cluster. The cluster takes approximately 15 minutes to launch.</p></li><li><p>Create the node group using the nodeGroup config file. We are using multiple nodeGroups of different
sizes to adapt Spot best practice of diversification. Replace the <<Policy ARN>> string using the ARN string
from the previous step.</p></li><li><p>Download the Cluster Autoscaler and edit it to add the cluster-name.</p></li></ul></div></p></li><li><p><div class="SIntrapara">Install the Cluster AutoScaler using the following command: kubectl apply -f cluster-autoscaler-autodiscover.yaml
</div><div class="SIntrapara"><ul><li><p>Using EKS Managed node groups requires significantly less operational effort compared to using self-managed
node group and enables: 1) Auto enforcement of Spot best practices. 2) Spot Instance lifecycle management. 3)
Auto labeling of Pods.</p></li><li><p>eksctl has integrated <a href="https://github.com/aws/amazon-ec2-instance-selector">amazon-ec2-instance-selector</a>
to enable auto selection of instances based on the criteria passed.</p></li></ul></div></p></li><li><p><div class="SIntrapara">Create service accounts for Flink
</div><div class="SIntrapara"><blockquote class="Rfilebox"><p class="Rfiletitle"><span class="Rfilename"><span class="emph">shell</span></span></p><blockquote class="Rfilecontent"><p><div class="sourceCode" id="cb1"><pre class="sourceCode bash"><code class="sourceCode bash"><span id="cb1-1"><a href="#cb1-1" aria-hidden="true" tabindex="-1"></a>$ <span class="ex">kubectl</span> create serviceaccount flink-service-account</span>
<span id="cb1-2"><a href="#cb1-2" aria-hidden="true" tabindex="-1"></a>$ <span class="ex">kubectl</span> create clusterrolebinding flink-role-binding-flink <span class="kw">\</span></span>
<span id="cb1-3"><a href="#cb1-3" aria-hidden="true" tabindex="-1"></a><span class="ex">--clusterrole</span>=edit --serviceaccount=default:flink-service-account</span></code></pre></div>
</p></blockquote></blockquote></div></p></li><li><p>Deploy Flink</p><p>This install folder here has all the YAML files required to deploy a standalone Flink cluster. Run the install.sh
file. This will deploy the cluster with a JobManager, a pool of TaskManagers and a Service exposing JobManager’s
ports.</p><ul><li><p>This is a High-Availability(HA) deployment of Flink with the use of Kubernetes high availability service.</p></li><li><p>The JobManager runs on OnDemand and TaskManager on Spot. As the cluster is launched in Application Mode,
if a node is interrupted only one job will be restarted.</p></li><li><p>Autoscaling is enabled by the use of Reactive Mode. Horizontal Pod Autoscaler is used to monitor the CPU
load and scale accordingly.</p></li><li><p>Check-pointing is enabled which allows Flink to save state and be fault tolerant.</p></li></ul></li></ol><h5>5<tt> </tt><a name="(part._.Conclusion)"></a>Conclusion</h5><p>In this post, we demonstrated how you can run Flink workloads on a Kubernetes Cluster using Spot Instances,
achieving scalability, resilience, and cost optimization.</p><h5><a name="(part._.Reference)"></a>Reference</h5><ol><li><p>Kinnar Sen. Optimizing Apache Flink on Amazon EKS using Amazon EC2 Spot Instances. AWS Compute Blog. 11 NOV 2021. <a href="https://aws.amazon.com/blogs/compute/optimizing-apache-flink-on-amazon-eks-using-amazon-ec2-spot-instances/">https://aws.amazon.com/blogs/compute/optimizing-apache-flink-on-amazon-eks-using-amazon-ec2-spot-instances/</a>.</p></li></ol><div class="navsetbottom"><span class="navleft"><div class="nosearchform"></div> </span><span class="navright"> <a href="flink_ec2.html" title="backward to "Flink on EC2"" data-pltdoc="x">← prev</a> <a href="Benchmarks.html" title="up to "Benchmarks"" data-pltdoc="x">up</a> <a href="flink_kinesis.html" title="forward to "Amazon Kinesis Data Analytics for Apache Flink"" data-pltdoc="x">next →</a></span> </div></div></div><div id="contextindicator"> </div></body></html>