ETL in AWS - Glue, Amazon Data Pipeline

-
https://aws.amazon.com/blogs/big-data/implement-continuous-integration-and-delivery-of-serverless-aws-glue-etl-applications-using-aws-developer-tools/

https://aws.amazon.com/blogs/compute/implementing-dynamic-etl-pipelines-using-aws-step-functions/


https://aws.amazon.com/blogs/big-data/etl-processing-using-aws-data-pipeline-and-amazon-elastic-mapreduce/





Komentarze

Prześlij komentarz

Popularne posty z tego bloga

Kubernetes

-
Obraz
I. Introduction 1. Kubernetes Overview - Kubernetes = popular container orchestrator  * it lets you schedule containers on a cluster of machines  * you can run multiple containers on one machine  * you can run long running services (like web applications)  * Kubernetes will manage the state of those containers:    ** Can start the container on specific nodeskubectl create -f pod-demo.yml    ** Will restart a container when it gets killed    ** Can move containers from one dnode to another node - you can run Kubernetes anywhere:   *on-premise (own datacenter)   *public (Google cloud, AWS)   *hybrid: public & private -highly modular -open source -backed by google -Container Orchestration = Make many servers act like one -Released by Google in 2015, maintained by large community -Runs on top of Docker (usually) as a set of APIs in containers -Provides API/CLI to managed containers across server...
Czytaj więcej

Helm

-
 0. Overivew - Packet manager 1. Helm Provenance and Integrity - https://github.com/technosophos/k8s-helm/blob/master/docs/provenance.md - Helm has provenance tools which help chart users verify the integrity and origin of a package. Using industry-standard tools based on PKI, GnuPG, and well-respected package managers, Helm can generate and verify signature files. a) Overview   -  I ntegrity is established by comparing a chart to a provenance record. Provenance records are stored in   provenance files , which are stored alongside a packaged chart. For example, if a chart is named   myapp-1.2.3.tgz , its provenance file will be   myapp-1.2.3.tgz.prov .  -  Provenance files are generated at packaging time ( helm package --sign ... ), and can be checked by multiple commands, notable   helm install --verify . b) The workflow This section describes a potential workflow for using provenance data effectively. Prerequisites: A valid PGP keyp...
Czytaj więcej

Ansible Tower / AWX

-
- Ansible Tower provides a web server interface to ansible.  - System rewuierments are somewhat heavy.  - Tower is only free for minimal use. Working with more than a few system requires a paid lincense.  - The two keys benefots of Ansible Tower are user permissioning and the audit trail (only provided with license.  - Only touched on in EX4077  - How is ansible Tower installed on your system? Ansible Tower is provided in a tarball containing binaries, config files, and an installation script that must be ran.  - To populate an Ansible Tower project with source files can be used:        * file system of the Ansible Tower server        * git        * subversion  - features:        * an interface for running Ansible plays and playbooks against target hosts.       * Separate user accounts permissioned for selective access to Tower-managed ...
Czytaj więcej