DevSecOps Notes

EBS - Diffrent Volume Types:  - General Purpose (SSD) - gp2  - Provisioned IOPS (SSD) -io1  - Throughput Optimized (HDD) -st1  - Cold (HDD) -sc1 Compare Volume Types A) Solid-State Drives (SSD)

System Status Checks (Checks Host) -Loss of network connectivity -Loss of system power -Software issues on the physical host -Hardware issues on the physical host -Best way to resolve issues is to stop and then start the VM again Instance Status Checks (Checks VM) -Failed system status checks -Misconfigured networking or startup configuration -Exhausted memory -Corrupted file system -Incompatible kernel -Best way to trouble shoot is by rebooting the instance or by making  modification in your operating system

1. )Amazon Cloud Watch - is monitoring service to monitor your AWS resources, as well as the application that you run on AWS. CloudWatch can monitor things like: a) Compute -Autoscaling Groups -Elastic Load Balancers -Route53 Health Checks b) Storage & Content Delivery -EBS Volumes -Storage Gateways -CloudFront c) Databases & Analytics -DynamoDB -Elasticache Nodes -RDS Instances -Elastic MapReduce Job Flows -Redshift d) Other -SNS Topics -SQS Queues -Opsworks -CloudWatch Logs -Estimated Charges on your AWS Bill Host Level Metrics Consist of: -CPU   * CPUUtilization -Network  * NetworkIn  * NetworkOut  * NetworkPacketsIn  * NetworkPacketsOut -Disk  * DiskReadOps  * DiskWriteOps  * DiskReadBytes  *D iskWriteBytes -Status Check  * StatusCheckFailed  * StatusCheckFailed_Instance  * StatusCheckFailed_System Tip: -RAM Utilization is a custom metric. -By default EC2 m...

https://aws.amazon.com/certification/certified-sysops-admin-associate/ 80 min 70% secure pass exam 1) Monitoring & Metrics 2) High Availability 3) Analysis 4) Deployment & Provisioning 5) Data Management 6) Security 7) Networking

https://circleci.com/blog/testing-docker-images-with-circleci-and-goss/

https://aws.amazon.com/blogs/database/viewing-amazon-elasticsearch-service-slow-logs/

https://hackernoon.com/run-bash-or-any-command-in-a-docker-container-9a1e7f0ec204 https://stackoverflow.com/questions/18497688/run-a-docker-image-as-a-container https://docs.docker.com/v1.7/reference/run/ networking : https://stackoverflow.com/questions/17157721/how-to-get-a-docker-containers-ip-address-from-the-host docker exec -i -t container_name /bin/bash

1. PostgresSQL Overview a) Database and RDBMS Introduction - Database - " A database is an organized collection of data, generally stored  and access electronically from a computer system." - RDBMS - A relational database follows the relational model proposed by E.f. Codd in 1970. It's a collection of data organized into tables, columns, and rows. These tables have well-defined relationships. A relational database management system (RDBMS) is software that enables us to interact with and manage relational databases RDBM Server = Instance + Database Instance = Memory + Processes Database = Disk ( Data, Logs, Config) Primary Key Foreign Key  b) PostgreSQL History - 1986 - Micheal Stonebraker leads the University of California at Berkeley POSTGRES project. - 1994 - An SQL language interpreter is added , and Postgres95 is released to the web. - 1996 - The name is change to "PostgreSQL" and new features are added by the community over time. c) PostgreSQL Architect...

https://itglue.com/

https://www.reddit.com/r/devops/comments/8sqgp3/6_months_to_try_get_a_devops_role/ http://www.devopsmastery.com/ https://devopscube.com/

https://aws.amazon.com/blogs/database/get-started-with-amazon-elasticsearch-service-use-amazon-cognito-for-kibana-access-control/?nc1=b_rp

https://www.npmjs.com/package/mongoosastic#indexing-an-existing-collection

https://aws.amazon.com/elasticsearch-service/faqs/ Curator lambda for es: 1) https://github.com/gergo-dryrun/curator-lambda-aws-es/blob/master/code/curator_lambda/curator_lambda.py 2) https://gist.github.com/cjuroz/d45f4d73e74f068892c5e4f3d1c7fa7c/revisions 3) https://github.com/elastic/curator/issues/804 4) https://gist.github.com/jarpy/f65b24209e0a9240207599a73ce99bc9#file-serverless-curator-py 5) https://www.elastic.co/guide/en/elasticsearch/client/curator/current/ex_snapshot.html Serverless Elasticsearch Curator on AWS Lambda: https://www.elastic.co/blog/serverless-elasticsearch-curator-on-aws-lambda Lambkin: 1) https://github.com/jarpy/lambkin 2) http://pydoc.net/lambkin/0.3.5/lambkin.aws/ 3) https://github.com/jarpy/lambkin/commit/b59450e144a53d5737596a30d966b1ee6f8129bf 4)

https://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/aes-ri.html

1.) CloudWatch -Compute -storage -CD -DB -Analysis 2.) Defaults for free -5min intervals -CPU, Network, Disk, Status(Instance, and system) 3.) Advanced -RAM -1 min intervals 4.) Baseline

Free services: -VPC -Elastic Bean Stack -Cloud Formation -IAM -Auto Scaling -OpsWorks -Considliating biling You paying for: -compute -storage -data transfer out AWS

Networking Security: 1.) SSL 2.) IPsec 3.) Isolation 4.) DDOS and more To do port scanning you have to contact Amazon. -Credentials (key pairs, MFA, security keys) -Trusted Advisor -Instance Isolation -Guest OS (Encryption, Integrity check, Authentication, Audit) -ELB (Proxy-Auth) Policy Stymulation

1)AWS IAM attributes: a) Users/Groups/Promissions b) Roles/Fed Users c) Cloud Trial d) Integration (IAM <-> AD) e) Compliance f) Automation 2)Inline policy 3) security scheme MFA token -> Time-Based Ohe-Time Password (TOTP) virtual MFA device uses a software app that generates six-digit auth. codes are compatible with the TOTP standard, as deseribed inRFC 6238 4) Instance profile -is a container for a IAM role that you can use to pass role inf. to an EC2 instance when the instance stars . 5) The signature Version 4 - signing process describes how to add auth. inf. to AWS requests. For security , most request to AWS must be signed with an access key (Access Key ID (AKI) and Secret Access Key (SAK)). If you use the  AWS CLI or one of the SDK, those tools automaticly sigh request for you based on credentials that you specify when you configure the tools. However, if you make direct HTTP or HTTPS calls to AWS, you must sign the requests yourself.

https://medium.com/oscar-tech/building-our-own-continuous-integration-system-adcdf0cb6df1

https://www.phusionpassenger.com/library/admin/apache/overall_status_report.html

https://httpd.apache.org/docs/2.4/rewrite/remapping.html https://httpd.apache.org/docs/trunk/es/rewrite/remapping.html

https://rockyj.in/2015/09/06/docker_capistrano.html https://stackoverflow.com/questions/39457603/how-to-integrate-capistrano-with-docker-for-deployment

https://programistanaswoim.pl/uczymy-sie-elasticsearch-004-indeks-odwrocony-apache-lucene/

http://blog.akademiakodu.pl/kurs-java/podstawy-klas/

https://stackoverflow.com/questions/1783405/how-do-i-check-out-a-remote-git-branch https://stackoverflow.com/questions/2765421/how-do-i-push-a-new-local-branch-to-a-remote-git-repository-and-track-it-too https://danielkummer.github.io/git-flow-cheatsheet/ http://rogerdudler.github.io/git-guide/index.pl.html

http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-dg.pdf http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emr-dg.pdf http://docs.aws.amazon.com/AmazonCloudWatch/latest/DeveloperGuide/acw-dg.pdf http://docs.aws.amazon.com/sns/latest/dg/sns-dg.pdf http://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-ug.pdf https://d0.awsstatic.com/whitepapers/aws-securing-data-at-rest-with-encryption.pdf http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/dynamodb-dg.pdf http://docs.aws.amazon.com/awsaccountbilling/latest/aboutv2/awsaccountbilling-aboutv2.pdf http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-ug.pdf http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/as-dg.pdf http://docs.aws.amazon.com/ElasticLoadBalancing/latest/DeveloperGuide/elb-dg.pdf http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-ug.pdf http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/rds-ug.pdf http://docs.aws.amazon.com/st...

RTO - recovery time objective - amount of hours to get back and running in perfect world RPO - recovery point object - acceptable data lost measure by time (30 min of data) Elements which can help create excellent disaster recovery: a) Region -AZ b) Storage c) Compute d) Networking e) Databases f) Orchestration (ex. CloudFormation) g) Security 1) backup and restore 2) pilot light 3) warm standby

Taking snapshots: -console -api call -command line (ec2_instance.snapshot) Restore from snapshot: ec2-create-volume Unmount Ec2-detach-volume Ec2-attache-volume Remount Coping snapshots to another region. Problem :We should doing backup when little utilisation to have consistency Solution for DB -hot backup mode -read replica (back up of that, and remove) -EBS snapshot Solution for filesystem: -xfs freeze -unmount Problem multiple volume backup: -lvm -replica to one big volume   Backup databases: A) your own: -native tools   -snapshots -RAID -> replica B) RDS -Automated   -Snapshots AMI - base on snapshots Action -> create image On Premis: -S3 and Glacier via API -AWS Storage Gateway (iSCSI protocol)

1) Pre-config  -database parameter group -> adding your own settings 2) Monitoring -basic automatically -extra for advance -Cloud Watch 3) Automated backups -by default 4) Snapshots 5) IOPS -designated -1 000-10 000 IOPS 6) Scaling -dynamic 7) Host/Replication(outside AZ)/Isolation(e.x.VPC&connectivity via VPN) 8) Changing db instance (size, instance class) -> without stopping Instance Actions -> Modify -> "change" -> Apply limitedly 9) R1 -DB engine -DB instance class -Deployment type -Licens model -Region 10) Master -> async Read Replica -> syn Standby

https://circleci.com/blog/deep-diving-into-circleci-workspaces/

https://resources.infosecinstitute.com/#gref

https://en.wikipedia.org/wiki/Poka-yoke

https://adamdrake.com/command-line-tools-can-be-235x-faster-than-your-hadoop-cluster.html

https://www.twilio.com/

Perks -performance -fault tolerance -reliability 1) Multiple AZ - multiple instance 2) Single point of contact 3) Health Checks 4) Application -Classic Needs: Listeners + ports Classic Load Balancer has to be internet facing! ALB : Targets(ec2) -> target group Cross AZ LB - dispose load equally between instances Routing Algoritms: CLB - Round Robin ALB - rule of routing

https://docs.stackstorm.com/overview.html

1) VPC peering 2) Software VPN 3) Software to HW VPN 4) Hardware VPN 5) Direct Connect

BGP

1 or 10 Gbps pipe

Multiple endpoint for High Avability . IPSec - as a security technology. Boarder Gateway Protocol (BGP) -easy routing of traffic from customer network to your VPC

1) Subnet 2) Internet Gateway 3) NAT Gateway NAT Instance -> Disable Source/Destination Check 4) hardware VPN connection 5) Virtual Private Gateway 6) Coustomer gateway 7) Router 8) Peering connection VPC peering -> no transitive peering -> DNS supported -50 VPC peer -> deafult -125 -> by request 9) VPC endpoint (endpoint to S3) Endpoints S3, DynamoDB 10) Engress-only Internet Gateway (outbound only for IPv6) enable DNS hostname : https://docs.aws.amazon.com/AmazonVPC/latest/UserGuide/vpc-dns.html Public subnet: Subnet Actions -> modify auto-assign IP settings -> Enable auto-assign public IPv4 addresess 11) Adding Lb to VPC: -secifi 1 subnet/AZ -you must specifity from at least 2 AZ to increase of your LB -2 public subnets at least + 1 subnet/AZ 12) VPC Security 13) VPC Flow Log: - wybierane w CloudWqtch -3 levele:   1.VPC   2.subnet   3.network interface -nie można enablować flow logów dla ...

https://aws.amazon.com/blogs/security/how-to-control-access-to-your-amazon-elasticsearch-service-domain/

https://medium.com/oscar-tech/building-our-own-continuous-integration-system-adcdf0cb6df1

1) Pricing model     a) On Demand     b) Reserved     c) Spot 2)Enhance Networking C3, C4, D2, I2, R3, M4 3) EC2 Instance Virtualization a) HVM -> all b) PV -> M3, C3 4) Plecment Groups -a logical groupin of instance in a single AZ -can't span multiple AZ -unique name across AWS account -recomendet application that benefi from low-latancy, high bandwith or best -instance that support entance network C3 C4 D2 I2 M4 E3 -existing group cannot be moved into placment group -cannot merge -can span pcered VPC but you will not get full - bisection bandwith between instance -reserved instance supported but not for placment group https://docs.amazonaws.cn/en_us/AWSEC2/latest/UserGuide/ec2-ug.pdf

https://github.com/hashicorp/terraform-aws-vault

https://robomongo.org/

1. Can only use supported runtimes and versions (e.g. python-3.6.2 or python 2.7.13) 2. Can only use supported system and software libraries 3. Can only run web services (data stores and other services available only via paid add-ons) 4. Apps can't access the shell 5. Devs can't access servers via ssh 6; Local disk is read-only 7. Load balancing is HTTP/HTTPS only 8. Requests are limited to 30 seconds. 9. Limited to one AWS region 10. App must boot in 60 seconds or less 11. Apps can be at most 100MB 12. Build must take less than 15 min 13. Logs are limited to 1500 lines unless you use supported (paid) add-ons 14. Manual scaling only 15. Pricing gets very steep as you scale up Source: 1. https://www.youtube.com/watch?v=LVgP63BkhKQ

1) https://www.ybrikman.com/

du -sh /path/to/folder

https://medium.com/@jshamilton/ios-dev-setup-customize-your-terminal-6048e7a43041

https://cloudavail.com/2015/07/06/an-introduction-to-the-aws-ec2-container-service/

https://cloudavail.com/2014/03/05/running-fsck-on-ec2-part-1/

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html

1) Configuration:  /home/user/ aws configure AWS Access Key ID [None]:  AWS Secret Access Key [None]:  Default region name [None]:  Default output format [None]: 2) Security Groups:  https://docs.aws.amazon.com/zh_cn/cli/latest/reference/ec2/describe-security-groups.html Commands: aws ec2 describe-security-groups //list security groups

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html Code Name us-east-1 US East (N. Virginia) us-east-2 US East (Ohio) us-west-1 US West (N. California) us-west-2 US West (Oregon) ca-central-1 Canada (Central) eu-central-1 EU (Frankfurt) eu-west-1 EU (Ireland) eu-west-2 EU (London) eu-west-3 EU (Paris) ap-northeast-1 Asia Pacific (Tokyo) ap-northeast-2 Asia Pacific (Seoul) ap-northeast-3 Asia Pacific (Osaka-Local) ap-southeast-1 Asia Pacific (Singapore) ap-southeast-2 Asia Pacific (Sydney) ap-south-1 Asia Pacific (Mumbai) sa-east-1 South America (São Paulo)

To create print screen use Command + Shift + 3 (4). File with print screen will be saved on Desktop.

Click on write upper corner on your user name then on 'My Security Credentials'. Then from list on left choose 'Users'. Click on user which credential you need to get. Click on 'Security Credentials'. Press on 'Create access key'. Pop up will show up. You can download keys. 

1) Basic Information about Terraform - tool to provisioning infrastructure - write and compile in Golang - Infrastracture as code - Automation of infrastracture - Keep your infrastracture in a certain state (compliant) e.g. 2 web instances with 2 volumes, and 1 load balance - make your infrastracture auditable  * you can keep your infrastracture change history in a version control system like git -Ansible, chef Puppet , Saltstack have a focus on automationg the installation and configuration of software. Keep the machines in compliance , ina a certain state. -Terraform can automate provisioning of the infrastracture itself eg. Using the AWS, DigitalOcean, Azure API. Works well with automation software like ansible to install software after the infrastracture is provisioned. Terraform Use Cases a) Infrastracture as a Code -Use infrastracture as code to safely and efficiently provision and manage infrastracture at  any scale. b) Multi-Cloud Compliance & ...