HowTo - Fix windmill error in Linux EC2

HowTo - Fix windmill error in Linux EC2

HowTo - Get InstanceID of AWS EC2 for coding usage

HowTo - Get InstanceID of AWS EC2 for coding usage

Watch AWS service health check - HowTo

AWS Services - Health check dashboard - HowTo

Ansible playbook to deploy specific version pip - HowTo

Ansible playbook to deploy specific version pip

Pip installer is package management system used to install and manage software packages that are developed in python. 

Recenty, I have upgraded Apache airflow version to 1.8.1 and the Ansible playbook was raising error on PIP being the latest to 13.0.1. The catch here is that, v1.8.1 needs PIP  v9.0.3 and any higher version of PIP, raises error from playbooks

All being discussed, let us see, how to deploy specific version of pip from playbooks

Apache Airflow - What happens when version used isn't supported anymore ???

Apache Airflow - What happens when version used isn't supported anymore ???

Before, we start looking into the issue and resolution, let us get to know what-is-what

Apache Airflow: 

Airflow is a platform to programmatically author, schedule and monitor workflows.

Use airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed

For more information refer to the documentation from Apache at https://airflow.apache.org/ 

Ansible:

Ansible is IT automation tool, easy and efficient to do configuration management, deploy software to make Continuous Delivery, an easy journey. For more details, refer to https://docs.ansible.com/ansible/latest/index.html

Start AWS EC2 with automated Lambda script

Stop AWS EC2 with automated Lambda script

Start RDS clusters based on Tag values

Start RDS clusters based on Tag values

Amazon Redshift - How to fix commands ignored error

Amazon Redshift - How to fix commands ignored error

Amazon Redshift - How to fix query access error

Amazon Redshift - How to fix query access error

Amazon Redshift - Renaming the cluster identifier

Amazon Redshift - Renaming the cluster identifier

Pyspark error in Amazon Linux

Pyspark error in Amazon Linux

AWS CLI error in Amazon Linux

AWS CLI error in Amazon Linux

AWS CLI - is AWS Command Line Interface. It is a tool that brings in all the AWS provided services under a single roof, meaning, we will be able to create, modify and access resources using this command line interface via commands

We all know that we need to install in any flavor of Linux; recently, I faced an issue of accessing AWS CLI in Amazon linux AMI

Accessing aws from shell prompt returns "too few arguments" or aws --version returns awscli version installed in that instance

Let us see on resolving the same

Error:

[ec2-user@ip-xx.xx.xx] aws

ImportError: cannot import name HTTPClientError


Issue:

1. could be that system update was applied using "sudo yum update" which might have updated any dependencies or so

Resolution:

• re-install didnt work
• sudo pip install upgrade --awscli - resolved the issue....Yayyyyyyyyyy !!!

Jenkins Dynamic Slaves setup - Quick Tip #3

Jenkins Dynamic Slaves setup - Quick Tip #3

Jenkins Dynamic Slaves setup - Quick Tip #2

Jenkins Dynamic Slaves setup - Quick Tip #2

While setting up Amazon EC2 plugin configuration of dynamic slaves, i recently came across quite some errors/issues and would like to share it here based on the error messages rather clubbing resolutions provided, altogether in one single thread post

So, now lets move on to the Tip of the day

Jenkins Dynamic Slaves Setup - Quick Tip #3

Jenkins Dynamic Slaves Setup - Quick Tip #3

Jenkins Dynamic Slaves Setup - Quick tip #1

Jenkins Dynamic Slaves Setup - Quick tip #1

Jenkins, as a CI server, allows AWS EC2 instances to be launched as Slave nodes for jobs that are labelled to run on Slaves

Primarily, the Jenkins jobs which needs certain framework or apps and custom setup in order to execute the job steps/actions, needs customized instances with the expected setup. This is made possible with Jenkins - Amazon EC2 plugin

Airflow with MySQL raising error on libraries

Airflow with MySQL as back end will raise error on libmysqlclient.so.1020 and let us see Why and How to resolve this errors with Apache Airflow

Apache Airflow is an opensource tool for orchestrating complex workflows and data processing pipelines. Workflow in Airflow is designed as a directed acylic graph(DAGs) meaning the tasks in a workflow can be split and executed independently and later be logically merged in a task

AWS decode authorization message

AWS - How-to decode authorization error messages


Today I came across an interesting issue in AWS

I was mapping Jenkins to launch EC2 slaves dynamically[yes finally, i did it :-)]

The Launch configuration was giving some errors unreadable and was really confused about whats wrong and where

Finally identified, that the AWS Authorization messages thrown out on any request is bound to generate an encoded message, which is really confusing !!!

The encode message thus generated as part of EC2 access permissions error will have to deciphered by decoding the same using the AWS CLI utility. Yessss !!! AWS has it all !!!

AWS Community Day 2018 - Bengaluru - Session was a huge hit !!!!

AWS Community Day 2018 - Bengaluru

Flyer Released

Today, I received the Flyer for the speaker with the respective topics launched and published

Tour-de-force

AWS User Groups

AWS User Groups

Stop RDS clusters based on Tag values

AWS RDS is an easy to setup and operate relational Database available for MySQL, Postgress, MariaDB, Oracle and MS SQL Server and Amazon Aurora

AWS RDS is priced per hour based on the usage apart from this you can acutally control the pricing by taking up Reserved Instances for RDS or by starting/stopping based on the usage were in the latter is quite an overhead as we need to spend more time on admin activities rather actual worklog with RDS. Please visit the Amazon documentation on detailed pricing information on each type of the DB you launch https://aws.amazon.com/rds/mysql/pricing/

Terminate EMR cluster to save costs

Stop AWS EC2 with automated Lambda script

Welcome to the Cloudgoddess blog !!!