Category Archives: AWS

Jenkins email notifications using AWS SES

I have installed Jenkins on Amazon Linux EC2 instance and would like to setup notifications. Since AWS provides an email service, Simple Email Service (SES) that can accomplish this task. I can also use other SMTP providers such as Gmail, Outlook, Yahoo and Jango, but I like to keep everything in one environment.

SES in my AWS VPC was setup a while back, so I will use its settings.

Pre-requisite(s)

  1. AWS Account
  2. SES already configured (https://docs.aws.amazon.com/ses/latest/DeveloperGuide/setting-up-email.html)
  3. SES SMTP credentials (https://docs.aws.amazon.com/ses/latest/DeveloperGuide/smtp-credentials.html)

Steps

  1. Access Jenkins
  2. Click “Manage Jenkins” then “Configure System”
  3. Scroll to “Jenkins Location” and input a “noreply” email address for “System Admin e-mail address. I used something like noreply@my.domain.***
  4. Scroll all the way down to “E-mail Notification” section
  5. Apply SES SMTP settings:
    • SMTP Server = email-smtp.us-east-1.amazonaws.com
    • Check box to “Use SMTP Authentication” and fill in required data:
      • User Name: {Use SMTP credentials created when setting up SES}
      • Password: {Use SMTP credentials created when setting up SES}
      • Check box for “Use SSL”
      • SMTP Port: 465
      • Add a “Reply-To Address”; in my case, I added the value used in step #3.
    • Click “Apply” to save settings
    • Check box to “Test configuration by sending test e-mail”
      • Test e-mail recipient: {your email address}
      • Click “Test Configuration” button
      • If successful, you will see “Email was successfully sent” message
    • Click “Save” button

***Without setting this value, an error will occur: “554 Message rejected: Email address is not verified. The following identities failed the check in region US-EAST-1: address not configured yet <nobody@nowhere>, nobody@nowhere

Running Chef Server on Ubuntu 16

Since I have established a working software-VPN solution to connect to my AWS VPC, an internal-only configuration management system can now be implemented. The chosen system is Chef. This means my chef server is not available to the public, and can be only accessed via VPN.

Below will summarize the commands to get a Chef Server up and running. For more information, read Chef’s article: https://docs.chef.io/install_server.html, and credit goes to Chef.

Pre-requisites

– Have a running Ubuntu EC2 (t2.medium is a good choice)

– Appropriate Security Group rules (port 22, 443)

– Set a hostname on the EC2, and its best that it be the FQDN of what it will be in DNS, if      used.

Commands

Commands below are exactly what was run on the ubuntu server as root (Chef expects its package(s) to be installed this way, otherwise it will complain).

 

1. Download chef server core package for Ubuntu OS: “wget https://packages.chef.io/files/stable/chef-server/12.17.15/ubuntu/16.04/chef-server-core_12.17.15-1_amd64.deb”

2. Install package: “dpkg -i chef-server-core_12.17.15-1_amd64.deb”

3. Get backend services up and running: “chef-server-ctl reconfigure” This process took about 4mins 15secs.

4. Create hidden chef directory to place chef-related files: “mkdir -p .chef”

5. Create admin user: “chef-server-ctl user-create admin Admin User admin@xyz.net ‘{password}’ –filename /root/.chef/chef.pem” Of course, password purposely not provided.

6. Create an organization and associate user created in step #5: “chef-server-ctl org-create esn ‘MyXingfu Net’ –association_user admin –filename /root/.chef/esn-validator.pem”

At this point, you have a working chef server running on Ubuntu 16, but there is no web UI. I like using the UI and installed the frontend for Chef Server, called Chef Manage, in three steps:

1. chef-server-ctl install chef-manage

2. chef-server-ctl reconfigure

3. chef-manage-ctl reconfigure –accept-license

Chef Server WebUI: https://<Private-IP or FQDN>; login with created user.

I also like Chef’s reporting mechanism, so installed that package as well:

1. chef-server-ctl install opscode-reporting

2. chef-server-ctl reconfigure

3. opscode-reporting-ctl reconfigure –accept-license

Login to chef server UI and click on “Reporting” tab.

Connect to your AWS VPC via OpenVPN

Since I cannot afford Direct Connect, one thought of connecting into my AWS environment is to use software VPN such as OpenVPN Access Server. Google search resulted in many articles covering this software and AWS.

I chose one article that outlines the setup and testing:

http://www.1strategy.com/blog/2017/02/21/securely-connecting-to-your-aws-environment-using-openvpn-access-server/

My VPC has both Public and Private subnets. The VPN EC2 instance is in the Public subnet with an Elastic IP (EIP).

The above article left the security group wide open in the rules purely for quick testing purposes. It was recommended to revisit the rules for the VPN security group, and make them more secure. I decided to restrict these rules to my Public IP as the source after successfully testing:

  1. VPN Connection
  2. SSH connection to a test Linux EC2 instance.

Tools for finding Public IP addresses include https://www.ipchicken.com/https://www.whatismyip.com/, etc.

After making the changes of the security group rules, test #2 failed. I knew that the private NACL needed to be updated, but what would the source be for the SSH rule(s)? I started using OpenVPN’s Dynamic IP Address Network for the SSH rules and that did not work. I found the answer through VPN Routing settings that states “NAT is preferred for client access to private networks.” Since NAT is used, the VPN server’s private IP address needed to be added as the source. Instead of adding the IP address, I added the subnet CIDR corresponding to all public subnets to the private NACL. Test #2 passed successfully.

To go one step further, I decided to edit the security group rule for the private EC2 instance. Instead of leaving it wide open, I changed the source to my VPC’s private network, and test #2 passed again.

Now unto some devopsy stuff via my VPN connection…

 

Install AWS CLI and AWS Python SDK – CentOS 7

The easiest method is to use pip as python is installed with Linux such as CentOS 7. This article covers only v2 of python only.

AWS CLI installation

  1. Download pip install script using curl: “curl -O https://bootstrap.pypa.io/get-pip.py
  2. Install pip using python: “python get-pip.py –user”
  3. Install AWS CLI using pip: “pip install awscli –upgrade –user”
  4. For successful installation, check version of the CLI: “aws –version”

AWS Python SDK installation

  1. Use pip to install boto3: “sudo pip install boto3”
  2. If a message appears stating that pip is not found, copy pip command from /root/.local/bin to /usr/bin
  3. Re-execute step 1

Credits: Amazon Web Services

http://docs.aws.amazon.com/cli/latest/userguide/awscli-install-linux.html#awscli-install-linux-pip

https://aws.amazon.com/sdk-for-python/

 

Using Vagrant on Windows 10

Step One – Installing Vagrant

  1. Download Vagrant
  1. Install after download completes.
  • This step assumes familiarity installing Windows applications.
  1. Reboot as informed by the installer.
  2. Launch PowerShell (Choose Run as Administrator)
  3. Install AWS-provider plugin:
  • vagrant plugin install vagrant-aws
  1. Create a project folder on root of C:\
  • mkdir project
  1. Run “vagrant init”
  2. Add dummy.box from Mitchell Hashimoto.

Step Two – Using Vagrant

  1. Use Scott Lowe’s blog on how use vagrant with AWS plugin.
  1. Create Vagrantfile and instances.yml similar to those of Scott Lowe’s github project.
  1. Since the Vagrantfile is looking in the user environment for AWS related key_id and secret_key, use set command to add this info to the environment:
  • set AWS_ACCESS_KEY_ID=
  • set AWS_SECRET_ACCESS_KEY=
  1. Customize instances.yml for your needs.
  2. Save files and run “vagrant up”.
  3. Check AWS Console under EC2 for running instance.
  4. Use Putty-gen to convert keypair.pem to keypair.ppk.
  • This step assumes familiarity with Putty.
  1. Setup a connection in Putty to connect to running instance.
  • This step assumes familiarity with Putty.
  1. SSH to instance using connection setup in previous step.
  2. Once done, run “vagrant destroy” to delete instance.
  3. Verify status of instance is “Terminated” on AWS Console.
  4. Logout of AWS Console.