- CloudSpokes Kiva Loan application [Java]: Data are loaded from a JSON feed into DynamoDB. The application supports item loading, and the display and filtering (by country name) of DynamoDB data.
The source code for this project is available on github. - Sample bulletin board application [PHP]: A discussion board application that supports DynamoDB data creation and review using web forms. The Readme file includes instructions that are sufficient for getting the application online, though Apache configuration knowledge is useful. Note that if you want to use the AWS Console to review table data, you'll need to set the appropriate DynamoDB endpoint in file SimpleAmazonDynamoDB.php. (See this page for a listing of DynamoDB endpoints.)
The source code for this project is available on github; I've created a fork with some customizations on github.
Showing posts with label amazon web services. Show all posts
Showing posts with label amazon web services. Show all posts
Friday, July 20, 2012
Amazon DynamoDB - application examples
I've been doing some development work with Amazon's DynamoDB recently. Here's info on two github repositories that enable you to get started with DynamoDB and that show, at least at some level, the kinds of apps that DynamoDB can be used for.
Thursday, January 5, 2012
procedure for increasing the root disk size of an EC2 instance
Last night, I used this procedure, written by Eric Hammond, to increase the boot disk size on the instance used to support the nrhdb. It requires use of the Amazon EC2 command line tools, working from another computer (i.e., a computer other than the server that's being resized).
I only had to make a couple of modifications to use it:
I only had to make a couple of modifications to use it:
- When running the ec2-run-instances command: because I wanted to start the instance in a specific availability zone and apply the desired firewall settings, I added two parameters not included in the procedure: --availability-zone and --group (this reference document provides more info)
- My initial attempt to run the resize2fs command did not resize the disk in the OS; I had to reboot the server, and at that point, resize2fs ran successfully
Friday, December 23, 2011
Amazon EC2 reserved instance - one year later
My first Amazon Web Services EC2 reserved instance, which supports the Naval Reactors History Database online resource, expired today. This is something of a milestone for me. I had used EC2 instances for several months prior to purchasing a reserved instance last December. This reserved instance enables me to run an EC2 instance 24/7/365 at a reduced cost over what I would pay for an on-demand AWS EC2 instance.Based upon my year's experience, I have these thoughts:
- First, my goal is to maintain the nrhdb resource online at the lowest possible cost. While it would be possible (though technically clunkier) for me to run the nrhdb on a Windows server, the EC2 prices for Linux versus Windows make it clear that the purchase and use of a Linux EC2 reserved instance is the most cost-effective choice.
- Second, the Amazon Web Services outage that occurred in April made me take a second look at the way that I manage my instance's data. I posted on this incident earlier this year. I now maintain a copy of my server's volume in another availability zone in the East region at all times and update the snapshot/volume every week or so. Although I could use EC2 API tools to automate this process, I'm still doing it manually using the AWS Management Console.
- I wound up changing my server OS during the year from Fedora to Amazon Linux (using the Amazon Linux AMI). Overall, my experience with this AMI has been positive - the Amazon Linux instance comes with fewer preinstalled packages and the ongoing installation of updates is very seamless.
- Finally, EC2 reserved instances are not only specific to an AWS region, but to an availability zone as well. Thus, you're locked into that AZ for the term period. Which can be a problem if, for example, there's an incident like the one that impacted the East region, but was centered in one availability zone.
Tuesday, May 3, 2011
using Amazon Web Services features to improve EC2 and EBS resource durability
I run the Naval Reactors History Database, a hobby project, on Amazon Web Services resources. This includes a Linux server, persistent storage, and an IP address. I run them out of the AWS East region, which was the region at the center of the recent and significant AWS outage. While my online resource never had an outage, to the best of my knowledge, it's clear that it could have been impacted because of the EBS control plane that supports availability zones across the region. Also, Amazon announced that a small amount of EBS volume data had been lost in the affected Availability Zone.
So, this weekend I spent some time thinking about preserving the work that I've done with my Naval Reactors project. This is in the context of an online database that I'm slowly building, with objects being added and updated as I find the time on weekends and evenings. In short, it's a fairly static resource. In his book on Amazon Web Services, Jeff Barr notes the importance of creating lists. That's what I hope to have out of my own work - a set of lists that I create and can use in order to recover from AWS outages like the one that occurred last month.
So, to begin. First scenario: I am running an m1.small Linux instance in the us-east-1d Availability Zone (AZ). It's possible for me to launch and test a copy of my current server in another East availability zone. All of this work is done in the Amazon Web Services console, so it's quite quick and easy:
1. From Instances: Create an AMI from the running instance (what I'll call the production instance). There is a short (estimate 1-2 minutes) of server downtime as the AMI is generated.
2. From AMIs: Choose to launch an instance from the newly-created AMI. When going through the creation steps, I change the default selection for the AZ and choose to run the new instance in us-east-1b. I choose to keep the same Key Pair Name and Security Group as I have for the production instance.
After launching the instance, I have an EBS-backed Linux instance running in us-east-1d (production) and an instance running in us-east-1b (backup). The AZs have independent power and network connectivity. While the incident report describes how problems in one AZ can potentially impact others in the region, having this server running in another AZ provides a resource backup and a method for bringing my online database back online in the event of an outage.
3. Using the public DNS address, I test access to the Tomcat-based Naval Reactors History Database on the backup server - with success.
Note: I didn't generate an Elastic IP address for this instance. First, it wouldn't make sense in the context of my use - I would use the Elastic IP address currently mapped to the production server and would map this address to the backup server in the event of an outage. But second, you should be aware that you will be charged for an unused Elastic IP address that you've allocated to your account.
4. Stop the EBS-backed backup server instance.
Result: production server running in AZ us-east-1d; backup server stopped, but ready to start and serve resources, in AZ us-east-1b.
I performed all of these steps successfully today.
This is one method of providing redundancy. But I want to come up with something a little more sophisticated - in part because I am interested in moving to a new server OS and more robust EC2 platform in the future. Here's a second scenario, in which I build a new instance to host the collection and attach a volume with the needed data to it - all in a different AZ than the production server is running in.
Steps 1-5 and 8-9 below are performed in the AWS Management Console - including step 5, which I'll comment on later.
1. From Volumes: Create a snapshot from the production server's EBS volume.
2. From Snapshots: Create a volume from the EBS snapshot. Again, since the production server is running in us-east-1d, I create the new volume in us-east-1b.
3. From AMIs: Find the right AMI for the future production server. In my case, I'm looking for a Linux OS AMI that I'm comfortable with, preferably with Apache Tomcat preloaded.
4. From AMIs: Launch an instance using the AMI found in step 3. I'll be mounting the volume created in step 2, so I will manually set the instance's AZ to us-east-1b.
5. From Volumes: Here, I will attach the volume created in step 2 to the instance launched in step 4.
So, this weekend I spent some time thinking about preserving the work that I've done with my Naval Reactors project. This is in the context of an online database that I'm slowly building, with objects being added and updated as I find the time on weekends and evenings. In short, it's a fairly static resource. In his book on Amazon Web Services, Jeff Barr notes the importance of creating lists. That's what I hope to have out of my own work - a set of lists that I create and can use in order to recover from AWS outages like the one that occurred last month.
So, to begin. First scenario: I am running an m1.small Linux instance in the us-east-1d Availability Zone (AZ). It's possible for me to launch and test a copy of my current server in another East availability zone. All of this work is done in the Amazon Web Services console, so it's quite quick and easy:
1. From Instances: Create an AMI from the running instance (what I'll call the production instance). There is a short (estimate 1-2 minutes) of server downtime as the AMI is generated.
2. From AMIs: Choose to launch an instance from the newly-created AMI. When going through the creation steps, I change the default selection for the AZ and choose to run the new instance in us-east-1b. I choose to keep the same Key Pair Name and Security Group as I have for the production instance.
After launching the instance, I have an EBS-backed Linux instance running in us-east-1d (production) and an instance running in us-east-1b (backup). The AZs have independent power and network connectivity. While the incident report describes how problems in one AZ can potentially impact others in the region, having this server running in another AZ provides a resource backup and a method for bringing my online database back online in the event of an outage.
3. Using the public DNS address, I test access to the Tomcat-based Naval Reactors History Database on the backup server - with success.
Note: I didn't generate an Elastic IP address for this instance. First, it wouldn't make sense in the context of my use - I would use the Elastic IP address currently mapped to the production server and would map this address to the backup server in the event of an outage. But second, you should be aware that you will be charged for an unused Elastic IP address that you've allocated to your account.
4. Stop the EBS-backed backup server instance.
Result: production server running in AZ us-east-1d; backup server stopped, but ready to start and serve resources, in AZ us-east-1b.
I performed all of these steps successfully today.
---
This is one method of providing redundancy. But I want to come up with something a little more sophisticated - in part because I am interested in moving to a new server OS and more robust EC2 platform in the future. Here's a second scenario, in which I build a new instance to host the collection and attach a volume with the needed data to it - all in a different AZ than the production server is running in.
Steps 1-5 and 8-9 below are performed in the AWS Management Console - including step 5, which I'll comment on later.
1. From Volumes: Create a snapshot from the production server's EBS volume.
2. From Snapshots: Create a volume from the EBS snapshot. Again, since the production server is running in us-east-1d, I create the new volume in us-east-1b.
3. From AMIs: Find the right AMI for the future production server. In my case, I'm looking for a Linux OS AMI that I'm comfortable with, preferably with Apache Tomcat preloaded.
4. From AMIs: Launch an instance using the AMI found in step 3. I'll be mounting the volume created in step 2, so I will manually set the instance's AZ to us-east-1b.
5. From Volumes: Here, I will attach the volume created in step 2 to the instance launched in step 4.
6. In the new server's Linux OS, Create the mount point location and mount the attached volume:
mkdir /mnt/prodata
mount /dev/sdf /mnt/prodata
7. Copy the XTF and Naval Reactors History Database files from the just-mounted volume to the Tomcat location on the new production server.
8. After testing, use this server as the new production server and map the Elastic IP address to the server.
9. Stop, and terminate when comfortable doing so, the previous production server.
8. After testing, use this server as the new production server and map the Elastic IP address to the server.
9. Stop, and terminate when comfortable doing so, the previous production server.
I had initially planned to perform step 5 using the EC2 command line tools, but it's vastly easier to use the AWS Management Console.
My conclusions: The second method provides an important foundation for ensuring the durability of my EC2-hosted online collection. Amazon's detailed report on last month's outage includes this statement: "For example, when running inside a Region, users have the ability to take EBS snapshots which can be restored in any Availability Zone...."
---
My conclusions: The second method provides an important foundation for ensuring the durability of my EC2-hosted online collection. Amazon's detailed report on last month's outage includes this statement: "For example, when running inside a Region, users have the ability to take EBS snapshots which can be restored in any Availability Zone...."
I'm still exploring how to best automate the process of creating snapshots and restoring a volume in an AZ that's different than the production server is running in. What I have, as a primary protection at this point: From the second procedure, my production service running in one AZ and an EBS volume containing my application and online collection data has been restored and is available for use in a second AZ.
Also, I plan to do more reading on AWS best practices. I'm sure that I can improve upon the above procedures, but this is what I came up with based upon my current knowledge.
Subscribe to:
Posts (Atom)