Amazon DynamoDB - application examples

I've been doing some development work with Amazon's DynamoDB recently.  Here's info on two github repositories that enable you to get started with DynamoDB and that show, at least at some level, the kinds of apps that DynamoDB can be used for. 

• CloudSpokes Kiva Loan application [Java]:  Data are loaded from a JSON feed into DynamoDB.  The application supports item loading, and the display and filtering (by country name) of DynamoDB data. 
  
  The source code for this project is available on github. 


• Sample bulletin board application [PHP]:  A discussion board application that supports DynamoDB data creation and review using web forms.  The Readme file includes instructions that are sufficient for getting the application online, though Apache configuration knowledge is useful.  Note that if you want to use the AWS Console to review table data, you'll need to set the appropriate DynamoDB endpoint in file SimpleAmazonDynamoDB.php.  (See this page for a listing of DynamoDB endpoints.)
  
  The source code for this project is available on github; I've created a fork with some customizations on github.  

XTF customizations for Naval Reactors History Database are on github

I've posted my XTF 3.0 customizations for the Naval Reactors History Database search interface to github.  They are available at URL https://github.com/alncornish/xtf-nrhdb.

adding Open Graph Protocol metadata to XTF subject search results

Postscript:  After encountering some technical problems with the use of Facebook on the database site, I've removed all but the site-level OGP data.  The example code functions.  

Description of problem

I'm using Open Graph Protocol metadata to enhance user engagement with the Naval Reactors History Database. I've coded the top-level site with OGP metadata, which is a very straightforward process. The XTF platform is used to support discovery for this database. OGP metadata is simply embedded in the HTML markup for the main form, as defined in the XTF file searchForms.xsl. The outputted metadata makes the site's search page an object in the Facebook social graph and enables users to express a Like relationship with the site.

I am also interested in embedding OGP structured metadata into topical pages within the NRHDB site, which is a much more challenging task. In short, I want to display OGP metadata for some subjects prominently covered in the database, using the database's controlled vocabulary to identify each subject.

Solution

The key XTF file for this task is resultFormatter.xsl and I used both XTF's debug mode support and the Facebook Linter in implementing this functionality.

Here's a rough outline of the steps that I performed for one specific topical search:

1. Identify the simplest URL that can be used to pull up a subject search (example: http://navalreactorshistorydb.info:8080/xtf/search?f1-subject=S1W).

2. Open this URL in XTF's debug mode, step 4.

3. Save the raw XML results for the search to local disk; this is the XML that is fed to the resultFormatter.xsl stylesheet.

4. Analyze the XML structure, particularly the data returned under the /crossQuery/parameters/param elements. This will be used to guide the creation of the XPath expressions used in step 6.

5. Open resultFormatter.xsl and add a local parameter.


6. Add the following code, immediately after the head section's title element. This XSLT code performs a test on the XML data to determine, first, if it's a controlled vocabulary search and, second, if it's a search for the specific term. If both conditions are met, OGP metadata for the topic is inserted into the HTML output displayed on the results screen.


The OGP documentation should be consulted on structuring this metadata and for other rules, such as thumbnail size requirements.

The results can be viewed using the Facebook Linter. I'm still exploring how to best implement subsite or topical page OGP metadata for the Naval Reactors History Database and welcome any comments on this approach and procedure.

----

Addendum

This excerpt provides another example - adding the Open Graph metadata only when viewing the first set or screen of results. This requires the addition of another local parameter (countInfo, mapped to /crossQueryResult/query) and a second xsl:if statement.



Thus far, with some of the major subjects coded in the resultsFormatter.xsl file, performance doesn't seem to be a problem, in terms of the time to display the results screen.

thoughts on AWS DynamoDB

I am impressed with Amazon's vision for its DynamoDB NoSQL database service.  This FAQ document provides a good introduction to the service.  DynamoDB has tremendous potential.  Most importantly, it enables a company or institution to deploy a service and scale it as needed, both in terms of the number of items and the read/write throughput for a table.  (As noted in the FAQ, the ability to scale up write throughput to any level that's needed is a significant break from the capabilities of Amazon Web Service's SimpleDB service, another NoSQL service option.)  Having a service that can scale up for success and scale down as needed again illustrates the economic advantages of cloud-based services.

On the negative side:  DynamoDB is still a relatively new service.  Earlier this month, the ability to view, add, and delete table items was added to the AWS Management Console, a respectable add-on, most notably, enabling a spreadsheet view of a table's data without using the API.  I found the Developer Guide documentation examples for read and scan operations to be barely adequate at this point- though equivalent for the Java, .NET, and PHP SDKs.  In short, these examples are well-organized and useful, but limited.  Additionally, searching on the web for code examples for read and scan operations didn't yield as much information as I expected.

moving Facebook Social Plugins from IFrame to HTML5

This weekend, I migrated Facebook Social Plugins for the Naval Reactors History Database site from the legacy IFrame apps to HTML5 apps. I found this procedure to be helpful in accomplishing the migration. I'm embedding this JavaScript snippet into XTF's XSLT at appropriate points (the search and results pages code):

There are a number of benefits of moving from IFrame to HTML5 apps. As one example, I'm now including the Send button, along with the Like button, on the nrhdb site.

Note that you need to have claimed your Facebook e-mail account before the Send button will function. Here's a Facebook Developers discussion on this point; a poster describes this requirement in the thread.

This was a pretty easy upgrade and I like the result, though embedding the plugins in the context of XTF's structure made it more challenging.

Postscript: I obtained much better results for pulling in the comments stream for the Comments plug-in by mounting the information pages on an Apache server, instead of Tomcat 6. When the info pages were served by Tomcat, the comments stream pages were frequently truncated, with a status bar appearing on the screen indicating the content was being loaded.

procedure for increasing the root disk size of an EC2 instance

Last night, I used this procedure, written by Eric Hammond, to increase the boot disk size on the instance used to support the nrhdb. It requires use of the Amazon EC2 command line tools, working from another computer (i.e., a computer other than the server that's being resized).

I only had to make a couple of modifications to use it:

• When running the ec2-run-instances command: because I wanted to start the instance in a specific availability zone and apply the desired firewall settings, I added two parameters not included in the procedure: --availability-zone and --group (this reference document provides more info)
  

• My initial attempt to run the resize2fs command did not resize the disk in the OS; I had to reboot the server, and at that point, resize2fs ran successfully

Summary: This procedure describes how to increase the size of the root disk on an EC2 server with very little downtime.

Amazon EC2 reserved instance - one year later

My first Amazon Web Services EC2 reserved instance, which supports the Naval Reactors History Database online resource, expired today. This is something of a milestone for me. I had used EC2 instances for several months prior to purchasing a reserved instance last December. This reserved instance enables me to run an EC2 instance 24/7/365 at a reduced cost over what I would pay for an on-demand AWS EC2 instance.

Based upon my year's experience, I have these thoughts:

• First, my goal is to maintain the nrhdb resource online at the lowest possible cost. While it would be possible (though technically clunkier) for me to run the nrhdb on a Windows server, the EC2 prices for Linux versus Windows make it clear that the purchase and use of a Linux EC2 reserved instance is the most cost-effective choice.
  
  
• Second, the Amazon Web Services outage that occurred in April made me take a second look at the way that I manage my instance's data. I posted on this incident earlier this year. I now maintain a copy of my server's volume in another availability zone in the East region at all times and update the snapshot/volume every week or so. Although I could use EC2 API tools to automate this process, I'm still doing it manually using the AWS Management Console.
  
  
• I wound up changing my server OS during the year from Fedora to Amazon Linux (using the Amazon Linux AMI). Overall, my experience with this AMI has been positive - the Amazon Linux instance comes with fewer preinstalled packages and the ongoing installation of updates is very seamless.
  
  
• Finally, EC2 reserved instances are not only specific to an AWS region, but to an availability zone as well. Thus, you're locked into that AZ for the term period. Which can be a problem if, for example, there's an incident like the one that impacted the East region, but was centered in one availability zone.
  

Overall, I'm glad that I'm using Amazon Web Services EC2 and I believe that I'm getting a good value from AWS as a whole. The EC2 home page notes that EC2 "is designed to make web-scale computing easier for developers" and that is certainly my take on it. I've been able to maintain an online service reliably for a year and the purchase of the reserved instance has made it affordable.