Hadoop 使用Whirr启动EC2集群

目前我正在学习杰弗里·布林的教程。当我想用Whirr启动ec2集群时遇到了一些麻烦。我使用cloudera演示vm cdh3u4

我下载了whirr的0.8.1版

以下是我运行的所有命令：

$ wget http://mirror.switch.ch/mirror/apache/dist/whirr/whirr-0.8.1/whirr-0.8.1.tar.gz
$ tar zxf whirr-0.8.0.tar.gz
$ export PATH="~/whirr-0.8.0/bin:$PATH"

$ export AWS_ACCESS_KEY_ID=MY ACCESS KEY
$ export AWS_SECRET_ACCESS_KEY=MY SECRET ACCESS KEY

$ ssh-keygen -t rsa -P hadoop-ec2

然后有人问我应该在哪个文件中保存密钥，我键入：hadoop-ec2

$ whirr launch-cluster --config hadoop-ec2.properties

…问题是：没有启动任何实例！我得到了以下信息：

Exception in thread "main" org.apache.commons.configuration.ConfigurationException: Cannot locate configuration source hadoop-ec2.properties
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:249)
        at org.apache.commons.configuration.AbstractFileConfiguration.load(AbstractFileConfiguration.java:229)
        at org.apache.commons.configuration.AbstractFileConfiguration.<init>(AbstractFileConfiguration.java:149)
        at org.apache.commons.configuration.PropertiesConfiguration.<init>(PropertiesConfiguration.java:252)
        at org.apache.whirr.command.AbstractClusterCommand.getClusterSpec(AbstractClusterCommand.java:122)
        at org.apache.whirr.cli.command.LaunchClusterCommand.run(LaunchClusterCommand.java:57)
        at org.apache.whirr.cli.Main.run(Main.java:69)
        at org.apache.whirr.cli.Main.main(Main.java:102)

这是朝着正确的方向迈出的一步吗？如果是的话，我该怎么做才能让它工作

我是一个非常初学者，所以我非常感谢你的帮助，如果可能的话，我会尽可能的“清楚”，因为我是一个初学者，正如我所说的

下一步是运行此命令：

$ whirr run-script --script install-r+packages.sh --config hadoop-ec2.properties 

我真的希望能在这里找到一些帮助，这样我就可以继续学习教程了

Whirr配置文件：

whirr.cluster-name=hadoop-ec2

# Change the number of machines in the cluster here
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,5 hadoop-datanode+hadoop-tasktracker
# whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,1 hadoop-datanode+hadoop-tasktracker

# Uncomment out these lines to run CDH
# You need cdh3 because of the streaming combiner backport
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop
# make sure java is set up correctly, requires Whirr >= 0.7.1
whirr.java.install-function=install_oab_java 

# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=${env:DFD...(mycode)..DFDSDF}
whirr.credential=${env:df342.(mycode)..3434324}

# The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
whirr.hardware-id=m1.large
# whirr.hardware-id=c1.xlarge

# select recent, 64-bit CentOS 5.6 AMI from RightScale
whirr.image-id=us-east-1/ami-49e32320

# here's what Cloudera recommends:
# whirr.image-id=us-east-1/ami-ccb35ea5

# If you choose a different location, make sure whirr.image-id is updated too
whirr.location-id=us-east-1

# You can also specify the spot instance price
# http://aws.amazon.com/ec2/spot-instances/
# whirr.aws-ec2-spot-price=0.109

# By default use the user system SSH keys. Override them here.
# whirr.private-key-file=${sys:user.home}/.ssh/id_rsa
# whirr.public-key-file=${sys:user.home}/.ssh/id_rsa.pub

# Expert: override Hadoop properties by setting properties with the prefix
# hadoop-common, hadoop-hdfs, hadoop-mapreduce to set Common, HDFS, MapReduce
# site properties, respectively. The prefix is removed by Whirr, so that for
# example, setting 
# hadoop-common.fs.trash.interval=1440
# will result in fs.trash.interval being set to 1440 in core-site.xml.

# Expert: specify the version of Hadoop to install.
#whirr.hadoop.version=0.20.2
#whirr.hadoop.tarball.url=http://archive.apache.org/dist/hadoop/core/hadoop-${whirr.hadoop.version}/hadoop-${whirr.hadoop.version}.tar.gz

是删除

#

此处示例：

whirr.private−key−file=${sys:user.home}/.ssh/whirr_id_rsa
whirr.public−key−file=${sys:user.home}/.ssh/whirr_id_rsa.pub

用您的文件名重新安装

whirr\u id\u rsa

。详情如下：

---

您的错误表明凭据有问题。我可以指给你看。。希望这有帮助。您需要创建一个新的SSH密钥对以与whirr一起使用，希望您已经创建了它。如果不是这样，那可能就是问题所在。除此之外，请检查您的AWS凭据。它们不是您的常规用户名密码。我使用以下命令创建了ssh密钥：[cloudera@localhost~]$ssh keygen-t rsa-P“我在上面的帖子中添加了配置文件。也许你可以检查一下是否一切正常？谢谢。您需要在配置文件中提供私钥和公钥。在您当前的配置文件中，这些行都有注释。您可能还知道我下面问题的答案吗？太好了！

whirr.private−key−file=${sys:user.home}/.ssh/whirr_id_rsa
whirr.public−key−file=${sys:user.home}/.ssh/whirr_id_rsa.pub