lein-aws
This is very use case specific plugin to suit my own needs, but I'd like to expand it to be a more general purpose AWS plugin; pull requests welcomed!
Right it it supposed uploading files to s3 via the s3-put subtask,
running and terminating jobflow via the run-flow and terminate-job
subtask respectively. It is based on the awesome
amazonica library.
Job flows are defined and loaded from a file containing a clojure definition of a keyword identified to an amazonina run-job-flow map.
{:my-flow
{:name "my-job-flow"
:log-uri "s3n://emr-logs/logs"
:instances
{:instance-groups [
{:instance-type "m1.large"
:instance-role "MASTER"
:instance-count 1
:market "SPOT"
:bid-price "0.10"}]}
:steps [
{:name "my-step"
:hadoop-jar-step
{:jar "s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/hadoop-jobs/bigml"
:main-class "bigml.core"
:args ["s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/data" "output"]}}]}
}Since job flows are parsed as clojure, you can factor out common steps:
(let [enable-debugging
{:name "Enable Debugging"
:hadoop-jar-step
{:jar "s3://elasticmapreduce/libs/script-runner/script-runner.jar"
:args ["s3://elasticmapreduce/libs/state-pusher/0.1/fetch"]}}
master-m1-xlarge
{:instance-type "m1.xlarge"
:instance-role "MASTER"
:instance-count 1}]
{:stage-job-flow
{:name "Run job flow in stage"
:log-uri "s3n://emr-logs/logs"
:instances
{:hadoop-version "1.0.3"
:instance-groups
[master-m1-xlarge]}
:steps
[enable-debugging
{:name "run-step-in-staging"
:hadoop-jar-step
{:jar "s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/hadoop-jobs/bigml"
:main-class "bigml.core"
:args ["s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/data" "staging"]}}]}
:prod-job-flow
{:name "Run job flow in prod"
:log-uri "s3n://emr-logs/logs"
:instances
{:hadoop-version "1.0.3"
:instance-groups
[master-m1-xlarge]}
:steps
[enable-debugging
{:name "my-step"
:hadoop-jar-step
{:jar "s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/hadoop-jobs/bigml"
:main-class "bigml.core"
:args ["s3n://beee0534-ad04-4143-9894-8ddb0e4ebd31/data" "prod"]}}]}})See the amazonica docs for more information.
Usage
You need to let the plugin know about your aws credentils:
:aws {:access-key "ACCESSKEY"
:secret-key "SECRETKEY"
:email "aws-login-email"}Use this for user-level plugins:
Put [lein-aws "0.1.4-SNAPSHOT"] into the :plugins vector of your
:user profile, or if you are on Leiningen 1.x do lein plugin install
lein-aws 0.1.4-SNAPSHOT.
Use this for project-level plugins:
Put [lein-aws "0.1.4-SNAPSHOT"] into the :plugins vector of your project.clj.
$ lein help aws
License
Copyright © 2013 Soren Macbeth
Distributed under the Eclipse Public License, the same as Clojure.
