twitter/scalding


A Scala API for Cascading

http://twitter.com/scalding

License: Apache-2.0

Language: Scala


Scalding

Build status Coverage Status Latest version Chat

Scalding is a Scala library that makes it easy to specify Hadoop MapReduce jobs. Scalding is built on top of Cascading, a Java library that abstracts away low-level Hadoop details. Scalding is comparable to Pig, but offers tight integration with Scala, bringing advantages of Scala to your MapReduce jobs.

Scalding Logo

Word Count

Hadoop is a distributed system for counting words. Here is how it's done in Scalding.

package com.twitter.scalding.examples

import com.twitter.scalding._
import com.twitter.scalding.source.TypedText

class WordCountJob(args: Args) extends Job(args) {
  TypedPipe.from(TextLine(args("input")))
    .flatMap { line => tokenize(line) }
    .groupBy { word => word } // use each word for a key
    .size // in each group, get the size
    .write(TypedText.tsv[(String, Long)](args("output")))

  // Split a piece of text into individual words.
  def tokenize(text: String): Array[String] = {
    // Lowercase each word and remove punctuation.
    text.toLowerCase.replaceAll("[^a-zA-Z0-9\\s]", "").split("\\s+")
  }
}

Notice that the tokenize function, which is standard Scala, integrates naturally with the rest of the MapReduce job. This is a very powerful feature of Scalding. (Compare it to the use of UDFs in Pig.)

You can find more example code under examples/. If you're interested in comparing Scalding to other languages, see our Rosetta Code page, which has several MapReduce tasks in Scalding and other frameworks (e.g., Pig and Hadoop Streaming).

Documentation and Getting Started

Please feel free to use the beautiful Scalding logo artwork anywhere.

Contact

For user questions or scalding development (internals, extending, release planning): https://groups.google.com/forum/#!forum/scalding-dev (Google search also works as a first step)

In the remote possibility that there exist bugs in this code, please report them to: https://github.com/twitter/scalding/issues

Follow @Scalding on Twitter for updates.

Chat: Gitter

Get Involved + Code of Conduct

Pull requests and bug reports are always welcome!

We use a lightweight form of project governence inspired by the one used by Apache projects. Please see Contributing and Committership for our code of conduct and our pull request review process. The TL;DR is send us a pull request, iterate on the feedback + discussion, and get a +1 from a Committer in order to get your PR accepted.

The current list of active committers (who can +1 a pull request) can be found here: Committers

A list of contributors to the project can be found here: Contributors

Building

There is a script (called sbt) in the root that loads the correct sbt version to build:

  1. ./sbt update (takes 2 minutes or more)
  2. ./sbt test
  3. ./sbt assembly (needed to make the jar used by the scald.rb script)

The test suite takes a while to run. When you're in sbt, here's a shortcut to run just one test:

> test-only com.twitter.scalding.FileSourceTest

Please refer to FAQ page if you encounter problems when using sbt.

We use Travis CI to verify the build: Build Status

We use Coveralls for code coverage results: Coverage Status

Scalding modules are available from maven central.

The current groupid and version for all modules is, respectively, "com.twitter" and 0.17.2.

Current published artifacts are

  • scalding-core_2.11, scalding-core_2.12
  • scalding-args_2.11, scalding-args_2.12
  • scalding-date_2.11, scalding-date_2.12
  • scalding-commons_2.11, scalding-commons_2.12
  • scalding-avro_2.11, scalding-avro_2.12
  • scalding-parquet_2.11, scalding-parquet_2.12
  • scalding-repl_2.11, scalding-repl_2.12

The suffix denotes the scala version.

Adopters

  • Ebay
  • Etsy
  • Sharethrough
  • Snowplow Analytics
  • Soundcloud
  • Twitter

To see a full list of users or to add yourself, see the wiki

Authors:

Thanks for assistance and contributions:

A full list of contributors can be found on GitHub.

License

Copyright 2016 Twitter, Inc.

Licensed under the Apache License, Version 2.0

Project Statistics

Sourcerank 15
Repository Size 17.8 MB
Stars 3,080
Forks 658
Watchers 342
Open issues 309
Dependencies 0
Contributors 142
Tags 92
Created
Last updated
Last pushed

Top Contributors See all

P. Oscar Boykin Ian O'Connell Ruban Monu Argyris Zymnis Kevin Lin Alex Levenson Sid Kapur Sam Ritchie Mansur Ashraf Piyush Narang Eddie Xie Ben Pence Avi Bryant koertkuipers Aaron Siegel Sriram Krishnan Jonathan Coveney Adam Pingel Anton Panasenko Richard Whitcomb

Packages Referencing this Repo

com.twitter:scalding-commons_2.12
scalding-commons
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:scalding-db_2.12
scalding-db
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:scalding-parquet-scrooge_2.12
scalding-parquet-scrooge
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:scalding-json_2.12
scalding-json
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:scalding-core_2.12
scalding-core
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:scalding-thrift-macros_2.11
scalding-thrift-macros
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:scalding-hadoop-test_2.12
scalding-hadoop-test
Latest release 0.17.4 - Updated - 3.08K stars
scalding-parquet-cascading
scalding-parquet-cascading
Latest release 0.16.1-cascading3-RC3 - Updated - 3.08K stars
scalding-hraven
scalding-hraven
Latest release 0.17.4 - Updated - 3.08K stars
scalding-db
scalding-db
Latest release 0.17.4 - Updated - 3.08K stars
scalding-avro
scalding-avro
Latest release 0.17.4 - Updated - 3.08K stars
scalding-serialization
scalding-serialization
Latest release 0.17.4 - Updated - 3.08K stars
scalding-args
scalding-args
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:execution-tutorial_2.11
execution-tutorial
Latest release 0.17.4 - Updated - 3.08K stars
com.twitter:scalding-serialization-macros_2.10
scalding-serialization-macros
Latest release 0.15.0 - Published - 3.08K stars
com.twitter:scalding-db_2.10
scalding-db
Latest release 0.15.1-RC9 - Updated - 3.08K stars
com.twitter:execution-tutorial_2.10
execution-tutorial
This package is no longer available on Maven
com.twitter:scalding-hadoop-test_2.10
scalding-hadoop-test
This package is no longer available on Maven
com.twitter:scalding-json_2.11
scalding-json
This package is no longer available on Maven
com.twitter:scalding-date_2.9.2
scalding-date
This package is no longer available on Maven

Recent Tags See all

v0.17.4 December 22, 2017
v0.17.3 September 30, 2017
v0.17.2 July 14, 2017
0.17.2 July 14, 2017
v0.17.1 July 10, 2017
v0.17.0 April 11, 2017
v0.16.1-cascading3-RC4 October 05, 2016
v0.16.1-cascading3-RC3 September 14, 2016
v0.16.1-cascading3-RC2 August 10, 2016
v0.16.1-cascading3-RC1 July 20, 2016
v0.16.1-RC3 June 29, 2016
v0.16.1-RC2 June 06, 2016
v0.16.1-RC1 June 03, 2016
v0.16.0 May 03, 2016
v0.16.0-RC6 April 05, 2016

Interesting Forks See all

stripe/scalding
A Scala API for Cascading
Scala - Apache-2.0 - Last pushed - 3 stars - 2 forks
azymnis/scalding
A Scala API for Cascading
Scala - Apache-2.0 - Last pushed - 3 stars
tresata/scalding
A Scala API for Cascading
Scala - Apache-2.0 - Last pushed - 2 stars
delftswa2014/scalding
A Scala API for Cascading
Scala - Apache-2.0 - Updated - 1 stars
agilemobiledev/scalding
A Scala API for Cascading
Scala - Apache-2.0 - Updated - 1 stars

Something wrong with this page? Make a suggestion

Last synced: 2017-04-11 05:23:56 UTC

Login to resync this repository