sourcerer

Library to programatically genrate python source code


Keywords
sourcerer
License
Apache-2.0
Install
pip install sourcerer==1.0b7

Documentation

Sourcerer

image

Programatically generate PEP8 python source code

Running examples

shark@tack ~/sourcerer/examples $ python swagger_to_flask.py ../sample_data/uber.yaml
shark@tack ~/sourcerer/examples $ python call_example.py

Generate code from code

Let’s start with the absolute basics

from sourcerer import Document, Statement

# The most important class in Sourcerer is the statement. Just about everything is a statement.
# Statements hold source code, and have a child scope. They can hold other statements.
# Even a document is just a special kind of Statement.
# A Document job is to hold statements and then output its contents.
doc = Document()

# Now we make a generic statement to assign 1 to x
s = Statement("x = 1")

# Add it to the  document.
# add_child() is a member function of Statement. It will append to a Statements child scope.
doc.add_child(s)

# Now output the current document
# output() without an output_file_name will output to standard out.

doc.output()

Output:

x = 1

Let’s use some of the purpose built tools in Sourcerer to make this easier to generate.

from sourcerer import Document, Name

doc = Document()

# Names are variable/function/class/etc... names
# We'll use a good name
good_name = Name("descriptive_name")

# and a bad name
bad_name = Name("1@*plz_help")

# Add both the children at once

doc.add_children([good_name, bad_name])

doc.output()

Output:

descriptive_name
plz_help

Notice the bad name has been transformed into a valid python name, this behavior can be turned off by setting validate=False

Let’s get back to that naive assignment we first made. We can improve it using Name and Assignment. We’ll also use Num just for good practice.

from sourcerer import Document, Name, Assignment, Num

doc = Document()

# We'll wrap this up in one line because it's not that long.
# Num can take a string, int, long, float, etc...
a = Assignment(Name("x"), Num("1"))

doc.add_child(a)

doc.output()

Output:

x = 1

Now that we’re warmed up, let’s do something more interesting. How about some functions?

from sourcerer import Document, FunctionDef, Return, Str, Num, Name, Assignment, DecoratorDef, Call

doc = Document()

# A function that returns 0
func_a = FunctionDef(name=Name("get_a_zero"))
ret_a = Return(Num("0"))

func_a.add_child(ret_a)
doc.add_child(func_a)

# A function that passes. We'll put it in an list for easier consumption later
func_b = [FunctionDef(name=Name("just_pass")),
          Return(_type="pass")
]

# Cascade the list of statements
doc.create_lineage(func_b)

# A function with args, and a *arg
func_c = [FunctionDef(name=Name("so_many_args"), arg_names=["a1", Name("a2")], varargs="args"),
          Return(Str("Not enough time"))
]

doc.create_lineage(func_c)

# A function with kwargs, and a **
func_d = [FunctionDef(name=Name("so_many_kwargs"), kwarg_pairs={Name("a1"):"val"}, keywords="kwargs"),
          Return()
]

doc.create_lineage(func_d)

# A function decorated function. Philosophy: If things get complicated, just make them a list.
func_e = [DecoratorDef(name=Name("fancy")),
          FunctionDef(name=Name("pants")),
          Return(Str("Hello World!"))
]

doc.create_lineage(func_e)

doc.output()

Output:

def get_a_zero():
    return 0


def just_pass():
    pass


def so_many_args(a1, a2, *args):
    return "Not enough time"


def so_many_kwargs(a1=val, **kwargs):
    return


@fancy()
def pants():
    return "Hello World!"

Here is an example that generates an extremely rough flask Blueprint from a swagger (http://swagger.io/) yml doc

from yaml import load
from sourcerer import Document, FunctionDef, DecoratorDef, Return, Str, Name, Call, Assignment, Attribute
from sys import argv

# Create a document to put our code in
doc = Document()

# Open our yml file and read it in
api = load(open(argv[1], 'r').read())

blueprint = Name(api['basePath'])

bp = Assignment(blueprint,
                Call(name="Blueprint",
                     arg_names=[Str(blueprint), '__name__'],
                     kwarg_pairs={'template_folder': Str('templates')}))

doc.add_child(bp)

for path in api['paths']:
    route = [DecoratorDef(name=Attribute(caller_list=[blueprint], name=Name('route')),
                          arg_names=[Str(path)]), # A decorator: @routename("mypath")
             FunctionDef(name=Name(path)), # A function: def routename():
             Return()] # A return statement: return

    doc.create_lineage(route) # Cascade these objects into the main document scope
                              # ...
                              # @routename("mypath")
                              # def routename():
                              #     return
                              # ...

doc.output() # Send output to standard out (output to file optional)

####################
# Without inline comments:
##

from yaml import load
from sourcerer import Document, FunctionDef, DecoratorDef, Return, Str, Name, Call, Assignment, Attribute
from sys import argv

doc = Document()

api = load(open(argv[1], 'r').read())

blueprint = Name(api['basePath'])

bp = Assignment(blueprint,
                Call(name="Blueprint",
                     arg_names=[Str(blueprint), '__name__'],
                     kwarg_pairs={'template_folder': Str('templates')}))
doc.add_child(bp)

for path in api['paths']:
    route = [DecoratorDef(name=Attribute(caller_list=[blueprint], name=Name('route')),
                          arg_names=[Str(path)]),
             FunctionDef(name=Name(path)),
             Return()]
    doc.create_lineage(route)
    
doc.output()

Output:

v1 = Blueprint("v1", __name__, template_folder="templates")


@v1.route("/products")
def products():
    return


@v1.route("/estimates/price")
def estimatesprice():
    return


@v1.route("/history")
def history():
    return


@v1.route("/me")
def me():
    return


@v1.route("/estimates/time")
def estimatestime():
    return

Technologies

  • YAPF formatted output to produce pep8 compliant code

Upcoming Features

Generate code from Spellbooks

Source code can also be generated by ingesting and parsing a config document (ex. yaml, json, xml…), known as a Spellbook. Spellbooks can be parsed into source code be defining a schema, called a Syntax Map.

Example Spellbook (YAML):

functions:
    func1:
        args: ['thing1', 'thing2']
        kwargs: {"key1": "val1"}
        varargs: false
        keywords: false
        ret:
            value:
                true

Example Syntax Map to parse this Spellbook:

# Without inline comments
{"functions": {'type': FunctionDef,
               'key': 'name',
               'value_map': {'args': 'arg_names',
                             'kwargs': 'kwarg_pairs',
                             'varargs': 'varargs',
                             'keywords': 'keywords'},
               'children':{'ret':'return'}},
 "return": {'type': Return,
             'value_map': {'value':'val'}}
}

# With inline comments
{"functions": {'type': FunctionDef, # Each top level entry under functions is a FunctionDef
               'key': 'name', # The functions key (func1) is the value to the name argument for the FunctionDef
               'value_map': {'args': 'arg_names',  # Define what args are called in the markups schema
                             'kwargs': 'kwarg_pairs',
                             'varargs': 'varargs',
                             'keywords': 'keywords'},
               'children':{'ret':'return'}}, # When top-level objects are seen their values will be 
                                             # pared as well, building a new object from the mapping 
                                             # they specify and then appended to this object
 "return": {'type': Return,
             'value_map': {'value':'val'}}
}

Building a Syntax Map for a Spellbook:

Your Syntax Maps top-level keys define what your Spellbook top-level sections are containing. The values of your Syntax Map top-level keys are dictionaries defining how to handle the contents of your Spellbook sections.

In the given example, the only top-level Spellbook section is ‘functions’. In the Syntax Map, the ‘functions’ key’s value says several things:

  1. For each child node encountered, create a new FunctionObj (defined by ‘type’)
  2. The key defining each child node is the ‘name’ argument for the FunctionObj
  3. The sub-keys of the child node are properties of the FunctionObj. The values of those sub-keys are can be one of two things:
    • If the value is in the value map, it is an argument to FunctionObj
    • If the value is in the children map, it should be placed into the scope of the FunctionObj. The value will be looked up in the Syntax Map top-level to see if it can be be instantiated into a new sourcerer object.

The Syntax Map schema should consist of:

  • type (required): The class name to instantiate
  • key (required): what the key for the node represents
  • value_map (required): map properties to arguments to the class
  • children: values that should be instantiated and placed into the current nodes child scope

Using a Syntax Map and Spellbook to generate your source:

Based on the example Syntax Map and the Example YAML, the following will write the resulting source code to stardard out

from sourcerer import YAMLProcessor

gen = YAMLProcessor()
gen.load('sample_data/sample.yml')
gen.output()