Python Wrapper for the Yahoo! Query Language. Allowing to run YQL queries, fetch financial data and create YQL Open Tables


Keywords
myql, yql, yahoo, query, language
License
MIT
Install
pip install myql==1.2.7

Documentation

mYQL

Build Status Documentation Status Code Health PyPI PyPI PyPI PyPI PyPI Coverage Status PyPI

mYQL is a Python wrapper of the Yahoo Query Language.

Yahoo! Query Language Documentation and Support

Features

  • Simple YQL Query
  • Authenticated YQL Query ( OAuth )
  • StockScraper
  • YQL Open Table (Classes and Metaclasses) Generator
  • Response prettyfier

Installation

$ pip install myql

Quick Start

It's important to know that response is a just requests.models.Response object. Yes indeed, mYQL uses requests 😄

By default, you have access to the community tables. If for whatsoever reason you would like to not have access to those tables

>>> import myql
>>> yql = myql.MYQL(community=False)

Changing response format (xml or json)

The response format is by default json.

>>> import myql
>>> from myql.utils import pretty_json, pretty_xml
>>> yql = myql.MYQL(format='xml', community=True)
>>> resp = yql.raw_query('select name, woeid from geo.states where place="Congo"')
>>> print(pretty_xml(resp.content))
<?xml version="1.0" encoding="utf-8"?>
<query xmlns:yahoo="http://www.yahooapis.com/v1/base.rng" yahoo:count="11" yahoo:created="2015-06-07T11:56:11Z" yahoo:lang="en-US">
    <results>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Cuvette-Ouest Department</name>
            <woeid>55998384</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Cuvette Department</name>
            <woeid>2344968</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Plateaux District</name>
            <woeid>2344973</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Sangha</name>
            <woeid>2344974</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Lekoumou</name>
            <woeid>2344970</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Pool Department</name>
            <woeid>2344975</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Likouala Department</name>
            <woeid>2344971</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Niari Department</name>
            <woeid>2344972</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Brazzaville</name>
            <woeid>2344976</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Bouenza Department</name>
            <woeid>2344967</woeid>
        </place>
        <place xmlns="http://where.yahooapis.com/v1/schema.rng">
            <name>Kouilou</name>
            <woeid>2344969</woeid>
        </place>
    </results>
</query>
<!-- total: 33 -->
<!-- pprd1-node1003-lh3.manhattan.bf1.yahoo.com -->

>>> resp = yql.raw_query('select name, woeid from geo.states where place="Congo"', format='json')
>>> print(pretty_json(resp.content))
{
    "query": {
        "count": 11, 
        "created": "2015-06-07T11:58:20Z", 
        "lang": "en-US", 
        "results": {
            "place": [
                {
                    "name": "Cuvette-Ouest Department", 
                    "woeid": "55998384"
                }, 
                {
                    "name": "Cuvette Department", 
                    "woeid": "2344968"
                }, 
                {
                    "name": "Plateaux District", 
                    "woeid": "2344973"
                }, 
                {
                    "name": "Sangha", 
                    "woeid": "2344974"
                }, 
                {
                    "name": "Lekoumou", 
                    "woeid": "2344970"
                }, 
                {
                    "name": "Pool Department", 
                    "woeid": "2344975"
                }, 
                {
                    "name": "Likouala Department", 
                    "woeid": "2344971"
                }, 
                {
                    "name": "Niari Department", 
                    "woeid": "2344972"
                }, 
                {
                    "name": "Brazzaville", 
                    "woeid": "2344976"
                }, 
                {
                    "name": "Bouenza Department", 
                    "woeid": "2344967"
                }, 
                {
                    "name": "Kouilou", 
                    "woeid": "2344969"
                }
            ]
        }
    }
}

>>>

Methods

use(yql_table_url,name=yql_table_name)

Maps a table name to the URL of an Open Data Table.

>>> yql.use('http://www.josuebrunel.org//users.xml', name='myusers') 

desc(tablename)

Returns table description

>>> response = yql.desc('weather.forecast')
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1, 
        "created": "2015-06-07T12:00:27Z", 
        "lang": "en-US", 
        "results": {
            "table": {
                "hash": "aae78b1462a6a8fbc748aec4cf292767", 
                "meta": {
                    "author": "Yahoo! Inc", 
                    "description": "Weather forecast table", 
                    "documentationURL": "http://developer.yahoo.com/weather/", 
                    "sampleQuery": "select * from weather.forecast where woeid=2502265"
                }, 
                "name": "weather.forecast", 
                "request": {
                    "select": [
                        {
                            "key": [
                                {
                                    "name": "location", 
                                    "required": "true", 
                                    "type": "xs:string"
                                }, 
                                {
                                    "name": "u", 
                                    "type": "xs:string"
                                }
                            ]
                        }, 
                        {
                            "key": [
                                {
                                    "name": "woeid", 
                                    "required": "true", 
                                    "type": "xs:string"
                                }, 
                                {
                                    "name": "u", 
                                    "type": "xs:string"
                                }
                            ]
                        }
                    ]
                }, 
                "security": "ANY"
            }
        }
    }
}

>>>

raw_query(query)

Allows you to directly type your query

>>> response = yql.raw_query("select * from geo.countries where place='North America'")
>>> # deal with the response

select(table, fields, limit).where(filters, ...)

NB : A simple select doesn't return any data. Use GET instead.

>>> response = yql.select('geo.countries', ['name', 'code', 'woeid']).where(['name', '=', 'Canada'])
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1, 
        "created": "2015-06-07T12:10:39Z", 
        "lang": "en-US", 
        "results": {
            "place": {
                "name": "Canada", 
                "woeid": "23424775"
            }
        }
    }
}

>>> ...
>>> response = yql.select('geo.countries', ['name', 'woeid'], 2).where(['place', 'in', ('Africa', 'Europe')])
>>> from myql.utils import dump
>>> dump(response)
{
    "query": {
        "count": 2, 
        "created": "2015-06-07T12:27:04Z", 
        "lang": "en-US", 
        "results": {
            "place": [
                {
                    "name": "Algeria", 
                    "woeid": "23424740"
                }, 
                {
                    "name": "Angola", 
                    "woeid": "23424745"
                }
            ]
        }
    }
}

>>>

get(table, fields, limit)

Same as SELECT, but instead returns data.

REMINDER : Some tables require a where clause, therefore GET won't work on those tables, use select(...).where(...) instead .

>>> from myql.utils import dump
>>> response = yql.get('geo.countries', ['name', 'woeid'], 1)
>>> dump(response)
{
    "query": {
        "count": 1, 
        "created": "2015-06-07T12:29:01Z", 
        "lang": "en-US", 
        "results": {
            "place": {
                "name": "Sao Tome and Principe", 
                "woeid": "23424966"
            }
        }
    }
}

>>>

insert(table, (field1, field2, ..., fieldN),(value1, value2, ..., valueN))

Insert values into a table. Arguments 2 and 3 may be tuples or list.

>>> from myql.utils import pretty_json
>>> response = yql.insert('yql.storage.admin',('value',),('http://josuebrunel.org',))
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1,
        "created": "2015-05-14T13:25:56Z",
        "lang": "en-US",
        "results": {
            "inserted": {
                "execute": "store://KkkC5xDw4v32IcWWSQ4YRe",
                "select": "store://Zc5LHXcmYM7XBfSbo9tzFL",
                "update": "store://Rqb5fbQyDvrfHJiClWnZ6q"
            }
        }
    }
}

update(table,[field1, ..., fieldN],[value1, ..., ...valueN]).where(filters, ...)

Update fields values. This method is always followed by where(). Arguments 2 and 3 may be tuples or list.

>>> from myql.utils import pretty_json
>>> response = yql.update('yql.storage',('value',),('https://josuebrunel.org',)).where(['name','=','store://Rqb5fbQyDvrfHJiClWnZ6q'])
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1,
        "created": "2015-05-14T13:32:52Z",
        "lang": "en-US",
        "results": {
            "success": "Updated store://KkkC5xDw4v32IcWWSQ4YRe"
        }
    }
}

delete(table).where(filters, ...)

Delete records

>>> from myql.utils import pretty_json
>>> response = self.yql.delete('yql.storage').where(['name','=','store://Rqb5fbQyDvrfHJiClWnZ6q'])
>>> print(pretty_json(response.content))
{
    "query": {
        "count": 1,
        "created": "2015-05-14T13:38:28Z",
        "lang": "en-US",
        "results": {
            "success": "store://Rqb5fbQyDvrfHJiClWnZ6q deleted"
        }
    }
}

Using OAuth

mYQL comes with yahoo_oauth, which is an OAuth library for Yahoo! APIs.

>>> from yahoo_oauth import OAuth1
>>> oauth = OAuth1(None, None, from_file='credentials.json') # only consumer_key and consumer_secret are required.
>>> from myql import MYQL
>>> yql = MYQL(format='xml', oauth=oauth)
>>> response = yql.get_guid('josue_brunel') # Deal with the response

Stocks Scraper

The full documentation on StockScraper is here

Release Notes

1.2.6

1.2.5

  • camelCase dropped for underscore
  • Support for substitution variable i.e @myvar
  • Support of Remote Filters
  • Support of Post Query Filters
1.2.4

  • Weather module added
  • StockScraper now under Finance namespace
1.2.3

  • Fixed issue related to date in StockRetriver.get_historical_info #107
  • Fixed issue with IN condition in where clause #106
  • Fix definition of raw_input for python3 #105
  • Yahoo-OAuth included as main oauth library #112
1.2.2

  • Python3 support OK #71
  • PyPy/PyPy3 support OK
  • Fixed issue with IN condition in where clause
  • Fixed issue when passing an empty list/tuple ([]/()) in a where clause besides first argument
  • Import of StockParser from Gurchet Rai OK #68
  • Insert, Update, Delete methods added #67
  • Dummy try/except removed from main module
  • Fixed Invalid OAuth Signature when using a refreshed token #64
  • Fixed misused of MYQL.use(...) #76
  • Fixed format issue #82
  • Added useful functions in utils #81
  • Default access to community tables
  • Response prettyfier : pretty_json, pretty_xml
v 1.2.1

  • Multiple requests while using OAuth fixed
1.2.0

  • OpenTable classes
  • Access to resources requiring authentication
0.5.6

  • fetch data
  • access to community data
  • select data format (xml/json)
  • change data source
  • filter data
  • fix handling of default response format on the fly
  • fix limit on select(...).where(...) when no limit value is passed
  • fix limit on get(...)