lxmlrpc_monkey

xmlrpclib patch to reduce its memory consumption


Keywords
xmlrpc, xmlrpclib, xml-rpc
License
GPL-2.0+
Install
pip install lxmlrpc_monkey==1.0.0

Documentation

lxmlrpc - XMLRPClib patch

This module monkeypatch python's xmlrpclib to use lxml based parser to reduce memory consumption on big xmlrpc requests / responses (100+ Mb)

NOTE

This module is useful only for *python2.7*.

Use it only if you suffer from high memory consumption of xmlrpclib

Install

This module is hosted on PyPI so it could be easily installed via pip:

Usage

To use this module just do following

Benchmarks (how to run)

To run benchmarks:

  1. install memory_profiler
  2. prepare data with python generate_data.py --path <demo data path> --size 50000000
  3. run benchmarks with python benchmark.py --path <demo data path>

Benchmark results (50 Mb (real 65 Mb) data file)

look at `p.feed(data) call in loads` function of xmlrpclib

*Running unpatched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents

1134 1135 1136 1137 1138 1139 1140 1141 1142

104.7 MiB

0.0 MiB

def loads(data, use_datetime=0):

"""data -> unmarshalled data, method name

Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present).

If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """

1143

104.7 MiB

0.0 MiB

p, u = getparser(use_datetime=use_datetime)

>1144

622.4 MiB 517.7 MiB

p.feed(data)

1145

558.0 MiB -64.4 MiB

p.close()

1146

558.0 MiB

0.0 MiB

return u.close(), u.getmethodname()

*Running patched loads*

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents

1134 1135 1136 1137 1138 1139 1140 1141 1142

106.9 MiB

0.0 MiB

def loads(data, use_datetime=0):

"""data -> unmarshalled data, method name

Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present).

If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """

1143

106.9 MiB

0.0 MiB

p, u = getparser(use_datetime=use_datetime)

>1144

235.9 MiB 129.0 MiB

p.feed(data)

1145

171.5 MiB -64.4 MiB

p.close()

1146

171.5 MiB

0.0 MiB

return u.close(), u.getmethodname()

Filename: bechmark.py

Line # Mem usage Increment Line Contents

13 14

104.7 MiB

0.0 MiB

@profile def bench_load(xmldata):

15

104.7 MiB

0.0 MiB

print ("Running unpatched loads")

16 17

106.9 MiB

2.2 MiB

loads(xmldata)

18 19

106.9 MiB

0.0 MiB

lxmlrpc.patch_xmlrpclib()

20

106.9 MiB

0.0 MiB

print ("Running patched loads")

21

107.1 MiB

0.2 MiB

loads(xmldata)

Benchmark results (100 Mb (real 129 Mb) data file)

*Running unpatched loads*

---

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents

1134 1135 1136 1137 1138 1139 1140 1141 1142

169.2 MiB

0.0 MiB

def loads(data, use_datetime=0):

"""data -> unmarshalled data, method name

Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present).

If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """

1143

169.2 MiB

0.0 MiB

p, u = getparser(use_datetime=use_datetime)

>1144

1203.0 MiB 1033.8 MiB

p.feed(data)

1145

1074.2 MiB -128.8 MiB

p.close()

1146

1074.2 MiB

0.0 MiB

return u.close(), u.getmethodname()

*Running patched loads*

---

Filename: /usr/lib/python2.7/xmlrpclib.py

Line # Mem usage Increment Line Contents

1134 1135 1136 1137 1138 1139 1140 1141 1142

171.6 MiB

0.0 MiB

def loads(data, use_datetime=0):

"""data -> unmarshalled data, method name

Convert an XML-RPC packet to unmarshalled data plus a method name (None if not present).

If the XML-RPC packet represents a fault condition, this function raises a Fault exception. """

1143

171.6 MiB

0.0 MiB

p, u = getparser(use_datetime=use_datetime)

>1144

429.4 MiB

257.8 MiB

p.feed(data)

1145

300.6 MiB

-128.8 MiB

p.close()

1146

300.6 MiB

0.0 MiB

return u.close(), u.getmethodname()

Filename: bechmark.py

Line # Mem usage Increment Line Contents

13 14

169.2 MiB

0.0 MiB

@profile def bench_load(xmldata):

15

169.2 MiB

0.0 MiB

print ("Running unpatched loads")

16 17

171.6 MiB

2.4 MiB

loads(xmldata)

18 19

171.6 MiB

0.0 MiB

lxmlrpc.patch_xmlrpclib()

20

171.6 MiB

0.0 MiB

print ("Running patched loads")

21

171.8 MiB

0.2 MiB

loads(xmldata)