erlenv
Goal
Make retrieved configuration parameters used by the application super fast and stable.
In general way, the value of a configuration parameter is retrieved by calling application:get_env/1-2-3.
Cache configuration parameters from ets to beam to replace application:get_env/2-3.
Rationale
Application store configuration in ets(ac_tab) with {{env, AppName, Key}, Value},
application:get_env/2-3 should lookup ets(ac_tab) by {env, AppName, Key} every time.
The retrieve speed become slow and unstable when lots of processes lookup ac_tab table at the same time.
This project dynamic compile configuration to beam. so we can speed up significantly.
PS: Only suitable for situations with very low write needs and very high read concurrency.
- configuration in
sys.config.
[
{app_name1, [{ip, "127.0.0.1"}]},
{app_name2, [{port, 8080}]}
].- Those configuration store in
ac_tabets as:
[{{env, app_name1, ip}, "127.0.0.1"},{{env, app_name2, port}, 8080}].lookup by application:get_env(app_name1, ip).
3. Those configuration load into beam by dynamic compile as:
# eenv_router.beam
-module(eenv_router).
-export([get/2]).
get(app_name1, Key) -> eenv_app_name1:get(Key);
get(app_name2, Key) -> eenv_app_name2:get(Key);
get(_, _) -> unload.
# eenv_app_name1.beam
-module(eenv_app_name1).
-export([get/1]).
get(ip) -> {ok, "127.0.0.1"};
get(_) -> undefined.
# eenv_app_name2.beam
-module(eenv_app_name2).
-export([get/1]).
get(port) -> {ok, 8080};
get(_) -> undefined.
lookup by eenv:get(app_name1, ip).
Every loaded application will generate beam file call eenv_'Application'.beam to store self configuration.
HowTo
- Load eenv when your project application start.
# YourProject_app.erl
-behaviour(application).
%% Application callbacks
start(_StartType, _StartArgs) ->
eenv:load(your_project_name),
your_project_sup:start_link().
stop(_State) ->
eenv:unload(your_project_name),
ok. - Replace.
-
application:get/2-3=>eenv:get/2-3. -
application:set/2-4=>eenv:set/2-4. -
application:unset/2-3=>eenv:unset/2-3.
Benchmark
Benchmark Type Spec
-
dictcopy env into process dictionary, every process has its own env dictionary. -
appread env byapplication:get/2. -
eenvread env byeenv:get/2. -
coderead env by static beam.
Benchmark detail
- Pids = [spawn worker_process ||_ <- lists:seq(1, ProcNum)].
- [send(Pid, start) ||Pid<- Pids].
- worker_process: {Cost, _} = timer:tc(fun() -> do_benchmark_loop(Count) end, send(MasterPid, Cost).
- Master collect
Timefrom all processes(Time = sum(process_1...process_n)).
More information see benchmark/eenv_benchmark.erl.
➜ make benchmark
Starting benchmark...
Erlang/OTP 20 [erts-9.0] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:false]
Eshell V9.0 (abort with ^G)
1> Run 1 processes 10000 count/process. EmptyLoop: 0.00600000us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.039| 386.000| 1.00x dict | 1.00x dict |
eenv| 0.070| 703.000| 1.82x dict | 1.82x dict |
code| 0.252| 2521.000| 6.53x dict | 3.59x eenv |
app| 0.311| 3112.000| 8.06x dict | 1.23x code |
--------------------------------------------------------------------------------------------------------
Run 10 processes 10000 count/process. EmptyLoop: 0.01020000us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.030| 301.900| 1.00x dict | 1.00x dict |
eenv| 0.180| 1804.700| 5.98x dict | 5.98x dict |
code| 0.195| 1949.900| 6.46x dict | 1.08x eenv |
app| 0.988| 9877.500| 32.72x dict | 5.07x code |
--------------------------------------------------------------------------------------------------------
Run 20 processes 10000 count/process. EmptyLoop: 0.01045000us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.025| 254.050| 1.00x dict | 1.00x dict |
code| 0.165| 1649.500| 6.49x dict | 6.49x dict |
eenv| 0.209| 2094.250| 8.24x dict | 1.27x code |
app| 1.559| 15586.650| 61.35x dict | 7.44x eenv |
--------------------------------------------------------------------------------------------------------
Run 30 processes 10000 count/process. EmptyLoop: 0.00893333us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.040| 396.700| 1.00x dict | 1.00x dict |
eenv| 0.279| 2790.300| 7.03x dict | 7.03x dict |
code| 0.300| 2995.900| 7.55x dict | 1.07x eenv |
app| 1.780| 17804.033| 44.88x dict | 5.94x code |
--------------------------------------------------------------------------------------------------------
Run 40 processes 10000 count/process. EmptyLoop: 0.01080000us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.041| 406.775| 1.00x dict | 1.00x dict |
code| 0.341| 3406.975| 8.38x dict | 8.38x dict |
eenv| 0.384| 3841.500| 9.44x dict | 1.13x code |
app| 2.246| 22458.725| 55.21x dict | 5.85x eenv |
--------------------------------------------------------------------------------------------------------
Run 50 processes 10000 count/process. EmptyLoop: 0.01022000us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.041| 410.880| 1.00x dict | 1.00x dict |
eenv| 0.305| 3053.020| 7.43x dict | 7.43x dict |
code| 0.378| 3780.360| 9.20x dict | 1.24x eenv |
app| 2.787| 27866.280| 67.82x dict | 7.37x code |
--------------------------------------------------------------------------------------------------------
Run 1000 processes 10000 count/process. EmptyLoop: 0.01233100us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.028| 275.264| 1.00x dict | 1.00x dict |
code| 7.323| 73225.561| 266.02x dict | 266.02x dict |
eenv| 7.795| 77952.363| 283.19x dict | 1.06x code |
app| 60.979| 609787.675| 2215.28x dict | 7.82x eenv |
--------------------------------------------------------------------------------------------------------
Run 2000 processes 10000 count/process. EmptyLoop: 0.01063850us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.032| 316.159| 1.00x dict | 1.00x dict |
eenv| 17.672| 176722.472| 558.97x dict | 558.97x dict |
code| 19.349| 193485.290| 611.99x dict | 1.09x eenv |
app| 167.384| 1673840.405| 5294.29x dict | 8.65x code |
--------------------------------------------------------------------------------------------------------
Run 3000 processes 10000 count/process. EmptyLoop: 0.01191800us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.029| 285.431| 1.00x dict | 1.00x dict |
eenv| 26.524| 265244.345| 929.28x dict | 929.28x dict |
code| 27.744| 277437.449| 971.99x dict | 1.05x eenv |
app| 257.351| 2573505.252| 9016.21x dict | 9.28x code |
--------------------------------------------------------------------------------------------------------
Run 5000 processes 10000 count/process. EmptyLoop: 0.01039020us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.029| 290.650| 1.00x dict | 1.00x dict |
code| 49.074| 490737.413| 1688.41x dict | 1688.41x dict |
eenv| 56.239| 562389.132| 1934.93x dict | 1.15x code |
app| 465.479| 4654793.599| 16015.11x dict | 8.28x eenv |
--------------------------------------------------------------------------------------------------------
Run 10000 processes 10000 count/process. EmptyLoop: 0.01086880us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.029| 285.223| 1.00x dict | 1.00x dict |
code| 76.661| 766614.083| 2687.77x dict | 2687.77x dict |
eenv| 97.440| 974403.123| 3416.29x dict | 1.27x code |
app| 842.297| 8422969.156| 29531.21x dict | 8.64x eenv |
--------------------------------------------------------------------------------------------------------
Run 15000 processes 10000 count/process. EmptyLoop: 0.00994253us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.030| 299.813| 1.00x dict | 1.00x dict |
code| 112.322| 1123222.431| 3746.41x dict | 3746.41x dict |
eenv| 126.063| 1260629.846| 4204.73x dict | 1.12x code |
app| 1108.777| 11087767.736| 36982.32x dict | 8.80x eenv |
--------------------------------------------------------------------------------------------------------
Run 20000 processes 10000 count/process. EmptyLoop: 0.01120275us/loop
--------------------------------------------------------------------------------------------------------
Type| Time/(ProcNum*Count) us| Time/ProcNum us| Time/MinTime| Time/PrevMinTime|
dict| 0.028| 281.017| 1.00x dict | 1.00x dict |
code| 117.057| 1170565.159| 4165.46x dict | 4165.46x dict |
eenv| 148.957| 1489573.352| 5300.65x dict | 1.27x code |
app| 1268.070| 12680703.050| 45124.31x dict | 8.51x eenv |
--------------------------------------------------------------------------------------------------------Conclusion
- Put env in dictionary is fastest and less CPU, but it cost lots of memory(N copy data with N processes running) and hard to update.
-
eenvandstatic code beamalmost the same, depend on the clause order and test sample. -
eenvis ~7x faster thanapplication:get_env/2-3and cost less CPU resources. - If CPU resources enough,
eenv(0.03us) slower thendict(0.18us) 6 times, and cost more CPU.
Question
-
Why eenv and code time is increase? it should be constant.
It exhausts CPU resources when lots of processes running in busy circle, so time trend to increase.
-
But why dict time almost constant?
It only increase not obvious way. Because fetch data from memory cost less CPU, and super fast(0.0x us) make it's hard to create lots of processes running at same time. You should see increase when run
make benchmark50000.
ChangeLog
License
MIT.
