Coverage and Profiling instrumentation library
profiling and coverage will instrument your code to include calls to profile and coverage data collector logic.
for that purpose instrument will disable all inlining of included types except for abstracts. additionally it will switch off all null safety metadata on types and fields included for instrumentation.
for an example use of instrument run haxe build.hxml and play with different cli options (see commented out lines in build.hxml)
requires Haxe 4.1.x or higher
Coverage
-lib instrument
--macro instrument.Instrumentation.coverage([include packages], [include folders], [exclude packages])- include packages - takes an array of package names you want to collect coverage data on. names are matched against fully qualified type names using
StringTools.startsWith - include folders - array of folders containing code for coverage data collection.
Compiler.includeis used to make sure your types get included - exclude packages - array of packages to ignore, e.g. because they are for a different target. again using
StringTools.startsWithto match fully qualified type names against list
make sure you include your source folder(s), otherwise you will only see coverage stats of types touched during run.
coverage instrumentation runs independently of a test framework. so you can create a coverage report from a "normal" run of your code if you like.
confirmed to work with utest and munit.
coverage uses a resource named coverageTypeInfo to store type information it collects during compiletime so it's available for runtime coverage report generation.
console coverage reporters
-D coverage-console-summary-reporter - prints a summary of coverage stats to your console
=====================================
Coverage summary
=====================================
packages 6/7 (85.71%)
types 31/37 (83.78%)
fields 177/201 (88.05%)
branches 399/504 (79.16%)
expressions 1398/1451 (96.34%)
files 31/33 (93.93%)
lines 1472/1540 (95.58%)
=====================================
Overall: 91.55%
=====================================
overall coverage is (covered fields + covered branches + covered expressions) / (total fields + total branches + total expressions)
-D coverage-console-package-summary-reporter - prints a package summary of coverage stats to your console
=========================================================================================================================================================
| Files | Types | Fields | Branches | Expression | Lines | Overall
Package | Rate Num | Rate Num | Rate Num | Rate Num | Rate Num | Rate Num |
=========================================================================================================================================================
demo | 0% 0/2 | 0% 0/4 | 0% 0/18 | 0% 0/0 | 0% 0/0 | 0% 0/0 | 0%
instrument | 100% 3/3 | 100% 3/3 | 85.71% 6/7 | 80% 40/50 | 96.96% 64/66 | 94.8% 73/77 | 89.43%
instrument.coverage | 100% 9/9 | 100% 9/9 | 100% 34/34 | 100% 50/50 | 100% 235/235 | 100% 217/217 | 100%
instrument.coverage.reporter | 100% 8/8 | 80% 8/10 | 98.52% 67/68 | 85.78% 175/204 | 98.59% 700/710 | 97.78% 793/811 | 95.92%
instrument.profiler | 100% 1/1 | 100% 1/1 | 100% 7/7 | 58.92% 33/56 | 90.78% 128/141 | 88.96% 129/145 | 82.35%
instrument.profiler.reporter | 100% 7/7 | 100% 7/7 | 97.91% 47/48 | 66.93% 83/124 | 88.23% 195/221 | 86.15% 168/195 | 82.69%
instrument.profiler.summary | 100% 3/3 | 100% 3/3 | 84.21% 16/19 | 90% 18/20 | 97.43% 76/78 | 96.84% 92/95 | 94.01%
=========================================================================================================================================================
Total: | 93.93% 31/33 | 83.78% 31/37 | 88.05% 177/201 | 79.16% 399/504 | 96.34% 1398/1451 | 95.58% 1472/1540 | 91.55%
=========================================================================================================================================================
-D coverage-console-file-summary-reporter - prints a file by file summary of coverage stats to your console
======================================================================================================================================================================
| Types | Fields | Branches | Expression | Lines | Overall
FileName | Rate Num | Rate Num | Rate Num | Rate Num | Rate Num |
======================================================================================================================================================================
[src/instrument/]
FileBaseReporter.hx | 100% 1/1 | 100% 2/2 | 65% 13/20 | 100% 20/20 | 88.88% 8/9 | 83.33%
Instrumentation.hx | 100% 1/1 | 66.66% 2/3 | 70% 7/10 | 88.23% 15/17 | 84.21% 16/19 | 80%
InstrumentationType.hx | 100% 1/1 | 100% 2/2 | 100% 20/20 | 100% 29/29 | 100% 49/49 | 100%
coverage/BranchInfo.hx | 100% 1/1 | 100% 4/4 | 100% 2/2 | 100% 10/10 | 100% 8/8 | 100%
coverage/BranchesInfo.hx | 100% 1/1 | 100% 5/5 | 100% 4/4 | 100% 29/29 | 100% 30/30 | 100%
coverage/Coverage.hx | 100% 1/1 | 100% 3/3 | 0% 0/0 | 100% 15/15 | 100% 16/16 | 100%
coverage/ExpressionInfo.hx | 100% 1/1 | 100% 4/4 | 100% 2/2 | 100% 10/10 | 100% 8/8 | 100%
coverage/FieldInfo.hx | 100% 1/1 | 100% 5/5 | 100% 4/4 | 100% 40/40 | 100% 45/45 | 100%
coverage/FileInfo.hx | 100% 1/1 | 100% 3/3 | 100% 4/4 | 100% 27/27 | 100% 25/25 | 100%
coverage/LineCoverage.hx | 100% 1/1 | 100% 3/3 | 100% 16/16 | 100% 32/32 | 100% 21/21 | 100%
coverage/PackageInfo.hx | 100% 1/1 | 100% 3/3 | 100% 8/8 | 100% 35/35 | 100% 31/31 | 100%
coverage/TypeInfo.hx | 100% 1/1 | 100% 4/4 | 100% 10/10 | 100% 37/37 | 100% 33/33 | 100%
coverage/reporter/CodecovCoverageReporter.hx | 100% 1/1 | 100% 3/3 | 100% 12/12 | 100% 45/45 | 100% 47/47 | 100%
coverage/reporter/ConsoleCoverageFileSummaryReporter.hx | 100% 1/1 | 100% 13/13 | 76.82% 63/82 | 97.07% 166/171 | 94.32% 183/194 | 90.97%
coverage/reporter/ConsoleCoveragePackageSummaryReporter.hx | 100% 1/1 | 100% 9/9 | 80% 16/20 | 96.96% 64/66 | 97.56% 120/123 | 93.68%
coverage/reporter/ConsoleCoverageSummaryReporter.hx | 100% 1/1 | 100% 4/4 | 50% 2/4 | 95.45% 21/22 | 95.45% 21/22 | 90%
coverage/reporter/ConsoleMissingCoverageReporter.hx | 100% 1/1 | 100% 10/10 | 93.75% 30/32 | 100% 72/72 | 98.59% 70/71 | 98.24%
coverage/reporter/EMMACoverageReporter.hx | 50% 1/2 | 100% 10/10 | 100% 6/6 | 100% 81/81 | 100% 80/80 | 100%
coverage/reporter/JaCoCoXmlCoverageReporter.hx | 50% 1/2 | 100% 12/12 | 100% 16/16 | 100% 118/118 | 100% 127/127 | 100%
coverage/reporter/LcovCoverageReporter.hx | 100% 1/1 | 85.71% 6/7 | 93.75% 30/32 | 98.51% 133/135 | 98.63% 145/147 | 97.12%
profiler/Profiler.hx | 100% 1/1 | 100% 7/7 | 58.92% 33/56 | 90.78% 128/141 | 88.96% 129/145 | 82.35%
profiler/reporter/CPUProfileReporter.hx | 100% 1/1 | 100% 8/8 | 59.09% 26/44 | 87.14% 61/70 | 92.53% 62/67 | 77.86%
profiler/reporter/CSVSummaryReporter.hx | 100% 1/1 | 100% 7/7 | 62.5% 10/16 | 89.65% 26/29 | 81.81% 18/22 | 82.69%
profiler/reporter/ConsoleDetailReporter.hx | 100% 1/1 | 100% 6/6 | 0% 0/0 | 100% 6/6 | 100% 7/7 | 100%
profiler/reporter/ConsoleHierarchyReporter.hx | 100% 1/1 | 100% 7/7 | 77.77% 14/18 | 96.42% 27/28 | 96.77% 30/31 | 90.56%
profiler/reporter/ConsoleMissedExitReporter.hx | 100% 1/1 | 83.33% 5/6 | 50% 3/6 | 66.66% 18/27 | 53.84% 14/26 | 66.66%
profiler/reporter/ConsoleSummaryReporter.hx | 100% 1/1 | 100% 7/7 | 62.5% 10/16 | 89.65% 26/29 | 81.81% 18/22 | 82.69%
profiler/reporter/D3FlameHierarchyReporter.hx | 100% 1/1 | 100% 7/7 | 83.33% 20/24 | 96.87% 31/32 | 95% 19/20 | 92.06%
profiler/summary/FlatSummary.hx | 100% 1/1 | 80% 4/5 | 100% 4/4 | 100% 13/13 | 100% 18/18 | 95.45%
profiler/summary/HierarchicalSummary.hx | 100% 1/1 | 80% 4/5 | 0% 0/0 | 100% 19/19 | 100% 29/29 | 95.83%
profiler/summary/HierarchyCallData.hx | 100% 1/1 | 88.88% 8/9 | 87.5% 14/16 | 95.65% 44/46 | 93.75% 45/48 | 92.95%
[srcDemo/demo/]
Hello.hx | 0% 0/1 | 0% 0/1 | 0% 0/0 | 0% 0/0 | 0% 0/0 | 0%
MyTestApp.hx | 0% 0/3 | 0% 0/17 | 0% 0/0 | 0% 0/0 | 0% 0/0 | 0%
======================================================================================================================================================================
Total: | 83.78% 31/37 | 88.05% 177/201 | 79.16% 399/504 | 96.34% 1398/1451 | 95.58% 1472/1540 | 91.55%
======================================================================================================================================================================
-D coverage-console-missing-reporter - prints a line for every type, field, branch and expression with no coverage
======================
== missing coverage ==
======================
srcDemo/demo/Hello.hx:3: type Hello not covered
srcDemo/demo/Hello.hx:4: field main not covered
srcDemo/demo/Hello.hx:5: expression not covered
srcDemo/demo/MyTestApp.hx:35: branch not covered
srcDemo/demo/MyTestApp.hx:36: expression not covered
srcDemo/demo/MyTestApp.hx:37: branch not covered
srcDemo/demo/MyTestApp.hx:38: expression not covered
srcDemo/demo/MyTestApp.hx:39: branch not covered
srcDemo/demo/MyTestApp.hx:40: expression not covered
srcDemo/demo/MyTestApp.hx:53: branch not covered
srcDemo/demo/MyTestApp.hx:54: expression not covered
srcDemo/demo/MyTestApp.hx:59: branch not covered
srcDemo/demo/MyTestApp.hx:60: expression not covered
srcDemo/demo/MyTestApp.hx:65: branch not covered
srcDemo/demo/MyTestApp.hx:66: expression not covered
srcDemo/demo/MyTestApp.hx:68: branch not covered
srcDemo/demo/MyTestApp.hx:74: branch not covered
srcDemo/demo/MyTestApp.hx:131: branch not covered
srcDemo/demo/MyTestApp.hx:134: expression not covered
srcDemo/demo/MyTestApp.hx:138: branch not covered
srcDemo/demo/MyTestApp.hx:151: field download not covered
srcDemo/demo/MyTestApp.hx:152: expression not covered
srcDemo/demo/MyTestApp.hx:154: type Role not covered
lcov coverage reporter
writes coverage data using lcov format to a file. filename defaults to lcov.info in your workspace root. you can set a different name and folder. lcov reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
includes full coverage data down to lines and branches (visualisation of partially covered branches might depend on your tooling, some may show it, some won't).
-D coverage-lcov-reporter
-D coverage-lcov-reporter=lcov.infoCodecov coverage reporter (untested)
writes coverage data using Codecov's Json coverage foramt. filename defaults to codecov.json in your workspace root. you can set a different name and folder. Codecov reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
includes line coverage for each file, partial branches show up as "1/2" or "3/4".
-D coverage-codecov-reporter
-D coverage-codecov-reporter=codecov.jsonemma coverage reporter (untested)
writes coverage data using emma xml format to a file. filename defaults to emma-coverage.xml in your workspace root. you can set a different name and folder. emma reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
only supports coverage down to method level (no branch and line coverage)
-D coverage-emma-reporter
-D coverage-emma-reporter=emma-coverage.xmlJaCoCo Xml coverage reporter (untested)
writes coverage data using JaCoCo xml format to a file. filename defaults to jacoco-coverage.xml in your workspace root. you can set a different name and folder. JaCoCo reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
includes line coverage for each file.
-D coverage-jacocoxml-reporter or -D coverage-jacocoxml-reporter=jacoco-coverage.xmlProfiling
-lib instrument
--macro instrument.Instrumentation.profiling([include packages], [include folders], [exclude packages])- include packages - takes an array of package names you want to collect coverage data on. names are matched against fully qualified type names using
StringTools.startsWith - include folders - array of folders containing code for coverage data collection.
Compiler.includeis used to make sure your types get included - exclude packages - array of packages to ignore, e.g. because they are for a different target. again using
StringTools.startsWithto match fully qualified type names against list
profiling was written with multithreading in mind, but's currently untested
console profiling reporters
-D profiler-console-detail-reporter - prints entry and exit of every function as it happens including call id, location type and function name and duration of call
>>> [2] srcDemo/demo/Hello.hx:4: Hello.main
srcDemo/demo/Hello.hx:5: Hello Haxe
<<< [2] srcDemo/demo/Hello.hx:4: Hello.main 0.080108642578125ms
-D profiler-console-summary-reporter - prints a summary listing all locations, fields, number of invocations and duration spent inside (includes duration of calls to other functions)
==================
== Call Summary ==
==================
srcDemo/demo/MyTestApp.hx:143: MyTestApp.main 1 3.10015678405761719ms
srcDemo/demo/MyTestApp.hx:6: MyTestApp.new 1 1.54900550842285156ms
srcDemo/demo/MyTestApp.hx:117: MyTestApp.noBody4 1 0.174999237060546875ms
srcDemo/demo/MyTestApp.hx:148: MyTestApp_Fields_.moduleBody 7 0.14591217041015625ms
srcDemo/demo/MyTestApp.hx:71: MyTestApp.opBoolOr 3 0.145196914672851562ms
srcDemo/demo/MyTestApp.hx:89: MyTestApp.initTranslations 1 0.134944915771484375ms
srcDemo/demo/MyTestApp.hx:102: MyTestApp.noBody3 2 0.129222869873046875ms
srcDemo/demo/MyTestApp.hx:19: MyTestApp.<anon-1> 2 0.116825103759765625ms
srcDemo/demo/MyTestApp.hx:82: MyTestApp.sortColsFunc 1 0.0898838043212890625ms
srcDemo/demo/MyTestApp.hx:57: MyTestApp.switchVal 2 0.0722408294677734375ms
srcDemo/demo/MyTestApp.hx:111: MyTestApp.noBody2 3 0.051021575927734375ms
srcDemo/demo/MyTestApp.hx:122: MyTestApp.getInt 2 0.0469684600830078125ms
srcDemo/demo/MyTestApp.hx:99: MyTestApp.noBody 1 0.04482269287109375ms
srcDemo/demo/MyTestApp.hx:114: MyTestApp.noBody5 3 0.03910064697265625ms
srcDemo/demo/MyTestApp.hx:130: MyTestApp.noCover 1 0.0278949737548828125ms
srcDemo/demo/MyTestApp.hx:137: MyTestApp.whileLoop 1 0.0209808349609375ms
srcDemo/demo/MyTestApp.hx:43: MyTestApp.<anon-2> 1 0.0159740447998046875ms
srcDemo/demo/MyTestApp.hx:83: MyTestApp.<anon-3> 1 0.0140666961669921875ms
srcDemo/demo/MyTestApp.hx:4: MyTestApp.<anon-arrow-0> 1 0.011920928955078125ms
-D profiler-console-missing-reporter - prints a list of all calls that didn't exit properly. (might indicate an issue in instrument library)
-D profiler-console-hierarchy-reporter - prints a hierachical summary of calls showing which function was called from what function, again listing all locations, fields, number opf invocations and duration
====================
== Call Hierarchy ==
====================
+ <root> 9.17577743530273438ms
---+ thread-0 9.17577743530273438ms
------+ srcDemo/demo/MyTestApp.hx:141: MyTestApp.main 1 4.44388389587402344ms
---------+ srcDemo/demo/MyTestApp.hx:6: MyTestApp.new 1 4.13799285888671875ms
------------+ srcDemo/demo/MyTestApp.hx:97: MyTestApp.noBody 1 0.030994415283203125ms
------------+ srcDemo/demo/MyTestApp.hx:109: MyTestApp.noBody2 1 0.0269412994384765625ms
------------+ srcDemo/demo/MyTestApp.hx:100: MyTestApp.noBody3 2 0.159025192260742188ms
---------------+ srcDemo/demo/MyTestApp.hx:146: MyTestApp_Fields_.moduleBody 2 0.0522136688232421875ms
------------+ srcDemo/demo/MyTestApp.hx:112: MyTestApp.noBody5 1 0.02193450927734375ms
------------+ srcDemo/demo/MyTestApp.hx:146: MyTestApp_Fields_.moduleBody 5 1.22761726379394531ms
------------+ srcDemo/demo/MyTestApp.hx:4: MyTestApp.<anon-arrow-0> 1 0.02288818359375ms
------------+ srcDemo/demo/MyTestApp.hx:115: MyTestApp.noBody4 1 0.294923782348632812ms
---------------+ srcDemo/demo/MyTestApp.hx:19: MyTestApp.<anon-1> 2 0.210046768188476562ms
------------------+ srcDemo/demo/MyTestApp.hx:112: MyTestApp.noBody5 2 0.03719329833984375ms
------------------+ srcDemo/demo/MyTestApp.hx:109: MyTestApp.noBody2 2 0.0393390655517578125ms
------------+ srcDemo/demo/MyTestApp.hx:87: MyTestApp.initTranslations 1 0.142812728881835938ms
------------+ srcDemo/demo/MyTestApp.hx:128: MyTestApp.noCover 1 0.031948089599609375ms
------------+ srcDemo/demo/MyTestApp.hx:120: MyTestApp.getInt 2 0.0612735748291015625ms
------------+ srcDemo/demo/MyTestApp.hx:135: MyTestApp.whileLoop 1 0.0259876251220703125ms
------------+ srcDemo/demo/MyTestApp.hx:41: MyTestApp.<anon-2> 1 0.0240802764892578125ms
------------+ srcDemo/demo/MyTestApp.hx:80: MyTestApp.sortColsFunc 1 0.072956085205078125ms
---------------+ srcDemo/demo/MyTestApp.hx:81: MyTestApp.<anon-3> 1 0.019073486328125ms
------------+ srcDemo/demo/MyTestApp.hx:55: MyTestApp.switchVal 2 0.118017196655273438ms
------------+ srcDemo/demo/MyTestApp.hx:69: MyTestApp.opBoolOr 3 0.70667266845703125ms
CSV profiling reporter
writes profiling data to a CSV file using thread;invocations;total time in ms;class;function;location columns. filename defaults to summary.csv in your workspace root. you can set a different name and folder. csv reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
-D profiler-csv-reporter
-D profiler-csv-reporter=summary.xmlD3 Flamegraph profiling reporter
writes a json file compatible with d3-flame-graph javascript library.f ilename defaults to flame.json in your workspace root. you can set a different name and folder. d3 reporter will try to create output folder. folder name is relative to your workspace root (or whereever you run Haxe from).
-D profiler-d3-reporter
-D profiler-d3-reporter=flame.json.cpuprofile profiling reporter (WIP)
work in progress - supposed to write a file in .cpuprofile format to be used in e.g. vscode-js-profile-flame VSCode extension.
-D profiler-cpuprofile-reporter
-D profiler-cpuprofile-reporter=profiler.cpuprofileend instrumentation detection
instrument tries to find your main function and all calls to Sys.exit to set up automatic detection of program exit. however if you exit your program by any other means or if your main function is excluded from instrumentation, then you might have to add a call to instrument's endProfiler or endCoverage functions:
#if instrument
instrument.coverage.Coverage.endCoverage(); // when measuring coverage
instrument.profiler.Profiler.endProfiler(); // when profiling
#endyou can invoke end instrumentation functions multiple times, results should include all data since start of application up to your end instrumentation call. file based reporters will overwrite their files with latest results.
exclude types or fields from instrumentation
you can exclude types / fields from instrumentation by adding metadata:
-
@ignoreInstrumentor@:ignoreInstrumentwill ignore a type / field from all instrumentation -
@ignoreCoverageor@:ignoreCoveragewill ignore a type / field from coverage instrumentation -
@ignoreProfileror@:ignoreProfilerwill ignore a type / field from profiler instrumentation
