TSL – Text Scraping Language
Sublime Text plugin for processing of a scraping language in pseudo-code
The TSL Sublime Text package allows you to write and execute pseudo-code style language to process text files with Regular expressions and simple logic. This gives an easy entry to data mining to non-programmers.
Once dropped in your Sublime Text's package directory, files with the extension .tsl
will automatically get syntax-highlighting and a build system to run them. Force building by choosing Tools > Build System > TSL
from the ST3 menu.
Example:
... This will read all lines from stats/milestones.csv
, take all lines, splits them into columns, selects the second column and saves the corresponding row into a file labeled by said column (e.g. stats/31-03-2019.txt
).
Index
How does it work?
Setup
Available Commands
Templating
How does it work?
TSL runs through the script line by line and executes corresponding Python code in the background. File handling, complex data types, and templating are built-in for rapid prototyping. Every line starts with a command followed by a space and space-separated arguments.
Most commands support optional clauses like as ...
(storage variable) or in ...
(file handle) to supply further information.
A command's inputs and outputs can be strings or collections of strings. In ladder case, TSL iterates over a collection's strings and applies the command to each of them. The commands as
, remember
, split
, and for every
loops change the context to the provided variable. This means you can omit as
clauses in the following commands, always automatically referring to the context. To reference variables rather than strings use square brackets. log something
will log the string "something", while log [something]
will log the content of the variable called something.
Setup
drop the TSL folder into %USERPROFILE%/AppData/Roaming/Sublime Text 3/Packages
to install the plugin.
Available Commands
File & system operations
<command>
as <variable>
bash Runs a bash command and saves the returned output to a variable.
Example:
bash git branch as branches
[<filepath>]
empty Opens up a file and deletes all its content.
Example:
in wordbag.txt
empty
<path/to/textfile.txt>
in Opens up a file and reads all its lines. You can log the lines using log line
All future file operations are refering to this one until your next "in" statement.
You'll usually see this followed by a take
or find all
command
Example:
in stats/01092019.txt
<path/to/folder>
in Creates the nested directory structure if it doesn't exist. Otherwise, the path will be used as context for future operations.
Example:
in "/Sublime Text/Packages"
count files as fileCount
log [fileCount]
[as <filepath>]
save Saves the latest collection in the given filename.
Example:
save as runner/cleaned_userinputs.txt
[<variable>]
write Writes given variable (or the results of the last find all
) into the last file opened with in
Example:
write [userIds]
<string | variable>
[to <filepath>
]
add Appends content to a file different from the currently open one
Example:
add [libraries] to libs.txt
Selections
[input]
]
select nth [of Selects a specific item of a collection, given its index.
Example:
in bigrams.txt
select 4th
[input]
][as <output>
]
select words [of Selects all words found in the last opened file.
Example:
in utterances.txt
select words
<string | RegEx | int>
] [to <string | RegEx | int>
]
select [from Selects the range from the indicated string/RegEX/number until the indicated string or regular expression or number. Note that we start counting with 1 to keep it natural
Example:
select from "accessibilityApp" to "[v:"
select from \s to \s
select from 1 to "[v:samsung.tvSearchAndPlay.Genres:drama]"
select two of [bigrams]
<string | RegEx | integer>
select from Selects the range from the indicated string / regular expression / number until the end of the line
**Example:
select from "dateTime"
select from \d\d\d
select from 122
<string | RegEx | integer>
select to Selects the range from the beginning of the line to the indicated string / regular expression / number.
Example:
select to "dateTime"
select to \W
select to 5th
select to 370
Debugging & calculations
<property>
be Sets one of the following properties of TSL to true:
verbose
| active
operation
as <variable>
calculate Calculates mathematical operations
Example:
calculate (5 * 4) / 2 as ratio
<variable | string>
log Prints to the console. Use strings with template tags (e.g. "here is: [varName]") for variables
<variable>
as <countVariable>
count Stores the count of lines in a selection.
Example:
count [entries-per-day] as frequency
log [frequency]
<files | folders>
in <path/to/dir>
as <countVariable>
count Stores the count of files or folders in a directory.
Example:
count files in "C:\Windows" as systemFiles
log "Exactly [systemFiles] system files found."
Manipulation
<varName>
to <formula>
change Iterates over a collection and changes all entries according to the template tag. Use brackets to tag variables, like so: [varName]
Example:
change [salute] to "Hi, [salute] #[i]"
will e.g. change "my name is Dan" to "Hi, my name is Dan #1"
<setName>
with <setName>
as <varName>
combine Merges two sets and stores it in a new variable.
Example:
combine [vowels] with [consonants] as letters
<string | RegEx>
[in <varName>
] [as <varName>
]
find all Finds all occurrences of a string or regular expression in the lines of the currently open file or a stored collection. The results of this search are automatically stored in a variable found
Example:
in corpus_de.txt
take lines as utterances
find all [aeiou]+ in [utterances]
log [found]
remove lines
Removes the last selected lines (e.g. the ones found using a find all
)
<string | RegEx>
by <string>
[in <variable>
]
replace Replaces given string or regular expression by another string, optionally in a particular collection.
Example:
replace \W+ by "_"
<varName>
]
sort [Sorts either the supplied or last referenced collection alphanumerically (in ascending order).
<string|RegEx>
by <delimiter>
as <variable>
split Splits a string into a collection using delimiter.
Example:
split apples;bananas;oranges by ; as fruits
log [fruits]
unique lines
Removes all duplicate lines from the last referenced collection.
Memory
<string | variable>
as <variableName>
remember Stores a string or variable in a new variable.
<lines | results | files | folders>
[as <name>
]
take Changes the selected collection to whole lines (take lines as ...
), results of a find all
directive, or to the files found in a folder specified with a preceding in <folderPath>
directive.
Example:
in source.txt
find all <[^>]+>
take lines as htmlLines
log [htmlLines]
in libraries/de
take files as germanLibs
log [germanLibs]
Flow
<variable>
for every ---
Loops through a collection, populating the variable i
with the current index. From within the loop, the item of the collection can be accessed using the variable name in singular (books -> book, babies -> baby).
If a collection is empty, the for-loop is skipped. This becomes useful to create conditional flows.
Always terminate a loop with three consecutive hyphens in a separte line.
Example:
in corpus.txt
find all [^\b]+\b[^\b]+ as bigrams
for every [bigram]
log "#[i]: [bigram]"
---
path/to/script.tsl
run Runs another TSL file
The external TSL file will receive the same scope as inlined code.
Templating
Templates are enclosed in square brackets and can appear in quoted strings, file paths, and even within regular expressions:
{
remember "\CommNetwork" as domain
in user.txt
find all \b[domain][^:]: as user
for every [user]
select from 0 to -1
in "/users/[user]/credentials.txt"
change [user] to "[user]:pleaseresetme"
add [user]
---
}
If the variables can not be found, the template tags remain untouched, including square brackets. This allows us to easily mix them with regular expressions.