WarpParse is designed for Observability, Security, Real-time Risk Control, Data Platform teams' high-performance ETL engine. Focus on logs/event ingestion, parsing and transformation, provides high-throughput parsing (WPL), transform (OML), routing, unified connectors API and minimalist DevOps experience.
curl
-sSf
https://get.warpparse.ai/setup.sh
|
bash
specialized for high-performance log processing and dataTransformdesign
in many scenariosComprehensively surpass in performance Vector, provides industry-leading log processing capability, meet high throughput scenario requirements.
WPL (parsing DSL) + OML (transform DSL), far more readable than regex and Lua, makes rule writing simpler.
Based on wp-connector-api, easy for community ecosystem extension, supports multiple data sources and output targets.
Single binary deployment with configuration-based management; provides wproj, wpgen, wprescue toolkits to simplify DevOps work.
Support SQL queries through in-memory database to enrich data and make it more valuable.
Route data based on rules and transform models, support multi-path copy and filters, flexibly control data flow direction.
WarpParse vs Vector Benchmark Results (AWS EC2 / Ubuntu 24.04 / 8 vCPU / 16 GiB RAM)
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
In Linux mixed log scenarios, provides 3.4x parse performance advantage, File and TCP reception link performance stands out.
Peak CPU is higher, but total CPU time required for equivalent data volume is less.
Small rule files for quick hot updates; strong single-machine processing capability reducing central pressure.
CPU and Memory Comparison at 20,000 EPS Fixed Rate (AWS EC2 / Ubuntu 24.04 / Mixed Log / TCP → BlackHole)
Average / Peak (%)
Average / Peak (MB)
54% Average CPU
Reduced compared to Vector-VRL by 68.8%，Reduced compared to Logstash by 80.4%
60 MB Average Memory
Reduced compared to Vector-VRL by 63.0%，Reduced compared to Logstash by 95.0%
At the same throughput, WarpParse resource consumption is far lower than other engines, very suitable for resource-constrained edge environments and large-scale deployment scenarios.
WarpParse vs Vector-VRL vs Vector-Fixed vs Logstash (Mac M4 Mini / 10C / 16G RAM)
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
EPS (Events Per Second) Comparison
Provides 1.4-8.8x performance advantage on Mac M4 platform.
Peak CPU is higher, but total CPU time required for equivalent data volume is less.
Small rule files for quick hot updates; strong single-machine processing capability reducing central pressure.
Industrial-grade data governance with strongly-typed parsing DSL, more precise and efficient than regular expressions
Complete more complex parsing tasks with less code, intuitive and readable syntax.
Built-in alt (alternative fallback), opt (optional matching), some_of (loop detection) and other meta-information, providing far superior fault tolerance than regex.
Integrate cleaning and parsing in a single expression, supports preprocessing like decode/base64, unquote/unescape, decode/hex.
{"date":1767006286.778936,"log":"180.57.30.149 - - [21/Jan/2025:01:40:02 +0800] \"GET /nginx-logo.png HTTP/1.1\" 500 368 \"http://207.131.38.110/\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.142 Safari/537.36\" \"-\""}rule nginx { (
json | take(log) | json_unescape() | (
ip:sip,
2*_,
time:recv_time<[,]>,
http/request",
http/status,
digit,
chars",
http/agent",
_"
)
) }obj = parse_json!(string!(.message))
. = object!(obj)
.|= parse_regex!(
string!(.log),
r'^(?P<sip>\S+)\s+\S+\s+\S+\s+\[(?P<recv_time>\d{2}\/[A-Za-z]{3}\/\d{4}:\d{2}:\d{2}:\d{2})\s+[+\-]\d{4}\]\s+"(?P<http_request>[^"]*)"\s+(?P<status>\d{3})\s+(?P<digit>\d+)\s+"(?P<chars>[^"]*)"\s+"(?P<http_agent>[^"]*)"\s+"[^"]*"\s*$'
)
.status = to_int!(.status)
.digit = to_int!(.digit)
del(.log)
del(.message)filter {
json {
source => "message"
}
dissect {
mapping => {
"log" => '%{sip} - - [%{recv_time} %{+recv_time}] "%{http_request}" %{status} %{digit} "%{chars}" "%{http_agent}" "%{ignore_tail}"'
}
tag_on_failure => ["nginx_dissect_failure"]
}
mutate {
convert => {
"status" => "integer"
"digit" => "integer"
}
}
mutate {
remove_field => ["log", "message", "ignore_tail","@timestamp","@version","event"]
}
}Declarative syntax, automatically reorganize data fields into complex structured business models
Describe what you want, not how to do it. Simply define the final output data structure without writing low-level processing logic.
read (non-destructive read) and take (destructive read) modes, support strong-type safety and automatic inference.
Native SQL integration, directly query database for data enrichment when building objects.
size : digit = take(size);
status : digit = take(status);
match_chars = match read(option:[wp_src_ip]) {
ip(127.0.0.1) => chars(localhost);
!ip(127.0.0.1) => chars(attack_ip);
};
str_status = match read(option:[status]) {
digit(500) => chars(Internal Server Error);
digit(404) => chars(Not Found);
};
* : auto = read();.status = to_int!(parsed.status)
.size = to_int!(parsed.size)
if .host == "127.0.0.1" {
.match_chars = "localhost"
} else if .host != "127.0.0.1" {
.match_chars = "attack_ip"
}
if .status == 500 {
.str_status = "Internal Server Error"
} else if .status == 404 {
.str_status = "Not Found"
}Used for writing WPL parsing and OML transform rules
Based on wp-connector-api, support multiple data sources and output targets
Experience the power of high-performance ETL engine right now