Model
Quantization
Median Latency (ms)
95th Quantile Latency (ms)
Throughput (rps)
Bert
FALSE
260
300
7.8
Bert
TRUE
150
210
11.8
DistilBert
FALSE
150
180
13.4
DistilBert
TRUE
82
100
23.7
by
Sheetsu.com