From cafc25dc82d5ecb5fa0da4ea80f14bf2ad550f3c Mon Sep 17 00:00:00 2001 From: David Schlachter Date: Wed, 7 Jan 2026 22:53:22 -0500 Subject: Add performance notes --- README.md | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 README.md (limited to 'README.md') diff --git a/README.md b/README.md new file mode 100644 index 0000000..04c7dbc --- /dev/null +++ b/README.md @@ -0,0 +1,22 @@ + +## Initial import speed + +Problem: current import speed is too slow. + +Current import speed with encoding/json: (1780000-990000)/(22:37:09-20:46:10) + 790000/((22*3600+37*60+9)-(20*3600+46*60+10)) + 119 inserts per second + +What if we: + +1) use goccy/go-json for decoding? + (40000)/(46*60+9)-(40*60+25) = 116 inserts per second + Looks like the database is our bottleneck. +2) parallelize? +3) other performance optimizations? + - https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite + - wrap all inserts in one transaction: + 410000/(29-13) = 25,625 inserts per second!! Much, much better! + (using plain old encoding/json instead of goccy: about 20,000 per second) + +Decided on using goccy to unmarhsal, and doing everything in one SQLite transaction. \ No newline at end of file -- cgit v1.2.3