Add performance notes

1 files changed, 22 insertions, 0 deletions

diff --git a/README.md b/README.md
new file mode 100644
index 0000000..04c7dbc
--- /dev/null
+++ b/README.md
@@ -0,0 +1,22 @@
+
+## Initial import speed
+
+Problem: current import speed is too slow.
+
+Current import speed with encoding/json:        (1780000-990000)/(22:37:09-20:46:10)
+                                                790000/((22*3600+37*60+9)-(20*3600+46*60+10))
+                                                119 inserts per second
+
+What if we:
+
+1) use goccy/go-json for decoding?
+    (40000)/(46*60+9)-(40*60+25) = 116 inserts per second
+    Looks like the database is our bottleneck.
+2) parallelize?
+3) other performance optimizations?
+    - https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite
+    - wrap all inserts in one transaction:
+        410000/(29-13) = 25,625 inserts per second!! Much, much better!
+        (using plain old encoding/json instead of goccy: about 20,000 per second)
+
+Decided on using goccy to unmarhsal, and doing everything in one SQLite transaction.
\ No newline at end of file