diff options
| -rw-r--r-- | README.md | 25 |
1 files changed, 5 insertions, 20 deletions
@@ -3,24 +3,9 @@ You need a copy of https://kaikki.org/frwiktionary/raw-wiktextract-data.jsonl.gz -## Initial import speed +# TODO -Problem: current import speed is too slow. - -Current import speed with encoding/json: (1780000-990000)/(22:37:09-20:46:10) - 790000/((22*3600+37*60+9)-(20*3600+46*60+10)) - 119 inserts per second - -What if we: - -1) use goccy/go-json for decoding? - (40000)/(46*60+9)-(40*60+25) = 116 inserts per second - Looks like the database is our bottleneck. -2) parallelize? -3) other performance optimizations? - - https://stackoverflow.com/questions/1711631/improve-insert-per-second-performance-of-sqlite - - wrap all inserts in one transaction: - 410000/(29-13) = 25,625 inserts per second!! Much, much better! - (using plain old encoding/json instead of goccy: about 20,000 per second) - -Decided on using goccy to unmarhsal, and doing everything in one SQLite transaction.
\ No newline at end of file +- height of viewport should be dynamic +- no hardcoded paths, or Anki info +- better setup instructions (intall plugin, set deck & card, properties, etc) +- clear the screen after we finish the initial import, probably?
\ No newline at end of file |