Results 1 to 8 of 8

Thread: JSON.decode a large JSON string takes forever

  1. #1

    Exclamation JSON.decode a large JSON string takes forever

    I am trying to decode this JSON and it takes forever:

    Code:
    function onLoad()
      WebRequest.get("https://swdestinydb.com/api/public/cards/", self, "cardsResponse")
    end
    
    function cardsResponse(response)
      cards = JSON.decode(response.text)
    end
    Am I doing something wrong?

  2. #2
    Well it is over 500k chars... JSON format bloats it horribly, it could be compressed to a fraction of that simply having codes
    (instead of strings like "set_codes", "affiliation_code", "affiliation_name" et cetera repeated hundreds of times) and one dictionary to decode it later.

    If you really want to use that API directly, refer to specific cards using a sublink like [url]https://swdestinydb.com/api/public/card/01001[/url] and decode them one by one in a coroutine.

  3. #3
    I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
    It does still however take a lot of time, about 1 minute to read all cards in all sets.

    I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.

  4. #4
    Decoding json files in regular Lua console works waay faster for me too. As far as TTS goes, just slapping some JSON module in (did this one [url]https://github.com/LuaDist/dkjson[/url]) speeded up decoding by ~20%. It's not the most lightweight one out these so figure some other could do even better. But all in all preprocessing those files before decoding should be able to reduce times A LOT anyway, I'd try that if it takes too long.

  5. #5

    JQ parse way faster... JSON PARSING SLOW WHY!?!?!

    Quote Originally Posted by Xorbis View Post
    I was able to load each set of cards individually by first calling the sets/ endpoint in their API.
    It does still however take a lot of time, about 1 minute to read all cards in all sets.

    I have seen 100x faster JSON decoders in NodeJS, must have something to do with the LUA JSON module Tabletop Simulator is using.
    I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

    06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
    "Durmacar"

    real 0m0.246s
    user 0m0.234s
    sys 0m0.017s

  6. #6
    Quote Originally Posted by staticrealm View Post
    I just his this myself... what took about 15 seconds to parse in TTS.... in command line with jq its way fast. Really disappointing.

    06:27:48 ~/dnd/strahd$ time cat durmacar.json | jq '.name'
    "Durmacar"

    real 0m0.246s
    user 0m0.234s
    sys 0m0.017s

    I just went and pulled in code from [url]https://github.com/rxi/json.lua/blob/master/json.lua[/url] and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.

  7. #7
    Join Date
    Sep 2016
    Posts
    219
    Quote Originally Posted by staticrealm View Post
    I just went and pulled in code from https://github.com/rxi/json.lua/blob/master/json.lua and made it inline. Now it runs in a respectable manner.. this reallly needs to be fixed.
    This helped me fix the problem with decoding in my mod. Thanks! Worth noting that the encoder wasn't as forgiving as the one included with TTS. It can still be used, but might require tables to be more consistent (no sparse arrays, no mixing of data types).

    Supposedly this is the code used for JSON encoding: http://regex.info/code/JSON.lua

    I noticed it uses a lot of table.sort calls to sort keys, which could be part of the reason why it takes so long.

    It's worth noting that using JSON is not the only way to store your data. I had a long discussion with others on Discord and it was pointed out that any string can be stored and loaded in TTS. One suggestion was to store your data as lua and then use dynamic.eval to load the data (http://www.moonsharp.org/additions.html)

    Personally, I find using the the decoder above is enough to make load times much more reasonable. Though I'm still looking for a way to more quickly encode without writing custom code for it.
    Last edited by Sancho; 04-26-2020 at 07:01 PM.

  8. #8
    Join Date
    Sep 2016
    Posts
    219
    Sorry to bump this old thread, but I wanted to share my progress with this. I finally took a stab today at getting the encoder in https://github.com/rxi/json.lua/blob/master/json.lua working. I basically rewrote some of the logic in encode_table to work with sparse arrays and mixed data types by utilizing the object_or_array method from the original parser (http://regex.info/code/JSON.lua). I also removed the table.sort calls from this method since I didn't think it was necessary for the encoder to be deterministic.

    The result was a massive improvement in speed. What previously took 1.7s for the parser to write is now taking only 0.8s. I compared the before and after LuaScriptState in both files and found them to be identical in structure (before I removed the table.sort calls). Your mileage may vary, though.

    If anyone is in need of the code I have it in my github here: https://github.com/DinnerBuffet/TTSCarcassonne/blob/master/scripts/json.ttslua

Similar Threads

  1. [ADDED] JSON Object
    By Abarden in forum Scripting Suggestions
    Replies: 4
    Last Post: 05-08-2018, 02:33 AM
  2. [CONNECTION ISSUES] Server browser takes forever to load
    By nightfuryninja in forum Technical Support
    Replies: 1
    Last Post: 03-09-2018, 01:54 AM
  3. Read JSON from URL
    By FoaS in forum Scripting
    Replies: 0
    Last Post: 09-04-2017, 09:47 AM
  4. .json Error: Out of Memory
    By Eskander in forum Technical Support
    Replies: 2
    Last Post: 12-19-2016, 07:38 PM
  5. [SOLVED] json memory error message
    By Maven in forum Technical Support
    Replies: 3
    Last Post: 11-25-2016, 05:21 PM

Tags for this Thread

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •