[nfbcs] locating hard to find memory leaks

[Aaron Cannon]

Once you create the XML tree with TinyXML, are you doing anything with
it besides just writing it out?  If not, why not just use a streaming
writer.  There's no need to store the whole tree in memory.

Basically, you sort of want the opposite of a SAX parser.  A SAX
parser, instead of reading an entire XML file into memory as a tree,
only reads a little bit of the document at a time.  It then fires
different events based on what it sees.  For example, you might create
a class that has a interface like:

class DataLoader {
  startTag (string tagName, hashMap attributes)
  endTag(string tagName)
  textFragment(string text)
}

You would then give that class (or more likely an instantiated
instance of that class) to the parser, which would call the various
methods on your class as it came across each item in the XML.

So, for XML like

<a><b x="1">Hello world<c/></b></a>

the parser would call your class as follows:
startTag("a", null)
startTag("b", {x: "1"})
textFragment("Hello world")
startTag("c", null)
endTag("c")
endTag("b")
endTag("a")

As you can see, the parser doesn't have to store huge chunks of data
in memory.  It just needs to consume enough to, at most,  hold a
complete starting tag, and a list of currently opened tags, for error
checking.

Of course, you want a writer, not a reader.  It should provide a
convenient interface to allow you to write the file a little bit at a
time, rather than creating the entire structure in memory, and then
writing it out.

I looked around a bit to see if I could find one for C++, and the only
one I found is this one:
http://www.firstobject.com/c++-xml-writer-creates-large-xml-file.htm

Many other folks appear to be just rolling their own, which seems
unnecessarily risky to me, but whatever works I suppose.

Good luck.

Aaron

On 10/20/15, Littlefield, Tyler via nfbcs <nfbcs at nfbnet.org> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Hello all,
> I am working on my mud engine:
> http://github.com/sorressean/aspen
> There appears to be a really weird issue when
> serializing/deserializing a lot of data. When the data is loaded, I
> end up increasing my memory to hold a 250x250 grid of rooms and exits
> from 100 KB to around 500 MB. I'm using TinyXML for this.
> Does anyone have any ideas in terms of how to track this down? Running
> valgrind doesn't seem to produce any results. This shouldn't be using
> this much memory at all--the objects are compact, which you can
> understand if you can hold 65000 rooms plus exits in memory. Any tips
> here would be awesome. Also if you're interested in a run, I provided
> a world generated grid here:
> https://dl.dropboxusercontent.com/u/10204868/wilderness.tgz
> to use after you clone aspen:
> mkdir -p aspen/data/areas
> put the extracted wilderness file there and run.
> Any tips/advice would be greatly appreciated.
> Thanks,
> - --
> Take care,
> Ty
> twitter: @sorressean
> web:http://tysdomain.com
> pubkey: http://tysdomain.com/files/pubkey.asc
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v2
>
> iQEcBAEBAgAGBQJWJmG+AAoJEAdP60+BYxejZt4H/0wXby2sRfzG/MbssA52K3I7
> YKFxLfKGwgdPOG0ADrME2rv19z/lIWEPewI7uwYuQiQZydVsF4FAOQeghj4vnqjZ
> V7xMADTatlvmM3cznpmsg+JAtIW3XYhZq62halnTA15XVJFna5wDtLr6BZbQdGhC
> R84QYqQi6JmyM1idQQS9Q9k47xGGVkT9v8GR2pkzbnCemy8QhqTdoTAUbvFg4l/U
> CRr1PCbt31lsgCir8dqPzH6gx/+ujn812WCOhXJTTm93gTF5uUVR8x6wYeBz0w1g
> c9OyhmmMVa1LhwdoGHl7nOoJa/OIldaNl/8psAc1AQ5kIP+aKgJUDsiYWUtdQ3U=
> =kDiq
> -----END PGP SIGNATURE-----
>
> _______________________________________________
> nfbcs mailing list
> nfbcs at nfbnet.org
> http://nfbnet.org/mailman/listinfo/nfbcs_nfbnet.org
> To unsubscribe, change your list options or get your account info for
> nfbcs:
> http://nfbnet.org/mailman/options/nfbcs_nfbnet.org/cannona%40fireantproductions.com
>