kicking the ruby garbage collector’s trash [how to improve memory usage for MRI]

Here’s some pointers I’ve found helpful:

fork off child processes to do a lot of memory usage then die–that way you for sure reclaim that memory
use GC in another child process [email me for details]
Use temp files — Marshal.dump large quantities of data to them while you’re not using it, then Marshal.load it when it is desired.
slim_attributes helps with mysql related usage.

Here’s some more intense options:
rewrite erb to re-use strings
write a “slim array” that encompasses simple ints and strings and floats more memory efficiently than Ruby’s currently does.

65 thoughts on “kicking the ruby garbage collector’s trash [how to improve memory usage for MRI]

  1. Evan’s find_by_id…puts that whole thing into memcached based on ID or what?

    There’s basically view fragments [expire them yourself]
    whole page fragments [sweep to expire them [auto-expire on change]]

    or “run this block on change of table X–also that view.”

    Or fragment caching view RSI or whatever it’s called–same thing.

    So maybe I could make an in-memory cache somehow, too, to avoid any file I/O shtuffs [well I guess memcache maybe?]

    I need a problem to solve!

    Mephisto?

    My slant is without cacheing, though…no cacheing really allowed [though I suppose you could actually monitor one and see if it relies on tables x, y, and zand if it has no Time.now “auto-cache” it.
    Cacheing seems pretty straight-forward, with some coolness thrown in.

  2. Could mark some tables as “change infrequent” then

    if the controller only “uses” those, can auto-skip them

    the view whole sections that “only use those” can be auto-cached.

    Obviously if nothing in the db is changing much and there’s no Time.now, you could cache everything…hmm. In reality you could only cache it based on queries.

    Sometimes you wouldn’t want to cache. If everyone’s query is the same.

    The kicker is if it uses some of both…

    anyway could basically put an Interlock around the whole action, whole view, always, well, based on what incoming parameters are used tho. Ex: POST.

    dang that’s tough. cacheing assumes some shared. Like storing view fragments in the DB table along with those fellas.

    low hanging: mark some as RO then in memory (or memcached) cache (trivial) queries’ replies :) like a view fragment but not. Make sure it works with auto-lookup :)

    Evan’s just takes out the initial queries even to see if anything has changed [I guess?]

    in reality many templates could be somehow cached [based on @'s used, locals]. Just do it in RAM to not worry about nuffin :) Still not too low hanging there.

  3. optional “should session stuffs count against cacheing?”

    1) which tables used -> skip controller, view [from RAM].
    [cache on get params, etc.]
    2) individual @this = Table by…x,y,z theoretically block off the view where that was used, cache on…??? If it’s find :all that’s fine, if it’s hard coded, then that’s fine, else cache on all params used [but you can still cache, right?]

  4. I guess I really have to cache against all its preceding parameters [sniff].
    Also might want to combine filters/views all together like good children [plus it's faster anyway, perhaps a tidge :P]. So then we have lruify, lruify whole actions, combine all together, then [if time], lruify partial actions/partial views.

  5. re: previous post on it [something 2008] — most people think that speed is gained solely by profiling your code and removing your own bottlenecks. Or that you should scale out like crazy!

    Note: just say “we could do this or that [python, auto partial views]–what do you think?”

  6. also gc related: show how it goes down with forking, etc…

    I suppose I could write my own write barrier style fella: newly created objects “must” be referenced by broken segments or they are “gone” and the old can just stay there I guess. The question is how many gen’s…–note this as future work :)

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>