how to compile ruby with increased garbage size

You may have noticed when you’re running long-ish rails processes in windows that it sometimes pauses briefly, then continues on its merry way.

I have, at least.
The problem can be attributed to the garbage collector.
Ex: with one operation large_table.find(:all).each{||…}
I would receive something like 10 0.10 second pauses before it finished–increasing my overall time by at least 25%
Tweaking the malloc limit [the amount it will grow before doing a GC limit] changed that to something like 2 0.25 second pauses–which caused things to speedup considerably.
How:
install msysgit
c:> git://github.com/oneclick/rubyinstaller.git
cd rubyinstaller
set CCFLAGS=-DGC_MALLOC_LIMIT=60000000
rake # build ruby 1.8 in sandbox/ruby18_mingw
rake ruby19 # build ruby 1.9 in sandbox/ruby19_mingw
Installers hopefully coming soon.
Another option for speeding things up is building with GCC 4.4 instead of the 3.4.5 currently used.
Here’s a download that does that for the adventurous:
http://all.faithpromotingstories.org/ruby_distros/ruby19_mingw_44.tgz
Anyway, watch this area as I’ll post various ruby executables with different (faster) compiler options, and experimental like using tcmalloc, etc.
Thanks and good luck to all your windows+ruby users (like myself) out there!
Also much thanks to Luis and the one click group for making it possible to use these cool tools.

7 thoughts on “how to compile ruby with increased garbage size

  1. how to proxy to a port when your firewall blocks that port [using mod_proxy instead]
    For me, it was getting some external server to forward all requests on to that port.

    I realize you can use ssh -Rport:host:port as well (through some other server), however ssh tunnelling was at times interrupted and didn’t have a retry or restart, it seemed, at least that I knew of.

    Anyway the final forward was:

    1) create new subdomain that will forward it on.

    2) edit apache’s conf [apache.conf in my example] to forward things on

    Order deny,allowAllow from allServerName audio-mp3.ibiblio.org.8000.doachristianturndaily.infoProxyPass / http://audio-mp3.ibiblio.org:8000/DocumentRoot /home/rdp/www/faithpromotingstories.org

    Order deny,allow

    Allow from all

    ServerName audio-mp3.ibiblio.org.8000.doachristianturndaily.info

    ProxyPass / http://audio-mp3.ibiblio.org:8000/

    3) install forwarders:

    sudo a2enmod proxy

    sudo a2enmod proxy_http

    4) restart apache

    5) enjoy the tunes [in my case].

  2. how to setup a remote git repository using ssh
    Finding the normal instructions slightly lacking on how to create a brand-new, empty git repository via SSH for a central location, here they are:

    $ ssh myserver.com
    Welcome to myserver.com!
    $ mkdir /var/git/myapp.git && cd /var/git/myapp.git
    $ git –bare init
    Initialized empty Git repository in /var/git/myapp.git
    $ exit
    Bye!

    Add the remote repository to your existing local git repo and push:

    $ cd ~/Sites
    $ mkdir myapp
    $ git init
    $ git remote add origin ssh://myserver.com/var/git/myapp.git
    $ touch README
    $ git add README
    $ git commit -m “initial commit”
    $ git push origin master
    Posted by Roger Pack at 7:43 AM 2 comments Links to this post

  3. How to compile using visual studio for ruby
    Extension building in ruby on windows is a little complicated, because the “mainstream” one click installer uses mingw for its build, which is based on a rather outdated msvcrt.dll [version 6 I believe--the one used by Visual Studio 6, no longer in production].

    It is, however, according to Charlie Savage, possible to mix code built from newer compilers into it [1].

    Mixing runtime libraries
    ========================

    The most obvious criticism of this plan is that it will lead to mixing of microsoft runtime c libraries. From my experience this works as long as extension developers follow the rules described here:

    http://msdn.microsoft.com/en-us/library/ms235460(VS.80).aspx

    To be more concrete, this boils down to two simple rules:

    * If you call ALLOC or ALLOC_N, use xfree and not free
    * Don’t call sprintf or printf in an extension, instead use rb_f_sprintf/rb_vsprintf/rb_io_printf

    If an extension violate these two rules then its obvious, a segmentation fault happens. Thus these bugs are easy to find and easy to fix.

    Since VC6 is thankfully no longer available, supporting msvc absolutely, positively requires mixing c runtime libraries and therefore extension writers must follow these two simple rules.

    (Note that if you use rb_x_malloc and its siblings, they will use the right free call for you).

    i.e. (summarizing) “If it uses ALLOC, xfree, doesn’t use STDIN/STDOUT, it can link against any runtime it wants” [though nobu did note there may be a problem with errno's differing, probably not a problem.]

    That being said it would appear that you need to also avoid the /MD compiler directive in MS compilers, as well [2].

    Also, users may need to download a redistributable for the version of C compiler you used.

    However, it should work.
    Note that for distributing gems, you can use Luis Lavena’s “rake compiler” to build your binaries for you [3].
    Good luck!
    [1] http://blade.nagaokaut.ac.jp/cgi-bin/scat.rb/ruby/ruby-core/22727
    [2] http://groups.google.com/group/thin-ruby/browse_thread/thread/c15b2472eb97c2ba/ea7c5127035d193b
    [3] http://github.com/luislavena/rake-compiler/tree/master ex: ext.cross_platform = ['i386-mswin32', 'i386-mingw32']

    with comments:
    and:

    Using /MD should be fine.

    Actually, it sounded more like to me that they didn’t compile the dll correctly. The dll has to have a manifest embedded in it – which the the regular version of Visual Studio does but nmake does not. Or they didn’t have the redistibutables installed (don’t do in manually, run the Microsoft installation program).

    9:51 AM

    Roger Pack said…
    might need to install the redistributable for [whatever compiler you want to use, like vc2009].

    2:45 PM

    Melissa said…
    http://blogs.law.harvard.edu/hoanga/2006/12/14/getting-a-ruby-c-extension-to-compile-on-windows/
    might be useful, as well

    6:22 AM

    Comment deleted
    This post has been removed by a blog administrator.

    5:02 PM

    Roger Pack said…
    http://groups.google.com/group/comp.lang.python/msg/e3c0937557dc2a7c

    might be helpful, too

    3:43 PM

    Roger Pack said…
    apparently “long doubles” are incompatible

    http://groups.google.com/group/rubyinstaller/browse_thread/thread/c948e28a72c7f81a/450ddb1540d7f411?lnk=gst&q=python#450ddb1540d7f411

    but ruby “doesn’t seem to use those”
    http://mail.python.org/pipermail/python-dev/2009-July/090739.html

    6:05 AM

    rogerdpack2 said…
    more fodder:

    There might be complications if you use time_t, as well.
    “> Also, somewhat unrelated, it appears that long doubles differ in size
    > between mingw/msvc 6, and the size of time_t varies from MSVC compiler
    > to compiler. Does this mean that using dll’s created by those
    > compilers might not work with mingw created binaries?

    If the APIs of the DLLs includes time_t or long double parameters or
    fields in structs pointed to by parameters, etc, then yes indeed.”

    But I really don’t know all the guts.
    Presumably if you only used these internally, that would be ok. Something like that.

  4. How to save MUCH RAM when running rails (linode/slicehost) and mod_rails passenger
    If you’re using mod_rails on a VPS with little RAM [ex: linode, slicehost], then there are a few things you’ll want to do to save RAM when running rails. Here are a few things I did to allow me to run multiple rails processes on one linode slice.

    1: Install a 32-bit OS

    Ruby uses twice as much RAM if you’re in 64-bit than in 32-bit [and most other things do, too]. Use 32-bit! [linode has options to do this easily].

    Savings: 50%

    2: Don’t use as many processes per rails app.
    If you have low volume sites (and most of us do), then don’t create too many processes per app.

    PassengerMaxInstancesPerApp 1

    3: If you are only using one process per rails app, then turn off the spawner process–it does you no help.

    By default mod_rails spins up one “spawner process” and then “x” “actual working processes” The spawner thread just preloads rails so that it can be shared amongst instances of that app.

    In my case, from top (~1 app process):

    total mem, RSS, … name117m 49m 3156 S 0 13.8 0:00.93 ruby # spawner 49M171m 89m 2240 S 0 24.7 0:00.61 ruby # instance 89m
    fix:
    use
    RailsSpawnMethod conservative
    in your apache2.conf

    this results in [slower startup times and]

    RAM, RSS…process name143m 72m 3480 S 0 20.1 0:01.69 rubySavings: (assuming you only want one process per rails app): 49m (on 64-bit).

    If you’re worried about slow startup speed, you can set your rails processes to never expire[4].

    4: If you use multiple processes per rails app, then set your spawner process to die quickly.

    Though I haven’t used it, theoretically the spawner process will be killed eventually–so set it to die quickly, to free up that expensive RAM[5].

    You could do something like
    ab -n10 -c10 http://yourhost
    to ensure it fires up all the processes from the spawner, then the spawner is free to die quickly.

    If you don’t use REE, then don’t use the spawner at all–little savings there RAM wise.

    I have a suspicion that even with REE, you don’t see much RAM savings, but having not used it, I can’t say for certain.

    4: use the MBARI patches to MRI.

    Originally, one of my rails apps started at 89MB RSS then grew in total RAM usage by ~ 8MB per request, linearly. Odd? Yes. Fix: I recompiled ruby using the MBARI patches to 1.8.6/1.8.7. It now starts at 60MB RSS and stays there solid [1]. That’s right–it stays solid at LESS than an unpatched ruby starts at. And with higher speeds [6].

    Also avoid the ruby that comes bundled with ubuntu–though it uses the same amount of RAM as normal, it is compiled with pthreads enabled so it is slower.

    Using 1.9 might also yield RAM savings like this. Haven’t tried it though.

    Savings: unknown since it appeared to growing forever (a lot though).

    Overall Result:

    With these suggestions in place, I can now run 3 or 4 rails apps on my “cheap grade” 360MB linode. Much better than the 1 I was able to originally.

    Enjoy!

    Other potential tricks:

    use nginx instead of apache–faster, much better RAM usage. Potential savings:

    Tweak mysql to use less memory [3]

    Possibly tweak GC settings (37signals’, evan weaver [2], etc.) though this appears to not increase speed too much over the MBARI patches [6].

    refs:

    http://groups.google.com/group/phusion-passenger/browse_thread/thread/df1fc1073dbef38
    [1] http://www.ruby-forum.com/topic/170608#new
    [2] http://blog.evanweaver.com/articles/2009/04/09/ruby-gc-tuning/
    [3] http://articles.slicehost.com/2007/9/11/ubuntu-feisty-mysql-and-ror
    [4] http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerPoolIdleTime
    [5] http://www.modrails.com/documentation/Users%20guide%20Apache.html#_railsframeworkspawneridletime_lt_integer_gt
    [6] http://www.nabble.com/-ruby-core:19846—Bug–744–memory-leak-in-callcc–to20447794.html#a21140287

    make sure to not use the 1.8 that ships by default with ubuntu/debian–it isn’t RAM friendly and it runs slowly!

    9:13 AM

    Roger Pack said…
    wait a second malloc_limit should grow tons if you’re using 168MB RAM, shouldn’t it? With more RAM use you want less frequent GC, I think.

    1:30 PM

    Roger Pack said…
    2K GC blocks, too

    1:30 PM

    Roger Pack said…
    if you use apache, this link
    http://blog.spikesource.com/apache_keepalive.htm
    might be useful

    10:43 AM

    Anonymous said…
    Okay, but how does this affect performance, especially under load?

    5:17 PM

    Roger Pack said…
    Depends what you mean.
    For me, I’m running low traffic sites, and it runs just fine.

    In general the MBARI patches don’t slow things down [1] (they speed things up). It is true that you’ll only have “the equivalent of one mongrel process per site” but if you need more, you can add more now, since you have more RAM available. In general on linode [for me at least] it’s RAM that’s in short supply, not CPU time.

    [1] http://www.nabble.com/-ruby-core:19846—Bug–744–memory-leak-in-callcc–to20447794.html#a21140287

    7:15 AM

    Hongli Lai said…
    Please, do not use ‘top’ to measure memory usage! ‘top’ includes shared memory, which shouldn’t be counted. You should use passenger-memory-stats instead, which reports memory usage correctly. This is the main reason why passenger-memory-stats was created in the first place.

    With REE you’ll save approximately 33% RAM if you have more than 1 process per application.

    12:36 AM

    Hi,

    after loads of research this was the first tutorial that did the job for me (not being a Ruby/apache expert).

    Thanks!

  5. state of the art in ruby compilation/JIT
    There are several levels that one can take compilation of Ruby code to.
    Ex: yarv compiles ruby code to yarv internal byte code. But there are other levels, which we hope to exploit in order to make a faster ruby. We’ll discuss some different style of compilation (JIT and otherwise).

    There are a few existent libraries that do translation.

    ruby2c: translates code like

    def fact(n) n*(fact(n-1))endfact(1000) # it needs this as a hint to know what type the fact method takes
    to standard C (at least in all examples I’ve seen)

    int fact(int n) { return(n*fact(n-1));}

    So their ansiC aspect is the most hard-core “I don’t want this to lany of this in Ruby at all after it’s done.”

    pros: fast as can be. cons: see below.

    rubyinline, interestingly, wraps something like

    “int fact(int n) { return(n*fact(n-1));}”
    with converters to call and return to and from Ruby, so you can call fact(1000) in your ruby code and it will work.

    One interesting idea would be to pipe the output of ruby2c to rubyinline. Alas, ruby2c seems somewhat broken currently.

    cons: you have to write in C. That is not what most of us want to ever have to do again.

    I think where this perspectives fall apart is that they’re not as flexible/dynamic as normal Ruby. It might have trouble with:

    a = [/abc/, 'abc', 33]b = 33a.each{|thing| puts thing.inspect; b+= 1} # dynamic arrays probably not well handled, as well as ruby internal methods like ‘inspect’, blocks, etc.or a = [/abc/, /def/]a.each{|reg| if “abc” =~ reg then puts ‘yes'; end} # regex probably isn’t supported, a lot of the rest of the stdlib

    Then again, I haven’t tried it since ruby2c doesn’t even work for me.

    pros: as fast as fast can be. also comes with a dependency walker so that it can “surmise” which classes of methods you’ll be passing in, then it programs for just those.
    cons: doesn’t handle all ruby (AFAIK).
    Maybe in its current state they’re useful for doing mathematically intense operations? Dunno. Rubyinline is a little more useful, but requires writing in C.

    Ruby2Cext takes a “ruby to ruby C” approach. Its original aim was to produce ruby compatible code equivalents in C.

    i.e. the ruby:

    def go 3end
    translates to

    VALUE go_c(VALUE self) { return INT2NUM(3); // uses all ruby types}void init_File { rb_method_register(someClass, “go”, go_c, 0);}
    It also translates blocks and everything to its ruby c equivalents (using ruby c syntax) . For 1.8 that was said to yield a 2X speed increase.

    They also added some plugins, one gives you the ability to “freeze” certain C method calls, i.e.
    if it encounters String#strip it always calls straight to the C function for String#strip [thus avoids doing a ruby method call]

    ex:

    def go “”.stripend
    Generates code that looks up String#strip and “saves off” the function’s exact location [who would want to override String#strip, right?].

    Thus the above ruby is converted roughtly to something like:

    void *string_strip_function = Qundef;VALUE go_c(VALUE self) { VALUE str = rb_str_new(“”); return strip(str);}VALUE strip(VALUE fromThis) { switch(GET_TYPE(fromThis)) { case STRING : return *(string_strip_function)(fromThis);// this avoids rb_funcall to lookup then call strip else: return rb_funcall3(rb_intern(“strip”), fromThis); }} void init_File { rb_method_register(someClass, “go”, go_c, 0); strip_function = lookup_internal_ruby_c_method(rbString, “strip”);}
    So you can see this avoids a few rb_funcalls to built-in methods, and is still almost entirely ruby compatible. This results in “up to 5x speedup” or so it says.

    drawbacks to rb2cext: 1.9 compiles ruby to bytecode–perhaps rb2xext won’t have as much a gain if used with a 1.9 VM, since it already does some of this work, though profiling would help evaluate this.

    All the above examples were static compilers. Run once before runtime [or at eval time].

    Another class would be JIT dynamic compilers. Jruby is really the only one that has anything like that currently, and yet somehow it doesn’t yet seem to be quite as fast as 1.9 without a JIT [1][4].
    There exists a “demo” JIT using ruby2c, as well [2]. It optimizes a single method, but at least shows you how you could do something more dynamic.

    So where can/should the future of ruby interpreters lie? There’s lots of things you could try.
    A few are making it tighter in C, or making it more JIT’y, or making it moree “dependency walking” so it can pre-optimize the paths that are guaranteed to only have a certain class passed in to them.

    re: making it tighter in C

    One drawback currently to ruby2cext is that if you define

    class A def this_method calls_this_other_method end def calls_this_other_method endendit will translate this (loosely) asVALUE calls_this_other_method = rb_intern(“calls_this_other_method”); // cache the symbol awayVALUE this_method_c(VALUE this) { check_right_parameters(); return rb_funcall(calls_this_other_method);endVALUE calls_this_c(VALUE this) { check_right_parameters(); return Qnil;}

    Note that it did not optimize the call between this_method and calls_this_other_method, but required ruby do rb_funcall. This is the perfectly valid ruby way–you can override calls_this_other_method later and it will work with the new method, but in most cases after a warm up phase methods aren’t overridden, so the rb_funcall could be avoided by calling the method directly. So we could hard code the calls to the other known ruby (now C) methods.

    If we were to assume that class method definitions were “frozen” after a specific setup phase, then a few more optimizations would be available, the above being one of them. [3]
    In other words, rb2cext could become more C-y, with direct calls from method to method.

    Interestingly, ruby2c also has a RubytoRubyC component (I never saw any examples of it posted, and couldn’t get it to run) which might attempt something similar.

    At the same time, ruby2c could become more ruby-y, i.e. integrating with the stdlib [i.e. if you know a string is being passed in, and the command is

    def go a a < < '3'end
    then you could call rb_str_concat directly.

    So there are two ways of action: one to make ruby to c translators more c-y, one to make them more ruby-y. Static compile time type analysis might make it so you can guarantee an object's class and optimize for it. That's an option.

    Another option would be to create something more JIT'y. Profile to discover the "hot paths", then write C that optimizes for the common path (then you could make direct C calls outside your current class, too).

    So what to do? Thoughts?
    -=r

    [1] http://blog.pluron.com/2009/05/ruby-19-performance.html
    [2]
    http://github.com/seattlerb/zenhacks/tree/master
    [3] For instance you could c-ify every ruby method in existence, with direct calls to any C methods within the same class (since we assume they all exist now). Ludicrous allows for this: http://betterlogic.com/roger/?p=1534 though I haven’t experimented speed-wise.
    [4] http://groups.google.com/group/ruby-benchmark-suite/browse_thread/thread/f56b4335cfd3ec57/c7babfb676d71450?lnk=gst&q=patch+gc#c7babfb676d71450 shows how 1.9 with a GC patch can be competitive to jruby.

    comments:

    appears that ruby2c also uses “very conservative” rb_funcall3 for all its funcalls–this cannot be fast [and yet it's ruby's default] :)

    6:19 AM

    Roger Pack said…
    rb2cext doesn’t have a built in Fixnum#times? [doesn't optimize it?]

    6:23 AM

    Roger Pack said…
    note that you don’t need a closure block if there aren’t any local variables…and you know it won’t actually become a named block in any methods calls..maybe? :)

    6:24 AM

    Roger Pack said…
    rb2cx appears to currently use
    rb_define_method(class, “name”, method_c_name, -1) always is that right? Is there a reason for that?
    huh?

    6:27 AM

    Roger Pack said…
    here’s another post about ruby2c’s limitations.
    http://www.ruby-forum.com/topic/188401#new
    appears I’m not the only one that believes it doesn’t translate the std libs et al. (it’s original goal was to be used in the core of the interpreter, i.e. not for ruby code itself but for stuff to RUN ruby code, so I guess that helps you understand why).
    Not going that direction, at least not if I can help it.

    11:51 AM

    Roger Pack said…
    http://betterlogic.com/roger/?p=1568
    shows how rb2cext could improve

    12:25 PM

    Roger Pack said…
    note from
    http://betterlogic.com/roger/?p=1581
    rb2cext can *almost* do what I want it to :)

    11:00 AM

    Roger Pack said…
    here’s [finally] a working ruby2c example

    http://betterlogic.com/roger/?p=1849

  6. how to install sqlite3 from source on ruby mingw (1.8 or 1.9)
    here they are, inspired by a question from Charles Roper
    1) install wget or curl in path
    2) install unzip.exe (http://gnuwin32.sourceforge.net/packages/unzip.htm — binaries link) in path, then put \bin in your path.

    3) install rake compiler [gem install rake-compiler]
    3.1) add the bin folder of GCC from the devkit to your path [i.e. if you installed the straight devkit [1] then “set path=C:\ruby\devkit\gcc\3.4.5\bin;%PATH%”]
    3.5) install msysgit, then git clone git://github.com/luislavena/sqlite3-ruby.git, then cd into it.

    4) rake vendor:sqlite3
    5) rake native gem
    6) gem install pkg/xxx.gem

    [1] http://programming-gone-awry.blogspot.com/2009/05/ruby-19-one-click-installer.html

    comments:

    note before number 5 is
    5) install msysgit
    6) git clone git://github.com/luislavena/sqlite3-ruby.git
    cd into it

    Here’s how to install the binaries [non from source]
    http://blog.mmediasys.com/2009/07/06/getting-started-with-rails-and-sqlite3/

  7. mingw ruby 1.9/1.8 how to install mysql from source

    inspired by a question from JD[1], here’s the instructions:

    Looks like the only way I know of is (assuming ruby 1.9 with devkit [2]) 1) download mysql 5.0.x windows “without installer” 2) unzip, put/rename it to c:\mysql 3) download mysql 2.8.1 binary from tmtm http://www.tmtm.org/en/mysql/ruby/ 4) edit extconf so that anywhere it says mswin32 it now says mswin32|mingw 4a) make sure that mysql’s bin are *not* in your path [might not be ncessary, but just in case] 5) run ruby extconf.rb — –with-mysql-include=c:/mysql/include –with-mysql-lib=c:/mysql/lib/opt 6) run make, then run make install 7) make sure libmysql.dll is in your path or in current directory 8) run ruby -e ‘require “mysql”‘ — should work.[1] http://www.ruby-forum.com/topic/188199#new2 http://programming-gone-awry.blogspot.com/2009/05/ruby-19-one-click-installer.html

    here’s how to install it just the binaries:
    http://blog.mmediasys.com/2009/07/06/getting-started-with-rails-and-mysql/

Leave a Reply

Your email address will not be published. Required fields are marked *


nine − 1 =

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>