I added a function called ‘all_hashes’ in a separate GIT the master branch. Seems slightly faster than the normal of:

a = []
res.each_hash {|h| a << h}
Ruby version:
0.68s — is ruby thread friendly.

C version:
0.58s — is not ruby thread friendly [i.e. blocks for those 0.58s, while using .45s user--i.e. still spends some time IO waiting, but...not too much].

I’m not sure which one I like more

Probably the quicker one since note: IO wait time is 0.25. all_hashes for the same is 0.58s. This means that if we are running all_hashes our parsing is slower than the IO wait is for those results, which means that we are effectively being “CPU efficient” by parsing them as they come in [CPU wait, not IO wait], which means we may as well parse them fast [doing them at the time requires the use of .use_query_result = false and .use_result].

It says somewhere in http://www.ruby-forum.com/topic/41160
That it’s because they don’t actually point to ‘real’ memory–GC’d VALUE’s…so…it doesn’t make as much sense to ++ them, since it would just be masking an extra assignment. Except I would like it so much. I know it breaks the rules, but please make it easy on us.
-=R

Take your queues from the eventmachine ‘em.cpp’ file — the function must accept only one argument. Not sure about the rest [does it need to be void? what about return values?] etc.
Good luck!
Some find it convenient to pass a struct in as the argument, in order to be able to pass more parameters.

rb_thread_blocking_region()

Patch:
download from the versions tab : http://directory.fsf.org/project/patch/
unzip, ./configure, make && make install

http://blog.evanweaver.com/articles/2008/02/05/valgrind-and-ruby/

are instructions on how to install valgrind.

looks like you have to set
:allow_concurrency
in your database.yml file, then it will use the asynchronous calls.
Mysql doesn’t have this option yet, but hopefully it will once people start to use mysqlplus they will add this option, since it’s just starting to be available for mysql.

http://oldmoe.blogspot.com/2008/08/neverblock-mysql-and-mysqlplus.html

Turns out that it doesn’t really exist [since there are no sockets to wait on...makes sense]

There is an
“asynchronous” setting

http://osdir.com/ml/db.sqlite.general/2004-03/msg00168.html

http://www.sqlite.org/pragma.html

set it to off to speed up DB access [but not non-block on it].

And also you could program something in the busy_handler to do something while it is waiting for the lock on the current database to be relinquished.

Also I suppose if you were devious you could have a secondary process or thread and connect to it and have it do the processing while you asynchronously use that. Ugh.

so my latest thought is the sempiternal “hey can’t we just inline everything?”

Some justifications:

some of the most expensive Ruby calls made are:

“three most expensive methods were rb_eval, rb_call0, and rb_call"[1] — and they are somewhat expensive–they have a long method for themselves in the C code

Also note this:

@a = 0

def go
 @a += 1
end

>> Benchmark.measure { 300000.times { go} }
=> Benchmark::Tms:0x3435138 @real=0.64>

>> Benchmark.measure { 300000.times { @a += 1}}
=> Benchmark::Tms:0x3431580 @real=0.40>

Anyway so if you can inline a function then it’s faster–for whatever reason.

It may be possible to speedup other constructs, like passed blocks, even.  Just turn it into one huge method with lots of variables.

blocks are slow’ish:

def yieler; yield; end
>> Benchmark.measure { 300_000.times{ yieler { @a += 1} }}
=> Benchmark::Tms:0x342dad0 @real=1.5

You also might be able to optimize out unnecessary named blocks [see ex in rails recently [2]].

notes:

[1] http://kfahlgren.com/blog/ — also notes some speedup from compiler parameters [heh].

[2] http://blog.pluron.com/2008/02/rails-faster-as.html

Stefan Kaes inlined some rhtml templates with some good results: http://railsexpress.de/blog/articles/2006/08/15/rails-template-optimizer-beta-test [though it's mostly used for routes].

ludicrous does something–wonder what it’s relationship would be to this project.  Could they be combined?

You might could specify expected class types for input parameter when you ‘inline freeze’ a method–then you might could inline calls to parameters.  Not sure if it would help since you’d have to call instance.instance_variable_get and instance.instance_variable_set for instance variables, but maybe it would help speed-wise.

Inlining methods might have a chance to make them more compatible with ruby2c, too [or ludicrous].

UnifiedRuby does some re-writes.  http://rewrite.rubyforge.org/ does, too.

This would help for 1.8.6 but what about ‘real’ Ruby — 1.9? What then?  [Maybe ruby_parser plus an ability to find where in the code functions were defined--or translate iseq's back to ruby?? Maybe run it in 1.8.6, have it create optimized methods for you, use them ].

What other bottlenecks then present themselves after this?

How far to roll out recursive calls? [mostly for the alioth benchmarks to help them along].

May be able to translate to faster constructs: ex: for in instead of .each{} — whichever’s faster.

Note also someone said that compiling with arch flags helps. Huh?

sometimes when running a ruby script with two files that require each other, there can be confusion [at least in 1.8.6].

Fix: have the first file require itself, then it won’t be re-required, skip the require to the second file and run itself before the second file has recursed.
ex:
require File.dirname(__FILE__) + ‘/individual_graph’ # within individual graph itself, where this line is also called within a second file
overcame this error:

./individual_graph.rb:175:in `initialize’: uninitialized constant IndividualGraph::ClientLogContainerWithStats (NameError)

GL. Hopefully not such a big deal with 1.9. One can hope.
-=R

Here’s the cool ones I know of:

footnotes

http://github.com/drnic/rails-footnotes/wikis

it shows at the bottom of all your rendered pages ‘footnotes’ like the session params.

andand allows you to substitute ‘instance && instance.method’ with ‘instance.andand.method’

http://weblog.raganwald.com/2008/01/objectandand-objectme-in-ruby.html

shameless self-promotion:

english like queries allow you to replace things like People.find :first, :conditions => ["name LIKE '%?'", name]

with

People.first_where :name_ends_with => name

very addicting.

any other indispensable ones out there?

asynchronous C Mysql: it’s somewhat possible [from my friend Mohammed of 'oldmoe' fame]

appears that a couple of c api functions offer you ‘pseudo’ asynchronous ability–take a look at
send_query
and
read_query_result

where send_query returns immediately. And you can call read_query_result after the conn->net->fd becomes readable. So somewhat asynchronous.
Rock on.

ex: with ruby bindings [mysqlplus]

@connections = {}

20.times do |i|
  c = Mysql.real_connect('localhost','root',nil)
  @connections[IO.new(c.socket)] = c
end

@connections.each_value do |c|
   c.send_query("select sleep(1)")
end

@sockets = @connections.keys

loop do
  res = select(@sockets,nil,nil,nil)
  if res
    res.first.each{|c|@connections[c].get_result.each{|r| p "#{c.to_i}:#{r}" }}
  end
end