'search -r' is very slow

[baniszew]

"search -r" is far slower than e.g. filtering using a custom filter on the body. The root cause is that searching renders each message, which results in a callout to perl, rather than just staying in C and examining fields. I suspect the right answer is to define search to search over (a fixed set of? all?) message fields, rather than the rendered body.

[asedeno@MIT.EDU]

This is due to BarnOwl needing to render (in memory, not on screen) all messages from the current message back to the first message as it searches. I'm curious to see how this churns the fmtext cache.

[nelhage@MIT.EDU]

Do we actually know that this is due to the rendering, or because it has to swap all the messages into memory, or what? I'd be curious to see some profiling.

[nelhage@MIT.EDU]

search -r apparently eats _insane_ amounts of memory. I suspect this is somehow the fmtext cache's fault, but I don't know. I need to do some debugging.

[nelhage@MIT.EDU]

I can't find any leaks ... I think we may just be seeing horrible heap thrashing by all the back-and-forth between C and perl. I'm going to hope asedeno's utf-8 fmtext fixes mean we can ditch the fmtext cache and won't have this problem as much...

[nelhage@MIT.EDU]

Bleh. There sure seems to be a memory leak here. I'll try to look into it.

[andersk@MIT.EDU]

The memory leak is fixed in commit 1639ff733f7f12836c6bcf7bbf77a9cf4d225b10 and commit e574a0048546c4051f73d18b63169008df045689. search isn’t significantly faster now, but it is a hell of a lot less dangerous.

[geofft@mit.edu]

Is this specific to search -r, or is this just about the fact that formatting messages for searching (either forwards or backwards) is slow?