FAQ
I have SOLR up and running since few months. Recently I ran into the issue
where the SOLR is taking atleast 15 seconds to return results for a query.

Initially, the "autowarmcount" was set to "0" for "queryResultCache" in
"solrconfig.xml" file was set to "0". I increased it to "512" but this
returns results quickly only for cached queries. It still returns results
after 15 seconds for new queries.

Right now, the SOLR index is 3.3 GB and there are 1.7 million documents.

Any suggestions on how to resolve this issue ?


--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2717761.html
Sent from the Lucene - General mailing list archive at Nabble.com.

Search Discussions

  • Ted Dunning at Mar 22, 2011 at 11:55 pm
    How much memory do you have on the machine?

    What is the heap size for SOLR?

    What does it look like if you look at gc logs?
    On Tue, Mar 22, 2011 at 3:41 PM, manish6886 wrote:

    I have SOLR up and running since few months. Recently I ran into the issue
    where the SOLR is taking atleast 15 seconds to return results for a query.

    Initially, the "autowarmcount" was set to "0" for "queryResultCache" in
    "solrconfig.xml" file was set to "0". I increased it to "512" but this
    returns results quickly only for cached queries. It still returns results
    after 15 seconds for new queries.

    Right now, the SOLR index is 3.3 GB and there are 1.7 million documents.

    Any suggestions on how to resolve this issue ?


    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2717761.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Manish6886 at Mar 23, 2011 at 1:08 am
    I have 4GB memory assigned to the Linux machine where SOLR is running.

    The heap size for SOLR is 2GB

    Can you please tell me where can I find the gc logs ?

    --
    View this message in context: http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2718419.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Ted Dunning at Mar 23, 2011 at 1:59 am

    On Tue, Mar 22, 2011 at 6:07 PM, manish6886 wrote:

    I have 4GB memory assigned to the Linux machine where SOLR is running.

    The heap size for SOLR is 2GB

    Can you please tell me where can I find the gc logs ?
    Search google for [java gc]

    Click on top hit:
    http://christiansons.net/mike/blog/2008/12/java-garbage-collection-logging/

    Read blog entry.
  • Manish6886 at Mar 23, 2011 at 7:05 pm
    Just to give an update, I have SOLR configured in the Master - Slave
    configuration.

    There is 1 Master and 2 slaves and both the slaves are under a VIP.

    The contents of the garbage collector log file are as follows,

    ------------------------------------------------
    50.442: [GC [PSYoungGen: 261226K->43630K(305856K)]
    261226K->57572K(1004928K), 0.0976060 secs] [Times: user=0.09 sys=0.10,
    real=0.10 secs]
    79.552: [GC [PSYoungGen: 305838K->42430K(305856K)]
    319780K->56372K(1004928K), 0.0683510 secs] [Times: user=0.08 sys=0.06,
    real=0.07 secs]
    80.981: [GC [PSYoungGen: 304638K->42392K(305856K)]
    318580K->56335K(1004928K), 0.0296180 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    82.557: [GC [PSYoungGen: 304600K->42648K(305856K)]
    318543K->56591K(1004928K), 0.0293540 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    84.495: [GC [PSYoungGen: 304856K->42811K(305024K)]
    318799K->56753K(1004096K), 0.0293380 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    86.291: [GC [PSYoungGen: 305019K->44154K(282880K)] 318961K->58097K(981952K),
    0.0299640 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]
    87.882: [GC [PSYoungGen: 282554K->1987K(240448K)] 296497K->58300K(939520K),
    0.0712360 secs] [Times: user=0.06 sys=0.09, real=0.07 secs]
    89.322: [GC [PSYoungGen: 240387K->3372K(277440K)] 296700K->59718K(976512K),
    0.0020620 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
    90.459: [GC [PSYoungGen: 217068K->1421K(215168K)] 273414K->59127K(914240K),
    0.0054630 secs] [Times: user=0.00 sys=0.01, real=0.00 secs]
    91.584: [GC [PSYoungGen: 215117K->2650K(277184K)] 272823K->61441K(976256K),
    0.0067010 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
    92.502: [GC [PSYoungGen: 209434K->2688K(209472K)] 268225K->64615K(908544K),
    0.0139990 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
    93.529: [GC [PSYoungGen: 209472K->2910K(278720K)] 271399K->67490K(977792K),
    0.0097700 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
    94.590: [GC [PSYoungGen: 210142K->2686K(278016K)] 274722K->70161K(977088K),
    0.0143320 secs] [Times: user=0.01 sys=0.02, real=0.01 secs]
    96.004: [GC [PSYoungGen: 209918K->2946K(281536K)] 277393K->73075K(980608K),
    0.0108360 secs]
    ------------------------------------------------


    --
    View this message in context: http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2721791.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Ted Dunning at Mar 23, 2011 at 7:37 pm
    This is 46 seconds of data.

    Did you have a slow-down during this time?
    On Wed, Mar 23, 2011 at 12:05 PM, manish6886 wrote:

    Just to give an update, I have SOLR configured in the Master - Slave
    configuration.

    There is 1 Master and 2 slaves and both the slaves are under a VIP.

    The contents of the garbage collector log file are as follows,

    ------------------------------------------------
    50.442: [GC [PSYoungGen: 261226K->43630K(305856K)]
    261226K->57572K(1004928K), 0.0976060 secs] [Times: user=0.09 sys=0.10,
    real=0.10 secs]
    79.552: [GC [PSYoungGen: 305838K->42430K(305856K)]
    319780K->56372K(1004928K), 0.0683510 secs] [Times: user=0.08 sys=0.06,
    real=0.07 secs]
    80.981: [GC [PSYoungGen: 304638K->42392K(305856K)]
    318580K->56335K(1004928K), 0.0296180 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    82.557: [GC [PSYoungGen: 304600K->42648K(305856K)]
    318543K->56591K(1004928K), 0.0293540 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    84.495: [GC [PSYoungGen: 304856K->42811K(305024K)]
    318799K->56753K(1004096K), 0.0293380 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    86.291: [GC [PSYoungGen: 305019K->44154K(282880K)]
    318961K->58097K(981952K),
    0.0299640 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]
    87.882: [GC [PSYoungGen: 282554K->1987K(240448K)] 296497K->58300K(939520K),
    0.0712360 secs] [Times: user=0.06 sys=0.09, real=0.07 secs]
    89.322: [GC [PSYoungGen: 240387K->3372K(277440K)] 296700K->59718K(976512K),
    0.0020620 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
    90.459: [GC [PSYoungGen: 217068K->1421K(215168K)] 273414K->59127K(914240K),
    0.0054630 secs] [Times: user=0.00 sys=0.01, real=0.00 secs]
    91.584: [GC [PSYoungGen: 215117K->2650K(277184K)] 272823K->61441K(976256K),
    0.0067010 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
    92.502: [GC [PSYoungGen: 209434K->2688K(209472K)] 268225K->64615K(908544K),
    0.0139990 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
    93.529: [GC [PSYoungGen: 209472K->2910K(278720K)] 271399K->67490K(977792K),
    0.0097700 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
    94.590: [GC [PSYoungGen: 210142K->2686K(278016K)] 274722K->70161K(977088K),
    0.0143320 secs] [Times: user=0.01 sys=0.02, real=0.01 secs]
    96.004: [GC [PSYoungGen: 209918K->2946K(281536K)] 277393K->73075K(980608K),
    0.0108360 secs]
    ------------------------------------------------


    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2721791.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Ted Dunning at Mar 23, 2011 at 7:38 pm
    Moreover, this is 46 seconds of data from one machine.
    On Wed, Mar 23, 2011 at 12:37 PM, Ted Dunning wrote:

    This is 46 seconds of data.

    Did you have a slow-down during this time?

    On Wed, Mar 23, 2011 at 12:05 PM, manish6886 wrote:

    Just to give an update, I have SOLR configured in the Master - Slave
    configuration.

    There is 1 Master and 2 slaves and both the slaves are under a VIP.

    The contents of the garbage collector log file are as follows,

    ------------------------------------------------
    50.442: [GC [PSYoungGen: 261226K->43630K(305856K)]
    261226K->57572K(1004928K), 0.0976060 secs] [Times: user=0.09 sys=0.10,
    real=0.10 secs]
    79.552: [GC [PSYoungGen: 305838K->42430K(305856K)]
    319780K->56372K(1004928K), 0.0683510 secs] [Times: user=0.08 sys=0.06,
    real=0.07 secs]
    80.981: [GC [PSYoungGen: 304638K->42392K(305856K)]
    318580K->56335K(1004928K), 0.0296180 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    82.557: [GC [PSYoungGen: 304600K->42648K(305856K)]
    318543K->56591K(1004928K), 0.0293540 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    84.495: [GC [PSYoungGen: 304856K->42811K(305024K)]
    318799K->56753K(1004096K), 0.0293380 secs] [Times: user=0.06 sys=0.00,
    real=0.03 secs]
    86.291: [GC [PSYoungGen: 305019K->44154K(282880K)]
    318961K->58097K(981952K),
    0.0299640 secs] [Times: user=0.06 sys=0.00, real=0.03 secs]
    87.882: [GC [PSYoungGen: 282554K->1987K(240448K)]
    296497K->58300K(939520K),
    0.0712360 secs] [Times: user=0.06 sys=0.09, real=0.07 secs]
    89.322: [GC [PSYoungGen: 240387K->3372K(277440K)]
    296700K->59718K(976512K),
    0.0020620 secs] [Times: user=0.01 sys=0.00, real=0.00 secs]
    90.459: [GC [PSYoungGen: 217068K->1421K(215168K)]
    273414K->59127K(914240K),
    0.0054630 secs] [Times: user=0.00 sys=0.01, real=0.00 secs]
    91.584: [GC [PSYoungGen: 215117K->2650K(277184K)]
    272823K->61441K(976256K),
    0.0067010 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
    92.502: [GC [PSYoungGen: 209434K->2688K(209472K)]
    268225K->64615K(908544K),
    0.0139990 secs] [Times: user=0.02 sys=0.00, real=0.01 secs]
    93.529: [GC [PSYoungGen: 209472K->2910K(278720K)]
    271399K->67490K(977792K),
    0.0097700 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]
    94.590: [GC [PSYoungGen: 210142K->2686K(278016K)]
    274722K->70161K(977088K),
    0.0143320 secs] [Times: user=0.01 sys=0.02, real=0.01 secs]
    96.004: [GC [PSYoungGen: 209918K->2946K(281536K)]
    277393K->73075K(980608K),
    0.0108360 secs]
    ------------------------------------------------


    --
    View this message in context:
    http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2721791.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Manish6886 at Mar 23, 2011 at 7:40 pm
    Yeah, I had a slow down during this time and the query took 18 seconds to
    return results back.

    I also observed that whenever I query the Slave machine, the Garbage
    Collection takes place.

    Is it that the Garbage Collection is causing slowness in SOLR ?

    --
    View this message in context: http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2721935.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Matthew Runo at Mar 23, 2011 at 7:44 pm
    Garbage collection will likely stop the whole JVM for the duration if you're not doing some JVM tuning in order to prevent or reduce it.

    Thanks for your time!

    Matthew Runo

    On Mar 23, 2011, at 12:40 PM, manish6886 wrote:

    Yeah, I had a slow down during this time and the query took 18 seconds to
    return results back.

    I also observed that whenever I query the Slave machine, the Garbage
    Collection takes place.

    Is it that the Garbage Collection is causing slowness in SOLR ?

    --
    View this message in context: http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2721935.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Manish6886 at Mar 25, 2011 at 5:29 pm
    After much investigation, I found out that there were stale documents in the
    index which were marked for deletion but hadn't been deleted.

    So, I executed the optimize command on the SOLR index and that reduced the
    index size by almost half.

    Now, my index size is 1.67 million. After this change, the search queries
    were taking 7-8 seconds instead of 18 seconds.

    I tried tweaking the JVM as well with the following options,
    -Xms2048M -Xmx2048M -XX:+UseAdaptiveGCBoundary -XX:+UseParallelGC
    -XX:NewSize=512M -verbose:gc -Xloggc:/taser/lib/gc.log
    -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

    With these changes, the search results were taking about 5-6 seconds.

    Are there any other tweaks or changes which I can try because ideally, the
    SOLR should return the results in milliseconds.

    --
    View this message in context: http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2731382.html
    Sent from the Lucene - General mailing list archive at Nabble.com.
  • Manish6886 at Mar 25, 2011 at 5:31 pm
    After much investigation, I found out that there were stale documents in the
    index which were marked for deletion but hadn't been deleted.

    So, I executed the optimize command on the SOLR index and that reduced the
    index size by almost half.

    Now, my index size is 1.67 million. After this change, the search queries
    were taking 7-8 seconds instead of 18 seconds.

    I tried tweaking the JVM as well with the following options,
    -Xms2048M -Xmx2048M -XX:+UseAdaptiveGCBoundary -XX:+UseParallelGC
    -XX:NewSize=512M -verbose:gc -Xloggc:gc.log -XX:+PrintGCTimeStamps
    -XX:+PrintGCDetails

    With these changes, the search results were taking about 5-6 seconds.

    Are there any other tweaks or changes which I can try because ideally, the
    SOLR should return the results in milliseconds.

    --
    View this message in context: http://lucene.472066.n3.nabble.com/SOLR-search-slowing-down-tp2717761p2731390.html
    Sent from the Lucene - General mailing list archive at Nabble.com.

Related Discussions

Discussion Navigation
viewthread | post
Discussion Overview
groupgeneral @
categorieslucene
postedMar 22, '11 at 10:42p
activeMar 25, '11 at 5:31p
posts11
users3
websitelucene.apache.org

People

Translate

site design / logo © 2018 Grokbase