memory leaks - Java - Full GC (Garbage Collector) happening a lot in short interval causing performance hit -

i seeing abnormal behavior in our prod environment causing high thread count on server running tomcat. heap size of 10,092,544k divided between new generation , tenure generation 2,752,512k + 7,340,032k = 10,092,544k.

i confused why gc running multiple times when there enough memory available on heap (new , old gen both) (full gc [psyounggen: 0k->0k(2752512k)] [paroldgen: 2748534k->2748529k(7340032k)] )

as can see 0k->0k young gen , .27g -> .27g old gen means hardly objects getting gc'd , there of memory available. (heap size 10g).

since full gc running multiple times during short interval, causing slow performance , hence application can't handle incoming user requests , hence high threads on server , have restart server come out of situation.

can please explain happening here.

here output on gc.log.

---

. .

more...... . .

jul 18 14:52:38 fwprodcontent03 gc.log: 3172.122: [gc [psyounggen: 0k->0k(2752512k)] 2750855k->2750855k(10092544k), 0.0515920 secs] [times: user=0.32 sys=0.01, real=0.06 secs]

jul 18 14:52:42 fwprodcontent03 gc.log: 3172.174: [full gc [psyounggen: 0k->0k(2752512k)] [paroldgen: 2750855k->2749937k(7340032k)] 2750855k->2749937k(10092544k) [pspermgen: 262143k->262115k(262144k)], 4.1571260 secs] [times: user=44.29 sys=0.06, real=4.15 secs]

jul 18 14:52:42 fwprodcontent03 gc.log: 3176.361: [gc [psyounggen: 9071k->1280k(2752512k)] 2759008k->2751217k(10092544k), 0.0989600 secs] [times: user=0.64 sys=0.01, real=0.10 secs]

jul 18 14:52:46 fwprodcontent03 gc.log: 3176.461: [full gc [psyounggen: 1280k->0k(2752512k)] [paroldgen: 2749937k->2748847k(7340032k)] 2751217k->2748847k(10092544k) [pspermgen: 262143k->262121k(262144k)], 3.1331910 secs] [tim es: user=31.25 sys=0.01, real=3.14 secs]

jul 18 14:52:46 fwprodcontent03 gc.log: 3179.616: [gc [psyounggen: 2161k->288k(2752512k)] 2751008k->2749135k(10092544k), 0.0688890 secs] [times: user=0.42 sys=0.02, real=0.07 secs]

jul 18 14:52:49 fwprodcontent03 gc.log: 3179.686: [full gc [psyounggen: 288k->0k(2752512k)] [paroldgen: 2748847k->2748930k(7340032k)] 2749135k->2748930k(10092544k) [pspermgen: 262143k->262143k(262144k)], 3.2369940 secs] [time s: user=32.23 sys=0.02, real=3.23 secs]

jul 18 14:52:49 fwprodcontent03 gc.log: 3182.923: [gc [psyounggen: 0k->0k(2752512k)] 2748930k->2748930k(10092544k), 0.0607190 secs] [times: user=0.39 sys=0.01, real=0.06 secs]

jul 18 14:52:57 fwprodcontent03 gc.log: 3182.984: [full gc [psyounggen: 0k->0k(2752512k)] [paroldgen: 2748930k->2748528k(7340032k)] 2748930k->2748528k(10092544k) [pspermgen: 262143k->262141k(262144k)], 8.5377730 secs] [times: user=98.30 sys=0.57, real=8.54 secs]

jul 18 14:52:58 fwprodcontent03 gc.log: 3191.533: [gc [psyounggen: 202k->128k(2752512k)] 2748731k->2748656k(10092544k), 0.1088430 secs] [times: user=0.67 sys=0.00, real=0.11 secs]

jul 18 14:53:02 fwprodcontent03 gc.log: 3191.642: [full gc [psyounggen: 128k->0k(2752512k)] [paroldgen: 2748528k->2748534k(7340032k)] 2748656k->2748534k(10092544k) [pspermgen: 262143k->262143k(262144k)], 3.1761780 secs] [time s: user=31.11 sys=0.02, real=3.18 secs]

jul 18 14:53:02 fwprodcontent03 gc.log: 3194.820: [gc [psyounggen: 0k->0k(2752512k)] 2748534k->2748534k(10092544k), 0.0589010 secs] [times: user=0.38 sys=0.01, real=0.06 secs]

jul 18 14:53:05 fwprodcontent03 gc.log: 3194.879: [full gc [psyounggen: 0k->0k(2752512k)] [paroldgen: 2748534k->2748529k(7340032k)] 2748534k->2748529k(10092544k) [pspermgen: 262143k->262143k(262144k)], 3.0554520 secs] [times: user=30.72 sys=0.03, real=3.05 secs] . . .more.....

---

thanks in advance.

have looked @ gc stats?

[full gc [psyounggen: 0k->0k(2752512k)] [paroldgen: 2748534k->2748529k(7340032k)] 2748534k->2748529k(10092544k) [pspermgen: 262143k->262143k(262144k)], 3.0554520 secs] 

the important part [pspermgen: 262143k->262143k(262144k)]. permgenspace exhausted , therefore full gc done. therefore increase permgenspace further solve problems (perhaps decreasing heap space little bit).

also don't deploy on production system without restarting tomcat exhaust permgenspace fast.