Some investors are pulling back from the bank space in response to a return to a zero-rate environment and loan portfolios riddled with credit quality questions. But others are sticking with the sector, seeing opportunity in depressed valuations.

Castine Capital Management LLC, a community bank investment fund, shut down and sent a letter to shareholders, calling the banking sector "un-investable" and expressing skepticism about the reliability of banks' financial statements. Other investment funds have reduced their holdings of banks. Banc Funds Co. LLC exited positions in 15 banks, but the fund added five to its portfolio and the total value of its holdings stayed pat. Warren Buffett also withdrew from some banks, with Berkshire Hathaway Inc. selling its stake in Goldman Sachs Group Inc. and slashing exposure to JPMorgan Chase & Co. and Wells Fargo & Co.

"Interest in the space is incredibly low," said Christopher McGratty, an analyst for Keefe Bruyette & Woods. "It's a combination of the cyclical nature of banks in a recession, some credit cost uncertainty, interest rates are near zero again — that's a big ding on revenues — and dividend sustainability is top of mind for investors."

But not all investment funds are backing out of banks. Some see opportunity in banks' fortress balance sheet, buttressed by thick capital cushions, and depressed valuations with many banks trading below book value. Another historically famous investor, George Soros, saw opportunity; Soros Fund Management LLC initiated stakes of more than $25 million in Bank of America Corp., Morgan Stanley and Citigroup Inc. Combined, the top 10 largest global hedge funds by equity assets increased their positions in financials by more than any other sector.

Smaller funds with more of a focus on community banks made some moves too. Basswood Capital Management LLC increased its financial stock portfolio by more than 19%, including an initiation into Texas Capital Bancshares Inc. Stieven Capital Advisors LP increased its bank holdings in the second quarter, including new positions in Home BancShares Inc. and Ameris Bancorp, pushing the fund's market value to $433.3 million as of June 30.

Stieven Capital CEO Joe Stieven described the pullback of some funds from the sector as part of the natural ebb-and-flow of an inherently volatile stock market.

"If you work in the stock market, there's always some level of concern, whether it's in the best of times or the worst of times, it doesn't matter," Stieven said in an interview. "I would say the concerns that are out there right now seem pretty well-known."

Stieven said he has been focused on deferral rates and, in particular, recent disclosures from banks on re-deferral rates, a measure of how many borrowers are requesting a second deferral period after the expiration of the initial deferral period. Since the end of the second quarter, several banks have reported dramatic decreases in the portion of loans on deferral. Most recently, HomeTrust Bancshares Inc. reported that 4.3% of its total loan portfolio was on deferral as of Aug. 31, down from 20.5% as of June 30.

Some larger regional banks have released deferral disclosures too, with Synovus Financial Corp. reporting 2% of loans on deferral, down from a peak of 15%, and Regions Financial Corp. also disclosing a deferral rate of 2%, down from 6% at the end of the second quarter.

"I think you'll see a lot of banks put out data intraquarter on their deferral activity, and if it's in the neighborhood of what's gone on with Regions, I think you'll have investors very pleased," said Stieven, who has a stake in Regions Financial.

KBW's McGratty said the investor apprehension is likely more driven by concerns about earnings than credit quality. With robust capital buffers, McGratty said, he is not concerned about banks' ability to weather the recession, but when banks can start to deliver robust profits remains in doubt.

"I think investors can form opinions on what is a reasonable loss estimate," McGratty said. "I think the challenge is: If we see zero rates for the next several years, what's a reasonable [return on equity] banks can earn? And the reality is that it's not great."