Investors are getting worried about Trump

— Wall Street experts are expressing growing concern that President Donald Trump isn't going to be everything that investors had hoped for.

U.S. stocks soared right after his election last fall, as investors were eager for the corporate tax reform, regulatory relief and infrastructure spending he promised. But his first few weeks in office suggest that some of those goals may be farther away than they thought.

Instead, Trump's efforts have been focused on controversial issues that businesses generally oppose, such as dropping out of trade deals and imposing new limits on immigration. And it may be hard to win approval for the parts of Trump's agenda that Wall Street does like amid stark political polarization.

Recent notes from Wall Street analysts and economists signal worries about the risks that come with a Trump presidency. Some have scaled back their stock forecasts.

"The Trump Agenda presents risks in both directions; tax cuts and infrastructure funding could boost growth but could be offset by the negative effects of restrictions on trade and immigration," said a note from Goldman Sach's economists. "One month into the year, the balance of risks is somewhat less positive in our view."

"The recent difficulty congressional Republicans have had in moving forward on Obamacare repeal does not bode well for reaching a quick agreement on tax reform or infrastructure funding, and reinforces our view that a fiscal boost, if it happens, is mostly a 2018 story," it added.

A separate Goldman note on Monday said that it now forecasts the S&P to rise to 2,400 by the end of March, but fade to 2,300 by the end of this year, near where it now stands.

Also this week, Wells Fargo pointed to the problems posed by the protests and political unrest that has marked the early weeks of the Trump administration.

"Controversial policies and geopolitical events could impede the progress of U.S. and global economic recoveries," it said in a note on Monday. "The potential for disruption is ever-present. The possibility of a near-term pullback in equity and commodity markets has us more conservatively positioned in most portfolios than in recent quarters."

Wells Fargo also expects markets to end 2017 about where they are today.

A Bank of America note out Monday suggests that the relatively low U.S. unemployment rate will make it difficult for the economy to kick into a higher gear, as predicted by Trump. The report added that the president's plans for spending on infrastructure may simply lead to higher wage inflation and a stronger dollar, which in turn could make it harder for U.S. goods to compete overseas.

"The risks of a recession rise as an economy approaches full employment and begins to overheat," it said.

Of course Wall Street analysts have been wrong about the market and Trump before.

Before the election, many analysts predicted a decline for U.S. stocks if Trump won. Citi analysts specifically forecast an immediate 3% to 5% drop in U.S. stocks. Macroeconomic Advisers predicted an 8% drop.

But when he did win, U.S. stocks took off on a "Trump rally." The S&P climbed about 7% between election day and his Jan. 20 inauguration. The Dow climbed nearly 10% to reach 20,000 for the first time in history in the five days after he took office.

But that rally has essentially stalled in the weeks since then.