China's reliance on military muscle to achieve its goals "is not a way to make long-term collaboration the rule of the road in a region that's important to China's future", Mattis said, when asked to elaborate.
The South China Sea, a key trade route, is subject to overlapping claims by six countries. The Pentagon said the decision to disinvite the Chinese Navy was triggered by what it called strong evidence that China has deployed weapons systems on the islands.
Beijing has deployed a range of military hardware including anti-ship missiles, surface-to-air missiles and electronic jammers across the South China Sea, where it has built islets and other maritime features into hardened military facilities.
"Despite China's claims to the contrary, the placement of these weapon systems is tied directly to military use for the purposes of intimidation and coercion", Mr Mattis said.
China's state-run Global Times newspaper said in an editorial that tension in the South China Sea was "due to the U.S. continuing to increase its military presence in the region, forcing China to naturally upgrade its defensive weapons on the islands".
China claims almost the entire South China Sea, despite overlapping claims by countries including Vietnam, the Philippines, Brunei, Malaysia and Taiwan.
Senior Colonel Zhao Xiaozhuo said a United States move to send two warships into China's "territorial waters" was a violation of law, and an "obvious provocation to China's national security and territorial integrity".
He warned that America's recent move to disinvite China from a multinational naval exercise this summer was an "initial response" to the militarization of the islands.
"The U.S. will continue to pursue a constructive, results-oriented relationship with China, cooperation whenever possible will be the name of the game and competing vigorously where we must. of course we recognise any sustainable Indo-Pacific order has a role for China", he said.
Beijing has dismissed the USA official's claims of militarization.
"I understand the disagreement, but it is not one on which we are unstudied", Mattis shot back, noting that the FONOPs take place in global waters. "This is our priority theater", Mattis said.
The Chinese general, however, said Beijing's actions were aimed at "national defense". "Certainly, we could have had the dispute resolution go in a peaceful way".
Gen Mattis was speaking just 10 days before President Donald Trump is scheduled to meet North Korean leader Kim Jong-un in Singapore. Trump announced Friday that the meeting was back on.
"Our objective remains the complete, verifiable, and irreversible denuclearisation of the Korean peninsula". But what's less clear is what Washington is prepared to offer Pyongyang.
Last month China said it had for the first time landed bombers on Woody Island in the Paracel Islands, prompting U.S. warnings that it was destabilising the region.
Mattis' address in Singapore was the second time he had attended the summit since becoming Pentagon chief.
"The hopes of the world are on these talks", Mattis said.
South Korea's Defence Minister Song Young-moo arrives to speak next to his Japanese counterpart Itsunori Onodera at the IISS Shangri-la Dialogue in Singapore June 2, 2018.