Malaysia can claim all of the South China Sea: 17th century British map of the Chinese Empire extends the Indian Ocean into the area now known as the South China Sea and beyond; map probably based on East India Company control over Indian sphere of influence which included South East Asia

 by Ganesh Sahathevan 

The  British map  above of the Chinese Empire was produced in the late 17th century. As with all maps, it represents the information gathered from sources available to the mapmakers at that point of time. 

It clearly shows the Indian Ocean stretching well past modern day Thailand. Thailand and the rest of South East Asia were of course at various times part of the Indian sphere of influence. 

The East India Company on behalf of the British  took over and expanded the administration of the Indian sphere of influence, via treaties  with local chieftains , or by the use of arms. Penang,  and Manila are examples that come to mind.  

It follows then that the British Government would consider all of South East Asia to be part of the Indian sphere of influence, and the Indian Ocean to extend well past Thailand. The British Government were opposed in their expansion by the Dutch, but never the Chinese. In other words, the Chinese never sought to assert  their sovereignty  over  the South Chin Sea , over which they claim historical rights. 

Malaysia is the successor state to that part of South East Asia that was formerly part of the British Empire, bounded by the Malay Peninsula and the island of Borneo, but for Brunei and Kalimantan which is part of Indonesia. The South China Sea lies between those two limits. and hence Malaysia's sovereignty over the South China Sea, which was once considered part of the Indian Ocean.  China, as noted above, has only laid claim to the area in the mid 20th century. 

END