[Reason] After some search on Google, it is said that Selenium does not provide a convenient way to switch between tabs. That might be the reason it will ignore the option "open new windows in a new tab" in profile, and open always in new window.
[Workaround] 1. We can use Ctrl+Click the link to open the page in new tab. Then we can use Ctrl+PageUp or Ctrl+PageDown or Ctrl+Tab to switch between tabs. 2. Change the option "open new windows in a new tab" by automation script, input keys "Alt+T" and "O", then use "Tab" to choose that option, and finally use "Space" key to select it.
[Note] For how to open a link in new tab for Firefox browser and switch between tabs, I have written an example, please refer to method testBrowserTabs() in regression.testcases.MiscTests.java of Regression Test Project. In that method, I have also shown how to switch between windows.