Ah I see sorry I understand now (sorry got a cold this week so head not quite with it today).
I assume the 5 in your example is an error and you just want to manipulate the middle column and multiply those by 2.
Yes AFAIK you would have to grab the subset you want and call .mul(2) on that and then concat result back in as you hinted at in your original post.
Also Tensorflow Tensors (both Python and TFJS) are immutable so you can not change a tensor without creating a new version of it in memory anyhow. So I think to do something like this you would need to be working at the array level in your language of choice.