That would require the hashrate on testnet4 to be orders of magnitude larger than on mainnet. That won't happen.
That's not what I meant. I meant that the marginal increase in hashrate of testnet4 increases difficulty by multiple times more than it would in mainnet, if both had the same hashrate.
Without ASIC blocks, CPU miners can't increase the difficulty any furthe, as they'll be stuck with 1 block every 20 minutes.
But, there will be ASIC blocks mined. In the past week, around 20% of the blocks were ASIC mined. If for each ASIC mined block, there are around 3 CPU mined blocks after it, then around half of the CPU mined blocks are mined due to time travel. If 20% drops to 10%, then CPU mined blocks will also decline by 25%, and then difficulty will decline abruptly again.