A government that borrows competes with firms for the same savings
When a government spends more than it collects in tax, it borrows the difference by selling bonds. The funds it borrows must come from somewhere: households might save more, private firms might borrow less, or the money might come from foreign investors. Whatever the budget deficit takes, one of those three must give. If private saving and inflows from abroad stay the same, a larger deficit leaves less financial capital for firms to invest in machines and buildings.
The government borrows an extra £50 billion to pay for new spending. What do you think happens to interest rates?
The government's borrowing adds to the demand for financial capital while the supply of savings is unchanged, so lenders can charge more. The Bank sets Bank Rate, but longer-term market rates also respond to borrowing.