Mr Budden said the work would result in “faster, smoother and more reliable journeys” in the future.
The work includes overhead line power upgrades as part of the Midland Mainline electrification programme and the replacement of a 160-year-old road bridge over a railway in Agar Grove, in Camden.
The third part of the project includes works for the future expansion of freight operations in Radlett, near Watford.
Philippa Cresswell, customer service director at EMR, said the firm would run up to 12 buses per hour between Bedford and Milton Keynes in both directions.
Ms Cresswell said the works would be “worth it” ahead of the introduction of the new Aurora bi-mode fleet, which would be able to run on electric and diesel, and provide modern interiors and 45% more seats than other current fleets.