Been there done that with a really early Mk2 Golf GTI, bought for similar reasons. Got over excited and paid £1500 too much for what turned out to be a car beyond crap. Probably sunk another 2.5k over a year into it to keep it alive. It had been rebuilt with loads of new parts by the previous owner but every part was shit and broke / didn't work. Clutch, gearbox, welding captive bolts for the front subframe, wheel bearings, brakes, springs, shocks, exhaust, every engine sensor, radiator, all hoses... It never lived up to expectations and sold on eBay for £800, I felt really sick until it drove off, then just relief and haven't really looked back.
If you can't easily remove the mods is it worth going the other way with it - if it isn't fun in its current guise go all out and turn it into a track car? Less comfort, more noise and more occasion may actually make it more enjoyable (if you like to suffer).