I would recommend a complete "tune-up" on your brake system. You didn't say if you have disc brakes or not. If you do, it will be easier to diagnose your problem. Find a spot with plenty of clean space to work in. Get a fresh/sealed quart bottle of quality brake fluid, a few cans of spray Brake-clean, plenty of clean shop towels or equivalent, tube of white lithium grease, brake adjusting spoon tool, wheel cylinder/caliper bleeder wrench. 100/150 grit emery cloth strip (comes on a roll) strip, or sand paper to clean shoes/drum friction surfaces. Brake bleeding supplies, transparent hose to attach to bleeders. Transparent, plastic container to drain dirty fluid into.
Remove drums. Inspect shoes for wear/contamination from leaking seals/wheel cylinders. If leakage is found, repair before going any further. Repair as needed, if cylinders, replace. rebuilding is slightly cheaper but you must do it properly, use more tools, ( hone, drill etc..) New, quality brand..no more problems. Replace brake system parts in pair, always. Inspect hardware and springs for damage, weakness. Replace if unsure . Not very expensive. Remember that the brake system is constantly exposed to heat and this weakens metal parts and springs. Check brake drums for being within mfg. specs. Spec will be cast into drum. 10.060 for example. May be 9" If out of spec, replace in pairs. The will keep braking forces equal on both sides. Spray the brake dust with a spray bottle of water to keep dust from flying in the air. DO NOT BREATH. The you can use the Brake cleaner to get the stubborn dirt off. Be sure to lube the backing plate at the brake shoe contact areas with a small amount of the Lithium grease when reassembling. Adjust the drum brakes to spec. There is more but you should get a service manual to get all details. After making sure that the mechanicals are repaired /verified in good working order we can now totally flush the brake hydraulic system. Check inside of master cylinder for sediment. Check outside of M/C and areas where the M/C attaches to the booster for evidence of any fluid leakage. If there is, get a NEW M/C. Not a rebuilt. Check booster for vacuum leakage. Replace if faulty. You can use a quality rebuilt Booster if necessary. Drain fluid from M/C. Use a baster or you can use paper towels to soak up fluid. Fill M/C with virgin fluid, go to the furthest point usually the r/r and open bleeder screw with bleeder tube attached. Have someone slowly pump the brake pedal, put something under the pedal to keep it from bottoming out. This can cause damage to the M/C. Make sure to keep M/C topped off while bleeding the fluids. Keep going until clean, clear fluid drains into old fluid container. Close brake bleeder. Continue to the l/r, r/f, l/f until complete system has on cleaner new, clear fluid coming from bleeders. I also like the Hydro-boost system for a lot of reasons. But it is expensive to convert and would be a better upgrade for a car with more performance upgrades that would require a brake system upgrade.