Finally saying no (almost)

I feel pulled in 100 directions, with no time at all for myself (or cleaning the house).

We haven't been reffing soccer games (by our request) and were asked if we could help.
I said "no" but it's hard to not help when it's needed, and you see others scheduled for multiple games, with schedules that I know are crazy too.

C's job (yay! he has a job!!) is mainly based at home, which is very convenient, but also means he's very available. Also, any production changes have to be done on the weekend, and it seems like he's working 1 or 2 weekends a month lately. With lots of all-nighters.

It seems like we don't have many chances to do things where we're all together.
When the kids were small, we only had D's schedule to deal with, and M and P (and A & S) came along and watched. Even though we were at the park or running around, we were spending lots of time together as a family.

Last year, and in the fall during soccer, it seemed like we were often leaving one kid at the park and running off somewhere else to another task.

I think part of it is having our exchange student living we us. We've been doing a lot of college tours and planning. And then M's school is closing, so we're visiting schools and trying to figure out where she's going to be next year. And P's high school classes and teachers are great, but his grades are horrible, so that's a nightly battle / discussion. And I'm still involved in a math/science club at the grammar school which has a big event May 11-12, that we're not ready for. And P is in Boy Scouts, with all the meetings and camping and merit badges. And my full-time job. And lots more things happening "occasionally" but it's something every week. So, even something as minimal as getting to the park for an extra hour seems like too much.

Why is it so hard to say no?
Do I push myself past collapse before setting limits?
I'm having trouble sleeping, and lie sad and emotional, just trying to rest.