It took a while, but I found that the code which handles the debug option for gcc is in the set_debug_level function. Specifically, the "write_symbols" variable is used to store the debug level. We can test that to see if the file line directives should be used or not.
One drawback of this is that by enabling source line information with .file and .loc directives, the dwarf .debug_line section is not created. I'm not sure why that is, but this is hard-coded behavior and not configurable. This just sounds dumb, so I'm overriding this. If you want debug sections, you now get all of them. I just like the idea of being able to use objdump to view the source code.
So I'm not sure what else to do right now. It might be time to do a release.