Removing custom firmware (CFW) from a 3DS is not recommended for most cases.
Fixing a broken game
Nearly all cases where a game doesn't work is not due to CFW alone. Most commonly, a corrupt game file or a corrupt save file. Removing CFW would it harder to diagnose the problems with a game. What Luma3DS does is expose more details about why a game doesn't work, allowing for more experienced people to figure it out, while stock would only show a generic "An error has occurred" screen.
Another possible case is due to a plugin or mod that is outdated or for the wrong version of the game. In this case, removing the plugin or mod would fix the game.
There are a few edge cases where games can break due to CFW settings. In particular, a few games crash on New 3DS if 804MHz and L2 cache are enabled. Usually these can be worked around by temporarily disabling the setting that is breaking the game.
Fixing broken system applications
Sometimes a built-in part of the firmware might stop working for some reason. Assuming the system was not updated to a point where Luma3DS is currently not compatible, keeping CFW would mean you can fix the corruption, usually with a CTRTransfer. Removing CFW will probably keep the system in a broken state with no way to fix it without ntrboot.
Concerns about bricks
CFW has become exceptionally safe to use on the 3DS over the years, both during and after setup. It is nearly impossible to brick on accident. GodMode9, the most common tool for interacting with the 3DS NAND, requires unlocking writing with a button combination when you are about to do something that could cause damage. The button combination requires deliberate inputs and is impossible to do by accident.
If Nintendo ever releases a new system update, there is a chance that Luma3DS will be incompatible with it. This is generally not regarded as a brick as it only breaks Luma3DS trying to boot into the firmware, it does not break running other tools such as GodMode9. Usually this is fixed in about a day and is fixed by replacing boot.firm on the SD card.