The 3GX file format is an evolution of the old PLG file format that is used in newer game plugins. You can use them via Luma3DS's built-in plugin loader.
Copy the boot.firm to the root of the SD card, replacing the boot.firm already there
Insert the SD card into your system and power it on
On the Luma Configuration screen, enable any options that you need, then press Start
Hint: If you don't know what the options are, you don't need them
Using 3GX plugins
Plugin location
The 3GX plugin loader will look for plugins in the luma -> plugins directory. Each plugin must be inside a folder named with the game's Title ID. Inside the folder, place your .3gx file.
You can get the game's title ID from here (cartridges), here (digital games), or from the FBI -> Titles menu. For example, Taiko no Tatsujin: Dokodon! Mystery Adventure's title ID is 0004000000190E00, so if you wanted to use a plugin for that game, you'd create a folder named 0004000000190E00
The creator of the plugin may have already created a folder with the correct name for you, in which case you can just copy the title ID folder from the plugin's zip/rar/7z archive to the plugins folder
If you want the plugin to run for all games (for example, for the CTRPluginFramework blank plugin), rename the plugin to default.3gx and just leave it in the plugins folder
An example directory for a plugin affecting one game would be luma -> plugins -> 0004000000190E00 -> plugin.3gx.
An example directory for a plugin affecting all games would be luma -> plugins -> default.3gx.
Launching 3GX plugins
Open the Rosalina menu (Left Shoulder+D-Pad Down+SELECT by default)
Scroll down to the plugin loader option, then press A to enable it
Exit the Rosalina menu by pressing B a couple of times
Open your game of choice
The screen should flash another color to indicate that the plugin has successfully loaded
Look at the information on the plugin's webpage/README for usage instructions - usage will differ for each plugin
Usage notes
Most plugins have a menu that can be launched by pressing SELECT while the plugin is loaded