The 3GX file format is an evolution of the old PLG file format that is used in newer game plugins. You can use them via Luma3DS's built-in plugin loader.
Launch Universal-Updater from the HOME Menu, or from the Homebrew Launcher
Select the magnifying glass on the bottom screen, then search for Luma3DS
Press A while hovering over Luma3DS from the list of homebrew
Choose boot.firm and wait for the install to finish
Reboot your console
If you see the Luma3DS configuration menu, enable any options that you need, then press Start
Hint: If you don't know what the options are, you don't need them
What you need
The latest release of Luma3DS (the Luma3DS .zip file)
Installation
Insert your SD card into your computer
Copy boot.firm from the Luma3DS .zip to the root of your SD card, replacing boot.firm if it is already there
Reinsert your SD card into your console and power on
If you see the Luma3DS configuration menu, enable any options that you need, then press Start
Hint: If you don't know what the options are, you don't need them
Using 3GX plugins
Plugin location
The 3GX plugin loader will look for plugins in the luma -> plugins directory. Each plugin must be inside a folder named with the game's Title ID. Inside the folder, place your .3gx file.
You can get the game's title ID from here (cartridges), here (digital games), or from the FBI -> Titles menu. For example, Taiko no Tatsujin: Dokodon! Mystery Adventure's title ID is 0004000000190E00, so if you wanted to use a plugin for that game, you'd create a folder named 0004000000190E00
The creator of the plugin may have already created a folder with the correct name for you, in which case you can just copy the title ID folder from the plugin's zip/rar/7z archive to the plugins folder
If you want the plugin to run for all games (for example, for the CTRPluginFramework blank plugin), rename the plugin to default.3gx and just leave it in the plugins folder
An example directory for a plugin affecting one game would be luma -> plugins -> 0004000000190E00 -> plugin.3gx.
An example directory for a plugin affecting all games would be luma -> plugins -> default.3gx.
Launching 3GX plugins
Open the Rosalina menu (Left Shoulder+D-Pad Down+SELECT by default)
Scroll down to the plugin loader option, then press A to enable it
Press B to exit the Rosalina menu
Open your game of choice
The screen should flash another color to indicate that the plugin has successfully loaded
Look at the information on the plugin's webpage/README for usage instructions - usage will differ for each plugin
Usage notes
Most plugins have a menu that can be opened by pressing SELECT while the plugin is loaded