This page details how installing software/homebrew works on the 3DS.
3DS software formats (CIA, 3DSX, etc.)
Software on the 3DS comes in two main formats: CIA and 3DSX.
CIA.cia format software is usually installed with FBI/GodMode9 and will end up on HOME Menu as a launchable title. Both homebrew and retail game dumps are available in the CIA format.
3DSX.3dsx format software is installed by placing the 3DSX file in the 3ds folder (NOT Nintendo 3DS) and can only be launched from Homebrew Launcher. All 3DSX format software is homebrew.
No format is inherently better than the other; it just depends on how you want to launch your software. If you'd prefer your application on HOME Menu, CIA format is the way to go. Otherwise, 3DSX is probably better.
There is also NDS.nds format software, which can only be used with TWiLightMenu++/nds-bootstrap and can be launched by placing the .nds file anywhere on the SD card visible to TWiLightMenu++. This format of homebrew is also compatible with the DSi and with DS flashcarts. Alternatively, you can create a forwarder that points to the .nds file on your SD card, which will show up on your HOME Menu. For instructions, see here.
The 3DS HOME Menu has a limit of 300 3DS titles and 40 DSiWare titles. The 3DS title limit can be bypassed with 3DSBank. The DSiWare limit itself cannot be bypassed, but if you are primarily installing .nds forwarders, you can use YANBF to make use of the 3DS title limit.
Some other file formats you might encounter (but not as often):
3DS.3ds is used for cartridge dumps, which you can make/install using GodMode9. This file format is also often used with Citra.
CXI.cxi is a generic executable format sometimes used for digital file dumps / preservation.
APP.app is the packed application format that you will see in the Nintendo 3DS folder. You won't be able to directly manipulate these very easily.
In summary, if you want to install software on your 3DS, you should be looking for CIA or 3DSX format applications — and sometimes NDS files.
Installing .cia files
There are 3 applications that are typically used to install 3DS software and homebrew:
FBI is an application that can install CIA files. To use it:
Place your .cia file anywhere on your SD card. (You may want to create a cias folder for organizational purposes.)
Open the FBI application (if you followed 3ds.hacks.guide, it will be on your HOME Menu)
Navigate to SD Card -> (wherever you put the .cia file)
Navigate to Install and delete CIA
The CIA file itself does not need to be kept after it is installed, hence "Install and delete"
Wait for the CIA to install
Once the install has finished, close FBI and the application should appear as a gift box to open on your HOME Menu. If it doesn't immediately appear, you may have to power your console off and on to get it to appear.
GodMode9 is an application that can install CIA files. To use it:
Place your .cia file anywhere on your SD card. (You may want to create a cias folder for organizational purposes.)
Power on your 3DS while holding Start
Navigate to [0:] SDCARD and press A
Navigate to the location of the .cia file you want to install, then press A
Navigate to CIA Image Options... and press A
Navigate to Install game image and press A
Unlock write permissions if prompted
Wait for the CIA to install
Press A when Install success appears
Relock write permissions if prompted
Once the install has finished, reboot your console by pressing Start
If you followed 3ds.hacks.guide to install custom firmware on your 3DS, you probably also have Universal Updater, which can be used as a homebrew app store. To use it:
Open the Universal Updater application (the blue down arrow icon)
Use the D-Pad to navigate between applications and the A button to select an application
Use the Search or Filter function (the magnifying glass) to look for a specific application
Under Available Downloads, select either the .cia, .3dsx, or .nds file, then press A again to install the application
Check the queue (the button with the number in it) to see download progress
CIA files will appear on your HOME Menu in a gift box after you exit Universal-Updater
3DSX files will become available in the Homebrew Launcher application
Once the install has finished, press Start to exit the application, if you installed a CIA file and it doesn't immediately appear, you may have to power your console off and on to get it to appear.
Running .3dsx files
To run a piece of homebrew in .3dsx format:
Place your .3dsx file in the 3ds folder (NOT Nintendo 3DS) on your SD card
Some pieces of homebrew may require additional support files. You may have to consult the README / documentation of that software if this is the case
The homebrew should now appear in the list of homebrew that appears when you open the Homebrew Launcher (the blue "h" icon) on your HOME Menu
Navigate between homebrew applications / folders using the D-Pad and press the A button to launch it
Reinstalling software from the eShop
See this page to redownload content from the 3DS eShop.