sm64Import v0.1.1

[David]

This is my own personal data importer for Super Mario 64. This tool can be used to import .obj files, basic geometry layouts, binary files, tweak (.xml) files, and raw hex data from a string. You can import as many files as you want, so you can create a level with multiple areas or manage a custom RAM bank with multiple object models easily. This tool is open-source and released under the MIT license, so feel free to use this in any of your projects.

However, I do want to mention that this tool does not generate any level scripts. If you're going to be using this tool for level importing, then you will have to create your own level scripts and import them as a binary file. You can update pointers using this tool, so you don't have to worry about having to change those afterward. If all you need is a level importer, then you should use Skelux's SM64 Editor 2.0 tool.

tl;dr: You can think of this tool as a self-contained Custom Importer that can also be used by other programs.

​Note: 
As of version 0.1.1, you will need to download nconvert.exe as part of the program. I can't legally include it into the project, but you can download it here: http://www.xnview.com/en/nconvert/

​Download sm64Import v0.1.1 (win32)
​Source code (Github)

Spoiler: Options

​-a ROM_ADDRESS {SEG_POINTER}
Set the base ROM address (and optional segmented pointer) to import to. The segmented pointer is required for importing obj files.
​
​-c SIZE
Sets the max data cap. If the following import data size is larger than this, then the program will return an error.
​
​-center BOOLEAN
Centers the follwing obj model(s) if true.
​
​-ct DATA
Allows you to add color to textures. DATA = "Name:Color:DarkScale". DarkScale is a float multiplier from the original color value that is used for lighting, so 0.8 would be 80% as bright as the set color.
​Example: -ct "TextureName:0xFF0000FF:0.8"
​
​-df BOOLEAN X1_POS Z1_POS X2_POS Z2_POS Y_POS
Creates a death floor for the level.
​
​-ib BIN_PATH {OFFSET SIZE} {NAME}
Import binary file into the ROM. Optional parameters for importing only a specific section of the file into the ROM. The optional NAME parameter is used with the -put cmd. Example: -ib "A_Binary_File.bin" 0x1000 0x10
​
-ig OBJ_NAME GEO_TYPE {NAME}
Creates a geo layout from data from a previous -io cmd. GEOTYPE = 0 for objects & GEOTYPE = 1 for levels.
​
-ih HEX_STRING {NAME}
Import raw hex from a string into the ROM. Example: -ih "FF 00 00 FF FF 00 00 FF"
​
-io OBJ_PATH {COL_DATA} {NAME}
Import .obj file as f3d level data (Textures, colors, verticies, and f3d scripts). If you define the optional second parameter, {COL_DATA}, then collision data will be generated for the OBJ.
​
-ioc OBJ_PATH COL_DATA {NAME}
Import .obj file as collision data. COL_DATA is for defining each material with a collision type. Note: You don't have to define every material with COL_DATA, the default value is set to 0 (enviorment default) for each material.
​
-it XML_PATH
Imports a tweak file into the ROM.
​
-n NAME
Sets the name for the next import. The name is used for the -put option.
​
-o OFFSET_X OFFSET_Y OFFSET_Z
Set offset model/collision data.
​
-option OPTION VALUE
Changes an importer value. Currently the only option to change is the vertex reduction level. Example: -option rvl 2 will set the vertex reduction level to 2.
​
-fog ENABLED {FOG_TYPE RED_VALUE GREEN_VALUE BLUE_VALUE}
Enables/Disables fog in the current level.
​
-r ROM_PATH
Defines the ROM to import data to
​
-run PROGRAM
Runs an exe program with arguments. Example: -run "ApplyPPF3.exe a "Path/To/Your/ROM/sm64.ext.z64" "obj_import195S.ppf""
​
-s SCALE_AMOUNT
Scale factor for model.
​
-sv LIGHT_VALUE DARK_VALUE
Set shading value for levels.
​
-sgm DATA
Sets the geometry mode for a particular material. Example: -sgm "TextureName:0x00060205"
​
-tt DATA {EXTRA}
Allows you to define the type of texture a material is, default type is rgba16. Example: -tt "TextureName:rgba32". The currently supported texture types are: RGBA16, RGBA32, I4, I8, IA4, IA8, and IA16. You can make I4 & I8 textures have an alpha mask by setting the optional {EXTRA} parameter to be "a" like this: -tt "i4_tex:i8:a"
​
-put NAME SEG_DATA ROM_ADDRESS {ADD_OFFSET}
Puts the segmented pointer of SEG_Data into the ROM. Useful for updating pointers in your levelscript automatically. SEG_DATA can be (Names are not case-sensative): START, SOLID, ALPHA, TRANS, GEO, or COL. The optional parameter {ADD_OFFSET} will add a value onto the pointer.
​
-vft BOOLEAN
Flip all textures vertically if true.
​
-wb BOX_TYPE_ID X1_POS Z1_POS X2_POS Z2_POS HEIGHT
Creates a water box in the level. Used with -io* & -ioc. *Water box data is grouped with collision data, so you have to define if your just using -io.
​
-wbc
Clears the water box list.
​
-wd
Imports water data for the current level. Note: You MUST import this at the segmented address 0x19001800 with the current patches.

Spoiler: Todo list for future versions

• ​Support CI textures & pallets
  
• ​Add music importing
  
• ​Improve water boxes
  
• ​Improve console compatibility
  

Spoiler: Q&A

​How do I import a .obj file? How do I import it as collision data?

There are two options for importing .obj files. -io is used to import model data and/or collision data, and -ioc is used to import collision data only. You must set the starting segmented pointer with the previous -a option.

-io example (no collision):
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj"

-io example (with collision):
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj" "Material1:0"

-ioc example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -ioc "Test.obj" "Material1:0"

---

​
​How do I setup a geometry layout for my imported model/level?

You can use the -ig option to create a basic geometry layout. The second parameter should be 1 for levels and 0 for object models.

example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj" "Material1:0" -a 0x18E1700 0x19001700 -ig "Test.obj" 1

---

​
​How do I import a binary file?

The -ib option is used to import binary files. You can specify the offset and size properties to only import a specific part of that file. Set the size value to -1 to calculate the size to the end of the file.

Importing a level script example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18E0000 -ib "LevelScript.bin"

Copying a level script from another ROM example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18E0000 -ib "Path/To/Another/ROM/sm64.ext2.z64" 0x18E0000 0x2000

---

​
​How do I import hex data?
​
You can import hex data from a string with the -ih option. This is useful for when you want to modify small amount of bytes in a specific location. Each byte must be separated by a space. If you want to import a large amount of bytes, then I would recommend using -ib instead if you can.

example:
​sm64Import.exe -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x2AC0F8 -ih "00 10 00 19 01 8E 00 00 01 8E 20 00 19 00 00 1C" 

---

​
​How do I import a tweak file?

Tweak files are very simple to import because they are self-contained. All you have to do is use the -it option with the path to the .xml file.

Importing multiple tweaks example:
​sm64Import.exe -r "Path/To/Your/ROM/sm64.ext.z64" -it "Tweaks/LevelImporterPatches.xml" -it "Tweaks/ExtendLevelBoundaries.xml" -it "Tweaks/RCVI hack.xml"

---

​
​How do I update my pointers?

​Updating segmented pointers within the ROM file is set by the -put option. There are 6 different kinds of pointers that you can put:

• ​START - The starting point of a named pointer group.
  
• ​SOLID - Pointer to the solid display list in a model. Created by -io
  
• ​ALPHA - Pointer to the alpha display list in a model. Created by -io
  
• ​TRANS - Pointer to the transparent display list in a model. Created by -io
  
• ​GEO - Pointer to the geometry layout. Created by -ig
  
• ​COL - Pointer to the collision data. Created by -io or -ioc
  

You can either use the .obj path name or a specific name given by the optional {NAME} parameter to get the specific pointer you need.

Updating level-script 0x2E collision pointer example (The 0x2E cmd is located at offset 0x370 in my level script file):
​sm64Import.exe -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18E0000 0x19000000 -c 0x1500 -ib "LevelScript.bin" -a 0x18f0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj" "" -a 0x18E1700 0x19001700 -c 0x100 -ig "Test.obj" 1 -put "Test.obj" col 0x18E0374

If you have any questions or suggestions, then feel free to leave a reply below. I will try to add more documentation when I have the free time/energy to do so.

[RobiNERD]

(10-07-2016, 04:58 AM)David Wrote: This is my own personal data importer for Super Mario 64. This tool can be used to import .obj files, basic geometry layouts, binary files, tweak (.xml) files, and raw hex data from a string. You can import as many files as you want, so you can create a level with multiple areas or manage a custom RAM bank with multiple object models easily. This tool is open-source and released under the MIT license, so feel free to use this in any of your projects.

However, I do want to mention that this tool does not generate any level scripts. If you're going to be using this tool for level importing, then you will have to create your own level scripts and import them as a binary file. You can update pointers using this tool, so you don't have to worry about having to change those afterward. If all you need is a level importer, then you should use Skelux's SM64 Editor 2.0 tool.

tl;dr: You can think of this tool as a self-contained Custom Importer that can also be used by other programs.

​Download sm64Import v0.1 (win32)
​Source code (Github)

Spoiler: Options

​-a ROM_ADDRESS {SEG_POINTER}
Set the base ROM address (and optional segmented pointer) to import to. The segmented pointer is required for importing obj files.
​
​-c SIZE
Sets the max data cap. If the following import data size is larger than this, then the program will return an error.
​
​-center BOOLEAN
Centers the follwing obj model(s) if true.
​
​-ct DATA
Allows you to add color to textures. DATA = "Name:ColorarkScale". DarkScale is a float multiplier from the original color value that is used for lighting, so 0.8 would be 80% as bright as the set color.
​Example: -ct "TextureName:0xFF0000FF:0.8"
​
​-df BOOLEAN X1_POS Z1_POS X2_POS Z2_POS Y_POS
Creates a death floor for the level.
​
​-ib BIN_PATH {OFFSET SIZE} {NAME}
Import binary file into the ROM. Optional parameters for importing only a specific section of the file into the ROM. The optional NAME parameter is used with the -put cmd. Example: -ib "A_Binary_File.bin" 0x1000 0x10
​
-ig OBJ_NAME GEO_TYPE {NAME}
Creates a geo layout from data from a previous -io cmd. GEOTYPE = 0 for objects & GEOTYPE = 1 for levels.
​
-ih HEX_STRING {NAME}
Import raw hex from a string into the ROM. Example: -ih "FF 00 00 FF FF 00 00 FF"
​
-io OBJ_PATH {COL_DATA} {NAME}
Import .obj file as f3d level data (Textures, colors, verticies, and f3d scripts). If you define the optional second parameter, {COL_DATA}, then collision data will be generated for the OBJ.
​
-ioc OBJ_PATH COL_DATA {NAME}
Import .obj file as collision data. COL_DATA is for defining each material with a collision type. Note: You don't have to define every material with COL_DATA, the default value is set to 0 (enviorment default) for each material.
​
-it XML_PATH
Imports a tweak file into the ROM.
​
-n NAME
Sets the name for the next import. The name is used for the -put option.
​
-o OFFSET_X OFFSET_Y OFFSET_Z
Set offset model/collision data.
​
-option OPTION VALUE
Changes an importer value. Currently the only option to change is the vertex reduction level. Example: -option rvl 2 will set the vertex reduction level to 2.
​
-fog ENABLED {FOG_TYPE RED_VALUE GREEN_VALUE BLUE_VALUE}
Enables/Disables fog in the current level.
​
-r ROM_PATH
Defines the ROM to import data to
​
-run PROGRAM
Runs an exe program with arguments. Example: -run "ApplyPPF3.exe a "Path/To/Your/ROM/sm64.ext.z64" "obj_import195S.ppf""
​
-s SCALE_AMOUNT
Scale factor for model.
​
-sv LIGHT_VALUE DARK_VALUE
Set shading value for levels.
​
-sgm DATA
Sets the geometry mode for a particular material. Example: -sgm "TextureName:0x00060205"
​
-tt DATA {EXTRA}
Allows you to define the type of texture a material is, default type is rgba16. Example: -tt "TextureName:rgba32". The currently supported texture types are: RGBA16, RGBA32, I4, I8, IA4, IA8, and IA16. You can make I4 & I8 textures have an alpha mask by setting the optional {EXTRA} parameter to be "a" like this: -tt "i4_tex:i8:a"
​
-put NAME SEG_DATA ROM_ADDRESS {ADD_OFFSET}
Puts the segmented pointer of SEG_Data into the ROM. Useful for updating pointers in your levelscript automatically. SEG_DATA can be (Names are not case-sensative): START, SOLID, ALPHA, TRANS, GEO, or COL. The optional parameter {ADD_OFFSET} will add a value onto the pointer.
​
-vft BOOLEAN
Flip all textures vertically if true.
​
-wb BOX_TYPE_ID X1_POS Z1_POS X2_POS Z2_POS HEIGHT
Creates a water box in the level. Used with -io* & -ioc. *Water box data is grouped with collision data, so you have to define if your just using -io.
​
-wbc
Clears the water box list.
​
-wd
Imports water data for the current level. Note: You MUST import this at the segmented address 0x19001800 with the current patches.

Spoiler: Todo list for future versions

• ​Support CI textures & pallets
  
• ​Add music importing
  
• ​Improve water boxes
  
• ​Improve console compatibility
  

Spoiler: Q&A

​How do I import a .obj file? How do I import it as collision data?

There are two options for importing .obj files. -io is used to import model data and/or collision data, and -ioc is used to import collision data only. You must set the starting segmented pointer with the previous -a option.

-io example (no collision):
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj"

-io example (with collision):
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj" "Material1:0"

-ioc example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -ioc "Test.obj" "Material1:0"

---

​
​How do I setup a geometry layout for my imported model/level?

You can use the -ig option to create a basic geometry layout. The second parameter should be 1 for levels and 0 for object models.

example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj" "Material1:0" -a 0x18E1700 0x19001700 -ig "Test.obj" 1

---

​
​How do I import a binary file?

The -ib option is used to import binary files. You can specify the offset and size properties to only import a specific part of that file. Set the size value to -1 to calculate the size to the end of the file.

Importing a level script example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18E0000 -ib "LevelScript.bin"

Copying a level script from another ROM example:
​sm64Import -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18E0000 -ib "Path/To/Another/ROM/sm64.ext2.z64" 0x18E0000 0x2000

---

​
​How do I import hex data?
​
You can import hex data from a string with the -ih option. This is useful for when you want to modify small amount of bytes in a specific location. Each byte must be separated by a space. If you want to import a large amount of bytes, then I would recommend using -ib instead if you can.

example:
​sm64Import.exe -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x2AC0F8 -ih "00 10 00 19 01 8E 00 00 01 8E 20 00 19 00 00 1C" 

---

​
​How do I import a tweak file?

Tweak files are very simple to import because they are self-contained. All you have to do is use the -it option with the path to the .xml file.

Importing multiple tweaks example:
​sm64Import.exe -r "Path/To/Your/ROM/sm64.ext.z64" -it "Tweaks/LevelImporterPatches.xml" -it "Tweaks/ExtendLevelBoundaries.xml" -it "Tweaks/RCVI hack.xml"

---

​
​How do I update my pointers?

​Updating segmented pointers within the ROM file is set by the -put option. There are 6 different kinds of pointers that you can put:

• ​START - The starting point of a named pointer group.
  
• ​SOLID - Pointer to the solid display list in a model. Created by -io
  
• ​ALPHA - Pointer to the alpha display list in a model. Created by -io
  
• ​TRANS - Pointer to the transparent display list in a model. Created by -io
  
• ​GEO - Pointer to the geometry layout. Created by -ig
  
• ​COL - Pointer to the collision data. Created by -io or -ioc
  

You can either use the .obj path name or a specific name given by the optional {NAME} parameter to get the specific pointer you need.

Updating level-script 0x2E collision pointer example (The 0x2E cmd is located at offset 0x370 in my level script file):
​sm64Import.exe -r "Path/To/Your/ROM/sm64.ext.z64" -a 0x18E0000 0x19000000 -c 0x1500 -ib "LevelScript.bin" -a 0x18f0000 0x0E000000 -c 0x150000 -s 500 -io "Test.obj" "" -a 0x18E1700 0x19001700 -c 0x100 -ig "Test.obj" 1 -put "Test.obj" col 0x18E0374

If you have any questions or suggestions, then feel free to leave a reply below. I will try to add more documentation when I have the free time/energy to do so.

​
Cool program, as I've tested, it makes it easy to import more than 15 levels.

[queueRAM]

Hey, I was able to successfully import a level using sm64Import 0.1, but I had to make some changes to get it to work. First, when I tried to run sm64Import.exe from your build, it crashed with my inputs. I am not sure what went wrong, but it could be related to some weird values in my .obj file (see stof() info below). The crash details are:

Spoiler: crash log

"sm64Import.exe has stopped working"
sm64Import -r sm64.import.z64 -a 0x18F0000 0x0E000000 -c 0x150000 -s 500 -io simple.obj "Grass:0" -a 0x18E1700 0x19001700 -ig simple.obj 1
Problem signature:
 Problem Event Name: APPCRASH
 Application Name: sm64Import.exe
 Application Version: 0.0.0.0
 Application Timestamp: 5781a995
 Fault Module Name: ucrtbase.DLL
 Fault Module Version: 10.0.10240.16390
 Fault Module Timestamp: 55a5bf73
 Exception Code: 40000015
 Exception Offset: 0007d85a
 OS Version: 6.1.7601.2.1.0.256.48
 Locale ID: 1033
 Additional Information 1: 374c
 Additional Information 2: 374c4fb566fde8984c7c52eaf424b4f4
 Additional Information 3: db22
 Additional Information 4: db22e5aaac42e049cf03a56a9aaf157d

Some of the changes I had to make were just due to building with gcc or running on Linux. I'm attaching my updated source.cpp (renamed source.txt for forum) with all the changes I made and a Makefile (renamed Makefile.txt). A description of the changes follows below. Let me know if you have any questions or prefer a pull request on github for these changes.

getline:
getline() works great if you are using text files created on the platform you are running this code, however, it keeps the '\r' in Windows text files in the string when run on a platform that uses different line endings, which causes things like the mtllib filename and texture filenames to keep the '\r' at the end and open() fails (isn't C++ fun?). I used one of the solutions from here: http://stackoverflow.com/questions/6089231

byteswap:
_byteswap_ushort()/_byteswap_ulong() are specific to MSVC. GCC and clang have their own variety. I added some ifdefs and defines for these, but I think the application still might be wrong. For example, if sm64Import were to be built on a system that is already big endian, it would be swapping to little endian. I prefer to keep integers as native endian and then explicitly pack them as big endian in byte arrays.

stof:
I didn't even know this could be a problem until it died on a .obj file that Sketchup produced. I am pretty sure it is because of the really small float values (practically zero) that Sketchup likes to use for vertex normals (e.g. "vn 4.6208e-032 1 -7.08593e-033")
terminate called after throwing an instance of 'std::out_of_range'
 what():  stof
I just converted them all to stod() and let the implicit casts do their thing.

c_str():
Most of the printf()s have this, except the error message at the end and some commented out debug along the way. I went ahead and added it there, otherwise you get garbage output.

sprintf:
I expanded some of the sprintf()s to use %02X instead of %X just to make them nicer for debugging. I would recommend adding a debug flag that can be passed in the command line and uncomment all the printf() debug statements so if you run into trouble with a released build, you can see how far it is getting and where it is choking up.

[David]

Updated the tool to version 0.1.1.

The tool now uses nconvert.exe to convert image files. This allows you to use any type of image file, instead of just only PNG, JPEG, and BMP files. However, you do have to download the program separately since I can't legally include it in the file (Download link included on the main post). If you use the N64Rip program, then you already have it on your system. Just make sure that nconvert.exe is in the same file path as sm64Import.exe

This update also includes a bunch of fixes from queueRAM, so now hopefully you will be able to build the program with GCC without any issues.