n64split v0.4a: ROM splitter and build system

[Fendoroid]

Thank you very much for the n64split, queueRam, it's a very valuable tool!  
After a frustrating trial and error process with the gnu-windows 'patch' command line tool, I figured out how to patch the diff example to the .asm file.
But the best moment was after running "make": when the ROM popped out, I had the same feeling of using a build toolchain to produce an executable from source. Maybe I'm wrong, but I've never seen something similar in the history of SM64 hacking. When you use more than one ROM patch contemporary, there's the risk of breaking code. Here instead, it's possible to build an entire ROM from assets.
I think that this tool, in an ideal condition, could be used on the top of the others, meaning that other tools manipulate assets instead of the ROM, and then the builder makes the ROM from the assets.
Maybe it's just because I'm new in asm hacking, but the results of your work left me STUNNED!
For today it's too late, but tomorrow I'll try to write in some asm instead of simply patching an example.
Thanks again, and good work on enhancing the splitter!

EDIT: Would it be possible to automate patching and copyfile through Makefile, instead of doing it manually at every build? Maybe it's a silly question, but I'm not so much in with them...

[queueRAM]

(04-10-2015, 10:21 PM)Fendoroid Wrote: After a frustrating trial and error process with the gnu-windows 'patch' command line tool, I figured out how to patch the diff example to the .asm file.

​
Thanks for trying out n64split, Fendoroid!  I don't spend much time in Windows, so there may be some things I overlooked in testing.  Can you explain specifically what frustrated you about the patching process?  I want to improve it so it is more friendly to new users.

(04-10-2015, 10:21 PM)Fendoroid Wrote: ​
EDIT: Would it be possible to automate patching and copyfile through Makefile, instead of doing it manually at every build? Maybe it's a silly question, but I'm not so much in with them...  

​

It is certainly possible to automate the patching and file copying with a Makefile.  I may just add a batch file to automate patching these examples in the next release.  That should make things easier to understand than trying to interpret my instructions in the README. Thanks for the feedback.

[Fendoroid]

(17-10-2015, 06:46 AM)queueRAM Wrote: Can you explain specifically what frustrated you about the patching process?  I want to improve it so it is more friendly to new users.

​
Well, Windows does not provide a tool for applying diff patches. Thus, I had to install a utility that replicates the GNU patch tool. But now I'm considering to install mingw and msys, to get full GNU integration under Windows. I've did that some years ago, but at the time I was such like a newbie programmer, and have done a disaster (something like installing a bunch of compilers). Then I think that installing msys before using n64split could be a good idea, so you can use the same commands on both platforms, or just use the patch command under Windows.

Also, after some testing under Windows, I found some issues. Maybe some info could be useful to you:

1. ​when I run the command "make clean", all build files are deleted but sm64.map. Is the program supposed to do that?
   
2. ​This bug is more tricky: I replace a texture through copy and substitution, in the textures folder and then build the ROM. Then, in game appears the older texture. In order to get things right, I have to open the texture file, save it and then build the ROM. Maybe the make process was not aware of the file substitution and thus used the old data from gen/bin folder instead from the textures one (you know the builder better than me, I don't know if actually things go like this).
   

PS: I also have built under VS2015 the splitter from source, with pain. lol, I had to change the names of every dependency library in the linker input settings, but after all, it is an alpha project.

[queueRAM]

(17-10-2015, 08:21 AM)Fendoroid Wrote:

(17-10-2015, 06:46 AM)queueRAM Wrote: ​
Can you explain specifically what frustrated you about the patching process?  I want to improve it so it is more friendly to new users.

​
Well, Windows does not provide a tool for applying diff patches. Thus, I had to install a utility that replicates the GNU patch tool. But now I'm considering to install mingw and msys, to get full GNU integration under Windows. I've did that some years ago, but at the time I was such like a newbie programmer, and have done a disaster (something like installing a bunch of compilers). Then I think that installing msys before using n64split could be a good idea, so you can use the same commands on both platforms, or just use the patch command under Windows.

​
Oh, yes, I don't inlcude a copy of GNU patch!  I should do that in the future to make the examples easier to test.  By the way, I also struggle with msys and mingw on Windows.  I have found better support for unix utilities under cygwin, but even then it is very difficult to build portable programs.

(17-10-2015, 08:21 AM)Fendoroid Wrote: ​
Also, after some testing under Windows, I found some issues. Maybe some info could be useful to you:

1. ​when I run the command "make clean", all build files are deleted but sm64.map. Is the program supposed to do that?
   
2. ​This bug is more tricky: I replace a texture through copy and substitution, in the textures folder and then build the ROM. Then, in game appears the older texture. In order to get things right, I have to open the texture file, save it and then build the ROM. Maybe the make process was not aware of the file substitution and thus used the old data from gen/bin folder instead from the textures one (you know the builder better than me, I don't know if actually things go like this).
   

​
Much appreciated!  The "make clean" doesn't remove "sm64.map" but it should.  I will fix this in the next version.

I have experienced the texture problem you are having as well.  'make' is looking at the timestamps of files to determine what to rebuild.  Unfortunately, Windows does not update the modified timestamp of a file that is copied, so it retains the timestamp of the file from it's original location.  Under unix, a 'cp source dest' will update the modification time of the 'dest' file to the current time so 'make' would pick it up.  The only workaround I found is forcing Windows to update the modification time like you are doing or using "copy /b filename +,," to update the timestamp (which is the closest I can find to 'touch').

(17-10-2015, 08:21 AM)Fendoroid Wrote: ​
PS: I also have built under VS2015 the splitter from source, with pain. lol, I had to change the names of every dependency library in the linker input settings, but after all, it is an alpha project.

​
Wow, that is impressive!  Are you using the cmake files to generate the VS project files or did you create your own?  I use cmake to generate mine and have a local directory where I've built all the dependencies.  I want to simplify this process so it is easier to build and contribute to, so let me know if you have suggestions.

[Fendoroid]

(17-10-2015, 09:16 AM)queueRAM Wrote: Are you using the cmake files to generate the VS project files or did you create your own?

​

Summarily:

1. ​About the dependencies, I used cmake to generate all project files but libconfig, that does not use cmake   .
   
2. ​I have built the dependencies and installed them.
   
3. ​I used cmake to generate sm64tools project files.
   
4. ​I've put libconfig into external/lib folder, as suggested by the additional input directory.
   
5. ​In projects 'mipsdisasm', 'n64split' and 'n64graphics' I've renamed the input libraries in the linker settings to match the names of the installed libraries.
   
6. ​I've changed the runtime library to /MDd (in debug), to get rid of most lnk4098 warnings. It still remains one about MSVCRT in 'n64split' project
   

Also, I've noted that some cmake projects searches for environment variables to link automatically deps to the generated project (like libpng does with zlib). But when I install a library, cmake does not create them, maybe because of admin privileges, and then I have to put manually the library path in the cmake-gui configuration window, but it's still better than inserting manually the directories and input libraries for each configuration and for each project into the VS IDE.

[queueRAM]

Update 19 Dezember 2015: n64split v0.4a

• ​fix build system issues reported by Fendoroid
  
• ​add GNU patch and batch files for examples to Windows release
  
• ​new config files: SM64 (J), SM64 (E), MK64 (U)
  
• ​add IA4 and IA1 graphics types
  
• ​add command line options: large textures, proc table, output dir
  
• ​disassemble %hi/%lo macros and BAL instructions
  
• ​add automatic config file detection based on header checksum
  
• ​corrected a lot of textures in the SM64 configs
  
• ​add music sequence bank and instrument set parsing
  
• ​add collision data parser and export to Wavefront OBJ
  
• ​add sm64walk tool to parse through SM64 level scripts
  
• ​correct SM64 behavior script lengths
  

[vexiant]

Lovely to see you add Mario Kart in the mix. How many games do you ultimately intend on making compatible with your tool?

[queueRAM]

(27-12-2015, 07:44 AM)vexiant Wrote: Lovely to see you add Mario Kart in the mix. How many games do you ultimately intend on making compatible with your tool?

​

There are only a few games that I personally want to add, but I am trying to write n64split in such a way that new games could be added with only a minimal config file. I am happy to add configurations for other games if someone is so inclined to create them. The hard part comes from custom game assets such as level data or compression algorithms. The tool will either need customization to support them or a plugin system to make including them easier. Texture data and disassembly is supported out of the box though, which made putting together the Mario Kart 64 config file relatively straightforward. As of n64split 0.4, the MK64 config still lacks a lot of the pieces like track data and has some issues being reassembled. I hope to use the tool to be able to contribute to the Mario Kart 64 hacking thread.

[Luke]

This tool is amazing i'm going to use it to try and hack Mario Kart 64. I am having some problems with getting it to work though when I drop a rom into split.bat nothing happens. I looked into the split.bat code and found that it was.

:: change to batch file's directory so tools and config file can be referenced
pushd %~dp0
:: split input ROM using verbose mode and let it autodetect the config file
tools\n64split.exe -v -c configs\sm64.u.config %1
popd
:: wait for user
pause

The program would do the first 2 lines of code but on the "pause" it just stay paused and I had no way to unpasue it. I could not type or do anything. Any help would be greatly appreciated.

[queueRAM]

(28-12-2015, 12:44 AM)Luke Wrote: This tool is amazing i'm going to use it to try and hack Mario Kart 64. I am having some problems with getting it to work though when I drop a rom into split.bat nothing happens. I looked into the split.bat code and found that it was.

:: change to batch file's directory so tools and config file can be referenced
pushd %~dp0
:: split input ROM using verbose mode and let it autodetect the config file
tools\n64split.exe -v -c configs\sm64.u.config %1
popd
:: wait for user
pause

The program would do the first 2 lines of code but on the "pause" it just stay paused and I had no way to unpasue it. I could not type or do anything. Any help would be greatly appreciated.

​

Remove the "-c configs\sm64.u.config" part and it should work on Windows. I went as far as to change the comment to say "autodetect" but forgot to remove the hard-coded configuration. Sorry about that. Once this is done, running it through the batch file should result in:

Code:

​config.name: Mario Kart 64 (U)
config.basename: mk64.u
config.memory: 2 entries
config.ranges: 89 entries
config.labels: 136 entries
Checking config file 'configs/mk64.u.config' (3E5055B6, 2E92DA52)
Using config file: configs/mk64.u.config
Splitting into "mk64.u.split" directory
Running first pass disassembler...

The pause at the end just leaves the cmd window open until you hit any key so you can view the output (only tested on Windows 7). If you have any further problems, let me know. Keep in mind that the Mario Kart 64 config and splitting is still a work in progress, so I could use help improving on it.