n64split v0.4a: ROM splitter and build system

[queueRAM]

(07-11-2017, 02:39 PM)mitchasdf Wrote: I don't fully understand what you mean by there is "unassociated data with it at the end" and also don't know what you mean by "This region should be the start of a new section."

I am interested in finding a fix. One thing that does stand out to me:
section#1 (0x80246000 to 0x8033A57F)
section#2 (0x8016F000 to 0x8071C1FF)

Not only are they overlapping, section#1 is completely encased by section#2. Is this right?

​

Those are the sections I was referring to and no, that isn't how it should be. The problem is the data immediately after the section that gets loaded into 0x8016F000 is just raw data but shouldn't also get loaded at into that address space. So the linker just keeps trying to put the data there and then it does some checks to make sure sections don't overlap which is what the error is coming from. Apparently binutils 2.25 that I was using to test before didn't perform these additional checks.

(07-11-2017, 02:39 PM)mitchasdf Wrote: ​
Is there any way at all to stop this from happening? Such as make it so section#1 is not part of section#2?

​

A quick fix is to add another '.section .datatables' or something before the " .space 0x00500, 0x01 # game_over_level" and add a section in the linker script to load it there. I haven't tested this yet, but something like the following may work:

Code:

​.datatables : {
 * (.datatables);
}

The reason I'm not so concerned about this is the next version of n64split will more than likely use armips as its assembler and I won't have deal with managing linker script.

[mitchasdf]

Thanks for all your help QueueRAM.

I have found a fix!

I learnt a bit more about these unix shells and have found a way to build binutils 2.25 for mips64-elf!

I simply copied the output files from my binutils/bin/ folder into the top-most n64toolchain/bin/, and copied the files from within bin and lib within binutils/mips64-elf/(bin and lib) folders into the n64toolchain/mips64-elf/(bin and lib), replacing some files which were from the latest build from cen64.

Cheering!

[Parashoe]

While preferable that the overlap be fixed, you can get mips64-elf-ld.exe to ignore the section overlap by adding "--no-check-sections" to LDFLAGS defined in the compiler options of the Makefile. Build completes using split files from n64split-0.4a and tools from the current build of n64chain.

Spoiler: Nothing important

I've been interested in SM64 hacking for a long while and been a community lurker since frauber/messian (spelling may be incorrect, what happed to this guy anyway? I remember him moving to starfox hacking?) began documenting and creating importer tools for game models.

While I played around with those tools and was able to make a couple stages, the crude nature of the tools always made me feel it wasn't worth the time. One major turn off for me was the file/info management: so many duplicates of rom files, patches, etc. getting lost as to where things are, trying to move data from one rom to another, rom padding, etc. for making a decently sized mod it's honestly horse****. This low level management is not suitable for game design.

Modding is a hobby, and I appreciate all the work put into the current tools and the necessary reversing process/documentation that said tools are built upon. I find it incredible that the likes of Kaze and Skelux, who have made the current tools work to create the projects they have, put up with the current workflow, but I personally find it not worth the time when there should be better ways. I feel this project is a definite step in the right direction. I really appreciate your contribution to the community.

I've found myself with more free time and am considering spending part of it poking around with SM64 again. Perhaps if I really do get into it, make the full mod that's been looming in my head over the last few years

While I have decent computer skills and some familiarity with scripting, I am no programmer. I am however interested in checking the current state of n64split, but am unable to build sm64tools from source. Running GNU Make 3.81 on the Makefile fails. I'm guessing there are dependencies I don't have or something else I don't know? (hopefully nothing too embarrassing )
How do I build sm64tools? I'm on Windows 10.

[mitchasdf]

(09-11-2017, 10:27 PM)Parashoe Wrote: While preferable that the overlap be fixed, you can get mips64-elf-ld.exe to ignore the section overlap by adding "--no-check-sections" to LDFLAGS defined in the compiler options of the Makefile. Build completes using split files from n64split-0.4a and tools from the current build of n64chain.

​

Handy to know! I spent 4 days trying to build it. It kept failing at every turn.

(09-11-2017, 10:27 PM)Parashoe Wrote: ​
How do I build sm64tools? I'm on Windows 10.

​

Here's what I ended up doing for my windows 10:

Download the latest prebuilt toolchain from cen64.com and store that where ever you want (preferably somewhere you won't have to move it from at any point, because you need to link the toolchain\tools\bin folder in your "Path" environment variable)
**If you would rather try building the toolchain you can go ahead, this MinGW msys may be able to do that - I have no idea, I haven't tried.
**Just omit all of the building binutils steps.

Do away with Cygwin or MSys2 (as standalone shells) and download this (it is a minimalist GNU for windows, it comes with an included older version of MSys and the option for many packages which are required for what we're trying to do) it just does not have any POSIX functionality which wasn't required for me to build binutils:

http://www.mingw.org/
When installing, make sure you mark all 7 or 8 boxes (options including MinGW Base (Meta), Objective C compiler, C++ compiler etc) for installation, it will download and link the included msys with the packages..

After it is installed, navigate into its installation folder (Install_Drive:\MinGW)

Now locate msys.bat, this will be your "MinGW-connected" msys shell console. It will be located within the msys\1.0\ folder.

I personally, did not have to do any additional environment variables or fiddling with any files in order for this to begin building packages. Which leads me to believe it may just work if you wish to build n64tools.

--Building binutils-2.25:

I downloaded "binutils_2.25.orig.tar.gz" from https://launchpad.net/ubuntu/+source/binutils/2.25-5ubuntu7 and extract the contents into your
\MinGW\msys\1.0\home\*YOUR USERNAME*\binutils-2.25\ (you will probably have to create the binutils-2.25 folder first)

While you are at it, create a folder within \MinGW\msys\1.0\home\*YOUR USERNAME*\ called "build-binutils-2.25"

Now from within the msys console, $ cd build-binutils-2.25

Once your target location is ~/build-binutils-2.25
Type this in:
$ ..\binutils-2.25\configure --prefix=/tools/mips64-elf --with-sysroot=$LFS --target=mips64-elf

After it has built the Makefile, just type "$ make" with no additional options
**This may take several minutes

After it has made, just "$ make install" with no additional options

The output files should go into the "tools" folder within where msys.bat is located.

There should be a mips64-elf folder in the tools folder. Within this folder, there will be another mips64-elf folder among other folders, that means we're in the right directory.

Where you see "bin, mips64-elf, share" and no other folders, go into "bin" and copy everything out of there into:
\n64toolkit\tools\bin\*here*, replacing all duplicates.

Now leave bin, and go into the next mips64-elf folder. Copy everything out of "bin" folder within here into:
\n64tools\tools\mips64-elf\bin\*here*, replacing all duplicates.

Now leave that bin folder and go into "\lib\ldscripts\" and copy everything from there into:
\n64tools\tools\mips64-elf\lib\ldscripts\*here*, replacing all duplicates.

This replaces the later version of binutils included within n64tools.

[Parashoe]

Thank you mitchasdf, although I already have working GNU binutils. binutils comes already built with MinGW in <MinGW Install Directory>\bin. The mips64 binutils also come prebuilt with n64chain. I do not need an older version of binutils because the section check added in the newer versions, which was terminating the build of the rom, can be disabled using the --no-check-sections flag when executing the linker. (what my previous post was about)

What I am trying to do, without success, is building "sm64tools" (including dev n64split) from source. I've figured out it has to do with dependencies. capstone, libyaml, zlib, libpng are not included in the repository. I've been able to obtain them and made progress toying around with the Makefile, but something I'm doing is wrong. I've gotten near to the end of the build when ld.exe can't find -lcapstone, -lyaml, -lz. I know those are supposed to be linked bin files for the associated dependencies, but I'm getting a bit confused here.

[mitchasdf]

(10-11-2017, 03:48 AM)Parashoe Wrote: Thank you mitchasdf, although I already have working GNU binutils. binutils comes already built with MinGW in <MinGW Install Directory>\bin. The mips64 binutils also come prebuilt with n64chain. I do not need an older version of binutils because the section check added in the newer versions, which was terminating the build of the rom, can be disabled using the --no-check-sections flag when executing the linker. (what my previous post was about)

​

My apologies, I miss-read your original post. I thought I'd put up instructions on how the get the n64split working again with the current n64-toolchain anyway, since it took so long for me to figure out, it may help other newcomers at some point. But I didn't know you could set that flag when running the linker (I thought you meant during "building" binutils) - that's probably an easier fix.

(10-11-2017, 03:48 AM)Parashoe Wrote: ​
What I am trying to do, without success, is building "sm64tools" (including dev n64split) from source. I've figured out it has to do with dependencies. capstone, libyaml, zlib, libpng are not included in the repository. I've been able to obtain them and made progress toying around with the Makefile, but something I'm doing is wrong. I've gotten near to the end of the build when ld.exe can't find -lcapstone, -lyaml, -lz. I know those are supposed to be linked bin files for the associated dependencies, but I'm getting a bit confused here.

​

This honestly sounds like the trouble I was having trying to figure out how to use the unix shells, until I found MinGW. Darn. I wouldn't have a clue, sorry.

Since I got n64split to work I gave up playing with all this stuff (although the rom runs just fine in Nemu64, it won't even boot in PJ64 (just says it's running at 1000 frames per second on a black screen) - maybe I have to recalculate checksum or something?).

[queueRAM]

(09-11-2017, 10:27 PM)Parashoe Wrote: While preferable that the overlap be fixed, you can get mips64-elf-ld.exe to ignore the section overlap by adding "--no-check-sections" to LDFLAGS defined in the compiler options of the Makefile. Build completes using split files from n64split-0.4a and tools from the current build of n64chain.

Thanks for letting me know about this. I'll recommend "--no-check-sections" as a workaround for n64split 0.4a from now on.

(09-11-2017, 10:27 PM)Parashoe Wrote: ​Running GNU Make 3.81 on the Makefile fails. I'm guessing there are dependencies I don't have or something else I don't know? (hopefully nothing too embarrassing )
How do I build sm64tools? I'm on Windows 10.

I don't spend much time on Windows, so I'm not 100% sure on this. When I generate win32 builds for release, I use the following setup.
Install: msys2 following the steps on msys2's website
Using pacman package manager in msys2, install the following packages: git, make, mingw-w64-i686-gcc, mingw-w64-i686-capstone, mingw-w64-i686-libyaml, mingw-w64-i686-zlib
Use x86_64 instead of i686 if you want win64 builds. There are also cmake build system that I used with Visual Studio back in the day, but I manually build the dependencies to generate static libraries for capstone, libpng, and libconfig (latter two are no longer needed).

I can hop on a Windows 10 machine this weekend to verify if you're still having issues.

Note, n64split from git master currently builds, but won't generate complete assembly code. I am in the middle of adding assembly data references which I hope to complete in the next week or two. mib is also helping me add in a lot of audio processing code, so lots of big changes happening.

(10-11-2017, 05:31 AM)mitchasdf Wrote: ​
the rom runs just fine in Nemu64, it won't even boot in PJ64 (just says it's running at 1000 frames per second on a black screen) - maybe I have to recalculate checksum or something?).

It wouldn't run at all with the incorrect checksum. I don't think I really did much testing on PJ64. It runs on console, cen64, and mupen64plus.

[edgelordslader]

does anyone know why this happens every single time