tool Pokemon Mystery Dungeon 2 - Psy_commando's Tools and research notes

[psy_commando]

Sounds like you guys don't really need me on this. With access to what you already have, I may be able to poke around the ROM and see what stores what, but there's not much I could do beyond that.

Alright then.

Are you familiar with C++ at least a little ? Because I could send you my latest compilable working copy of my library if you want ? Its pretty embarrassing right now, but if it could help poke around the rom, it might just be worth it!

You should consider using some sort of source control for the ROM editor. I recommend CodePlex, as it supports both Git and TFS. If you use Visual Studio, TFS is very convenient. We could also use the documentation features to document what stores what in the ROM, as well as file structures. Github may also be viable, if you'd prefer that. Using source control, we could all collaborate on each project, or at least provide testing without the need for releases.

Right now I'm using a local repository with tortoise svn on my pc for my code, and I don't know what Nerketur uses currently.

And is codeplex/github free and not too capricious with licenses ?

Because those were the issues I had with those kinds of sites..

DeSmuMe already has code to allow directly loading the Arm9 and Arm7, but I'd imagine that it's those binaries that are expecting the files to be in the ROM itself. Maybe if we had the ROM stored in RAM? A hack-ish DeSmuMe may be able to do that, decreasing load and run time a little.

From what I saw, it can only load .elf executables. And we don't have those for pmd2, we only got the obfuscated .bin..

Well, the thing is, the rom data is accessed via requesting data at specific addresses in the rom from the gamecart. And then the gamecart sends the data in groups of 4 bytes at a time. So that makes it all really problematic..

I guess our best bet would be to edit the rom directly. But I don't know if I should focus on something else than PMD2 right now.

Update:

Also, I finally got the thing fully working, and tested. Let me know if there are any bugs!

Though, just to be clear, you need to feed this thing files that begins with either the PKDPX header, or the AT4PX header, without a single byte before it, or it will refuse to do anything!

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_unpx_0.1.zip

I'll update the front page as well!

[evandixon]

Alright then.

Are you familiar with C++ at least a little ? Because I could send you my latest compilable working copy of my library if you want ? Its pretty embarrassing right now, but if it could help poke around the rom, it might just be worth it!

Right now I'm using a local repository with tortoise svn on my pc for my code, and I don't know what Nerketur uses currently.

And is codeplex/github free and not too capricious with licenses ?

Because those were the issues I had with those kinds of sites..

From what I saw, it can only load .elf executables. And we don't have those for pmd2, we only got the obfuscated .bin..

Well, the thing is, the rom data is accessed via requesting data at specific addresses in the rom from the gamecart. And then the gamecart sends the data in groups of 4 bytes at a time. So that makes it all really problematic..

I guess our best bet would be to edit the rom directly. But I don't know if I should focus on something else than PMD2 right now.

Update:

Also, I finally got the thing fully working, and tested. Let me know if there are any bugs!

Though, just to be clear, you need to feed this thing files that begins with either the PKDPX header, or the AT4PX header, without a single byte before it, or it will refuse to do anything!

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/ppmd_unpx_0.1.zip

I'll update the front page as well!

Works great! We'll have to analyze the actual image encoding, but it couldn't have happened without that decompression. I can do some research tomorrow. It's probably similar to the sprite encoding implemented in the python file you linked to.

Codeplex doesn't seem to have any weird license issues. For each project, there's a variety of licenses available, like Apache, GPL, MIT, and several others. If you post your source somewhere like that I may take a look, but I know only a very little C++ at the moment and couldn't do much with it. Having read up on it somewhat today, I may be familiar enough with it to understand what's going on, if not now, in a few days.

Attached: image of a decompressed bpg from the BACK directory.

Capture.zip

[psy_commando]

Good to know ! I was slightly worried I might have broken something while cleaning up

I'm fairly sure the palettes for that image are stored somewhere else, probably in the FONT folder.

Also, they're most likely 8 bpp.

But that's pretty much all I can say without messing with the files myself!

I'll give Codeplex a try I guess. I'll look at it tomorrow.

And C++ is not all that complex syntactically, so I'm guessing you probably shouldn't have too much problem to read it. In the worst cases, I don't mind explaining some things! Plus, I tend to leave myself a lot of notes and comments

Its probably easier to understand than the Python script Zhorken made. Simply because Python has a complex syntax and isn't strongly typed like C and C++ are, which is handy when writing code, but not when trying to read it ^^;

But now that this is done, I gotta refocus on a single goal right now.. Not sure on what I should focus though. I could try finding out how to re-compress things, figure out the sprite data, now that we got access to every kinds of sprites around, or add support for the Kaomado file maybe.. And of course, there's the music and sound effects that I'm really close to figuring out, but I'd need to focus on it to have a decent breakthrough..

I actually was able to get decent results from extracting sound effects from the game. Samples are compressed using a variant of ADPCM. But I can't find anything at all anywhere that deals with ADPCM extraction from NDS files.. I really should stop using google search..

Though, I opened the samples using audacity, importing them as raw data, then picking VOX ADPCM, mono, little endian, and the sample rate of the sample is written on 2 bytes in the "wavi" chunk of the swdl files that contains the sample data. But it usually ranges from 32000 to ~16000. Most of the time its closer to 22050 though. But yeah, that yielded some decent results. Even though I can't replicate it using the same VOX / Diaphonic ADPCM decoding algorithm I got on the web

[evandixon]

I think the bgp files are the same format as the ground sprites you researched earlier. Besides the palate, there's the same issue with the inconsistent length of lines for the tiles. For the Nintendo logo and ESRB stuff at the beginning, it's definitely 4bpp, with some extra data before it. I'd imagine all that data is how all the chunks fit together, and perhaps the palate.

0x420-A1F may be how the chunks are located on the screen. The DS's resolution is 256x192, giving 49152 square pixels. Since each chunk is 64 square pixels, that gives 768 total chunks on the screen, and since the block I mentioned is 0x600 bytes, that gives 2 bytes per chunk to be located. (And seeing that most of the screen is white and most of that block is "01 00 01 00 01 00...") I'll test that later, but that may be how bgp files work.

Update:

Using the following VB code, I got close:

Imports System.Drawing

Module Module1

   Sub Main()
       Const filename As String = "s09p05a"
       Dim file As Byte() = IO.File.ReadAllBytes("C:\Users\Evan\Desktop\sky\data\BACK\" & filename & ".rawimg")
       Dim imageData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &HC40, file.Length - 1)
       Dim mapData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &H420, &HA1F + 1)
       Dim chunks As New List(Of Bitmap)
       For count As Integer = 0 To imageData.Count - 1 Step 32
           Dim i = (ChunkToImage(GenericArrayOperations(Of Byte).CopyOfRange(imageData, count, count + 31)))
           chunks.Add(i)
           'i.Save("C:\Users\Evan\Desktop\sky\data\BACK\temp\" & count / 32 & ".png")
       Next
       Dim outputBitmap As Bitmap = ProcessMapping(mapData, chunks)
       outputBitmap.Save("C:\Users\Evan\Desktop\sky\data\BACK\" & filename & ".png")
   End Sub

   Function ProcessMapping(Data As Byte(), Chunks As List(Of Bitmap)) As Bitmap
       Dim i As New System.Drawing.Bitmap(256, 192)
       Dim g As Graphics = Graphics.FromImage(i)
       Dim dataIndex As Integer = 0
       For y As Integer = 0 To 23
           For x As Integer = 0 To 31
               Dim index As Integer = Data(dataIndex * 2)
               If Chunks.Count >= index - 1 Then
                   Select Case (Data(dataIndex * 2 + 1) Or &HF0) - &HF0
                       Case 0
                           g.DrawImage(Chunks(index - 1), New Point(x * 8, y * 8))
                       Case 1, 2
                           index = BitConverter.ToUInt16({Data(dataIndex * 2), (Data(dataIndex * 2 + 1) Or &HF0) - &HF0}, 0)
                           If Chunks.Count > index Then
                               g.DrawImage(Chunks(index - 1), New Point(x * 8, y * 8))
                           Else
                               Console.WriteLine(index)
                           End If
                       Case 4
                           Dim icopy = Chunks(index - 1).Clone
                           icopy.RotateFlip(RotateFlipType.RotateNoneFlipX)
                           g.DrawImage(icopy, New Point(x * 8, y * 8))
                       Case 8
                           Dim icopy = Chunks(index - 1).Clone
                           icopy.RotateFlip(RotateFlipType.RotateNoneFlipY)
                           g.DrawImage(icopy, New Point(x * 8, y * 8))
                       Case &HC
                           Dim icopy = Chunks(index - 1).Clone
                           icopy.RotateFlip(RotateFlipType.RotateNoneFlipXY)
                           g.DrawImage(icopy, New Point(x * 8, y * 8))
                       Case Else
                           g.DrawImage(Chunks(index - 1), New Point(x * 8, y * 8))
                   End Select
               Else
                   Console.WriteLine(index)
               End If
               dataIndex += 1
           Next
       Next
       Return i
   End Function

   Function ChunkToImage(Data As Byte()) As Bitmap
       Dim i As New System.Drawing.Bitmap(8, 8)
       Dim g As System.Drawing.Graphics = System.Drawing.Graphics.FromImage(i)
       Dim colors As New List(Of Byte)
       For Each b In Data
           colors.Add(((b) Or &HF0) - &HF0)
           colors.Add(((b >> 4) Or &HF0) - &HF0)
       Next
       Dim colorIndex = 0
       For y As Byte = 0 To 7
           For x As Byte = 0 To 7
               g.FillRectangle(Pen(colors(colorIndex)), x, y, 1, 1)
               colorIndex += 1
           Next
       Next
       g.Save()
       Return i
   End Function

   Function Pen(Color As Byte) As Brush
       Dim out As New SolidBrush(System.Drawing.Color.FromArgb(255, Color * 16, Color * 16, Color * 16))
       Return out
   End Function

   Public Class GenericArrayOperations(Of T)
       Public Shared Function CopyOfRange(ByteArr As T(), Index As Integer, EndPoint As Integer) As T()
           Dim output(Math.Max(Math.Min(EndPoint, ByteArr.Length) - Index, 0)) As T
           For x As Integer = 0 To output.Length - 1
               output(x) = ByteArr(x + Index)
           Next
           Return output
       End Function
   End Class

End Module

The mapping chunk starting at 0x420 has two bytes for every 64 square pixel chunk. The first byte and lowest few bits of the second byte, for the most part, appears to be the index of the chunk. The second byte's greatest few bits are a bitfield of what operations to do, to flip on the x, y, or both axises.

I haven't investigated the palate yet, this is a grayscale version.

[update]

There are 4 palettes in bgp files, each chunk can use one of the 4. Still investigating how that works.

Edited October 3, 2014 by evandixon
Learned more about the mapping

[psy_commando]

EDIT:

For some reasons the beginning of my message disapeared

Anyways, that's some nice finds!

I took a quick look to see if I couldn't recognize anything in there:

I took a quick look at an extracted bgp.

There seems to be a header going from 0x0 to 0x20 exclusively.

The value at 0x00 seems to be the length of the header, or a pointer to the beginning of the palette data.

The value at 0x04 seems to be the length of the palette data.

The value at 0x08 seems to be the offset from the end of the header where the actual image starts. or maybe just the offset where the image starts

The value at 0x0C is a really big value. but still within the file's size.. In the middle of a ton of zeros though..

The value at 0x10 seems to point to the first position after the palette.

The value at 0x14 might be the image's length maybe ?

In n_logo :

The palette begins at 0x20 and continue for the amount of bytes specified at offset 0x10.

Its easy to tell its a palette because of the 0x80 byte its 4 bytes per color, with the last byte unused. Most likely RGBX stored in little endian order.

One odd thing is that the size of the section after the palette to the image pointer seems to be 0x800, one of the value in the header. And that value is also very close to the apparent length of the image data section.

Just some quick observations.

[evandixon]

I did it. For a while I thought there were only 4 palettes. There's actually 64 possible palettes, all 16 color sequential portions of the 256 color palette.

The following code will convert a *.rawimg into a png:

Imports System.Drawing

Module Module1

   Sub Main()
       'for all raw img's
       For Each item In IO.Directory.GetFiles("C:\Users\Evan\Desktop\sky\data\BACK", "*.rawimg")
           Dim file As Byte() = IO.File.ReadAllBytes(item)
           Dim imageData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &HC40, file.Length - 1)
           Dim mapData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &H420, &HA1F + 1)
           Dim palData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &H20, &H420)
           Dim chunks As New List(Of Byte())
           For count As Integer = 0 To imageData.Count - 1 Step 32
               'Dim i = (ChunkToImage(GenericArrayOperations(Of Byte).CopyOfRange(imageData, count, count + 31), palData))
               'chunks.Add(i)
               'i.Save("C:\Users\Evan\Desktop\sky\data\BACK\temp\" & count / 32 & ".png")
               chunks.Add(GenericArrayOperations(Of Byte).CopyOfRange(imageData, count, count + 31))
           Next
           Dim outputBitmap As Bitmap = ProcessMapping(mapData, chunks, palData)
           outputBitmap.Save(item.Replace(".rawimg", ".png"))
       Next

       'for just one
       'Const filename As String = "s09p06a"
       'Dim file As Byte() = IO.File.ReadAllBytes("C:\Users\Evan\Desktop\sky\data\BACK\" & filename & ".rawimg")
       'Dim imageData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &HC40, file.Length - 1)
       'Dim mapData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &H420, &HA1F + 1)
       'Dim palData As Byte() = GenericArrayOperations(Of Byte).CopyOfRange(file, &H20, &H420)
       ''For count As Integer = 0 To 500 Step 2 '713 Step 2
       ''    Dim b8 = New SkyEditorBase.Bits8(mapData(count * 2 + 1))
       ''    Dim index As Integer = BitConverter.ToUInt16({mapData(count), (mapData(count + 1) Or &HFC) - &HFC}, 0)
       ''    Dim flipX As Boolean = If(b8.Bit3, True, False)
       ''    Dim flipY As Boolean = If(b8.Bit4, True, False)
       ''    Dim ExtraData As Integer = (mapData(count + 1) >> 4 Or &HF0) - &HF0
       ''    Console.WriteLine("{0} - Flip X: {1}, Flip Y: {2}, Extra Data: {3}", count, flipX, flipY, ExtraData)
       ''Next
       'Dim chunks As New List(Of Byte())
       'For count As Integer = 0 To imageData.Count - 1 Step 32
       '    'Dim i = (ChunkToImage(GenericArrayOperations(Of Byte).CopyOfRange(imageData, count, count + 31), palData))
       '    'chunks.Add(i)
       '    'i.Save("C:\Users\Evan\Desktop\sky\data\BACK\temp\" & count / 32 & ".png")
       '    chunks.Add(GenericArrayOperations(Of Byte).CopyOfRange(imageData, count, count + 31))
       'Next
       'Dim outputBitmap As Bitmap = ProcessMapping(mapData, chunks, palData)
       'outputBitmap.Save("C:\Users\Evan\Desktop\sky\data\BACK\" & filename & ".png")
       'Console.ReadLine()
   End Sub

   Function ProcessMapping(Data As Byte(), Chunks As List(Of Byte()), PalData As Byte()) As Bitmap
       Dim i As New System.Drawing.Bitmap(256, 192)
       Dim g As Graphics = Graphics.FromImage(i)
       Dim dataIndex As Integer = 0
       For y As Integer = 0 To 23
           For x As Integer = 0 To 31
               Dim index As Integer = BitConverter.ToUInt16({Data(dataIndex * 2), (Data(dataIndex * 2 + 1) Or &HFC) - &HFC}, 0) 'Data(dataIndex * 2)
               If Chunks.Count >= index - 1 AndAlso index > 0 Then
                   Dim b8 = New SkyEditorBase.Bits8(Data(dataIndex * 2 + 1))
                   Dim palette As Integer = ((Data(dataIndex * 2 + 1) >> 4 Or &HF0) - &HF0)
                   Dim icopy = ChunkToImage(Chunks(index - 1), PalData, palette) 'Chunks(index - 1).Clone
                   If b8.Bit3 Then
                       icopy.RotateFlip(RotateFlipType.RotateNoneFlipX)
                   End If
                   If b8.Bit4 Then
                       icopy.RotateFlip(RotateFlipType.RotateNoneFlipY)
                   End If
                   g.DrawImage(icopy, New Point(x * 8, y * 8))
               Else
                   Console.WriteLine(index)
               End If
               dataIndex += 1
           Next
       Next
       Return i
   End Function

   Function ChunkToImage(Data As Byte(), PalData As Byte(), PalIndex As Byte) As Bitmap
       Dim i As New System.Drawing.Bitmap(8, 8)
       Dim g As System.Drawing.Graphics = System.Drawing.Graphics.FromImage(i)
       Dim colors As New List(Of Byte)
       For Each b In Data
           colors.Add(((b) Or &HF0) - &HF0)
           colors.Add(((b >> 4) Or &HF0) - &HF0)
       Next
       Dim colorIndex = 0
       For y As Byte = 0 To 7
           For x As Byte = 0 To 7
               g.FillRectangle(Pen(colors(colorIndex), PalData, PalIndex), x, y, 1, 1)
               colorIndex += 1
           Next
       Next
       g.Save()
       Return i
   End Function

   Function Pen(Color As Byte, palData As Byte(), PalNumber As Byte) As Brush
       Dim r = palData(PalNumber * 64 + Color * 4 + 0)
       Dim g = palData(PalNumber * 64 + Color * 4 + 1)
       Dim b = palData(PalNumber * 64 + Color * 4 + 2)
       'Dim out As New SolidBrush(System.Drawing.Color.FromArgb(255, Color * 16, Color * 16, Color * 16))
       Dim out As New SolidBrush(System.Drawing.Color.FromArgb(255, r, g, b))
       Return out
   End Function

   Public Class GenericArrayOperations(Of T)
       Public Shared Function CopyOfRange(ByteArr As T(), Index As Integer, EndPoint As Integer) As T()
           Dim output(Math.Max(Math.Min(EndPoint, ByteArr.Length) - Index, 0)) As T
           For x As Integer = 0 To output.Length - 1
               output(x) = ByteArr(x + Index)
           Next
           Return output
       End Function
   End Class

End Module

If you don't have a copy of Sky Editor's Utilities.vb, the above uses the Bit8 class:

Public Class Bits8
   Public Property Value As Byte
#Region "Bits 1-8"
   Public Property Bit1 As Boolean
       Get
           Return ((Value Or 254) - 254) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 1) - 1) + (bit * 1)
       End Set
   End Property
   Public Property Bit2 As Boolean
       Get
           Return ((Value Or 253) - 253) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 2) - 2) + (bit * 2)
       End Set
   End Property
   ''' <summary>
   ''' Bit that adds 4 to the value
   ''' </summary>
   ''' <value></value>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Property Bit3 As Boolean
       Get
           Return ((Value Or 251) - 251) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 4) - 4) + (bit * 4)
       End Set
   End Property
   ''' <summary>
   ''' Bit that adds 8 to the value
   ''' </summary>
   ''' <value></value>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Property Bit4 As Boolean
       Get
           Return ((Value Or 247) - 247) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 8) - 8) + (bit * 8)
       End Set
   End Property
   ''' <summary>
   ''' Bit that adds 16 to the value
   ''' </summary>
   ''' <value></value>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Property Bit5 As Boolean
       Get
           Return ((Value Or 239) - 239) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 16) - 16) + (bit * 16)
       End Set
   End Property
   ''' <summary>
   ''' Bit that adds 32 to the value
   ''' </summary>
   ''' <value></value>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Property Bit6 As Boolean
       Get
           Return ((Value Or 223) - 223) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 32) - 32) + (bit * 32)
       End Set
   End Property
   ''' <summary>
   ''' Bit that adds 64 to the value
   ''' </summary>
   ''' <value></value>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Property Bit7 As Boolean
       Get
           Return ((Value Or 191) - 191) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 64) - 64) + (bit * 64)
       End Set
   End Property
   ''' <summary>
   ''' Bit that adds 128 to the value
   ''' </summary>
   ''' <value></value>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Property Bit8 As Boolean
       Get
           Return ((Value Or 127) - 127) > 0
       End Get
       Set(value As Boolean)
           Dim bit As Byte = 0
           If value Then bit = 1
           Me.Value = ((Me.Value Or 128) - 128) + (bit * 128)
       End Set
   End Property
#End Region
   ''' <summary>
   ''' Bit order: 87654321
   ''' So, FromBits(False,False,True,False,False,False,False,False)=4
   ''' </summary>
   ''' <param name="Bit1"></param>
   ''' <param name="Bit2"></param>
   ''' <param name="Bit3"></param>
   ''' <param name="Bit4"></param>
   ''' <param name="Bit5"></param>
   ''' <param name="Bit6"></param>
   ''' <param name="Bit7"></param>
   ''' <param name="Bit8"></param>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Shared Function FromBits(Bit1 As Boolean, Bit2 As Boolean, Bit3 As Boolean, Bit4 As Boolean, Bit5 As Boolean, Bit6 As Boolean, Bit7 As Boolean, Bit8 As Boolean) As Bits8
       Dim out As New Bits8(0)
       If Bit1 Then out += 1
       If Bit2 Then out += 2
       If Bit3 Then out += 4
       If Bit4 Then out += 8
       If Bit5 Then out += 16
       If Bit6 Then out += 32
       If Bit7 Then out += 64
       If Bit8 Then out += 128
       Return out
   End Function
   ''' <summary>
   ''' Returns a Bit8 from an array of byte
   ''' </summary>
   ''' <param name="Binary">Byte array to get value from</param>
   ''' <param name="ByteIndex">Index of the byte to get the value from</param>
   ''' <param name="BitIndex">Index of the bit (0 to 7) to start getting the value from.  If the binary is 11110000 00001111, a BitIndex of 2 would return 11000000.</param>
   ''' <returns></returns>
   ''' <remarks></remarks>
   Public Shared Function FromBinary(Binary As Byte(), ByteIndex As Long, BitIndex As Long) As Bits8
       Dim output As Bits8 = Nothing
       If BitIndex > 7 Then
           ByteIndex += Math.Floor(BitIndex \ 8)
           BitIndex = BitIndex Mod 8
       End If
       Dim byte1 As Bits8 = Binary(ByteIndex)
       Dim byte2 As Bits8 = 0
       If Binary.Length > ByteIndex + 1 Then
           byte2 = Binary(ByteIndex + 1)
       End If
       Select Case BitIndex
           Case 0
               Dim out As Bits8 = 0
               If byte1.Bit1 Then out += 1
               If byte1.Bit2 Then out += 2
               If byte1.Bit3 Then out += 4
               If byte1.Bit4 Then out += 8
               If byte1.Bit5 Then out += 16
               If byte1.Bit6 Then out += 32
               If byte1.Bit7 Then out += 64
               If byte1.Bit8 Then out += 128
               output = out
           Case 1
               Dim out As Bits8 = 0
               If byte2.Bit8 Then out += 1
               If byte1.Bit1 Then out += 2
               If byte1.Bit2 Then out += 4
               If byte1.Bit3 Then out += 8
               If byte1.Bit4 Then out += 16
               If byte1.Bit5 Then out += 32
               If byte1.Bit6 Then out += 64
               If byte1.Bit7 Then out += 128
               output = out
           Case 2
               Dim out As Bits8 = 0
               If byte2.Bit7 Then out += 1
               If byte2.Bit8 Then out += 2
               If byte1.Bit1 Then out += 4
               If byte1.Bit2 Then out += 8
               If byte1.Bit3 Then out += 16
               If byte1.Bit4 Then out += 32
               If byte1.Bit5 Then out += 64
               If byte1.Bit6 Then out += 128
               output = out
           Case 3
               Dim out As Bits8 = 0
               If byte2.Bit6 Then out += 1
               If byte2.Bit7 Then out += 2
               If byte2.Bit8 Then out += 4
               If byte1.Bit1 Then out += 8
               If byte1.Bit2 Then out += 16
               If byte1.Bit3 Then out += 32
               If byte1.Bit4 Then out += 64
               If byte1.Bit5 Then out += 128
               output = out
           Case 4
               Dim out As Bits8 = 0
               If byte2.Bit5 Then out += 1
               If byte2.Bit6 Then out += 2
               If byte2.Bit7 Then out += 4
               If byte2.Bit8 Then out += 8
               If byte1.Bit1 Then out += 16
               If byte1.Bit2 Then out += 32
               If byte1.Bit3 Then out += 64
               If byte1.Bit4 Then out += 128
               output = out
           Case 5
               Dim out As Bits8 = 0
               If byte2.Bit4 Then out += 1
               If byte2.Bit5 Then out += 2
               If byte2.Bit6 Then out += 4
               If byte2.Bit7 Then out += 8
               If byte2.Bit8 Then out += 16
               If byte1.Bit1 Then out += 32
               If byte1.Bit2 Then out += 64
               If byte1.Bit3 Then out += 128
               output = out
           Case 6
               Dim out As Bits8 = 0
               If byte2.Bit3 Then out += 1
               If byte2.Bit4 Then out += 2
               If byte2.Bit5 Then out += 4
               If byte2.Bit6 Then out += 8
               If byte2.Bit7 Then out += 16
               If byte2.Bit8 Then out += 32
               If byte1.Bit1 Then out += 64
               If byte1.Bit2 Then out += 128
               output = out
           Case 7
               Dim out As Bits8 = 0
               If byte2.Bit2 Then out += 1
               If byte2.Bit3 Then out += 2
               If byte2.Bit4 Then out += 4
               If byte2.Bit5 Then out += 8
               If byte2.Bit6 Then out += 16
               If byte2.Bit7 Then out += 32
               If byte2.Bit8 Then out += 64
               If byte1.Bit1 Then out += 128
               output = out
               'Case Else
               '    Throw New ArgumentException("BitIndex must be a value from 0 to 7.")
       End Select
       Return output
   End Function
   Public Shared Function FromBinary(Binary As Byte(), ByteIndex As Long, BitIndex As Long, ByteCount As Long) As Byte()
       Dim out(ByteCount - 1) As Byte
       For count As Integer = 0 To ByteCount - 1
           out(count) = FromBinary(Binary, ByteIndex + count, BitIndex)
       Next
       Return out
   End Function

   Public Sub New(Value As Byte)
       Me.Value = Value
   End Sub
#Region "Operators"
   Public Shared Narrowing Operator CType(ByVal x As Byte) As Bits8
       Return New Bits8(x)
   End Operator
   Public Shared Widening Operator CType(ByVal x As Bits8) As Byte
       Return x.Value
   End Operator
   Public Shared Operator +(ByVal x As Bits8, ByVal y As Bits8) As Bits8
       Return x.Value + y.Value
   End Operator
   Public Shared Operator =(ByVal x As Bits8, ByVal y As Byte) As Boolean
       Return x.Value = y
   End Operator
   Public Shared Operator <>(ByVal x As Bits8, ByVal y As Byte) As Boolean
       Return x.Value <> y
   End Operator
#End Region

   Public Overrides Function ToString() As String
       Return Value.ToString
   End Function
End Class

The next step will be to reverse the process to convert png's into a raw image, then bgp. That, or analyze other file formats.

[psy_commando]

Nice.

Though, I wouldn't call those ".rawimg". Its just a generic extension I gave to images extracted from AT4PX containers.

Given the image format stored in those may be different from one another.

Though, if I can have more details on the formats, like from those extracted from BGP files, I could probably handle the file extensions better.

And what do you mean by 64 possible palettes ?

I thought it looked like a regular single 256 colors palette. Unless I missed something?

What is strange about this, is that the portraits aren't using the same format..

Why store the palettes before the AT4PX if they could put them inside with that file structure?

Anyways, I guess they're not always doing very logical things..

[evandixon]

Each 8x8 chunk/tile has its own palette, 16 of the 256 colors, starting at a particular offset, possibly different for each chunk/tile.

The MAP_BG files appear to have all the data in different files. I have no idea why. I'll have to look at it further, but each file is probably similar to the blocks in bgp.

[psy_commando]

Each 8x8 chunk/tile has its own palette, 16 of the 256 colors, starting at a particular offset, possibly different for each chunk/tile.

The MAP_BG files appear to have all the data in different files. I have no idea why. I'll have to look at it further, but each file is probably similar to the blocks in bgp.

That's interesting! So each tile has its own 16 colors palette ?

That works with the 4bpp format then!

It would have taken me a while to figure that one out

You seem pretty handy with image data! If you get the chance, do you think you could compare the character sprites from either "/MONSTER/m_ground.bin", "/MONSTER/m_attack.bin" or "/MONSTER/monster.bin" to the sprites stored inside ".wan" files? (there are a bunch inside the "/FONT/" folder)

Both of those are using the same structure, stored within a SIR0 container. They differ slightly, probably because the pokemon sprites are animated I'd say, but I can't seem to figure out at all what a big part of the structure in those is supposed to be there for..

The SIR0 header's first pointer points to a little 12 bytes chunk at the very end of the file.

And there, the first 4 bytes is a pointer to a part that contains other pointers to a bunch of "tables" and details about those.

And the next 4 bytes is a pointer to another part that contains pointers to the actual frames table and palette, along with a couple of other values.

I go into more details in my notes, but, this first part that doesn't seems to have anything to do with the actual images data/frames is the part I just, can't figure out..

I've known about it for a long while, but I can't figure out what's going on in there

But feel free to focus on something else.

I just think that, having someone else take a look at it could maybe help solve that mystery!

[Blob55]

So is it possible to turn certain Pokémon Shiny? Like all female Pikachu as shiny.

[evandixon]

Looking at MAP_BG, I have absolutely no idea what the format for .bga and .bma is, and it's not possible to render the image without knowing that.

.bpl is the palette, the same format as in the block in .bgp, and .bpc contains the chunks/tiles (whichever term you prefer) in the same format as in the block in the .bgp.

As for the Pokémon sprite files, I can't tell where one block starts and another begins. Tomorrow, I'll take another look and re-read through your notes in case I missed something.

The wan files would possibly be easier to figure out if I had some idea of what it actually contained.

So is it possible to turn certain Pokémon Shiny? Like all female Pikachu as shiny.

Not quite yet. In PMD, shiny Pokémon are considered different Pokémon

Like Celebi from the future and the green Celebi featured later on, actually have different Pokémon ID's.

We currently don't know how to extract Pokémon sprites, let alone change them. But we're making progress.

[psy_commando]

So is it possible to turn certain Pokémon Shiny? Like all female Pikachu as shiny.

Not quite yet. In PMD, shiny Pokémon are considered different Pokémon

We currently don't know how to extract Pokémon sprites, let alone change them. But we're making progress.

Well, we can extract the sprites, I already did. I got code that can decode the individual rle encoded sprite frames and re-encode them easily.

Putting them back together into a sprite file is indeed a little more complicated.. Given we have to figure out the whole sprite file format to insert modified images/frames back into the sprite file without breaking anything..

Looking at MAP_BG, I have absolutely no idea what the format for .bga and .bma is, and it's not possible to render the image without knowing that.

.bpl is the palette, the same format as in the block in .bgp, and .bpc contains the chunks/tiles (whichever term you prefer) in the same format as in the block in the .bgp.

As for the Pokémon sprite files, I can't tell where one block starts and another begins. Tomorrow, I'll take another look and re-read through your notes in case I missed something.

The wan files would possibly be easier to figure out if I had some idea of what it actually contained.

I can take a look at those bga and bma, but if I'd have to guess, one of those might just be data on whether one tile can be walked on or not, or something like that.

The sprites use a very particular structure. You have to follow the pointers in the header and sub-header to find anything at all in there. Bookmarks and the highlighter tool are your friends. The data is very similar in many place, but yet not for the same purpose. The wan file have the exact same structure as the pokemon sprites, they only differ in content.

But, yeah.. The notes for the sprites are really a mess right now.. I wish I didn't have to use vague letters to identify every parts of the file.. If there's anything unclear, let me know and I'll try to fix it/explain it better.

If you want a quick idea of what the wan files contain, just run them through yy-chr or tiledggd, the images will be a little deformed, but you'll be able to see a little better what they contain. You can also locate and extract the palette, using my notes, and load it to get a slightly better result.

I could also write a quick little utility to export all frames from a sprite file? Most of the code is there, but I didn't really see the point until I could figure out how to extract the rest of the data.

I guess I'll just code that utility regardless.. I've been cleaning up my, early, sprite handling code anyways..

[psy_commando]

Alright, so most of the code works this far, but I can't find anything on Handling 16x32 sprites..

Reading those in YY-CHR with 16x32 as size seems to work, but when I try to emulate the 4x8 pattern it seems to use, I don't get good results at all..

Anybody knows of a good documentation on handling sprites like that ? Or maybe just have a tip to handle these ?

[evandixon]

Are you sure it's that pattern, and not square blocks? Perhaps there's some sort of positioning section like in .bgp, telling the relative location of each tile.

(Having not looked in a while, I don't know, just something to consider.)

[psy_commando]

Nevermind everything I said.. I just realized I mixed up a couple of things..

And the reason the 16x32 images weren't coming out right was because I put the wrong variable in the wrong place..

However, I found a new problem, it seems image resolution for sprites isn't determine by the image's byte/pixel total size alone. Because I stumbled on some 32x16 sprites, and they come out wrong..

If you want I can give you a copy of the program in its current flawed state, so you can try it out for yourself ? Because I can't figure how we're supposed to find info on the resolution..

[psy_commando]

Anyways, here's a not fully working version of the sprite extractor for those interested.

Once we find out a sure fire way of telling the images' resolution, I'll make an "official" non-broken release and put it in the first post !

It works just like the last utility I posted here. Pass an input path, and optionally an output folder. Or just drag and drop a sprite file contained inside an SIR0

(Those are currently extracted with the extension ".sir0" by my extractor(but a lot of non-sprite files can get a ".sir0" extension!), but elsewhere in the rom's file, they have a ".wan" file extension, and I think I even saw another with a different file extension.. But the utility will extract frames from those as well !)

Oh, and the images are exported as indexed 8bpp PNGs. And the palette is included in every resulting PNGs!

There you go:

If there are any bugs, and if anybody finds anything about the sprite files, please lets us know !

ppmd_sprex_broken.zip

[evandixon]

Good work! A few of the Bulbasaur are a little odd, but unless there's something like in .bgp that controls an x or y flip (which I don't think would help), it may or may not be garbage. It may be beneficial to look at this site if you haven't already. The misaligned sprites didn't seem to appear on that sheet.

The wan files in the FONT folder don't work with this tool, while in the GROUND folder, the aXXX.wan files store footprints (extracted as 4 images), and all the rest appear to be part of dungeons, which don't seem to make it all the way through this program, probably because of a slight file format variation (or something you haven't seen yet).

[psy_commando]

Good work! A few of the Bulbasaur are a little odd, but unless there's something like in .bgp that controls an x or y flip (which I don't think would help), it may or may not be garbage. It may be beneficial to look at this site if you haven't already. The misaligned sprites didn't seem to appear on that sheet.

Thanks, and yeah, those are most likely the ones that are wider than their height, and that's the reason why I named the thing "broken" because I currently have no way of figuring out the resolution, other than guessing from the total amount of pixels.

The wan files in the FONT folder don't work with this tool, while in the GROUND folder, the aXXX.wan files store footprints (extracted as 4 images), and all the rest appear to be part of dungeons, which don't seem to make it all the way through this program, probably because of a slight file format variation (or something you haven't seen yet).

That's an interesting find!

That means that maybe comparing those with the others sprite files that works might reveal what part of the file mentions format specific info !

EDIT:

I just thought about it, but, is there any particular error messages that pops up when the program stops midway ?

[evandixon]

because I currently have no way of figuring out the resolution, other than guessing from the total amount of pixels.

Looking at it again, that's the problem, it's only a few pixels off, so part of the sprite is cut off and put on the next line. Is SIR0 + Sprite in CommonFileFormats.txt up to date? If so, I can take a look and see if I can find anything.

I just thought about it, but, is there any particular error messages that pops up when the program stops midway ?

Nothing from within the program, just the generic "ppmd_sprex_broken.exe has stopped working...". This time I chose to debug it, and the error message is:

Unhandled exception at 0x00B5E74C in ppmd_sprex_broken.exe: 0xC0000094: Integer division by zero.

This is GROUND/v21p05a1.wan. GROUND/v39p01a1.wan had some tiles come out before crashing.

[psy_commando]

I should have included the ".pdb" before to make debugging easier I guess ^^;

Here's the version with the pdb / symbol file (its a zipped 7z, because zip couldn't make it small enough, and the forums doesn't like 7z files):

Its kinda odd you'd get a division by zero error like that.. It should crash way before it does any divisions.. D:

I'll have to look into it..

And yes, its more or less up to date.

But I mean, with the new findings, the meaning of a bunch of things in there could change radically ! So take anything in there not as the absolute truth.

EDIT:

Its funny, I can't find the file you've mentionned "GROUND/v21p05a1.wan"

I can only see "/GROUND/v21p02a1.wan". Are you using Explorers of Time / Darkness ?

EDIT2:

I traced back where it crashes. It was actually something I left in there to notice any unusual total image size.. Its because it stumbles on a 256 pixels image, and I never saw one of those inside a sprite file before.. So, we got another discovery

EDIT3:

I made a more graceful error handling:

https://dl.dropboxusercontent.com/u/13343993/my_pmd_utilities/tmp/Debug_SpriteEx.7z

Its compiled in debug currently, so its noticeably slower. It shouldn't die on 256 pixels sprites anymore with the changes I made.

It will throw an exception on never seen before images sizes !

ppmd_sprex_broken_debug.zip

Edited October 11, 2014 by psy_commando

[TruePikachu]

Meanwhile, I've been hacking away at decoding .smd files. Got quite a bit of work done, in fact. http://cdusto.selfip.com/pmdMus/smd_format.txt is my textual version of my notes on the format (based off psy_commando's work, but I have some changes and a bit more discovered). Additionally, I wrote up a couple of very-alpha tools in http://cdusto.selfip.com/pmdMus/pmd2mid-trunk.tar.gz - the dumper tool (smdDump) is pretty useful for looking over the files (and one can attempt to transcribe from them), while smd2mid, very much work-in-progress, somewhat manages to make .mid files (albeit with incorrect instruments) from .smd files.

There are still some bugs that need to be worked out, partially on the MIDI encoding side of things, partially on the SMD decoding side. I get desynchronization in BGM0003 (Marowak Dojo), and BGM0085 (I Don't Want To Say Goodbye)...well...

1:47 AM - TruePikachu: ........................................

1:47 AM - TruePikachu: oh....my....I might not be able to sleep now

1:47 AM - TruePikachu: I just tried BGM0085, "B_EVENT_SEPARAT"

1:48 AM - TruePikachu: one of the saddest pieces in the game

1:48 AM - TruePikachu: with the octave falling and wrong instruments, it sounded really creepy

1:48 AM - TruePikachu: /me shudders

Anyway, I might pull a friend into helping with figuring out some of the unknown commands (supplying said friend with .ogg and .smd versions of some songs, since they don't play PMD). They already managed to help quite a bit, finding problems in the MIDI files I was generating.

EDIT: Since not everyone will probably want to try to compile the tools, http://cdusto.selfip.com/bgm0003.mid should play in Windows Media Player, as proof-of-concept (and an older version of the tools)

[evandixon]

Meanwhile, I've been hacking away at decoding .smd files. Got quite a bit of work done, in fact. http://cdusto.selfip.com/pmdMus/smd_format.txt is my textual version of my notes on the format (based off psy_commando's work, but I have some changes and a bit more discovered). Additionally, I wrote up a couple of very-alpha tools in http://cdusto.selfip.com/pmdMus/pmd2mid-trunk.tar.gz - the dumper tool (smdDump) is pretty useful for looking over the files (and one can attempt to transcribe from them), while smd2mid, very much work-in-progress, somewhat manages to make .mid files (albeit with incorrect instruments) from .smd files.

There are still some bugs that need to be worked out, partially on the MIDI encoding side of things, partially on the SMD decoding side. I get desynchronization in BGM0003 (Marowak Dojo), and BGM0085 (I Don't Want To Say Goodbye)...well...

Anyway, I might pull a friend into helping with figuring out some of the unknown commands (supplying said friend with .ogg and .smd versions of some songs, since they don't play PMD). They already managed to help quite a bit, finding problems in the MIDI files I was generating.

EDIT: Since not everyone will probably want to try to compile the tools, http://cdusto.selfip.com/bgm0003.mid should play in Windows Media Player, as proof-of-concept (and an older version of the tools)

Good work! I've occasionally looked around the web to see if anyone did anything like this, and you're the first person to accomplish a (somewhat) working program in the many years the game's been out.

On some tracks, there's some sort of integer exception that causes one or two tracks to be locked to the same note, but this is great progress nonetheless. Given more time, I'm sure you'll improve it.

Its funny, I can't find the file you've mentionned "GROUND/v21p05a1.wan"

I can only see "/GROUND/v21p02a1.wan". Are you using Explorers of Time / Darkness ?

I must have mistyped that, I only see v21p02a1.wan now that I look at it.

Once again, good work everyone!

[TruePikachu]

Good work! I've occasionally looked around the web to see if anyone did anything like this, and you're the first person to accomplish a (somewhat) working program in the many years the game's been out.

On some tracks, there's some sort of integer exception that causes one or two tracks to be locked to the same note, but this is great progress nonetheless. Given more time, I'm sure you'll improve it.

If you are speaking about the .mid on my server, know that it does not reflect the current state of the program; IIRC it was working on the assumption that a NOTE_PLAY event had 1-2 extra bytes (and it would guess about the existance of the 2nd extra byte from the value), rather than the more-accurate checking the flag nibble for length setting. It might also not be taking into account what might have, at that time, been believed to be echo (actually volume).

As I said in my version of the specification, it is all thanks to psy_commando that I managed as much as I had. I have, in the past, reverse engineered sequencer formats for the GameBoy (partially by use of a disassembler), so I have that experience to help, but I don't want to even try (and I'm not sure my software even can) disassembling ARM assemblies, so the work here is pretty much just listen, guess, and check.

I actually came across this thread (and, by extension, the notes) while searching for a tool to dump the BGs from Sharpedo Bluff; I was trying to get a 3D model assembled in Blender, but eventually noticed that the three images that I had (beach view, exterior, and interior) don't actually match up...I blame space/time getting screwed up (which also explains the mysterious disappearing dorsal fin...). I think Google picked up on your message at the top of the 3rd page (*.rawimg), so you are partially to blame for progress

[psy_commando]

Meanwhile, I've been hacking away at decoding .smd files. Got quite a bit of work done, in fact. http://cdusto.selfip.com/pmdMus/smd_format.txt is my textual version of my notes on the format (based off psy_commando's work, but I have some changes and a bit more discovered). Additionally, I wrote up a couple of very-alpha tools in http://cdusto.selfip.com/pmdMus/pmd2mid-trunk.tar.gz - the dumper tool (smdDump) is pretty useful for looking over the files (and one can attempt to transcribe from them), while smd2mid, very much work-in-progress, somewhat manages to make .mid files (albeit with incorrect instruments) from .smd files.

There are still some bugs that need to be worked out, partially on the MIDI encoding side of things, partially on the SMD decoding side. I get desynchronization in BGM0003 (Marowak Dojo), and BGM0085 (I Don't Want To Say Goodbye)...well...

Anyway, I might pull a friend into helping with figuring out some of the unknown commands (supplying said friend with .ogg and .smd versions of some songs, since they don't play PMD). They already managed to help quite a bit, finding problems in the MIDI files I was generating.

EDIT: Since not everyone will probably want to try to compile the tools, http://cdusto.selfip.com/bgm0003.mid should play in Windows Media Player, as proof-of-concept (and an older version of the tools)

Nice work !

I was going to write a little something to attempt converting to midi. But, I was busy with the sprites

(Which I still have no ideas how to guess the resolution from those..)

Ironically though, yesterday I got some enlightenment on what a few messages codes meant, and on the delay between notes, and updated a little my notes, which were not very accurate ^^;

I'm under the impression that, the delay that can optionally be prefixed to events is probably dependent on the tempo. So that might be one of the issue you're having.

Also, if you want, we could probably split tasks, so we wouldn't all work on the same thing! Since you got smd well on its way, I could probably focus on the swd that contain the samples?

I made some nice finds on those recently !

Speaking of which, you wouldn't happen to know of a good library of re-usable code for decoding whatever format of ADPCM the NDS is using ? I tried writing my own algorithm, but, it only gives static.. I had some success exporting samples using Audacity's Dialogic/VOX ADPCM decoder to read the samples in PMD:EoS, but, I can't find code for any of that, and I'm pretty sure they didn't use Dialogic's specification for encoding those.. Probably something close, but not the same..

IIRC it was working on the assumption that a NOTE_PLAY event had 1-2 extra bytes (and it would guess about the existance of the 2nd extra byte from the value), rather than the more-accurate checking the flag nibble for length setting.

Yeah, I knew about it not guessing it just like that, but I didn't bother updating the notes until yesterday ironically ^^;

I noticed that the ( 0100 ) bit in the high nybble was always 1 when there were two parameters. Really sorry about that

At one point I was pretty much convinced nobody besides me would use those notes

It might also not be taking into account what might have, at that time, been believed to be echo (actually volume).

What do you mean though ?

0xE3 is for the channel's volume. While 0xE0 seems to have some echo/reverb-like effect on the notes.

I'm just asking, just to make sure I'm not missing something.

And would you mind if I'd update my notes with your findings, and your own notes ? I'd mention that you contributed of course !

As I said in my version of the specification, it is all thanks to psy_commando that I managed as much as I had. I have, in the past, reverse engineered sequencer formats for the GameBoy (partially by use of a disassembler), so I have that experience to help, but I don't want to even try (and I'm not sure my software even can) disassembling ARM assemblies, so the work here is pretty much just listen, guess, and check.

I actually came across this thread (and, by extension, the notes) while searching for a tool to dump the BGs from Sharpedo Bluff; I was trying to get a 3D model assembled in Blender, but eventually noticed that the three images that I had (beach view, exterior, and interior) don't actually match up...I blame space/time getting screwed up (which also explains the mysterious disappearing dorsal fin...). I think Google picked up on your message at the top of the 3rd page (*.rawimg), so you are partially to blame for progress

No problems

And if you try the PX extractor, you might be able to extract those BGs! You just need to find them in the game files first ^^;

Though, I think Nerketur could probably figure out what is the id for the scripts related to sharpedo buff, and in turn make it easy to find the associated BG.

Also, I don't recommend using a disassembler to figure out this stuff, well at least with PMD2. It would have been way more trouble than it would be worth it

What I did was simply look at the midi format, and think, "what do I need to play music in a sequencer?", basically.. I just looked at the features that midi offered, and looked for values that would fall within the same range in the smd file. And since most sequencers are event based, it was easy to guess it would be the same thing here ! Especially since they probably wrote the song in midi first before transferring it to the game. Which meant they'd logically want something that supports all common midi features!

And, if you want to guess what all events do, I strongly suggest taking an existing song in the game, removing all tracks but track 0 and another one. And just replace the content of the track that isn't track 0 (unless you really want to) manually with what you want to try, set the file length in the header, and set the track length in the track chunk, properly set the end padding of the track.

And there you go, your own poor man's sequencer !

I'd have tried everything out myself, but I wanted to focus on sprites, and other file format first. But its probably the best way to do, even though its a little time consuming.

I'll also say that, plugging in my midi keyboard and just looking at the events and trying to play what notes the track seemed to contain along with the actual music revealed a lot !

And I transcribed a few of them by ear, so it made things easier

https://dl.dropboxusercontent.com/u/13343993/MyMidis/idwtsg_49.ogg

https://dl.dropboxusercontent.com/u/13343993/MyMidis/otbad_6.mp3

https://dl.dropboxusercontent.com/u/13343993/MyMidis/Main%20Menu.ogg

Those are probably not 100% accurate, but if it can help, I could probably share the midis for those !

[evandixon]

I think Google picked up on your message at the top of the 3rd page (*.rawimg), so you are partially to blame for progress

Yay, I'm helping!

I actually came across this thread (and, by extension, the notes) while searching for a tool to dump the BGs from Sharpedo Bluff; I was trying to get a 3D model assembled in Blender, but eventually noticed that the three images that I had (beach view, exterior, and interior) don't actually match up...I blame space/time getting screwed up (which also explains the mysterious disappearing dorsal fin...).

I think that would be stored in one of the GROUND\vXXpXXaXX.wan files. psy_commando's sprite tool can extract some tiles from them, and will probably support them much better in the near future.

If you are speaking about the .mid on my server, know that it does not reflect the current state of the program; IIRC it was working on the assumption that a NOTE_PLAY event had 1-2 extra bytes (and it would guess about the existance of the 2nd extra byte from the value), rather than the more-accurate checking the flag nibble for length setting. It might also not be taking into account what might have, at that time, been believed to be echo (actually volume).

As I said in my version of the specification, it is all thanks to psy_commando that I managed as much as I had. I have, in the past, reverse engineered sequencer formats for the GameBoy (partially by use of a disassembler), so I have that experience to help, but I don't want to even try (and I'm not sure my software even can) disassembling ARM assemblies, so the work here is pretty much just listen, guess, and check.

I was actually talking about the source you just posted. I compiled it and besides line 41 of smdRead.cpp, which doesn't seem to be used anywhere, some tracks like Beach Cave and Drenched Bluff (bgm0021 and bgm0022) both had some sort of int exception (I didn't look too far into it, just let the program continue running, which resulted in the position on the track it failed on holding the same note until the end of the song, while the other tracks play.)

Out of curiosity, what compiler do you use? I'm not that familiar with C++ for the time being, and haven't found how to compile using the makefiles you included, I just put the cpp files into Visual Studio to let it compile them.