Pmd2 SIR0 - Revision history

UniqueGeek: Redirect

2017-10-14T21:25:35Z

Redirect

UniqueGeek at 00:08, 3 February 2017

2017-02-03T00:08:09Z

← Older revision		Revision as of 00:08, 3 February 2017
Line 2:		Line 2:
	[[Category:ROM Hacking]]		[[Category:ROM Hacking]]
	[[Category:Technical References]]		[[Category:Technical References]]
			[[Category:3DS Mystery Dungeon]]
			[[Category:NDS Mystery Dungeon]]
			[[Category:Game Data]]

	{{DISPLAYTITLE:SIR0/SIRO File Format}}		{{DISPLAYTITLE:SIR0/SIRO File Format}}
	The SIR0 format is a pretty common wrapper file format with as primary function wrapping other file formats. It provides a pointer to the format's "entry point", along with a list of file offsets to the pointers which need to be translated to NDS memory when the file is loaded.		The SIR0 format is a pretty common wrapper file format with as primary function wrapping other file formats. It provides a pointer to the format's "entry point", along with a list of file offsets to the pointers which need to be translated to NDS memory when the file is loaded.

Psy commando: Removed from the incomplete category. This is pretty much complete, there is nothing to add.

2015-08-26T04:29:33Z

Removed from the incomplete category. This is pretty much complete, there is nothing to add.

← Older revision		Revision as of 04:29, 26 August 2015
Line 2:		Line 2:
	[[Category:ROM Hacking]]		[[Category:ROM Hacking]]
	[[Category:Technical References]]		[[Category:Technical References]]
	~~[[Category:Incomplete Articles]]~~
	{{DISPLAYTITLE:SIR0/SIRO File Format}}		{{DISPLAYTITLE:SIR0/SIRO File Format}}
	The SIR0 format is a pretty common wrapper file format with as primary function wrapping other file formats. It provides a pointer to the format's "entry point", along with a list of file offsets to the pointers which need to be translated to NDS memory when the file is loaded.		The SIR0 format is a pretty common wrapper file format with as primary function wrapping other file formats. It provides a pointer to the format's "entry point", along with a list of file offsets to the pointers which need to be translated to NDS memory when the file is loaded.

Psy commando: Changed title, re-worded several things.

2015-03-22T19:15:27Z

Changed title, re-worded several things.

← Older revision		Revision as of 19:15, 22 March 2015
Line 3:		Line 3:
	[[Category:Technical References]]		[[Category:Technical References]]
	[[Category:Incomplete Articles]]		[[Category:Incomplete Articles]]
			{{DISPLAYTITLE:SIR0/SIRO File Format}}
			The SIR0 format is a pretty common wrapper file format with as primary function wrapping other file formats. It provides a pointer to the format's "entry point", along with a list of file offsets to the pointers which need to be translated to NDS memory when the file is loaded.

	~~The~~ SIR0 ~~format is a pretty common wrapper~~ file ~~format. Its also fairly simple.~~		When a SIR0 file gets loaded into memory, its magic number turns from SIR'''0''' to SIR'''O''', and all the pointers in the entire file are modified to be offset relative to the NDS's memory. The second pointer in the SIR header is also set to null when the file has been loaded and turned into a SIR'''O'''.
	~~When it~~ gets loaded into memory, its magic number turns from SIR'''0''' to SIR'''O''', and all the pointers in the entire file are modified to be offset relative to the NDS's memory.

	== File Structure ==		== File Structure ==
Line 99:		Line 100:
	Its a "compressed" list of all the offsets of all the pointers stored in the file. This includes both the SIR0 structure, and the contained structure.		Its a "compressed" list of all the offsets of all the pointers stored in the file. This includes both the SIR0 structure, and the contained structure.
	The game use this list to change the value of each pointers in the file after it has been loaded in memory, so they're relative to NDS memory.		The game use this list to change the value of each pointers in the file after it has been loaded in memory, so they're relative to NDS memory.
	The list will always begin with 04 04, as those are the encoded offsets of the 2 pointers in the SIR0 header.		The list will always begin with 04 04, as those are the encoded offsets of the 2 pointers in the SIR0 header. Each encoded offset values are relative to the previous offsets.

			==== Encoding ====

	If a byte has its highest bit (1000 0000) set to 1, then we have to "append" the next byte. Here are the 3 possible cases, using example values:		If a byte has its highest bit (1000 0000) set to 1, then we have to "append" the next byte. Here are the 3 possible cases, using example values:
Line 105:		Line 108:
	0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12		0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12
	0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06		0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06
	Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also note that~~, while decoding~~, we get rid of the highest bit's value using the bitmask 0x7F!		Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also note that, we always get rid of the highest bit's value using the bitmask 0x7F, after checking if its toggled on!

	If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F (0111 1111) bitmask.		If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F (0111 1111) bitmask.

Psy commando: /* Pointer Offsets List */

2015-02-16T19:35:32Z

Pointer Offsets List

← Older revision		Revision as of 19:35, 16 February 2015
Line 188:		Line 188:
	using namespace std;		using namespace std;
	//...		//...

	void EncodeSIR0PtrOffsetList( const std::vector<uint32_t> &listoffsetptrs, std::vector<uint8_t> & out_encoded )		void EncodeSIR0PtrOffsetList( const std::vector<uint32_t> &listoffsetptrs, std::vector<uint8_t> & out_encoded )
	{		{
	uint32_t offsetSoFar = 0; //used to add up the sum of all the offsets up to the current one		uint32_t offsetSoFar = 0; //used to add up the sum of all the offsets up to the current one

	for( const auto & anoffset : listoffsetptrs )		for( const auto & anoffset : listoffsetptrs )
	{		{
	uint32_t offsetToEncode = anoffset - offsetSoFar;		uint32_t offsetToEncode = anoffset - offsetSoFar;
	bool hasHigherNonZero = false; //This tells the loop whether it needs to encode null bytes, if at least one higher byte was non-zero		bool hasHigherNonZero = false; //This tells the loop whether it needs to encode null bytes, if at least one higher byte was non-zero
	offsetSoFar = anoffset; //set the value to the latest offset, so we can properly subtract it from the next offset.		offsetSoFar = anoffset; //set the value to the latest offset, so we can properly subtract it from the next offset.

	//Encode every bytes of the 4 bytes integer we have to		//Encode every bytes of the 4 bytes integer we have to
	for( int32_t i = 4; i > 0; --i )		for( int32_t i = 4; i > 0; --i )
	{		{
	uint8_t currentbyte = ( offsetToEncode >> (7 * (i - 1)) ) & 0x7Fu;		uint8_t currentbyte = ( offsetToEncode >> (7 * (i - 1)) ) & 0x7Fu;

	if( i == 1 ) //the lowest byte to encode is special		if( i == 1 ) //the lowest byte to encode is special
	{		{
Line 217:		Line 217:
	}		}
	}		}

	//Append the closing 0		//Append the closing 0
	out_encoded.push_back(0);		out_encoded.push_back(0);
	}		}
	</code>		</code>

Psy commando: /* Pointer Offsets List */ Fixed a serious mistake I only figured out 2 days ago. No zeros are dropped if there's at least a 0x80 flagged byte before. And a zero byte only ends the list when its not right after a 0x80 flagged byte!

2015-02-16T19:34:43Z

Pointer Offsets List: Fixed a serious mistake I only figured out 2 days ago. No zeros are dropped if there's at least a 0x80 flagged byte before. And a zero byte only ends the list when its not right after a 0x80 flagged byte!

← Older revision		Revision as of 19:34, 16 February 2015
Line 100:		Line 100:
	The game use this list to change the value of each pointers in the file after it has been loaded in memory, so they're relative to NDS memory.		The game use this list to change the value of each pointers in the file after it has been loaded in memory, so they're relative to NDS memory.
	The list will always begin with 04 04, as those are the encoded offsets of the 2 pointers in the SIR0 header.		The list will always begin with 04 04, as those are the encoded offsets of the 2 pointers in the SIR0 header.

	~~An encoded byte with a value of 0 indicates the end of the list.~~

	If a byte has its highest bit (1000 0000) set to 1, then we have to "append" the next byte. Here are the 3 possible cases, using example values:		If a byte has its highest bit (1000 0000) set to 1, then we have to "append" the next byte. Here are the 3 possible cases, using example values:
Line 107:		Line 105:
	0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12		0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12
	0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06		0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06
	Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also note that, we get rid of the highest bit's value using the bitmask 0x7F!		Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also note that, while decoding, we get rid of the highest bit's value using the bitmask 0x7F!

	If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F (0111 1111) bitmask.		If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F (0111 1111) bitmask.

	Also, each times we ~~compute~~ an offset using either of the above, we add its value to the sum of all the previous offsets to get the actual offset.		An encoded byte with a value of 0 indicates the end of the list, but only when the byte that came before didn't have its highest bit set to 1 (1000 0000).

			Also, each times we decode an offset using either of the above, we add its value to the sum of all the previous offsets to get the actual offset.
	This is why the list starts with 04 04, and not 04 08 for example. Because they're added to each others as we process them.		This is why the list starts with 04 04, and not 04 08 for example. Because they're added to each others as we process them.

	~~However consider this case:~~

	~~This a valid encoded value:~~
	~~0xCC 0x80 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7~~
	~~It results in 0x130000 once decoded. Note how there are only 2 encoded bytes, but the decoded integer spans 3 bytes.~~

	~~And if the last byte's content of that base value were to change, it would end up being encoded this way:~~
	~~0xCC 0x80 0x01 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7 \| 0x01~~
	~~The encoded value is stored on 3 bytes. And it results in 0x130001, which is 1 more than the value above, that was encoded on only 2 bytes !~~


	'''Example:'''		'''Example:'''
Line 140:		Line 129:
	0x914		0x914
	0x928		0x928

			Those are the offsets relative to the beginning of the file "/FONT/frame0.wte", where pointers are stored.

	'''Some Code:'''<br/>		'''Some Code:'''<br/>
Line 153:		Line 144:
	{		{
	vector<uint32_t> decodedptroffsets( ptroffsetslst.size() ); //worst case scenario		vector<uint32_t> decodedptroffsets( ptroffsetslst.size() ); //worst case scenario
	auto itcurbyte = ptroffsetslst.begin();		auto itcurbyte = ptroffsetslst.begin();
	auto itlastbyte = ptroffsetslst.end();		auto itlastbyte = ptroffsetslst.end();
	uint32_t offsetsum = 0; //This is used to sum up all offsets and obtain the offset relative to the file, and not the last offset		uint32_t offsetsum = 0; //This is used to sum up all offsets and obtain the offset relative to the file, and not the last offset
	uint32_t buffer = 0; //temp buffer to assemble longer offsets		uint32_t buffer = 0; //temp buffer to assemble longer offsets
	uint8_t curbyte = *itcurbyte;		uint8_t curbyte = *itcurbyte;
			bool LastHadBitFlag = false; //This contains whether the byte read on the previous turn of the loop had the bit flag indicating to append the next byte!
	decodedptroffsets.resize(0); //preserve alloc, and allow pushbacks		decodedptroffsets.resize(0); //preserve alloc, and allow pushbacks

	while( itcurbyte != itlastbyte && (~~curbyte =~~ *itcurbyte) != 0 )		while( itcurbyte != itlastbyte && ( LastHadBitFlag \|\| (*itcurbyte) != 0 ) )
	{		{
			curbyte = *itcurbyte;
	//Ignore the first bit, using the 0x7F bitmask, as its reserved. And append or assign the next byte's value to the buffer.		//Ignore the first bit, using the 0x7F bitmask, as its reserved. And append or assign the next byte's value to the buffer.
	buffer \|= curbyte & 0x7Fu;		buffer \|= curbyte & 0x7Fu;
Line 167:		Line 160:
	if( (0x80u & curbyte) != 0 )		if( (0x80u & curbyte) != 0 )
	{		{
			LastHadBitFlag = true;
	//If first bit is 1, bitshift left the current buffer, to append the next byte.		//If first bit is 1, bitshift left the current buffer, to append the next byte.
	buffer <<= 7u;		buffer <<= 7u;
Line 172:		Line 166:
	else		else
	{		{
			LastHadBitFlag = false;
	//If we don't need to append, add the value of the current buffer to the offset sum this far,		//If we don't need to append, add the value of the current buffer to the offset sum this far,
	// and add that value to the output vector. Then clear the buffer.		// and add that value to the output vector. Then clear the buffer.
Line 197:		Line 192:
	{		{
	uint32_t offsetSoFar = 0; //used to add up the sum of all the offsets up to the current one		uint32_t offsetSoFar = 0; //used to add up the sum of all the offsets up to the current one

	for( const auto & anoffset : listoffsetptrs )		for( const auto & anoffset : listoffsetptrs )
	{		{
	uint32_t offsetToEncode = anoffset - offsetSoFar;		uint32_t offsetToEncode = anoffset - offsetSoFar;
	bool hasHigherNonZero = false; //This tells the loop whether it needs to encode null bytes,		bool hasHigherNonZero = false; //This tells the loop whether it needs to encode null bytes, if at least one higher byte was non-zero
	//if at least one higher byte was non-zero
	offsetSoFar = anoffset; //set the value to the latest offset, so we can properly subtract it from the next offset.		offsetSoFar = anoffset; //set the value to the latest offset, so we can properly subtract it from the next offset.

	//Encode every bytes of the 4 bytes integer we have to		//Encode every bytes of the 4 bytes integer we have to
	for( int32_t i = 4; i > 0; --i )		for( int32_t i = 4; i > 0; --i )
Line 213:		Line 207:
	{		{
	//If its the last byte to append, leave the highest bit to 0 !		//If its the last byte to append, leave the highest bit to 0 !
	~~if( currentbyte != 0 )~~		out_encoded.push_back( currentbyte );
	out_encoded.push_back( currentbyte );
	~~//If the last byte to append is null, we don't need to append anything~~
	~~// as the automatic bitshift of the last byte will take care of that if a higher byte was non-zero.~~
	~~// In any other cases, null pointers are not to be messed with.~~
	}		}
	else if( currentbyte != 0 \|\| hasHigherNonZero ) //if any but lowest byte!		else if( currentbyte != 0 \|\| hasHigherNonZero ) //if any bytes but the lowest one! If not null OR if we have encoded a higher non-null byte before!
	{		{
	//Set the highest bit to 1, to ~~signify~~ that the next byte must be appended		//Set the highest bit to 1, to signifie that the next byte must be appended
	out_encoded.push_back( currentbyte \| 0x80u );		out_encoded.push_back( currentbyte \| 0x80u );
	hasHigherNonZero = true;		hasHigherNonZero = true;
Line 227:		Line 217:
	}		}
	}		}

	//Append the closing 0		//Append the closing 0
	out_encoded.push_back(0);		out_encoded.push_back(0);
	}		}
	</code>		</code>

Psy commando: Forgot to update the table.

2015-01-25T08:00:05Z

Forgot to update the table.

← Older revision		Revision as of 08:00, 25 January 2015
Line 47:		Line 47:
	\| A list containing the offsets to every pointers in the entire file.		\| A list containing the offsets to every pointers in the entire file.
	\|-		\|-
	\| After ~~End of File Data Block~~		\| After Pointer Offsets List
	\| Varies		\| Varies
	\|		\|
Line 84:		Line 84:
	\| little		\| little
	\| uint32		\| uint32
	\| Pointer to ~~End of File Data~~		\| Pointer to Pointer Offsets List
	\| A pointer to ~~a block of data~~ located after the contained data.		\| A pointer to the [[#Pointer Offsets List\|Pointer Offsets List]] located after the contained data.
	\|-		\|-
	\| 0x0C		\| 0x0C

Psy commando: /* Pointer Offsets List */ reworded and added a few missing details.

2015-01-19T20:34:54Z

Pointer Offsets List: reworded and added a few missing details.

Psy commando: reworded

2015-01-19T20:29:11Z

reworded

← Older revision		Revision as of 20:29, 19 January 2015
Line 107:		Line 107:
	0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12		0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12
	0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06		0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06
	Note that, since the ~~size~~ are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also not that, we get rid of the highest bit's value using the bitmask 0x7F!		Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also not that, we get rid of the highest bit's value using the bitmask 0x7F!

	If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F bitmask.		If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F bitmask.

Psy commando: /* Pointer Offsets List */ Re-worded and added mention of a special case that is easy to miss.

2015-01-07T23:32:56Z

Pointer Offsets List: Re-worded and added mention of a special case that is easy to miss.

← Older revision		Revision as of 23:32, 7 January 2015
Line 97:		Line 97:
	=== Pointer Offsets List ===		=== Pointer Offsets List ===
	This list is what makes the SIR0 container what it is.		This list is what makes the SIR0 container what it is.
	Its a "compressed" list of all the offsets of all pointers in the ~~files~~. ~~Both~~ the SIR0 structure, and the contained structure.		Its a "compressed" list of all the offsets of all the pointers stored in the file. This includes both the SIR0 structure, and the contained structure.
	The game use this list to change the value of each pointers in the file after it has been loaded in memory, so they're relative to NDS memory.		The game use this list to change the value of each pointers in the file after it has been loaded in memory, so they're relative to NDS memory.
	The list will always begin with 04 04, as those are the offsets of the 2 pointers in the SIR0 header.		The list will always begin with 04 04, as those are the encoded offsets of the 2 pointers in the SIR0 header.

	A byte with a value of 0 indicates the end of the list.		An encoded byte with a value of 0 indicates the end of the list.

	If a byte has its highest bit (1000 0000) set to 1, then we have to "append" the next byte. Here are the possible cases, using example values:		If a byte has its highest bit (1000 0000) set to 1, then we have to "append" the next byte. Here are the 3 possible cases, using example values:
	0x80 0x81 0x82 0x75 => (0x80 & 0x7F) << 21 \| (0x81 & 0x7F) << 14 \| (0x82 & 0x7F) << 7 \| 0x75		0x80 0x81 0x82 0x75 => (0x80 & 0x7F) << 21 \| (0x81 & 0x7F) << 14 \| (0x82 & 0x7F) << 7 \| 0x75
	0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12		0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12
Line 111:		Line 111:
	If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F bitmask.		If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F bitmask.

	Also, each times we compute an offset using the above, we add its value to the sum of all the previous offsets to get the actual offset.		Also, each times we compute an offset using either of the above, we add its value to the sum of all the previous offsets to get the actual offset.
	This is why the list starts with 04 04, and not 04 08 for example. Because they're added to each others as we process them.		This is why the list starts with 04 04, and not 04 08 for example. Because they're added to each others as we process them.

			However consider this:
			This a valid encoded value:
			0xCC 0x80 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7
			It results in 0x130000 once decoded. Note how there are only 2 encoded bytes, but the decoded integer "uses" 3 bytes.

			And if the last byte's content of that base value were to change, it would end up being encoded this way:
			0xCC 0x80 0x01 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7 \| 0x01
			And this results in 0x130001 !


	'''Example:'''		'''Example:'''

← Older revision		Revision as of 20:34, 19 January 2015
Line 107:		Line 107:
	0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12		0x80 0x81 0x12 => (0x80 & 0x7F) << 14 \| (0x81 & 0x7F) << 7 \| 0x12
	0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06		0x80 0x06 => (0x80 & 0x7F) << 7 \| 0x06
	Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also ~~not~~ that, we get rid of the highest bit's value using the bitmask 0x7F!		Note that, since the offsets are stored as 32 bits integer, chaining more than 4 bytes is impossible. Also note that, we get rid of the highest bit's value using the bitmask 0x7F!

	If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F bitmask.		If the byte's highest bit (1000 0000) is set to 0, we use the byte as is, still applying the 0x7F (0111 1111) bitmask.

	Also, each times we compute an offset using either of the above, we add its value to the sum of all the previous offsets to get the actual offset.		Also, each times we compute an offset using either of the above, we add its value to the sum of all the previous offsets to get the actual offset.
	This is why the list starts with 04 04, and not 04 08 for example. Because they're added to each others as we process them.		This is why the list starts with 04 04, and not 04 08 for example. Because they're added to each others as we process them.

	However consider this:		However consider this case:

	This a valid encoded value:		This a valid encoded value:
	0xCC 0x80 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7		0xCC 0x80 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7
	It results in 0x130000 once decoded. Note how there are only 2 encoded bytes, but the decoded integer ~~"uses"~~ 3 bytes.		It results in 0x130000 once decoded. Note how there are only 2 encoded bytes, but the decoded integer spans 3 bytes.

	And if the last byte's content of that base value were to change, it would end up being encoded this way:		And if the last byte's content of that base value were to change, it would end up being encoded this way:
	0xCC 0x80 0x01 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7 \| 0x01		0xCC 0x80 0x01 => (0xCC & 0x7F) << 14 \| (0x80 & 0x7F) << 7 \| 0x01
	And ~~this~~ results in 0x130001 !		The encoded value is stored on 3 bytes. And it results in 0x130001, which is 1 more than the value above, that was encoded on only 2 bytes !