collector {dt.:Sammler}

MS SQL generated XML – cleanup and sanitize

All recent versions of MS SQL are able to generate the XML representation of a SQL query result on the fly:

>select * from dbo.foo
id  foo
1  dead
2  beef

>select * from dbo.foo for XML RAW, ELEMENTS

<row>
  <id>1</id>
  <foo>dead</foo>
</row>
<row>
  <id>2</id>
  <foo>beef</foo>
</row>

This is nothing new and special (in the age of NO-SQL and document centric databases) and works well. If you go to use the SQL server as a data source in a business integration orchestration the XML representation of the SQL results turns the mapping of the data between different entities into a point and click game…

While playing with the example databases of Microsoft NAV I recognised some funny things.

UTF-16 conversion of column names

Some smart guy decided to use spaces and braces within the column names. To turn these column names into valid XML element names Microsoft decided to convert these characters into their UTF-16 representation surrounded by underscores. By applying this rule a column named ‚Ali Baba‘ will turn into ‚Ali_x0020_Baba‘. It turned out that the conversation is applied to nearly every special character used within column names:

<test.dbo.foo xmlns:colon="uri">
  <space_x0020_sign>1</space_x0020_sign>
  <exclamation_x0021_sign>1</exclamation_x0021_sign>
  <doublequote_x0022_sign>1</doublequote_x0022_sign>
  <hash_x0023_sign>1</hash_x0023_sign>
  <dollar_x0024_sign>1</dollar_x0024_sign>
  <percent_x0025_sign>1</percent_x0025_sign>
  <amp_x0026_sign>1</amp_x0026_sign>
  <singlequote_x0027_sign>1</singlequote_x0027_sign>
  <openparentheses_x0028_sign>1</openparentheses_x0028_sign>
  <closingparentheses_x0029_sign>1</closingparentheses_x0029_sign>
  <mul_x002A_sign>1</mul_x002A_sign>
  <plus_x002B_sign>1</plus_x002B_sign>
  <comma_x002C_sign>1</comma_x002C_sign>
  <minus-sign>1</minus-sign>
  <dot.sign>1</dot.sign>
  <slash_x002F_sign>1</slash_x002F_sign>
  <colon:sign>1</colon:sign>
  <semicolon_x003B_sign>1</semicolon_x003B_sign>
  <lesserthen_x003C_sign>1</lesserthen_x003C_sign>
  <equal_x003D_sign>1</equal_x003D_sign>
  <biggerthe_x003E_sign>1</biggerthe_x003E_sign>
  <questionmark_x003F_sign>1</questionmark_x003F_sign>
  <at_x0040_sign>1</at_x0040_sign>
  <backslash_x005C_sign>1</backslash_x005C_sign>
  <caret_x005E_sign>1</caret_x005E_sign>
  <underscore_sign>1</underscore_sign>
  <pipe_x007C_sign>1</pipe_x007C_sign>
  <opencurly_x007B_sign>1</opencurly_x007B_sign>
  <closingcurly_x007D_sign>1</closingcurly_x007D_sign>
  <tilde_x007E_sign>1</tilde_x007E_sign>
</test.dbo.foo>

This is a minor problem cause it just creates ugly element names. If the data is only exported (and no ‚translation‘ back in the database is needed), these encoded entities can be replaced with anything looking … better – like the underscore. This is just for cosmetic purposes – nothing a machine will take a notice of…

IF OBJECT_ID (N'dbo.cleanXMLElements', N'FN') IS NOT NULL
    DROP FUNCTION dbo.cleanXMLElements
GO

-- replace any occurence of _x00<VALUE@REPLACE>_ with the string given by the second argument
-- warning: this is a pure pattern matching - if the sequence _x00<VALUE@REPLACE>_ is part of your payload - good luck!
-- note: to increase performance - remove any values from @REPLACE not needed
-- param_0: xml document
-- param_1: replacement string
-- return: cleaned xml document
CREATE FUNCTION dbo.cleanXMLElements (@XMLDoc XML, @REPLACE_WIDTH VARCHAR(10))
RETURNS xml
AS
BEGIN
  DECLARE @XMLString NVARCHAR(MAX)
  DECLARE @REPLACE VARCHAR(100)
  -- hex-values of elements that should be replaced, match is done on _x00<value>_
  SET @REPLACE = '20,21,22,23,24,25,26,27,28,29,2a,2b,2c,2f,3b,3c,3d,3e,3f,40,5c,5e,7c,7b,7d,7e'
  SET @XMLString = CAST(@XMLDoc as NVARCHAR(MAX))

  WHILE LEN(@REPLACE) > 0
  BEGIN
    SET @XMLString = REPLACE(@XMLString, '_x00' + LEFT(@REPLACE, CHARINDEX(',', @REPLACE + ',')-1) + '_', @REPLACE_WIDTH)
    SET @REPLACE = STUFF(@REPLACE, 1, CHARINDEX(',', @REPLACE + ','), '')
  END
  RETURN CAST(@XMLString as XML)
END
GO

XML name spaces

If a column name contains a colon the XML export will fail:

>SELECT * FROM dbo.foo for XML AUTO, ELEMENTS, TYPE

Msg 6846, Level 16, State 1, Line 4
XML name space prefix 'colon' declaration is missing for FOR XML column name 'colon:sign'.

In XML the construct A:B defines B as an element living in the name space of A (here: sign is part of the name space colon). Since the generated XML representation lacks the needed name space definition so the document is not well formed…

Unhappily it is very likely that the colon is part of the values contained in the query/document so the simple approach to just replace/strip all colons is very error prone. It is also not possible to generate the name space definition on the fly (eg. by parsing the table definition). The only remaining solution for that kind of problem is to add the needed name space definition manually:

WITH XMLNAMESPACES ('uri' as colon)
SELECT * FROM dbo.foo for XML AUTO, ELEMENTS, TYPE

This eliminates the option to export large databases automatically – if there is a colon in a column name. Of course its still possible to generate the needed code…

invalid characters in XML document

Back to the NAV example database… I tried to insert the rows of the Item table as XML into another table (trigger based logging) – and the command failed caused by some illegal characters contained into the values of some fields (example):

USE test
GO

IF OBJECT_ID (N'dbo.log', N'U') IS NOT NULL
    DROP TABLE dbo.log
GO

CREATE TABLE dbo.log (
  id int IDENTITY(1,1),
  tab_name VARCHAR(20),
  data XML
);
GO
insert into dbo.log values('Item', (SELECT * FROM [Demo Database NAV (8-0)].[dbo].[CRONUS AG$Item] for xml raw, elements, BINARY BASE64))

Msg 9420, Level 16, State 1, Line 10
XML parsing: line 1, character 5458, illegal xml character

I dumped the XML and found &#x04 at the position given in the error message. The ASCII code 0x04 represents the special character EOT – End Of Transmission (I have no idea how the guy that created the example table was able to put that special char into the table data). Since 0x04 is not allowed in XML documents the conversion fails. Maybe I’m to demanding – but why the hell is the freaking tool that converts the SQL result to its XML representation not able to handle/filter that kind of special char? Hu? Have a look at http://www.w3.org/TR/xml/#charsets to get an idea what is allowed – and what not.

To workaround this misbehaviour just remove these illegal characters from the XML document before any further processing. Since the generated XML is not valid it is not possible to handover the converted SQL result directly to the sanitizer function so an explicit cast to nvarchar is needed here.

USE test
GO

IF OBJECT_ID (N'dbo.sanitizeXML', N'FN') IS NOT NULL
    DROP FUNCTION dbo.sanitizeXML
GO

-- remove any occurrence of &#x<VALUE@REMOVE>;
-- warning: this is a pure pattern matching - if the sequence &#x<VALUE@REMOVE> is part of your payload - good luck!
-- note: to increase performance - remove any values from @REMOVE not needed
-- param_0: text to be sanitized
-- return: sanitized string
--CREATE FUNCTION dbo.sanitizeXML (@XMLDoc XML)
CREATE FUNCTION dbo.sanitizeXML (@XMLString NVARCHAR(max))
RETURNS XML
AS
BEGIN
  --DECLARE @XMLString NVARCHAR(MAX)
  DECLARE @REMOVE VARCHAR(100)
  -- hex-values of elements that should be removed, match is done on &#x<value>;
  -- 0x05 -> ENQ
  -- 0x04 -> EOT
  -- full set: 00,01,02,03,04,05,06,07,08,0b,0c,0e,0f,10,11,12,13,14,15,16,17,18,19,1a,1b,1c,1d,1e,1f
  SET @REMOVE = '04,05'

  WHILE LEN(@REMOVE) > 0
  BEGIN
    SET @XMLString = REPLACE(@XMLString, '&#x' + LEFT(@REMOVE, CHARINDEX(',', @REMOVE + ',') - 1) + ';', '')
    SET @REMOVE = STUFF(@REMOVE, 1, CHARINDEX(',', @REMOVE + ','), '')
  END
  RETURN CAST(@XMLString as XML)
END
GO

insert into dbo.log (tab_name, data) values('Item', (select test.dbo.sanitizeXML(CAST((SELECT * FROM [Demo Database NAV (8-0)].[dbo].[CRONUS AG$Item] for xml raw, elements, BINARY BASE64) AS NVARCHAR(MAX)))))

For performance reasons the sanitizeXML currently only handles the special chars I have to deal with. In theory every char from 0x00 up to 0x1f (except 0x09, 0x0a, 0x0d) must be removed.

Scripts and examples: sanitize_and_cleanup_xml_sql.zip

collected: own shiny kernel on the tolino shine

The (patched) kernel used in the Tolino Shine was made available (by accident) a few weeks ago. I mirrored the archive here:

md5	size/mb	file
90e78f2f7fdfffe460c85651cae431a3	109	kernel.tar.gz

Extract the archive, export the path to the toolchain (if not done yet) and compile the kernel:

>export PATH=$PATH:/home/devel/projects/tolino_shine/own_uboot/kobo/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/
>make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf-

U-boot uses a little header in front of images (kernel, initrd, …) to store informations about the content (so it knows how to handle the image data). Normally you could compile the kernel directly as u-boot compatible image by using the make target uImage – but lets do that by hand to see how it works (we need that u-boot header thingy again when it comes down to the initrd – so it is not about wasting time, won’t hurt and is done with one line… 🙂 ).

_note: The mkimage tool is located in the tools folder of the u-boot sources so we need to export that path.

>export PATH=$PATH:/home/devel/projects/tolino_shine/own_uboot/uboot-imx/tools/
>mkimage -A arm -O linux -T kernel -C none -a 0x70008000 -e 0x70008000 -n "Shine Kernel" -d zImage shine_kernel.uboot
Image Name:   Shine Kernel
Created:      Sun Jun 23 16:13:38 2013
Image Type:   ARM Linux Kernel Image (uncompressed)
Data Size:    3275528 Bytes = 3198.76 kB = 3.12 MB
Load Address: 70008000
Entry Point:  70800000

_note: I took the address parameters -a (load address) and -e (execute address) from the original u-boot.

From ntx_comm.c (#define SD_OFFSET_SECS_KERNEL 2048) and the boot log (MMC read: dev # 0, block # 2048, count 8192 partition # 0 ) we know that the boot-loader expects the kernel at 0x800*0x200 = 0x100000 on the SD card.

_note: the #define DEFAULT_LOAD_KERNEL_SZ 8192 sets the maximum for the kernel size (here 4MB)

So lets write our u-boot header equipped kernel to that position on the disk…

#>dd if=shine_kernel.uboot of=/dev/SDCARD seek=2048 bs=512

It boots… and crashes:

MMC read: dev # 0, block # 1, count 1 partition # 0 ... 
1 blocks read: OK
hwcfg rootfstype : 2
hwcfg partition type : 7
## Booting kernel from Legacy Image at 70800000 ...
   Image Name:   Shine Kernel
   Created:      2013-06-23  16:05:55 UTC
   Image Type:   ARM Linux Kernel Image (uncompressed)
   Data Size:    3275528 Bytes =  3.1 MB
   Load Address: 70008000
   Entry Point:  70008000
   Loading Kernel Image ... OK
OK
-
Starting kernel ...
-
Uncompressing Linux... done, booting the kernel.
Initializing cgroup subsys cpu
Linux version 2.6.35.3 (devel@bigplay) (gcc version 4.8.1 20130401 (prerelease) (crosstool-NG linaro-1.13.1-4.8-2013.04-20130417 - Linaro GCC 2013.04) ) #109 PREEMPT Sun Jun 23 00:15:35 CEST 2013
CPU: ARMv7 Processor [412fc085] revision 5 (ARMv7), cr=10c53c7
..................
mxc_epdc_fb_fw_handler-3619] 
drivers/video/mxc/mxc_epdc_fb.c(3654):EPD 1024x758 
Unable to handle kernel NULL pointer dereference at virtual address 00000026
pgd = 80004000
[00000026] *pgd=00000000
Internal error: Oops: 5 [#1] PREEMPT
last sysfs file: 
Modules linked in:
CPU: 0    Not tainted  (2.6.35.3 #109)
PC is at epdc_firmware_func+0x10c/0x2a4
LR is at epdc_firmware_func+0x68/0x2a4
................

The last regular output prints a file (drivers/video/mxc/mxc_epdc_fb.c) and a line number (3654). A few lines later (3694) some data is copied from waveform… remember the u-boot output @ boot:

...........
MMC read: dev # 0, block # 14335, count 1 partition # 0 ... 
1 blocks read: OK
no "waveform" bin header
...........

Yeah, it is missing- and the code will crash if it is not there. Learned: it seems the waveform contains the frame buffer configuration. Same procedure as every year: extract the waveform image from the SD card backup and put it on our new card… The header including the magic number starts at 0x6ffff0. The size is given at mem(magic number + 8) -> (AF 66 11 00). Interpreted as an unsigned long we got a size of 1140399 bytes (but we copy 1140431 := 1140399 + 32 byte header). The waveform data starts at 0x700000. Copy now!

>dd if=first_16_mb_sd_card.img of=waveform.img bs=1 skip=7340016 count=1140431
#>dd if=waveform.img of=/dev/SDCARD bs=1 seek=7340016 obs=512

md5	size/mb	file
7d440e0d24b2ee89c280d83ecf55452d	1	waveform.img

Insert the card, cross the fingers – boot… works :). On boot the display is initialized – the screen turns black for two times. But now the kernel dies with:

VFS: Cannot open root device "mmcblk0p1" or unknown-block(179,1)
Please append a correct "root=" boot option; here are the available partitions:
b300         1966080 mmcblk0 driver: mmcblk
Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(179,1)
[<8003d250>] (unwind_backtrace+0x0/0xe0) from [<8046c620>] (panic+0x68/0xd8)
[<8046c620>] (panic+0x68/0xd8) from [<80008e50>] (mount_block_root+0x1c0/0x200)
[<80008e50>] (mount_block_root+0x1c0/0x200) from [<800090c0>] (prepare_namespace+0x128/0x17c)
[<800090c0>] (prepare_namespace+0x128/0x17c) from [<80008aac>] (kernel_init+0x120/0x17c)
[<80008aac>] (kernel_init+0x120/0x17c) from [<80039954>] (kernel_thread_exit+0x0/0x8)

But that is okay and will be fixed in the next part. (Complete boot log including kernel: here)

collected: build u-boot for Tolino Shine

I took the u-boot sources for imx507 from the official repo of the KOBO reader at git://github.com/kobolabs/Kobo-Reader.git cause I was unable to find a working u-boot repository that contains the mx50/imx507. Hey Deutsche Telekom, you should publish something… The u-boot source is located in Kobo-Reader/hw/imx507/u-boot-2009.08.tar.gz. To build the boot-loader you also need a matching tool-chain – happily you can find one also in the KOBO repo – this time in Kobo-Reader/toolchain/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux.tar.bz2. You could get both archives here (cause the full blown KOBO-Reader-Repo is currently about 3.5G – and you only need these two parts to start):

md5	size/mb	file
244920ee3a01d3f8cd187d41ea06a718	14	u-boot-2009.08.tar.gz
998fb1f00b5f7eaea1cc2491960892fb	80	gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux.tar.bz2

Copy both archives to a common directory and then:

>tar xzvf u-boot-2009.08.tar.gz
>tar xjvf gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux.tar.bz2
>export PATH=$PATH:/home/devel/projects/tolino_shine/own_uboot/kobo/gcc-linaro-arm-linux-gnueabihf-4.8-2013.04-20130417_linux/bin/

_note 0: replace the exported path with yours!
_note 1a: if you use the files from the git repo – change the name of the used toolchain/gcc-prefix in u-boot-2009.08/build_mddr_256.sh and u-boot-2009.08/build.sh from arm-fsl-gnueabi- to arm-linux-gnueabihf-
_note 1b: The names are already changed in archives linked above.
_note2: For running the build_mddr_256.sh the program dialog (console based dialogs & menus) must be installed on your system.

Now you are ready to build:

>cd u-boot-2009.08/
>./u-boot-2009.08/build_mddr_256.sh

and choose E60610 board, K4X2G323PD – Go! The build process creates a file named u-boot_mddr_256-E60610-K4X2G323PD.bin (just the renamed u-boot.bin) – thats the boot-loader. Easy, huh?

Connect your SD card to your PC – use dmesg to make sure you take the right device for the next steps.
_warning: all data on the SD will be lost!
_note: tested it with 2GB – must not be a SDHC – old cards are working

The bootcode of the CPU expects the code at 0x400 on the SD card. The created image is already padded to fit this needs – so no seeking is needed – just write the image to the card:

#>dd if=u-boot_mddr_256-E60610-K4X2G323PD.bin of=/dev/SDCARD bs=512

The drawback of using the image with padding is, that if you write it to the SD card also the MBR is wiped out – means your partition table is also deleted. For development it is handy to just replace the u-boot and not the 0x400 bytes in front of it – so lets strip the image.

>dd if=u-boot_mddr_256-E60610-K4X2G323PD.bin of=u-boot_mddr_256-E60610-K4X2G323PD.bin.stripped bs=512 skip=2

Writing the stripped image is straight forward – and it keeps the MBR untouched/intact:

#>dd if=u-boot_mddr_256-E60610-K4X2G323PD.bin.stripped of=/dev/SDCARD bs=512 seek=2

_note: before starting developing you should clean the SD card entirely to be save from side-effects caused by old data:
#>dd if=/dev/zero of=/dev/SDCARD bs=512

Plug the card into the shine, connect serial adapter and power on… the first boot-up reveals some other stuff that is needed to boot the shine properly.

MMC read: dev # 0, block # 1023, count 1 partition # 0 ...
1 blocks read: OK
no "hwcfg" bin header
-
MMC read: dev # 0, block # 18431, count 1 partition # 0 ...
1 blocks read: OK
no "logo" bin header
-
MMC read: dev # 0, block # 14335, count 1 partition # 0 ...
1 blocks read: OK
no "waveform" bin header
boot normal : no hwconfig !
_init_tps65185_power(): cannot init without hwconfig !

At the moment I think the tps65185 is the most important – cause without the display powered up we will see nothing. The magic behind the hardware configuration is partly done in freescale/mx50_rdp/ntx_common.c. The array gszNtxBinMagicA contains the magic number (FF F5 AF FF) that is checked within the function _load_ntx_bin_header. The information about the size of the hardware configuration is the location of the magic number + 8 – so if we find the magic number in the image we could easily extract the hardware config of the shine to use it in our own build…

Looking at block 1023 (=0x7fe00) of the backup image for finding the magic number, according to the code…

Magic = 0x7fe00 + 0x200 – 0x10 -> 0x7FFF0 -> Gotcha!

BinSize = Magic + 8 -> 0x7FFF8 = 0x6e -> 110

…strange cause the size of the header is

+0x10 tagNTXHWCFG_HDR
+0x23 tagNTXHWCFG_VAL
———–
0x33

…and only that amount of memory is copied. Strange. But now we could extract the configuration from the backup image of the shine and put it on our SD card.

>dd if=first_16_mb_sd_card.img of=tolino_shine_hw_config.img skip=524272 bs=1 count=67
...
>hexdump -C tolino_shine_hw_config.img
00000000 ff f5 af ff 78 56 34 12 6e 00 00 00 00 00 00 00 |....xV4.n.......|
00000010 48 57 20 43 4f 4e 46 49 47 20 76 31 2e 36 00 26 |HW CONFIG v1.6.&|
00000020 1e 0d 00 00 07 00 00 08 04 06 06 00 00 18 01 00 |................|
00000030 02 00 02 02 07 01 68 02 e4 00 00 02 02 02 00 01 |......h.........|
00000040 05 00 00 |...|
00000043

Now lets write the hardware configuration on the card and check what happens on boot.

#>dd if=tolino_shine_hw_config.img of=/dev/SDCARD seek=524272 bs=1

On boot now we get

....................
Kernel RAM visiable size=255M->255M
init TPS65185 power ...
....................

means the power control for the display is working now 🙂  You can find the complete boot log here.

_note: have a look at the source files in board/freescale/mx50_rdp – it is a great playground (eg. ntx_hwconfig.h explains what is stored inside the hw config blob)

Next step: adding a kernel and a initrd.

Useful to understand what the heck is done when compiling u-boot for a specific board: freescale IMX50 user guide

collected: exploring the tolino shine

I played some days weeks with the Tolino Shine – an ebook reader offered by a consortium of different booksellers and developed by the Deutsche Telekom.
Two results: I’m confused about the idea behind Kafkas „Die Verwandlung“ and the reader is save – in the meaning of „lets enable ADB or gain root the easy way“.
More or less the girls/guys of the Telekom did a good job – only one little hole – but so far no major issue. I explored the shine in hard and software and this post is a sum-up of my findings regarding to this. On some points also some Android background knowledge pops up.

reading the file system

The Tolino Shine exports two drives over USB. One is linked to the internal flash (and named „Mein Tolino“ – with a space in the middle – idiots). The second drive appears only if an micro sd is inserted – as  an
unnamed device. I placed files with a fixed name on that drives and tried to access them via the browser by using an URL starting with file://sdcard/… and voilà, that worked well. Its a known issue for old browsers –
see CVE-2010-4804. This hole gives you access to all local files (if rights granted) and could be used to explore the Shine a little… it is possible to read parts of /proc – that gives you access to meminfo, cpuinfo and so on… a lot of useful informations – if you know nothing about a device. Some samples…

proc/partitions
major minor #blocks name
179 0 3872256 mmcblk0
179 1 2263552 mmcblk0p1
179 2 393216 mmcblk0p2
179 3 1 mmcblk0p3
179 4 262144 mmcblk0p4
179 5 524288 mmcblk0p5
179 6 393216 mmcblk0p6
179 8 1933312 mmcblk1
179 9 1933244 mmcblk1p1

/proc/cpuinfo
Processor : ARMv7 Processor rev 5 (v7l)
BogoMIPS : 799.53
Features : swp half thumb fastmult vfp edsp neon vfpv3
CPU implementer : 0x41
CPU architecture: 7
CPU variant : 0x2
CPU part : 0xc08
CPU revision : 5
Hardware : Freescale MX50 Reference Design Platform
Revision : 50011
Serial : 0000000000000000

/system/bin/upgrade_check.sh (cutted)
#!/system/bin/bash -x
i=0
rm /data/no_sd_upgrade
upgrade_check(){
sleep 6
if [ -e /mnt/sdcard/extsd/update.zip ] ; then
echo "---> recovery mode setting ---------------------------------------"
rm /cache/downloadfile*
sync
sync
if [ -e /mnt/sdcard/extsd/update_mark ] ; then
# mv /mnt/sdcard/extsd/update.zip /mnt/sdcard/extsd/update.zip_old
rm /mnt/sdcard/extsd/update.zip
rm /mnt/sdcard/extsd/update_mark
upgrade_check
......
fi
elif [ -e /mnt/sdcard/extsd/waveform.bin ]; then
........
# xiaobo begin of adb_open
# elif [ -e /mnt/sdcard/extsd/adb_open ]; then
# setprop persist.service.adb.enable 1
# sync
# sync
# elif [ -e /mnt/sdcard/extsd/adb_close ]; then
# setprop persist.service.adb.enable 0
# sync
# sync
# xiaobo end of adb open
elif [ -e /mnt/sdcard/extsd/recovery.img ] || [ -e /cache/upgrade/recovery.img ]; then
echo "---> Programming partition RECOVERY ----------------------------------"
......
am start -a android.intent.action.ACTION_REQUEST_SHUTDOWN
fi
}
upgrade_check

It is even possible to read binary files like the init or config.gz – that holds the config for building a kernel… a lot of files – a lot of typing – and it was really a pain to enter all that by hand. So I crafted a page that uses java script to read a local file, encode it into a JSON-string (big overhead – but that is encoding save in every way) and sends it to a server. The page contains a free form text field for entering file names directly, a list of known files (not all are working), and a word book attack that could be run in different locations (the word book was formed by creating permutations from a list single words and separators – not very clever… sorry). You need to place the .php-file on a server the Shine can access – and set
the upload password to something else then the given one – or some strange guys will use your server as some kind of drop box replacement. The upload is limited by the allowed POST-size of your server. Change the encoding of the transmitted data or add multipart file transfer if you like… here the sources, use on your own risk: readout Just copy shine.html and wl.js on the drive name „Mein Tolino“ and point the browser to file://sdcard/shine.html and you should get the following page:

Be creative… and report new files you have found on the device. Edit: don’t do it. Continue reading!

the update.zip

After getting a lot of informations from the device I tested its reaction on a file named update.zip by just ‚touching‘ it on the drives. After restarting the reader  he tried to install the update-file.
The file was empty – so a seek failed message was printed. A good sign that the shines update/recovery system uses the stock Android verifier code… In these days the long awaited update was made
available – but intercepting the update by sniffing the traffic turned out as ***nearly impossible*** – cause they use an SSL connection for transferring the update file. By sniffing the communication
I made two observations:

1. wikipedia.de is called – I think this is the way they check …are we online?!?…
2. a suspend-jpg is downloaded from http://www.pageplace.de/media/eink-reader/sleep_screen.jpg – no idea why (conspiracy theory: there is some hidden data in the image…)

But back to the update.zip… For security reasons normally an update-container is signed by the creator. That means that SHA1 was used on the data part to create an unique fingerprint and then the supplier uses a/his private (RSA) key to encrypt the hash. This encrypted hash is then placed as a comment (in a PKCS1.5 container) at the end of the update file (see verifier.cpp – gives an idea how comments are placed in zip files). On startup the device extracts the encrypted hash out of the update.zip, loads the public key (update.zip file res/keys contains an example) stored on the device, decrypts the encrypted hash and compares the result with a SHA1 hash created over the data part of the offered update.zip file… if both hashes are equal the update.zip is accepted. If not an „failed to verify whole-file signature“-error is printed and the device continues to boot. (btw: if you got a „signature is too short“ error the the update contains a comment – but not an PKCS1.5 container)

This information is very important cause most articles I found about repacking and signing update.zip files do not outline that the signature used for checking authenticity and validity  is based on a/some public key(s) stored on the device. To use self signed update files the recovery program must be replaced by a version that ignores the signature check or a new public key (for a known private key) must be installed on the device. Both ways are not open for now. So if you change something within the update file you are out of luck.

If you are going to create your own certificates for signing update-files, remember to use 3 as exponent for the RSA  (…at least for the Tolino – if I got it right e was changed back to 65537 in later Android versions).

According to the license pdf contained in the archive the Shine is updated to Android 2.3.4 – the license file contained in firmware 1.0.1 shows 2.3.3 as the used Android version.

shiny kernel

A nice guy (thank you, Hajo) shared a link to the kernel sources of the Tolino Shine. I think they were published by accident cause the download location is a formerly unprotected WordPress upload directory… The Shine uses a 2.6.35.3 Kernel – a diff against the sources from kernel.org returns the changes made for the Shine… mainly patches from freescale to support the MX-series, the Android parts and some additions made by the Shine firmware developers (Joseph? , Angela?, Daniel?) . You can compile and start it using qemu – but thats all – a booting kernel – nothing special.

at the end…

I was a little frustrated …I opened the shine – looking for a JTAG connector or at least a serial port. It was not that easy to open the cover cause the display frame is glued to the upper half of the enclosure using double sided tape… but after some sweating it was done. The entire board – the PCB and the display – is mounted to the bottom cover with 4 screws.

Tolino board mounted on backcover

After removing them I was able to pull out the whole electronic body – nice.

The first thing that catches my attention was the micro SD located on the back of the board… wow, what a find. I did a backup of the whole card (after running the restore to factory process) – you can find the image here:

md5	size/mb	file
9a9427be4988ac7050c77a9a023f0c1e	673	backup_internal_sd_shine_after_sys_recover.img.gz
0ec840258ec4b8161736fa55f1d45c09	16	first_16_mb_sd_card.img

Instead of using a dedicated NAND flash chip an SD card is used to store all persistent data… what the heck… that means: you can replace the complete system image without the hazel of a soldered NAND chip / JTAGing cause you can write the system to the SD an try it out… and there should be no way to brick the device cause you could replace the (persistent) memory  easily… and I was looking for a JTAG port… happiness. The partition table of the internal SD card as shown by parted and enriched with infos from recovery.fstab:

Number  Start   End     Size    Type      File system  Flags
1      15.7MB  2334MB  2318MB  primary   fat32                      /sdcard vfat /dev/block/mmcblk0p1   "Mein Tolino"
2      2334MB  2736MB  403MB   primary   ext4                       /system ext4 /dev/block/mmcblk0p2
3      2736MB  3679MB  943MB   extended
5      2737MB  3274MB  537MB   logical   ext4                       /data ext4 /dev/block/mmcblk0p5
6      3275MB  3678MB  403MB   logical   ext4                       /cache ext4 /dev/block/mmcblk0p6
4      3679MB  3947MB  268MB   primary   ext4                       recovery

There are 15 megs of space in front of the first partition – here my findings:

0x000000 - 0x0001bf filled with 0 - means: no boot loader
0x0001c0 - 0x0001ff partition table (ends with 55aa)
0x000200 - 0x00020a serial number (device type)
0x00020b            just a colon
0x00020c - 0x00022b another serial/a hash? (32 byte)
0x000400 - 0x02695f uboot (contains also the recovery)
0x07fff0 - 0x080035 HW Config v 1.6 - version string?
0x0e0002 - 0x0e07ff binary blubber
0x0f0002 - 0x0f07ff binary blubber
0x0ffff0 - 0x0fffff binary blubber
0x100000 - 0x10003f uboot header
0x100040 - 0x43122e kernel
0x5ffff0 - 0x6236ff initramfs
0x6ffff0 - 0x81658e binary blubber...

After browsing the endless hex-file and writing the fun down I found recovery:/system/bin/upgrade.sh – it shows a partly different partition scheme – but contains also useful informations.

shiny hardware

Here comes a list of the chips I found on the board:

plate	function	manufacturer	data sheet/info
K4X2G323PD8GD8	DRAM 64Mx32 = 256 MB, 800 Mhz, 1.8V	SAMSUNG
MCIMX507CVM8B	Cortex A8, up to 800 Mhz	Freescale	www.freescale.com/webapp/sps/site/prod_summary.jsp?code=i.MX507
zForce NN1001	optical touch controller	neonode & TI
TPS 65185	eInk power supply	TI	http://www.ti.com/product/tps65185
430G2333	micro controller (reading battery state?)	TI	http://www.ti.com/product/msp430g2333
430V325	micro controller (charging?)	TI	http://www.ti.com/product/msp430p325
wc121	single chip WLAN module	cybertan	http://www.cybertan.com.tw/products/WC121.html

serial reader

There are 3 serial ports on the board. The one next to the CPU (right side of the board) is … the serial port that is connected to the CPU. Using a proper interface with 3.3V signal level and a terminal program running at 115200 8N1 you can sniff the boot messages of u-boot and the kernel. Don’t forget to cross RX and TX and again: use 3.3V signal levels!

U-Boot 2009.08 ( 1月 25 2013 - 15:04:09)
-
CPU: Freescale i.MX50 family 1.1V at 800 MHz
mx50 pll1: 800MHz
mx50 pll2: 400MHz
mx50 pll3: 216MHz
ipg clock : 66666666Hz
ipg per clock : 66666666Hz
uart clock : 24000000Hz
ahb clock : 133333333Hz
axi_a clock : 400000000Hz
axi_b clock : 200000000Hz
weim_clock : 100000000Hz
ddr clock : 200000000Hz
esdhc1 clock : 80000000Hz
esdhc2 clock : 80000000Hz
esdhc3 clock : 80000000Hz
esdhc4 clock : 80000000Hz
Board: MX50 RDP board
Boot Reason: [POR]
Boot Device: SD
I2C: ready
DRAM: 256 MB
......................
MMC read: dev # 0, block # 14336, count 2228 partition # 0 ...
2228 blocks read: OK
Kernel RAM visiable size=254M->254M
Detecting HOME+POWER key for recovery ...
Hit any key to stop autoboot: 0
-
MMC read: dev # 0, block # 2048, count 8192 partition # 0 ...
8192 blocks read: OK
-
MMC read: dev # 0, block # 12288, count 768 partition # 0 ...
768 blocks read: OK
## Booting kernel from Legacy Image at 70800000 ...
Image Name: Linux-2.6.35.3-hg9facadc85de9-di
Created: 2013-01-25 7:01:22 UTC
Image Type: ARM Linux Kernel Image (uncompressed)
Data Size: 3346940 Bytes = 3.2 MB
Load Address: 70008000
Entry Point: 70008000
Verifying Checksum ... OK
## Loading init Ramdisk from Legacy Image at 70d00000 ...
Image Name: ntxinitramfs
Created: 2013-02-05 5:38:45 UTC
Image Type: ARM Linux RAMDisk Image (uncompressed)
Data Size: 145089 Bytes = 141.7 kB
Load Address: 70308000
Entry Point: 70308000
Verifying Checksum ... OK
Loading Kernel Image ... OK
OK
...................................
Starting kernel ...
Uncompressing Linux... done, booting the kernel.
Initializing cgroup subsys cpu
Linux version 2.6.35.3-hg9facadc85de9-dirty (antony@antony-desktop) (gcc version 4.4.3 (GCC) ) #168 PREEMPT Fri Jan 25 15:01:19 CST 2013
CPU: ARMv7 Processor [412fc085] revision 5 (ARMv7), cr=10c53c7f
...................................

Of course – you can interrupt the boot process by pressing a key.. here comes the output of printenv and bdinfo:

eBR-1A # printenv
bootdelay=1
baudrate=115200
loadaddr=0x70800000
netdev=eth0
ethprime=FEC0
uboot=u-boot.bin
kernel=uImage
nfsroot=/opt/eldk/arm
bootargs_base=setenv bootargs console=ttymxc0,115200
bootargs_nfs=setenv bootargs ${bootargs} root=/dev/nfs ip=dhcp nfsroot=${serverip}:${nfsroot},v3,tcp
bootcmd_net=run bootargs_base bootargs_nfs; tftpboot ${loadaddr} ${kernel}; bootm
bootargs_mmc=setenv bootargs ${bootargs} ip=dhcp root=/dev/mmcblk0p2 rootwait
bootcmd_mmc=run bootargs_base bootargs_mmc; bootm
rd_loadaddr=0x70D00000
bootcmd_SD=mmc read 0 ${loadaddr} 0x800 0x2000; mmc read 0 ${rd_loadaddr} 0x3000 0x300;
bootcmd=run bootcmd_SD; bootm ${loadaddr} ${rd_loadaddr}
bootargs=console=ttymxc0 init=/init androidboot.console=ttymxc0 keypad video=mxc_elcdif_fb:off calibration
stdin=serial
stdout=serial
stderr=serial
-
eBR-1A # bdinfo
arch_number = 0x00000BAC
env_t       = 0x00000000
boot_params = 0x70000100
DRAM bank   = 0x00000000
-> start    = 0x70000000
-> size     = 0x10000000
baudrate    = 115200 bps

You can download the complete bootlog here: tolino_shine_boot_log.txt

After booting you are put to a bash shell but the console is cluttered by a continuously stream of some redraw informations and the logging of temperature state of the display voltage controller. If you are lucky you can get some clean output…

bash-3.2# ps
USER PID PPID VSIZE RSS WCHAN PC NAME
root 1 0 324 188 800ec05c 0000875c S /init
root 2 0 0 0 800823d8 00000000 S kthreadd
root 3 2 0 0 80071a24 00000000 S ksoftirqd/0
root 4 2 0 0 8007ec3c 00000000 S events/0
root 5 2 0 0 8007ec3c 00000000 S khelper
root 8 2 0 0 800890f0 00000000 S async/mgr
root 9 2 0 0 8007ec3c 00000000 S pm
root 12 2 0 0 8007ec3c 00000000 S suspend
root 76 2 0 0 80057a2c 00000000 S usb_wakeup thre
root 77 2 0 0 80057a2c 00000000 S usb_wakeup thre
root 234 2 0 0 800bb414 00000000 S sync_supers
root 236 2 0 0 800bbef4 00000000 S bdi-default
root 238 2 0 0 8007ec3c 00000000 S kblockd/0
root 252 2 0 0 8007ec3c 00000000 S mxc_spi.2
root 259 2 0 0 8007ec3c 00000000 S otg_switch/0
root 265 2 0 0 8029d5c8 00000000 S khubd
root 282 2 0 0 8007ec3c 00000000 S kmmcd
root 312 2 0 0 8031c4d4 00000000 S pmic-event-thre
root 336 2 0 0 8007ec3c 00000000 S rpciod/0
root 345 1 0 0 80051524 00000000 D swapper
root 350 2 0 0 8007ec3c 00000000 S zq_calib
root 356 2 0 0 800b5abc 00000000 S kswapd0
root 404 2 0 0 8007ec3c 00000000 S aio/0
root 414 2 0 0 8007ec3c 00000000 S nfsiod
root 418 2 0 0 8007ec3c 00000000 S crypto/0
root 446 2 0 0 8007ec3c 00000000 S submit/0
root 457 2 0 0 8007ec3c 00000000 S tps65185_PWRGOO
root 459 2 0 0 8007ec3c 00000000 S tps65185_INT/0
root 1039 2 0 0 802346f8 00000000 S kapmd
root 1105 2 0 0 802c4260 00000000 S file-storage
root 1149 2 0 0 8007ec3c 00000000 S kstriped
root 1154 2 0 0 8007ec3c 00000000 S kconservative/0
root 1157 2 0 0 8007ec3c 00000000 S mxc_chg
root 1158 2 0 0 8007ec3c 00000000 S mc13892_battery
root 1165 2 0 0 8007ec3c 00000000 S esdhc_wq/0
root 1170 2 0 0 8007ec3c 00000000 S esdhc_wq/0
root 1172 2 0 0 8007ec3c 00000000 S esdhc_wq/0
root 1182 2 0 0 8007ec3c 00000000 S usbhid_resumer
root 1185 2 0 0 8007ec3c 00000000 S binder
root 1217 2 0 0 8007ec3c 00000000 S l2cap
root 1218 2 0 0 80459624 00000000 S krfcommd
root 1226 1 300 156 800ec05c 0000875c S /sbin/ueventd
root 1243 2 0 0 80336e9c 00000000 S mmcqd
root 1309 446 0 0 8006ff84 00000000 Z submit/0
root 2013 2 0 0 80336e9c 00000000 S mmcqd
root 2015 2 0 0 8015c29c 00000000 S jbd2/mmcblk0p2-
root 2018 2 0 0 8007ec3c 00000000 S ext4-dio-unwrit
root 2019 2 0 0 800fbf4c 00000000 S flush-179:0
root 2020 2 0 0 8015c29c 00000000 S jbd2/mmcblk0p5-
root 2021 2 0 0 8007ec3c 00000000 S ext4-dio-unwrit
root 2022 2 0 0 8015c29c 00000000 S jbd2/mmcblk0p6-
root 2023 2 0 0 8007ec3c 00000000 S ext4-dio-unwrit
root 2024 1 1428 780 8006f710 000a8fa8 S /system/bin/bash
system 2025 1 816 256 8034b580 6fd0b6fc S /system/bin/servicemanager
root 2026 1 3868 588 ffffffff 6fd0bdac S /system/bin/vold
root 2027 1 3844 560 ffffffff 6fd0bdac S /system/bin/netd
root 2028 1 820 324 800ec05c 6fd0b844 S /system/bin/dispd
root 2029 1 676 260 803834dc 6fd0c0cc S /system/bin/debuggerd
root 2030 1 99812 27668 800ec05c 6fd0b844 S zygote
media 2031 1 17216 4212 ffffffff 6fd0b6fc S /system/bin/mediaserver
root 2032 1 824 316 804057bc 6fd0b45c S /system/bin/installd
keystore 2033 1 1752 428 803834dc 6fd0c0cc S /system/bin/keystore
shell 2036 1 3400 164 ffffffff 00008294 S /sbin/adbd
system 2088 2030 165876 36432 ffffffff 6fd0b6fc S system_server
app_3 2140 2030 116008 20024 ffffffff 6fd0c51c S com.android.inputmethod.latin
system 2146 2030 108160 14940 ffffffff 6fd0c51c S ntx.power
system 2152 2030 113392 24008 ffffffff 6fd0c51c S com.android.systemui
system 2153 2030 138684 35756 ffffffff 6fd0c51c S de.telekom.epub
app_6 2192 2030 112424 19224 ffffffff 6fd0c51c S android.process.acore
app_4 2220 2030 111196 17944 ffffffff 6fd0c51c S android.process.media
app_5 2230 2030 108752 16912 ffffffff 6fd0c51c S com.android.providers.calendar
root 2250 2 0 0 800fbf4c 00000000 S flush-179:8
root 2257 2024 904 316 00000000 6fd0b45c R ps

And btw – about „rooting“ that device:

bash-3.2# id
uid=0(root) gid=1007(log)

I also tried to get some data from the remaining serial ports – but nothing happens here. Maybe they use a protocol based on polling or they are not for reading data bur for programming the two MSP430 processors… follow the traces and tell us if you found something great…

ADB

I made some changes on the SD card to enable ADB by activating some uncommented lines in /system/bin/upgrade_check.sh. Remember – this file was one of my first findings in the beginning of the journey. It gives me some (late) satisfaction to use that file (even if I had to open the Shine for it – and of curse, there are other ways to do that – just remove the disabled flag for the ADB service…)

# xiaobo begin of adb_open
elif [ -e /mnt/sdcard/extsd/adb_open ]; then
setprop persist.service.adb.enable 1
sync
sync
elif [ -e /mnt/sdcard/extsd/adb_close ]; then
setprop persist.service.adb.enable 0
sync
sync
xiaobo end of adb open

After that I touched a file named adb_open to /sdcard/extsd. If you had a look at the boot-log you already found a line containing the following:

warning: `adbd' uses 32-bit capabilities (legacy support in use)
enabling adb

I can’t give you any information about using ADB in windows. But if you are running Linux you are fine eg. under Ubuntu sudo apt-get install android-tools-adb will provide you with the needed tool. If not and your distro does not offer a package that contains ADB go and download the right part from the  Android SDK. Good luck.

If you have installed ADB the remaining part is straight forward:

# get the vendor id from syslog (maybe dmesg | grep idVendor): idVendor=1f85 then...
#> echo 'SUBSYSTEM=="usb", ATTR{idVendor}=="1f85", MODE="0666", GROUP="plugdev"' >> /etc/udev/rules.d/51-android.rules
#> mkdir ~/.android/
#> echo 0x1f85 > ~/.android/adb_usb.ini
#> adb server-start
#> adb devices
-
List of devices attached
20030394 device
-
#> adb shell
$ ls
config
cache
sdcard
acct
mnt
vendor
d
etc
sys
init.rc
ueventd.rc
default.prop
sbin
init.goldfish.rc
ueventd.freescale.rc
system
ueventd.goldfish.rc
data
init
proc
init.freescale.rc
root
dev
....

The rest is up to you. With ADB enabled you could close the enclosure and use USB for communication… just use the serial line to remove permission restrictions before:)

next steps

Exploring the Tolino Shine was fun. I ended up with a device that was opened… not really rooted by using some software – some special kind of escalation. But looking for a JTAG interface and finding an easy to change SD card is… win. Having ADB and serial connection now is nothing to be proud of – it was inside the device all the time. And to be honest: I feel a little bit sad that I was not able to achieve at least the working ADB without opening the device. Now some scripts have to be done to enable ADB all the time and remove all permission restrictions on that device (and: publish new SD images!). Maybe someone could change the Public Key (one is located in the recovery partition – just look for key, use the informations about creating keys and the key file out of this ) in the image by on of a well known keypair… a lot of stuff if you like to continue in a eco system called „Tolino Shine“. For me the device is now just an easy to change and powerful system with a excellent touch screen, a serial port and more or less space inside the enclosure to add new hardware. So there are two big possibilities to continue on:

• analyzing the system by looking at the whole system description – in hard- and software, find the weak point and root the device without opening it
• create your own system running on the shines hardware

The latter one is the way I choose – the goal is a sd card image that holds all the stuff to run the shine as piratebox… But with the announcement of new Tolino hardware for the end of 2013 the price should go down… more Shines, more developers… Lets see what happens. Good luck!

relevant links

hardware:
http://www.freescale.com/webapp/sps/site/prod_summary.jsp?code=IMX50EVK

ARM knowledege base:
http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.faqs/

Porting kernel to arm:
http://www.linux-arm.org/pub/LinuxKernel/WebHome/aleph-porting.pdf

freescale downloads:
http://git.freescale.com/git/

ARM toolchain:
https://wiki.linaro.org/Resources/ToolchainInstall-10/4.4
http://www.linaro.org/downloads/
https://github.com/AdiPat/Android_Toolchains

uboot:
http://www.compulab.co.il/workspace/mediawiki/index.php5/U-Boot:_Images
http://balau82.wordpress.com/2010/03/10/u-boot-for-arm-on-qemu/

qemu – arm:
http://www.linuxforu.com/2011/06/qemu-for-embedded-systems-development-part-1/
http://wiki.qemu.org/Download
http://balau82.wordpress.com/2010/03/22/compiling-linux-kernel-for-qemu-arm-emulator/
http://mmmyddd.freeshell.net/wiki/embed/linuxonarmonqemu.html
http://qemu.weilnetz.de/qemu-doc.html#ARM-System-emulator
https://wiki.edubuntu.org/ARM/RootfsFromScratch/QemuDebootstrap

problems with ARM on qemu:
https://community.freescale.com/thread/281256

good intro about using gdb with qemu to debug a running system:
http://files.meetup.com/1590495/debugging-with-qemu.pdf

kernel debug:
https://www.kernel.org/doc/htmldocs/kgdb/EnableKGDB.html
http://elinux.org/Debugging_by_printing#Debugging_early_boot_problems

gdb kernel stub:
http://wiki.osdev.org/GDB

android ndk download:
http://developer.android.com/tools/sdk/ndk/index.html

.apk – run and reverse:
http://stackoverflow.com/questions/4512458/androidhow-to-run-apk-file-on-emulator
https://code.google.com/p/android-apktool/

android security:
http://www.ibm.com/developerworks/library/x-androidsecurity/

android filesystem overview:
http://anantshri.info/andro/file_system.html

android – relevant sources:
https://android.googlesource.com/platform/bootable/recovery/+/master/bootloader.cpp
https://android.googlesource.com/platform/bootable/recovery/+/master/verifier.cpp
https://android.googlesource.com/platform/bootable/recovery/+/master/install.cpp

update.zip:
http://images.thalia.de/md2/ereader/update/update.zip
http://www.sslshopper.com/certificate-decoder.html
http://www.londatiga.net/it/how-to-sign-apk-zip-files/
http://developer.android.com/tools/publishing/app-signing.html

keys & co – tools, infos:
https://raw.github.com/cgjones/android-system-core/master/libmincrypt/tools/DumpPublicKey.java
http://www.bouncycastle.org/latest_releases.html
http://en.wikipedia.org/wiki/RSA_%28algorithm%29

uRamdisk, initrd & co:
http://forum.xda-developers.com/showthread.php?t=1477845
http://www.glomationinc.com/doc/HowToEditRAMDisk.PDF
http://www.isysop.com/unpacking-and-repacking-u-boot-uimage-files/
http://www.unixboard.de/vb3/showthread.php?5913-initrd-img-bearbeiten

CVE:

http://www.cvedetails.com/product/19997/Google-Android.html?vendor_id=1224
http://www.cvedetails.com/cve/CVE-2010-4804/
http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2010-1807
http://thomascannon.net/blog/2010/11/android-data-stealing-vulnerability/

other readers but related:
http://pastie.org/pastes/6112039

collected: compile Qt on Ubuntu 12.04 and other snippets from work…

After some sporadic crashes of a Qt-application (not my fault – gdb says it comes from the QT core…, Ubuntu 12.04/32, QT SDK 4.8.1 binary) I decided to build Qt from the sources… but configure failes with „missing xlib“-errro. solution:

sudo apt-get install libxext-dev

for using the full beauty and power of Qt the following additional devel-deps should be installed:

sudo apt-get install libXrender-dev libicu-dev libfontconfig1-dev libxrandr-dev libxinerama-dev libglib2.0-dev libgtk2.0-dev

… easy. huh?!?
for proper fonts & style run configure with the parameters „-gtkstyle -fontconfig -glib“ – or it looks wired…

configure throws some warnings/errors about missing dependencies… that reminds me that self-checking for a given library can be done with

pkg-config --cflags <libname>

it returns the compile-flags (mostly paths) or an error if the lib is not present. by replacing the –cflags with –libs the needed linker flags are dumped.

the following snipped was/is useful for dumping log-data:

cat `ls  -lrt | sort -k6,8 -r | awk 'NR==1{ print $9 }'`

it prints the oldest file in the directory to stdout (ohh… we assume there are only regular files).

why writing this down?!? Qt is still compiling… nothing left to do… should read a book.

collected: screen

this is a plain collector article about gnu screen. in the early days I used screen only for running processes that should last after I’ve logged out. now-days it’s comfortable for debugging programs and/or running [virtual] kernels/machines in one terminal. to screen-able your shell simply run screen (>screen). the following snipped gives a short overview about the most common shortcuts you could use in screen…

(C = CTRL)

C+a c          new screen-session
C+a A          set name for the window
C+a w          show list of all windows in status bar
C+a "          show list of all windows (up/down for select, enter to open)
C+a N          show number and title of current window
C+a '          switch to given window (0..n + enter)
C+a 1…n        switch to window with given number
C+a n          switch to next window
C+a p          switch to previous window

C+a S          split current window horizontally
C+a |          split current window vertically
C+a TAB        switch to the next region
C+a X          close the current region
C+a Q          close all regions but the current
C+a q          send C+q to the current region

C+a d          detach window from terminal
               reattach with screen -r <pid_of_screen>

C+a ESC       
C+a [          start copy and scroll mode, use cursor up/down for scrolling/move cursor
               press enter to start marking text, press enter second time->marked text is copied to buffer
C+a ]          paste current buffer to stdout
C+a >          paste current buffer to /tmp/screen-exchange
C+a <          read data from /tmp/screen-exchange

C+a :          screen command prompt
C+a ?          list key-bindings  
C+a v          outputs screen version
C+a K          exit&close screen session

you can also define your own bindings and beside the already bounded commands there are tons of it you can use in the screen-prompt (have a look to the man’s, most used by me is hardcopy -h – it simply dumps the current content of the region into a file).

module development for GRUB2 – playground

Since testing modules for GRUB by compile/reboot/test/boot/… is time consuming, frustrating – simply not very efficient, there was an utility shipped with the sources called grub-emu that (like the name states) emulates the grub console. But: it was/is not a perfect emulation (stm), no modules are loaded/loadable so every module must be hand-coded-added to the built and… it does not build (like descibed here).

The playground – world:

And thats is okay cause there is another great tool: grub_mkrescue – an image builder that creates floppy/cdrom-images with… grub on it. (and it supports the preloading of modules) Sample:

$m[2]

Gives a GRUB inside qemu with hello.mod already loaded. Fine. Put the lines into the Makefile of the module currently developed (maybe as run-target) and testing becomes a lot easier…

The playground – garden:

Assume you have checked out the (latest) sources of grub2, and now you stay into the source-dir grub-1.98. Your new module (for the firsttime a copy/paste/change of hello/hello.c) goes to grub-1.98/simplemod.

Building a module could be done two ways: the slower one by adding a probate entry in conf/common.mk (simple copy/paste an entry for a given module like hello) – results in building everything again, and again when calling make clean/make… or the good, fast and due to the handcrafted standalone-Makefile ugly one.
No more filling phrases… here is the makefile (to be placed in grub-1.98/simplemod/Makefile):

$m[2]

Okay, on the first time you must create the surrounding grub2-ecosystem (grub-1.98/configure;make;make install). But after that you only need to compile your new module. That is efficient.

Fedora 14 & GRUB2

Forced by a project@work I had to install GRUB2 in Fedora 14. Here is a simple description of what I have done to got it running:

1. install the new „grand unified bootloader package“ by

$m[2]

Attention: at this time, Fedora uses multislot – GRUB Legacy (0.9x) and GRUB2 could be installed parallel.

2. edit/create the new /boot/grub2/grub.cfg

2.1 My old /boot/grub/grub.conf looks like:

$m[2]

In the snipped above you could see that GRUB2 is allready inserted into the grub.conf of your old GRUB. You could use it for boot into the GRUB command-line.

2.2. create a new grub.cfg
Important: in GRUB Legacy (the old GRUB) the partition-numbering starts with 0. In GRUB2 the numbering starts with 1. So hd0,0 from above becomes hd0,1 eg.
Check/note on witch device/partition your /boot and root is located and witch UUID it has assigned (mount gives you the device /dev/sd…, blkid the associated UUID), sample:

$m[2]

3. Now run

$m[2]

The output is placed into /boot/grub2/grub.cfg.new – do a cat on it and compare the UUID for /boot and root with the ones from above (and maybe compare against your old grub.conf) … and yes, hd0,0 is now hd0,1 .

4. The final step: install the new GRUB on the disc…
First only test that everything is fine… (without writing to MBR assuming the target drive is /dev/sda):

$m[2]

Now lets use the real bullet:

$m[2]

5. Finally

$m[2]

And it works. For me. Now.

GRUB2 offers 1000nds new possibilities (ohh, it has a LUA-shell, there is module-support, submenus, theming, …)

For learning: boot the core.img, load kernel and ramdisk by hand. boot. check console commads ls & cat.

Have a look at:
http://www.gnu.org/software/grub/manual/
http://ubuntuforums.org/showthread.php?t=1195275