Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

Hi,

Our ARM32 Linux embedded system consists of these:
* Linux Kernel: 4.14
* Processor: Qualcomm Arm32 Cortex-A7
* Storage: NAND 512MB
* Platform: Simple busybox
* Filesystem: UBIFS, Squashfs
* Consists of nand raw partitions, squashfs ubi volumes.

My requirement:
We wanted to use dm-verity at boot time to check the integrity of
squashfs-rootfs before mounting.

Problem:
dm-0 is not able to locate and mount the squash fs rootfs block.
The same approach is working when emulating with ext4 but fails with squashfs.

Logs:
[....]
[    0.000000] Kernel command line: [...] verity="96160 12020
d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681 0 "
rootfstype=squashfs root=/dev/mtdblock34 ubi.mtd=30,0,30 [...]
root=/dev/dm-0 dm="system none ro,0 96160 verity 1 /dev/mtdblock34
/dev/mtdblock39 4096 4096 12020 8 sha256
d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681
aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7"
[....]
[    4.693620] vreg_conn_pa: disa▒[    4.700662] md: Skipping
autodetection of RAID arrays. (raid=autodetect will force)
[    4.700713] device-mapper: init: attempting early device configuration.
[    4.708224] device-mapper: init: adding target '0 96160 verity 1
/dev/mtdblock34 /dev/mtdblock39 4096 4096 12020 8 sha256
d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681
aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7'
[    4.714979] device-mapper: verity: sha256 using implementation
"sha256-generic"
[    4.737808] device-mapper: init: dm-0 is ready
[....]
[    5.278103] No filesystem could mount root, tried:
[    5.278107]  squashfs
[    5.280477]
[    5.287627] Kernel panic - not syncing: VFS: Unable to mount root
fs on unknown-block(253,0)
[...]

Not sure, why is it still locating block "253" here which seems like a
MAJOR number ?

Working logs on ext4:
[....]
[    4.529822] v▒[    4.534035] md: Skipping autodetection of RAID
arrays. (raid=autodetect will force)
[    4.534087] device-mapper: init: attempting early device configuration.
[    4.550316] device-mapper: init: adding target '0 384440 verity 1
/dev/ubiblock0_0 /dev/ubiblock0_0 4096 4096 48055 48063 sha256
a02e0c13afb31e99b999c64aae6f4644c24addbc58db5689902cc5ba0be2d15b
aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
restart_on_corruption ignore_zero_blocks use_fec_from_device
/dev/ubiblock0_0 fec_roots 2 fec_blocks 48443 fec_start 48443'
[    4.572215] device-mapper: verity: sha256 using implementation
"sha256-generic"
[    4.610692] device-mapper: init: dm-0 is ready
[    4.720174] EXT4-fs (dm-0): mounted filesystem with ordered data
mode. Opts: (null)
[    4.720438] VFS: Mounted root (ext4 filesystem) readonly on device 253:0.
[    4.737256] devtmpfs: mounted
[....]

Questions:
a) Is dm-verity supposed to work on squashfs block devices ?
b) Are there any known issues with dm-verity on Kernel 4.14 ?
c) Are there any patches that we are missing ?


Thanks,
Pintu

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Mikulas Patocka]

[-- Attachment #1: Type: TEXT/PLAIN, Size: 3391 bytes --]

Hi

Try to set up dm-verity with block size 512 bytes.

I don't know what block size does squashfs use, but if the filesystem 
block size is smaller than dm-verity block size, it doesn't work.

Mikulas



On Tue, 20 Jul 2021, Pintu Agarwal wrote:

> Hi,
> 
> Our ARM32 Linux embedded system consists of these:
> * Linux Kernel: 4.14
> * Processor: Qualcomm Arm32 Cortex-A7
> * Storage: NAND 512MB
> * Platform: Simple busybox
> * Filesystem: UBIFS, Squashfs
> * Consists of nand raw partitions, squashfs ubi volumes.
> 
> My requirement:
> We wanted to use dm-verity at boot time to check the integrity of
> squashfs-rootfs before mounting.
> 
> Problem:
> dm-0 is not able to locate and mount the squash fs rootfs block.
> The same approach is working when emulating with ext4 but fails with squashfs.
> 
> Logs:
> [....]
> [    0.000000] Kernel command line: [...] verity="96160 12020
> d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681 0 "
> rootfstype=squashfs root=/dev/mtdblock34 ubi.mtd=30,0,30 [...]
> root=/dev/dm-0 dm="system none ro,0 96160 verity 1 /dev/mtdblock34
> /dev/mtdblock39 4096 4096 12020 8 sha256
> d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681
> aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7"
> [....]
> [    4.693620] vreg_conn_pa: disa▒[    4.700662] md: Skipping
> autodetection of RAID arrays. (raid=autodetect will force)
> [    4.700713] device-mapper: init: attempting early device configuration.
> [    4.708224] device-mapper: init: adding target '0 96160 verity 1
> /dev/mtdblock34 /dev/mtdblock39 4096 4096 12020 8 sha256
> d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681
> aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7'
> [    4.714979] device-mapper: verity: sha256 using implementation
> "sha256-generic"
> [    4.737808] device-mapper: init: dm-0 is ready
> [....]
> [    5.278103] No filesystem could mount root, tried:
> [    5.278107]  squashfs
> [    5.280477]
> [    5.287627] Kernel panic - not syncing: VFS: Unable to mount root
> fs on unknown-block(253,0)
> [...]
> 
> Not sure, why is it still locating block "253" here which seems like a
> MAJOR number ?
> 
> Working logs on ext4:
> [....]
> [    4.529822] v▒[    4.534035] md: Skipping autodetection of RAID
> arrays. (raid=autodetect will force)
> [    4.534087] device-mapper: init: attempting early device configuration.
> [    4.550316] device-mapper: init: adding target '0 384440 verity 1
> /dev/ubiblock0_0 /dev/ubiblock0_0 4096 4096 48055 48063 sha256
> a02e0c13afb31e99b999c64aae6f4644c24addbc58db5689902cc5ba0be2d15b
> aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
> restart_on_corruption ignore_zero_blocks use_fec_from_device
> /dev/ubiblock0_0 fec_roots 2 fec_blocks 48443 fec_start 48443'
> [    4.572215] device-mapper: verity: sha256 using implementation
> "sha256-generic"
> [    4.610692] device-mapper: init: dm-0 is ready
> [    4.720174] EXT4-fs (dm-0): mounted filesystem with ordered data
> mode. Opts: (null)
> [    4.720438] VFS: Mounted root (ext4 filesystem) readonly on device 253:0.
> [    4.737256] devtmpfs: mounted
> [....]
> 
> Questions:
> a) Is dm-verity supposed to work on squashfs block devices ?
> b) Are there any known issues with dm-verity on Kernel 4.14 ?
> c) Are there any patches that we are missing ?
> 
> 
> Thanks,
> Pintu
> 

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

On Tue, 20 Jul 2021 at 17:12, Mikulas Patocka <mpatocka@redhat.com> wrote:
>
> Hi
>
> Try to set up dm-verity with block size 512 bytes.
>
> I don't know what block size does squashfs use, but if the filesystem
> block size is smaller than dm-verity block size, it doesn't work.
>
Okay thank you so much for this clue,
It seems we are using 65536 as the squashfs block size:
==> mksquashfs [...] - comp xz -Xdict-size 32K -noI -Xbcj arm -b 65536
-processors 1

But for dm-verity we are giving block size of 4096
==> [    0.000000] Kernel command line:[..] verity="96160 12020
d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681 0 "
rootfstype=squashfs root=/dev/mtdblock34 ubi.mtd=30,0,30 [...]
root=/dev/dm-0 dm="system none ro,0 96160 verity 1 /dev/mtdblock34
/dev/mtdblock39 4096 4096 12020 8 sha256
d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681
aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7"

Now, we are checking by giving squashfs block size also as 4096

In case, if this does not work, what else could be the other option ?
Can we try with initramfs approach ?

Thanks,
Pintu

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Mikulas Patocka]

On Wed, 21 Jul 2021, Pintu Agarwal wrote:

> On Tue, 20 Jul 2021 at 17:12, Mikulas Patocka <mpatocka@redhat.com> wrote:
> >
> > Hi
> >
> > Try to set up dm-verity with block size 512 bytes.
> >
> > I don't know what block size does squashfs use, but if the filesystem
> > block size is smaller than dm-verity block size, it doesn't work.
> >
> Okay thank you so much for this clue,
> It seems we are using 65536 as the squashfs block size:

65536 is the compression block size - it is unrelated to I/O block size.

There's a config option SQUASHFS_4K_DEVBLK_SIZE. The documentation says 
that it uses by default 1K block size and if you enable this option, it 
uses 4K block size.

So, try to set it. Or try to reduce dm-verity block size down to 1K.

> ==> mksquashfs [...] - comp xz -Xdict-size 32K -noI -Xbcj arm -b 65536
> -processors 1
> 
> But for dm-verity we are giving block size of 4096
> ==> [    0.000000] Kernel command line:[..] verity="96160 12020
> d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681 0 "
> rootfstype=squashfs root=/dev/mtdblock34 ubi.mtd=30,0,30 [...]
> root=/dev/dm-0 dm="system none ro,0 96160 verity 1 /dev/mtdblock34
> /dev/mtdblock39 4096 4096 12020 8 sha256
> d7b8a7d0c01b9aec888930841313a81603a50a2a7be44631c4c813197a50d681
> aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7"
> 
> Now, we are checking by giving squashfs block size also as 4096
> 
> In case, if this does not work, what else could be the other option ?
> Can we try with initramfs approach ?

Yes - you can try initramfs.

Mikulas

> Thanks,
> Pintu
> 

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

On Wed, 21 Jul 2021 at 22:40, Mikulas Patocka <mpatocka@redhat.com> wrote:

> > >
> > > Try to set up dm-verity with block size 512 bytes.
> > >
> > > I don't know what block size does squashfs use, but if the filesystem
> > > block size is smaller than dm-verity block size, it doesn't work.
> > >
> > Okay thank you so much for this clue,
> > It seems we are using 65536 as the squashfs block size:
>
> 65536 is the compression block size - it is unrelated to I/O block size.
>
> There's a config option SQUASHFS_4K_DEVBLK_SIZE. The documentation says
> that it uses by default 1K block size and if you enable this option, it
> uses 4K block size.
>
Okay it seems this config is set in our case:
CONFIG_SQUASHFS_4K_DEVBLK_SIZE=y
So, with this the squashfs and dm-verity block size exactly matches (4K)

> So, try to set it. Or try to reduce dm-verity block size down to 1K.
>
Okay we are trying this.
Thank you so much!!

Regards,
Pintu

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

On Wed, 21 Jul 2021 at 22:59, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> On Wed, 21 Jul 2021 at 22:40, Mikulas Patocka <mpatocka@redhat.com> wrote:
>
> > > >
> > > > Try to set up dm-verity with block size 512 bytes.
> > > >
> > > > I don't know what block size does squashfs use, but if the filesystem
> > > > block size is smaller than dm-verity block size, it doesn't work.
> > > >
> > > Okay thank you so much for this clue,
> > > It seems we are using 65536 as the squashfs block size:
> >
> > 65536 is the compression block size - it is unrelated to I/O block size.
> >
> > There's a config option SQUASHFS_4K_DEVBLK_SIZE. The documentation says
> > that it uses by default 1K block size and if you enable this option, it
> > uses 4K block size.
> >
> Okay it seems this config is set in our case:
> CONFIG_SQUASHFS_4K_DEVBLK_SIZE=y
> So, with this the squashfs and dm-verity block size exactly matches (4K)
>
> > So, try to set it. Or try to reduce dm-verity block size down to 1K.
> >
Hi,

Sorry for coming back to this again..
Unfortunately, none of the options is working for us with squashfs
(bootloader, initramfs).
initramfs have different kinds of challenges because of the partition
size issue.
So, our preferred option is still the bootloader command line approach..

Is there a proven and working solution of dm-verity with squashfs ?
If yes, please share some references.

The current problem with squashfs is that we could not append the
verity-metadata to squashfs, so we store it on a separate volume and
access it.
By specifying it like : /dev/mtdblock53

Then we get the error like this:
{
[    4.950276] device-mapper: init: attempting early device configuration.
[    4.957577] device-mapper: init: adding target '0 95384 verity 1
/dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
restart_on_corruption ignore_zero_blocks use_fec_from_device
/dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026'
[    4.975283] device-mapper: verity: sha256 using implementation
"sha256-generic"
[    4.998728] device-mapper: init: dm-0 is ready

[    5.614498] 1f35            1736 mtdblock53
[    5.614502]  (driver?)
[    5.621098] fc00           61504 ubiblock0_0
[    5.621102]  (driver?)
[    5.627661] fd00           47692 dm-0
[    5.627665]  (driver?)
[    5.633663] No filesystem could mount root, tried:
[    5.633667]  squashfs
[    5.636009]
[    5.643215] Kernel panic - not syncing: VFS: Unable to mount root
fs on unknown-block(253,0)
}

Do you see any other problem here with dm-verity cmdline or with squashfs ?

Is squashfs ever proved to be working with dm-verity on higher kernel version ?
Currently our kernel version is 4.14.

Or, another option is to use the new concept from 5.1 kernel that is:
dm-mod.create = ?
But, currently I don't know how to use it with squashfs either...
Any reference example will be helpful..

Thanks,
Pintu

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Thomas Petazzoni]

Hello,

On Mon, 30 Aug 2021 21:55:19 +0530
Pintu Agarwal <pintu.ping@gmail.com> wrote:

> Sorry for coming back to this again..
> Unfortunately, none of the options is working for us with squashfs
> (bootloader, initramfs).
> initramfs have different kinds of challenges because of the partition
> size issue.
> So, our preferred option is still the bootloader command line approach..
> 
> Is there a proven and working solution of dm-verity with squashfs ?
> If yes, please share some references.
> 
> The current problem with squashfs is that we could not append the
> verity-metadata to squashfs, so we store it on a separate volume and
> access it.

Here, it definitely worked to append the hash tree to the squashfs
image and store them in the same partition.

> By specifying it like : /dev/mtdblock53
> 
> Then we get the error like this:
> {
> [    4.950276] device-mapper: init: attempting early device configuration.
> [    4.957577] device-mapper: init: adding target '0 95384 verity 1
> /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
> 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
> aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
> restart_on_corruption ignore_zero_blocks use_fec_from_device
> /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026'
> [    4.975283] device-mapper: verity: sha256 using implementation
> "sha256-generic"
> [    4.998728] device-mapper: init: dm-0 is ready

Could you show the full kernel command line ?

> Do you see any other problem here with dm-verity cmdline or with squashfs ?
> 
> Is squashfs ever proved to be working with dm-verity on higher kernel version ?
> Currently our kernel version is 4.14.

I confirm we used squashfs on dm-verity successfully. For sure on 4.19,
perhaps on older kernels as well.

> Or, another option is to use the new concept from 5.1 kernel that is:
> dm-mod.create = ?

How are you doing it today without dm-mod.create ?

Again, please give your complete kernel command line.

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

On Mon, 30 Aug 2021 at 22:25, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Mon, 30 Aug 2021 21:55:19 +0530
> Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> > Sorry for coming back to this again..
> > Unfortunately, none of the options is working for us with squashfs
> > (bootloader, initramfs).
> > initramfs have different kinds of challenges because of the partition
> > size issue.
> > So, our preferred option is still the bootloader command line approach..
> >
> > Is there a proven and working solution of dm-verity with squashfs ?
> > If yes, please share some references.
> >
> > The current problem with squashfs is that we could not append the
> > verity-metadata to squashfs, so we store it on a separate volume and
> > access it.
>
> Here, it definitely worked to append the hash tree to the squashfs
> image and store them in the same partition.
>
> > By specifying it like : /dev/mtdblock53
> >
> > Then we get the error like this:
> > {
> > [    4.950276] device-mapper: init: attempting early device configuration.
> > [    4.957577] device-mapper: init: adding target '0 95384 verity 1
> > /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
> > 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
> > aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
> > restart_on_corruption ignore_zero_blocks use_fec_from_device
> > /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026'
> > [    4.975283] device-mapper: verity: sha256 using implementation
> > "sha256-generic"
> > [    4.998728] device-mapper: init: dm-0 is ready
>
> Could you show the full kernel command line ?
Shared below

> > Do you see any other problem here with dm-verity cmdline or with squashfs ?
> >
> > Is squashfs ever proved to be working with dm-verity on higher kernel version ?
> > Currently our kernel version is 4.14.
>
> I confirm we used squashfs on dm-verity successfully. For sure on 4.19,
> perhaps on older kernels as well.

ohh that means we already have a working reference.
If possible can you share the details, even 4.19 or higher will be
also a good reference.

> > Or, another option is to use the new concept from 5.1 kernel that is:
> > dm-mod.create = ?
> How are you doing it today without dm-mod.create ?
I think in 4.14 we don't have dm-mod.create right ?

> Again, please give your complete kernel command line.
>
Here is our kernel command line:

[    0.000000] Kernel command line: ro rootwait
console=ttyMSM0,115200,n8 ....  verity="95384 11923
16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026
" rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0
.... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity
1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
restart_on_corruption ignore_zero_blocks use_fec_from_device
/dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ...

Do you see any issue here ?
Can you share your command line for squashfs to compare ?

Thank you,
Pintu

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Thomas Petazzoni]

Hello,

On Mon, 30 Aug 2021 23:48:40 +0530
Pintu Agarwal <pintu.ping@gmail.com> wrote:

> ohh that means we already have a working reference.
> If possible can you share the details, even 4.19 or higher will be
> also a good reference.
> 
> > > Or, another option is to use the new concept from 5.1 kernel that is:
> > > dm-mod.create = ?  
> > How are you doing it today without dm-mod.create ?  
> I think in 4.14 we don't have dm-mod.create right ?

No, but you can backport it easily. Back at
http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html
I provided backports of this feature to OpenWrt, for the 4.14 and 4.19
kernels.

> Here is our kernel command line:
> 
> [    0.000000] Kernel command line: ro rootwait
> console=ttyMSM0,115200,n8 ....  verity="95384 11923
> 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026
> " rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0
> .... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity
> 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
> 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
> aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
> restart_on_corruption ignore_zero_blocks use_fec_from_device
> /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ...

I don't see how this can work without the dm-mod.create feature. Are
you sure the verity= and dm= kernel arguments exist?

Best regards,

Thomas
-- 
Thomas Petazzoni, co-owner and CEO, Bootlin
Embedded Linux and Kernel engineering
https://bootlin.com

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

Hi,

On Tue, 31 Aug 2021 at 00:42, Thomas Petazzoni
<thomas.petazzoni@bootlin.com> wrote:
>
> Hello,
>
> On Mon, 30 Aug 2021 23:48:40 +0530
> Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> > ohh that means we already have a working reference.
> > If possible can you share the details, even 4.19 or higher will be
> > also a good reference.
> >
> > > > Or, another option is to use the new concept from 5.1 kernel that is:
> > > > dm-mod.create = ?
> > > How are you doing it today without dm-mod.create ?
> > I think in 4.14 we don't have dm-mod.create right ?
>
> No, but you can backport it easily. Back at
> http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html
> I provided backports of this feature to OpenWrt, for the 4.14 and 4.19
> kernels.
>
Yes, I can backport it to our 4.14 Kernel.
Can you share the list of patches to be backported to make it work on 4.14 ?
If it's backported also I need to report to our internal kernel, but
it might be slightly easier.
Please share the details.

> > Here is our kernel command line:
> >
> > [    0.000000] Kernel command line: ro rootwait
> > console=ttyMSM0,115200,n8 ....  verity="95384 11923
> > 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026
> > " rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0
> > .... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity
> > 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
> > 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
> > aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
> > restart_on_corruption ignore_zero_blocks use_fec_from_device
> > /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ...
>
> I don't see how this can work without the dm-mod.create feature. Are
> you sure the verity= and dm= kernel arguments exist?

Sorry, I am not a security guy and this was done by someone from the
security team.
But, I know that this is already working with ext4.
The moment we change to squashfs, it does not work.

The only difference with squashfs are:
=> verity metadata are kept on separate volume
=> The rootfstype and related stuff are different
=> verity command line related stuff are almost the same.

Also, you mentioned:
>>> Here, it definitely worked to append the hash tree to the squashfs
>>> image and store them in the same partition.
Can you share some details about it ?
How it can be done since squashfs is readonly.
Do, we need to change some parameters during squashfs image generation ?
{
  $(STAGING_DIR_HOST)/bin/mksquashfs4 $(call mkfs_target_dir,$(1)) $@ \
- -nopad -noappend -root-owned \
+ -noappend -root-owned \
}

Also, for the above cmdline, is there any problem with the block size ?
As @Mikulas said before that the block size could be the issue

Also, for squashfs we are passing like this for root=. Is it fine ?
rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0

I see that dm-0 is already passed elsewhere so do we really need it ?
I suspect it is not required as a block device.


Thanks,
Pintu

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

Dear Thomas, Mikulas,
Need your help in root causing my dm-verity issue with squashfs.
Please see my comments inline.

On Tue, 31 Aug 2021 at 18:49, Pintu Agarwal <pintu.ping@gmail.com> wrote:

> > No, but you can backport it easily. Back at
> > http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html
> > I provided backports of this feature to OpenWrt, for the 4.14 and 4.19
> > kernels.
> >
> Yes, I can backport it to our 4.14 Kernel.
> Can you share the list of patches to be backported to make it work on 4.14 ?
> If it's backported also I need to report to our internal kernel, but
> it might be slightly easier.
> Please share the details.
>

I am interested to backport dm-mod.create related patches to our 4.14 kernel.
Please let me know where can I find all the patches ?
Is it already part of mainline 4.14 ?
Please share the list of commits (from mainline) that we need to pull
and backport.

> > > Here is our kernel command line:
> > >
> > > [    0.000000] Kernel command line: ro rootwait
> > > console=ttyMSM0,115200,n8 ....  verity="95384 11923
> > > 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3 12026
> > > " rootfstype=squashfs ubi.mtd=40,0,30 ubi.block=0,0 root=/dev/dm-0
> > > .... init=/sbin/init root=/dev/dm-0 dm="rootfs none ro,0 95384 verity
> > > 1 /dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
> > > 16da5e4bbc706e5d90511d2a3dae373b5d878f9aebd522cd614a4faaace6baa3
> > > aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
> > > restart_on_corruption ignore_zero_blocks use_fec_from_device
> > > /dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026" ...
> >
> > I don't see how this can work without the dm-mod.create feature. Are
> > you sure the verity= and dm= kernel arguments exist?
>
I checked a little further and yes there is "dm=" command line in
kernel available.
This is already working with ext4 glue, but was never tried with squashfs.
I think it is mainline derived from Android.
https://patchwork.kernel.org/project/dm-devel/patch/2c01b2a43a46fab760208d7af3a7af37eec8c41a.1537936397.git.helen.koike@collabora.com/
https://github.com/projectceladon/device-androidia-kernel/blob/master/init/do_mounts_dm.c

Mostly, this is the main repo where our source might be derived:
https://github.com/android-linux-stable/msm-4.14

Can we backport the patches here ?
If I get the list I can try it.

>
> Also, you mentioned:
> >>> Here, it definitely worked to append the hash tree to the squashfs
> >>> image and store them in the same partition.
> Can you share some details about it ?
> How it can be done since squashfs is readonly.
Can you share your reference, how are you appending the hash tree ?
Let me try the same.

But it seems like the underlying concept is the same for both
"dm-mod.create" and "dm=".
However, I am not sure if there are any changes required for squashfs
as block device..

Errors:
Currently, we are getting this in boot logs:

[    4.962188] device-mapper: init: attempting early device configuration.
[    4.969699] device-mapper: init: created device '253:0'
[    4.975503] device-mapper: init: adding target '0 95384 verity 1
/dev/ubiblock0_0 /dev/mtdblock53 4096 4096 11923 8 sha256
8fc2e4bb751f4b3145a486a0f4f1b58149ba3eedc2a67312f31fbee131380dab
aee087a5be3b982978c923f566a94613496b417f2af592639bc80d141e34dfe7 10
restart_on_corruption ignore_zero_blocks use_fec_from_device
/dev/mtdblock53 fec_roots 2 fec_blocks 12026 fec_start 12026'
[    4.992323] device-mapper: verity: sha256 using implementation
"sha256-generic"
[    5.015568] device-mapper: init: dm-0 is ready
[   10.080065] prepare_namespace: dm_run_setup - done
[   10.080093] prepare_namespace: saved_root_name: /dev/dm-0
[   10.083903] prepare_namespace: Inside: name_to_dev_t
[   10.089605] prepare_namespace: Calling - mount_root() ...
[   10.094519] [PINTU]: mount_block_root: called with input name:
/dev/root, fs_names: squashfs
[   10.263510] [PINTU]: do_mount_root: sys_mount failed: err: -22
[   10.263544] [PINTU]: mount_block_root: do_mount_root: err: -22, p:
squashfs, flags: 32769, root_mount_data: (null)
[..]
[   10.745672] No filesystem could mount root, tried:
[   10.745676]  squashfs
[   10.748015]
[   10.755232] Kernel panic - not syncing: VFS: Unable to mount root
fs on unknown-block(253,0)

It seems the rootfs could not mount due to invalid arguments.
Not sure which arguments are invalid here...


Thanks,
Pintu

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

Hi,

On Mon, 6 Sept 2021 at 21:58, Pintu Agarwal <pintu.ping@gmail.com> wrote:

> On Tue, 31 Aug 2021 at 18:49, Pintu Agarwal <pintu.ping@gmail.com> wrote:
>
> > > No, but you can backport it easily. Back at
> > > http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html
> > > I provided backports of this feature to OpenWrt, for the 4.14 and 4.19
> > > kernels.

Can you please let me know where to get the below patches for
backporting to our kernel:
 create mode 100644
target/linux/generic/backport-4.14/390-dm-add-support-to-directly-boot-to-a-mapped-device.patch
 create mode 100644
target/linux/generic/backport-4.14/391-dm-init-fix-max-devices-targets-checks.patch
 create mode 100644
target/linux/generic/backport-4.14/392-dm-ioctl-fix-hang-in-early-create-error-condition.patch
 create mode 100644
target/linux/generic/backport-4.14/393-Documentation-dm-init-fix-multi-device-example.patch

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Greg KH]

On Wed, Sep 08, 2021 at 04:57:45PM +0530, Pintu Agarwal wrote:
> Hi,
> 
> On Mon, 6 Sept 2021 at 21:58, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> 
> > On Tue, 31 Aug 2021 at 18:49, Pintu Agarwal <pintu.ping@gmail.com> wrote:
> >
> > > > No, but you can backport it easily. Back at
> > > > http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html
> > > > I provided backports of this feature to OpenWrt, for the 4.14 and 4.19
> > > > kernels.
> 
> Can you please let me know where to get the below patches for
> backporting to our kernel:
>  create mode 100644
> target/linux/generic/backport-4.14/390-dm-add-support-to-directly-boot-to-a-mapped-device.patch
>  create mode 100644
> target/linux/generic/backport-4.14/391-dm-init-fix-max-devices-targets-checks.patch
>  create mode 100644
> target/linux/generic/backport-4.14/392-dm-ioctl-fix-hang-in-early-create-error-condition.patch
>  create mode 100644
> target/linux/generic/backport-4.14/393-Documentation-dm-init-fix-multi-device-example.patch

If you are stuck on an older kernel version, then you need to get
support from the vendor that is forcing you to be on that kernel
version, as you are already paying them for support.  Please take
advantage of that, as no one knows what is really in "your kernel".

thanks,

greg k-h

Re: Kernel 4.14: Using dm-verity with squashfs rootfs - mounting issue

[Pintu Agarwal]

Hi All,

On Wed, 8 Sept 2021 at 17:38, Greg KH <gregkh@linuxfoundation.org> wrote:

> > > > > No, but you can backport it easily. Back at
> > > > > http://lists.infradead.org/pipermail/openwrt-devel/2019-November/025967.html
> > > > > I provided backports of this feature to OpenWrt, for the 4.14 and 4.19
> > > > > kernels.
> >
> > Can you please let me know where to get the below patches for
> > backporting to our kernel:
> >  create mode 100644
> > target/linux/generic/backport-4.14/390-dm-add-support-to-directly-boot-to-a-mapped-device.patch
> >  create mode 100644
> > target/linux/generic/backport-4.14/391-dm-init-fix-max-devices-targets-checks.patch
> >  create mode 100644
> > target/linux/generic/backport-4.14/392-dm-ioctl-fix-hang-in-early-create-error-condition.patch
> >  create mode 100644
> > target/linux/generic/backport-4.14/393-Documentation-dm-init-fix-multi-device-example.patch
>
> If you are stuck on an older kernel version, then you need to get
> support from the vendor that is forcing you to be on that kernel
> version, as you are already paying them for support.  Please take
> advantage of that, as no one knows what is really in "your kernel".
>

This is to update this thread that now I am able to successfully
bring-up dm-verity with NAND+ubiblock+squashfs on our 4.14 kernel
itself without backporting the patches.
Now, I am able to boot dm-verity using both initramfs and bootloader approach.
The initramfs booting issue was our internal issue which was related
to Kernel size configuration in UEFI.
The bootloader approach issue was related to system image size issue,
where we need to pass the exact image size to find the verity-metadata
offset at the end of system image.

However, I felt that dm-verity documentation still needs to be
enhanced further with a better example.
With the 5.4 Kernel, I may further explore the option of using
dm-mod.create option, then I might get more clarity on how best to use
it.

Thank you all for your help and support!

Regards,
Pintu