From nobody Wed Aug 2 16:22:58 2023 X-Original-To: soc-status@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RGHMz0GYbz4qj3V for ; Wed, 2 Aug 2023 16:23:14 +0000 (UTC) (envelope-from raghavself28@gmail.com) Received: from mail-io1-f47.google.com (mail-io1-f47.google.com [209.85.166.47]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RGHMh4tH2z4F73 for ; Wed, 2 Aug 2023 16:23:12 +0000 (UTC) (envelope-from raghavself28@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of raghavself28@gmail.com designates 209.85.166.47 as permitted sender) smtp.mailfrom=raghavself28@gmail.com; dmarc=none Received: by mail-io1-f47.google.com with SMTP id ca18e2360f4ac-77a62a84855so285709939f.1 for ; Wed, 02 Aug 2023 09:23:12 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1690993390; x=1691598190; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=EnnifQZj2e6uxw0TY/Ygvl3MO+6AWZFazn62l45zHNY=; b=egKduxEeG+0dV3gO5QxhJQt5DtZ2/3f9pO9xcgJ77MUmuUzxS0jovUBDtwU+j74Hff HDF6F3cPQOtWAiNFc5h2rs9l9FkJW2+gMip5tv4aFZCwwNkSFtLRExm0Zodo0g9GcPyR 3ePLmTEniZJP64JBPpljIoP6NVIe4BDFpL/IKEpeOXqTGyqm8hdjlP3PEwamnp4crZH0 KU5t3zrbg7nQNdiS2nBYc416gbPIf1fShnC8p7l1d8NS97neU0EB26zBbK/a9a94ciEk 3L9C29k6RoTwRE3tTTTL/fhc8FMnnH87llX3i8jcVMW8dXV8lBsR1DIAJXUx1wErzzQt ukMg== X-Gm-Message-State: ABy/qLYVQKDpIGDS+oJErJbmL167RT0PES5aT8Z3qbXB9NCnN7xiKN7i 2z+MRlTvZQ7mkxdUwMW6pBYTZ94mW4Q= X-Google-Smtp-Source: APBJJlHQzK2BiT4EUEunyMw4uL5QIsNX/LJgat/Ys5k5fjNr+VD4T7ZD1poUQrlPMSXO0G9Wnp7abA== X-Received: by 2002:a5d:9bd1:0:b0:786:4795:30c9 with SMTP id d17-20020a5d9bd1000000b00786479530c9mr13906640ion.20.1690993390093; Wed, 02 Aug 2023 09:23:10 -0700 (PDT) Received: from mail-il1-f172.google.com (mail-il1-f172.google.com. [209.85.166.172]) by smtp.gmail.com with ESMTPSA id x3-20020a02ac83000000b0043021113e09sm4331679jan.75.2023.08.02.09.23.09 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 02 Aug 2023 09:23:09 -0700 (PDT) Received: by mail-il1-f172.google.com with SMTP id e9e14a558f8ab-3491568afd4so13712925ab.0 for ; Wed, 02 Aug 2023 09:23:09 -0700 (PDT) X-Received: by 2002:a05:6e02:20c5:b0:346:49bd:7b6c with SMTP id 5-20020a056e0220c500b0034649bd7b6cmr16786186ilq.22.1690993389450; Wed, 02 Aug 2023 09:23:09 -0700 (PDT) List-Id: Summer of Code Status Reports and Discussion List-Archive: https://lists.freebsd.org/archives/soc-status List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-soc-status@freebsd.org X-BeenThere: soc-status@freebsd.org MIME-Version: 1.0 From: Raghav Sharma Date: Wed, 2 Aug 2023 21:52:58 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: SquashFS port status update To: soc-status@freebsd.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4RGHMh4tH2z4F73 X-Spamd-Bar: -- X-Spamd-Result: default: False [-2.67 / 15.00]; NEURAL_HAM_LONG(-1.00)[-1.000]; NEURAL_HAM_SHORT(-0.96)[-0.956]; NEURAL_HAM_MEDIUM(-0.72)[-0.719]; FORGED_SENDER(0.30)[raghav@freebsd.org,raghavself28@gmail.com]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17:c]; MIME_GOOD(-0.10)[text/plain]; ARC_NA(0.00)[]; MLMMJ_DEST(0.00)[soc-status@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; R_DKIM_NA(0.00)[]; FREEMAIL_ENVFROM(0.00)[gmail.com]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.166.47:from]; FROM_NEQ_ENVFROM(0.00)[raghav@freebsd.org,raghavself28@gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RCVD_IN_DNSWL_NONE(0.00)[209.85.166.47:from,209.85.166.172:received]; MIME_TRACE(0.00)[0:+]; RCVD_COUNT_THREE(0.00)[4]; FROM_HAS_DN(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; RCVD_TLS_LAST(0.00)[]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[soc-status@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DOM_EQ_FROM_DOM(0.00)[] Hello! Apologies for not writing much but here's the writeup on progress for our squashfs port. I started my work by debugging umount() hook issue, which was running for an infinite loop. After spending some time understanding umount() hook behavior realized it was waiting for a lock on a vnode and vnops_reclain() hook. I implemented both and we finally got working mount() and umount() hooks for squashfs. The PR I created for my mentor to review my code was unfortunately too large so my mentor suggested breaking it down into several small PRs and then pushing for review. I followed his advice and spend some time creating a new review branch and pushing small changes PRs. The first major change my mentor suggested was to not depend on external algorithms for compiling kernel drivers. Squashfs is a compressed file system which means it uses compression algorithms like zlib, zstd, lzma, etc... In the beginning, I used external files like zlib.h to get those compression techniques which is not good for compiling and linking the squashfs driver. My mentor advised me to look at the subr_compressor.c file and check if I can support some compression algorithms defined there for squashfs. I explored that and found we can support zlib and zstd compressions since they are implemented for our kernel. So for now squashfs supports the above two compressions and if time permits we can add support for more from geom. In the meantime, on addressing review comments I Implemented vnops_open() and close() hooks. They are not much complex so it was easy to implement them. To get directories read support, implementing the lookup() hook is mandatory. I managed to implement that and currently finishing up my work on readdir() hook. Once that is tested and is in working condition our directories read support will be completed. Thanks Raghav From nobody Fri Aug 25 16:37:02 2023 X-Original-To: soc-status@mlmmj.nyi.freebsd.org Received: from mx1.freebsd.org (mx1.freebsd.org [IPv6:2610:1c1:1:606c::19:1]) by mlmmj.nyi.freebsd.org (Postfix) with ESMTP id 4RXQbK3kHCz4qVyd for ; Fri, 25 Aug 2023 16:37:17 +0000 (UTC) (envelope-from raghavself28@gmail.com) Received: from mail-il1-f175.google.com (mail-il1-f175.google.com [209.85.166.175]) (using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 (128/128 bits) key-exchange X25519 server-signature RSA-PSS (4096 bits) server-digest SHA256 client-signature RSA-PSS (2048 bits) client-digest SHA256) (Client CN "smtp.gmail.com", Issuer "GTS CA 1D4" (verified OK)) by mx1.freebsd.org (Postfix) with ESMTPS id 4RXQbJ40r5z4sKr for ; Fri, 25 Aug 2023 16:37:16 +0000 (UTC) (envelope-from raghavself28@gmail.com) Authentication-Results: mx1.freebsd.org; dkim=none; spf=pass (mx1.freebsd.org: domain of raghavself28@gmail.com designates 209.85.166.175 as permitted sender) smtp.mailfrom=raghavself28@gmail.com; dmarc=none Received: by mail-il1-f175.google.com with SMTP id e9e14a558f8ab-34ca2fcdf9fso3747015ab.0 for ; Fri, 25 Aug 2023 09:37:16 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1692981435; x=1693586235; h=to:subject:message-id:date:from:mime-version:x-gm-message-state :from:to:cc:subject:date:message-id:reply-to; bh=O0QMdCdmjFUnSPyipteVJTBRUwp2Wy704dgm/1iqlkU=; b=Ll2s89iDyQ5XvPr5Brr3h1GuKAtlxkXtyw9xBHdBlBEGfob9Vj0idZss5+dQ06oVx2 sXn4XzAy2raT8Aj/AoOvY9V5YfEpTdGt+g/uqX29LWZdE807nbKmUEHQYcOkwmByJTiZ NuX8XLe+4EefxGY49HyXIUPj3FYVeEvhjhjo+XGr6hPoS6kmPrOMqxVh6mzY9IPAJh+d KL49hSZcw06eSOs2Te3xUkwNboz9mRHaLseEd1bwhLfFH3I2NRMIuCFvAe68xpROnk0a w4bzcT2XIYQSeUv1a4JBm7liVfuEje8Lnd+obIh2xUzkdrX4G+moAEtP3hP522crfWQ1 kTWQ== X-Gm-Message-State: AOJu0YyRSFze0aKfwawhFexvR5AqL1E8H5oo00rgtuVZ+N2d+QWk+bwB zkVLlYcMeuMLYHvPSNMQktQZYSYRDHs= X-Google-Smtp-Source: AGHT+IGLET0dB3sv2mTSAb2AhPGLselnes2Q8k3jShGHVtIFRkmlhW9c4sOn8piI5zmYsHLq6u9xiA== X-Received: by 2002:a05:6e02:13cc:b0:345:c77a:b5eb with SMTP id v12-20020a056e0213cc00b00345c77ab5ebmr8046214ilj.18.1692981434577; Fri, 25 Aug 2023 09:37:14 -0700 (PDT) Received: from mail-il1-f175.google.com (mail-il1-f175.google.com. [209.85.166.175]) by smtp.gmail.com with ESMTPSA id b13-20020a920b0d000000b00345cce526cdsm623446ilf.54.2023.08.25.09.37.14 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 25 Aug 2023 09:37:14 -0700 (PDT) Received: by mail-il1-f175.google.com with SMTP id e9e14a558f8ab-34ca22e3b3dso3672975ab.3 for ; Fri, 25 Aug 2023 09:37:14 -0700 (PDT) X-Received: by 2002:a05:6e02:160b:b0:345:d583:5c64 with SMTP id t11-20020a056e02160b00b00345d5835c64mr11680131ilu.24.1692981433824; Fri, 25 Aug 2023 09:37:13 -0700 (PDT) List-Id: Summer of Code Status Reports and Discussion List-Archive: https://lists.freebsd.org/archives/soc-status List-Help: List-Post: List-Subscribe: List-Unsubscribe: Sender: owner-soc-status@freebsd.org X-BeenThere: soc-status@freebsd.org MIME-Version: 1.0 From: Raghav Sharma Date: Fri, 25 Aug 2023 22:07:02 +0530 X-Gmail-Original-Message-ID: Message-ID: Subject: SquashFS port final report To: soc-status@freebsd.org Content-Type: multipart/alternative; boundary="000000000000f980b10603c1f447" X-Spamd-Result: default: False [-1.64 / 15.00]; NEURAL_HAM_MEDIUM(-0.97)[-0.969]; NEURAL_HAM_LONG(-0.95)[-0.946]; FORGED_SENDER(0.30)[raghav@freebsd.org,raghavself28@gmail.com]; NEURAL_SPAM_SHORT(0.27)[0.274]; R_SPF_ALLOW(-0.20)[+ip4:209.85.128.0/17]; MIME_GOOD(-0.10)[multipart/alternative,text/plain]; RCVD_IN_DNSWL_NONE(0.00)[209.85.166.175:received,209.85.166.175:from]; FREEMAIL_ENVFROM(0.00)[gmail.com]; ASN(0.00)[asn:15169, ipnet:209.85.128.0/17, country:US]; RWL_MAILSPIKE_POSSIBLE(0.00)[209.85.166.175:from]; MLMMJ_DEST(0.00)[soc-status@freebsd.org]; DMARC_NA(0.00)[freebsd.org]; R_DKIM_NA(0.00)[]; MIME_TRACE(0.00)[0:+,1:+,2:~]; RCVD_TLS_LAST(0.00)[]; RCVD_COUNT_THREE(0.00)[3]; FROM_HAS_DN(0.00)[]; ARC_NA(0.00)[]; RCVD_VIA_SMTP_AUTH(0.00)[]; FROM_NEQ_ENVFROM(0.00)[raghav@freebsd.org,raghavself28@gmail.com]; TO_MATCH_ENVRCPT_ALL(0.00)[]; TO_DN_NONE(0.00)[]; PREVIOUSLY_DELIVERED(0.00)[soc-status@freebsd.org]; RCPT_COUNT_ONE(0.00)[1]; TO_DOM_EQ_FROM_DOM(0.00)[] X-Spamd-Bar: - X-Rspamd-Queue-Id: 4RXQbJ40r5z4sKr --000000000000f980b10603c1f447 Content-Type: text/plain; charset="UTF-8" Hello! The last 3 weeks have been amazing for our SquashFS port. We completed directories, files, and symlinks support. I completed xattrs support too but implementing vnops() hooks for them is remaining as I was spending this last week on testing our driver. A more detailed report for each feature: * Directories - SquashFS directories are simple, we have a header for the current directory which keeps the count of the number of entries and offset of block in the Directory table. Since I was done implementing their traversing functions most of the work remaining was of implementing VOP_LOOKUP() and VOP_READDIR() hooks. I spent some time understanding those hooks and implemented them completing directories read support. * Files - Now this was a little special, SquashFS files are in fragments which in turn are in the Fragment table. Since there wasn't much documentation available I tried to make sense of SquashFuse source code to understand files. Once I got its design I began porting them. However, I got stuck while testing due to constant page faults that occurred while reading files. After a lot of debugging, I realized that we cannot use uio vectors directly for reading and should use uio helper functions like uiomove(), uiomove_frombuf(), etc... to handle those operations. Once I used them the issue was resolved. * Symlinks - This was simple, the symlink target was specified in inode metadata and all we needed to do was read it. I quickly implemented the VOP_READLINK() hook and symlinks support was done as well. * Extended Attributes - I spent some time understanding SquashFuse's implementation of them, and after that I ported all necessary functions for operations like getting extended attributes, getting their value, etc... The work for implementing VOP_() hooks for them remains since I spent this last week testing more of our driver. I am glad I did a lot of testing where I encountered two issues: 1. We can't list the first entry of the root inode. 2. The inode data leaks memory, I don't know how to free that data once we are not using that inode. I think the second issue could be resolved soon once I see which function gets called for freeing inode data. This is it for this final report. Thanks a lot for reading it this far. --Raghav --000000000000f980b10603c1f447 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Hello!

The last 3 weeks have been amazing for our S= quashFS port.
We completed directories, files, and symlinks support.
= I completed xattrs support too but implementing vnops() hooks for them is r= emaining as I was spending this last week on testing our driver.

A m= ore detailed report for each feature:

* Directories - SquashFS direc= tories are simple, we have a header for the current directory which keeps t= he count of the number of entries and offset of block=C2=A0in the Directory= table.
Since I was done implementing their traversing functions most of= the work remaining=C2=A0was of implementing VOP_LOOKUP() and VOP_READDIR()= hooks.
I spent some time understanding=C2=A0those hooks and implemented= them completing directories read support.

* Files - Now this was=C2= =A0a little special, SquashFS files are in fragments which in turn are in t= he Fragment table.
Since there wasn't much documentation available I= tried to make sense of SquashFuse source code to understand files.
Once= I got its design I began porting them.

However, I got stuck while t= esting due to constant page faults that occurred while reading files.
Af= ter a lot of debugging, I realized that we cannot use uio vectors directly = for reading and should use uio helper functions like uiomove(), uiomove_fro= mbuf(), etc... to handle those operations.
Once I used them the issue wa= s resolved.

* Symlinks - This was simple, the symlink target was spe= cified in inode metadata and all we needed to do was read it.
I quickly = implemented the VOP_READLINK() hook and symlinks support was done as well.<= br>
* Extended Attributes - I spent some time understanding SquashFuse&#= 39;s implementation of them, and after that I ported all necessary function= s for operations like getting extended attributes, getting their value, etc= ...
The work for implementing VOP_() hooks for them remains since I spen= t this last week testing more of our driver.

I am glad I did a lot o= f testing where I encountered two issues:

1. We can't list the f= irst entry of the root inode.
2. The inode data leaks memory, I don'= t know how to free that data once we are not using that inode.
I think t= he second issue could be resolved soon once I see which function gets calle= d for freeing inode data.

This is it for this final report.

T= hanks a lot for reading it this far.

--Raghav
--000000000000f980b10603c1f447--