Docs: add ci/kas, quick start, and release information

Signed-off-by: Jon Mason <jon.mason@arm.com>

README.md

@@ -19,19 +19,23 @@ Other Directories
 
 * ci
 
-  This directory contains gitlab continuous integration configuration files (KAS yaml files) as well as scripts needed for this
+  This directory contains gitlab continuous integration configuration files (KAS yaml files) as well as scripts needed for this.
+
+* documentation
+
+  This directory contains information on the files in this repository, building, and other relevant documents.
 
 * kas
 
-  This directory contains KAS yaml files to describe builds for systems not used in CI
+  This directory contains KAS yaml files to describe builds for systems not used in CI.
 
 * scripts
 
-  This directory contains scripts used in running the CI tests
+  This directory contains scripts used in running the CI tests.
 
 Mailing List
 ------------
-To interact with the meta-arm developer community, please email the meta-arm mailing list at meta-arm@lists.yoctoproject.org
+To interact with the meta-arm developer community, please email the meta-arm mailing list at <meta-arm@lists.yoctoproject.org>.
 Currently, it is configured to only allow emails to members from those subscribed.
 To subscribe to the meta-arm mailing list, please go to
 https://lists.yoctoproject.org/g/meta-arm
@@ -42,7 +46,7 @@ Currently, we only accept patches from the meta-arm mailing list.  For general
 information on how to submit a patch, please read
 https://www.openembedded.org/wiki/How_to_submit_a_patch_to_OpenEmbedded
 
-E-mail meta-arm@lists.yoctoproject.org with patches created using this process. You can configure git-send-email to automatically use this address for the meta-arm repository with the following git command:
+E-mail <meta-arm@lists.yoctoproject.org> with patches created using this process. You can configure git-send-email to automatically use this address for the meta-arm repository with the following git command:
 
 `$ git config --local --add sendemail.to meta-arm@lists.yoctoproject.org`
 
@@ -55,6 +59,12 @@ The component being changed in the shortlog should be prefixed with the layer na
 
 >  arm-toolchain/gcc: enable foobar v2
 
+All contributions are under the [MIT License](/COPYING.MIT).
+
+For a quick start guide on how to build and use meta-arm, go to [quick-start.md](/documentation/quick-start.md).
+
+For information on the continuous integration done on meta-arm and how to use it, go to [continuous-integration-and-kas.md](/documentation/continuous-integration-and-kas.md).
+
 Backporting
 --------------
 Backporting patches to older releases may be done upon request, but only after a version of the patch has been accepted into the master branch.  This is done by adding the branch name to email subject line.  This should be between the square brackets (e.g., "[" and "]"), and before or after the "PATCH".  For example,
@@ -66,15 +76,21 @@ https://www.kernel.org/doc/html/latest/process/submitting-patches.html#submittin
 Releases and Release Schedule
 --------------
 We follow the Yocto Project release methodology, schedule, and stable/LTS support timelines.  For more information on these, please reference:
-https://docs.yoctoproject.org/ref-manual/release-process.html
-https://wiki.yoctoproject.org/wiki/Releases
-https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS
+* https://docs.yoctoproject.org/ref-manual/release-process.html
+* https://wiki.yoctoproject.org/wiki/Releases
+* https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS
+
+For more in-depth information on the meta-arm release and branch methodology, go to </documentation/releases.md>.
 
 Reporting bugs
 --------------
-E-mail meta-arm@lists.yoctoproject.org with the error encountered and the steps
+E-mail <meta-arm@lists.yoctoproject.org> with the error encountered and the steps
 to reproduce the issue.
 
+Security and Reporting Security Issues
+--------------
+For information on the security of meta-arm and how to report issues, please consult [SECURITY.md](/SECURITY.md).
+
 Maintainer(s)
 -------------
 * Jon Mason <jon.mason@arm.com>

SECURITY.md

@@ -18,7 +18,7 @@ number), please contact the meta-arm mailing list at
 meta-arm@lists.yoctoproject.org and arm-security@arm.com.
 
 If you are dealing with a not-yet released or urgent issue, please send a mail
-to the maintainers (see README.md) and arm-security@arm.com, including as much
+to the maintainers \(see [README.md](/README.md)\) and arm-security@arm.com, including as much
 detail as possible.  Encrypted emails using PGP are welcome.
 
 For more information, please visit https://developer.arm.com/support/arm-security-updates/report-security-vulnerabilities.
@@ -27,11 +27,20 @@ For more information, please visit https://developer.arm.com/support/arm-securit
 ## Branches maintained with security fixes
 
 meta-arm follows the Yocto release model, so see
-[https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS Stable release and
-LTS] for detailed info regarding the policies and maintenance of stable
+[Stable release and LTS](https://wiki.yoctoproject.org/wiki/Stable_Release_and_LTS)
+for detailed info regarding the policies and maintenance of stable
 branches.
 
-The [https://wiki.yoctoproject.org/wiki/Releases Release page] contains a list of all
+The [Release page](https://wiki.yoctoproject.org/wiki/Releases) contains a list of all
 releases of the Yocto Project. Versions in grey are no longer actively maintained with
 security patches, but well-tested patches may still be accepted for them for
 significant issues.
+
+
+# Disclaimer
+
+Arm reference solutions are Arm public example software projects that track and
+pull upstream components, incorporating their respective security fixes
+published over time. Arm partners are responsible for ensuring that the
+components they use contain all the required security fixes, if and when they
+deploy a product derived from Arm reference solutions.

documentation/continuous-integration-and-kas.md

@@ -0,0 +1,67 @@
+# **CI for Yocto Project and meta-arm**
+# **CI for Yocto Project**
+The Yocto Project has an autobuilder that performs nightly builds and image tests on all of the defined QEMU machines, including qemuarm and qemuarm64  Also, it currently runs builds on the hardware reference platforms including genericarm64 and meta-arm mahines fvp-base and sbsa-ref.  More information on the autobuilder can be found at <https://autobuilder.yoctoproject.org/>.
+
+More information on the image tests can be found at <https://wiki.yoctoproject.org/wiki/Image_tests>.
+
+The Yocto Project also has the ability to have individual package tests, ptests.  For more information on those, go to <https://wiki.yoctoproject.org/wiki/Ptest>.
+# **CI for meta-arm**
+meta-arm is using the Gitlab CI infrastructure.  This is currently being done internal to Arm, but an external version can be seen at <https://gitlab.com/jonmason00/meta-arm/-/pipelines>.
+
+This CI is constantly being expanded to provide increased coverage of the software and hardware supported in meta-arm.  All platforms are required to add a kas file and `.gitlab-ci.yml` entry as part of the initial patch series.  More information on kas can be found at <https://github.com/siemens/kas>.
+
+To this end, it would be wise to run kas locally to verify everything works prior to pushing to the CI build system.
+## **Running kas locally**
+### **Install kas**
+kas can be installed with pip, for example:
+```
+$ pip3 install --user kas
+```
+
+See <https://kas.readthedocs.io/en/latest/userguide/getting-started.html> for information on the dependencies and more.
+
+This assumes that the kas path ($HOME/.local/bin) is in $PATH.  If not, the user will need to manually add this or the kas command will not be found.
+
+### **Run kas locally**
+```
+$ cd ~/meta-arm/
+$ kas build kas/juno.yml
+```
+
+By default kas will create a build directory under meta-arm to contain the checked out layers, build directory, and downloads.  You can change this by setting environment variables.  DL\_DIR and SSTATE\_DIR are respected so these can point at existing directories, and setting KAS\_WORK\_DIR to the directory where repositories are already cloned will save having to re-fetch.  This can look something like:
+```
+$ SSTATE_DIR=/builds/persist/sstate DL_DIR=/builds/persist/downloads kas build ci/qemuarm64.yml:ci/testimage.yml
+```
+
+See the [quick start guide](/documentation/quick-start.md) for more information on how to set this up.
+
+## **Locked Revisions in CI with lockfiles**
+The CI in meta-arm will generate a kas "lock file" when it starts to ensure that all of the builds checkout the same revision of the various different layers that are used. If this isn't done then there's a chance that a layer will be modified upstream during the CI, which results in some builds failing and some builds passing.
+
+This lock file is saved as an artefact of the update-repos job by the CI, and only generated if it doesn't already exist in the repository.  This can be used to force specific revisions of layers to be used instead of HEAD, which can be useful if upstream changes are causing problems in development.
+
+The lockfile.yml can be downloaded manually, but there's a script in meta-arm to fetch the lock file for the latest successful build of the specified branch:
+
+```
+$ ./ci/download-lockfile.py --help
+usage: download-lockfile.py [-h] server project refspec
+
+positional arguments:
+  server      GitLab server name
+  project     meta-arm project name
+  refspec     Branch/commit
+
+$ ./ci/download-lockfile.py https://gitlab.com/jonmason00/meta-arm master
+Fetched lockfile.yml
+Commit this lockfile.yml to the top-level of the meta-arm repository and the CI will use it automatically.
+```
+# **Relevant Links for kas, CI, and testing**
+<https://github.com/siemens/kas.git>
+
+<https://wiki.yoctoproject.org/wiki/Oe-selftest>
+
+<https://wiki.yoctoproject.org/wiki/Image_tests>
+
+<https://wiki.yoctoproject.org/wiki/Ptest>
+
+<https://wiki.yoctoproject.org/wiki/BSP_Test_Plan>

documentation/quick-start.md

@@ -0,0 +1,105 @@
+# **Yocto Project quick start for Arm system software developers**
+If you want to read the The Yocto Project official quick start documentation, go to <https://docs.yoctoproject.org/brief-yoctoprojectqs/index.html>
+
+If that looks like too much reading, then here is how to do it even faster!
+# **Step 0: Install build deps and kas**
+```
+$ sudo apt install gawk wget git diffstat unzip texinfo gcc build-essential chrpath socat cpio python3 python3-pip python3-pexpect xz-utils debianutils iputils-ping python3-git python3-jinja2 libegl1-mesa libsdl1.2-dev python3-subunit mesa-common-dev zstd liblz4-tool file locales libacl1
+
+$ pip install kas
+```
+OR, if you prefer to use a docker will all that stuff already installed:
+
+```
+$ sudo docker run -it --name kas-test --volume /mnt/yocto/:/builds/persist ghcr.io/siemens/kas/kas /bin/bash
+```
+
+> **_NOTE:_**
+> the “--volume” is the directory where your persistent stuff (like downloads and build artifacts) will go to help speed up your builds and can be sharable amongst your builds/containers.  If you want to go completely clean-room, feel free to remove it
+# **Step 1: clone meta-arm and build meta-arm**
+```
+$ git clone https://git.yoctoproject.org/meta-arm
+$ cd meta-arm/
+$ SSTATE_DIR=/builds/persist/sstate DL_DIR=/builds/persist/downloads kas build ci/fvp-base.yml:ci/testimage.yml
+```
+> **_NOTE:_**
+> “ci/testimage.yml” will cause the build to run some basic system tests.  If you don’t care about verifying basic functionality, then remove it and it should be faster (a few less programs will be added to the system image and the 2-3mins that it takes to run the test will not happen).
+
+> **_NOTE:_**
+> You may wish to add the Yocto Project SSTATE Mirror (especially the first time) to speed up the build by downloading the build fragments (built by the Yocto Project autobuilder) from the internet.  This can be done by adding "ci/sstate-mirror.yml" in kas or adding the relevant lines to your local.conf.  Using the above example:
+
+```
+$ SSTATE_DIR=/builds/persist/sstate DL_DIR=/builds/persist/downloads kas build ci/fvp-base.yml:ci/sstate-mirror.yml
+```
+
+> **_NOTE:_**
+> This only fetches the parts necessary for your build and may take several minutes depending on your internet connection speed.  Also, it only fetches what is available.  There may still be a need to build things depending on your configuration.
+
+For more information on kas and various commands, please reference <https://kas.readthedocs.io/en/latest/>.
+
+Depending on what software you are building, fvp-base might not be the machine you want to build for.
+The following website provides an EXTREMELY rough way to tell what software is in what machines, and what versions are being run:
+<https://gitlab.com/jonmason00/meta-arm/-/jobs/artifacts/master/file/update-report/index.html?job=pending-updates>
+
+If, as an example, we’re wanting to develop trusted-firmware-a; then fvp-base will work for us. 
+
+### **Okay, you are done!  VICTORY!**
+### **Oh, you actually wanted to mess around with the system software source code?**
+# **Step 2: use devtool to get your source**
+Setup your environment via the (non-kas) Yocto Project tools
+
+```
+$ source poky/oe-init-build-env
+```
+
+Use devtool to checkout the version of software being used on the machine above (in the above example, this will be trusted-firmware-a for fvp-base).
+
+```
+$ devtool modify trusted-firmware-a
+```
+
+This will download the source, hopefully in git (depending on how the Yocto Project recipe was written), and should print a path at the end where the source code was checked out.  In the trusted-firmware-a example, I got:
+
+> /builder/meta-arm/build/workspace/sources/trusted-firmware-a
+
+Inside of that directory, you should see the relevant source code.  In this example, it is a standard git tree.  So, you can add remotes, checkout different SHAs, etc
+
+Ok, so you are set with your changes and want to build them.
+
+```
+$ devtool build trusted-firmware-a
+```
+
+This should build the software in question, but it is not yet integrated into a system image.  To do that, run:
+
+```
+$ devtool build-image core-image-sato
+```
+
+The image should match the image being used on your machine above.  Most of them in meta-arm are set to core-image-sato.  
+
+Also, if you used testimage above, it will run testimage now
+### **Okay, you are done!  VICTORY!**
+# **Step 3.  Testing your patches outside of devtool**
+At this point I will assume you have a patch and want to add it to the base recipe.  Using the above example, in the devtool directory:
+```
+$ git format-patch -1
+0001-example.patch
+$ mv 0001-example.patch ~/meta-arm/meta-arm/recipes-bsp/trusted-firmware-a/files/
+$ cd ~/meta-arm
+$ devtool reset trusted-firmware-a
+$ echo ‘SRC_URI:append = " file://0001-example.patch" >> meta-arm/recipes-bsp/trusted-firmware-a/trusted-firmware-a_2.10.3.bb
+```
+
+> **_NOTE:_**
+> there is a space before the “file” and yes it matters very much
+
+At this point, you can go back using kas and verify that the patch works in a clean-ish tree.
+
+```
+$ SSTATE_DIR=/builds/persist/sstate DL_DIR=/builds/persist/downloads kas build ci/fvp-base.yml:ci/testimage.yml
+```
+
+There is obviously much more that can be done and other ways to do similar things.
+
+## **If there are issues or questions then please ask them on the #meta-arm irc channel on libera.chat**

documentation/releases.md

@@ -0,0 +1,43 @@
+# **meta-arm Releases and Branching**
+## **Release and Branching background**
+The Yocto Project releases twice a year (April and October): "stable" releases are made every six months and have a lifetime of seven months to allow for migration, while "long term support" (LTS) releases are picked every two years starting from Dunfell in April 2020. The standard practice for all Yocto Compatible layers is to create a "named" branch consistent with the code name of that release. For example, the “dunfell” release of the Yocto Project will have a branch named “dunfell” in the official git repository, and layers compatible with dunfell will have a branch named “dunfell”. Thus, a customer can easily organize a collection of appropriate layers to make a product.
+
+In the Yocto Project, these named branches are “stable”, and only take bug fixes or security-critical upgrades. Active development occurs on the master branch. However, this methodology can be problematic if mimicked with the compatible layers. Companies, like Arm, may not wish to release a snapshot of the relevant “master” branches under active development, due to the amount of testing, fixing, and hardening necessary to make a product from a non-stable release. Also, changes to keep the master branch of a layer working with the upstream master branch of the Yocto Project may result in that branch no longer being compatible with named branches (e.g., it might not be possible to mix and match master and dunfell). So, a decision must be made on the branching policy of meta-arm.
+
+## **Adding new Hardware or Software features**
+There are many different ways to resolve this issue. After some discussion, the best solution for us is to allow new hardware enablement (and relevant software features) to be included in LTS named branches (not just bug fixes). This will allow for a more stable software platform for software to be developed, tested, and released. Also, the single branch allows for focused testing (limiting the amount of resources needed for CI/CD), lessens/eliminates code diverging on various branches, and lessens confusion on which branch to use. The risk of making this choice is a potentially non-stable branch which will require more frequent testing to lessen the risk, and not following the “stable” methodology of the core Yocto Project layers (though it is not uncommon for BSP layers to behave this way).
+
+## **Process**
+The process for patches intended on being integrated into only the master branch is the normal internal process of pushing for code review and CI, approval and integration into upstream meta-arm master branch. 
+For patches intended on being included in an LTS named branch, the preferred process is to upstream via the master branch, rebase the patch (or series against the intended LTS branch) and send email with the release name in the subject line after the "PATCH" (e.g., "[PATCH dunfell] Add foo to bar").
+
+If there is a time crunch and the preferred way above cannot be completed in time, upstreaming via the LTS branch can occur. This follows the normal process above but without the master integration step.  However, any patches upstreamed in this manner must be pushed to master in a timely fashion (after the time crunch). Nagging emails will be sent and managers will be involved as the time grows.
+
+## **Testing**
+See [continuous-integration-and-kas.md](/documentation/continuous-integration-and-kas.md) for information how the layer is tested and what tests are run. It is presumed that all code will be compiled as part of the CI process of the gerrit code review. Also, testing on virtual platforms and code conformity checks will be run when enabled in the process.
+
+## **Branching strategy and releases**
+Named branches for meta-arm will be released as close as possible to the release of the YP LTS release. Meta-arm named branches will be created from the meta-arm master branch.
+
+To minimize the additional work of maintaining multiple branches it is assumed that there will only be two active development branches at any given time: master and the most recent Long Term Stable (LTS) as the named branch. All previous named LTS branches will be EOLed when a new LTS has been released. Any branches that are EOLed will still exist in the meta-arm, but bug fix patches will be accepted. Limited to no testing will occur on EOL’ed branches. Exceptions to this can be made, but must be sized appropriately and agreed to by the relevant parties.
+
+Named branch release will coincide with Yocto Project releases. These non-LTS branches will be bug fix only and will be EOLed on the next release (similar to the YP branching behavior).
+
+### **Branch transitions**
+When YP is approaching release, meta-arm will attempt to stabilize master so that the releases can coincide. 
+* T-6 weeks - Email is sent to meta-arm mailing list notifying of upcoming code freeze of features to meta-arm
+* T-4 weeks - Code freeze to meta-arm. Only bug fixes are taken at this point.
+* T-0 - Official upstream release occurs. With no outstanding critical bugs, a new named branch is created based on the current meta-arm master branch. Previous named branches are now frozen and will not accept new patches (but will continue to be present for reference and legacy usage).
+
+## **Tagging**
+### **Branch Tagging**
+When each branch is released, a git tag with the Yocto Project version number will be added.  For example, `4.3`.  Also, this tag version number will be prepended with "yocto" in a duplicate tag (e.g., "yocto-4.3").
+
+Conciding with the Yocto Project release schedule, every branch which has one or more changes added to it in the previous 6 months will get a minor versioned tag (e.g., "4.3.1" and "yocto-4.3.1").
+
+### **BSP Release Tagging**
+BSP releases for those boards supported in meta-arm-bsp maybe have an additional tag to denote their software releases.  The tag will consist of the board name (in all capital letters), year, and month.  For example, "CORSTONE1000-2023.11".
+The release schedule for this is outside the standard Yocto Project release candence, but is generally encouraged to be as close to these releases as possible.  Similarily, it is recommended the BSP releases be based on the latest LTS branch.
+
+# **Relevant Links**
+<https://wiki.yoctoproject.org/wiki/Releases>